I have a signal from EEG sensors and I try to denoise it from AC frequencies. For that reason, I estimated PSD of my signal and found that 50 Hz and 100 Hz are likely to represent noise. I constructed Butterworth filter of order four and got much clearer signal, but at the start ([0:150] segment) there is even more distortion. Why is it so? If it helps, I use lfilter from scipy.signal.

enter image description here

Besides, in the future I want to break a signal into smaller pieces (say, length of 100). I have tried denoising them already and it seems like such filters do not work on short segments. What can I do with this?

enter image description here


All real-time filtering (as opposed to post processing) wirh FIR and IIR filters will have start up transitions based on the state of the filter at start up. For optimum rejection of AC noise , Instead of a Butterworth Filter consider using an 2nd order IIR notch filter with the notch set at your AC frequency (such as 50 Hz). A design for this is further explained on this site and is simple and effective for this purpose, and happens to be demonstrated with a notch at 50 Hz. (Transfer function of second order notch filter).

To have shorter segments, partition into segments at the output of the filter so that the filter’s memory can be maximized to meet the filtering requirement as designed (a filter’s transition bandwidth between passing a frequency of interest and rejection is dependent on the span of its memory—- for tight rejection long memory is required.)

If the application is for very short durations at completely different instances in time, that can make no use of the prior immediate time domain signal, then such traditional FIR and IIR filtering techniques will not be suitable beyond what they are capable of for the given time span. That said, the IIR nulling approach with its rejection optimized at the interference frequency may be sufficient and simple.

  • $\begingroup$ Thanks! Could you please clarify one more point for me: For example, I had a real-time signal coming which I'd like to denoise. Do I understand it correctly that for a clear denoising process with minimum distortion I need to 'collect' the signals in chunks, e.g. size 1000, and apply filtering over this chunk? The longer a chunk, the less distortion would be. Is that a correct interpretation? $\endgroup$ Jul 31 '20 at 9:50
  • $\begingroup$ @Ilya No I didn't imply that. Do you need to collect in chunks for your own reasons or can you stream all the data with no gaps (and then divide into chunks after filtering)? $\endgroup$ Jul 31 '20 at 9:51
  • $\begingroup$ No, I do not need to collect data in chunks, in fact, it would kill the whole idea. The second option is what I seek for, but how do I implement it? $\endgroup$ Jul 31 '20 at 10:14
  • $\begingroup$ @IlyaZisman You can use the 2nd order IIR nulling filter implementation that I linked. If you don't know how to implement IIR filters, start with that. You want to stream the data and filter it, then after filtering select your blocks. $\endgroup$ Jul 31 '20 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.