2
$\begingroup$

I am new to DSP, but I've been attempting apply a butterworth high-pass filter on a 1D array (Values peeking at 300~) to eliminate any noise, yet when applying a high pass filter, I'm ending up with an attenuated result?

So far I have attempted to approach the problem with different specs, but it seems that my lack of experience in Signal Processing is holding me back. 


def butter_highpass(highcut, fs, order):
    nyq = 0.5 * fs
    high = highcut / nyq
    b, a = butter(order, high, btype="highpass")
    return b, a

fs      = 30.0            #Sampling rate Hz
b, a = butter_highpass(5, fs, 5)
filtered      = signal.lfilter(b, a, data)

Result: enter image description here

My Goal : Reduce all noise to 0 while retaining the peak values.

Any suggestions or recommendations are more than welcome

Improve this question
$\endgroup$
10
  • $\begingroup$ Well a high pass filter will pass all frequencies above a certain frequency and start attenuating frequencies below that with a particular roll-off that depends on the filter. From your goal this is not what you want. To achieve your goal you would need bandpass filters around each of the spectral peaks. The FFT is essentially a bank of filters so you could approach this by setting a threshold on your FFT output, selecting the bins above the threshold and then doing an inverse FFT (frequency sampling approach to filter design) However what is it you are really trying to do and why? $\endgroup$
    – Dan Boschen
    Commented Nov 28, 2019 at 17:26
  • $\begingroup$ I'm attempting to preprocess a data column from a .csv output which represents Eye Movement Landmark on the Y axis. The peaks represent a certain movement which I'd like to isolate while reducing the surrounding noise, preferably down to 0. The input data is acquired through processing video footage (hence the FPS axis). Could you advise on how to choose the bin threshold with this kind of input? or perhaps other preprocessing steps I might've missed? Thank you, $\endgroup$
    – CalEl
    Commented Nov 28, 2019 at 17:46
  • $\begingroup$ Have you tried zooming in on the result? Your samples are offset by about 250 while your peaks have an amplitude of about 10. This means that when you start running you will see a massive spike, but after that it should work more/less as expected. You might get clever and remove the initial offset too. $\endgroup$
    – Dan Szabo
    Commented Nov 28, 2019 at 17:52
  • $\begingroup$ You could consider an “N-sigma” approach where you take the standard deviation of your complex FFT output and set a threshold at a magnitude level that is a multiple of this standard deviation away. Remove the bins above the threshold (but maintain the index as your “selector”) and then repeat the standard deviation (sigma) computation and again set the threshold at the same multiple of sigma. Keep doing this until sigma converges to a allowable small error from one iteration to the next and you have made an intelligent selector between your noise and signal components. $\endgroup$
    – Dan Boschen
    Commented Nov 28, 2019 at 17:55
  • $\begingroup$ If you set N too low you will get “false alarms” calling noise signal and if you set N too high you will get “false detections” calling signal noise. $\endgroup$
    – Dan Boschen
    Commented Nov 28, 2019 at 17:56

2 Answers 2

Reset to default
1
$\begingroup$

The correct answer to this question has been provided by: Stanley Pawlukiewicz but has since been removed.

Your filter attenuates the high frequencies of the total timeseries, signal and noise.

Your results are expected.

You really can’t perfectly remove just the noise.

When the noise and signal overlap in time and frequency, you have to make a tradeoff between attenuating signal and attenuating noise, by adjusting the cutoff frequency.

Improve this answer
$\endgroup$
1
$\begingroup$

A high pass filter (ideally) only lets through the higher frequencies. The low frequencies are what determine the local average of the signal. A high pass filter will remove those and set the local average to $0$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.