# How to filter out noise in high frequency signal?

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy import signal

sig = df['1']

b, a = signal.butter(4, 0.1, analog=False)
sig_ff = signal.filtfilt(b, a, sig)
plt.plot(sig, color='silver', lw=0.3, label='Original')
plt.plot(sig_ff, color='red', lw=0.3, label='filtfilt')
plt.xlim(0, sig.index[-1])
plt.legend()
plt.show()


As you can see the noise reduction works but it also cuts off the impulse signal. I think the impulse frequenzy might be as high as the noise frequency. How can I filter out the noise of the signal without deleting the impulse?

Edit:

To give a further explanation what I am trying to do: I am trying to determine the Impulse properties like raise time and half-life time. Although it is possible to do with the noise in the system, I thought it might influence the Impulse. The Impulse I have shown is one of the larger ones that I have. It gets harder and harder to analyze smaller Impulses like this one:

As you see, this Impulse is partly concealed by the noise signal. I was wondering if it would be possible to remove the noise sufficiently to also be able to analyze these small Impulses.

The Spectrum (np.fft.rfft) of the larger Impulse (Gray/Red) is this:

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

fft = pd.DataFrame(np.abs(np.fft.rfft(df['1'])))   #y
n = df['0'].size
unit_freq = 1000000000 #Giga
sample_rate = 10000000000   #10 GS/s
freq_sample_fact = sample_rate/unit_freq
freq = np.fft.rfftfreq(n, 1/freq_sample_fact)       #x
fft.index = freq
fft.values[0] = 0
fft.plot(grid = 1,
color = (255/255,0,0),
linewidth = 0.3,
figsize = (10,5),
legend = False,
xlim = [fft.index[0], fft.index[-1]*0.8],
ylim = 0,
xticks = np.arange(0, freq_sample_fact/2 + 0.1, round(freq_sample_fact/2/10, 1)))
##plt.semilogy(fft)
plt.xlabel('Frequenz / GHz')
plt.ylabel('Signalstärke')
plt.title('FFT')
plt.show()


The FFT of the smaller Impulse (blue) is:

Noise

This is the spectrum of the noise:

I believe the y axis (Signalstärke) represents the "amount" of signal found on one frequency. The noise signal has a pretty high amount compared to the other ones because I took a wider time frame of the noise.

• filtfilt is almost never the function you want to use, and a Butterworth filter usually isn't the type of filter you'd want to use to use in a computer, either. What were your filter design criteria (i.e. why did your forward/backward filter with a Butterworth filter, and why did you parameterize that as you did?) – Marcus Müller Oct 7 '19 at 17:05
• I am not that experienced in signal processing. I asked this question in the signal processing stack and someone answered: "low-pass filter, and compensating for the filter group delay (which is a constant if you use a linear-phase FIR filter)". So I looked up lowe-pass filters ans FIR and ended up with the filtfilt function. – Artur Müller Romanov Oct 7 '19 at 17:19
• ah, OK, so maybe the question here is "how do I low-pass filter this signal to achieve…" (and I'd really recommend you expand on what you want to achieve, in the long run, for what reason you need to detect what)? Just to address your question: an impulse doesn't have a single frequency, it's inherently very wide-band, but when I commented that you should low-pass filter, I was assuming that you've already determined that the noise had only high-frequency components, so that a low-pass filter would over-proportionally dampen the noise. – Marcus Müller Oct 7 '19 at 17:21
• Could you try to plot the spectrum of your overall signal? And maybe give a very strong zoom in on the noise, so that we can get a feeling for it? – Marcus Müller Oct 7 '19 at 17:22
• I will do it tomorrow. Thank you for your help – Artur Müller Romanov Oct 7 '19 at 17:30