# Filtering of data in signal processing

I am processing the data from serial. I have to filter the data to remove ripples. I have tried with the following code. However, I can't get the expected results. Suggest me which type of filter I have to use in order to get expected results?

def graph_plot():
plt.xlabel("samples")
plt.ylabel("data")
plt.xlim([0, 2048])
plt.ylim([0, 255])
return plt

plt.ion()
fig = plt.figure()
ax.minorticks_on()
ax.grid(1, which = 'both', axis = 'y', markevery = 5)

actual_rx_data = rx_data[4:2052] #rx_data is input from serial
N = len(actual_rx_data)
rx_data = [actual_rx_data[i] for i in range (0, N)]
rx_data = np.reshape(rx_data, (2048, 1))

smoother = ConvolutionSmoother(window_len = 20, window_type = 'ones')
smoother.smooth(rx_data)

plt = graph_plot()

ax.plot(rx_data, color = 'red')  #input
ax.plot(smoother.smooth_data[0], linewidth = 2, color = 'blue')  #output
ax.clear()


Result obtained by the above code https://i.stack.imgur.com/xS6ct.jpg

Expected results in blue color https://i.stack.imgur.com/RoxlY.jpg

Fourier spectrum https://i.stack.imgur.com/6L3Vc.jpg

• Hi Maheswari! What is your expected output? Jul 17 '21 at 11:56
• I would hope that those two plots are what you expect, and what you're actually getting -- could you edit your question to label which is which? Or in the text, say "the first plot is ..." and "the second plot is ...". Jul 17 '21 at 19:41
• It is very hard, if not impossible to perfectly filter out the noise from any signal. The best approach in these situations, is to plot the frequency spectrum of the signal you want to filter, and then decide on the most appropriate filter to use. Edit your answer and add said spectrum so that we can provide additional help. Jul 18 '21 at 15:47
• @TimWescott, Now I edited the question...I didn't notice it...Sry Jul 19 '21 at 4:44
• @Marcus Müller, second one is the expected result... I need sharpness as input signal in steep slope Jul 19 '21 at 4:51

First, in order to perform a good FFT you have to know your sampling frequency, then to prepare a frequency vector for having a relevant plot (I'm not sure what is the maximum frequency on your frequency spectrum axis, something like 1.6.10^7 ? ):

import numpy as np
import matplotlib.pyplot as plt

fs = ? #sampling frequency

freq = np.linspace(0,fs/2,number of sample to divide your frequency axis)#frequency axis from 0 to fs/2 ([Nyquist-Shannon Theorem][1])

plt.plot(freq, np.abs(data_fft))#plot it
plt.show()


It's possible that you get a huge value on zero of the frequency-domain(see this clear explanation : MATLAB - remove the frequency at zero in FFT).

So before computing the FFT you can just do:

your_data = your_data - np.mean(your_data)


Now, since you want to eliminate the little fluctuations of your signal, a low-pass filter seems to be appropriate here. Once you performed your FFT and evaluated which bandwith you are interested in, you can try for instance a Butterworth lowpass filter (Butterworth filter):

b, a = signal.butter(4, cut_off_frequency/fs, 'lowpass')# computation of the filter's coefficients, 4 is the order of the filter (do not increase it too much)

filtered_data = signal.lfilter(b,a, your_data)

plt.plot(filtered_data)

plt.show()


This is just as an example but it does the job, filters are very tricky so it's necessary to read about it to not make mistakes. Also try different order, beginning by the weaker one. And be gentle with the cut off frequency, don't set it to close to the frequency you want to keep, it could do weird things ( this zone might become unstable depending on the order of the filter you set ).

• Sampling frequency of my data is 15MHZ. I have tried lowpass butter worth filter with various orders and cutoff frequency. I have edited fourier spectrum. Finally, I worked with 1st order low pass with cut off frequency of 1.5MHZ which is better than others. However, It has little fluctuations which i don't want. Aug 11 '21 at 6:05
• I edited my post and added a few words about how to remove the zero frequency of the FFT (it could be helpful). About the fluctuations, things like that might happen with filters, sometimes depending on the data. This would deserve to be asked on another topic. Aug 16 '21 at 12:01