Filtering of data in signal processing

Question

I am processing the data from serial. I have to filter the data to remove ripples. I have tried with the following code. However, I can't get the expected results. Suggest me which type of filter I have to use in order to get expected results?

def graph_plot():
    plt.xlabel("samples")
    plt.ylabel("data")
    plt.xlim([0, 2048])
    plt.ylim([0, 255])
    return plt

plt.ion()
fig = plt.figure()
ax = fig.add_subplot(111)
ax.minorticks_on()
ax.grid(1, which = 'both', axis = 'y', markevery = 5)

actual_rx_data = rx_data[4:2052] #rx_data is input from serial
N = len(actual_rx_data)
rx_data = [actual_rx_data[i] for i in range (0, N)]
rx_data = np.reshape(rx_data, (2048, 1))

smoother = ConvolutionSmoother(window_len = 20, window_type = 'ones')
smoother.smooth(rx_data)

plt = graph_plot()

ax.plot(rx_data, color = 'red')  #input 
ax.plot(smoother.smooth_data[0], linewidth = 2, color = 'blue')  #output
ax.clear()

Result obtained by the above code https://i.stack.imgur.com/xS6ct.jpg

Expected results in blue color https://i.stack.imgur.com/RoxlY.jpg

Fourier spectrum https://i.stack.imgur.com/6L3Vc.jpg

Answer 1

First, in order to perform a good FFT you have to know your sampling frequency, then to prepare a frequency vector for having a relevant plot (I'm not sure what is the maximum frequency on your frequency spectrum axis, something like 1.6.10^7 ? ):

import numpy as np
import matplotlib.pyplot as plt

fs = ? #sampling frequency

freq = np.linspace(0,fs/2,number of sample to divide your frequency axis)#frequency axis from 0 to fs/2 ([Nyquist-Shannon Theorem][1])

data_fft =  np.fft.rfft(your_data)#perform your fft

plt.plot(freq, np.abs(data_fft))#plot it
plt.show()

It's possible that you get a huge value on zero of the frequency-domain(see this clear explanation : MATLAB - remove the frequency at zero in FFT).

So before computing the FFT you can just do:

your_data = your_data - np.mean(your_data)

Now, since you want to eliminate the little fluctuations of your signal, a low-pass filter seems to be appropriate here. Once you performed your FFT and evaluated which bandwith you are interested in, you can try for instance a Butterworth lowpass filter (Butterworth filter):

b, a = signal.butter(4, cut_off_frequency/fs, 'lowpass')# computation of the filter's coefficients, 4 is the order of the filter (do not increase it too much)

filtered_data = signal.lfilter(b,a, your_data)

plt.plot(filtered_data)

plt.show()

This is just as an example but it does the job, filters are very tricky so it's necessary to read about it to not make mistakes. Also try different order, beginning by the weaker one. And be gentle with the cut off frequency, don't set it to close to the frequency you want to keep, it could do weird things ( this zone might become unstable depending on the order of the filter you set ).