Detecting and Removing Noise from Signal using Python

Question

Through this platform, I want to ask that how can I remove unwanted noise from the signal when you do not have much information regarding the frequency at which they appear? Data is collected from an inductive sensor and sampling frequency is 30000 Hz. There are a lot of electrical noises in the signal, the easily distinguishible have already been removed using a Notch Filter. However, there are some electrical noises which are not easily distinguishible from the other parts of the signal. I tested the following approaches:

• Removing easily visible electrical noises using Notch Filter
• Taking FFT and applying binning to visualize other noisy parts
• Using find_peaks() function to detect and remove noisy parts

However, when I appled ifft I could not get the filtered original time series data.

This is the Original Time Series Signal:

This is the complex Fourier Transform before binning and find_peaks():

And this is the fourier transform after binning:

And this is the result obtained after find_peaks():

And this is the inverse FFT result that I have obtained after so called processing (FFT, binnig, find_peaks() technique)

Can anyone please help me understand where I went wrong?

This is the reference data obtained usinf Eddy Current Sensor. I suppose my data should also look likr this after denoising.

Answer 1

The train moves at speed $v_t$. This (absolute) speed reasonably will have a supremum $v_{t\max}$. It obviously has an infimum of $0$, but there is nothing to measure with a train standing around. So we declare some minimum speed we are interested in and call that $v_{t\min}$.

The rail fasteners are spaced at a distance of $d$, which can be assumed as constant for these purposes. The transitions at the "heart" points can with some reason be assumed to be in the same order of magnitude as $d$.

From this, we can derive the part of the spectrum we are interested in. For the lower corner frequency, we take $v_{t\min}$ and add some margin, say, a decade, below. That gives us $$f_l = 0.1\frac{v_{t\min}}{d}$$ We do the same for the upper corner frequency and set the margin again to a decade, above this time of course: $$f_u = 10\frac{v_{t\max}}{d}$$ With these two corner frequencies, design a bandpass to filter the signal. You should be able to see a lot more than before. From there further steps can be taken.

Remark on your attempts so far: The operations you performed on the spectrum where nonlinear, so the inverse FFT was bound to produce rubbish.