I am trying to detect (not remove) noise from the digital signal to get meaningful information out of it.
The signal I need is in the frequency range of 0.1 Hz to 10 Hz. Anything beyond this is just noise (and useless) for me. So I want to find parts of a signal that do not contain my necessary data and is pure noise.
Once again, to make it clear - I want to detect parts of a signal that is PURE Noise. If it has noise mixed with my required signal, I can pick out the meaningful information from it. But I want to completely remove parts that are PURE noise.
What have I tried till now? The below steps summarize what I have tried till now:
- Scale the data using robust scaler (because the signal can be at different gains or amplifications, so I am scaling it over a fixed window size. And in that window size the gain or amplification is same)
- Pass the data through a band pass filter of 0.1-10 Hz
- Find autocorrelation of the signal with itself over different lags
And some more steps after this which have turned out to be of no help.
The autocorrelation of the desired data after filtering and noise after filtering is helping me differentiate between the 2, but is failing to do so over a considerable number of files.
I want to make an algorithm, that works for all varieties of data - over different amplifications, different frequencies, whether the data has pure meaningful information or pure noise or both mixed, my algorithm should be able to detect if that particular window of the signal is pure noise or has some meaningful information.
So I thought, that when I am filtering a signal having some desired information, to 0.1Hz to 10Hz, what will the effect on its energy? Because then I can find the ratio of energy of the filtered signal to the energy of the raw signal & set a threshold for it. If this ratio is greater than the threshold, the signal has meaningful information & if it is lesser, then it is pure noise.
Thus, to summarize -
- I want to detect parts of my signal with pure noise. Is there a better way to build a universal algorithm to do it?
- What is the effect of filtering a signal on its energy when compared to the raw signal?
- How to find the energy in a signal - in Python?
I know my question might not be clear, but if you can answer the last three questions, it would be great.
Thanks in advance.