0
$\begingroup$

I'm playing with FFT at the moment and I try to get periods from noisy signals by recreating this example. While experimenting, I've noticed that after smoothing a quite noisy signal, the result of fft() is actually the same signal again - which is what I don't understand.

Here is a full example which can be run in an IPython Notebook (You can create a notebook here and run the code if you want).

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

figsize = (16,8)
n = 500
ls = np.linspace(0,2*np.pi, n)

x_target = np.sin(12*ls) + np.sin(52*ls)

x = np.sin(12*ls) + np.sin(52*ls) + np.random.rand(n) * 3.5
x = x - np.mean(x)

x_smooth = pd.rolling_mean(pd.DataFrame(x), 14).replace(np.nan, 0.0).as_matrix()
x_smooth = x_smooth - np.mean(x_smooth)

x_smooth = np.roll(x_smooth, -7)

# Getting shwifty and showing what we've got
plt.figure(figsize=(16,8))
plt.scatter(ls, x, s=3, c=[1.0,0.0,0.0,1.0])
plt.plot(ls, x_target, color=[1.0,0.0,0.0, 0.3])
plt.plot(ls, x_smooth)

plt.legend(["Target", "Smooth", "Noisy Data"])

# Target
x_fft = np.abs(np.fft.fft(x_target))
pd.DataFrame(x_fft).plot(figsize=figsize)

# Looks like it should
x_fft = np.abs(np.fft.fft(x))
pd.DataFrame(x_fft).plot(figsize=figsize)

# Plots the same signal?
x_fft = np.abs(np.fft.fft(x_smooth))
pd.DataFrame(x_fft).plot(figsize=figsize)

Below you find the resulting plots of this script.

Noisy data with smoothed signal and target function:

enter image description here

FFT of the target

enter image description here

FFT of the noisy data

enter image description here

FFT of the smoothed data

enter image description here

I don't really get why this is the case here. Can somebody explain this to me or am I doing something wrong here?

$\endgroup$
  • $\begingroup$ can you show the FFT of the target? because the FFT of the "noisy" data looks like the FFT of the target to me, also, what is the scale of the FFT? is it in dB or is it magnitude in a linear scale or is it magnitude squared? $\endgroup$ – robert bristow-johnson Sep 19 '16 at 16:04
  • $\begingroup$ @robertbristow-johnson Well, I don't understand why this looks so "un-noisy" to you but I've added the plot of fft(x_target). I'm not sure but I think you can run my code here .. $\endgroup$ – displayname Sep 19 '16 at 16:33
  • $\begingroup$ well, i originally interpreted the vertical scale to be in dB. and if your noise was 150 or 200 dB below the signal, it's pretty clean. so now i am guessing that your vertical scale is not dB. still, the noisy data with green dots appears to me to be so much noisier than the FFT of it. $\endgroup$ – robert bristow-johnson Sep 19 '16 at 16:49
  • $\begingroup$ @robertbristow-johnson But the question is why FFT of the smoothed data is not showing the same two spikes. $\endgroup$ – displayname Sep 19 '16 at 16:52
  • $\begingroup$ actually, today the smoothed data doesn't look so bad, considering the very large amount of noise that appears to be added to the target signal. $\endgroup$ – robert bristow-johnson Sep 19 '16 at 17:52
1
$\begingroup$

The answer is that the resulting vector from

x_smooth = pd.rolling_mean(pd.DataFrame(x), 14).replace(np.nan, 0.0).as_matrix()

was not flat.

I just had to change

x_smooth = x_smooth - np.mean(x_smooth)

to

x_smooth = (x_smooth - np.mean(x_smooth)).flatten()

which changes a vector from

[...
 [  3.53890989e-01]
 [  1.21347337e-01]
 [ -3.39968386e-01]
 [ -5.37617124e-01]
 [ -5.20015023e-01]
 [ -5.57602781e-01]
 [ -5.89304486e-01]
 [ -6.27991220e-01]
 [ -7.61544946e-01]
 ...]

to something like:

[...
 -0.56731051 -0.59901221 -0.63769895 -0.77125267 -0.89067055 -1.28890512
 -1.49325225 -1.59707353 -1.6298505  -1.42489726 -1.296161   -1.01562861
 -0.76996748 -0.75370003 -0.80788025 -0.8353555  -0.78678532 ...]

and we finally get

enter image description here

$\endgroup$
0
$\begingroup$

Smoothing usually involves some kind of low past filter effect (depending of how you do it, there might also be some non-linear effects). So high frequencies information gets removed from the data.

If your actual signals happened to be in the high frequency range, then it will get removed, but if it is in the lower frequency range than it would be (almost) unaffected.

$\endgroup$
  • $\begingroup$ Well, in my post you can see that I smooth with a simple rolling mean. However, the original periods of 12 months and 52 is still quite visible to the naked eye and yet FFT is returning just the same signal. I don't see how your answer responds to my question. $\endgroup$ – displayname Sep 18 '16 at 22:54
  • $\begingroup$ @displayname The FFT should not return the same signal, this might just be a coincidence. You probably get a different result if you would use newly generated noise. However I do wonder what is up with the numpy FFT function, because in the first one can clearly see that it is mirrored at 250, which it should if it also shows the negative frequencies, however the second one isn't. $\endgroup$ – fibonatic Sep 18 '16 at 23:15
  • $\begingroup$ @displayname It looks like x_smooth is pure noise. Somehow your filter is removing the desired signals from x. $\endgroup$ – MBaz Sep 19 '16 at 0:46
  • $\begingroup$ First of all if you take a look at the plot you can clearly see that x_smooth is defenitely less noisy as x, the original signal. Second you can see that fft(x) has peaks at 12 and 52 which is what you expect because fft(x_target) is created that way. Third, x_target follows x quite well and the peaks at 12 and 52 should be even more visible for fft(x_smooth). If not, then the plot(fft(x_smooth)) should at least be a symmetric plot and not look the same as plot(x_smooth). $\endgroup$ – displayname Sep 19 '16 at 11:04
0
$\begingroup$

From you FFT result, the raw data ("noisy") is two sinusoids where one has frequency nearly exactly 4 times that of the other. that adds up to a periodic or nearly periodic function of time. Then you are adding to that a bunch of white noise that makes it look really dotty, but is spread out among all frequencies, so it brings up the noise floor a little.

except that doesn't completely make sense because the sinusoids appear nearly 200 dB above the noise floor. if that is really the case, your "noisy" plot shouldn't look very noisy, as it does.

my original diagnostication was that the "smoothing algorithm" wasn't very good and made a nasty approximation of the original two sinusoids.

$\endgroup$
  • $\begingroup$ Yes, the noisy plot looks very noisy which is actually intended. The "smoothing algorithm" is just a rolling average and as you can see in my updated plot, it is actually working pretty well. The only question is why fft(x_smooth) is not a perfectly symmetric plot that shows two peaks at 12 and 52. $\endgroup$ – displayname Sep 19 '16 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.