0
$\begingroup$

I have data which when plotted looks like the green line in the below image. The data is from the walking-motion of a reinforcement-learning-model in a simulation and therefore the values do not really correspond to time. But the dataset is 10000 steps long. I used the steps as the time variable. This is just a part of the plot.

input

To me the signal looks like a periodic signal with a lot of noise. Therefore my idea was to use Fourier Transform to identify a few underlying main frequencies and then simplify the signal. Thus cancelling out the noise.

I create the FFT like this:

s = ppo2_df['obs_4'].values

fft = np.fft.fft(s)
T = 1           # sampling interval 
N = s.size

# 1/T = frequency
f = np.linspace(0, 1 / T, N)

plt.figure(figsize=(20, 5))
plt.ylabel("Amplitude")
plt.xlabel("Frequency [1/Step]")
# plot only half the spectrum
barlist = plt.bar(f[:N // 2], np.abs(fft)[:N // 2], align='edge', width=0.001)

max_inds = np.argsort(np.abs(fft)[:N // 2])[-5:]
for i in max_inds:
    barlist[i].set_color('r')

plt.show()

FFT

Next I try to reconstruct the signal from the FFT using IFFT using the 5 strongest frequencies discovered. My issue is that the eventual plot does not fluctuate around 0 like the original plot but around 0.4 instead. I don't know the mistake I must have made:

# filter n max amplitude frequencies
max_freq = f[max_inds]
max_fft = np.zeros(N)
max_fft[max_inds] = fft[max_inds]

x = np.fft.ifft(max_fft)
fig = px.line(y=np.abs(x[:1000]))
fig.show()

IFFT

Can you tell me what I did wrong? (e.g. I am unsure if i chose a sensible sampling interval T = 1)

EDIT:

The data can be copied from here:

https://justpaste.it/3st5t

$\endgroup$
1
  • $\begingroup$ Also I suggest not plotting the absolut value of the complex values of the reconstructed signal, but only the real part fig = px.line(y=x.real[:1000]). The output signal seemed a lot less accurate because of this $\endgroup$ – Philipp May 27 '20 at 21:04
2
$\begingroup$

Without having the original data it is difficult to confirm, but the DC offset is due to bin 0 in the DFT. Bin 0 corresponds to the DC term or average offset value for the sequence. The plot in the DFT result shows a large value for bin 0 while the time domain waveform appears to be closer to 0 average. I would need to see the original data to determine how the DFT bin 0 occurred.

Update: Upon reviewing the original data from the OP, the average of this data is 0.4123 which is consistent with the result achieved.

$\endgroup$
11
  • $\begingroup$ But where did I omit it? The offset does not exist in the original signal, therefore I was surprised to find it in the frequency spectrum created by the FFT. I assumed I had made a mistake in the construction of this spectrum? I will upload the data... $\endgroup$ – Philipp May 24 '20 at 16:04
  • 1
    $\begingroup$ Looking at it now- i like that justpaste.it link, will have to remember that one $\endgroup$ – Dan Boschen May 24 '20 at 16:15
  • 1
    $\begingroup$ Yes it looks correct. Not necessarily the best approach to remove noise but the result appears to represent the dominant frequencies present. $\endgroup$ – Dan Boschen May 24 '20 at 16:55
  • 1
    $\begingroup$ It would require knowledge about the signal you are looking for, but there are other posts here that detail why nulling bins is a poor approach to filtering (I will link then if I can find it). They discourage back and forth discussion in the comments, so if suggest posting a new question about optimum filtering approaches assuming you do have more information about what you expect your signal to look like if there was no noise (is it frequency constrained or are there any other characteristics you are hoping to extract?) $\endgroup$ – Dan Boschen May 24 '20 at 17:02
  • 1
    $\begingroup$ @Philipp also see this recent post by MattL dsp.stackexchange.com/q/67808/21048 $\endgroup$ – Dan Boschen May 25 '20 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.