I am designing a frequency lowpass filter using FFT&IFFT. My test data is a time series with a large DC component (the pink curve). Before the FFT, the time series was detrended (curve fitting) to remove the trend (the green curve). Test time series enter image description here

My problem is: the detrended time series has discontinuity at the two ends which will cause severe spectral leakage. To improve the result, I tried several approaches:

1) applying windows (e.g., Hann window), it helps to force the ends to zeros. However, the windowing causes the amplitude reduced in the time domain, the frequency amplitude changes as well, as result, the restored signal using IFFT will have lower amplitude, and it cannot be recovered by multiplying certain coefficients (because the windowing procedure is not reversible).

Dividing the whole time series into small segments with 50% overlap, applying like Hann window to each sub-segment helps to solve the above amplitude change issue, but my actual project scenario requests me to do FFT using the longest data length, so this is not a way to go.

2) Instead of windowing, another approach I tried is simply align the beginning and end point to remove the discontinuity, this reduce the spectral leakage caused by discontinuity, but because of the alignment, the DC component is not zero any more, which also introduce leakage to the low frequency band.

Based on my limited knowledge, I cannot figure out a better idea, looking for experts' suggestions, thanks

  • $\begingroup$ Only a single cycle of the sinusoid (green curve)? $\endgroup$ – random_dsp_guy Apr 9 '15 at 4:17
  • $\begingroup$ The de-trended curve (green) is close to but not a complete cycle, see the second figure $\endgroup$ – EastSunshine Apr 9 '15 at 19:12

You don't remove DC by de-trending. The DC component is removed by de-meaning or subtracting the average. The detrending/curve fitting process alters your time series and will only distort your results.

You do not need to remove the DC component prior to the FFT.

Just by visual inspection of your pink curve, I'd say that you have less than a single cycle of a sinusoid. You need multiple cycles...

| improve this answer | |
  • $\begingroup$ After de-trending, the DC and the big trend both are gone. but, because the de-trended curve is not an integer number of a complete cycle, begin and end point has discontinuity. To force these two ends to zero, I simply fit a straight line connecting the begin-end points, the resulting curve is the green one (zero at both ends, but DC is not zero then). If de-meaning on the green curve, the two ends won't be zero then. $\endgroup$ – EastSunshine Apr 9 '15 at 19:18
  • $\begingroup$ used to be i thought i knew something about math, LTI systems and filters, DSP, etc. it's like it's a different language here. can't find neither "de-trending" nor "de-meaning" (except in a social context, i am a demeaned old curmudgeon that eschews trends) in the lit. $\endgroup$ – robert bristow-johnson Apr 9 '15 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.