4
$\begingroup$

I have a series of signals sampled at 1200 Hz that are 200-300 samples long. I'd like to analyze them in the frequency domain in 1 Hz bins in MATLAB. I'm primarily interested in the frequencies below about 20 Hz, since this is where virtually all of the signal power is contained.

However, I'm finding conflicting opinions on whether it's appropriate to subtract the mean value from the signal before I perform the FFT function. When I don't subtract the mean, I get a strong 0Hz component, which makes sense given the shape of the original signal. But some sources say you should zero-mean the data so you don't get distortion of the low frequencies, which are the ones I care about. If I do subtract the mean, the FFT is markedly different, and appears to contain massive sinc-like distortions.

Here is the original signal in blue, and the zero-mean version in red, created using detrend(mySignal, 'constant').

enter image description here

As expected, the detrended signal has a mean of zero. But when I do the FFT and look at the spectral distribution, this is what I get:

enter image description here

Note that X-axis is frequency and I've truncated the axes to the first 50 frequency bins (0-49 Hz).

Is it inappropriate to zero-mean a signal like this? And what's the meaning of this distortion in the zero-meaned signal, if anything? Any help is much appreciated!

Improve this question
$\endgroup$
5
$\begingroup$

I bet you zero-padded the data before FFT. That's not going to work well because the results will be dominated by the discontinuity between your data and the zeros. It is not equivalent to subtract the mean before zero padding vs. after.

Try subtracting the mean, then multiplying the data with a window function, and then zero-padding.

The 0 Hz bin will still typically have a non-zero value. You can get rid of that by calculating the windowed, 0 Hz subtracted signal as (MATLAB/Octave notation):

(signal - mean(signal.*window)/mean(window)).*window

sum can be used in place of mean because the numbers of samples cancel.

Improve this answer
$\endgroup$
  • $\begingroup$ Yes, this exactly what happened! I was zero-padding to get the bin width to 1 Hz. One more question--after I subtract the mean, then multiply by a window function, my data no longer has a mean of zero. Is this still going to cause problems? I think I'll still get a 0 Hz component when I do the FFT of the data, after having subtracted the mean, multiplied by a window function, and zero-padded. $\endgroup$ – user33685 Feb 4 '18 at 18:31
  • $\begingroup$ I don't know which is better, to allow some 0 Hz or to eliminate it. $\endgroup$ – Olli Niemitalo Feb 4 '18 at 21:41
2
$\begingroup$

If you remove a constant amount from your signal, it will only affect the DC (0Hz) bin. You are using detrend.

"Description. detrend removes the mean value or linear trend from a vector or matrix"

If you remove a slanted line from your signal, it will affect the other bins. I don't see any reason for you to perform this operation on your data, just ignore (or zero out) the zero bin. I also don't see the FFT as being that applicable to your data. It is best for periodic signals. In other words, I don't understand what you are trying to achieve by a "frequency analysis" with that shaped curve.

Hope this helps,

Ced

=======================================

Followup

Looking at your data closer, you may benefit from removing a best fit half a sine wave, then do a FFT on the remainder to analyze those squiggles. It looks like you have a small peak at bin 9. The major portion of you blue FFT can be explained as the leakage of a tone in between bin 0 and bin 1.

To remove the best fit of the half sine wave, or any curve, create a basis vector for it (an array with the function values). Call it $\vec B$. Your signal is $\vec S$. The goal is to find a coefficient ($c$) gives the closest fit.

$$ \vec S = c \vec B $$

Dot both sides with $ \vec B $. If your basis is complex valued, use the conjugate.

$$ \vec S \cdot \vec B = c \vec B \cdot \vec B $$

It is now a scalar equation and c and be easily solved for.

$$ c = \frac{ \vec S \cdot \vec B }{ \vec B \cdot \vec B } $$

Finally, subtract the best fit from your signal

$$ \vec R = \vec S - c \vec B $$

Now, the results of the FFT applied to the remainder ($\vec R$) may be more interesting and not masked in the FFT.

===================================

Another followup.

I didn't notice the 'constant' modifier, so this isn't a detrend issue. My apologies to Mark Leeds.

What is probably happening is a framing issue. Something like the dtrend shifting the signal where the values exist, but the fft is being applied to a larger frame. In essence you are adding a step function to your signal. Make sure your fft is only happening on the part of the signal that you have values for.

What I said about removing the best fit half sine wave remains true.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ @Cedric: Based on the picture, I think the mean is ~1000 and it's being subtracted from the original signal. So, he is subtracting a constant rather than de-trending. I think. But I don't know the answer to his question. I would think the FFT should be able to handle the constant without the need for subtracting it out. Thanks for any insights. $\endgroup$ – mark leeds Feb 4 '18 at 4:09
  • $\begingroup$ @mark leeds, It is s property of the DFT that adding or subtracting any constant signal will only affect the DC bin. This is due to the DFT being an orthogonal basis and a DC signal is orthogonal to all the other bin basis vectors. You can find a mathematical proof, and graphical display, of this property from the DFT definition at Figure 17 in my blog article called "DFT Graphical Interpretation: Centroids of Weighted Roots of Unity" at dsprelated.com/showarticle/768.php. Hopefully, the whole article will give you some insight. The concept behind the blog was a huge "aha moment" for me. $\endgroup$ – Cedron Dawg Feb 4 '18 at 4:30
  • $\begingroup$ Hi Cedric: I plan on checking out many of your articles because your answers are always beautiful. The problem is that when you're learning DSP for the first time, it's like an avalanche and you have to decide which snow boulders to run through and which to get out of the way of !!!!! Thanks. P.S: my comment was just to point that he is subtracting a mean and not de-trending.. $\endgroup$ – mark leeds Feb 4 '18 at 16:42
  • $\begingroup$ @mark leeds, Thanks for the kind words. Your observation was spot on, I had misdiagnosed the problem. BTW, it's "Cedron", no big deal. DFTs are only part of DSP, but they are the main focus of my blogs. There are two traditional pathways to learning about the DFT. The first is as a special case of the continuous FT, which requires Calculus and Real Analysis to truly understand. The second is in a Linear Algebra context, which requires an understanding of Linear Algebra to understand. Both are unnecessarily difficult in my opinion. $\endgroup$ – Cedron Dawg Feb 4 '18 at 17:42
  • $\begingroup$ @mark leeds, (con't) My articles strive to give an understanding of the DFT based solely on the definition. The only prerequisites are ordinary algebra, trigonometry, and an understanding of complex numbers. Some articles use a little bit of Linear Algebra too. All the formulas are derived from scratch, and usually the article is about explaining the derivation. You won't find most of the formulas I derive anywhere else. $\endgroup$ – Cedron Dawg Feb 4 '18 at 17:42
1
$\begingroup$

dtrend() does more than just remove the average over the window, it subtracts a linear fit from the data. While puzzling about the differences in the spectrum can say a lot, look at your data in the time domain and puzzle on that was well. It might be that 256 points are not enough samples to resolve what you are interested in, using a DFT. One common mistake people make is to think that a power of 2 FFT is written in stone. A modern FFT library is likely to be based on FFTW or have a similar user interface. You can pick a window size that works for what you want to see.

Another common misconception is that AR, ARMA, proney, and DFT spectral estimation is an either/or mutually exclusive proposition. It's not that hard to get that impression because most papers like to compare what they invented, with what was before but there is no general always best technique. Spectral analysis, like life, isn't a one course meal and the literature, by its nature is doesn't reflect that. You can often mix and match.

You don't say why you picked 256 point DFTs. It might be a hard constraint or it might not. My experience is that dtrend() has more subtle effects as your window gets longer. There are times when I use it and times when I don't. My general criteria favours using it if I'm more interested in the mid and high sections of my FFT based spectrum. It helps reduce the often present "junk" near DC. If you are interested in those low frequencies, I don't recommend using it.

Improve this answer
$\endgroup$
  • $\begingroup$ The detrend was my initial suspicion as well. The OP uses "detrend(mySignal, 'constant')", so it is doing a clean subraction of the mean (I think). Olli's explanation sounds quite reasonable. If the OP is zero padding, but shifts before zeropadding, the difference between the padded original signal and the padded shifted signal will be a step function, which would explain the sinc like output when it is DFTed. $\endgroup$ – Cedron Dawg Feb 4 '18 at 18:22
  • $\begingroup$ Olli did make a good call. I think your suggestion was good too, although the OP should look at the time domain data when choosing a model to fit. A half sine is a good general suggestion. I'll try it when I have a reason. $\endgroup$ – user28715 Feb 4 '18 at 18:31
  • $\begingroup$ It seems the OP agrees, Olli got the check mark. My suggestion about the half sine was based on the appearance of the single example the OP gave. It may not be appropriate in general. I was just trying to make the DFT more relevant to the problem. $\endgroup$ – Cedron Dawg Feb 4 '18 at 18:39
  • $\begingroup$ actually the more i think about it, the more i like it. subtracting the average is like a notch filter at DC. Subtracting a half cycle sine, corresponding to the window length, is like a notch between the zero and first bin. Doing a DFT on a time series where the linear fit has been removed is like doing a DFT on the residual error of the linear fit which makes sense if the linear fit is your purpose but the half sine makes more sense if the harmonic estimate is your actual purpose. So DC and half sin or just the half sine? $\endgroup$ – user28715 Feb 4 '18 at 19:19
  • $\begingroup$ Sort of, kind of. If you want to remove the 1/2 cycle component properly you need to build a basis with a half sine ($\vec{B_1}$) and a half cosine ($ \vec{B_2} $) vector set. $$ \vec S = c_1 \vec{B_1} + c_2 \vec{B_2} $$ Dot this with both basis vectors and you get a system of two equations with two unknowns ($c_1$, $c_2$). This is the essence of what the DFT operation is anyway, except $ \vec{B_1} $ and $ \vec{B_2} $ are always orthogonal for whole integer frequencies, making their dot product zero and solving for the coefficients trivial. $\endgroup$ – Cedron Dawg Feb 4 '18 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy