2
$\begingroup$

I'm digging into some (quite popular in my field) analytical practice that I find suspicious. The problem is to remove the trend from some very short (eg. 18-40 data points representing 200 - 800 ms) time series. The common approach is to either use a moving average or a (something around 11-) order FIR filter. I already know that moving-avg is an "exceptionally bad low-pass filter "[1], but I'm concerned about potential artifacts that this procedure can introduce into the filtered signal. When I simulate filters used by other researchers with python-mne, I notice that the filter ringing takes many times longer than the signal they filtered. I have also heard about the existence of ringing artifacts and although from what I understand the problem is mainly about disturbing impulse and step signals, I wonder if this phenomenon can introduce artifacts in other types of signals as well. In particular, I have noticed that most filters in my field stop attenuating around 4 Hz, and a bit further, around 5 Hz, researchers often find an increase in frequency strength and then report the oscillating nature of the signal they are studying. I wonder this increase may be due to "overshoot" - like with ringing artifacts. I would love to understand this problem well, so please kindly explain to me if I am reasoning correctly and provide materials from which I could learn more about this phenomenon. Thank you very much in advance!

EDIT: To clarify: by the whole signal I mean an overall number of samples that were collected after a certain event, not that I slice a smaller subsample from a longer signal or I point out a number of non-zero values in a long time series.

[1]: The Scientist and Engineer's Guide to Digital Signal Processing By Steven W. Smith, Ph.D.

Improve this question
$\endgroup$
6
  • $\begingroup$ I'm confused. What exactly do you mean by de-trending? Normally that involves a high-pass filter, not a low-pass $\endgroup$
    – Hilmar
    Mar 9, 2022 at 13:09
  • $\begingroup$ @Hilmar I know that, and I also find it quite weird, but it's literary how they argue a usage of moving-average. What's even more confusing - some people applied mov-avg without explaining any reason at all and then they make an additional step of removing trend by fitting n-th order polynomial and subtracting it. I get the impression that they do it without further reflection, just copying preprocessing from classic publications. That is why it all seems suspicious to me. $\endgroup$
    – Bartek Kroczek
    Mar 9, 2022 at 13:19
  • $\begingroup$ Apart from what Hilmar suggests, the usual approach to removing a DC or wandering trend is a DC blocker. These can have a long impulse response, though, so may not be best for short data segments. There is an example implementation here.. $\endgroup$
    – Peter K. ♦
    Mar 9, 2022 at 13:54
  • $\begingroup$ When you say your signal is 18-40 samples long, do you mean that measurement starts at some external event (dropping a rock, pouring acid into a strong base, telling your wife you just bleached her little black dress, whatever) and then just that many samples were collected? Or do you mean that there's samples being measured continuously, and there's a non-zero signal that's present for 18 to 40 of them? $\endgroup$
    – TimWescott
    Mar 10, 2022 at 0:29
  • 1
    $\begingroup$ Sorry, I should have put this in my comment -- please edit your question to clarify this. Stackexchange has the quirk that it wants the full question to be stated, even if it takes edits, rather than having details buried in the comments. $\endgroup$
    – TimWescott
    Mar 10, 2022 at 20:38

1 Answer 1

Reset to default
1
$\begingroup$

The easiest way to detrend a short signal is to simply subtract out the mean, i.e.

$$y[n] = x[n] - \frac{1}{N} \sum_{k = 0}^{N-1} x[k]$$

No ringing, no filter tweaking required and guaranteed to be me mean free.

Any other method will require some trade offs that depend the signal the characteristics of the trend your specific application requirements. There are lots of different methods: DC blocking filters, parametrical models, trend fitting etc.

Trade-off involves time domain ringing, spectral damage to the signal, effectiveness of de-trending, causality (or not), transient preservation, length of "usable" data, etc.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Sorry but I don't get that. What if a signal is non-stationary? Probably I misuse the term "detrending" I guess I make my question unclear. I made a simulation of the preprocessing described above, and the artifact appeared in the results. Now I try to understand why. Is it because the filter ringing is longer than the signal itself? Is this an example of overshoot? Thank you very much for your help. $\endgroup$
    – Bartek Kroczek
    Mar 9, 2022 at 14:40
  • $\begingroup$ Fair question: but how do you even define "stationary" for a signal that's just 30 samples long ? $\endgroup$
    – Hilmar
    Mar 9, 2022 at 20:58
  • $\begingroup$ Maybe a picture would help. !Detrending So, I'm dealing with this detrending/lowpass problem by fitting low-order poly and extracting residuals. It seems to work fine, and my simulations show that my preprocessing does not create an artificial effect in the filtered data, while the process I described above does. Now he is trying to understand why this is happening. $\endgroup$
    – Bartek Kroczek
    Mar 10, 2022 at 8:43
  • $\begingroup$ Yes, the picture helps. In this case you would need indeed a high pass filter OR a lowpass to determine the trend as a function of time and subtract it out. There are many algorithms on how to do this but the best choice depends on the properties of the desired data and the trend. In this case I would probably do a 2nd or 3rd order polynomial fit and subtract this out. The trend looks a bit sinusoidal, so could also try a high resolution spectral analysis and subtract out the low frequency peak. In most case, you just need to a try a few and see which is "best" for your needs $\endgroup$
    – Hilmar
    Mar 11, 2022 at 8:49
  • $\begingroup$ But yes, I agree. A conventional filter here would be a poor choice: given the low cutoff frequency, it would take too long to get into gear and you will lose significant amounts of data at both ends $\endgroup$
    – Hilmar
    Mar 11, 2022 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.