1
$\begingroup$

I have a certain signal that has a linear trend that should be removed:

enter image description here

The problem is that the signal needs to be processed in real time(sampling rate is 20Hz). If I only apply the detrending per window of say, 20 samples and then concatenate detrending results, the signal is disrupted.

I wonder if there is some efficient technique to detrend the data per window in real time and then concatenate it so the final signal will make sense? Or should I just detrend every new incoming data with the older data which is far from being efficient.

I would like to note that taking the entire signal and processing it offline using MATLAB's detrend works correctly. The problem occurs when breaking the signal into certain fixed size windows and then applying detrend on each.

Thanks, Mike

$\endgroup$
  • $\begingroup$ Any luck, Mike? $\endgroup$ – Peter K. Sep 3 '15 at 19:55
  • $\begingroup$ Votes or best answer validation are required $\endgroup$ – Laurent Duval Jul 28 at 11:59
1
$\begingroup$

I've had success using DC blocking filters to do this.

The blue graph below shows a noisy trend. The red graph shows the DC blocking filter applied.

Below is some scilab code to implement it. Scilab is quite close to matlab (but not quite!).

enter image description here

// 25189
N=1024;
t=[0:N-1];
trend = 0.1*t;
noisy_trend = trend + 10*rand(1,1024,"normal");

clf
plot(t,noisy_trend)
dc_blocked = [];
for idx = [2:N]
    dc_blocked(idx) = noisy_trend(idx) - noisy_trend(idx-1) + 0.9*dc_blocked(idx-1)
end
plot(t,dc_blocked,'r')
$\endgroup$
  • $\begingroup$ Interesting. I will check it with my signal. I think this is pretty much what I'm looking for. $\endgroup$ – mike Aug 14 '15 at 16:29
1
$\begingroup$

What is the nature of the trend?

If the trend is due to DC drift (perhaps due to the signal acquisition apparatus), you may be able to remove it with a simple high-pass filter around 0.1 - 0.5 Hz. This would eliminate the need for any real-time post-processing.

DC drift gives a slowly rising/falling DC component, such as in the first figure here:

enter image description here

$\endgroup$
  • $\begingroup$ The signal looks more like this walterbressert.com/Images/AboutWB/interview2.gif. And high pass filter didn't do the jobreally. $\endgroup$ – mike Aug 11 '15 at 13:15
  • $\begingroup$ SE.DSP wishes you a happy new year 2017, with a kind reminding signal that your question or its answers may require some action from you (edit, update, votes, acceptance, etc.) $\endgroup$ – Laurent Duval Jan 15 '17 at 16:25
0
$\begingroup$

Let me suggest you to perform the filtering in sliding blocks, not separate. Assume you want to filter the signal $\{ x_i\}$. At each index $k$, you can estimate the linear trend on the block $$X_k=[ x_{k-K_l},\ldots,x_k,\ldots, x_{k+K_r}],$$ taken at indices $$I_k=[ {k-K_l},\ldots,k,\ldots, {k+K_r}],$$ in the form $\beta_k x + \alpha_k$, and remove $\beta x_k + \alpha$ from $x_k$. If $<Y_k>$ denotes the average of the block $Y_k$, then $\beta_k = \frac{<I_k.X_k> - <I_k><X_k>}{<X_k^2>-<X_k>^2}$, and $\alpha_k = <X_k> - \beta_k <I_k>$. All the averages $<I_k>$, $<X_k>$, $<I_k.X_k>$ and $<X_k^2>$ can be computed recursively with few computations.

You will end up with an implementation of a local linear regression, also known as LOESS or LOWESS smoothing.

Then you can think of alternatives like the Hodrick–Prescott filters for instance.

$\endgroup$
  • $\begingroup$ Thanks, I will try this. The only problem with HP filter is that it might not just remove the trend but also smooth the signal. The unwanted smoothing also appears when I applied highpass 2nd order IIR filter of 0.05Hz to the signal. $\endgroup$ – mike Aug 12 '15 at 9:42
  • $\begingroup$ I am not a big fan of HP, just wanted to mention this as one of the many alternatives. I have started implementing the above solution in Matlab, with a variable look-ahead possibility. It seems to work. By the way, you will have smoothing effects anyway. $\endgroup$ – Laurent Duval Aug 12 '15 at 11:42
  • $\begingroup$ Question is if it will be worse than doing high pass filtering. Doing matlab's detrend on the whole data doesn't seem to smooth the data, or at least it is not noticeable. I haven't tried your suggestion yet, should I expect it to smooth the data? $\endgroup$ – mike Aug 12 '15 at 16:51
  • $\begingroup$ BTW, I don't really see why removing a trend from a signal should also smooth it. $\endgroup$ – mike Aug 12 '15 at 16:58
  • $\begingroup$ Because "smoothing" and "detrending" are ill-defined concepts, unless you have a clear and sound model of the data, the trend and the noise, and the a priori you assume to separate them. In what I describe, everything is somewhat linear, so you'll get a kind of adaptive filter. It "could" be better than some fixed high-pass filter $\endgroup$ – Laurent Duval Aug 12 '15 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.