1
$\begingroup$

I need to remove trend from my time-series which looks like the following images.

However, I want to estimate the trend before removing it. Hence directly removing it won't do it.

The simple polynomial models didn't work, because — I think — the signal is long and the trend is changing. Also, filtering the signal with low pass removed too much of the trend as well.


enter image description here


enter image description here


The files are available here.

$\endgroup$
18
  • 1
    $\begingroup$ high-pass filter? or compute sliding median and subtract that from the data? $\endgroup$ May 21, 2023 at 8:04
  • $\begingroup$ Which simple polynomial models? References, please $\endgroup$ May 21, 2023 at 8:49
  • $\begingroup$ @robertbristow-johnson, highpass? it is a low frequency phenomenon, no? $\endgroup$
    – Mark
    May 22, 2023 at 8:04
  • $\begingroup$ @RodrigodeAzevedo, i used matlab detrend mathworks.com/help/matlab/ref/detrend.html. $\endgroup$
    – Mark
    May 22, 2023 at 8:05
  • $\begingroup$ "trend" and "local" are contradictory terms. Which trend exactly do you want to remove ? (Please a more descriptive answer than "the one on the plot".) $\endgroup$
    – user67664
    May 22, 2023 at 9:12

3 Answers 3

5
$\begingroup$

Start simple: just use a 1-D median filter of an appropriate length.

If I do that with a length of 100 samples, I get the following for your first signal.

Signal, and median filtered signal.

The top plot shows the original signal (blue) and the median filtered signal (red). The bottom plot shows just the zoomed-in median filtered signal.

Another possibility is to use a DC Blocker.


Code Below

load signals_samples.mat
clf

for k=1:10

    figure(k)
    clf
    subplot(211)
    plot(signals_samples{k})
    hold on
    plot(medfilt1(signals_samples{k},100),'r')
    subplot(212)
    plot(medfilt1(signals_samples{k},100))

end
$\endgroup$
2
  • $\begingroup$ Any chance you add the link in your answer? $\endgroup$
    – Mark
    Jul 5, 2023 at 16:30
  • 1
    $\begingroup$ @Thomas D'oh! I see what you mean now. Thanks for the clarification. Done. $\endgroup$
    – Peter K.
    Jul 5, 2023 at 17:26
3
$\begingroup$

In A Self Supervised Learning for a Signal Denoising I wrote:

You may have a look at the method called JOT: A Variational Signal Decomposition Into Jump, Oscillation and Trend (You may access it in A Two Stage Signal Decomposition into Jump, Oscillation and Trend Using ADMM).

This method basically does what you're after, it decomposes the signal into 3 signals:

enter image description here

You may look on the results of a signal similar to yours:

enter image description here

The method is quite simple if you know ADMM.
In any way, they supply code.

You signals looks very long, so it might stress the solver for memory.
Still worth a try.

$\endgroup$
1
$\begingroup$

Similar to the JOT method suggested by @Royi, the BEADS algorithm decomposes a signal in three components:

  • a trend,
  • a sparse signal,
  • a residual (which can be used as quality control).

I did a quick test on your first signal. BEADS is not (yet) good at the beginning and end, but it is quite fast: 0.25 seconds for the first signal. It scales almost linearly with signal size. Its parameters allow to control the smoothness of the trend. Here are the results. Parameters follow.

BEADS trend removal

data1 = [signals_samples{1}];
% Filter parameters
fc = 0.0035;     % fc : cut-off frequency (cycles/sample)
d = 1;          % d : filter order parameter (d = 1 or 2)

% Positivity bias (peaks are positive)
r = 1;          % r : asymmetry parameter

% Regularization parameters
% amp = 0.8;      
amp = 0.1;      
lam0 = 1*amp;
lam1 = 5*amp;
lam2 = 2*amp;
 [data1Filt, dataBaseline, cost] = beads(data1, d, fc, r, lam0, lam1, lam2);
$\endgroup$
2
  • 1
    $\begingroup$ What can be done to improve this? $\endgroup$
    – Mark
    Jul 26, 2023 at 5:52
  • $\begingroup$ I am not sure of what you need to improve here. There are many options. The simplest is to perform a preprocessing such that both ends are tapper to zero $\endgroup$ Aug 4, 2023 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.