1
$\begingroup$

This signal represents the distance in an experimental device. As the speed is constant, the signal should be linear (untill it drops). However, for some reason the device showed interferences as shown in the figure below:

enter image description here

The question is how can I remove this high amplitude interferences, making the signal as linear as possible?

I've already tried the butterworth filter and read something about the Chebyshev, but I don't think it applies.

Any suggestions?

$\endgroup$
  • $\begingroup$ Can you share the data? $\endgroup$ – Laurent Duval Jun 21 '16 at 18:44
  • $\begingroup$ @LaurentDuval Yes, download a sample on this link $\endgroup$ – Paulo MiraMor Jun 21 '16 at 19:54
  • $\begingroup$ Same value -0.00082681 everywhere? $\endgroup$ – Laurent Duval Jun 21 '16 at 20:02
  • $\begingroup$ @LaurentDuval This is only the beginning of the file, represented by the horizontal line near zero in thebeginning of the plot. Note that this file has 512000 values. $\endgroup$ – Paulo MiraMor Jun 21 '16 at 20:36
  • $\begingroup$ Apparently this part is flat. No noise $\endgroup$ – Laurent Duval Jun 21 '16 at 20:46
5
$\begingroup$

There are a few simple things that you could try:

  • sliding-window median filtering with an appropriate window size
  • linear-regression in a sliding window
  • robust linear regression to the total signal, if you know the signal duration etc well enough

EDIT: Further idea after a comment from OP on another answer

Since you want the slope at 160000 be the same as before (i.e. no flat section in your signal), you could use a robust linear regression technique such as RANSAC with an appropriate number of sample points to analyze/filter your data.

$\endgroup$
  • 2
    $\begingroup$ Especially I would like to mention bout the nonlinearity of the required filter. Some nonlinear algorithms such as a variation of median filtering is necessary. A linear filter will have quite difficulty in eliminating spike type noise which has flat spectral distribution over the whole signal spectrum. $\endgroup$ – Fat32 Jun 21 '16 at 16:39
  • 2
    $\begingroup$ Yes, that is very correct. Thanks, I will remove the Fourier-suggestion since it really does not perform in this case. $\endgroup$ – M529 Jun 21 '16 at 18:39
3
$\begingroup$

Non-linearity or non-least-energy seems a key, as stated by @Fat32.

If you want to denoise first, a sliding weighted median (with positive weights) could help you remove impulse noise, while keeping a more accurate location of transitions than a pure median. A Matlab implementation here. If this does not suffice, I'll go for more general rank filters. Median is sort of rank $1/2$. If the impulse noise is too heavy, a $\max$ or a $\min$ filter (toward rank $1$ or $0$) may better follow the slope.

If around $160000$ you want to go through following the apparent (local) slope, then a weighted-median derivative, or a sliding robust linear regression (a robust LOWESS) could produce local derivatives without prior filtering.

$\endgroup$
  • $\begingroup$ Around 160000 I'm expecting that the curve follows the pattern, that is, a diagonal line at the same slope as before. $\endgroup$ – Paulo MiraMor Jun 21 '16 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.