3
$\begingroup$

Given the signal shown below, what is the best way to remove the steps and local maximas it contains. The signal contains some steps which can last up to 100 Samples before they return to about the same value as before the step (Marked with red circles). There are also some peaks which last only for one sample which should also be removed.

First Dataset: On this set, the median Filter works as expected (not shown here). Sample Signal, all peaks should be removed, also the ones wich are marked red.

On the following Dataset, the Medianfilter with the same settings fails. To many samples are replaced by the median value. Sample Signal, green: Median filter from one of the Answers

What I've tried already:

  • Work with a moving average. Advantage: Small peaks have less influence and are nicely removed. Disadvantage: Long lasting steps remain in the signal.
  • Get the first difference of the signal (diff(X)) and correct the value in X if a big positive change is followed by a negative change immediately. Advantages and disadvatages are the same as above.

What are some good methods to remove/correct these unwanted values?

$\endgroup$
9
$\begingroup$

The problem with the moving average is that the average is not robust to the presence of the outliers - so you would need a very large window size to "dilute" the outliers.

Try a non-linear filter instead, like a median filter: Apply a median filter on your signal - you would need a window size of at least 300 samples. Compute the difference between the original signal and the median-filtered version. If the difference is above a threshold, replace the signal by the median-filtered version.


Here is some scilab code that attempts to implement this suggestion. The results are plotted here; it seems to work nicely on the fabricated data.

enter image description here

function sm = smooth(x,len)
    sm = filter(ones(1,len)/len,1,x);
endfunction

N = 6000;
x = 0.1*rand(1,N,'normal');
y = cumsum(x);
y = smooth(y,100);

clf
subplot(211);
plot(y)


Njumps = 20;

jump_indices = round(rand(1,Njumps)*N);

jump_length = 0;

y2 = y;

for idx = jump_indices,
    y2(min(N,idx:(idx+jump_length))) = y(min(N,idx:(idx+jump_length))) + 1;
    //plot(min(N,idx:(idx+jump_length)),y2(min(N,idx:(idx+jump_length))),'r')
    jump_length = jump_length + 1;
end

plot(y2,'g');

filter_length = 10;
y3 = y;
for k = 1:N,
    y3(k) = median(y3(max(1,min(N,(k-filter_length/2):(k+filter_length/2)))));
end

plot(y3,'k')

subplot(212);
plot(y-y2);
plot(y-y3,'r');
$\endgroup$
  • $\begingroup$ This is a good first aproach, but sadly doesn't solve my problem. I've added a second dataset to the question on which the median filter also fails. On the first dataset, this method works very good. $\endgroup$ – jrast Nov 18 '13 at 11:32
  • $\begingroup$ OK, i'll accept this answer, as it brought me closer to a solution. If I apply two filters in sequence, the first to eliminate big offsets (with a big window, ~80 samples) and the second to eliminate small errors (small window, ~20 samples), I get a pretty good result on both example sets from above. $\endgroup$ – jrast Nov 18 '13 at 12:10
3
$\begingroup$

Using the diff approach, you could correct big positive changes and big negative changes separately, i.e. drop the condition that the second has to follow the first immediately. Compute the diff, detect outliers using a threshold, replace them by zero or by the mean of the non-outlier differences, compute the cumulative sum of the differences.

The result won't be perfect though. If you need a better result, I'd recommend investigating where the steps and peaks come from, and either avoid them from the start or design a correction method tailored to their properties.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.