0
$\begingroup$

I implemented the Hampel Filter (see here) for outlier removal, however I noticed in instances when the data in the sliding window is stationary and its standard deviation is very low, the filter will think even the tiniest deviation is an outlier.

For example, if we have sliding window equal to 7 (three values on each adjacent side), with the input series 2, 2, 2, 2.05, 2, 2 ,2 then the Median Absolute Deviation (MAD) = 0 and the filter thinks that the 2.05 is an outlier, independent of how many standard deviations the value should be off by.

Any advice on a workaround?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

What comes to mind here is to employ dithering before filtering and de-dithering afterwards. This will effectively increase the standard deviation of any sliding window to a minimum value the size of which depends on the power of the dithering noise. Non-stationary slices will virtually be not affected. After filtering, you can subtract the dithering noise again, which might even be unneccessary depending on your $S/N$ requirements.

Regarding implementation, you will not need a special function for that. Just add some random white noise of low power to your signal. Experiment with the power level and you should come to a satisfying solution.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Interesting. What do you mean by 'low power'? From my understanding, I could sample from a Normal distribution with mean = 0, and standard deviation = 1, or perhaps a different s.d.? $\endgroup$
    – MilTom
    Commented Mar 31, 2022 at 9:20
  • $\begingroup$ By low power i mean low compared to the signal, so that you keep the S/N high. You could say "low amplitude" or "low level". Normal distribution should work fine, just tweak the level, high enough so that the "false outliers" in stationary areas are kept untouched and low enough, so that your S/N does not suffer unduly. $\endgroup$
    – Max
    Commented Mar 31, 2022 at 10:16
  • $\begingroup$ Would I dither the sample I am looking at as well (i.e. the whole window), or exclude it? $\endgroup$
    – MilTom
    Commented Mar 31, 2022 at 12:31
  • $\begingroup$ I don't think it matters much outcomewise. It is probably simpler to dither the whole signal and not bother excluding the center sample. The workflow could be "dither whole signal -> run Hampel filter over it -> subtract dither signal again". If the signal is to long for such an approach, you can parallel pipe it any way you like. $\endgroup$
    – Max
    Commented Mar 31, 2022 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.