5
$\begingroup$

I have read in many places that Moving median is a bit better than Moving average for some applications, because it is less sensitive to outliers.

I wanted to test this assertion on real data, but I am unable to see this effect (green: median, red: average). See here:

enter image description here


Here's the sample audio data test.wav. Here's the Python code:

import numpy as np
from scipy.io.wavfile import read
import matplotlib.pyplot as plt

def median(lst): return np.median(np.array(lst))
def mean(lst): return sum(lst)/len(lst)

(fs, x) = read('test.wav')

x = abs(x)
env = np.zeros_like(x)
env2 = np.zeros_like(x)

for i in range(len(x)):
    env[i] = median(x[max(i-1000,0):i+1])
    env2[i] = mean(x[max(i-1000,0):i+1])

plt.plot(range(len(x)), env, color = 'green')
plt.plot(range(len(x)), env2, color = 'red')
plt.show()

I have tried with various values for Window width (here in the code : 1000), and it was always the same: the moving median is not better than moving average (i.e. not less sensitive to outliers).


The same with Window width = 10000 (10000 >> the spike width) :

enter image description here


Question:

Can you provide an example showing that moving median is less sensitive to outliers than moving average? And if possible using the sample .WAV file data-set (download link).

i.e. is it possible to do a moving median on this data such that the result is like this yellow curve? (i.e. no more spike!)

enter image description here

$\endgroup$
  • 3
    $\begingroup$ The glitch in your test data seems to be larger that 1000 samples long. Median filters only work well when the outliers are... Outliers, and there are not many of them. $\endgroup$ – Peter K. Nov 27 '15 at 23:54
  • $\begingroup$ @PeterK. I updated my question (see second screenshot) with window witdh = 10.000 >> spike width, and it's the same. Note: full data ~ 94.000 samples long $\endgroup$ – Basj Nov 28 '15 at 8:20
  • $\begingroup$ And another for window with = 20.000 >>>> spike width : screenshot. $\endgroup$ – Basj Nov 28 '15 at 8:29
  • $\begingroup$ I will try to play with the wav file... If it is true that the spike length is much less than 20,000 then something else is throwing off the median filter. As per Patrick's answer, you shouldn't see the spike at all. $\endgroup$ – Peter K. Nov 28 '15 at 18:45
5
$\begingroup$

Is moving median always less sensitive to outliers? Sometimes. It will work if you have a very short spike (preferrably shorter than the median/average sample size). However, if you have a large spike, then taking the median won't help eliminate the spikes.

Long spike

I have illustrated this using a dataset (with sample size = 5, taking into account 2 left entries and 2 right entries):

Orig. Median Avg.
Data  Filter Filter
-----------------
1       
2       
3      3    3
4      4    4
5      5    11.6
6      6    19
40     40   26.2
40     40   33.2
40     40   33.4
40     40   27
7      9    20.8
8      9    14.8
9      9    9
10     10   10
11      
12

enter image description here

Notice how the the median of the all the 40s is 40. For example, take the 1st 40. The left values are 5,6 and the right values are 40,40, so we get a sorted dataset of 5,6,40,40,40 (the bolded 40 becomes our median filter result).

Short spike

You also wanted an example for the median filter to work. So, we will have a short spike. Try this:

Orig. Median Average
Data  Filter Filter
-----------------
1       
2       
3      3    3
4      4    10.8
5      5    11.6
40     6    12.4
6      7    13.2
7      8    14
8      8    8
9       
10

enter image description here

You can see that the median filter did filter out the single large spike, while the spike skewed the average filter's results for entries 4,5,40,and,6.

$\endgroup$
  • $\begingroup$ Thanks for your answer @PatrickYu. But with so few samples, it's hard to tell if median or mean are so different. Would you have an example with something more like 100 samples? Something else: can you plot the original data as well on the images? Thanks again for your helpful answer. $\endgroup$ – Basj Nov 28 '15 at 8:23
3
$\begingroup$

This isn't really an answer, but I thought I'd report what I'm seeing and ask for more information.

I've loaded your test.wav file and I can see the signal plotted below.

enter image description here

So what you're getting in the plots you show is not so much the median value, but is more like an envelope of the signal.

The second issue is that the signal actually seems to be part of the signal. If I zoom into the blip then this is what I see:

enter image description here

What are you really trying to achieve?


Thanks for the feedback. Below is some R code that does the following:

  • Loads the wav file.
  • Takes the absolute value of the signal (it's loaded into the left channel by the waveR library).
  • Performs a 100-length moving average filter on the data to get something closer to the "envelope" (red signal).
  • Then applies a median filter of lengths 201, 2001, and 4001 to the result (blue signal).

From the plot below, the best performing is the 4001 length one. Otherwise the effect of the glitch is still present.

Median filtered versions for different window lengths

The only thing I can see wrong now is that the "envelope" doesn't match the true envelope as well as I'd like. A better envelope detector might improve this (e.g. the analytic signal or such).

Below is a plot of the calculated median-filtered envelope overlaid on the original signal.

enter image description here


R Code Below

#27349
#install.packages("tuneR")
library(tuneR)
x <- readWave("Q27349/test.wav")    
sig <- x@left    
abssig <- abs(sig)    
N <- 100
filtabssig <- filter(abssig, rep(1/N,N))    
plot(abs(filtabssig))    
med <- 0*filtabssig    

N2median <- 1000 # Window length is 2 * N2median + 1
for (k in 1:length(filtabssig))
{
  idxs = seq(max(1,k-N2median), min(length(filtabssig), k+N2median),1)
  med[k] = median(filtabssig[idxs])
}

par(mfrow=c(3,1))
plot(filtabssig, col="red")
lines(med200,col="blue")
title("Median filter length of 201")

plot(filtabssig, col="red")
lines(med1000,col="blue")
title("Median filter length of 2001")

plot(filtabssig, col="red")
lines(med2000,col="blue")
title("Median filter length of 4001")
$\endgroup$
  • $\begingroup$ Thanks for the time you spent on this question! In fact, I plotted the median and mean of the absolute value of the signal. And yes this gives an idea of the envelope, this is what I want. What I was expecting (thanks to median instead of mean) was something like this : screenshot. Unfortunately whatever the window width for the median, the spike is always here. Whereas many document say "the median has the advantage of lowering the effect of outliers values"... $\endgroup$ – Basj Nov 30 '15 at 14:01
  • $\begingroup$ Right, so the question I'm asking is: "outlier of what?". :-) Taking direct median or mean of the data you have will yield nonsensical results. Median filtering only works well when a few samples (in relation to the window length) are outside the expected range. Because the data you have is swinging positive and negative, the median isn't really appropriate for it. Taking the median of the envelope should work better. You may even want to put a simple low-pass filter on the absolute value to get a steadier envelope value. $\endgroup$ – Peter K. Nov 30 '15 at 14:05
  • $\begingroup$ Watch this space, I'll try to answer your follow-up question. $\endgroup$ – Peter K. Nov 30 '15 at 14:06
  • $\begingroup$ No the data is not swinging positive and negative, because I take the absolute value of the signal at the beginning of the code : (fs, x) = read('test.wav') x = abs(x) $\endgroup$ – Basj Nov 30 '15 at 14:09
  • $\begingroup$ See this answer, first bullet in the list, he suggests using median instead of moving average to have less effect of outliers. $\endgroup$ – Basj Nov 30 '15 at 14:09
0
$\begingroup$

What you said is correct - moving median is less sensitive to outliers, where an outlier is usually a single point in time series that is very different from all others, which may be due to some kind of error. Moving median filter simply removes outliers from the result, where moving mean/average always takes into account every point. However, moving median can be even more sensitive to short term significant spikes that span several points, especially when they span more than half of the moving window.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.