Moving average vs. Moving median

Question

I have read in many places that Moving median is a bit better than Moving average for some applications, because it is less sensitive to outliers.

I wanted to test this assertion on real data, but I am unable to see this effect (green: median, red: average). See here:

---

Here's the sample audio data test.wav. Here's the Python code:

import numpy as np
from scipy.io.wavfile import read
import matplotlib.pyplot as plt

def median(lst): return np.median(np.array(lst))
def mean(lst): return sum(lst)/len(lst)

(fs, x) = read('test.wav')

x = abs(x)
env = np.zeros_like(x)
env2 = np.zeros_like(x)

for i in range(len(x)):
    env[i] = median(x[max(i-1000,0):i+1])
    env2[i] = mean(x[max(i-1000,0):i+1])
 
plt.plot(range(len(x)), env, color = 'green')
plt.plot(range(len(x)), env2, color = 'red')
plt.show()

I have tried with various values for Window width (here in the code : 1000), and it was always the same: the moving median is not better than moving average (i.e. not less sensitive to outliers).

---

The same with Window width = 10000 (10000 >> the spike width) :

---

Question:

Can you provide an example showing that moving median is less sensitive to outliers than moving average? And if possible using the sample .WAV file data-set (download link).

i.e. is it possible to do a moving median on this data such that the result is like this yellow curve? (i.e. no more spike!)

Answer 1

This isn't really an answer, but I thought I'd report what I'm seeing and ask for more information.

I've loaded your test.wav file and I can see the signal plotted below.

So what you're getting in the plots you show is not so much the median value, but is more like an envelope of the signal.

The second issue is that the signal actually seems to be part of the signal. If I zoom into the blip then this is what I see:

What are you really trying to achieve?

---

Thanks for the feedback. Below is some R code that does the following:

• Loads the wav file.
• Takes the absolute value of the signal (it's loaded into the left channel by the waveR library).
• Performs a 100-length moving average filter on the data to get something closer to the "envelope" (red signal).
• Then applies a median filter of lengths 201, 2001, and 4001 to the result (blue signal).

From the plot below, the best performing is the 4001 length one. Otherwise the effect of the glitch is still present.

The only thing I can see wrong now is that the "envelope" doesn't match the true envelope as well as I'd like. A better envelope detector might improve this (e.g. the analytic signal or such).

Below is a plot of the calculated median-filtered envelope overlaid on the original signal.

---

R Code Below

#27349
#install.packages("tuneR")
library(tuneR)
x <- readWave("Q27349/test.wav")    
sig <- x@left    
abssig <- abs(sig)    
N <- 100
filtabssig <- filter(abssig, rep(1/N,N))    
plot(abs(filtabssig))    
med <- 0*filtabssig    

N2median <- 1000 # Window length is 2 * N2median + 1
for (k in 1:length(filtabssig))
{
  idxs = seq(max(1,k-N2median), min(length(filtabssig), k+N2median),1)
  med[k] = median(filtabssig[idxs])
}

par(mfrow=c(3,1))
plot(filtabssig, col="red")
lines(med200,col="blue")
title("Median filter length of 201")

plot(filtabssig, col="red")
lines(med1000,col="blue")
title("Median filter length of 2001")

plot(filtabssig, col="red")
lines(med2000,col="blue")
title("Median filter length of 4001")

Answer 2

Is moving median always less sensitive to outliers? Sometimes. It will work if you have a very short spike (preferrably shorter than the median/average sample size). However, if you have a large spike, then taking the median won't help eliminate the spikes.

Long spike

I have illustrated this using a dataset (with sample size = 5, taking into account 2 left entries and 2 right entries):

Orig. Median Avg.
Data  Filter Filter
-----------------
1       
2       
3      3    3
4      4    4
5      5    11.6
6      6    19
40     40   26.2
40     40   33.2
40     40   33.4
40     40   27
7      9    20.8
8      9    14.8
9      9    9
10     10   10
11      
12

Notice how the the median of the all the 40s is 40. For example, take the 1st 40. The left values are 5,6 and the right values are 40,40, so we get a sorted dataset of 5,6,40,40,40 (the bolded 40 becomes our median filter result).

Short spike

You also wanted an example for the median filter to work. So, we will have a short spike. Try this:

Orig. Median Average
Data  Filter Filter
-----------------
1       
2       
3      3    3
4      4    10.8
5      5    11.6
40     6    12.4
6      7    13.2
7      8    14
8      8    8
9       
10

You can see that the median filter did filter out the single large spike, while the spike skewed the average filter's results for entries 4,5,40,and,6.

Answer 3

What you said is correct - moving median is less sensitive to outliers, where an outlier is usually a single point in time series that is very different from all others, which may be due to some kind of error. Moving median filter simply removes outliers from the result, where moving mean/average always takes into account every point. However, moving median can be even more sensitive to short term significant spikes that span several points, especially when they span more than half of the moving window.