How should I remove obvious anomalies from spectral data?

Question

I'm working with raman spectra and it seems that our spectrometer produces some anomalies at very specific wave numbers and I'm looking to remove them from the dataset. Here's a picture of 27 spectra of the same substance and below it a picture of the variance at the individual wave numbers.

From looking at the variance graph it seems fairly obvious where the anomalies are (ca. at wave numbers 480, 800, 1500, 1650-1700 and 2150).

I ultimately want to use the data for machine learning. So far I've implemented an algorithm that looks where the variance exceeds a certain threshold and then removes outlier spectra at that point (it removes the entire spectrum) until the variance doesn't exceed the threshold at any point. The problem with this algorithm is that it removes around 60% of the spectra which is acceptable but not ideal.

I'm thinking about a different method where I delete the information at all the wave numbers where the variance exceeds a certain threshold (like cutting out a vertical strip from the first graph where the variance is too high). Since most of the anomalies are relatively "narrow" I think I would't lose much information.

I want to know if one of these methods is clearly better than the other and why. Also I would like to know if there are any common methods used for dealing with this kind of data.

Answer 1

This seems to be a textbook case for a narrow median filter. Your $S$ is very broadband compared to your $N$, so a median filter will not affect your signal, as it can be designed quite narrow. And by using median instead of averaging (i.e. lowpass) you minimize the error introduced by the anomalies. I would give it a try.