Idea for Noise Level Estimation / Automatic Thresholding in the Presence of Peaks

Question

I have the fft of some signal, and want a rough estimate of the noise level in order to choose an appropriate threshold for our peak detection algorithm. In general, the fft contains mostly noise with a handful of peaks (which are usually pretty high compared to said noise). For reference, I've attached a screenshot of a pretty typical fft:

Now to my proposed algorithm for noise level estimation, which is based on the following assumptions:

• What i want is the mean and standard deviation of the fft
• If the fft contained only (gaussian) noise, the median would be very close to the mean
• Peaks shouldn't affect the median too much
• The mean and standard deviation are independent of the order of the samples
• Therefore i can sort the fft and still get the same result

So the resulting algorithm looks like this:

• Sort the fft (ascending)
• Calculate the mean and standard deviation of the sorted fft, but stop as soon as the mean exceeds the median
• Use the calculated mean and standard deviation to choose the peak detection threshold
• Profit?

I've tested this algorithm on some typical signals and the results seem to be pretty decent (keeping in mind that i don't need the exact noise level, but just something robust to choose an appropriate threshold). Sorting is a little expensive of course, but i don't expect the noise level to change too much, so i don't need to run it for every fft.

That being said, the algorithm itself feels kind of "wrong", because i don't ask the actual question "What would the mean and standard deviation be without the peaks?" but rather "What would the standard deviation be if the mean was X?".

So what are your thoughts on this? Are there better algorithms for this kind of problem? (i bet there are)

Answer 1

I'd do some small adjustments to your idea (You really nailed them).

Assumptions

• The Signal Model - Signal + Additive White Gaussian Noise (AWGN)
  Probably we could generalize it more but this is beyond the scope of this question.
• The DFT of the signal contains Peaks with relatively small roll off
  This is important as we're almost saying the Signal is a combination of sparse number of Harmonic signals. As the lobes width will hide the noise data. So the ratio between the observation window, signal lobes width and sampling rate means that the DFT of the data creates sharp peaks.

Estimation

Under the above assumptions we know that:

• Noise is AWGN in the DFT (As Gaussian Vector stays Gaussian under linear transformations). It also suggests that the Mean Value and the Median are similar (Realization of Symmetric Distribution).
• Without the Signal Data the Mean and the Median value of the beans should be similar.
• Number of bins contaminated with signal (Funny to write this, ha?) is sparse.

So we can sort the bins, but sorting and evaluating the mean on sub set will create bias unless we make sure the sub set is symmetric around the Mean / Median. So, I'd sort the data and keep bins in the 25-75 Percentile range only. I'd use Mean on those.

Answer 2

There are indeed many peak detection algorithms, and no clear consensus on which ones are "good" or "bad". But for what it's worth, your approach makes sense. Using median or other quantiles to detect sparse signals is common, e.g. the "median clipping" stage in Lasseck (2014), Large-scale identification of birds in audio recordings. In effect, you're treating the FFT magnitudes as coming from a mixture of signal and noise, and since the signal is rare you assume that the median should be close enough to the true median of noise alone.

However, the standard deviation, like the mean, will be affected by the presence and magnitude of signal. Depending on your problem and assumptions, potential solutions could be to use a different frequency band or known silence periods to estimate the noise.

(It also seems like you don't need to calculate the mean at all, if you already assumed the noise median and mean to be similar.)