Onset Detection for Audio Signals

Question

I am having a problem understanding how the author gets the onset envelope of an audio song in this paper.

The entire section in question has been quoted below:

First the input sound is resampled to 8 kHz, then we calculate the short-term Fourier transform (STFT) magnitude (spectrogram) using 32 ms windows and 4 ms advance between frames. This is then converted to an approximate auditory representation by mapping to 40 Mel bands via a weighted summing of the spectrogram values [Ellis, 2005]. We use an auditory frequency scale in an effort to balance the perceptual importance of each frequency band. The Mel spectrogram is converted to dB, and the first-order difference along time is calculated in each band. Negative values are set to zero (half-wave rectification), then the remaining, positive differences are summed across all frequency bands. This signal is passed through a high-pass filter with a cutoff around 0.4 Hz to make it locally zero-mean, and smoothed by convolving with a Gaussian envelope about 20 ms wide. This gives a one- dimensional onset strength envelope as a function of time that responds to proportional increase in energy summed across approximately auditory frequency bands

My understanding of the procedure of the first part of the procedure is that our 44.1KHz audio signal is resampled to 8KHz and then we perform a overlapping widowed STFT operation on it so we get the Fourier transform coefficients.

But I really have no idea what is happening in the second part of the procedure:

I think we then alter those coefficients in such a way that the new Fourier transform coefficients are now balanced perceptually(ie the coefficients are now what a person's ear would perceive them to be).Then the so-called resultant "Mel spectrogram" is converted to dB, and the first-order difference along time is calculated in each band.Negative values are then set to zero (half-wave rectification),then the remaining, positive differences are summed across all frequency bands.

Why and how would we calculate the first order difference? What use is this? I have no extensive knowledge of the Mel scale and it's so-called "frequency bands" so this part is proving hard to understand.

This signal is passed through a high-pass filter with a cutoff around 0.4 Hz to make it locally zero-mean, and smoothed by convolving with a Gaussian envelope about 20 ms wide. This gives a one- dimensional onset strength envelope as a function of time that responds to proportional increase in energy summed across approximately auditory frequency bands

Why do we need the signal to be locally zero-mean?

Answer 1

First of all, the title of your question is misleading because onset detection is very different from envelope detection. The onset detection function won't tell you how long a note is played or how slow its attack is... the envelope will. If the problem you are trying to solve is envelope detection, you might get inspiration from onset detection literature, but at some points the methods diverge. Onset detection extracts a "cartoon" version of the envelope in which all sudden changes are emphasized to make their detection easy.

I think we then alter those coefficients in such a way that the new Fourier transform coefficients are now balanced perceptually(ie the coefficients are now what a person's ear would perceive them to be)

This is done by summing the energy (squared magnitude) of the FFT across 40 ranges of frequencies defined by the mel scale - usually with a triangular shape to emphasize a different center frequency.

Check point 4 of my answer to this question.

Why and how would we calculate the first order difference?

This is the $1 - z^-1$ filter.

So the first order difference is simply the energy in a given Mel band at time t minus the energy in the same Mel band at time t-1.

We are interested in detecting onsets. Onsets are mostly characterized by a sudden rise of energy across the entire spectrum. A positive value of the first order difference indicate that energy is rising in one band. When summing across all bands, this gives a value that is very likely to be high when there is a note onset.

We do this detection across 40 frequency bands rather than in the time domain to detect events that would be masked by loud sounds in the time domain. For example, imagine a recording in which a high-hat cymbal sound is played on top of a loud, droning bass synth. In the time domain, the envelope of the signal will be a solid line with just a tiny wiggle when the high-hat plays, because the most dominant sound is the bass. But in the frequency domain, the bass note will only occupy the lowest frequencies, and it'll be easy to notice the attack of the high-hat sound, extending from mids to high frequencies.

Why do we need the signal to be locally zero-mean?

It makes it easier to detect onsets by thresholding. The high-pass filter (or subtraction of median, which is another commonly used detrending method) will make sharp peaks in the detection function more prominent.