What is really the Mel-filterbank?

Question

After applying STFT to a signal, you typically need to convert the frequencies to the Mel scale. Typically the frequency values range from $0$ to $N/2$, where $N$ is the width of each sliding window.

I have read so many articles about Mel Spectrogram. Basically they all say to convert frequencies to Mel Scale in the following way:

1. Partition the Hz scale (i.e. partition a sequence of numbers ranging from $0$ to $N/2$). In this way we will have a number of bins. This number is for example indicated in librosa by "n_mel". These bins are called frequency bands.

2. Covert each bins into Mel Scale using the formula $2595*log(1+f/700).$

3. Application of triangular filters for each bins to capture the energy at each frequency band and approximate the spectrum shape.

It seems that when this is buried in the articles, nobody really knows why these operations are carried out. I find it difficult to fully understand:

1. Why divide the Hz scale into bins?

2. What exactly is meant by a frequency band?

3. Why do we apply triangular filters?

4. What exactly is a triangular filter? How is it defined mathematically and from what concept?

Can anyone explain exactly these things to me or recommend a book where these steps are explained in detail? There seems to be no book that explains these concepts exactly.

Answer 1

As I am a newbie in DSP and I thought this article has a clear explanation, I did not try to answer your questions myself. But you said you have read it and thought these questions are still ambiguous, then I want to try one more time to give my answers.

First, about the concepts of bins and bands, they confused me a lot in the beginning when I was trying to figure out what the Fourier Transform means, especially the Discrete Fourier Transform (DFT), which is also much related to your questions. Right now my understanding of frequency bins are specific frequency values, such as 2000, or 2001.5. A frequency band, for example from 1000 to 2000, covers a range of frequencies, and it is defined by the two frequency bins 1000 and 2000 (one is the floor, and one is the ceiling). In the physical world, the values of frequencies can take any real number, in other words, they are continuous. As to the above example, there are infinite values or frequency bins in the frequency band [1000,2000]. However, when comes to digital signal processing, in the time domain, we take finite (and integer) numbers of samples with a sampling rate, for example 44100 samples per second (i.e. 44100Hz). Later we do Discrete Fourier Transform (DFT) in ways of FFT or STFT to go to the frequency domain. When we do STFT, we do it in the unit of frame and apply hanning window to it. As you mentioned the width of the window equals to the frame size such as 2048 samples, and 2048 is your N.

As to your questions:

1. It is not that we further partition a frequency scale into bins, naturally we get N/2 frequency bins after FFT or STFT. If N = 2048, sampling_rate = 44100, then we have 1024 frequency bins [21.53, 43.07, 64.60, ..., 22028.47, 22050]. Our finite and discrete sampling in the time domain has already limited the frequencies we get to be finite and discrete too, as bins.

2. As explained above, usually we filter(maybe can be understood as combine or group) the frequency bins to get bands.

3. Naturally we have more than one way to group bins to bands, I think the triangular filter is just one specific (Mel) way to get Mel spectrogram.

4. As to this question, the above tutorial definitely explains better than me.

About the question 3, I am not sure on what level you are asking about the reason to do the triangular filter. If it is about the choice of the specific technique/algorithm, I think it is related to the question 4, and I will leave it to the tutorial. If it is about the bigger reason to do Mel transform, this comes to the field of human sensation and perception. When we talk about frequency and amplitude of the sound source,these are pure physical features of sound. Pitch and loudness are their corresponding human perceptual features. The point is our human perceptions do not linearly change with the values of the physical features. That is the reason we have Decibel scale and Mel scale, therefore our perception change linearly with the values in these scales. We only need to do some math transforms to find the real physical values corresponding to these scales or vice versa.

I hope these can offer some help.

Answer 2

This was my first google hit: https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53

It seems to explain both how and why?

Mel frequency binning is afaik only used when human perception is involved. Either directly, or for signals that have probably been adapted to human hearing (such as human speech). It is a way to represent signal information in a way that mimics «low level auditory perception», or could be seen as removing perceptually redundant information.

I have not understood why it is that we generally have the same temporal reolution in narrow low frequency bands, as in wide high frequency bands. Historically, it may have been due to limited processing power and the efficiency of a single STFT windowing process. And/or it could reflect the limited neuron firing rate after the acoustic-mechanical filtering carried out by our ear.

But from a dsp perspective, I would expect a variable time/frequency resolution, where high & wide bands gets more temporal resolution.

Answer 3

The below tutorial has the clearest explanation I have read about Mel, I think it can answer all of your questions.

Mel Frequency Cepstral Coefficient(MFCC) tutorial

Answer 4

A filterbank is a way to discretize a continuous frequency response into bins.

Which filterbank you choose depends on the type of application. For example, the Mel filterbank is preferred when the goal is human centric perception.

Librosa enables you to create different filterbanks. I would assume that

• a Mel filterbank might be appropriate for instrument transcription (e.g. transcribing a piano recital to the pitches played)
• A Chroma filterbank might be appropriate for pitch class (only 12 pitches, as opposed to 88 for piano transcription).

If you were trying to classify industrial sounds or even bird sounds, you may want to experiment with other filterbanks.

Staying with the choice of Mel filterbank, the number of bins you choose depends on the application. For recognizing speech, the paper linked above suggest 26 bins. For Piano transcription, this paper on Onsets and Frames suggests 229 bins.

Answer 5

@Franc, I can see from your question that you are very good at thinking. Let me extend your question

1. Why use mel spectrogram?
2. Why use filter banks?

I will reply to your question first for reference only.

set sr is samplate, fftLength is singal length，The effective data length after DFT or FFT is fftLength/2+1.

Re:1+2

1. Why divide the Hz scale into bins?
2. What exactly is meant by a frequency band?

First, bin can be understood as index,

import numpy as np

sr=32000 # samplate
fftLength=4096 # dataLength 0.128ms

freArr=np.linspace(0,sr/2,fftLength/2+1)

freArr is DFT's frequency band, bin is freArr's index.

Secondly, the first step for mel spectrogram is to calculate mel frequency band, the order is:

hz fmin, fmax ---> mel scale fmin, fmax ---> mel scale freArr ---> hz freArr

import numpy as np

sr=32000 # samplate
fftLength=4096 # dataLength 0.128ms

mel_num=512 # mel number

# fre range
fmin=0 # index/bin=0
fmax=16000 # index/bin=2049

freArr=np.linspace(0,sr/2,fftLength/2+1)

# hz fre ---> mel scale fre
fmin_mel=2595*np.log10(1+ fmin/700);
fmax_mel=2595*np.log10(1+ fmax/700);

# mel scale freArr
freArr_scale=np.linsapce(fmin_mel,fmax_mel,mel_num)

# mel scale freArr ---> mel frequency bands
freArr_mel=700*(np.power(10, freArr_scale/2595)-1)

# Why divide the Hz scale into bins?

I guess your problem may be this. This bin is freArr_mel corresponds to the closest index of' freArr '. The answer is that it will be used in subsequent business.

Re:3+4

1. Why do we apply triangular filters?
2. What exactly is a triangular filter? How is it defined mathematically and from what concept?

In the field of digital signal, filter is a basic and important concept. One of the filter design methods is the window function method. Of course, this is not about filter design, but assuming that you have this part of the basis, triangular window function formula is

$N$ is odd，

$\quad w(n)=\begin{cases} \cfrac{2n}{N+1}, & 1 \le n \le \cfrac {N+1}{2} \\ 2-\cfrac{2n}{N+1}， & \cfrac {N+1}2 \le n \le N \end{cases} $

$N$ is even，

$\quad w(n)=\begin{cases} \cfrac{2n-1}N, & 1 \le n \le \cfrac N{2} \\ 2-\cfrac{2n-1}N， & \cfrac{N}2 +1 \le n \le N \end{cases} $

This is what it looks like

Of course, you can not use triangular filter. These window functions, such as hann, hamm, etc, can also be used, even without any filter.

The following is a spectrum comparison diagram of different windows

In the figure, Slaney and ETSI are the renderings of triangular windows. If you are interested in research, you can use python's audioFlux try various contrast effects such as hann, hamm, gauss, etc. mel is a scale of auditory design, and of course there are also scales such as bar/erb/octave, audioFlux include the following types of spectrum.

• linear spectrogram
• mel spectrogram
• bark spectrogram
• erb spectrogram
• octave spectrogram
• log spectrogram