Confusion with regards to STFT and MFCCs

Question

I'm wondering if someone can clear the following up for me as I'm having a difficult time understanding the concept.

I'm looking to implement and learn about the processes and uses using short-time Fourier transforms. I have had experience using DFT's and wanted to expand my knowledge.

From research that I have read online, as well as in some text books I understand that the STFT is calculated by the following:

1. Take the inputted signal of $x[n]$

2. Multiply $x[n]$ by a window function (e.g. Hamming, Hanning, Triangular, .., Barlett) by sliding the block size over the time axis

3. Compute the DFT for the resulting values of step 2

Is this correct? I know the following steps above are for usually a continuous-time STFT and in the case of a Discrete-time STFT the data can be broken up into chunks.

The other thing that is confusing me:

For a project last year, I had to implement a MFCC algorithm. Inside this algorithm I computed the "Mel Scale" in Triangular filter bank and then multiplied each of these values against the FFT (DFT) computed on the inputted signal in Discrete-Time. Is this therefore a use of the STFT, just using the Mel-Frame over Hamming etc.. Or, would I still need to multiply the result of the FFT (DFT) with the Triangular filter bank if I was to use a STFT in order to calculate the MFCC values?

Thank yo for reading.

Answer 1

I am going to break down this answer into two main parts, one for a step-by-step computation of the STFT, and the other in regards to how to compute the MFCC's, once you have the STFT.

STFT: Let us say you wanted to compute the STFT of your signal $x[n]$, of total length $N=100$, and block size $M = 10$, with $50$% overlap, and using a hamming window. I will give you a step by step, (and redundant) flow, to get you started:

1. Take the first $M=10$ samples out, that is, samples $x[0], x[1], x[2] ... x[9]$.
2. Window them, by multiplying them, element by element, with your window coefficients.
3. Take the DFT of this result.
4. Take the next $M=10$ samples out, that is, samples $x[5], x[6], x[7], ... x[14]$.
5. Window them, by multiplying them, element by element, with your window coefficients.
6. Take the DFT of this result.

etc, etc. You get the idea. Let us say that you took a (same size) DFT in the above example. Then, you will end up with a $10$ x $19$ complex matrix. If you then simply take the absolute magnitude of the above matrix, you will end up with a real matrix, corresponding to the absolute magnitude of your complex matrix. This is called the Short-Time-Fourier-Transform, or STFT. We will use the STFT as the input into computation of the MFCCs.

Before that, let us get some formulas and terminology straight: Let us call our STFT matrix $S$. This matrix is composed of $B=19$ blocks, or columns, and $K=10$ frequency bands, or rows. To refer to a time-frequency point in $S$, we will say $S[k,b]$, where $b \in 1, 2, ... B$, temporal blocks, (columns) and $k \in 1, 2, ... K$ sub-bands, (rows).

MFCC:

Once you have the STFT computed, you can go ahead and use that as a stepping stone for computing the MFCC's. Regarding your question, you seem to be confusing the time domain windowing done on each block for the STFT, and the frequency domain windowing done in the MFCC. They are completely separate and unrelated.

Let us say that you computed your STFT using a hamming window, and you have a $K=500$ x $B=1000$ real matrix. Thus using our prior terminology, you have a real matrix $S_{K \text x B}$. Thus, you have $500$ sub-bands, (frequencies), and $1000$ blocks, (time).

Let $Y_{b}[k] = |S[k,b]|^2$. That is, we are letting $Y_{b}[k]$ represent the power-spectral density of a particular column/block of the STFT matrix $S$. Obviously, each $Y_{b}[k]$ is a $500$ x $1$ column vector.

Clearly, we have $1000$ $Y$'s. That is, we have $Y_{1}[k], Y_{2}[k], Y_{3}[k], ... Y_{1000}[k]$.

Now, you want to compute the MFCC of each column. Take the first column. This is $Y_{1}[k]$. Forget about all the other columns, and forget about the hamming window. The first column is now used in your MFCC algorithm. The first step in MFCC was getting a PSD column, which you have, because we have $Y_{1}[k]$.

The second step is the computation of, say, 10 power estimates. This is computed by windowing this column, by 10 overlapping windows, as per the mel-scale, like so:

Note how each (triangular) window is mostly zeros. So now you literally do a dot product of each window, with $Y_{1}[k]$. You will get $10$ numbers, for this column. Now take the log of those $10$ numbers. Now take the DCT of this result. Congratulations! You just computed the MFCC for this column. You just computed the MFCCs for $Y_{1}[k]$.

Now, go back, and repeat for all the other $Y_{2}[k]$. Then repeat for $Y_{3}[k]$, etc.