How to stitch together MFCCs from multiple frames?

I am trying to record a sound two times and compare the two recordings. I am playing exactly the same sound both times (for example, play a small .wav file twice and record it each time), so I hoped that my attempt would not rise to the complexity of Speech Recognition...

I have tried computing the DFT (using FFT) and comparing the two power spectrums of the signals (more precisely, comparing their MSE - Mean Squared Error), but the results were discouraging (sometimes for recordings of the same sound the error was greater than for different sounds)...

Then I found out about the MFCCs and computed them for each signal. That leaves me with 12 coefficients for each frame I devided my signal into (details below). How should I combine these groups of 12 coefficients to get some value(s) that describe the whole signal?

(Appending them and extracting the MSE for the two arrays did not return encouraging results either...)

Below are some details about the way I computed the MFCCs and values I chose for computing it. Maybe it's here that I'm doing something wrong:

(sample rate = 22000 Hz)

1. Pre-emphasis.
2. Framing with frame size = 512, frame overlap = 200. (because "each frame should be ~20-30ms long" and 512 / 22000 = 0,023 seconds = 23 ms. As "the overlap should be of ~10ms and 200 / 22000 = 0,009s = 9ms)
3. Apply Hamming window to each 512-size frame.
4. Apply FFT to each windowed 512-size frame => 512 magnitudes for my signal, from witch I use only the first 256. Domain: Hz from 0 to 22000 (to 11000, respectively)
5. Compute the Mel Filter Bank: (min frequency = 300 Hz, max frequency = 11000 Hz)
• Compute mels from min and max frequences and then compute 26 equally distanced values between these two mel values.
• Convert them back to Hz => array of 28 frequency filters.
• Compute a filterbank (filter triangle) for each three consecutive values => 28 - 2 = 26 filterbanks.
• Pass the whole power spectrum (256 magnitudes) from step 4 through each triangular filter to get a "filterbank energy" for each filter => 26-sized array of energies.
6. Apply log: ln(each energy) (they are still frequency-domain values)
7. Apply DCT to the logged energies => 26-sized array of (time-domain?) values.
8. Take only the first 12 => 12 MFCCs for each 215-sized frame of my signal.
• May I suggest an edit to the title? "interpret" made me think that the question was about "what do the MFCCs mean? what do they represent?". Instead, the question is more about "how to compare MFCCs?" – pichenettes Mar 22 '13 at 20:26
• Are your Mel filters linearly or logarithmically spaced? – hotpaw2 Mar 23 '13 at 4:04
• @hotpaw2 The initial filters (expressed in mels) are linearly spaced, but from these I compute a set of logarithmically spaced filters (expressed in Hz - the first filters converted from mels to Hz). It's these last ones (expressed in Hz, logarithmically spaced) that are used further on. Did I get something wrong in this whole process? – Ioanna Mar 23 '13 at 10:23
• @loanna 1. How do these two recordings differ? Is the recording environment different? 2. Can you share sample recordings (say over dropbox)? – user13107 Mar 25 '13 at 4:47
• @user13107 I'm using NAudio to record .wav files. As a first test, I'm simply playing a wave file, recording it twice and then comparing the results. – Ioanna Mar 25 '13 at 6:36

How should I combine these groups of 12 coefficients to get some value(s) that describe the whole signal?

I don't think "flattening" the sequence of $215 \times 12$ coefficients into a $1 \times N$ vector, and then computing the distance between these vectors to compare two sounds is a good idea. This approach collapses the temporal information - the way a sound evolves in time. For example, let us say that you decide to use the standard deviation and mean of each coefficient to "summarize" the sequence of MFCC vectors (this is a technique used in some applications). If you use this approach, both a sound and the same sound played backwards will yield exactly the same feature vector! This might work for simple tasks like recognizing different music instruments, but as soon as the temporal information needs to be preserved (distinguishing "halo" and "hola", or a drum instrument playing "boom boom tchik" vs "boom tchik boom"), thinking in terms of vectors rather than sequences no longer works.

This might not be the answer you expect, but I suggest you to stop reasoning in terms of vector comparison, and rather think of a way of comparing the sequences of MFCC frames themselves. Your approach - which is akin to computing the raw pairwise distance between sequence of vectors - is not robust to difference in timing between the two sequences. The solution is to use Dynamic Time Warping. It attempts to optimally align the two sequences in time (so that comparing "hoooolaa " with " holaaa" yields the same score as comparing "hola" and "hola"). This technique is good enough for basic voice command recognition.

Another approach would be to model each sound by a HMM with a few states; and to use the distance between models as a similarity metric. This can be more computationally efficient, and put you in control of how strict you want to respect temporal information (by increasing the number of states).

• Thank you, @pichenettes! Dynamic Time Warping was exactly what I was thinking to try next (first the idea came from this paper ). However, it is still unclear what I should compare: I understand that I should compare the two signals frame-by-frame, but should I take only the 12 MFCCs from the frames? They seem so few... Do you think that computing the whole 39 coeficients (with energies, deltas and double deltas) whould help? And then use DTW to compare each 39 coefs from each frame of one signal to the other? – Ioanna Mar 23 '13 at 10:45
• Given that this will multiply computation time by a factor of 3, you'd better define a performance metric for your task (such as a recognition rate) and evaluate if adding the deltas has an impact. – pichenettes Mar 23 '13 at 11:09
• Here is what I ended up doing: I computed the Euclidian Distance between the MFCCs from each possible couple of frames, one from the first signal, one from the second. That resulted in a 2-dimension array on which I applied the DTW algorithm. I took the last computed value as a measure of the difference between the two signals. The results seem promising, but I'll have to analyze them a little deeper. If I am doying something wrong please let me know. Thank you for your suggestions! – Ioanna Mar 23 '13 at 14:26
• Yes, this is the right approach. To save up on computation time you might replace all distances far away from the diagonal of the array by very large numbers. – pichenettes Mar 23 '13 at 14:30
• could you please share your code if possible, @loanna. I'm working on a similar problem but haven't made much progress. i want to use mfcc for sound classification and recognition. – kRazzy R Jan 24 '18 at 14:19

You could try homebrewing something similar to the basic idea of sound fingerprinting. Create a 2D array (maybe even graph it to take a look) of your Mel energies, pick out all the local maxima, and then compare the 2D distances between the nearest peak pairs between the two graphs as a similarity measure.

• If I understand well, this seems similar to applying DTW to the MFCCs (does it? :) ). This is what I just tried and it seems to work well, though I'll have to look into the results a little more. Thank you for the idea! – Ioanna Mar 23 '13 at 14:33
• could you please share your code if possible, @loanna. I'm working on a similar problem but haven't made much progress. i want to use mfcc for sound classification and recognition. – kRazzy R Jan 24 '18 at 14:20