How to stitch together MFCCs from multiple frames?

Question

I am trying to record a sound two times and compare the two recordings. I am playing exactly the same sound both times (for example, play a small .wav file twice and record it each time), so I hoped that my attempt would not rise to the complexity of Speech Recognition...

I have tried computing the DFT (using FFT) and comparing the two power spectrums of the signals (more precisely, comparing their MSE - Mean Squared Error), but the results were discouraging (sometimes for recordings of the same sound the error was greater than for different sounds)...

Then I found out about the MFCCs and computed them for each signal. That leaves me with 12 coefficients for each frame I devided my signal into (details below). How should I combine these groups of 12 coefficients to get some value(s) that describe the whole signal?

(Appending them and extracting the MSE for the two arrays did not return encouraging results either...)

---

Below are some details about the way I computed the MFCCs and values I chose for computing it. Maybe it's here that I'm doing something wrong:

(sample rate = 22000 Hz)

1. Pre-emphasis.
2. Framing with frame size = 512, frame overlap = 200. (because "each frame should be ~20-30ms long" and 512 / 22000 = 0,023 seconds = 23 ms. As "the overlap should be of ~10ms and 200 / 22000 = 0,009s = 9ms)
3. Apply Hamming window to each 512-size frame.
4. Apply FFT to each windowed 512-size frame => 512 magnitudes for my signal, from witch I use only the first 256. Domain: Hz from 0 to 22000 (to 11000, respectively)
5. Compute the Mel Filter Bank: (min frequency = 300 Hz, max frequency = 11000 Hz)
   • Compute mels from min and max frequences and then compute 26 equally distanced values between these two mel values.
   • Convert them back to Hz => array of 28 frequency filters.
   • Compute a filterbank (filter triangle) for each three consecutive values => 28 - 2 = 26 filterbanks.
   • Pass the whole power spectrum (256 magnitudes) from step 4 through each triangular filter to get a "filterbank energy" for each filter => 26-sized array of energies.
6. Apply log: ln(each energy) (they are still frequency-domain values)
7. Apply DCT to the logged energies => 26-sized array of (time-domain?) values.
8. Take only the first 12 => 12 MFCCs for each 215-sized frame of my signal.

Answer 1

How should I combine these groups of 12 coefficients to get some value(s) that describe the whole signal?

I don't think "flattening" the sequence of $215 \times 12$ coefficients into a $1 \times N$ vector, and then computing the distance between these vectors to compare two sounds is a good idea. This approach collapses the temporal information - the way a sound evolves in time. For example, let us say that you decide to use the standard deviation and mean of each coefficient to "summarize" the sequence of MFCC vectors (this is a technique used in some applications). If you use this approach, both a sound and the same sound played backwards will yield exactly the same feature vector! This might work for simple tasks like recognizing different music instruments, but as soon as the temporal information needs to be preserved (distinguishing "halo" and "hola", or a drum instrument playing "boom boom tchik" vs "boom tchik boom"), thinking in terms of vectors rather than sequences no longer works.

This might not be the answer you expect, but I suggest you to stop reasoning in terms of vector comparison, and rather think of a way of comparing the sequences of MFCC frames themselves. Your approach - which is akin to computing the raw pairwise distance between sequence of vectors - is not robust to difference in timing between the two sequences. The solution is to use Dynamic Time Warping. It attempts to optimally align the two sequences in time (so that comparing "hoooolaa " with " holaaa" yields the same score as comparing "hola" and "hola"). This technique is good enough for basic voice command recognition.

Another approach would be to model each sound by a HMM with a few states; and to use the distance between models as a similarity metric. This can be more computationally efficient, and put you in control of how strict you want to respect temporal information (by increasing the number of states).

Answer 2

You could try homebrewing something similar to the basic idea of sound fingerprinting. Create a 2D array (maybe even graph it to take a look) of your Mel energies, pick out all the local maxima, and then compare the 2D distances between the nearest peak pairs between the two graphs as a similarity measure.