I am trying to compare 2 speech samples and rate them on similarities. Think of someone trying to repeat a phrase, and then comparing those 2 audio files.

I started by implementing the MFCC (http://en.wikipedia.org/wiki/Mel-frequency_cepstrum) algorithm. I calculate the MFCCs of both audio samples, which gives me roughly 500 frames of audio (at 10ms each, with like 30% overlapping of the previous) having 14 or so MFCC coefficients. So a 500x14 matrix for each audio signal.

Then I do the naive approach of simply differencing the matrices. This does not give very promising results. Half of the time when I compare completely different audio samples (where different phrases are spoken), I get less difference than comparing the audio where I try to repeat the same phrase! This is clearly backwards and can't give me a good scoring algorithm.

How can I improve on this? I thought MFCCs were a really important part of speech processing, though clearly I need to do more with it.

  • $\begingroup$ Hey I am dealing with a similar problem, what is the approach if I should be comparing voices of two different speakers in order to obtain a similarity score ? Also , is there a way to directly compute if audio clips are originating from the same person or not ? $\endgroup$ May 31, 2019 at 9:55

1 Answer 1


First, you will have to correct for differences in timing. For example, if one utterance is "--heeelloooo---" and the other "hellooooooo----" (- representing silence), a direct pairwise comparison of MFCC frames will show differences simply because the two samples are not aligned. You can use Dynamic Time Warping to find the best alignment between the two sequence of feature vectors - and compute the corresponding distance.

The second problem is that if the two recordings are not from the same speaker, you will have to compensate for the differences in timbre. The MFCC of a female saying "aaa" are not the same as the MFCC of a male speaker saying the same phoneme! A relatively simple model to account for variations in voice timbre is to assume that there exists a linear transform $\Gamma$ which "maps" the MFCCs of one speaker onto the MFCCs of another speaker (to be fair, only one small subset of these transform accurately model how changing parameters such as age, gender, etc "shifts" the MFCC). Once two recordings have been aligned, even roughly, you can use a least square procedure to estimate $\Gamma$. This procedure is known as speaker normalization or speaker adaptation.

Your comparison procedure will thus consists of the following steps. $A$ and $B$ are your original MFCC sequences.

  • Align the two utterances using DTW. This yields $A'$, a matrix with the observations from $A$ warped/shifted to be aligned with the observations in $B$. You can stop here if $A'$ and $B$ are known to be from the same speaker.
  • Estimate the transform $\Gamma$ which minimizes the difference between $\Gamma A'$ and $B$.
  • Use the distance between $\Gamma A'$ and $B$ as your metric.

A last thing that comes to my mind is that you have to discard the first MFCC coefficient (which roughly expresses the signal loudness) to improve the capacity of your system to match utterances pronounced at a different volume/recording level.

  • $\begingroup$ Thank you for providing more directions for me. I did remove the first MFCC value as per your answer to my previous question. About using DTW to align utterances: is this supposed to be done on a frame-by-frame basis? I.e. in my 500x14 matrix I have 500 frames of 10ms audios, in both A and B. Would I use DTW row-by-row to match up A to B? $\endgroup$
    – YoungMoney
    Jan 18, 2013 at 21:32
  • $\begingroup$ I am not sure I understand your question... You use DTW to align the 500 vectors from A to the 500 vectors from B. $\endgroup$ Jan 19, 2013 at 9:07
  • $\begingroup$ What I mean is should I concatenate the 500 vectors for each A and B into one "long" vector (with 500x14 elements) and then apply DTW? Versus applying DTW 500 times on the 14 element vectors. The former makes more sense to me based on what I've read so far. $\endgroup$
    – YoungMoney
    Jan 19, 2013 at 19:18
  • 2
    $\begingroup$ The former indeed. You need to find the optimal alignment in the 500x500 matrix (populated with the pairwise distances between the 14-dimensional vectors) $\endgroup$ Jan 19, 2013 at 19:49
  • $\begingroup$ Ah. Well it looks like I'm off to the races again $\endgroup$
    – YoungMoney
    Jan 19, 2013 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.