# Speech comparison algorithm for rating on similarities

I am trying to compare 2 speech samples and rate them on similarities. Think of someone trying to repeat a phrase, and then comparing those 2 audio files.

I started by implementing the MFCC (http://en.wikipedia.org/wiki/Mel-frequency_cepstrum) algorithm. I calculate the MFCCs of both audio samples, which gives me roughly 500 frames of audio (at 10ms each, with like 30% overlapping of the previous) having 14 or so MFCC coefficients. So a 500x14 matrix for each audio signal.

Then I do the naive approach of simply differencing the matrices. This does not give very promising results. Half of the time when I compare completely different audio samples (where different phrases are spoken), I get less difference than comparing the audio where I try to repeat the same phrase! This is clearly backwards and can't give me a good scoring algorithm.

How can I improve on this? I thought MFCCs were a really important part of speech processing, though clearly I need to do more with it.

-

First, you will have to correct for differences in timing. For example, if one utterance is "--heeelloooo---" and the other "hellooooooo----" (- representing silence), a direct pairwise comparison of MFCC frames will show differences simply because the two samples are not aligned. You can use Dynamic Time Warping to find the best alignment between the two sequence of feature vectors - and compute the corresponding distance.

The second problem is that if the two recordings are not from the same speaker, you will have to compensate for the differences in timbre. The MFCC of a female saying "aaa" are not the same as the MFCC of a male speaker saying the same phoneme! A relatively simple model to account for variations in voice timbre is to assume that there exists a linear transform $\Gamma$ which "maps" the MFCCs of one speaker onto the MFCCs of another speaker (to be fair, only one small subset of these transform accurately model how changing parameters such as age, gender, etc "shifts" the MFCC). Once two recordings have been aligned, even roughly, you can use a least square procedure to estimate $\Gamma$. This procedure is known as speaker normalization or speaker adaptation.

Your comparison procedure will thus consists of the following steps. $A$ and $B$ are your original MFCC sequences.

• Align the two utterances using DTW. This yields $A'$, a matrix with the observations from $A$ warped/shifted to be aligned with the observations in $B$. You can stop here if $A'$ and $B$ are known to be from the same speaker.
• Estimate the transform $\Gamma$ which minimizes the difference between $\Gamma A'$ and $B$.
• Use the distance between $\Gamma A'$ and $B$ as your metric.

A last thing that comes to my mind is that you have to discard the first MFCC coefficient (which roughly expresses the signal loudness) to improve the capacity of your system to match utterances pronounced at a different volume/recording level.

-
Thank you for providing more directions for me. I did remove the first MFCC value as per your answer to my previous question. About using DTW to align utterances: is this supposed to be done on a frame-by-frame basis? I.e. in my 500x14 matrix I have 500 frames of 10ms audios, in both A and B. Would I use DTW row-by-row to match up A to B? – YoungMoney Jan 18 '13 at 21:32
I am not sure I understand your question... You use DTW to align the 500 vectors from A to the 500 vectors from B. – pichenettes Jan 19 '13 at 9:07
What I mean is should I concatenate the 500 vectors for each A and B into one "long" vector (with 500x14 elements) and then apply DTW? Versus applying DTW 500 times on the 14 element vectors. The former makes more sense to me based on what I've read so far. – YoungMoney Jan 19 '13 at 19:18
The former indeed. You need to find the optimal alignment in the 500x500 matrix (populated with the pairwise distances between the 14-dimensional vectors) – pichenettes Jan 19 '13 at 19:49
Ah. Well it looks like I'm off to the races again – YoungMoney Jan 19 '13 at 19:54