I am trying to compare 2 speech samples and rate them on similarities. Think of someone trying to repeat a phrase, and then comparing those 2 audio files.
I started by implementing the MFCC (http://en.wikipedia.org/wiki/Mel-frequency_cepstrum) algorithm. I calculate the MFCCs of both audio samples, which gives me roughly 500 frames of audio (at 10ms each, with like 30% overlapping of the previous) having 14 or so MFCC coefficients. So a 500x14 matrix for each audio signal.
Then I do the naive approach of simply differencing the matrices. This does not give very promising results. Half of the time when I compare completely different audio samples (where different phrases are spoken), I get less difference than comparing the audio where I try to repeat the same phrase! This is clearly backwards and can't give me a good scoring algorithm.
How can I improve on this? I thought MFCCs were a really important part of speech processing, though clearly I need to do more with it.