Algorithm to detect if a spoken phrase matches a previously spoken phrase

Question

I would like to create a very simple system that allows a user to train (once or multiple times) a spoken phrase (from 1 word to a whole sentence.)

Then the (same) user would speak the phrase back, and the system would score how close to the original the phrase was matched. If certain words were missed or incorrect, the score would be deducted.

I've seen several articles about audio fingerprinting (ie Shazaam) but these implementations are an overkill. I have a known sound pattern that I'm matching against, not search against a database of audio.

Here's what I've tried (using the R libraries DTW and tuneR):

1. recorded the phrase "Known as the father of the constitution, he was the first to arrive in Philadelphia for the Constitutional Convention." three times, however, the 3rd time instead of saying "Philadelphia" I said "New York".
2. took the MFCC of all three WAV files
3. took the DTW three times: Phili1 vs Phili2, Phili1 vs. NY, Phili2 versus NY
4. Plotted the alignment, question is...what do I do with this data, what metric should I use You can see in the plot that the first alignment is most linear (same phrase repeated exactly)

Phili1 vs Phili2 Phili1 vs. NY Phili2 vs. NY

Answer 1

MFCC is an established way of handling speech recognition. You will find plenty of libraries and tools (e.g. yaafe, SoX) that can extract them for you. Implementing them yourself is also quite easy. Yaafe provides plethora of other audio features you might like to play with.

Mind that MFCC are not very robust against noise, so you might like to apply some noise reduction beforehand if required. Some people advocate raising the log-mel-amplitudes to a power 2 or 3 before applying discreet cosine transform.

Some libraries, like scikit learn, next to all machine learning algorithms provide also DTW. On your place I would just try MFCC and see how it goes from there.

Answer 2

MFCC + DTW is the baseline/textbook-ish method for what you want to do. Looking at the best alignment path will allow you to find out which words are missing ; looking at the distances along the alignment path will allow you to find which frames are incorrectly pronounced. Note that this method is not very robust to changes in environment (say background noise, distance to microphone...), will give random results when the two sentences to match are from different speakers; and will be oblivious to some pronunciation errors (such as differences in pitch - since this aspect of prosody is very superficially captured by the MFCC).

Shazam uses features specifically crafted for music. I don't think the literature on audio fingerprinting is very relevant to what you are trying to do.