I am working on something like cartoon, but the character's mouth shape is automatically detected and matched by the word they are saying.

my guess is to use speech recognize algorithm to analyze the wav file, and get the content(word) at the exactly moment, for the exactly word, prepare corresponding mouth shape picture.

but it look like a dumb way... is there any convenient way to work this out?

I saw many games support this, how did they achieve that?


The simplest solution I can think of is to use an envelope follower, the higher the amplitude, the more open the mouth is. This is pretty simple but will one give tell you how 'open' the mouth should be, not the shape. This should already be quite effective for a cartoon. To make it more realistic you should probably try to detect vowels (Oh and Ah for instance) by measuring the formants. http://en.wikipedia.org/wiki/Formant gives a list of average frequency's for some vowels. Of course these are only average, so you need to experiment.

I don't think you need to recognize exactly which words are spoken, this will be much more complex, and there are quite a lot of words in each language, and quite a lot of languages... :-)

If you add consonant detection as well, you probably have all the information you need...

| improve this answer | |
  • $\begingroup$ currently I am using the "amplitude" way to do it, but only open and shut mouth is not good enough to me. it's just like "talking tom". want to find a more accurate way. thanks for the tip of formant, I will look into it. $\endgroup$ – springrider Aug 25 '14 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.