0
$\begingroup$

I know that usually ASR systems are evaluated using WER (word error rate), and I was wondering if it makes sense to use a more fine grained approach. E.g. transforming both the ASR hypothesis and the reference transcription to its phonetic representation (e.g. using an algorithm such as soundex or metaphone), and then taking the WER.

Some half-baked ideas for use cases:

  • perhaps this evaluation method can help to distinguish between errors that stem from a bad language model to those that stem from a bad acoustic model

  • when using third-party ASR APIs, where the there's no control over the language model, perhaps it can be used to pick the best hypothesis among the top results (especially when the speech contains non-standard names and other out-of-vocabulary) words

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ "Pea" and "pee" sound the same but do not have the same meaning. ASR benefits greatly from a language model. "Finally in the toilet I decided it was time to", now "Pea" is a noun so it would not even be considered there. You might want to have a look at Sphinx $\endgroup$
    – A_A
    Sep 23, 2016 at 13:23
  • $\begingroup$ Thanks, that's a good point. Speaking of Sphinx, I wonder whether this evaluation method can help to distinguish between errors that stem from a bad language model to those that stem from a bad acoustic model. $\endgroup$
    – dimid
    Sep 23, 2016 at 13:57

1 Answer 1

Reset to default
1
$\begingroup$

You have misconception that errors are caused by either language model or by acoustic model. There is no clear distinction because models are combined together in during search for best hypothesis. Interaction is complex and another major player is beam search pruning.

Most errors are originated from the acoustic model when it fails to give proper estimate for a certain frame, but the problem is that they propagate over next few words. For example, if you have an unknown word, it will not be recognized properly but the next several words will be wrong too because search state will not match the actual content. And phone-by-phone comparison will not be really meaningful.

So it's a bit more complex.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, do you think there is any way to mitigate this, e.g. looking at a window of N words, or phonetic algorithms aren't useful at all for ASR? $\endgroup$
    – dimid
    Sep 24, 2016 at 16:39
  • $\begingroup$ Phonetic algorithms are useful for ASR of course, you could evaluate acoustic model in phonetic decoding mode for example to estimate acoustic model accuracy. Most acoustic model methods are evaluated on phonetic decoding on TIMIT database first. You can also switch the language model to very good one with very small perplexity to estimate how good is that and how much does it contribute to accuracy. You can run decoding with very wide beam to estimate how much pruning errors do you have. It all depends on your final goal which you didn't mention in original question. $\endgroup$
    – Nikolay Shmyrev
    Sep 24, 2016 at 21:07
  • $\begingroup$ Thanks for the clarification, my main goal is to compare different ASR apis where I can't control internal parameters such as the beam's breadth. $\endgroup$
    – dimid
    Sep 25, 2016 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.