1
$\begingroup$

What kind of audio signal features/properties are appropriate for signal similarity measurement invariant to imitation?

Basically I would like to do the following:

  1. Having a template (e.g. a spoken word from a politician or from an actor as a recording)
  2. Another person trying to imitate the template (same word spoken)
  3. Estimation of imitation quality.

The estimation should give a high similarity value if the imitation is on a comedian level, so that a human can observe some similarity but typically hears that it's not the same person (if imitation isnt perfect) and sometimes some characteristics of that voice are even overhighlighted (parody).

So I think it is more about nasal sound etc.

Are there any appropriate features for this kind of similarity?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I think, what gives humans the impression of a "good" voice imitation is more about tempo, melody, accent and pronounciation, less about pitch and formants. Pitch and formants are important but will be looked over, if the former parameters are on spot. $\endgroup$
    – Max
    Commented Jan 3, 2022 at 10:25
  • $\begingroup$ I think the tempo is important for whole sentences, but accent, melody and pronounciation are important for single words. What kind of features or signal properties can be used to compare the accent, melody and pronounciation of two recordings (which assume to have the same spoken word)? $\endgroup$
    – Micka
    Commented Jan 3, 2022 at 10:52
  • 1
    $\begingroup$ I'm no expert in the field, but you should look into "acoustic phonetics" for detailed information about pronounciation and melody. (From a technical viewpoint, "accent" is just part pronounciation and part melody.) I think, a good starting point for comparison is the pronounciation of vowels. $\endgroup$
    – Max
    Commented Jan 3, 2022 at 11:06
  • $\begingroup$ @Max thank you, that's a great help for starting that task/experiments. I'm only expert in computer vision, so might be a long way. $\endgroup$
    – Micka
    Commented Jan 3, 2022 at 11:13

1 Answer 1

Reset to default
1
$\begingroup$

You could try something silly like combining the score of a speaker identification algorithm trained on the original speaker with a score from an algorithm that estimates "nasality". This would be doable if you happen to have enough recordings of the original speaker.

If not, then you could just use a voice-to-phonemes algorithm (so, the part of speech recognition that recognizes the sounds made) on both reference and parody, and use some difference metric on that as "closeness" measure. Then, use a "comedy" classifier.

The "comedy" or "nasality" estimators would sound like a prime job for a neural network, e.g. based on wavenet.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.