21
$\begingroup$

Is there any existing application to sample someone's voice and use it to modulate any other voice or synthesize a text to resemble the original one?

For example, this Text-to-Speech Demo by AT&T lets you choose a voice and a language from presets that I guess are based on some human voice that have been sampled.

How do you call this process? Is it voice modulation? Voice synthesis?

Improve this question
$\endgroup$
7
  • $\begingroup$ I guess if you had enough training samples, for particular words then it could be possible $\endgroup$
    – Phorce
    Feb 13, 2013 at 16:41
  • $\begingroup$ In the research community, this is called "voice conversion". $\endgroup$
    – pichenettes
    Feb 13, 2013 at 16:53
  • $\begingroup$ @user1582478 I have plenty of them, how would you proceed? $\endgroup$
    – clapas
    Feb 13, 2013 at 22:40
  • $\begingroup$ Is it possible to create a voice from say a friend then take that live on a phone call kind of like a voice morph but with the voice you've created? $\endgroup$
    – user11765
    Nov 18, 2014 at 21:51
  • $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. $\endgroup$
    – Matt L.
    Nov 19, 2014 at 10:35

4 Answers 4

Reset to default
27
$\begingroup$

A first note: Most modern text-to-speech systems, like the one from AT&T you have linked to, use concatenative speech synthesis. This technique uses a large database of recordings of one person's voice uttering a long collection of sentences - selected so that the largest number of phoneme combinations are present. Synthesizing a sentence can be done just by stringing together segments from this corpus - the challenging bit is making the stringing together seamless and expressive.

There are two big hurdles if you want to use this technique to make president Obama say embarrassing words:

  • You need to have access to a large collection of sentences of the target voice, preferably recorded with uniform recording conditions and good quality. AT&T has a budget to record dozens of hours of the same speaker in the same studio, but if you want to fake someone's voice from just 5 mins of recording it will be difficult.
  • There is a considerable amount of manual alignment and preprocessing before the raw material recorded is in the right "format" to be exploited by a concatenative speech synthesis system.

Your intuition that this is a possible solution is valid - provided you have the budget to tackle these two problems.

Fortunately, there are other techniques which can work with less supervision and less data. The field of speech synthesis interested in "faking" or "mimicking" one voice from a recording is known as voice conversion. You have a recording A1 of target speaker A saying sentence 1, and a recording B2 of source speaker B saying sentence 2, you aim at producing a recording A2 of speaker A saying sentence 2, possibly with access to a recording B1 of speaker B reproducing with his/her voice the same utterance as the target speaker.

The outline of a voice conversion system is the following:

  1. Audio features are extracted from recording A1, and they are clustered into acoustic classes. At this stage, it is a bit like having bags will all "a" of speaker A, all "o" of speaker A, etc. Note that this is a much simpler and rough operation than true speech recognition - we are not interested in recognizing correctly formed words - and we don't even know which bag contains "o" and which bag contains "a" - we just know that we have multiple instances of the same sound in each bag.
  2. The same process is applied on B2.
  3. The acoustic classes from A1 and B2 are aligned. To continue with the bags analogy, this is equivalent to pairing the bags from step 1 and 2, so that all the sounds we have in this bag from speaker A should correspond to the sounds we have in that bag from speaker B. This matching is much easier to do if B1 is used at step 2.
  4. A mapping function is estimated for each pair of bags. Since we know that this bag contains sounds from speaker A, and that bag the same sounds but said by speaker B - we can find an operation (for example a matrix multiplication on feature vectors) that make them correspond. In other words, we now know how to make speaker 2's "o" sound like speaker 1's "o".
  5. At this stage we have all cards in hand to perform the voice conversion. From each slice of the recording of B2, we use the result of step 2. to figure out which acoustic class it corresponds to. We then use the mapping function estimated at step 4 to transform the slice.

I insist on the fact that this operates at a much lower level than performing speech recognition on B2, and then doing TTS using A1's voice as a corpus.

Various statistical techniques are used for steps 1 and 2 - GMM or VQ being the most common ones. Various alignment algorithms are used for part 2 - this is the trickiest part, and it is obviously easier to align A1 vs B1, than A1 vs B2. In the simpler case, methods like Dynamic Time Warping can be used to make the alignment. As for step 4, the most common transform are linear transforms (matrix multiplication) on feature vectors. More complex transforms make for more realistic imitations but the regression problem to find the optimal mapping is more complex to solve. Finally, as for step 5, the quality of resynthesis is limited by the features used. LPC are generally easier to deal with a simple transformation method (take signal frame -> estimate residual and LPC spectrum -> if necessary pitch-shift residual -> apply modified LPC spectrum to modified residual). Using a representation of speech that can be inverted back to the time domain, and which provide good separation between prosody and phonemes is the key here! Finally, provided you have access to aligned recordings of speaker A and B saying the same sentence, there are statistical models which simultaneously tackle steps 1, 2, 3 and 4 in one single model estimation procedure.

I might come back with a bibliography later, but a very good place to start to get a feel for the problem and the overall framework used to solve it is Stylianou, Moulines and Cappé's "A system for voice conversion based on probabilistic classification and a harmonic plus noise model".

There is to my knowledge no widely piece of software performing voice conversion - only software modifying properties of the source voice - like pitch and vocal tract length parameters (For example IRCAM TRAX transformer) - with which you have to mess in the hope of making your recording sound closer to the target voice.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Awsome answer! I could probably have not been able to understand it without the bags analogy... There is only one thing that I do not understand after this so clear explanation: to my understanding, you can always have B1! Thank you very much. $\endgroup$
    – clapas
    Feb 13, 2013 at 22:53
  • $\begingroup$ Not if A and B speak a different language (there are uncanny applications of voice conversion in which TTS in another language is played back with your own voice!). Or if A and B are both famous persons, for which you can't find a sufficiently long common sentence in all publicly available recordings, and if you don't speak their language so you can't use a recording of your voice as a "bridge" between the two. $\endgroup$
    – pichenettes
    Feb 14, 2013 at 0:23
  • $\begingroup$ I see. Thanks again @pichenettes. I'll try to have a look at the menctioned book from Stylianou et al. Cheers $\endgroup$
    – clapas
    Feb 15, 2013 at 10:34
  • $\begingroup$ Please update your answer with links to the acronyms you used. E.g. LPC, VQ, GMM. $\endgroup$
    – aaronsnoswell
    Dec 8, 2015 at 23:40
  • $\begingroup$ In response to the comment from aaronsnoswell: LCP: linear predictive coding, VQ: vector quantization, GMM: Gaussian mixture model. It's not obvious that the mere expansion of these acronyms helps a lot since each is a complicated idea, but (roughly) each idea relates to modeling or explaning pst or future data from a set of existing samples. $\endgroup$
    – GregD
    Jun 25, 2016 at 2:06
4
$\begingroup$

You can use something like MorphVox. Here is a demonstration. The process is called voice morphing or conversion. If you are interested in the technical aspects, a recent article you can study is Voice Conversion Using Dynamic Kernel Partial Least Squares Regression.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Dark Vader's voice rocks mate, it was funny. Well, I have seen similar effects before. Thank you $\endgroup$
    – clapas
    Feb 13, 2013 at 23:03
1
$\begingroup$

I am looking for the same thing, but it can't be done. There is a company in Scotland called CereProc which does voice modeling, but they need someone at their lab recording hours of audio and the cost to model a single voice is around USD $30K.

Improve this answer
$\endgroup$
0
$\begingroup$

What you are looking for is called a vocoder.

Have you tried Audcity's vocoder? Audacity can be downloaded from: http://audacity.sourceforge.net/download. A demo on how to use it can be found at https://www.youtube.com/watch?v=J_rPEmJfwNs.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Vocoding consists in applying the spectral envelope of one audio recording (which is usually a voice) to filter another sound (which is usually a raw, unmodulated synthesizer sound). It cannot be used to fake someone's voice. $\endgroup$
    – pichenettes
    Feb 13, 2013 at 20:20
  • $\begingroup$ Yes, I know Audacity, but I didn't try the vocoder. I think you can only apply white noise to the envelope of the recorded voice. Thank you $\endgroup$
    – clapas
    Feb 13, 2013 at 22:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.