1
$\begingroup$

I am working on a speech synthesis model and I am looking to evaluate my synthesized speech. I found that most people use the Mel Cepstral Distortion (MCD) which can be calculated by the formula:

$$\frac{10}{ln10}\frac{1}{T}\sum_t\sum^{25}_i \sqrt{2||mc(t,i)-mc_{synth}(t,i)||^2}$$

I have ground truth speech and the synthesised version so I calculate the MFCCs for both (taking 25 coefficients) and then apply the formula. The result I get is around 300, which seems orders of magnitude larger than what I should get. I tried doing cepstral mean normalization which brings the number down to around 30 but I notice that most of the time this number is <10. The speech I synthesize is not so bad so I must be doing something wrong. Does anyone have any ideas?

P.S. The way I produce the speech ensures that it be aligned with the original(more or less). I wonder if a slight misalignment could cause this issue and if so should I use DTW to align the audios?

Edit Also If anyone knows of a library that calculated MCD when given an audio file then I could use that to compare my results.

$\endgroup$
  • $\begingroup$ Is the segmentation the same across audio streams? If you can generate the timestamps of the frames the coefficients are calculated on then you might be able to compare the corresponding frames more accurately. $\endgroup$ – A_A Apr 2 at 6:52
  • $\begingroup$ I think the two signals are aligned. Do you think that misalignment is the problem I'm experiencing? Also I'm using frame length 20ms and stride 10ms for the mfccs. Could this be the problem $\endgroup$ – MrHat Apr 2 at 7:22
  • $\begingroup$ OK, another way to cross check the error quantity is to subtract one waveform from another. Is the error as small as you would expect in that case? $\endgroup$ – A_A Apr 2 at 7:24
  • $\begingroup$ when subtracting one from another I get an average difference of 35. My signals are 16-bit integers so they have values range [-32767, 32767]. To me this seems reasonable but I have no idea what others get when they synthesize. Another problem is I can't seem to find two reference signals for which the MCD is known so that I can try to reproduce that. $\endgroup$ – MrHat Apr 2 at 7:32
  • $\begingroup$ I am having similar issues in computing MCD. I need to compare results with a paper - which seems to have mcds of order 1-10, but in my computations we get a number that is 10 times larger. I am aligning my sequences (this is for speech waveforms) using the dtwalign tool, and then compute the MCD using the same formula above, with librosa to create mels and mfccs. There seems to be two ways of getting the mfccs, either through the raw waveform or through the mel spectrogram with librosa's api. Could you describe how you managed to reduce the disparity, if possible? $\endgroup$ – kakrafoon Jun 18 at 21:49
0
$\begingroup$

Ok so here is what I found. The distance is dependent on the way that the mfccs are calculated. This makes sense to me and also explains why cepstral mean normalization affects the values of the MCD.

I found this implementation (https://github.com/MattShannon/mcd), which unfortunately did not support .wav files. I ran this and it gives results that are in the correct range for its test examples. I looked into the code and found that the the difference was not so much in the calculation of the MCD (which is straight forward) but rather the mfccs. The library I was using gave mfccs at a different scale (I tried python_speech_features and pyspeech). Using the mfccs from Matlab gave me mfccs in a similar range to that of the repo so I opted to use those.

A noteworthy point is that the first coefficient is quite large which is why it is ignored in (https://github.com/MattShannon/mcd) so I did the same. There are some artifacts in the speech I synthesise which likely accounts for why my MCD is still above 10 but its still lower than 30 now.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.