8
$\begingroup$

I have a set of songs for which I extracted the STFT (Short-Time Fourier Transform) and used the magnitude spectrum $|S|$ to calculate the mel spectrogram by using a mel filterbank matrix $M$, so $X=\log(M\times |S|)$. I want to know is there any method to reverse this process, i.e. convert from the mel spectrogram back to the spectrogram. I performed some dimensionality reduction on the mel spectrogram, and reconstructed the mel spectrogram from lower dimensions. Now I want to regenerate the audio signal from the reconstructed mel spectrogram, so I guess first reconstruct the spectrogram and then the audio signal.

The problem is that the mel filter bank matrix is not a square matrix, since we the reduce the no of frequency bins, so inverse of $M$ cant be used like this : $ \hat{S}=M^{-1}\exp(X)$. So is there any way to generate the inverse mapping, like some inverse transfer function that can convert from $X$ to $S$?

$\endgroup$
  • $\begingroup$ This is simply not possible unless you have a very strong prior model of what your sources sounded like. Without any prior information on your sources, there are indeed many different spectra that can be "collapsed" into the same set of mel frequency coefficients. $\endgroup$ – pichenettes Jul 29 '13 at 10:41
  • $\begingroup$ @pichenettes : Then how do reconstruct the original song, I hope you have understood the problem. I have a mel-spectrogram, and a reconstructed version of it from lower dimensions. Instead of reporting that "the error was " in terms of numeric figures, if there is a way I could somehow "hear" the reconstructed mel-spectrogram , then I can say that after the dimensionality reduction, by retaining less components and reconstructing the mel-spectrogram, this is how the audio sounds. $\endgroup$ – user76170 Jul 29 '13 at 14:01
  • 2
    $\begingroup$ It is simply not possible in the general case. It is like having 5 numbers and asking how to "reconstruct" the numbers form their sum. $\endgroup$ – pichenettes Jul 29 '13 at 14:20
  • $\begingroup$ We wanna make voice control system on the matlab but we don't know how we can do this.We have voice data but how can we put this data in the Matlab. e.g (eight1.mel....eight.mel) files . We want this audio english numbers recognition in matlab . 1-Must i convert ".mel-waw" files? 2-Which can i use codes? If you help us,we'd appreciate Thanks Regards $\endgroup$ – user21133 May 22 '16 at 15:36
5
$\begingroup$

Both taking a magnitude spectrogram and a Mel filter bank are lossy processes. Important information needed to reconstruct the original will have been lost. Thus you need to go back and use the original audio samples to do the reconstruction by determining a time or frequency domain filter equivalent to your dimensionality reduction.

You can make assumptions about the lost information, but those assumptions themselves usually sound inaccurate, artificial and/or robotic. Or you can use only specially synthesized input, where the assumptions will be correct by design of that input.

$\endgroup$
  • $\begingroup$ Is there an equivalent reverse transformation for Mel fiter bank ? I understand that it is a lossy process. and we cannot find $M^{-1}$ since M is not square. However, is it possible to skip that step, and directly perform PCA on magnitude spectrogram |M| ? Also store the phase information. Then reconstruct |M|, combine the phase information, and get back the original audio ? $\endgroup$ – user76170 Nov 4 '13 at 10:17
1
$\begingroup$

As already mentioned, in general it is not possible to perform exactly. Like if you have 2d representation of 3d scene, e.x. side of a car, you can not in principle say if it is very thin cartoon that resembles a car or it was photo of some real car but if you assume that nobody has tried to fool you, you can make some assumptions about real data, indeed this is how machine learning generative models work.

Regarding inverse transformation, since most of the filters are orthogonal (their inner product is zero), you can use transposition of the filter bank as an approximation of the inverse transformation, however scaling of the data will be wrong as you'll easily verify. You can use some statistics about original STFT extracted from some large dataset e.x. per channel (frequency) and normalize the $M^TMx$ to have the 'right' scale. When it comes to creating full spectrogram from the magnitude signal only, you can use Griffin-Lin algorithm to do some approximation. You can use something like WaveNet or Parallel Wavenet to do it more robustly, as it has been verified to work quite well with speech signal and other audio signals.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.