Converting mel spectrogram to spectrogram

Question

I have a set of songs for which I extracted the STFT (Short-Time Fourier Transform) and used the magnitude spectrum $|S|$ to calculate the mel spectrogram by using a mel filterbank matrix $M$, so $X=\log(M\times |S|)$. I want to know is there any method to reverse this process, i.e. convert from the mel spectrogram back to the spectrogram. I performed some dimensionality reduction on the mel spectrogram, and reconstructed the mel spectrogram from lower dimensions. Now I want to regenerate the audio signal from the reconstructed mel spectrogram, so I guess first reconstruct the spectrogram and then the audio signal.

The problem is that the mel filter bank matrix is not a square matrix, since we the reduce the no of frequency bins, so inverse of $M$ cant be used like this : $ \hat{S}=M^{-1}\exp(X)$. So is there any way to generate the inverse mapping, like some inverse transfer function that can convert from $X$ to $S$?

Answer 1

Both taking a magnitude spectrogram and a Mel filter bank are lossy processes. Important information needed to reconstruct the original will have been lost. Thus you need to go back and use the original audio samples to do the reconstruction by determining a time or frequency domain filter equivalent to your dimensionality reduction.

You can make assumptions about the lost information, but those assumptions themselves usually sound inaccurate, artificial and/or robotic. Or you can use only specially synthesized input, where the assumptions will be correct by design of that input.

Answer 2

Nowadays the easiest thing would be to use librosa for this task. It has the mel_to_stft function which does exactly what you want.

As others have mentioned, this reconstruction is lossy and only approximate solution can be found. In librosa it is done using the Non-negative Lease Squares algorithm.

A thing to keep in your mind: if you have extracted the mel energies using your own algorithm, you must ensure that the frequency warping is similar, otherwise you will end up with audio that sounds disharmonic (assuming that later on you want to synthesize the waveform from the STFT). In librosa there are two warping functions, you can use the second one by specifying htk=True.

Answer 3

As already mentioned, in general it is not possible to perform exactly. Like if you have 2d representation of 3d scene, e.x. side of a car, you can not in principle say if it is very thin cartoon that resembles a car or it was photo of some real car but if you assume that nobody has tried to fool you, you can make some assumptions about real data, indeed this is how machine learning generative models work.

Regarding inverse transformation, since most of the filters are orthogonal (their inner product is zero), you can use transposition of the filter bank as an approximation of the inverse transformation, however scaling of the data will be wrong as you'll easily verify. You can use some statistics about original STFT extracted from some large dataset e.x. per channel (frequency) and normalize the $M^TMx$ to have the 'right' scale. When it comes to creating full spectrogram from the magnitude signal only, you can use Griffin-Lin algorithm to do some approximation. You can use something like WaveNet or Parallel Wavenet to do it more robustly, as it has been verified to work quite well with speech signal and other audio signals.