0
$\begingroup$

I would like to know whether I am correct in my understanding of going from STFT to a spectrogram. My goal is to convert a spectrogram back to a wav file.

If I have my STFT:

audio, _ = librosa.load(f, sr=Fs)
stft = librosa.stft(samples, n_fft=NFFT, hop_length=HOP_LENGTH, win_length=window_length_samples)

and I wish to display a spectogram, I have to do:

spec = librosa.specshow(np.abs(stft))

However, since I have taken the modulus, it must be impossible to go from spec back to audio correct? So does that mean that librosa.istft does NOT convert a spectrogram to a wav file? My confusion arises because I have seen many answers to "spec to wav" questions suggesting the use of librosa.istft.

I am wishing to use a visual representation of audio that can be fed to a convolutional neural network, but still be converted back into audio. As an extension to the question, does anyone have experience using the stft numpy array as the input to a CNN?

Improve this question
$\endgroup$
3
  • $\begingroup$ if you don’t change your STFT results, it’s a lot easier to just keep a copy of the wave file and process it the way you want. $\endgroup$
    – user28715
    Dec 11 '19 at 14:30
  • $\begingroup$ I am actually using generative adversarial networks to try and produce an stft array. So the reason I am wondering about the conversion between STFT and wav is because I’d like the network to generate unique, unseen stft arrays. $\endgroup$
    – Harry Stuart
    Dec 12 '19 at 8:27
  • $\begingroup$ You are right in that it's impossible to get the time-domain signal from the spectrogram. (There are algorithms that aim at estimating it, e.g., the algorithm proposed by Griffin and Lim mentioned in an answer, but that's a different topic.) The reason for that is that all phase information is lost by taking the modulus. What you could try to do is to pass a second matrix, which contains the phase of the short-time DFT, to the CNN and see whether that brings you a step further. $\endgroup$
    – applesoup
    Apr 15 at 22:53
0
$\begingroup$

However, since I have taken the modulus, it must be impossible to go from spec back to audio correct?

You need to resort to some approximations like the Griffin-Lim algorithm or various vocoders (WaveNet,WaveGlow,etc.).

So does that mean that librosa.istft does NOT convert a spectrogram to a wav file?

It's a bit more complicated than that. An FFT ideally gives you the magnitude and phase spectra. If I'm not mistaken, what causes the confusion is that np.stft gives you an ndarray with a complex dtype, that is, it includes the magnitude and phase representation in one single ndarray. When you take the absolute value it tosses away the phase information, and that's an irreversible operation (even in the real domain np.abs would be irreversible). That's why you need a phase approximation scheme to get back to audio.

As an extension to the question, does anyone have experience using the stft numpy array as the input to a CNN?

You can do it. Actually, most CNNs use Mel spectrograms as their audio representation, so it is entirely possible. The best way to do it depends on your dataset, sadly.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.