# Generate audio data using VAE+GAN

I am trying to train a VAE+GAN model to generate sounds produced by honeybees. I build my model by slightly modifying this tutorial, which aims to generate new MNIST images. Since my data are 1D signals, I apply STFT to the input to generate image-like features whose height is time and width is frequency. However, the total loss quickly goes to inf or nan just after a few steps(~10). Any idea why? What is the legit way of using VAE+GAN model to generate audio data?

If you are applying the Short Time Fourier Transform (STFT) , then your data are more likely to look like a spectrogram, where the "height" is frequency and the "width" is time.

The..."legit" way to train an autoencoder type of network so that it learns to generate periodic-like sounds is to "teach" a bank of oscillators to "fire" in a predefined pattern. This "pattern" is what the VAE will "learn" from the training dataset.

By applying the STFT, you are segmenting your time into $$N$$ bins, where $$N$$ depends on the length of your STFT "window". Consequently, the STFT "window" also determines your frequency resolution. If you choose a low $$N$$, you will get very good temporal resolution but very poor frequency resolution too and therefore, this is a design choice.

Once you apply the STFT, you get the equivalent of a spectrogram. Taken a bit more liberally, the spectrogram gives you this information. Only that, in the case of the spectrogram, the "keyboard" is rotated 90 degrees clockwise (in a spectrogram the vertical axis is frequency and high pitch is upwards, while the horizontal axis is time with positive time extending to the right).

Every one of the keys is an oscillator. If you scan the spectrum at a given "time window" $$t_n$$ within a sound file and you find a high amplitude in that frequency then the key is "pressed" (the oscillator is active). Of course, the other parameter is how hard is the key pressed (amplitude).

So, your auto-encoder has to learn to predict the sequence by which it has to activate the oscillators and at what amplitude, in order to generate the sounds your are after.

This is the bigger picture. The "detail" to look for here is that the output of the STFT is $$\mathcal{C}$$omplex and you really need it to be to be able to take into account the phase that these oscillators are firing with respect to each other. Otherwise, you will be getting a baseline error below which your model would not be able to optimise any more.

More generally, I think that you would benefit from a little bit of background on speech (or music) generation by looking into what a vocoder is and how does it work, Prony's method by which signals can be synthesized from damped sinusoids (This is the closest to the piano analogy), speech synthesis and of course, the STFT and spectrogram links from above.

Hope this helps