How do I apply a binary mask and STFT to produce an audio file?

Question

So here's the idea: you can generate a spectrogram from an audio file using shorttime Fourier transform (stft). Then some people have generated something called a "binary mask" to generate different audio (ie. with background noise removed etc.) from the inverse stft.

Here's what I understand:

stft is a simple equation that is applied to the audio file, which generates the information that can easily be displayed a spectrogram. By taking the inverse of the stft matrix, and multiplying it by a matrix of the same size (the binary matrix) you can create a new matrix with information to generate an audio file with the masked sound.

Once I do the matrix multiplication, how is the new audio file created?

It's not much but here's what I've got in terms of code:

from librosa import load
from librosa.core import stft, istft
y, sample_rate = load('1.wav')
spectrum = stft(y)
back_y = istft(spectrum)

Thank you, and here are some slides that got me this far. I'd appreciate it if you could give me an example/demo in python

Answer 1

The idea of the short-time Fourier transform (STFT) in this case is to compute a representation of the input signal where we see how the frequency content of the signal evolves over time. If you aren't familiar with the STFT, I would suggest you read up on it a bit first. The Wikipedia article on the subject should get you started.

To your question. The STFT function in your code gives you a complex-valued matrix in the output. When you move up or down a column in the matrix you are moving in the frequency direction. When you move along a row in the matrix you are moving along the time direction. The magnitude of a single complex-valued element in the matrix gives you an estimate of the energy in a certain frequency band at a certain time position in the input signal.

I recommend you try to understand the parameters that are given to you in the function call. You should try to understand how the window size, hop size, number of FFT bins, and the windowing function used affect the output of the function.

If we denote your complex valued STFT with $$X = \textrm{STFT\{x(n)\}},$$ and the binary mask with $M$, the output signal in the time-frequency domain is given by:

$$Y = X \odot M,$$ where $\odot$ refers to element-wise multiplication. Then you want to convert your modified time-frequency representation $Y$ back into to a time signal. This is done by doing the inverse STFT:

$$y(n) = \textrm{ISTFT}\{Y\}.$$

This should get you going, but time-frequency processing is a complex topic and I highly recommend trying to find resources to help you really understand what is going on when you are doing things such as applying a binary mask in the STFT-domain.