How do I apply a binary mask and STFT to produce an audio file?

So here's the idea: you can generate a spectrogram from an audio file using shorttime Fourier transform (stft). Then some people have generated something called a "binary mask" to generate different audio (ie. with background noise removed etc.) from the inverse stft.

Here's what I understand:

stft is a simple equation that is applied to the audio file, which generates the information that can easily be displayed a spectrogram. By taking the inverse of the stft matrix, and multiplying it by a matrix of the same size (the binary matrix) you can create a new matrix with information to generate an audio file with the masked sound.

Once I do the matrix multiplication, how is the new audio file created?

It's not much but here's what I've got in terms of code:

from librosa.core import stft, istft
spectrum = stft(y)
back_y = istft(spectrum)

Thank you, and here are some slides that got me this far. I'd appreciate it if you could give me an example/demo in python

• Hi Blake! It seems you are a little bit naive about STFT, spectogram and audio files... I suggest you understand each block better before combining them to remove background noise as ascribed. Aug 2 '18 at 15:11
• binary masks in the freuency domain are not a good idea dsp.stackexchange.com/questions/6220/…
– user28715
Aug 2 '18 at 15:34
• Thanks! you'd be right that I'm new to this! Here, I was reading this, and it seems that they used applied a mask to the stft, then used the inverse to get back the audio with the voices separated. Am I talking about something else?arxiv.org/pdf/1705.04662.pdf Aug 2 '18 at 15:58

The idea of the short-time Fourier transform (STFT) in this case is to compute a representation of the input signal where we see how the frequency content of the signal evolves over time. If you aren't familiar with the STFT, I would suggest you read up on it a bit first. The Wikipedia article on the subject should get you started.

To your question. The STFT function in your code gives you a complex-valued matrix in the output. When you move up or down a column in the matrix you are moving in the frequency direction. When you move along a row in the matrix you are moving along the time direction. The magnitude of a single complex-valued element in the matrix gives you an estimate of the energy in a certain frequency band at a certain time position in the input signal.

I recommend you try to understand the parameters that are given to you in the function call. You should try to understand how the window size, hop size, number of FFT bins, and the windowing function used affect the output of the function.

If we denote your complex valued STFT with $$X = \textrm{STFT\{x(n)\}},$$ and the binary mask with $M$, the output signal in the time-frequency domain is given by:

$$Y = X \odot M,$$ where $\odot$ refers to element-wise multiplication. Then you want to convert your modified time-frequency representation $Y$ back into to a time signal. This is done by doing the inverse STFT:

$$y(n) = \textrm{ISTFT}\{Y\}.$$

This should get you going, but time-frequency processing is a complex topic and I highly recommend trying to find resources to help you really understand what is going on when you are doing things such as applying a binary mask in the STFT-domain.

• wow, this is exactly what I was looking for. I feel like it really fills a lot of the holes in terms of my understanding! Aug 2 '18 at 19:10