# What is the proper way to implement a real time spectrogram?

I am trying to implement a real time spectrogram and I am not sure that I am implementing it the right way. First I decided to use Matlab, I was taking a short recording of lets say 0.1 seconds and then perform the FFT on that array of numbers which basically consists of the amplitude of lets say a 11025 sampling rate in that 0.1 seconds. I was then plotting the result of the FFT after computing the magnitude. After this process, I would then record another 0.1 seconds and so on. This was being done in an infinite loop in order to try to achieve a real time result. I know that this is not a spectrogram is just a different representation of this information. The problem with this is that I am not getting a very good results from this calculation. For example I was giving a certain tone using a frequency tone generator, and I was seeing a peak at just some instances which shows that somewhere the algorithm is failing. I do not have to use MATLAB, I could also use other programming languages such as Java so I prefer if I do not utilize a ready made function such as 'spectrogram' which is only present in MATLAB.

So can someone tell me on how a real time FFT of audio should be performed please? So how real time spectrum analyzer achieve that quality and they are real time so if they are taking a short sample of time it must be really short and still manage to get a good representation out of it.

• are you trying to create these "waterfall" images? with time and frequency and power (dB) on the three axes? not so sure how short the snippets of audio are in a commercial real-time analyzer, but you should know that the frameLength (which is also the window width) and the frameHop are two different numbers if there is anything better than 0% overlap. you can have a long frame (with a long window) and still advance that by short hops everytime the FFT is evaluated. Oct 1, 2018 at 23:31

Spectrograms are normally computed using the Short-Time Fourier Transform (STFT), and can be done in real-time. It consists of splitting the signal into overlapped frames, using a window function (typically scaled cosine like Hann) and then computing the FFT for the frame.

The overlap and windowing is key to avoid 'spectral leakage', and sounds like what is missing in your case.

If you want more frequency resolution, you can use longer FFTs.

In order to update with low latency (closer to "real time"), you can input short buffers often (every few milliseconds). Then combine that data with one or more previous buffers to FFT a longer data window as often as the short buffers arrive (if the CPU can keep up). Making these longer FFT buffers out of shorter incoming data buffers is called overlapping, since the longer buffers consist of partially overlapped data. Common overlaps are 50% and 75%, allowing an FFT 2X and 4X longer.

In addition, you can apply a window function as well as zero-pad the data before the FFT to remove rectangular windowing artifacts.

• This is not how I understood the overlap approach: the overlap and add is done on the FFT result so does not result in a longer FFT but avoids the amplitude loss we would have over intervals in the time domain if the data was not overlapped. Jun 16 at 18:08
• Overlapping, and overlap-add for fast convolution, are 2 different things. At the extreme, overlapping all but 1 sample is the same as a running DFT, a new DFT result every input sample, with a DFT length chosen for frequency "resolution" (bin spacing). Jun 16 at 23:25
• Right - that is how I understood it for creating a spectrogram (such as explained here:tek.com/en/documents/primer/…) : each FFT is the length of the buffer, rather than overlapping the data in time to then FFT an effectively longer buffer as your answer is written; that wording confused me. A running DFT would not be the same as a longer DFT with regards to frequency resolution, right? Jun 17 at 1:19
• I think the word "buffer" is confusing our discussion. There's the amount of data gathered before the next process step (or spectrogram column), which can be a ton shorter than the amount of actual sample data fed the FFT (overlapped), which can be a lot shorter then the FFT length. A running DFT would allow a new (horizontally blurry) spectrogram column every sample (48k times/second audio). Jun 17 at 4:36
• yes you're right- that is what was confusing me. Thanks for the clarification. Jun 17 at 4:58

My understanding is that a spectrum analyzer is a graph of frequency vs magntiude and a spectrogram is a heatmap of frequency vs time. It might help to clarify which one you're after.

For audio, 1024 - 4096 is an adequate frame size to yield good results at 44.1kHz and extremly fast to compute. You will almost certainly need to window each frame.

For a spectrum analyzer, you'll have to decide if you want to skip samples for each animation frame (not to be consufsed with your FFT frame - lol) or make sure you compute multiple frames to cover all samples that have been output since the last animation frame and take, say, the max for each bin. This won't be an issue for a spectrogram as you'll decide how many lines of your heatmap you can draw since your last animation frame.

For a spectrum analyzer you'll need some ballistics so each bin can "fall".

Although you can use the STFT to calculate a spectrogram, you will have a time-frequency trade-off. The longer your frame, the better your frequency resolution at the expense of time resolution.

Alternative approaches:

• use the constant Q transform
• use a filterbank

It actually depends on what you are trying to achieve. I have two general points here to help you:

First, you need to know the specifications of your real-time system. How long do you need to update your spectrogram, $$\Delta t$$?

Second, you need to decide your STFT properties (especially the overlap percent) so you can match your computation time with your target real-time update, $$\Delta t$$. The least possible computation time is when you have an overlap of $$0$$ or hop size is $$L$$ ($$L$$ is the window size), meaning every sampling time, $$T_s$$, you will update your STFT property. If you have a powerful computation, matching $$T_s$$ with $$\Delta t$$ is fine but not recommended due to bulky memory. In my spectrogram below, done in python, I use an overlap of $$50\%$$ (which is the usual percent) so I have a waiting time more than $$T_s$$ to compute and update my STFT. The longest time possible is when you have overlap of $$L - 1$$ (minimum possible hop size is $$1$$).

By the way, the overlap percent determines the resolution of your spectrogram so you also need to plan it out.

• Nice visual. PS maximum possible hop size is $L$ (also "overlap of" -> "hop size of"). It does depend on window but it's not $L - 1$ in general either. Jun 16 at 16:06
• Do you mean the nonoverlapping STFT where $L$ is the hop size? Jun 16 at 16:20
• Invertibility breaks for $\geq L + 1$, yes. Jun 16 at 17:35
• Didn't you mean "hop size" instead of "overlap" in "least possible computation time is when you have an overlap of $L−1$"? Jun 16 at 17:49
• What I mean is, $L - 1$ takes longest time, not least. So "when you have an overlap of $0$". Jun 16 at 18:00