What is the proper way to implement a real time spectrogram?

Question

I am trying to implement a real time spectrogram and I am not sure that I am implementing it the right way. First I decided to use Matlab, I was taking a short recording of lets say 0.1 seconds and then perform the FFT on that array of numbers which basically consists of the amplitude of lets say a 11025 sampling rate in that 0.1 seconds. I was then plotting the result of the FFT after computing the magnitude. After this process, I would then record another 0.1 seconds and so on. This was being done in an infinite loop in order to try to achieve a real time result. I know that this is not a spectrogram is just a different representation of this information. The problem with this is that I am not getting a very good results from this calculation. For example I was giving a certain tone using a frequency tone generator, and I was seeing a peak at just some instances which shows that somewhere the algorithm is failing. I do not have to use MATLAB, I could also use other programming languages such as Java so I prefer if I do not utilize a ready made function such as 'spectrogram' which is only present in MATLAB.

So can someone tell me on how a real time FFT of audio should be performed please? So how real time spectrum analyzer achieve that quality and they are real time so if they are taking a short sample of time it must be really short and still manage to get a good representation out of it.

Answer 1

Spectrograms are normally computed using the Short-Time Fourier Transform (STFT), and can be done in real-time. It consists of splitting the signal into overlapped frames, using a window function (typically scaled cosine like Hann) and then computing the FFT for the frame.

The overlap and windowing is key to avoid 'spectral leakage', and sounds like what is missing in your case.

Answer 2

If you want more frequency resolution, you can use longer FFTs.

In order to update with low latency (closer to "real time"), you can input short buffers often (every few milliseconds). Then combine that data with one or more previous buffers to FFT a longer data window as often as the short buffers arrive (if the CPU can keep up). Making these longer FFT buffers out of shorter incoming data buffers is called overlapping, since the longer buffers consist of partially overlapped data. Common overlaps are 50% and 75%, allowing an FFT 2X and 4X longer.

In addition, you can apply a window function as well as zero-pad the data before the FFT to remove rectangular windowing artifacts.

Answer 3

My understanding is that a spectrum analyzer is a graph of frequency vs magntiude and a spectrogram is a heatmap of frequency vs time. It might help to clarify which one you're after.

For audio, 1024 - 4096 is an adequate frame size to yield good results at 44.1kHz and extremly fast to compute. You will almost certainly need to window each frame.

For a spectrum analyzer, you'll have to decide if you want to skip samples for each animation frame (not to be consufsed with your FFT frame - lol) or make sure you compute multiple frames to cover all samples that have been output since the last animation frame and take, say, the max for each bin. This won't be an issue for a spectrogram as you'll decide how many lines of your heatmap you can draw since your last animation frame.

For a spectrum analyzer you'll need some ballistics so each bin can "fall".

Although you can use the STFT to calculate a spectrogram, you will have a time-frequency trade-off. The longer your frame, the better your frequency resolution at the expense of time resolution.

Alternative approaches:

• use the constant Q transform
• use a filterbank

Answer 4

It actually depends on what you are trying to achieve. I have two general points here to help you:

First, you need to know the specifications of your real-time system. How long do you need to update your spectrogram, $\Delta t$?

Second, you need to decide your STFT properties (especially the overlap percent) so you can match your computation time with your target real-time update, $\Delta t$. The least possible computation time is when you have an overlap of $0$ or hop size is $L$ ($L$ is the window size), meaning every sampling time, $T_s$, you will update your STFT property. If you have a powerful computation, matching $T_s$ with $\Delta t$ is fine but not recommended due to bulky memory. In my spectrogram below, done in python, I use an overlap of $50\%$ (which is the usual percent) so I have a waiting time more than $T_s$ to compute and update my STFT. The longest time possible is when you have overlap of $L - 1$ (minimum possible hop size is $1$).

By the way, the overlap percent determines the resolution of your spectrogram so you also need to plan it out.