# How to develop a spectrogram (2D array) from audio signal?

I have developed a spectrogram in Python using Scipy.Signal.Spectrogram. But I need a complete understanding of data. Here I am not asking about plotting and color selection etc. I am more into data (numbers). I am attaching a picture, please have a look : In image 1, you can see that I have an array of frequency mapped to [0, Fs/2] i.e [0,24000] in my case.

In image 2, Time is mapped from [0,10sec] and the total array length is 2141.

In image 3, Spectrogram has been computed in a 2D array.

I want a clear understanding that how these array of Frequencies and Times have been developed. What is the data that the spectrogram holds in the 2D array? Is it a log magnitude of frequency domain components?

I need some clear steps as I want data in 2D array as it can be seen in the 3rd Image. Here I am not into plotting thing but more into a clear understanding of data behind the spectrogram. I want to get this same data in C++.

• I really don't care who downvoted my question, I believe this is a forum for noobs to pro to get the useful information with clear learning and understanding. I am glad there are people who understood the question and make it clear to me in the easiest way possible. Thank you DSP Rookie and Jithin. Apr 16 '20 at 14:24

The array of time is obtained from the Sampling frequency $$f_s$$ and spectrogram window length M, since time duration between consecutive samples will be $$T_s = \frac{1}{f_s}$$.

The array of frequency depends on both $$f_s$$ and DFT length N.

Explanation :

Suppose you have total $$L$$-length data sampled at $$f_s$$. The way a spectrogram is obtained is dividing the $$L$$-length data into overlapping N length windows and then take N-DFT of the windows. If the windows are overlapping, then it means, for every subsequent window, you are moving forward by $$M$$ samples in time-domain where $$M.

Total number of $$N$$-Length windows, you will have is $$\lfloor \frac{L}{N} \rfloor$$, so the total number of $$N$$-DFT to be taken is $$\lfloor \frac{L}{N} \rfloor$$. That means, you will have $$\lfloor \frac{L}{N} \rfloor$$ number of $$N$$-length DFT coefficients. You can arrange these $$\lfloor \frac{L}{N} \rfloor$$ number of $$N$$-DFT Coeffs in a Matrix of dimension $$N X \lfloor \frac{L}{N} \rfloor$$. Each column of this matrix in the $$N$$-DFT Coeffs.

Now, if we write the expression for spectrogram in a single equation, it will be: $$S[k,m] = \sum^{N-1}_{n=0}x[mM + n]e^{-j\frac{2\pi nk}{N}}$$, where $$m$$ denotes the $$m^{th}$$ window and $$k$$ denotes the $$k^{th}$$ DFT Coefficient. You can see that as $$m$$ increases, the time-domain data being picked up for DFT moves forward by $$M$$ samples.

If you want to compute the Spectrogram as Matrix Multiplication, you will have the following:

$$S = W_N \begin{pmatrix} x & x[M] & x[2M] & \cdots \\ x & x[M+1] & x[2M+1] & \cdots \\ \vdots & \vdots & \vdots & \cdots \\ x[N-1] & x[M+N-1] & x[2M+N-1] & \cdots \end{pmatrix}_{NX\lfloor \frac{L}{N} \rfloor}$$, where $$W_N$$ is the matrix of N-DFT basis vectors.

So, You will get the matrix in which values along column will give the frequency component in the data and values along row will give the variation along time index $$m$$. And, the value

Now, since you are moving forward by $$M$$ samples, each column of the Matrix above gives you the frequency domain picture keeping time constant, meaning $$m^{th}$$ column in the matrix $$S$$ gives frequency domain picture at $$time = mM.T_s$$. So, when you look at a column, time remain constant at $$mM.T_s$$, and digital frequency changes in steps of $$\frac{2\pi}{N}$$.

Similarly, $$k^{th}$$ row of the matrix $$S$$ gives you variation in intensity of digital frequency $$\omega = 2\pi \frac{k}{N}$$ as you move forward in time. So, when you look at a row, frequency remains constant at $$\omega = 2\pi \frac{k}{N}$$ and time moves forward in steps of $$M.T_s$$.

• Thanks alot. That's what I wanted. Cheers. Apr 15 '20 at 12:31
• I have understood how to get the incremental value for the frequency axis based on Fs and NFFT(DFT length N). I am still a little confused in getting step size for time axis. Ts in my case gives me value of 0.00002083 sec or 0.02083ms. But the incremental values in my data is 4.66ms/index. How do I get that ? Apr 15 '20 at 13:23
• @KhubaibAhmad If the time index is 4.66msec/index and $T_s = 0.02083msec$, then it means that the consecutive window is moving forward by $M=4.66/0.02083 = 224$(approx.) Apr 15 '20 at 13:31
• If I am not wrong (time ms /index) isn't dependent upon (NFFT(DFT length N)). Since I changed Nfft to 512. Frequency step size reduced to 93.75Hz/index. But my time remained same. Lets say I know, Fs , Ts, Nfft then how will I compute the step size on time axis ? Apr 15 '20 at 13:45
• @KhubaibAhmad That's exactly what I have explained. Step-size on time axis depends on M and $T_s$ and not on N. So, if you know M and $T_s$, then step-size on Time axis will be : $M.T_s$. It seems that for you $M = 224$. Apr 15 '20 at 13:52

Spectrogram is obtained using Short Time Fourier Transform technique. It is used to visualize frequency changes over time. Imagine you have a very long signal whose frequency content vary with time. You can be tempted to take the FFT of this whole data. But that will not show you the variation of frequency content over time. So you will have to split your signal into smaller chunks and take FFT of each chunk. That way you will be able to see how the frequency content varies over time. This is how the 2-D array is generated. For each chunk, you get the FFT output as one single column vector(in each column as per your figure). For consecutive chunks, you place them adjacent to each other. so as you move across row, you are moving across time.

How do you decide how to split your time signal into smaller chunks? Suppose your FFT size is $$N$$, you can split your whole signal into chunks of $$N$$ and take FFT of each without overlapping these chunks. So at time $$n$$, your FFT output is $$X(k,n) = \sum_{m=n}^{m=n+N}x[m]e^{-j2\pi km/N}$$ When taking FFT, we have not done any pre-processing on $$x[m]$$. We just did a rectangular windowing of $$x[m]$$ from $$m=n$$ to $$m=n+N$$ before taking FFT. But rectangular windowing is not a good windowing technique because it will increase spectral leakages. So instead of that, you will apply other windowing techniques like Hann or Hamming or Kaiser to mitigate this spectral leakage. Windowing is just point-by-point multiplication of $$x[m]$$. For window function $$w[n]$$ , $$0 \le n \le N-1$$, $$X(k,n) = \sum_{m=n}^{m=n+N}x[m]w[m-n]e^{-j2\pi km/N}$$ For the picture you showed, $$\text{Column, X[:,m]} \rightarrow \text{ Spectral Content at time index m}\\ \text{Row, X[k,:]} \rightarrow \text{ Variation of frequency index k across time index m}\\$$ Size of windowed $$x[m]$$ may be less than FFT size $$N$$ but for simplicity I am assuming same here.

There is one more parameter while obtaining this 2-D array. I mentioned earlier that the chunks do not overlap. But if the frequency content varies fast across each chunk, you may end up getting a jagged display when you plot the spectrogram. So it is imperative to provide some amount of overlap between chunks. The overlap size is $$L$$. So for each time index (row), the increment would be $$L/T_s$$, where $$T_s$$ is the sampling interval. To summarize, any two consecutive columns of the 2-D spectrogram array will be computed as $$X_r(k,n+rL) = \sum_{m=n+rL}^{m=n+rL+N}x[m]w[m-n-rL]e^{-j2\pi km/N}\\ X_{r+1}(k,n+(r+1)L)= \sum_{m=n+(r+1)L}^{m=n+(r+1)L+N}x[m]w[m-n-(r+1)L]e^{-j2\pi km/N}$$

• very detailed answer. Thanks alot. I wasn't able to understand some points from the above answer and you clarified that. Cheers. Apr 16 '20 at 12:50