Self studying, getting a quality spectrogram

Question

So I've been self studying signal processing, and been trying to use Python to get quality spectrograms, quality spectrogram like the one that would be produced with audacity etc. The steps I have done are:

1. Me saying 'computer', was recorded on my mac in .m4a, then I used ffmpeg converting it into .wav

   $ ffmpeg -i computer.m4a -acodec pcm_s16le -ac 1 -ar 16000 computer.wav

2. Imported into Python via:

   sample_rate, samples = wavfile.read('/path/to/computer.wav')

3. Now I just do a quick plot of the samples:

   plt.plot(samples)

   and I get this kinda graph:

   

   Why are my peaks capped?

4. Anyway, I carry on and as I understand it, I perform a Fourier transform that will take my signal, decompose it into the sine waves of different amplitudes and frequencies, and then get the real part of the complex numbers that are generated:

   samples_fft = np.fft.rfft(samples)
   frequencies = np.abs(samples_fft)

5. I now plot a graph of frequencies:

   

   This is what I guessed would be like, as vocal frequencies are typically around the range of 100 to a thousand, right?

6. A spectrogram is when I divide my samples into a certain amount of windows, and then take the Fourier transform of each of the windows, resulting in a time frequency graph.

   last_wind = 0
   windows = np.linspace(0,len(samples),1000)
   spectrogram = []
   for window in windows:
       if window != 0:
           samples_in_range = samples[last_wind:int(window)]
           samples_in_range_fft = np.fft.rfft(samples_in_range)
           frequencies_in_range = np.abs(samples_in_range_fft)
           spectrogram.append(list(frequencies_in_range))
           last_wind = int(window)
   spectrogram2 = np.asarray(spectrogram).T

7. Plot the spectrogram:

   audio_length = len(samples)/sample_rate
   audio_time_intervals = np.linspace(0,audio_length,999)
   tempfreq = np.linspace(0,1,19)
   plt.pcolormesh(audio_time_intervals, tempfreq, spectrogram2)

So now I have a spectrogram, but of very poor quality. but I've been watching youtube videos such as this one, how do I get better resolution and clarity so as to read stuff such as formants in vowels, i tried changing the size of my partitions, but still get very similar results, more windows get less data of frequencies in each window, and less windows get poor time resolution.

I am very new to signal processing, so any help would be greatly appreciated, such as a point in the right direction or pointers as to what I'm doing wrong, in order to get clear spectrograms so as to study.

Note, I have tried python packages such signal.spectrogram, but the results are very poor aswell. Plus I'd rather get to know the nitty gritties so I really understand what is going on.

Answer 1

Why are my peaks capped?

Your amplification gain is set to too high, or you are too close to the microphone. The amplifier is driven to its limits and it clips the output. Keep this recording and make another one where you are a little bit further away from the microphone to later compare the differences in the spectrogram. It will be interesting to see how this clipping manifests itself in the frequency spectrum.

...how do I get better resolution and clarity so as to read stuff such as formants in vowels, i tried changing the size of my partitions, but still get very similar results, more windows get less data of frequencies in each window, and less windows get poor time resolution.

On line 7 of your point #6, you make use of numpy.fft.rfft.

From the documentation ...the length of the transformed axis of the output is therefore n//2 + 1..

Your signal is about 2 seconds long at a sampling frequency (Fs) of 16kHz and you are dividing it into windows of 1000 samples that overlap by 965 samples. That's about 97% overlap.

This 97% overlap is a very high temporal resolution. What you are lacking is in frequency resolution and the only way to increase this is to increase your window size. This will give the Fast Fourier Transform (FFT) the opportunity to resolve more frequencies.

These are Audacity's parameters for deriving spectrograms:

There are two things to notice here:

1. All of Audacity's window sizes are powers of 2. This is done to exploit the structure of the calculations in computing the FT and do it F(ast).

2. The window size is inversely proportional to the resolution of the FFT. A size 8 window is marked as "most wideband" and a size 32768 is marked as "most narrowband". Of course, your window size is not independent of your Sampling Frequency and there are limits to how many frequency components it can resolve given the data that has already been captured at sampling. Therefore, higher true frequency resolution means capturing more data which means increasing your sampling frequency. With 16kHz you can resolve (theoretically) up to 8kHz.

So with the exception of a few details and optimisations here and there, you are down the right track with your code.

You are missing:

1. The windowing functionality (nothing to do with the length of the sliding window you apply in your code (1000)). This will become important later on if you want to resolve things like formants.

2. "Proper" plotting which sets the x and y axes of your plot to the time and frequency scales required to interpret the spectrogram. This is not something incredibly difficult to do, but you will find yourself needing it, otherwise you will be doing "back of the envelope" calculations every time you want to read frequencies and offsets from your plots.

Ultimately, you will add those to your code too and you will have replicated, to a large extent, the functionality of scipy.signal.spectrogram.

Therefore, I would recommend to you to focus your studying efforts on the specifics of the spectrogram in order to understand how it works and what the impact of each one of its parameters is to the quality of its output.

A good introduction to the spectrogram itself is this one. And of course, it is not the only one out there.

scipy.signal.spectrogram includes all of the key parameters and it will allow you to explore spectrograms comfortably.

Finally, if you are interested in audio analysis, there are some really good libraries in the Python space, that implement a lot of functionality you will need. Some examples are pyAudioAnalysis and librosa (and here for something closer to what you are asking). Many more pointers available here.

Hope this helps.

Answer 2

In addition to what the others have said, let me just add a couple of more things that I think are important for creating a detailed spectrogram (this information doesn't target Python or any specific audio library )

1. FFT frame (window) overlap--Your FFT frames should overlap by at least 50%. This means that if your FFT frame length is, say, 1024, your frames will start at 0, 512, 1024, 1536 and so on (for a 50% overlap) In order to compute the magnitudes you will average results from 3 consecutive frames (except for first and last frame where you average 2 frames). These averages magnitudes are then used for the actual spectrogram display.

   The overlapping described above compensates for the loss of information due to the windowing (well, to an extent) and it kind of increases the temporal resolution

2. The number of FFT bins is determined by the frame length used for the FFT. If your frame length is 1024, you'll get 512 bins (actually 1024 but only the first half is used). The actual spectrogram height may be less than 512 (the number of bins), in which case you obviously won't be able to display every bin. You should use peak magnitudes as that would make most sense (especially for things like formants etc). If you just average several bins or even skip some bins randomly, your spectrogram may lose (not show) important information.