Python audio analysis: which spectrogram should I use and why?

Question

I am doing my final project at university: pitch estimation from song recording using convolutional neural network (CNN). I want to retrieve pitches existed in a song recording. For CNN input, I am using a spectrogram.

I am using MIR-QBSH dataset with pitch vectors as data label. Before processing the audio to CNN (each audio has 8 sec duration in .wav files of 8 KHz, 8 bit, mono), I need to pre-process the audio into a spectrogram representation.

I have found 3 ways to generate a spectrogram, the code are listed below. Audio example I am using in this code is available here.

Imports:

import librosa
import numpy as np
import matplotlib.pyplot as plt
import librosa.display
from numpy.fft import *
import math
import wave
import struct
from scipy.io import wavfile

Spectrogram A

x, sr = librosa.load('audio/00020_2003_person1.wav', sr=None)

window_size = 1024
hop_length = 512 
n_mels = 128
time_steps = 384 

window = np.hanning(window_size)
stft= librosa.core.spectrum.stft(x, n_fft = window_size, hop_length = hop_length, window=window)
out = 2 * np.abs(stft) / np.sum(window)

plt.figure(figsize=(12, 4))
ax = plt.axes()
plt.set_cmap('hot')
librosa.display.specshow(librosa.amplitude_to_db(out, ref=np.max), y_axis='log', x_axis='time',sr=sr)
plt.savefig('spectrogramA.png', bbox_inches='tight', transparent=True, pad_inches=0.0 )

Spectrogram B

x, sr = librosa.load('audio/00020_2003_person1.wav', sr=None)
X = librosa.stft(x)
Xdb = librosa.amplitude_to_db(abs(X))
# plt.figure(figsize=(14, 5))
librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')

Spectrogram C

# Read the wav file (mono)
samplingFrequency, signalData = wavfile.read('audio/00020_2003_person1.wav')
print(samplingFrequency)
print(signalData)

# Plot the signal read from wav file
plt.subplot(111)
plt.specgram(signalData,Fs=samplingFrequency)
plt.xlabel('Time')
plt.ylabel('Frequency')

Spectrogram results are displayed below:

My question is, from the 3 spectrograms I have listed above, which spectrogram is best to use for input to CNN and why should I use that spectrogram type? I am currently having difficulty to find their differences, as well as their pros and cons.

Answer 1

I recommend a Synchrosqueezed Continuous Wavelet Transform representation, available in ssqueezepy. Synchrosqueezing arose in context of audio processing (namely speaker identification), and there's much literature on applying CWT for audio tasks.

Advantage over STFT is the inherently logarithmic nature of the feature extractor, matching audio structuring. That is, we perceive sound differences in relative terms, so 6kHz is to 3kHz what 24kHz is to 12kHz. STFT is unable to map such features without being very wasteful, as it lacks an adaptive window (your Spectrogram A is not logarithmic; it's a logarithmic display of linear data). Analytic CWT (CWT with analytic wavelet) further enjoys superior instantaneous frequency and amplitude representations (see below references).

Synchrosqueezing further enhances these representations, and can be thought of as an attention mechanism. Experiments have shown it to enhance EEG classifier performance, for example. Currently only CWT is supported, but I'll be adding STFT within a week or so. Also see:

• Excellent CWT tutorial (also comparing with STFT)
• Synchrosqueezing friendly overview
• CNN + CWT vs STFT comparison
• Synchrosqueezing paper

---

Examples on EEG data:

Answer 2

From the information you've given, I'd use spectrogram A as the frequency data is spaced logarithmically. This is advantageous for pitch detection applications as it gives you a greater amount of resolution around the fundamental frequencies.

Answer 3

It is preferable to plot Spectrogram using Praat. It is a very trusted historical source for calculating important features, also it is can be used with python. Check out this link : Parselmouth. Hand coding Spectrogram can be erroneous, So what I suggest is use the Praat Script to cross verify which one is correct.
Or use the Praat Scripting. Hope it helps