# Spectrograms for neural nets

General Question

Given an audio file, say a 16-bit wav, what are some standard methods to preprocess a spectrogram of this wav so that it may be fed into a neural net?

Context

In Lee et al's 2009 NIPS paper on applying deep belief networks (DBNs) to speech recognition, they briefly mention in Sections 2.2 and 3.1 preprocessing a spectrogram using PCA whitening.

NB: I don't have image-processing background as applied to machine learning...

Question

How may we reproduce the training data that is fed into the DBN in the above NIPS article, section 4.1 for speaker identification in the TIMIT dataset?

"We trained the CDBN on unlabeled TIMIT data (as described in Section 3.1)...We computed the first and second-layer CDBN features using the spectrogram as input."

Section 3.1 mentinons 20ms window sizes with 10ms overlaps to generate the spectrograms and then applying PCA whitenting. Does this spectrogram -> whitening process produce an $n \times m$ or $n \times m \times t$ array for each speaker? Or do we get an "image" that is somehow fed directly into a (C)DBN?

Outlook

Based on the NIPS paper, such preprocessing seems standard but I'm (obviously) new to using spectrograms in neural networks.

Any python references on spectrograms are appreciated as related to these questions, for example with librosa, pylab, or this tutorial.

After generating a spectrogram, perhaps the general steps in this PCA whitening overview can be applied to produce relevant input to a DBN?

Example

For many supervised classification tasks, we may fit a model via fit(X, y) where $X$ is an $n \times k$ matrix of $n$ samples with $k$ features and $y$ is an $n \times 1$ vector where each row is a corresponding label.

Using librosa's melspectogram method on an audio file, we produce an $m \times t$ matrix. With, say, $n$ labeled audio files we would have $n$ training samples each of shape $m \times t$ resulting in an $n \times m \times t$ matrix $X$ (and $n \times 1$ matrix $y$). How may we feed this $X$ into a DBN or neural network in general?

The "dimensions" of the spectrogram are not chosen based on where will the spectrogram be fed to but rather depend on your application. Therefore, it is key to understand the spectrogram itself first, as a means of generating features for one or more signals and to an extent, understand the Discrete Fourier Transform (DFT) as well, which is the key operation the spectrogram is based on. In fact, the "dimensions" you are referring to stem entirely from the way the DFT works and determine its spectral "resolution".

If you were to run DFT on some discrete time signal ($x[n]$), of maximum length $N$, you would get back a set of coefficients $X[k]$ that decompose the signal to a weighted sum of sinusoids at different frequencies.

The range of physical frequencies in $x[n]$ is $\left[ 0 \ldots \frac{F_s}{2} \right]$. However, the DFT decomposes the signal into $N_{FFT}$ frequency bins (indexed by $k$) with $N_{FFT} \le N$. Therefore, the physical frequency that is associated with the $k^{th}$ frequency bin is $f_k = F_s \cdot \frac{k}{N_{FFT}}$.

So what?! So, if you are creating a classifier that operates on some $x_m[n]$ sample obtained from the $m^{th}$ speaker and is trying to figure out the gender of the speaker via the range of frequencies audible, then you would have to tune your $N_{FFT}$ in such a way as to be able to assign different frequencies between 85Hz to 225Hz to different frequency bins. If you did not do that, then your spectral features (the set of $k$ frequency bins) would be insensitive to your class $C$ (here $C \in \left\{ male,female \right\}$ and the classification would report poor performance.

Alright, so, we keep this consideration of spectral resolution in mind. This actually considers the vertical dimension of the spectrogram. Let's have a look at the spectrogram itself now.

The spectrogram is the repetitive application of Short Time Discrete Fourier Transforms (STFT) to a signal in overlapping sections. Let's denote the overlap here with some factor $q \in \left\{ 0 \ldots 1\right)$. When $q=0$ there is no overlap and the spectrogram is evaluated independently on a set of windows (for example $n_{w1} \in \left\{0 \ldots 10 \right\}, n_{w2} \in \left\{11 \ldots 20 \right\}$ and so on). As $q$ approaches $1$, the STFT starts running on overlapping windows (for example $n_{w1} \in \left\{0 \ldots 10 \right\}, n_{w2} \in \left\{5 \ldots 25 \right\}$ and so on).

This overlap is desired in the STFT for a number of reasons that are related to the maths of the FT but also the signal itself and the desired analysis on it. The bottom line is that now we have to worry about one more parameter, $q$.

So what?!? So, to extend the previous example: Before, we were trying to understand what is the gender of the speaker just by examining a sample of their voice. Let's now assume that we have the recording of a conversation between males and females and we want to produce a list of time instances and their gender classification to examine (for example) if males or females tend to dominate a given discussion. Obviously, we have to tune $N_{FFT}$ from above but now we also have to tune $q$. Too much overlap means a high temporal resolution but also large amount of data (and processing). Too little overlap and we might miss some transition as some speaker was interrupted by some other speaker and then they resumed. If this phenomenon happens within the duration of a window, the STFT cannot resolve different males or females. It's a gray area that contains a mixture of both fundamental frequencies and it will produce some decision that might not be accurate.

As you can see, our discussion so far has not involved the classifier itself. The classifier is indeed some $C = f(x_m,\Theta)$ that takes the $m^{th}$ sample and maps it to one of the elements of the set of classes $C$ based on some parameter set $\Theta$. But the only thing we have been talking about so far is how do you construct this set of features that are derived from $x_m$ so that then the $f$ can do its mapping.

Let's take a look at that now: Remember how playing around with $q$ produced overlapping windows? Let's take this to the extreme. Let's think about what happens when $q=0.95$. Then, two neighbouring windows actually look very much alike. They differ only in 5% of the samples! Consequently, their spectra (adjacent vertical lines of the spectrogram) are very much alike. Therefore, it is as if we are feeding the classifier redundant data. We are showing the classifier lots of similar examples. Therefore, continuing with the above example, the classifier starts forming an opinion that MALE means all of these similar data that we are feeding into it when MALE, in our world, in the physical world, is conceptually all the different MALE voices across all recordings. A large STFT overlap could lead to overfitting!

To brake this similarity of adjacent samples, we introduce whitening. What does whitening do? It shuffles the input matrix, to reduce the covariance between adjacent samples, this is all that the whitening matrix $W$ does (but does it in a mathematical way that takes into account the covariance of the input. It produces an optimal shuffling, so optimal that any covariance between samples has been reduced to zero.)

Of course, when you bring Principal Component Analysis (PCA) into the mix, the game is changing slightly because PCA is still on the feature generation side. Applied to our previous example, PCA would enable us to get an indication of how many distinct speakers there might be in a given conversation and express the rest of the samples as coefficients to these distinct speakers (or waveforms).

So:

As a multi-dimensional array what dimensions should an audio sample's spectrogram have before PCA whitening;

It depends on your application and specifically the spectral ($N_{FFT}$) and temporal ($q$) resolutions necessary to resolve unique examples for the classifier.

What dimensions should the pre-preocessed object have so that it may be fed into a DBN?

It doesn't matter. You can choose to feed the whole thing and let the network "choose" the best input vectors by adjusting its weights OR you could run feature selection and reduce the dimensionality of your input (and the effort to train the network and the time it takes to train it, etc).

Hope this helps.

EDIT: The paper states "For the application of CDBNs to audio data, we first convert time-domain signals into spectrograms. However, the dimensionality of the spectrograms is large (e.g., 160 channels). We apply PCA whitening to the spectrograms and create lower dimensional representations. Thus, the data we feed into the CDBN consists of $n_c$ channels of one-dimensional vectors of length $n_V$ , where $n_c$ is the number of PCA components in our representation.". Therefore, the input can be considered as "2D".

• Thank you! Very generous and informative, esp for a newcomer like me. Some DBN's do seem to require X be "2D" i.e. n x m such as nolearn's dbn. I'll edit my question to specify which part of the NIPS paper I'd like to reproduce. Your insights though should put me on a good path. – Quetzalcoatl Oct 29 '16 at 20:11
• @Ivan You are welcome, glad it was helpful. I am not saying that the "2D" input is impossible, it was just the way I chose to build up to the spectrogram case via its "1D" components. Please see clarification. – A_A Oct 29 '16 at 21:17
• Thanks! So if understood correctly, for each audio sample of a speaker: produce a n x m spectrogram, apply pca whitening to it, and form a collection of n_c - many vectors where each vector has shape [n_v, 1]. Now, assuming one sample each for S speakers, our matrix X would then be a 2D matrix of shape [(n_c * S), n_v] since we have n_c *S number of "samples" each with n_v "features". – Quetzalcoatl Oct 29 '16 at 21:34
• last question: what does it mean to have 160 'channels' in a spectrogram? Do they mean 160 time frames or set of windows? – Quetzalcoatl Nov 6 '16 at 17:16
• $N_{FFT}=160$, equivalent to the "height" of the spectrogram. (The number of "time frames" is the "length" of the spectrogram, its $x$ axis). I am not sure what you mean by "windows" (?). Like, overlapping windows? That would again affect the "length" of the spectrogram. – A_A Nov 6 '16 at 17:35