# How to recognize patterns in a noisy audio frequency spectrum?

I have an audio spectrum (FFT) that looks like this:

101761.00 :                                 *
76309.98 :
57224.41 :
42912.26 :       *     *                     *
32179.65 :       *                                 *
24131.34 :                *      *       *
18095.95 :
13570.05 :          *             *     *
10176.10 :                 *                *  *
7631.00 :                                  *  *
5722.44 :                           *     *       *
4291.23 :        *     * **               *  ** *
3217.97 :        *                       *              *
2413.13 :         *            *                   *
1809.59 :    * *    *               *               *    *                                                    *
1357.00 :                  ***    **             *       *
1017.61 :                       *     **        *                                            *        *        **
763.10 :        * * * **          *                                                                     *     **
572.24 :               *   * *   *                  *      **     *    *             *
429.12 :     *              *       *         *             *    *                                             *
321.80 :   *                         *              *                                                      *
241.31 :                                *  *     *    **       * *               *                  *             *
180.96 :   *  *    *   *            * *       *                            *   *     **           *      * * *
135.70 :           **                                *             * *                *             *   *        *
101.76 :  * *                                        *     *  * *  *   *  *                       *       * *    *   *
76.31 :                  *                                                         *                 *         *
57.22 :  *                                 *   *                 *                             *   *
42.91 :                                              *   *     *   ***       ** *     *               *            *    *
32.18 : *                                                * *         **  *       **     **           *         *  *    **   *     *
24.13 :
18.10 : *                   **        *                            *   ***   *              *****      *          *   *    *  ***                  *       *
13.57 :
10.18 :     *                * *                        *   ***                *        *   ** **        **          **  *   *      *    * * **   *     *
7.63 :
5.72 :
4.29 :             *     *                       *           *   *         *    ***  *** **      *   * *          *   *          ** **     *        *  ****
3.22 :
2.41 :
1.81 :
1.36 :    *                                     *      *   *          ** **** *    *  * **   *     *            *  **   ********* ****************** ** ** **
========= : ======================================================================================================================================================
:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
:  0 0 0 1 1 1 2 2 2 3 3 3 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 6 6 6 7 7 7 8 8 8 9 9 9 0 0 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5
:  1 5 8 2 5 8 2 5 9 2 6 9 3 6 0 3 6 0 3 7 0 4 7 1 4 7 1 4 8 1 5 8 2 5 8 2 5 9 2 6 9 3 6 9 3 6 0 3 7 0 4 7 0 4 7 1 4 8 1 5 8 1 5 8 2 5 9 2 6 9 2 6 9 3 6
:  7 2 6 1 5 9 4 8 3 7 2 6 1 5 0 4 8 3 7 2 6 1 5 0 4 9 3 7 2 6 1 5 0 4 9 3 8 2 6 1 5 0 4 9 3 8 2 7 1 5 0 4 9 3 8 2 7 1 6 0 4 9 3 8 2 7 1 6 0 4 9 3 8 2 7


Basically, there's a broad peak near 500Hz, and a narrower peak around 1750Hz. And clearly there's a lot of noise in the data. (This is a snore, for those who are interested.)

A different person has a similar frequency distribution, only the peaks may be scaled up/down (based, presumably, on the resonant frequencies of the throat and mouth).

How would one distinguish that from a different sound that is more random in distribution? For instance, a choking sound looks about like this in one instance:

339889.00 :                                **
254880.80 :
191133.62 :
143330.00 :                                    *
107482.34 :                             *
80600.38 :                              *
60441.76 :                                                                          *
45324.93 :                              *   *
33988.90 :                         *        *                                    *
25488.08 :                          * *    *  *                                      *
19113.36 :                                        *                               **
14333.00 :                                 * * *  *                             *
10748.23 :                                *                                          *
8060.04 :                               *   *        *                               **
6044.18 :                     ***               * *     *                  *       *
4532.49 :                          **              *       *                   *     *
3398.89 :                 * *  *       **              *                      *
2548.81 :                     *  *            *                      *    *  *        *
1911.34 :           *                                  *       *  *                       *
1433.30 :              *        * *                  *        *    *  *      *    *
1074.82 :            *       *        *                 *   * * *                     *
806.00 :             *  * **                                  *       ***
604.42 :           *   *            *                     *    **   *      *           *
453.25 :                  *       *                     *       *      *                 *                                                  * ** *
339.89 :        *   *   *                     *  *   *             *      *                  * *   *                                         *  * *
254.88 :        *          *                       **       *      * ***                      *                                                *
191.13 :   *          *                           *         *    *                      **   **                                                     *
143.33 :                                                                               *            *             *
107.48 :    *    *                                                *     *                  *   *    *   *     *    *                    **      *   *     *
80.60 : *  *  *                                                                          *                         *                                *
60.44 :      *  *                                                                    *   *     *        *   **    *              *     *         **  *
45.32 :                                                                  *             *        ***    * *    *       **          *   *    *    *    *
33.99 :
25.49 :      *    *   *                                    *              *                    *    * * * **            *        *    *      *               *
19.11 : *   *           *    *               *          *                                  *     *       ***            *  ***  *    *  *             *
14.33 :
10.75 :       *  *         *      *             *         *   *                *                *      *       *     *   * * *  *  **                  ** *  *
8.06 :
6.04 :
4.53 : *** *  *    *          *              *   * *  **         *         *        *    ***  *   ****   * * ********** ** ***** ****  *****     *** *******
========= : ======================================================================================================================================================
:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
:  0 0 0 1 1 1 2 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 7 8 8 8 9 9 9 0 0 0 1 1 1 2 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 7 8 8 8 9 9 9 0 0 0 1 1 1 2 2 2 2 3
:  1 4 7 0 4 7 0 3 6 9 2 5 9 2 5 8 1 4 7 0 4 7 0 3 6 9 2 5 9 2 5 8 1 4 7 0 4 7 0 3 6 9 2 5 9 2 5 8 1 4 7 0 4 7 0 3 6 9 2 5 9 2 5 8 1 4 7 0 4 7 0 3 6 9 2
:  6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8 9 1 2 3 4 6 7 8


As you can see, it's a bit more disorganized (if that's possible), at least in some cases more "peaky", and, it appears, much less reproducible -- the next sound would be quite a bit different, while snores tend to look a lot alike (when standing 10 feet back, at least).

I can see attempting identification by breaking the sound up into bands and giving each band a score, then coming up with heuristics such as "band A and C should be relatively loud, while bands B and D should be relatively silent", but it would be nice if there was a more systematic approach.

Any ideas?

Until I get something better, here is a screen shot of the graph:

Here's a spectrum from Audacity of some similar data:

This is a different sound source, and I only approximated the time duration of the sample (which is 1024 samples at 8000 samples/second in the other images). The sample rate for this sound file is 44.1K, so it has much more high frequency data than is available in "real life". The first three peaks are at roughly 215Hz, 600Hz, and 2000Hz.

This next one is from a different sound file (& individual) where the sound level is much poorer. We especially have trouble with these. The peaks are about 124, 248, and 581.

## Update

I added code to build a "filter" by taking snapshots of the frequency spectrum during peak snore (as determined by an existing heuristic), summing together several successive snapshots, "smoothing" a bit, and "normalizing" to an average unity gain. Did the same with snapshots taken during periods of relative quiet. I then multiply the signal by both filters and sum (over the frequency range) the difference of the two products to produce a "delta". This seems to produce a fairly sensitive indicator of snoring.

(Did this description make sense?)

To the extent that you can understand the above description does it make sense? Is a simple multiplication the most reasonable way to apply a filter of this sort?

• Maybe the data would be more organized if you localized it by multiplying it with a window function (for example the Hann window), before computing the FFT. A multiplication in time space corresponds to a convolution in frequency space, so this would smooth out the frequency data. Select the window size depending on how accurate you want to determine the frequencies. – Thomas Klimpel Oct 15 '11 at 21:53
• @Thomas Klimpel -- I had a shot at applying a Hamming window function. It changed things a bit -- seemed to sharpen some of the peaks in the spectrum -- but didn't really provide any improvement or insight. – Daniel R Hicks Oct 18 '11 at 1:32

This is a far wider problem than merely how to compute and plot spectra correctly; I am merely trying to answer the latter.

First things first: You've got a WAV of this stuff, right? Just download Audacity and use its spectrum/spectrogram plots. They're inadequate in several ways — in particular, it has fairly poor control over window length, it's really slow, its colormap facilities are kinda poor — but it's free, it gets a lot of things right, and it gives you results pretty quickly.

Your own spectrum analysis, like Audacity's, should be using Welch's method (with 50% overlap) for improved quality. The optimal window length and function are typically very situational, but the Hann/Hanning window is usually a pretty safe choice of window function.

Note that a power spectrum plot may not even be the best way to go here — an autocorrelation approach might be better. (Sorry, I'm not a speech guy.)

Speech analysis uses frequently technique called "formant analysis" that tries to extract the areas of interest out of an audio spectrogram (e.g. http://www.phon.ucl.ac.uk/resource/sfs/howto/formant.htm). This may be overkill for what you need. A simpler approach would be

• bandpass filter to frequency region of interest
• normalize spectrum to overall energy in the spectrogram
• define a threshold, everything above the threshold is a "peak"

This should give a list of peaks in terms of position and width which you can use as a "pattern".

• What is the best way to build a simple bandpass filter? – Daniel R Hicks Oct 19 '11 at 16:23

It is customary to always plot audio spectra on a log-frequency scale unless you have a good reason not to do so. One thing that raises an alarm is that your corrupted speech seems to have an extremely low level, in fact it seems that it is of such low level that most of the speech is below the noise floor or what can be represented in Q15. That would also explain why your speech appear bandlimited.

Has the speech you record been processed in anyway?
Is the speech of such low level that your plot indicates?
What is the data type used to represent the samples (short, int, float)?

• I display all the charts with a linear frequency scale because the first were, and there's no practical way to do those log. (Had enough trouble doing the log power scale.) Yes, the sound level is extremely low, but still audible to the human ear. The data has not been processed other than FFT (with a "square" window). The original audio is 16bit, but the FFT was done in 32-bit float. But, as you can see from the printer plot, there's some granularity to the resulting data at low levels. I agree that the last plot is daunting, and is the extreme case -- the ultimate challenge, as it were – Daniel R Hicks Oct 19 '11 at 12:21
• In Audacity you can switch to log scale by just flipping the "axis" dropdown. My interpretation of what you're saying is that you can't rule out that the poor speech quality is caused by insufficient dynamic range or equivalently by very low input level. Because the original audio is 16bit damage has already been done even though the FFT is done in single-precision floating-point. If you can dump your original 16bit audio to disk (.raw is fine) and upload it here, I can tell you if the poor quality is caused by low input level (you can probably tell yourself). – niaren Oct 19 '11 at 13:05
• Certainly the poor quality is caused by low sound level and high noise -- there's no debate. I'm just trying to obtain the best results I can from poor quality input. I figure that if the human ear can hear something in the recording (which it barely can) then we should be able to extract something (which we can't quite at this stage). – Daniel R Hicks Oct 19 '11 at 16:21
• As I understand the problem, the low sound level is creating the noise/distortion. There doesn't seem to be a lot of noise in your input. Take a look at your spectrum of the degraded speech, do you see any significant noise? Your input seems to be bandlimited to approximately 1kHz which is quite severe. I doubt that even bandwidth extension techniques can improve quality. Is it not possible to upload a sample? Where is the speech coming from? Why is it of such low level? – niaren Oct 19 '11 at 20:14
• The level is so low because the sound is picked up by an iPhone microphone with the phone sitting on a table several feet from the person. The sampling rate is 8K. The frequency is weighted towards the low end because this is a male snoring -- it's not "voice" per se. – Daniel R Hicks Oct 20 '11 at 23:52

Your FFT is very poor quality, there is little point in trying to extract patterns using poor tools to gather the info.

You can check each point for it's nearest neighbour and do a dot to dot diagram.

You can use a gaussian filter to blur your spectrogram as they are not very well printed for viewing.

Your first graph has very little patterns to it at all. it is almost noise. you can perform tests to find how disparate the points are, the noisiness of it. for serious patterns you need serious detail.