How do I go about detecting whistles, pops and other sounds in live audio input?

Question

I've read many questions on SO, and frankly, each one of them is not describing any particular way to go about it. Some say "do FFT" and some say "zero crossing" etc. But I've only gone as far as understanding that the digital audio input consists of an array of amplitudes for a particular frequency and well.. Don't really know much beyond it.

Now I do know Nyquist theorem, frequency, amplitude, Fourier series etc, and that's from 2-3 years ago when I did that in my university program in some semester. But back then we really weren't taught the real world usage of Fourier much, and I didn't bother to dig up more on the subject beyond learning just enough to pass the subject. But now I'm gonna have to use all that stuff.

Here's a snapshot of the sounds I'm trying to detect:

Clearly the sounds have unique graphs. I just want to understand how to extract their particular unique characteristics for their unique spiky lines in the graph. Like what amplitude, frequency, etc. And for how much time - although that is trivial I guess.

I want a simple step by step, un-vague instruction list - I can google the terminology I won't understand.

Like perhaps this? --

1. Get input audio data

2. Plot spectogram

3. Get spectogram graph for the sound you want to detect in a noiseless environment

4. Study that graph - draw unique characteristics of that sound

5. Make some kind of function that can detect those particular characteristics in live audio feed, using the characteristics of the sound found in (4)

6. If a match is found, well - the job's done.

7. Polish the algorithm to weed out false negatives.

Where I am confused is - how do I go about 3,4,5. Particularly 3 and 4. Because I'm making a software where in the user would be able to record any sound as a gesture input to be used later, I would want my system to be able to record any sound and match it against live audio feed to detect that sound gesture. Like say, coughing, or finger snapping, or whistling, or clapping. Or saying consonants - $A$ or $B$ or $X$ or $Z$. Just about any sound.

I was thinking of making the user record the sound they want to store as a gesture in a quite environment. And the user would make the sound only in between the quiet time padding; 3 secs in the beginning and in the end of the recording.

So say, during the first 3 secs, my system would establish that the current input is the normal quiet background sound. And then a sudden change in the graph would be the sound input starting. And when that stops, the recording would go on for another 3 secs, the trailing quiet time padding. This would be done manually by the user. It would then automatically store the characteristics of only that part during which the sudden change in the graph lasted - somewhere between the padding times.

And thus the characteristics of that part would be saved as that sound's gesture data, which would be used to detect that particular sound in live audio feed later on.

Problem is, that I'm thinking all this in plain English. I need to think in maths and physics, to be able to implement it efficiently in my code. I'm just so damn clueless about what to write and where to write it in my code - even with so many libraries and questions on SO at my disposal.

And sorry if this was long.

Answer 1

While I agree with one of the comments that using speech recognition techniques might be a good start, these sounds are different and I'm not aware of anyone having done any research into categorizing them (the paper cited by Nathan appears only to distinguish between speech and noise), so unless someone presents something to contradict that, you are going to have to invent your own technique, and that's going to take a lot of learning and a lot of work. The best I can do is get you started.

First off, don't expect anyone to be able to produce some magic formula. There's no equation to get from sound to what the sound is. Humans and computers must learn by incorporating data to make educated guesses about what sounds are. The reason people say in their answers "use FFT" or "use zero crossing" is because these are some of the basic DSP building blocks used in speech recognition and related algorithms. But the FFT and zero crossing rate are usually just first steps in creating a set of parameters describing the sound. These parameters are then analyzed statistically (not through some magic function) to determine which category they most likely belong to. Note that I said "most likely": even the best speech detection (and the human brain!) has a high error rate at this stage and further processing using knowledge of the language and grammar is used to correct these mistakes.

So, some parameters you could look for include:

• zero-crossing rate
• frequency centroid
• envelope (this is actually a set of parameters, including, eg attack time)
• spectral envelope
• even/odd harmonic balance
• voiciness
• fundamental pitch

Once you have a set of parameters that you think will allow you to distinguish your sounds, you will need to use some statistical method to categorize them. Hidden Markov Model is often used in speech. You can also look into logistic regression, K-means, and I'm sure there are other choices, but I think HMM is tried and true.

Answer 2

I used this paper on entropy based end pointing when I was trying to ignore these kinds of sounds in phone calls, used for computer speech recognition, if the sounds you are try to capture is speech then entropy could work really well, for music it may not be useful.

Answer 3

I think Bjorn's recommendations are very good, but I want to provide some additional information. From your description, this sounds a lot like a timbre identification problem. There is some research in this area in computer music contexts (identifying different instruments is a useful thing to be able to do and telling the difference between a snap and clap is because of the sound's timbre). William Brent has done some research in this area (look for timbreID on his page) and created some software to use in Pure Data. In each of the situations you're looking for specific events, so doing some automatic segmentation by onset detection would be a good idea. Since you're already taking an STFT, determining an onset wouldn't require too much additional work (look into Spectral Flux onset detection). A system outline could be as such:

Training

• Determine appropriate features (e.g. zero-crossing rate, spectral centroid, MFCC's)
• Detect an onset
• Calculate these features on input audio (your gesture recordings)
• Keep a database of feature calculations. You will need to determine whether this is a supervised or unsupervised training situation. For example, do users specify beforehand that a snap is "sound 1" and a clap is "sound 2", or does your system try to cluster these after training.

Classification

• Detect an onset in real-time and calculate features on incoming audio
• Use a distance metric (euclidean distance or another Lp distance) to do a nearest neighbor or K-nearest neighbor classification to determine the "closest" sound from your database

This paper on percussive timbre identification could be of some use. It outlines definitions for potential features to calculate on the incoming sound and the author's classification method. This will work fairly well for percussive sounds, but might not work as well for something like speech (multiple syllables), in that case an HMM method would be more suitable. Similarly, your onset detection's accuracy will vary depending on the types of sounds you're looking for.

If you're particularly concerned about snaps vs. light switches, then spend some time figuring out which features could accurately discriminate between the two sounds.