While looking for an answer to this problem, I found this board so decided to cross post this question of mine from Stack Overflow.

I am searching for a method of determining the similarity between an audio segment and a human voice, which is expressed numerically.

I've searched quite a bit, but what I've found so far (detailed below) doesn't really fit what I need:

  • One method is to use speech recognition software to obtain words from an audio segment. However, this method is unable to come up with how "similar" audio is to human speech; it can often tell whether or not there are words in the audio, but if there are no definite words, it can't tell close the audio is to having such words.
    Examples: CMU Sphinx, Dragonfly, SHoUT

  • The more promising method is referred to as Voice Activity Detection (VAD). However, this tends to have the same problems: the algorithms / programs using VAD tend to just return whether or not the activity threshold has been reached, and no "similarity" value before or after such threshold. Alternatively, many just look for volume, not similarity to human speech.
    Examples: Speex, Listener, FreeSWITCH

Any ideas?

  • 1
    $\begingroup$ From your question it's not clear if you problem is A/ take an audio recording and says whether it contains human speech or not (example of application: detect and remove DJ talk from a recording of a radio show) ; or B/ take speech input and says how similar it sounds to a reference audio segment. In case it's B, on which criteria do you want to measure the similarity. On melody contour? (eg: matching a sung voice to a song). On rhythm and cluster classes? (eg: matching beatboxing/onomatopoeia to a drum loop). On timbre? (matching a voice to sound effect). Please tell us your application. $\endgroup$ Commented Jun 16, 2012 at 8:25
  • $\begingroup$ Sorry, my problem is what you detailed in A. I wish to determine whether or not an audio segment is human speech. $\endgroup$ Commented Jun 16, 2012 at 13:43
  • 1
    $\begingroup$ I've been working on a related problem -- trying to determine when the sounds of snoring/breathing have been "polluted" by speech or music. It is frustratingly difficult to do with any reliability, and without resorting to "advanced" speech recognition algorithms (if they, in fact, would even work). My one observation would be that speech tends to have an irregular rhythm, whereas music is (usually) regular. That and maybe "spectral flatness" is worth checking into (I'm still evaluating its merits for my purposes). $\endgroup$ Commented Jun 16, 2012 at 20:10
  • $\begingroup$ (A hair more detail: I find that the normalized standard deviation of spectral flatness computed from the FFT of the audio seems to reflect something of audio quality.) $\endgroup$ Commented Jun 18, 2012 at 20:58

1 Answer 1


This kind of problem is usually solved using machine learning techniques.

Break down the signal into a sequence of 20ms or 50ms frames. Extract features on each frame. MFCC are generally good for this kind of application, though there are feature more specific to voice detection (4 Hz modulation energy - which is roughly the rate at which people speak ; zero-crossing rate).

Then, using a training set of audio you have manually labelled as being speech / not speech, train a classifier (Gaussian mixture models, SVM...) on the frames features.

This will allow you to classify unlabelled frames into speech/non-speech classes. The last step consists in smoothing the decisions (a frame classified as non-speech surrounded by hundreds of speech frame is likely to be classification error), for example using HMMs, or just a median filter.

A few references:

Robust speech / music classification in audio documents (Pinquier & al) Speech / music discrimination for multimedia applications (El-Maleh & al) A comparison of features for speech / music discrimination (Carey & al)

Note that the features and classification techniques they describe are also relevant for the 1-class problem of detecting speech (instead of discriminating speech vs something else). In this case, you can use 1-class modeling techniques such as 1-class SVM, or just take the likelihood score out of a GMM trained on a speech data as a "speechiness" measure.

If, on the other hand, your problem is really discriminating speech vs something else (say music), you could also very well use unsupervised approaches which are focused on detecting the boundaries between similar audio content - rather than identifying this content itself.

  • $\begingroup$ Thanks, this helps a ton! What is the benefit of breaking down the signal into small windows? Because the output I am looking for is a numerical value that describes the entire audio segment, would it be better to extract features for the entire signal and not just specific windows? $\endgroup$ Commented Jun 16, 2012 at 14:37
  • 1
    $\begingroup$ Computing the features (particularly the spectral or cepstral ones) over very long windows would average or cancel some of the properties that make speech stand out. You can verify this yourself by mixing together many short fragments of speech - it will be hard to recognize the result as speech. It is thus better to perform the classification on small segments ; and, in your case, aggregate the scores (for example compute the average of the likelihood score given by a GMM ; or compute the % of frames classified as speech by a binary classifier). $\endgroup$ Commented Jun 16, 2012 at 15:19
  • $\begingroup$ To be more precise, keep in mind that the temporal dimension is "collapsed" when you look at a spectrum. For example, the power spectrum of a 500 Hz tone followed in time by a 1kHz tone is similar to the power spectrum of those two tones played simultaneously ; so the power spectrum, over a long window, of a signal that changes a lot might not look very representative of the signal's content. $\endgroup$ Commented Jun 16, 2012 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.