Detecting changes between voice and music

Question

I have a bunch of audio files that have both speech and music parts (think radio). For instance, a given file might consist of the following sections:

music --- speech --- music --- speech --- music --- speech --- music

My goal is to detect when the file changes between music and speech, and cut away the music parts leaving only the speech. However, I don't know what method to use to distinguish between speech and music. I was considering detecting the tempo, but some of the music is very slow or has a varying tempo.

(While we're at it, so that I don't have to write my own, are there any existing software libraries out there that can distinguish between speech and music?)

Answer 1

This is a well-studied problem, dating back from the mid 90s (DARPA/NIST broadcast transcription challenges). Search for "speech/music segmentation" or "audio segmentation" and you'll find thousands of research papers.

There are two broad approaches to solve this problem:

Supervised classification

Train a speech/music classifier, using a standard machine learning approach. You can use MFCCs as input features, along with other basic feature like zero-crossing rate, amplitude modulation at 4Hz, etc. Recently it became common to throw in as many features as possible, and using feature detection techniques to identify the most discriminant ones.

Any classification algorithm will do - support vector machines, gaussian mixture models, decision trees. Once the classification is done, you'll have misclassified frames (for example a tiny acapella segment in a song will be classified as speech; or a FX or jingle between speech will stand-out). This requires post-processing, the most common approach is to apply mode filtering (voting) on the sequence of classifier outputs. The classification/temporal smoothing are sometimes rolled into one through the use of hidden markov models for both classification and temporal smoothing.

Ref: Content-based audio classification and segmentation by using support vector machines, Lu et al.

Unsupervised segment change detection

Consider a 10s window sliding over the signal. Compute audio features on the first half, on the second half, and use a statistical test to decide which hypothesis is the most likely: the two sets of audio features are drawn from the same distribution, or are drawn from two different distributions. The output of the test will tell you how likely it is that the middle of the window corresponds to the boundary between a speech and a music segment. Select the points with the highest scores as the segment boundaries.

The same audio features as for the supervised approach (MFCC, ZCR, amplitude modulation at 4 Hz...) can be used.

"Textbook" criterion for the statistical test: bayesian information criterion (BIC).

Ref: Unsupervised Audio Stream Segmentation And Clustering Via The Bayesian Information Criterion, Zhou & Hansen (for an introduction to BIC).

Combined Supervised and Unsupervised approaches for automatic segmentation of radiophonic audio streams, Richard, Ramona & Essid (for more exotic change detection tests).