I have a bunch of audio files that have both speech and music parts (think radio). For instance, a given file might consist of the following sections:

music --- speech --- music --- speech --- music --- speech --- music

My goal is to detect when the file changes between music and speech, and cut away the music parts leaving only the speech. However, I don't know what method to use to distinguish between speech and music. I was considering detecting the tempo, but some of the music is very slow or has a varying tempo.

(While we're at it, so that I don't have to write my own, are there any existing software libraries out there that can distinguish between speech and music?)

  • $\begingroup$ Sounds like a Blind Source Separation problem $\endgroup$ – Phorce Apr 1 '14 at 22:52
  • $\begingroup$ The music and speech isn't mixed (well, it is, but only for a few seconds at the start and end of the speech sections), so I don't think so. $\endgroup$ – haroba Apr 1 '14 at 23:01
  • $\begingroup$ How many mics? What is the duration of each segment? If you want to look at segmentation properties, this is difficult since there are no defining characteristics, for example the person could talk for a longer period and the music could play for a shorter period than expected. Have you got any training data (of what speech looks like, and, what music looks like)? $\endgroup$ – Phorce Apr 1 '14 at 23:03
  • $\begingroup$ Depends on the type of music. But voiced speech usually has a single pitch. Whereas music can be polyphonic and can contain pitches outside human speaking ranges (bass to soprano). $\endgroup$ – hotpaw2 Apr 2 '14 at 5:00
  • 1
    $\begingroup$ @kRazzyR, I ended up writing a solution based on this paper: speech.kth.se/prod/publications/files/3437.pdf $\endgroup$ – haroba Dec 16 '17 at 9:45

This is a well-studied problem, dating back from the mid 90s (DARPA/NIST broadcast transcription challenges). Search for "speech/music segmentation" or "audio segmentation" and you'll find thousands of research papers.

There are two broad approaches to solve this problem:

Supervised classification

Train a speech/music classifier, using a standard machine learning approach. You can use MFCCs as input features, along with other basic feature like zero-crossing rate, amplitude modulation at 4Hz, etc. Recently it became common to throw in as many features as possible, and using feature detection techniques to identify the most discriminant ones.

Any classification algorithm will do - support vector machines, gaussian mixture models, decision trees. Once the classification is done, you'll have misclassified frames (for example a tiny acapella segment in a song will be classified as speech; or a FX or jingle between speech will stand-out). This requires post-processing, the most common approach is to apply mode filtering (voting) on the sequence of classifier outputs. The classification/temporal smoothing are sometimes rolled into one through the use of hidden markov models for both classification and temporal smoothing.

Ref: Content-based audio classification and segmentation by using support vector machines, Lu et al.

Unsupervised segment change detection

Consider a 10s window sliding over the signal. Compute audio features on the first half, on the second half, and use a statistical test to decide which hypothesis is the most likely: the two sets of audio features are drawn from the same distribution, or are drawn from two different distributions. The output of the test will tell you how likely it is that the middle of the window corresponds to the boundary between a speech and a music segment. Select the points with the highest scores as the segment boundaries.

The same audio features as for the supervised approach (MFCC, ZCR, amplitude modulation at 4 Hz...) can be used.

"Textbook" criterion for the statistical test: bayesian information criterion (BIC).

Ref: Unsupervised Audio Stream Segmentation And Clustering Via The Bayesian Information Criterion, Zhou & Hansen (for an introduction to BIC).

Combined Supervised and Unsupervised approaches for automatic segmentation of radiophonic audio streams, Richard, Ramona & Essid (for more exotic change detection tests).

  • $\begingroup$ Kind Sir/Ma'am could you please point me to any python libraries that may accomplish this "detection of regions of speech and regions of music etc in an audio" ? I am trying to solve this problem but haven't made much progress. $\endgroup$ – kRazzy R Dec 14 '17 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.