# Detection of regions of voice in music

I want to detect the regions of voice in a song. Currently, I am using the fact that the vocals in the song are recorded as center panned. So, I am removing all center panned audio from the song. However, this removes other instruments such as bass also. So, it sometimes gives false indication of voice being present.

I want a way ( better? than this naive approach?) to be able to detect portions of the song where a singer is singing. There is no specific type of music, I'd like an implementation for any type of song. A sample might be http://www.youtube.com/watch?v=w_Rut4qm33g but I up for any song for which this can be done, even if it has some specific properties.

My current implementation is -

[y, fs] = wavread('song.wav');

subplot(2, 1, 1);
plot(y)

left = y(:,1);
right = y(:,2);
yOut(:,1) = (left - right)/2;
yOut(:,2) = (left - right)/2;
wavwrite(yOut, fs, 'tmp2.wav');

fftL = fft(left);
fftR = fft(right);

N = size(fftL)
N = N(1);

M = size(fftR)
M = M(1);

N = min(N, M);

for i = 1:N
dif = fftL(i,1) / fftR(i,1);
dif = abs(dif);
if (dif > 0.7 && dif < 1.5)
fftL(i,1) = 0;
fftR(i,1) = 0;
end;

end;

leftOut = ifft(fftL);
rightOut = ifft(fftR);
yOut(:,1) = leftOut;
yOut(:,2) = rightOut;

subplot(2, 1, 2);
plot(yOut)

N = size(yOut);
N = N(1);
for i = 6:N
yOut(i) = 1/5* (yOut(i-1) + yOut(i-2) + yOut(i-3) + yOut(i-4) + yOut(i-5));
end
wavwrite(yOut, fs, 'tmp.wav');

• And the question is? – pichenettes Apr 14 '13 at 0:40
• Welcome to DSP.SE! You've stated a problem, but you haven't asked a specific questions. the StackExchange sites work best when asking specific questions. You might also want to supply more information about the sorts of music, even give a link to a short example. Ask a question, we'll try to answer it. – Peter K. Apr 14 '13 at 0:47
• Sorry, In my confusion I forgot to state the problem. Please see the edit. – ffledgling Apr 14 '13 at 18:40

Warning: this is still an active research topic so there is no "canned" solution.

As you have stated, middle channel suppression is not a very robust method. It can help in getting a very rough "Karaoke" version but you can't infer anything reliable from the resulting audio.

A simple approach would consist in extracting a bunch of audio features on each signal frame, and use machine learning techniques (for example support vector machines) to discriminate the "voice present"/"no voice" classes. This works relatively well (reported recognition rates in the literature are in the 70-80% range), but the downside is that you have to manually annotate material to get training data. You might try to use karaoke files (with precise timing of the lyrics) and align them with the audio to get a kind of automatic transcription, but remember that your system will only be as good as your training data. Your training database will have to cover all the possible genres of music on which your classification system will be unleashed. See Rocamora's Master's thesis or Ramona et al's paper. There are dozens of other papers presenting variations of this scheme - with different feature sets, post-processing/temporal-smoothing operations, etc. Follow the references in the papers.

If you can't afford to spend countless hours in Transcriber or Wavesurfer annotating data, there are still unsupervised methods that would reasonably work. I recommend you to look into lead melody separation methods such as:

• "REpeating Pattern Extraction Technique (REPET): A Simple Method for Music/Voice Separation" (Rafii / Pardo). This is a very simple method which works very well on "commercial" pop music (with a very stereotypical structure and modern computer-based production).
• "Source/Filter Model for Main Melody Extraction From Polyphonic Audio Signals" (Durrieu et al). Complex algorithm, but the code is available.
• If you want something really simple, just extract and track the most salient pitch and assume it is the lead melody. This works to some extent in recordings in which the singer voice is well isolated from the mix (other instruments EQ'ed so that they do not overlap much with it).

You don't need to bother here with resynthesis - just use the energy of the extracted spectrum / f0 track.

A last angle of attack is to use a multi-f0 estimator or a sinusoids tracker, and check if one of the tracks has frequency fluctuations typical of vibrato or glissando. In commercial pop music, most instruments have a very stable pitch contour (keyboards, fretted bass and guitars), and the voice is the only instrument with a very wobbly pitch contour - unless it has been doctored by autotune.