# Detecting delay and sample drift in audio files (wav, mp3, etc.)

In my application, I have an audio recording with background music. I know which song is playing in the background and have a copy of the original song (wav, mp3, etc.). The goal is to align the original song and the song in the recording with a high degree of accuracy so that I can subtract out the song from the recording. Unfortunately, most recorded songs will have some amount of time stretching due to sample drift in the playback device (e.g. up to $1-2\%$ in older CD players, meaning typical length songs can be $\pm 3\text{ to }5$ seconds longer). In addition, the background music will often overlap with other sounds (voices, noise, etc.).

I have a fairly robust solution for roughly estimating the time delay using image processing techniques on the spectrograms of the input signals (see code from this blog). Basically I take smaller chunks of the spectrogram from the original song (e.g. $10$ seconds) and slide it across the recording, looking for peaks in the normalized cross-correlation. I generally find a strong peak, as shown below, even with lots of voices talking, noise, and reverberation.

However, when it comes to estimating the relative time stretch between the signals, I'm at a loss.

• How do I know if peaks correspond to the correct portion of the song (and are not just caused by correlation between sections of the song)?
• How do I find secondary peaks with enough confidence that I can use those to calculate sample drift?
• How do I (with high confidence) know if there is even a song playing (there will possibly be recordings where the song isn't playing for long durations).
• Are there simpler time-domain techniques I'm missing here?
• so the one sound buffer is delayed about 110 samples from the other. if there is drift, apply the same cross-correlation to windowed sounds at a later time and see if the offset remains 110 samples. – robert bristow-johnson Dec 30 '16 at 18:05
• So this is the best approach I can think of and I'm going to try and implement this today. The main challenge here is finding "clean" sections of each buffer to compare; the recording will often have voices, noise, reverberation, etc.. I know I could employ something like Voice Activity Detection (VAD), but that's a lot of work (and computation). I also need to detect the drift with very high accuracy. According to my rough calculations, for a five minute song, the windowed sections of audio I cross-correlate need to be at least a couple seconds apart to estimate drift accurately enough. – user2348114 Dec 30 '16 at 19:18
• once you have a good idea of the delay (for a given segment of sound) there is an alg called the LMS adaptive filter (the Normalized LMS is a little better) that you can use to subtract the background sound. it might be useful to look that one up. but cross-correlation to frame up the background music with your recorded copy is useful to do first. – robert bristow-johnson Dec 30 '16 at 22:05
• Yeah already tried NLMS. Even with no sample drift, it's assumption that the channel is linear causes it to break down for many use cases. It works well for simulated data (even when noisy and reverberated), or when you have transducers with high linearity (e.g. expensive speakers and mic). But as soon as you start playing audio from some crap speaker or recording it from a little mic in a phone, the filter quickly diverges and produces garbage output. I've researched more advanced echo cancellers that deal with non-linearity, but of course companies don't release that source code. – user2348114 Dec 31 '16 at 1:18
• Additionally (ran out of space), the filter order of the NLMS filter determines the length of the delay it can detect. With sample drift, the peak in the impulse response indicating the delay quickly drifts out of range. So the NLMS filter would have to already know (or estimate) the delay and slow down as it went through the audio (possible but adds complexity). – user2348114 Dec 31 '16 at 1:30