7
$\begingroup$

I would like to implement an algorithm that synchronizes two audio tracks.

The two tracks are very similar; you can imagine they are two recordings of sound (not necessarily music) from two microphones in the same room. This means they appear as one signal that passed through two different (linear) channels, with different additive noise.

I understand the first idea might be to look for the peak of their cross-correlation, but I am particularly interesting in finding literature on the topic, but all the papers I could find are either synchronizing two musical tracks (using chroma-based audio features), or synchronizing audio with a score. They also assume there might be time-warping, which is an unnecessary assumption in my case.

$\endgroup$
4
$\begingroup$

Chroma-based approaches are used when the tracks to align are two performances of the same material with widely different instrumentations - in which case you need to find the "lowest common denominator" which is harmony, melody, chord progression etc. Chroma are good for that - they totally abstract timbre, recording techniques. But you don't need those in your case.

As you have guessed, cross-correlation of the raw audio would be 1/ not very robust to noise, 2/ not very robust to differences in transducers (the two microphones might have very different responses) 3/ very costly if you have minutes of audio.

I suggest you to:

  • Extract a sequence of MFCC vectors from both signals. This way you'll have something of lower dimensionality and a bit more robust to noise and differences in transducers.
  • Optionally normalize the MFCC ("by design" coefficient 0 has more variance than coefficient 1 and so on...)... Ideally you'd want each row of the MFCC matrix to have variance 1.
  • Compute the cross-correlation of two matrices you get, along the time axis (that is to say, compute the 2D cross-correlation and just keep the data for the time axis).
$\endgroup$
  • $\begingroup$ Interesting approach. What kind of time resolution can you get from a MFCC analysis? Is this limited by the spectral resolution? $\endgroup$ – Hilmar Feb 12 '12 at 1:04
  • $\begingroup$ Poster did not say which resolution he expected. This approach is limited to the resolution of the MFCC analysis, typically done at 50 to 100 frames per second. This is enough for applications involving speech and "everyday audio" (eg aligning takes from multiple non TCed cameras in video editing) ; probably not music. To get more resolution, I suggest doing a "coarse grained" alignment using MFCC to get an alignment at +/- 10ms (duration of a FFT frame) ; then to take a few chunks of 100ms of audio every 10s or so from both source ; and to do an exhaustive matching using those. $\endgroup$ – pichenettes Feb 12 '12 at 4:33
  • 2
    $\begingroup$ Actually, the cross-correlation method is usually quite insensitive to noise. The cross correlation of the non-noise part of the sound will create a spike in the cross-correlation. On the other hand, since noise is random, the noise in one recording is unlikely to correlate in any specific way with anything in the other recording, and hence will only give rise to more noise in the cross-correlation. This noise is very weak in comparison to the spike, and hence the precision when localizing the spike will be almost unaffected. $\endgroup$ – HelloGoodbye Feb 1 '16 at 10:11
  • $\begingroup$ I'm skeptic to whether slightly different responses will play such a big role either, for that matter; you would still get a spike in the cross-correlation, although perhaps translated very slightly, which would in that case be caused by a delay in one of the transducers. However, if you have such a delay, I don't see how MFCC would improve upon this. $\endgroup$ – HelloGoodbye Feb 1 '16 at 10:23
  • $\begingroup$ Finally, even if you do have minutes of audio, the cross-correlation can be calculated very efficiently using FFT. I suspect that extracting a sequence of MFCC vectors from both signals would take significantly longer time. $\endgroup$ – HelloGoodbye Feb 1 '16 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.