# Synchronization of two audio tracks via Mel-frequency cepstral coefficients (MFCCs)

I am trying to solve the (common) problem of synchronizing (i.e. measuring the delay between) two audio tracks recorded from different sources during a conference talk (about 1h of audio). One track is recorded with the not-so-good video-camera internal mic (but in sync with video), and the other one recorded directly from the mixer output (good audio quality). The final goal is obviously to superimpose the good audio from the mixer to the video track. By the way, I am trying to implement this on my own in python.

I have already tried to calculate the cross-correlation between selected segments (ranging from 2min to 20min) of the two (normalized) raw audio files, but it appears that the result is very noisy and extremely dependent on the choice of the segment length and position, making it difficult to search for well defined correlation peaks.

I have stumbled into the approach proposed here by @pichenettes and I found it really interesting:

• Extract a sequence of MFCC vectors from both signals. This way you'll have something of lower dimensionality and a bit more robust to noise and differences in transducers.
• Optionally normalize the MFCC ("by design" coefficient 0 has more variance than coefficient 1 and so on...)... Ideally you'd want each row of the MFCC matrix to have variance 1.
• Compute the cross-correlation of two matrices you get, along the time axis (that is to say, compute the 2D cross-correlation and just keep the data for the time axis).

However, I have trouble dealing with the result of this correlation. In my understanding, I would then search where the correlation is maximum in terms of frame index, and then calculate the delay from such index. The point is that after calculating the correlation, I am left to deal with this quantity for 13 cepstral coefficients, which also turns out to be strongly dependent on the coefficient itself. My question is: how can I deal with this array of 13 cross-correlations to calculate the track delay? Am I thinking this in the wrong way?

It is probably useful to mention that the delay between the tracks (measured by ear) is about 18-19s. The sync resolution I need would be something slightly below the 25fps of the original video, e.g. 10ms is definitely ok.

Thank you!

## 1 Answer

I think, you do not need to follow complicated steps of MFCC. I recommend you to preprocess the signal. So you can remain the portion between 300~4000Hz. (1. FFT, 2. assign zeros for the frequencies for smaller than 300 and larger than 4000Hz. 3. IFFT) And then you can do cross-correlation stuffs like what you did. Hope it helps. You can understand the idea of MFCC from here.

Then you will understand why I don't recommend you MFCC but to remain 300~4000Hz.

• MFCC might be useful if you are doing DTW alignment. – Aaron Oct 12 '17 at 4:04