2
$\begingroup$

I am trying to solve the (common) problem of synchronizing (i.e. measuring the delay between) two audio tracks recorded from different sources during a conference talk (about 1h of audio). One track is recorded with the not-so-good video-camera internal mic (but in sync with video), and the other one recorded directly from the mixer output (good audio quality). The final goal is obviously to superimpose the good audio from the mixer to the video track. By the way, I am trying to implement this on my own in python.

I have already tried to calculate the cross-correlation between selected segments (ranging from 2min to 20min) of the two (normalized) raw audio files, but it appears that the result is very noisy and extremely dependent on the choice of the segment length and position, making it difficult to search for well defined correlation peaks.

I have stumbled into the approach proposed here by @pichenettes and I found it really interesting:

  • Extract a sequence of MFCC vectors from both signals. This way you'll have something of lower dimensionality and a bit more robust to noise and differences in transducers.
  • Optionally normalize the MFCC ("by design" coefficient 0 has more variance than coefficient 1 and so on...)... Ideally you'd want each row of the MFCC matrix to have variance 1.
  • Compute the cross-correlation of two matrices you get, along the time axis (that is to say, compute the 2D cross-correlation and just keep the data for the time axis).

However, I have trouble dealing with the result of this correlation. In my understanding, I would then search where the correlation is maximum in terms of frame index, and then calculate the delay from such index. The point is that after calculating the correlation, I am left to deal with this quantity for 13 cepstral coefficients, which also turns out to be strongly dependent on the coefficient itself. My question is: how can I deal with this array of 13 cross-correlations to calculate the track delay? Am I thinking this in the wrong way?

It is probably useful to mention that the delay between the tracks (measured by ear) is about 18-19s. The sync resolution I need would be something slightly below the 25fps of the original video, e.g. 10ms is definitely ok.

Thank you!

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

I think, you do not need to follow complicated steps of MFCC. I recommend you to preprocess the signal. So you can remain the portion between 300~4000Hz. (1. FFT, 2. assign zeros for the frequencies for smaller than 300 and larger than 4000Hz. 3. IFFT) And then you can do cross-correlation stuffs like what you did. Hope it helps. You can understand the idea of MFCC from here.

Then you will understand why I don't recommend you MFCC but to remain 300~4000Hz.

Improve this answer
$\endgroup$
1
  • $\begingroup$ MFCC might be useful if you are doing DTW alignment. $\endgroup$
    – Aaron
    Oct 12, 2017 at 4:04
0
$\begingroup$

MFCC was developed for speech recognition, and works well for that purpose. That is to say, it discriminates well between phonemes. But phonemes are all part of human speech.

The raw Mel coefficients used in the MFCC are much less useful to discriminate between phonemes, but they're much better at discriminating between human voice and other audio sources.

Improve this answer
$\endgroup$
0
$\begingroup$

You have two distinct data sources, each with their own electronics (which are black boxes and can be non-linear).

I would use a DTW algorithm. With it you can match segments and find their respective delay.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.