2
$\begingroup$

I've got several (4) not-great audio recordings of the same short (a few minutes long) performance, each taken from different microphones. I'd like one high-quality clip of the event. This is... difficult. Likely impossible. I think I'm on a wild goose chase, hopefully there is a pre-existing way to do this?

Challenges:

  • Someone in the audience was talking near one of the microphones at the beginning.
  • The mics have slightly different characteristics.
  • The audio was compressed when it was first recorded. :(
  • I'm not mathy enough to know how to use beamforming.

Is there a better way than the following:

  1. Sync up the recordings using FFT (something like https://github.com/mileshenrichs/QuiFFT, I'm a little fuzzy on the details, but I bet I can get it to work)
  2. I'm not sure if the time-sync will be precise enough. People's voices go up to ~250 Hz, so to get the signals really aligned, I need like... 4x that? 1/1000 of a second precision? The recordings were all pretty compressed, but decoded to CD frequency, 44,100 Hz - yikes. That is a VERY precise alignment.
  3. Average out the synced-up recordings. Possibly using Mode? (Does that idea of getting the Average/Mode of a sound signal even apply here? I'm more used to stacking images!)
  4. Save out the averaged signal, recompressed along with the video to a common format that video-sharing sites can use. (ffmpeg is happy to export AV1 and Youtube is happy to accept it)
$\endgroup$
1
  • $\begingroup$ Maybe you could find some information in the paper "Automating Mixing of User-Generated Audio Recordings from the Same Event" by Stefanakis, Mastorakis, Alexandridis and Mouctaris (link: aes.org/e-lib/browse.cfm?elib=20452). Unfortunately I can't provide the paper, just a link to it. $\endgroup$
    – ZaellixA
    Apr 11, 2022 at 11:55

1 Answer 1

3
$\begingroup$

Problem Statement

It sounds like you are trying to solve the Cocktail Party Problem, which is to separate different signals (e.g., multiple people speaking simultaneously at a party) that have been mixed together and recorded on multiple channels. It's an example of a Blind Source Separation problem: you don't know the individual signals that were combined, and you don't know how they were combined either! Mathematically, this can be represented as: $$ Ax = b $$ where $x$ contains the desired source signals (unknown), $A$ is the unknown mixing matrix, and $b$ is the known, mixed signals.

We want to solve for $x$ to get the original unmixed signals, which would be fairly easy if we knew $A$, but we don't. The only way to solve this problem is to make some assumptions about the underlying signals and/or mixing in order to constrain the problem.

Independent Component Analysis

A common way to solve this is to use Independent Components Analysis. ICA constrains the problem by assuming the components have independent, non-Gaussian distributions. It often works quite well for the type of problem you describe, and there are straightforward computing packages such as Fast ICA in multiple languages (see the External Links section of the Wikipedia page). I would recommend giving that a try.

Things get complicated for ICA if there is a time delay between the signals, which is a form of the Convolutive Blind Source Separation problem (see Independent Component Analysis, by A. Hyvärinen, J. Karhunen, and E. Oja, pg. 361ff, available free here). So depending on how much delay there is, vanilla ICA might struggle.

Independent Vector Analysis (IVA)

IVA by Taesu Kim is supposed to be able to handle delays, and is considered one of the more robust BSS algorithms. Kim's IVA code is available here.

Beamforming

I know you wanted to avoid this, but another method would be beamforming, and especially adaptive beamforming, which is simpler than ICA in some ways. If you know the positions of the microphones relative to each other, and roughly the direction of the speaker of interest, it might be the most straightforward. The maximum SNR gain you could get under ideal conditions from combining 4 channels would be 6 dB.

Beamforming actually seems the closest to what you describe in your potential solution. In a nutshell, all it does is use the known geometry of the microphone array to compensate for the different relative delays of a signal from a particular direction. Once you compensate for the delays for the desired direction, you can sum the channels together. The signal coming from the direction of interest will sum coherently, and the noise won't, resulting in a net SNR gain by a factor of N (ideally).

If you have a loud interference signal coming from a different direction (e.g., the loud speaker closer to one of the microphones), adaptive beamforming can null the interference in addition to beamforming to the desired signal direction.

Practical Note

Since the signals are long, and the interference isn't the same throughout the recording, it would probably work best to break the signal into shorter chunks and do ICA, IVA, or adaptive beamforming on each segment separately, then put them back together.

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.