Algorithm(s) to mix audio signals without clipping

Question

I'd like to mix two or more PCM audio channels (eg recorded samples) digitally in an acoustically-faithful manner, preferably in near-real-time (meaning little or no peek-ahead).

The physically "correct" way to do this is summing the samples. However when you add two arbitrary samples, the resulting value could be up to twice the maximum value.

For example, if your samples are 16-bit values, the result will be up to 65536*2. This results in clipping.

The naive solution here is to divide by N, where N is the number of channels being mixed. However, this results in each sample being 1/Nth as loud, which is completely unrealistic. In the real world, when two instruments play simultaneously, each instrument does not become half as loud.

From reading around, a common method of mixing is: result = A + B - AB, where A and B are the two normalized samples being mixed, and AB is a term to ensure louder sounds are increasingly "soft-clipped".

However, this introduces a distortion of the signal. Is this level of distortion acceptable in high-quality audio synthesis?

What other methods are there to solve this problem? I'm interested in efficient lesser-quality algorithms as well as less-efficient high-quality algorithms.

I'm asking my question in the context of digital music synthesis, for the purpose of mixing multiple instrument tracks together. The tracks could be synthesised audio, pre-recorded samples, or real-time microphone input.

Answer 1

It's very hard to point you to relevant techniques without knowing any context for your problem.

The obvious answer would be to tell you to adjust the gain of each sample so that clipping rarely occurs. It is not that unrealistic to assume that musicians would play softer in an ensemble than when asked to play solo.

The distortion introduced by A + B - AB is just not acceptable. It creates mirror images of A on each side of B's harmonics - equivalent to ring-modulation - which is pretty awful if A and B have a rich spectrum with harmonics which are not at integer ratios. Try it on two square waves at 220 and 400 Hz for example.

A more "natural" clipping function which works on a sample-per-sample basis, is the tanh function - it actually matches the soft-limiting behavior of some analog elements. Beyond that, you can look into classic dynamic compression techniques - if your system can look ahead and see peaks comings in advance this is even better.

Answer 2

The physically "correct" way to do this is summing the samples. However when you add two arbitrary samples, the resulting value could be up to twice the maximum value. ... The naive solution here is to divide by N, where N is the number of channels being mixed.

That's not the "naive" solution, its the only solution. That's what every analog and digital mixer does, because it's what the air does, and it's what your brain does.

Unfortunately, this appears to be a common misconception, as demonstrated by these other incorrect non-linear "mixing" (distortion) algorithms:

• Mixing digital audio (the wrong way)
• A quick-and-dirty audio sample mixing technique to avoid clipping (don't do this)

The "dividing by N" is called headroom; the extra room for peaks that's allocated above the RMS level of the waveform. The amount of headroom required for a signal is determined by the signal's crest factor. (Misunderstanding of digital signal levels and headroom is probably partially to blame for the Loudness war and Elephunk.)

In analog hardware, the headroom is maybe 20 dB. In a hardware DSP, fixed-point is often used, with a fixed headroom; AD's SigmaDSP, for instance, has 24 dB of headroom. In computer software, the audio processing is usually performed in 32 bit floating point, so the headroom is enormous.

Ideally, you wouldn't need to divide by N at all, you'd just sum the signals together, because your signals wouldn't be generated at 0 dBFS in the first place.

Note that most signals are not correlated to each other, anyway, so it's uncommon for all the channels of a mixer to constructively interfere at the same moment. Yes, mixing 10 identical, in-phase sine waves would increase the peak level by 10 times (20 dB), but mixing 10 non-coherent noise sources will only increase the peak level by 3.2 times (10 dB). For real signals, the value will be between these extremes.

In order to get the mixed signal out of a DAC without clipping, you simply reduce the gain of the mix. If you want to keep the RMS level of the mix high without hard clipping, you will need to apply some type of compression to limit the peaks of the waveform, but this is not part of mixing, it's a separate step. You mix first, with plenty of headroom, and then put it through dynamic range compression later, if desired.

Answer 3

the formula

$$\text{result} = A + B - AB$$

doesn't make any sense, even if you mean something other than $AB = A*B$. One thing you need to think about is that sound varies above and below zero. A better way to think about it is like this:

$$\text{result} = g( A + B )$$

where $g\le 1$.

The most simple approach is to say $g = 0.5$, which is conservative, linear and always works, but may not be as "loud" as you want. A less conservative approach that "usually works" and is "louder", is $g = 1/\sqrt{2}$. Extensions to more channels with this approach work better.

Alternatively, $g$ can change over time, in which case it's usually the result of a compressor/limiter algorithm. Then you really have a difference equation:

$$\text{result}[i] = g[i]( A[i] + B[i] )$$

$g[i]$ is a then a function of previous $A$, $B$, $g$ and $\text{result}$.

Perhaps this:

$$g[i] = f( A[i] + B[i], g[i-1] )$$

UPDATE: As suggested by hotpaw2, you can delay the input signal but not the gain suppression. This is called a "look-ahead limiter".

Answer 4

One way this can be done for non-real-time mixing to use a look-ahead AGC, where the gain of one or both channels is lowered at a hard-to-perceive rate before the sum amplitude exceeds the clipping limit. The less look-ahead available, either the the AGC gain adjustment will become more audible, or the max gain for for a softer gain adjust ramp will get closer and closer to 0.5 per channel at the limit. For sound sources with some predictability, one could also use statistics regarding the envelope's behavior over time to adaptively guess a gain limit, but with some probability of failure (which would be an abrupt AGC gain adjustment).

Answer 5

I had talked with a mixer designer of the late 1990's and first 2000's that was going on the digital wave (after having tiptoed). I think the guy was a designer for SPL, but maybe not that big, I absolutely don't remember neither the name neither the brand, I just remember how really really big and expensive the machine was.

We spoke long, and finally spoke about the techniques for really guaranteeing that their 64/128 @ 24bits channels mixed together were remaining a 24 bits accurate mixed output channel without clipping.

The technique he explained was rather simple. The 64 tracks (on 24 bits) were added in a 48 bits channel, where the clipping cannot occur. Straight.

I cannot say how that signal was then dithered 48 back from to 24 bits. Maybe that's where the tricky kitchen recipes are applied.

And there maybe are a lot of techniques to achieve that, above all different whether done in real-time or with the all signal already recorded with high-peaks simple to determine... all kind of normalizations to imagine I think.

Answer 6

Lower the global volume. Impulse tracker classically outputs channels at about 33% volume max by default.

That seems to be both loud enough for music with few channels (4 channel Amiga MODs) and soft enough for songs with 50 channels (since channel contents are typically not correlated so volume doesn't add up that fast past a certain level... plus few channels will be outputting at max volume with that much stuff going on). It also leaves enough headroom for hard-left or hard-right panned channels (which use 66% of the range).

Also you don't want to add your channels together in 16bits, you want to add them together in 32bits, then clip the result and reduce to 16bits at the very end. You will need the higher range so that it doesn't wrap around while doing the math. Another option is to use 32bit floating point (which is convenient for doing filters, effects etc).

Answer 7

I think the key is, if you have 16 bit values and your adding 2 values together that could potentially be more then the maximum value, then you have 2 options:

1) cast both to 32 bit add then return the maximum value if the addition exceeds that value. Then cast it back to 16 bits. For example if your values are 32768 and 34567 it exceeds 65535 and the key is to then return 65535. You would do the same thing if using signed values at the minimum value end.

2) compress both values, then add them together.

The first is essentially hard clipping, the second is soft clipping. Analog systems are all hard clipping.

Answer 8

They will only be twice the space if the frequencies in both tracks take up the same space frequency wise. Use eq and compression to get around this by carving out areas of the frequency spectrum for each sound and controlling the transients and sustains of sounds so that everything pokes out where it should. Maybe that doesn't answer the question though. You could delay lower frequency signals by up to 2 ms. It won't cancel through phase because the wavelengths are longer than the higher frequencies and it will add space because the transients aren't completely in time with the power hungry bass signals. Something linearly adding more of a delay the lower the signal would be interesting to test out.

Answer 9

A + B + {
    (|A| = A) = (|B| = B) = true: -AB;
    (|A| = A) = (|B| = B) = false: AB;
    else: 0
}

That is, if both A and B share a sign, apply a limiting offset. The magnitude of the offset is the product of A and B. The direction of the offset is opposite to that of A and B.

If A and B do not share a sign, no limit is applied, as there is no way to overflow.

Answer 10

My suggestion:

1. Convert track audio format from 16bit fixed point to 32bit floating point.
2. Add the current sample value of all tracks to be mixed.
3. Don't do anything else.

The user may wish to process this mixed stream with compression and/or limiting prior to dithering and reconversion to 16bit fixed point format (assuming this conversion... mixdown to hand off to mastering engineers is usually left at a higher resolution format)