49
$\begingroup$

I'd like to mix two or more PCM audio channels (eg recorded samples) digitally in an acoustically-faithful manner, preferably in near-real-time (meaning little or no peek-ahead).

The physically "correct" way to do this is summing the samples. However when you add two arbitrary samples, the resulting value could be up to twice the maximum value.

For example, if your samples are 16-bit values, the result will be up to 65536*2. This results in clipping.

The naive solution here is to divide by N, where N is the number of channels being mixed. However, this results in each sample being 1/Nth as loud, which is completely unrealistic. In the real world, when two instruments play simultaneously, each instrument does not become half as loud.

From reading around, a common method of mixing is: result = A + B - AB, where A and B are the two normalized samples being mixed, and AB is a term to ensure louder sounds are increasingly "soft-clipped".

However, this introduces a distortion of the signal. Is this level of distortion acceptable in high-quality audio synthesis?

What other methods are there to solve this problem? I'm interested in efficient lesser-quality algorithms as well as less-efficient high-quality algorithms.

I'm asking my question in the context of digital music synthesis, for the purpose of mixing multiple instrument tracks together. The tracks could be synthesised audio, pre-recorded samples, or real-time microphone input.

$\endgroup$
  • $\begingroup$ I wonder how often it is possible to avoid clipping by just time-shifting the signals a little bit. $\endgroup$ – Sebastian Reichelt Oct 9 '12 at 23:00
  • $\begingroup$ Good idea, though I suspect it's not quite that simple, especially when you don't have much lookahead (eg in real-time). Problem is, you have to know the sample in advance to know what sort of time-shift would be appropriate. That said, in most music, you'd have a high probability of correlation, so a bit of random time-shifting might work very well. Anybody have any experience to draw on here? $\endgroup$ – bryhoyt Oct 14 '12 at 8:34
  • 2
    $\begingroup$ @bryhoyt: Real mixers sum the signals together. That's it. No time delays or non-linear processing required. Clipping isn't a problem because the original signals weren't that loud to being with. $\endgroup$ – endolith Jun 12 '13 at 15:20
  • 2
    $\begingroup$ 16 + 16bit = 17bits ;-) $\endgroup$ – nikwal Nov 23 '16 at 7:59
  • 1
    $\begingroup$ just divide by number of inputs, then clipping won't be possible. and if the sound is too quiet, turn up the amplifier… $\endgroup$ – Sarge Borsch Mar 5 '17 at 15:07

10 Answers 10

14
$\begingroup$

It's very hard to point you to relevant techniques without knowing any context for your problem.

The obvious answer would be to tell you to adjust the gain of each sample so that clipping rarely occurs. It is not that unrealistic to assume that musicians would play softer in an ensemble than when asked to play solo.

The distortion introduced by A + B - AB is just not acceptable. It creates mirror images of A on each side of B's harmonics - equivalent to ring-modulation - which is pretty awful if A and B have a rich spectrum with harmonics which are not at integer ratios. Try it on two square waves at 220 and 400 Hz for example.

A more "natural" clipping function which works on a sample-per-sample basis, is the tanh function - it actually matches the soft-limiting behavior of some analog elements. Beyond that, you can look into classic dynamic compression techniques - if your system can look ahead and see peaks comings in advance this is even better.

$\endgroup$
  • 4
    $\begingroup$ Additions and hard clipping. Just look at any open-source mod player. Using an addition for mixing, with inputs scaled appropriately to minimize clipping, and then a hard-limiter (optionally soft) is the norm, not the exception... $\endgroup$ – pichenettes Oct 7 '12 at 19:12
  • 4
    $\begingroup$ In most situations this is not the developer's responsibility to solve the problem. You give the user/composer the possibility of adjusting the volume of each channel, and it is up to the user to do the mix so that the clipping is acceptable to them. For example, in Renoise, by default, the gain of each instrument/note is 1 and things start clipping badly when adding tracks - it is up to the user to adjust the volume of the notes or instruments in the module to prevent clipping on the master track (unless it's desired). Here's a screenshot showing that: i.imgur.com/KVxDt.png. $\endgroup$ – pichenettes Oct 8 '12 at 9:39
  • 2
    $\begingroup$ IIRC, FastTracker was more conservative, in that it applied an attenuation on each track, and then had a global "makeup gain" in the preference dialog from x1 to x32. I remember that when I had to render all my modules to .WAV for a CD, I had to try values of the gain until I found the lowest one that did not cause clipping... $\endgroup$ – pichenettes Oct 8 '12 at 9:40
  • 2
    $\begingroup$ Regarding the attenuation level, in case you cannot have a user in the loop; 1/32 is an absolutely safe level (no clipping). Assuming that the channels are not correlated (which is not very true for music - more correct when mixing background ambiances), a value of 1/sqrt(32) would be a good compromise between loudness and clipping probability. The best solution would still be to use 1/32 and then post-process your samples with a dynamic compressor. $\endgroup$ – pichenettes Oct 8 '12 at 9:43
  • 3
    $\begingroup$ Additions. That's what hardware mixers do anyway and it's the way people expect things to behave. System-level mixers simply clip. It would be a big problem if the system drivers implemented any kind of non-linear processing - I would imagine the pain of mastering engineers trying to figure out whether what they hear is their compressor plug-in setting or some system-level dynamic processing. Music production software offers a wide palette of dynamics compression plug-ins, it's up to the users to make sure their mix does not clip. $\endgroup$ – pichenettes Oct 14 '12 at 9:04
17
$\begingroup$

The physically "correct" way to do this is summing the samples. However when you add two arbitrary samples, the resulting value could be up to twice the maximum value. ... The naive solution here is to divide by N, where N is the number of channels being mixed.

That's not the "naive" solution, its the only solution. That's what every analog and digital mixer does, because it's what the air does, and it's what your brain does.

Unfortunately, this appears to be a common misconception, as demonstrated by these other incorrect non-linear "mixing" (distortion) algorithms:

The "dividing by N" is called headroom; the extra room for peaks that's allocated above the RMS level of the waveform. The amount of headroom required for a signal is determined by the signal's crest factor. (Misunderstanding of digital signal levels and headroom is probably partially to blame for the Loudness war and Elephunk.)

In analog hardware, the headroom is maybe 20 dB. In a hardware DSP, fixed-point is often used, with a fixed headroom; AD's SigmaDSP, for instance, has 24 dB of headroom. In computer software, the audio processing is usually performed in 32 bit floating point, so the headroom is enormous.

Ideally, you wouldn't need to divide by N at all, you'd just sum the signals together, because your signals wouldn't be generated at 0 dBFS in the first place.

Note that most signals are not correlated to each other, anyway, so it's uncommon for all the channels of a mixer to constructively interfere at the same moment. Yes, mixing 10 identical, in-phase sine waves would increase the peak level by 10 times (20 dB), but mixing 10 non-coherent noise sources will only increase the peak level by 3.2 times (10 dB). For real signals, the value will be between these extremes.

In order to get the mixed signal out of a DAC without clipping, you simply reduce the gain of the mix. If you want to keep the RMS level of the mix high without hard clipping, you will need to apply some type of compression to limit the peaks of the waveform, but this is not part of mixing, it's a separate step. You mix first, with plenty of headroom, and then put it through dynamic range compression later, if desired.

$\endgroup$
  • 1
    $\begingroup$ I understand these concepts, but I'm not sure it's quite correct. Sure, if I add a bunch of 16-bit samples, 32 bits gives me heaps of numerical room. But I still have to play the resulting mix back at a normalised volume on a real-world sound system. I want 2 channels to sound louder than each channel played separately, but I don't want clipping. Doing my sums in 32 or even 64 bits doesn't help with this. Perhaps I'm starting to answer my own question: the original samples should be normalised at a quieter level than the maximum amplitude. As you suggest, leaving some mixing headroom. $\endgroup$ – bryhoyt Oct 14 '12 at 8:53
  • 3
    $\begingroup$ @bryhoyt: Yes, but you also have to remember that the waves are rarely correlated with each other, so adding together 5 sounds doesn't make peaks 5 times as high. $\endgroup$ – endolith Oct 15 '12 at 2:57
  • 1
    $\begingroup$ Thanks, @endolith, I guess that's really at the heart of all this, and explains to me why it's not quite as big a problem as I first thought. $\endgroup$ – bryhoyt Oct 15 '12 at 19:04
  • $\begingroup$ So if 10 non-coherent sources give 10 dB, would dividing by sqrt(number of of sources) be a reasonable heuristic? That is, if you've got 3 sources, sum them and divide by sqrt(3)? (sorry for commenting on an ancient thread) $\endgroup$ – nerdfever.com Jun 11 '17 at 18:42
  • $\begingroup$ @nerdfever.com That's how the RMS levels combine, so... probably? $\endgroup$ – endolith Jun 11 '17 at 21:42
6
$\begingroup$

the formula

$$\text{result} = A + B - AB$$

doesn't make any sense, even if you mean something other than $AB = A*B$. One thing you need to think about is that sound varies above and below zero. A better way to think about it is like this:

$$\text{result} = g( A + B )$$

where $g\le 1$.

The most simple approach is to say $g = 0.5$, which is conservative, linear and always works, but may not be as "loud" as you want. A less conservative approach that "usually works" and is "louder", is $g = 1/\sqrt{2}$. Extensions to more channels with this approach work better.

Alternatively, $g$ can change over time, in which case it's usually the result of a compressor/limiter algorithm. Then you really have a difference equation:

$$\text{result}[i] = g[i]( A[i] + B[i] )$$

$g[i]$ is a then a function of previous $A$, $B$, $g$ and $\text{result}$.

Perhaps this:

$$g[i] = f( A[i] + B[i], g[i-1] )$$

UPDATE: As suggested by hotpaw2, you can delay the input signal but not the gain suppression. This is called a "look-ahead limiter".

$\endgroup$
  • $\begingroup$ By "AB" I do mean "A * B". I do understand that the amplitude can be either positive or negative. You're right, my equation doesn't make much sense for combination involving negative amplitudes. $\endgroup$ – bryhoyt Oct 14 '12 at 8:43
  • $\begingroup$ I have to mix 8 to 10 (N) different sinus waves. Empirically I knew the right value was around 0.3... 1 / √N seems right... any link to why is that formula correct? $\endgroup$ – Zibri Sep 12 at 15:03
5
$\begingroup$

One way this can be done for non-real-time mixing to use a look-ahead AGC, where the gain of one or both channels is lowered at a hard-to-perceive rate before the sum amplitude exceeds the clipping limit. The less look-ahead available, either the the AGC gain adjustment will become more audible, or the max gain for for a softer gain adjust ramp will get closer and closer to 0.5 per channel at the limit. For sound sources with some predictability, one could also use statistics regarding the envelope's behavior over time to adaptively guess a gain limit, but with some probability of failure (which would be an abrupt AGC gain adjustment).

$\endgroup$
  • $\begingroup$ This is a look-ahead limiter, not a look-ahead AGC. $\endgroup$ – Bjorn Roche Oct 8 '12 at 17:34
  • 2
    $\begingroup$ @BjornRoche: Can't a limiter be considered a type of AGC? $\endgroup$ – endolith Apr 12 '13 at 18:04
  • $\begingroup$ Some limiters are AGCs, but a lookahead limiter is not an AGC. $\endgroup$ – Bjorn Roche Apr 12 '13 at 21:31
  • 1
    $\begingroup$ @BjornRoche well it is automatic and controls gain... $\endgroup$ – Olli Niemitalo Sep 13 '16 at 7:49
2
$\begingroup$

I had talked with a mixer designer of the late 1990's and first 2000's that was going on the digital wave (after having tiptoed). I think the guy was a designer for SPL, but maybe not that big, I absolutely don't remember neither the name neither the brand, I just remember how really really big and expensive the machine was.

We spoke long, and finally spoke about the techniques for really guaranteeing that their 64/128 @ 24bits channels mixed together were remaining a 24 bits accurate mixed output channel without clipping.

The technique he explained was rather simple. The 64 tracks (on 24 bits) were added in a 48 bits channel, where the clipping cannot occur. Straight.

I cannot say how that signal was then dithered 48 back from to 24 bits. Maybe that's where the tricky kitchen recipes are applied.

And there maybe are a lot of techniques to achieve that, above all different whether done in real-time or with the all signal already recorded with high-peaks simple to determine... all kind of normalizations to imagine I think.

$\endgroup$
2
$\begingroup$

Lower the global volume. Impulse tracker classically outputs channels at about 33% volume max by default.

That seems to be both loud enough for music with few channels (4 channel Amiga MODs) and soft enough for songs with 50 channels (since channel contents are typically not correlated so volume doesn't add up that fast past a certain level... plus few channels will be outputting at max volume with that much stuff going on). It also leaves enough headroom for hard-left or hard-right panned channels (which use 66% of the range).

Also you don't want to add your channels together in 16bits, you want to add them together in 32bits, then clip the result and reduce to 16bits at the very end. You will need the higher range so that it doesn't wrap around while doing the math. Another option is to use 32bit floating point (which is convenient for doing filters, effects etc).

$\endgroup$
0
$\begingroup$

I think the key is, if you have 16 bit values and your adding 2 values together that could potentially be more then the maximum value, then you have 2 options:

1) cast both to 32 bit add then return the maximum value if the addition exceeds that value. Then cast it back to 16 bits. For example if your values are 32768 and 34567 it exceeds 65535 and the key is to then return 65535. You would do the same thing if using signed values at the minimum value end.

2) compress both values, then add them together.

The first is essentially hard clipping, the second is soft clipping. Analog systems are all hard clipping.

$\endgroup$
0
$\begingroup$

They will only be twice the space if the frequencies in both tracks take up the same space frequency wise. Use eq and compression to get around this by carving out areas of the frequency spectrum for each sound and controlling the transients and sustains of sounds so that everything pokes out where it should. Maybe that doesn't answer the question though. You could delay lower frequency signals by up to 2 ms. It won't cancel through phase because the wavelengths are longer than the higher frequencies and it will add space because the transients aren't completely in time with the power hungry bass signals. Something linearly adding more of a delay the lower the signal would be interesting to test out.

$\endgroup$
-1
$\begingroup$
A + B + {
    (|A| = A) = (|B| = B) = true: -AB;
    (|A| = A) = (|B| = B) = false: AB;
    else: 0
}

That is, if both A and B share a sign, apply a limiting offset. The magnitude of the offset is the product of A and B. The direction of the offset is opposite to that of A and B.

If A and B do not share a sign, no limit is applied, as there is no way to overflow.

$\endgroup$
  • $\begingroup$ Note this is not commutative. If you want to mix more than 2 voices, you have to mix them all at once. In this case, you should "flatten" everything in one direction (if you're too high, flatten the positive values down with the negative ones; if you're too low, flatten the negative values down with the positive ones). Once you've accounted for the offset (applied proportionally across the remaining values); use the binary approach, but scale the limiter based on the number of mixed values. $\endgroup$ – Rich Remer Apr 8 '13 at 22:02
-1
$\begingroup$

My suggestion:

  1. Convert track audio format from 16bit fixed point to 32bit floating point.
  2. Add the current sample value of all tracks to be mixed.
  3. Don't do anything else.

The user may wish to process this mixed stream with compression and/or limiting prior to dithering and reconversion to 16bit fixed point format (assuming this conversion... mixdown to hand off to mastering engineers is usually left at a higher resolution format)

$\endgroup$
  • 2
    $\begingroup$ Hello, and welcome to DSP.se. We thank you for trying to contribute, but I don't feel like this answers the OP's question at all. The OP didn't mention "users" of his system: he might be playing with it on his own, or writing the program to specific requirements. I'm sorry to downvote: I'll be happy to revise my vote if you make your answer more to the point. Also, please take care of your formatting: take a look at the FAQ to see how to write good answers. $\endgroup$ – penelope Jan 21 '14 at 10:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.