2
$\begingroup$

TLDR: How do I mix network streams when streams might be unreliable?

I am working on a project which will allow musicians to practice/perform together over the internet with low latency. (Think JackTrip but more user friendly).

I am receiving raw PCM audio data via UDP over the network. As these come in, I identify who the sender is and pipe that stream into that particular person's buffer. I have a separate buffer for each person participating.

Periodically I then take one "chunk" at a time from each buffer, mix them together and output to the speakers of the device.

The issue I'm having is knowing when to take that "chunk". Some clients go offline and their buffers run empty. I cannot just wait for their buffer to refill as this will slow down the other clients.

Thoughts I had were to look at all the buffers at a particular point in time and only process a chunk when at least half of the buffers are populated.

Any ideas?

Other details:

  • My buffers are 128 samples big.
  • I am using a sample rate of 44100
  • I am using 16 bit signed integer pcm
  • Streams are MONO
  • I am using PortAudio and C++.
  • I am using ASIO audio drivers.
$\endgroup$
2
$\begingroup$

Generally you will need some sort time sync protocol.

Typically you want create locally one circular buffer for each network channel. As soon as a packet comes in, you shove it into circular buffer ort that channel at the write positions.

Playback is clocked by your local D/A clock. As soon as the D/A buffer is running empty, you grab a frame from all the client circular buffers at the current read position, mix them together and put them into the D/A buffer.

There are two things you need to manage.

  1. Drop out: That happens if you need a new playback frame and the circular buffer of a client is empty OR a packet got dropped (they should be numbered to detect this). There isn't a whole lot you can do about. You can play zeros or try to extrapolate from the previous buffer A drop out means that your network can't keep up.
  2. Clock drift. Even if everyone is playing at nominally 44.1kHz, no two clocks are identical and over time they will drift apart. To fix these you need to a) measure the clock drift and b) correct it. a) can be done through either a global time reference (something like NTP), by averaging over packet arrival times, or monitoring buffer levels. b) will require some sort of sample rate conversion: the most crude option would be to occasionally drop a frame insert a zero frame.

This is age old trade off: the lower the latency, the higher your drop out rate will be. So dialing this in "optimally" is the key here. It's easy enough to adjust: simply make your circular buffers smaller/bigger. However finding the optimum is hard since it depends very much on current network conditions so in practice this really should be adaptive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.