Realtime sample rate conversion from variable source

Question

Having all but given up on getting USB audio to work reliably, I'm looking now at taking the stereo 16-bit I2S stream from the Raspberry Pi 4's 40-pin GPIO header.
Unfortunately, that signal stops when not playing anything (some people have made a daemon that constantly plays silence, but I'm not sure that I want to: it seems like an excellent opportunity to poorly document how to set up another one), and varies to match the sample rate of the material being played at the moment. And it's jittery anyway, because the standard audio rates don't have a nice relationship to that system clock.
So I probably don't want to slave my DSP chain to that.

If my input is that unreliable, then I'd much rather use a fixed independent rate for my DSP chain - 48kHz only - and resample the input as needed to match that. There's this Q&A about a fixed ratio between the two rates, but I don't think I can guarantee that.

I've thought of two ways so far, to resample a variable, unknown at design time, input rate to a fixed internal one:

1. Run a general-purpose lowpass at X times my fixed rate (with the coefficients of course calculated for that rate, which is different from the rest of the DSP chain), and just have its high-rate output available for the rest of the DSP chain to grab as needed. (implicit decimation?) The input is the new sample if we just got one, zero otherwise, and it has no idea about the I2S stream itself. It's always running, and either sees a new sample this time or it doesn't.
   This (as it appears to me) makes a dynamic resampler that takes a wide range of possible input rates, even irregular ones, without reconfiguring. It even handles a stopped input, becoming actively silent until the input restarts.

   • Problems?
   • Opportunities to simplify or speed up? Most of its input samples are zero (but don't know which ones), most of its output samples aren't used (do know which of those), and it's running at a much higher rate than anything else...or at least, something is running faster than everything else, just to know when the new sample arrived. The actual lowpass code could be called at the normal rate and only produce one output sample, but use an X-sample buffer as its input instead of just one. (its coefficients still need to be calculated for the X-higher rate though)
     _{So far, I've used a bunch of 1st-order IIR lowpasses in various SAR ADCs' interrupt handlers (algebraic rearrangement of an exponential average, using integer bit shifts instead of multiplication/division), and I think I know enough about a biquad to choose a seemingly attractive form and write it (and copypasta the coefficient calculators), but I haven't actually used one of those in a project yet. Likewise for state-variable.}
     .
     ^{I've looked at IIR more because of its similarity to the analog world. I understand the theory behind FIR, but the analog comparisons just don't work as well. I've read (and completely agree) that a FIR lowpass can be heavily optimized for resampling, because it doesn't need to calculate every output value for the sole purpose of feeding it back into the algorithm like IIR does; it only needs to calculate the output values that are actually used downstream, and no more. But, this logic skips a critical step: yes, IIR must calculate every output sample, even if most of them are only used internally, but there's so little going on in the first place, compared to a similar-response FIR, that it may still come out ahead, even after the FIR is optimized. I've never seen that answered, always jumped over to say that "FIR is better for resampling," which I'm not convinced yet is always true. But I certainly could be convinced!}
     This is very much a learning experience!

2. Run it through a D/A/D conversion. Let an off-the-shelf DAC figure it out, and then pick that up with an ADC right next to it on the same custom PCB.
   I'd rather do it all digitally if I can, since I have another project in the idea phase that needs to receive 8 channels or so from a PC of some kind. A single stereo pair might be manageable with this method, but 4 of them seems like a bit much! I'd like to copy/paste the code from this project into that one if I can.

That other project will definitely need something better, but for this one, a stereo-to-2.1 converter with system processing and amps included, it seemed to me like it might fit on a $4 Raspberry Pi Pico. (datasheet) So that's what I'm running on so far.
Dual M0+, so it doesn't have a hardware FPU, but it does have a hand-optimized floating-point library in ROM. 133MHz.

---

Obvious frame-challenge:
Why can't I just use an off-the-shelf chip and library like Analog Devices' ADAU series and SigmaStudio?

I certainly could for the immediate project - stereo to 2.1 - but I also want to learn how to write my own DSP code, because I think I have to for the other one. The other one has a budget of maybe 20 samples total at 96kHz, analog to analog. (separate from the PC connection) That is, measured by an external stereo ADC with one channel probing the analog input and the other at the analog output, so it includes the converters' group delay as well. Meanwhile, all the libraries I've seen so far have used a bigger buffer than that, not counting the converters.

Answer 1

Since you mentioned USB audio, I assume that your input stream is sampled at a uniform rate, but that this rate not phase locked to your local audio clock and the data comes in "bursts" or "packages" with somewhat unpredictable timing. If that assumption is wrong, than my answer doesn't help.

In this case you need an asynchronous sample rate converters (ASRC). These are found in almost all USB clients and other digitally connected systems that have more than one clock.

ASRC consists of two parts.

1. A standard polyphase sample rate converter (SRC) perhaps with added phase interpolation
2. A control loop that dynamically adjusts the conversion ratio

The design of the originating lowpass filter for the polyphase filter is the same as it is for a synchronous SRC, i.e. defined by the requirements of your application for signal to noise ratio, passband flatness, modulation, etc. If your input sample rate spans a large nominal range (1k, 32k, 48k, 96k) you may need a different filter for each "class" of nominal input sample rates.

The control loop determines the exact conversion ratio. This is done by monitoring the level of either an input buffer or an output buffer over time and then decreasing or increasing the rate to keep the buffer level at the target level (typically the middle).

The speed of the control loop is often dynamic: when acquiring a new stream you want to adjust quickly to prevent over/underflows, but once you have a decent estimate, you can slow down the adaption to increase conversion quality.

The latency (and buffer size) are determined by the properties of your input stream and your operating system (if you are running one). The more "choppy" the input is and the more interrupt latency your OS has, the larger the buffer needs to be.

USB for example is "isynchronous" with a 1ms packet rate. If your input stream has a nominal rate of 44.1kHz, you know you will get one packet with either 44 or 45 samples once every millisecond. So a buffer of maybe 100 samples or so, should do the trick (provided your OS can serve that thread reliably once every 1ms).

Answer 2

This is not a complete answer, but as close as I can come up with at the moment for the understanding that I'm looking for.

---

Thinking mostly about the Problems? bullet point, eventually got me thinking about high sample rates compared to analog. And that led me to the Calc 1 way to introduce an integral as a progressively finer sum of smaller discrete blocks that all add up to the desired total. (I'm very much a graphical thinker) When you get to an infinite number of infinitely small blocks, that's an integral. Likewise, analog can be thought of (loosely) as an infinitely high discrete sample rate and bit depth.

Or if you start thinking about the quantum world, even analog signals become "digital" in the sense of having finite resolution and sample rate. You still have insanely high numbers for those parameters, but they are finite.

It's really not very useful to design an analog system from a quantum perspective, because other noise sources completely drown out the quantum stuff. But I think it can be used as a "bridge" of sorts between the clearly-discrete and practically-continuous worlds.

Thus, the idea of having an input lowpass with a really fast free-running sample rate, regardless of what input it has, starts to make a lot more sense. It really does become the equivalent of an analog anti-aliasing filter.

---

Add to that, the Nyquist sampling theorem (pedantically more than 2x the highest frequency is sufficient to recover completely), and the idea that the signal is zero everywhere between samples and NOT a step that lasts until the next sample, and the result is the real possibility of this pseudocode working:

//single samples, not arrays or buffers
sample intermediate;
sample conditioned_input;

//called whenever a new sample is ready, which could be any time or rate, even non-uniform
//if the input stream stops, then this simply doesn't run, while everything else below it still does
new_sample(sample s)
{
    intermediate = s;
}

//called at the high free-running rate (480kHz or higher?  4.8MHz?)
input_filter()
{
    sample raw_input = intermediate;
    intermediate = 0;
    
    //process raw_input to produce conditioned_input
}

//called at the normal DSP rate (48kHz, for example)
dsp()
{
    sample x = filter1(conditioned_input);
    //continue processing x
}

---

To "de-jitter" an input, simply requires delaying each sample by the appropriate amount to make them all line up again on a steady average. The initial, naive understanding requires advancement as well as delay, so that the average delay is zero. But the only difference between that, and filling a buffer at non-uniform intervals while reading it uniformly, is an overall delay.

If a "feedback control system" is added, as I now understand @robertbristow-johnson to say in a comment to the original question, the purpose of which is to adjust the uniform reading rate to keep the buffer at a particular size, then the intended sample rate can be determined, and used to choose a fixed-ratio resampler and then adjust the local master clock from there.

Or, that buffer could be placed just ahead of the intermediate sample in the above pseudocode, so that the input_filter() receives a uniform stream of samples at a potentially wacky rate compared to the unadjusted local clock, but that's okay because the input_filter() is running so much faster anyway.

---

At any rate, seeing that the conditioned_input is only ever used at the DSP's rate (48kHz or so), the input_filter() function doesn't actually have to run at the higher rate. It only has to produce a value for each output clock as if it were running that fast.

So for that purpose, the intermediate sample could be a buffer that is filled at the higher rate, using the logic shown above, just to keep the timing right and not introduce its own jitter. But that's ALL that happens that fast, while the actual filter is at the top of the dsp() function and loops until the buffer is empty to "catch up". Or it could be a FIR filter that just churns out the correct value in a single iteration.

(fill a buffer at 4.8MHz or so, using zero if a new sample isn't available at the moment; then at 48kHz, convolve the buffer once, with the desired impulse response to produce the "official" input sample to the rest of the DSP chain, and remove the oldest 100 samples from the buffer)