FIR filter design for resampling

Question

I am working on an audio resampler that is realized as a polyphase FIR filter. While testing, I found that it does not suppress aliasing as well as I would have expected. For testing, a sine sweep is upsampled by a factor of 8, distorted by $\tanh(kx)$, and downsampled to its original rate. For $k \lessapprox 3.7$, the resulting signal does not contain audible aliasing. At higher values for $k$, however, aliasing becomes increasingly noticeable (e.g. at 15000 Hz for $k = 4$). Audio examples:

• Original sweep: http://sndup.net/cjrg
• Distorted sweep ($k = 4$): https://sndup.net/xxjs

Additionally, when examining the distorted signal, its amplitude increases with time, even for lower $k$. I believe this is also due to aliased signal content:

I'm assuming that my polyphase implementation is correct, so the resampler's antialiasing performance is likely bounded by the filter. The filter that has been used for the examples above was obtained in Octave using fir1(799, 0.12, 'low', kaiser(800, 10)) .* 8 (the multiplication by 8 is done so that the upsampler does not have to perform gain compensation).

Since the resampler is intended to be used in real-time distortion audio effects for DAWs, I am looking for ways to generate a filter that better suppresses aliasing and, ideally, has fewer coefficients (currently 800).

Apart from fir1, I also tried fir2, this sinc filter generator by Nigel Redmon, and the Parks-McClellan method, but all of the filters I was able to generate were worse than the above filter in the following ways:

• incresed passband droop in the high frequencies, or
• worse stopband rejection in the low frequencies, or
• overall worse stopband rejection, leading to even more noticeable aliasing.

Is there a way to design a filter that has both fewer coefficients and better antialiasing performance or is this about as good as FIR resampling can get?

Answer 1

To optimize FIR filter rejection for resampling applications, I recommend using the least squares algorithm (firls in Matlab, Octave and Python scipy.signal) with multi-band targets concentrating the zeros at the image frequency locations. This will provide the maximum rejection where it matters for a given number of coefficients. The plots below depict a simple example of interpolate by 4.

The resulting coefficients are then used directly in the polyphase interpolator implementation as I have previously detailed at this post.