If you have a signal x[n] and it is interpolated by N and then decimated by M, how do you ensure minimum distortion in x[n] from the resampling process?
The answer to this is to design a filter that perfectly passes the original spectrum and completely rejects the images that the decimation and interpolation process creates. Such a filter is not achievable, so the outcome becomes a design constraint on how much distortion we can tolerate (as in all filter design). Helpful for this is to understand the mechanism that creates the images in this case so I am including some graphics below that I have that further explain this process.
Interpolation by inserting zeros
We upsample with minimum distortion by inserting zeros in between each sample in the time domain, and then follow this with an interpolation filter. The interpolation filter passes the original spectrum and rejects the images the zero-insert creates.
The graphic below shows the image replication of the spectrum by upsampling by 4 (insert 3 zeros between each sample). Note how we started off with just our desired spectrum in the "first Nyquist zone" which is the freqency from DC to half the sampling rate ($F_s/2$) for a real signal. (The same applies to the spectrum for a complex signal, just in that case the spectrum is unique from $F_s/2$ to $+F_s/2$ so we would include that or equivalently the signal from DC to $F_s$.) After the zero insert we end up with 4 additional spectral images in the new first Nyquist zone at the higher sampling rate, for the case of a 4x sampling rate increase. (So in your case you would have N such images).
The ideal interpolation filter would "grow" the zeros to the actual value needed for perfect interpolation. Such a filter could be a multiband filter (Matlab firls and firpm do these easily) as depicted in the graphic below, but a low pass filter would also be fine to use. The actual distortion depends on how much of the image frequencies we allow through and how much pass band distortion (either loss or ripple) we allow, this all becomes a design constraint on the complexity of the filter, tolerable delay etc...
Decimation by dropping samples
For decimation we reduce the sampling rate by simply removing samples (So if we decimate by M, we only use every Mth sample). The spectral challenge of images in the decimation process is very similar. While in interpolation we were concerned about clean-up in that we filter after we do the zero-insert; for decimation we are concerned with filtering prior to the dropping of samples due to aliasing; signal energy that will fold into our spectrum of interest if we don't filter prior to removing samples. The graphic below shows this for an example of decimating by 4. This is identical to the aliasing issues in A/D conversion (in fact A/D conversion is effectively decimating from an infinite sampling rate! So here we are doing a digital to digital conversion):
Like the interpolation filter, the ideal decimation filter would pass our spectrum of interest with no distortion while completely eliminating any signal that may exist in the higher spectral locations that can fold in. Note if you know by design that these spectral locations have already been filtered (including amplified noise floor), then filtering may not be necessary.
Like the interpolation filter design, a multiband filter approach would provide more rejection for the same number of taps (more efficient):
And finally a rational rate conversion (both decimate and interpolate) suggests that you can do the interpolation first so that the filtering can be shared between the two operations. This is a consideration dependent on the actual filtering design requirements for both operations, but feasible that one low pass filter design could cover both requirements. In some cases with high numbers for M and N it may also makes more sense to break up the interpolation and decimation operations further to simplify the filtering requirements.
In all cases a significant limit to what can be achieved is how much of the digital spectrum available we are using. The more excess spectrum available, the simpler the filter design becomes since the design is driven by how tight the transition bands need to be.