All the references I can find only describe how to design an idealized lowpass filter for doing resampling. For example, MATLAB's documentation describes how to choose a "cutoff frequency" (for rational-ratio resampling):
Something like this:
We can't implement an ideal filter, so a slightly better approximation would be to introduce a transition band:
So, how should I set the passband edge $f_{pass}$ and stopband edge $f_{stop}$?
The only idea I have is to consider halfband filters. In halfband filters, there is symmetry about $f_c$. So, for example, in a halfband decimator, we could set $f_{pass} = 0.9f_c$ and we would get $f_{stop} = 1.1f_c$. This works rather nicely because anything input between $f_c$ and $f_{stop}$ will only alias back into the transition band (so the passband $0$ - $f_{pass}$ is clean).
Is that a reasonable approach in general? We just place $f_{pass}$ and $f_{stop}$ symmetrically about the theoretical cutoff frequency $f_c$?