According to literature, the arctan-demodulation method commonly used in Radio-frequency radar systems is the one shown below:
Is there a rule of thumb regarding the cut-off frequency of these Low-pass filters?
Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this communityThe primary purpose of those filters is to pass the desired signal bandwith and reject the carrier feed-through at $\omega_h$ and the double frequency component at $2\omega$. Given that, I would recommend designing the filters with a passband and rejection requirement at those frequencies in mind with a goal of minimizing filter complexity (which means maximize the transition band between passband and rejection to the extent possible). Further filtering can be done on the $\phi(t)$ result itself based on the video bandwidth desired.
If a digital receiver, it would be best practice to use decimating low pass filters prior to the arctan computation to minimize the processing requirements there. This would also be based on the video bandwidth desired.
Further if an actual atan computation is needed, I would recommend the atan2(Q, I) function.