Filter Design near Nyquist frequency (complex curve fitting)

I'm fitting IIR filters to a complex transfer function with Matlab's invfreqz.m. My fitting results are very good, but when I come very close to the Nyquist frequency the phase response runs to 0 and I also get deviations in the magnitude response. I did some research but couldn't find anything regarding that phenomenon. Can anybody point me in the right direction, for which keywords I have to search in order to find something on that topic and how I can try to optimize the fitting near the Nyquist frequency?

• The phase of a discrete-time system always equals $0$ or $\pi$ at Nyquist; there's nothing you can do about it. Furthermore, the magnitude response must be symmetrical around Nyquist, i.e. $|H(\pi-\omega)|=|H(\pi+\omega)|$. So you'll always get larger approximation errors close to Nyquist than at other frequencies. In practice you have to choose the sampling frequency such that your frequency range of interest stops slightly below Nyquist. Have a look at this answer to see an approximation of an analog filter by a digital filter. Commented Apr 12, 2016 at 14:35
• Could you provide a plot of the desired response and its discrete-time approximation? Commented Apr 12, 2016 at 14:35
• thank you for your answer, i just couldnt come up with the reason for the phase response being equal to 0 at Nyquist, but when i read your comment it came to my mind instantly. Are there certain ways to keep the magnitude response as stable as possible as near to Nyquist as possible or respectively approximate as close as possible near to Nyquist? Commented Apr 12, 2016 at 14:56
• Unfortunatelly i have to use a curve fitting algorithm to derive my filter Commented Apr 12, 2016 at 15:05
• If the discrete-time filter doesn't have a zero at Nyquist, the derivative of its magnitude must be zero (because of the symmetry with respect to Nyquist); this is why the magnitude approximation at Nyquist can be worse than below Nyquist. Let me know your desired response and the chosen filter order and I can try to design a filter for you to have a comparison. BTW, have you checked if the filter returned by invfreqz is stable? Commented Apr 12, 2016 at 16:05