I'm fitting IIR filters to a complex transfer function with Matlab's invfreqz.m. My fitting results are very good, but when I come very close to the Nyquist frequency the phase response runs to 0 and I also get deviations in the magnitude response. I did some research but couldn't find anything regarding that phenomenon. Can anybody point me in the right direction, for which keywords I have to search in order to find something on that topic and how I can try to optimize the fitting near the Nyquist frequency?
You can't get a good fit very near the Nyquist frequency (or near DC) because information is lost due to the imaginary components around that frequency being of (almost) opposite phase and thus (nearly) cancelling out. Thus, there will be ambiguity errors due to trying to fit any curve to lost information.
If possible, you can try to gather more information near (but not at) the Nyquist frequency by sampling for a longer time span or at a higher sample rate. This will move the measurement point slightly farther from where there is complete cancellation of the imaginary component.