I've recently encountered a rather clever low-pass filter that appears to have been designed to sacrifice uniform group delay in the pass-band to get better frequency response relative to the filter length. What methods are available to design FIR filters that maximize frequency performance relative to length while allowing phase to become nonlinear (but still continuous) in the pass-band?
Although not a comprehensive list, the following are filter algorithms available in Matlab for designing non-linear phase FIR filters:
firlpnorm: Nonlinear phase equiripple or least squares, uses Lp norm: for equiripple filters p = infinity, for least squares p =2.
cfirmpm: complex and nonlinear phase equiripple filters.
See the Matlab documentation for full details on how to design non-linear phase using these algorithms.
One way to make a non-linear out of a FIR means solving for the zeroes, keeping only the inside/outside ones, then rebuilding the impulse response, and all this with equiripple filters due to the method of adding a DC to the middle coefficient (this for both same-N, and half-N filters). But this can be extended to non-equiripple filters, as well, see this, for example. My recommendation remains: IIRs are a better investment in terms of computing power, for the same, or better results, particularly since you, yourself, say that you will compensate the phase distortion afterwards. Of course, you have the last saying.
if you need me to explain it, lemme know in a comment and i will come back and try to be explicit about it.