As detailed in this post Can I set a constraint on the first tap of an FIR filter such that its inverse is stable? I show how Cauchy's Argument Principle can be used to easily confirm if an FIR filter is minimum phase, meaning all zeros are inside or on the unit circle, or the converse, maximum phase, simply by plotting the frequency response on a complex plane.
However, the approach I use is completely graphical, in that I have to plot the frequency response and then manually count encirclements of the origin. Each encirclement of the origin indicates a zero that is outside the unit circle.
My question what would be a compact and efficient algorithmic approach (not graphical) based on the Cauchy Argument Principal that would be more efficient than solving for the roots and determining that $|z|>1$ for all roots, or any other approach to count the number of zeros outside the unit circle?
The graphical approach is as follows, using the example from the linked post above with a maximum-phase FIR filter with coefficients [1 -3 -3 2 5]. The frequency response (magnitude and phase versus frequency) is plotted below on a complex plane. The graphical way to count encirclements is to draw a vector from the origin out toward infinity at any angle and count how many crossings of the frequency response take place. Below you would count 4 encirlements indicating all four zeros are outside the unit circle. (A minimum phase filter would have no encirclements). The easy way to count encirclements is to note a direction on the frequency response with a forward direction consistent with increasing ω, and then draw a vector from the origin out toward infinity at any angle and count how many crossings of the frequency response take place: if the cross is of a forward direction the count increases, and if of a negative direction the count decreases. Note given the crossing locations it is not as simple as just counting crossings on the Real or Imaginary axis (here the Real can't work but the Imaginary could, but that is not a general solution).