When we say that a Generalized Linear Phase System must satisfy the pole zero plot with the condition that a complex zero not on the unit circle exist's in a pair of 4. Then I understand that I need the conjugate pairs to have real coefficients. Why do we need the complex conjugate reciprocal pairs and what role do they play for the FIR filter? I am aware that we use complex conjugate reciprocal pairs in case of all pass systems but how does that relate to the Linear Phase System if it even does.
The impulse response of a linear phase FIR filter is odd or even symmetrical. This has as a consequence that if $z_0$ is a zero, $1/z_0^*$ must also be a zero. If in addition to the linear phase property you also have real-valued coefficients, complex zeros always occur in complex conjugate pairs. So if a real-valued linear phase filter has a complex root $z_0$, you automatically get $z_0^*$, $1/z_0$, and $1/z_0^*$ as roots.
Complex roots on the unit circle just occur in complex conjugate pairs (because $z_0=1/z_0^*$, and real-valued roots just occur in pairs $z_0$ and $1/z_0$, unless they're on the circle (i.e., $z_0=1$ or $z_0=-1$).
In sum, for real-valued linear phase FIR filters, there are four types of "clusterings" of roots:
- complex roots with $|z_0|\neq 1$ occur in groups of four: $z_0$, $z_0^*$, $1/z_0$, $1/z_0^*$
- complex roots on the unit circle occur as complex conjugate pairs $z_0$ and $z_0^*$
- real roots with $|z_0|\neq 1$ occur in pairs as $z_0$ and $1/z_0$
- real roots on the unit circle ($z_0=1$ or $z_0=-1$ ) don't imply another root
Mathematically, for a real-valued length $N$ FIR filter the linear phase condition in the frequency domain is
where $N$ is the filter length (number of taps). From $(1)$ it is clear that if $z_0$ is a zero of $H(z)$ then also $1/z_0$ must be a zero.