Using a spectral transformation a filter $G(z^{cap})$ is
designed to be a low pass filter from an existing lowpass filter $G_{lp}(z)$ via $G(z^{cap}) = G(z)z=H(z^{cap})$
. Which property should H(z) have in order to preserve
stability of the designed filter?
The filter is stable if the all the poles of the transformed filter, i.e., filter obtained after applying spectral transformation, lie inside the unit circle. For example, if $H(z)$ is the original filter and $F(z)$ is the filter response of the transformation filter, then the spectrally transformed filter response is given as \begin{align} G(z) = H(F(z)) = H(z)|_{z^{-1}\leftarrow 1/F(z)} \end{align} where \begin{align} \frac{1}{F(z)} = \frac{1-\beta z}{z - \beta}. \end{align}
Check if the poles of $G(z)$ lie inside the unit circle. If yes, then $G(z)$ is stable. Note that $F(z)$ is an all-pass filter. Therefore, the regions of stability and instability of $H(z)$ are preserved in $G(z)$.