Output of IIR filter not matching the frequency response - Proximity to fs/4

Question

After not having dealt with digital filters for a long time, I have now been playing around with filter design in octave and I am observing a behavior which I do not understand.

After designing a low-pass Chebychev filter with cheby1(1,3,0.4) and plot its frequency response, it can be seen that the -3dB point is exactly at the specified cutoff frequency.

However, when I test the filter with an input wave x=cos(2*pi*cutoff*t), using the filter function y=filter(b,a,x), the amplitude of the output signal can get significantly lower than the expected $0.707$, $0.63$ for this example in particular.

It seems that the output of the filter better approximates the plotted frequency response for cutoff frequencies either close to Nyquist frequency, or close to $0$, but as it gets closer to the middle point between these two, the output steps away from the expected attenuation.

What is the reason behind this?

Answer 1

From the octave documentation on cheby1.m you can see that the cut-off frequency in radians is given by pi*Wc, where Wc is the cut-off frequency input argument of cheby1.m. So for a function call [b,a] = cheby1(n,Rp,Wc), you should test the filter's gain at the cut-off frequency with an input sequence cos(Wc*pi*n). Furthermore, choose a sufficiently long input signal so that you can observe the filter's steady-state behavior.

Apart from that it's a matter of sampling the time domain function. Note that the samples will generally not occur at the maxima of the sinusoid. I.e., the maximum magnitude of the output signal

$$y[n]=|H(\omega_c)|\cos(n\omega_c+\phi(\omega_c))$$

will not necessarily attain its theoretical maximum value $|H(\omega_c)|$.