# How to smooth filter response?

I have a filter designed in matlab with the function cheby2( N, Rs, Ws, 'stop'). The filter would give nice frequency response with for a given parameter set when the filter order is 2 or 4 (N=4). But if I increase the filter order to say 14 the magnitude plot of filter response is not at all smooth in fact in the stop band it gives very high gain instead of attenuation. Any suggesetion what is going on. What I should look at? Do you need any more information to access the problem. The parameters I am using are: Rs = 40; Wc = 0.2; Wb = 1/128 Ws = Wc +0.5*Wb*[ -1, 1];

• I recommend you to use the matlab function fvtool. Besides the magnitude and phase it also shows you the Pole-Zero plot. Here you will see that your design results in a filter with poles outside the unite circle, which leads to the instability of the filter – Irreducible Oct 11 '19 at 6:23
• I suspect it's about numerical precision (and a bit of unorthodox design). Inverse Chebyshev crams the poles near the Y axis (zeroes, too), if the imposed constraints are too tight. For example, here you determine N based on attenuations and frequencies, then increase N manually -- this results in the filter adjusting either the attenuations, or the frequencies (most likely). The resulting filter should have very tight transition widths, which can result in tight placement of the poles/zeroes. – a concerned citizen Oct 11 '19 at 7:58

Another important point that you might not realize is that if you design a band pass or a band stop filter using the command cheby2 (and all other similar commands), then N is the order of the prototype low pass filter, and not the order of the resulting band pass or band stop filter. So if you choose N=14 then you've designed a band stop filter of order $$28$$. Such a filter is very hard, if not impossible to realize because of numerical problems.
Also note that calling cheby2 with [z,p,g] (zeros, poles, and gain) as output argument usually results in higher accuracy than using the polynomial coefficients.