I have a filter designed in matlab with the function cheby2( N, Rs, Ws, 'stop'). The filter would give nice frequency response with for a given parameter set when the filter order is 2 or 4 (N=4). But if I increase the filter order to say 14 the magnitude plot of filter response is not at all smooth in fact in the stop band it gives very high gain instead of attenuation. Any suggesetion what is going on. What I should look at? Do you need any more information to access the problem. The parameters I am using are: Rs = 40; Wc = 0.2; Wb = 1/128 Ws = Wc +0.5*Wb*[ -1, 1];
You have a very narrow stop band which means that all the poles are crammed in a very small area of the complex plane, close to the unit circle. This can result in severe numerical problems, even for relatively small filter orders, even with floating point arithmetic.
Another important point that you might not realize is that if you design a band pass or a band stop filter using the command
cheby2 (and all other similar commands), then
N is the order of the prototype low pass filter, and not the order of the resulting band pass or band stop filter. So if you choose
N=14 then you've designed a band stop filter of order $28$. Such a filter is very hard, if not impossible to realize because of numerical problems.
Also note that calling
[z,p,g] (zeros, poles, and gain) as output argument usually results in higher accuracy than using the polynomial coefficients.
I would actually suggest trying a notch filter instead of such a narrow band band stop filter.