Designing appropriate filter by choosing values for $A$ and $B$

Question

Let $$y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2]$$ Where $A$ and $B$ are constants. Also there are two sounds which have $400$ hertz noise and the sampling rate is $8000$ samples/sec. The task is determining $A$ and $B$ in order to eliminate the noise. I've written a MATLAB program which takes $A$ and $B$ as inputs and then plots the frequency response of the filter as output(using freqz command). Letting $A = 0.9$ and changing $B$ from $1.4$ to $1.9$ leads to shifting the band stop region to the left: Since we want to remove $400$ hertz noise choosing $B = 1.9$ is good because $0.1 \times \frac{8000}{2} = 400$. The main problem is finding appropriate $A$. Let $B = 1.4$ and increase $A$ from $1.1$. Part of magnitude plot will move downward: Conversely, if $B = 1.9$ increasing $A$ from $0.1$ to $0.9$ causes plot moves upward. If $A = 1$ then magnitude plot is a straight line It seems there is some sort of usual trade-off situation here. More elimination of noise is achieved if we select larger values for $A$ but this causes more attenuation in other frequencies as well. Is there any mathematical solution for finding the best values for $A$ and $B$? Also how we can design better filters for removing $400$ hertz noise? I mean alternative solutions instead of mentioned IIR filter.

Answer 1

Bottom Line:

$$A <1$$

$$B =2\cos(\omega_n)$$

Where $\omega_n$ is the normalized angular frequency of the desired notch location (in this case for the OP with a sampling rate of 8KHz and notch at 400 Hz this would be $\omega_n = 2\pi400/8000 = \pi/10$), resulting in $B \approx 1.902$ and $A$ is the frequency notch bandwidth parameter; the closer $A$ is to 1 the tighter the bandwidth of the notch.

Details:

The straight-forward approach to eliminating narrow band noise at 400 Hz would be a classic 2nd order notch filter. This is detailed at this post:

Transfer function of second order notch filter

This approach is done by simply placing a zero at the frequency of interest, and by adding a pole close to the zero, but inside the unit circle of course to be stable, we can adjust the bandwidth of the notch by the proximity of this pole to that zero, meaning $|p|<1$, but close to 1.

Using real coefficients, meaning complex conjugate zeros and poles as detailed in the linked post, this results in the transfer function:

$$ H(z) = \frac{1+a}{2}\frac{z^2-2z\cos(\omega_n)+1}{(z^2-2az\cos(\omega_n)+a^2)} $$

Which as a difference equation is exactly in the form that the OP has used and therefore given this is the same filter, answers the OP's question. This is is derived from the z-transform as follows (we can ignore the gain scaling parameter $(1+a)/2$ which for most cases is close to 1, and express in decreasing powers of z (and assuming I didn't make simple algebra errors!):

$$ \frac{Y(z)}{X(z)} = \frac{1-2z^{-1}\cos(\omega_n)+z^{-2}}{(1-2az^{-1}\cos(\omega_n)+a^2z^{-2})} $$

$$ Y(z) (1-2az^{-1}\cos(\omega_n)+a^2z^{-2})= X(z)(1-2z^{-1}\cos(\omega_n)+z^{-2})$$

$$Y(z)- Y(z)2a\cos(\omega_n)z^{-1}+Y(z)a^2z^{-2} = X(z)-X(z)2\cos(\omega_n) z^{-1} +X(z)z^{-2} $$

And then with the inverse Z transform we get the desired coefficients that the OP wanted:

$$y[n] - 2a\cos(\omega_n)y[n-1] + a^2y[n-2] = x[n]-2\cos(\omega_n)x[n-1] + x[n-2] $$

With comparison to OP's expression:

$$y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2]$$

Results in the following:

$$A = a$$

$$B =2\cos(\omega_n)$$

Where $\omega_n$ is the normalized angular frequency of the desired notch (for the OP's sampling rate and notch frequency this would be $\omega_n = 2\pi400/8000 = \pi/10$) and $a$ is the notch bandwidth parameter ($a<1$, the closer a is to 1 the tighter the notch, refer to the linked post for complete details).

Answer 2

This is meant as a stepping stone up to Dan's answer.

The units for frequency at the sample level are radians per sample. You've got:

$$ 400 \frac{cycles}{second} \cdot 2 \pi \frac{radians}{cycle} / 8000 \frac{samples}{second} = \frac{\pi}{10} \frac{radians}{sample} $$

That is your target $\omega_t$.

Sound is a real valued signal, so you can model it like this:

$$ s[n] = A \cos( \omega n + \phi ) = A \frac{ e^{i(\omega n + \phi)}+e^{-i(\omega n + \phi)} }{2} $$

Those are your two complex tones you want to zero out. Notice the the units work out as well.

$$ \omega \frac{radians}{sample} \cdot n\; samples = \omega n \; radians $$

Starting with:

$$ y[n] - ABy[n-1] +A^2y[n-2] = x[n] - Bx[n-1] +x[n-2] $$

Suppose that: $$ \begin{aligned} y[n] &= A_y e^{i(\omega n + \phi_y) } \\ x[n] &= A_x e^{i(\omega n + \phi_x) } \\ \end{aligned} $$

It follows that: $$ \begin{aligned} y[n+d] &= A_y e^{i(\omega (n + d) + \phi_y) } = A_y e^{i(\omega n + \phi_y }e^{i(\omega d) } = y[n]e^{i\omega d } \\ x[n+d] &= A_x e^{i(\omega (n + d) + \phi_x) } = A_x e^{i(\omega n + \phi_x }e^{i(\omega d) } = x[n]e^{i\omega d } \\ \end{aligned} $$

Plug those into your equation:

$$ \begin{aligned} y[n] - ABy[n-1] +A^2y[n-2] &= x[n] - Bx[n-1] +x[n-2] \\ y[n] \left[ 1 - AB e^{i\omega } + A^2 e^{i2\omega } \right] &= x[n] \left[ 1 - B e^{i\omega} + e^{i2\omega} \right] \\ \end{aligned} $$

For convenience, make a substitution:

$$ z = e^{i\omega} $$

$$ \begin{aligned} ( 1 - AB z + A^2 z^2 ) \cdot y[n] &= ( 1 - B z + z^2 ) \cdot x[n] \end{aligned} $$

Your $z_1$ and $z_2$ represent the two tones you want to zero. By putting the polynomial for $x[n]$ in factor form, your target values can be plugged right in.

$$ \begin{aligned} 0 = ( 1 - B z + z^2 ) = ( z - z_1 )(z - z_2 ) = z^2 - (z_1 + z_2)z + z_1 z_2 \end{aligned} $$

$$ \begin{aligned} z_1 &= e^{i\omega_t} \\ z_2 &= e^{-i\omega_t} \end{aligned} $$

Observe:

$$ z_1 z_2 = e^{i\omega_t} e^{-i\omega_t} = e^0 = 1 $$

Which fits in the equation without rescaling, therefore

$$ B = z_1 + z_2 = e^{i\omega_t} + e^{-i\omega_t} = 2 \cos(\omega_t) $$

Now, continue with Dan's answer.