Derivation of a non-ideal low-pass rectangular windowed FIR filter

Question

I have been trying to understand certain aspects of FIR filter design which have frankly annoyed me for some time such as exactly why the critical frequency $\omega_c$ in a low-pass FIR filter is mapped to the $-6\;\text{dB}$ point for a non-ideal low-pass FIR filter and also why the critical frequency is mapped to point halfway between the transition band $\omega_t$. Below I will show my logic up until now.

If we take the ideal low-pass filter in the frequency domain, $H_{ideal}(\omega)$, with the critical/crossing frequency $\omega_c$, and take the inverse discrete-time Fourier transform, we get some response $h_{ideal}(n)$.
If we window this function with a rectangular window for simplicity but it could be any other window $w(n)$, we get the following time-domain result: $$h(n) = h_{ideal}(n) \times w(n)$$ As we know, multiplication in the time-domain is convolution in the frequency domain and vice-versa, thus $$H(\omega) = H_{ideal}(\omega) * W(\omega)$$ This analytic solution gives all of the properties of the filter, such as for a window of length N, giving N coefficients, how can we derive the attenuation, the location of the shifted critical frequency and the length of the transition band.

I have tried to solve this convolution before and gave up as I essentially found myself in an infinite integration by parts loop. If the analytical solution is perhaps too complicated, could anyone point me in the direction of how the properties of a filter with a particular window are actually derived ?

Answer 1

The properties of a filter designed using the window method are dependent on two parameters:

• The length of the window
• The design window function

---

Long story short:

• The transition bandwidth is approximately equal to the window's main lobe width, which is itself dependent on the window function AND window length. For example, a rectangular window of length $M$ has main side lobe width $\dfrac{4\pi}{M}$ (including the negative half, see @RickarySanchez's comment), whereas a hamming window of length $M$ has main side lobe width $\dfrac{8\pi}{M}$. For the same length, the rectangular window hence has a sharper transition width.
• The attenuation depends on the window's side lobe height, which is itself dependent on the window function. For example, a rectangular window of length $M$ has first side-lobe height at $-13\,\text{dB}$ while a hamming window of length $M$ has first side-lobe height at $-41\,\text{dB}$. For the same length, the hamming window will therefore have better attenuation in the stop-band.
  Less important but worth noting: for a given window, the attenuation also depends on the transition bandwidth (the sharper the transition, the better the attenuation).
• For the cut-off $\omega_c$ to be at $-6 \,\text{dB}$, the transition bandwidth needs to be sharp enough. An illustration can be found in a previous answer of mine.

---

Further reading

1. Read this to understand the window method for FIR low-pass filters and how the parameters affect the resulting filter. This is also a great reference.
2. Read this to see example designs based on 2