The resolution of a window is typically defined by the width of its spectral mainlobe (3dB-width or null-to-null width). It is often stated that the rectangular window has the best resolution among all windows. In my work, I'd be looking for a reference where it is rigorously proved that there exists no window that has a better resolution. Does anyone know where I could find something like this (at least I could not find anything yet)?
Thanks for your help.
Wikipedia has a very nice article on window functions https://en.wikipedia.org/wiki/Window_function
The basic trade-offs in windows is main lobe width vs side lobe height and overall attenuation. The rectangular window is at one end of this spectrum: it is the "widest in time", i.e. has the most energy at the window edges (compared to the other "normal" windows. However, it has also very poor side lobe behavior.
rigorously prove that there exists no window that has a better resolution.
You can't prove this, since it's not true. Not unless you put some constraints on what exactly qualifies as a window. Per my argument above, the width of the main lobe is determined by how much energy is at the edges. If that's what you want, you don't need to stop at a rectangular window: you can crank it up some more. Raise the edges and reduce the middle.
For example you can use window that is $w[n] = 1-hanning[n]$. If you do the spectral analysis you see that this is actually narrower than a rectangular window. But, off course, the side lobes are terrible.
If we use a criteria of equivalent noise bandwidth under white noise condition, the rectangular window will always win. The equivalent noise bandwidth is the bandwidth of a brickwall filter that will have the same noise level as the window under conditions of filtering white noise (noise with the same spectral density at all frequencies. The Equivalent Noise Bandwidth (ENBW) is given as:
$$ENBW = N \frac{\displaystyle\sum_{n=0}^{N-1} {\big(w[n]^2\big)}}{\left(\displaystyle\sum_{n=0}^{N-1} {w[n]}\right)^2} \tag{1} \label{1}$$
And predicts the noise we would get if we integrated the Kernel (Discrete Time Fourier Transform) of the window. This formula comes about by recognizing that a window in time represents a weighted average of the time domain samples below the window in the Discrete Fourier Transform, and the Kernel is the frequency response of this averaging function. In such averaging of a white noise process, statistics shows us that the magnitude of the coherent samples converge toward the mean while the noise samples (with a magnitude given as the standard deviation) will go down by $\sqrt{N}$.
Thus for a DFT, we have the "coherent gain" and "non-coherent gain" of each bin as given by a window of $N$ samples, where the coherent gain is the expected growth in magnitude in the DFT of single tones or signals whose spectrum occupies less than the resolution bandwidth of the window, and the non-coherent gain is the growth of white-noise.
The coherent gain is given as follows which we may recognize to simply be the mean of the window:
$$G_c = \frac{\displaystyle\sum_{n=0}^{N-1} w[n]}{N} \tag{2} \label{2}$$
And similarly the non-coherent gain is given by the root-mean-square of the window:
$$G_{nc} = \sqrt{\frac{\displaystyle\sum_{n=0}^{N-1} w[n]^2}{N}} \tag{3} \label{3}$$
Whether we actually divide by $N$ or not in the above equations in predicting the coherent and non-coherent gain rests in the form of the DFT used. The common format in MATLAB, Octave and Python is not scaled by $N$, and in this case we would see that the coherent signals increase by $N$ while the magnitude (rms) of the white noise increases by $\sqrt{N}$. In either case, if we square each to put in power quantities, the division of the two would be the processing gain of the window, notably the change in $SNR$ and given as:
$$PG = \frac{G_c^2}{G_{nc}^2} \tag{4} \label{4}$$
As a power quantity when expressed in dB as commonly done for "processing gain" would be $10\log_{10}(G_c/G_{nc})$. We see that combining equations \ref{2}, \ref{3}, and \ref{4} for the case of a rectangular window results in a processing gain of $10\log_{10}{N}$ which may ring bells for those familiar with spread spectrum and CDMA.
This is clear when we see how we get this "processing gain" by filtering out noise, so in this case of white noise, we have removed $1/N$th of the total noise when using a rectangular window.
The formula for the equivalent noise bandwidth given above is simply the reciprocal of the processing gain, and we see by taking the reciprocal of \ref{4} we end up with \ref{1}.
What is well established is that under a white noise condition, the rectangular window provides the optimal "matched filter", and therefore would have the lowest equivalent noise bandwidth when compared to any other filter. This is very clear when we consider the underlying moving average operation- under a white noise condition, each sample is independent and identically distributed, so giving equal weight to each sample in determining our estimate of the average would then minimize the final deviation due to the noise versus giving preference to any particular sample- this is the theory of optimal ratio combining where we combine all samples based on their respective SNR. If we want to establish noise bandwidth as a metric of resolution, then in that case it is clear that the rectangular window is best.
Indeed we can recognize from equation \ref{1} given the numerator is increasing by a sum of the squares for the window coefficients, while the denominator is increasing by the square of the sum of the window coefficient, how a uniform rectangular window minimizes the ENBW to be $1/N$. This conclusion all changes for other signal and noise conditions other than this white noise case described where the signal to noise of each sample is the same; the signal components from sample to sample are coherent and the noise components from sample to sample are independent. This also leads to the general conditions for optimized matched filters.
I provide further details on the equivalent noise bandwidth for windows here.
