UPDATE: I completed a verification simulation of the initial formula provided with two further insights: (1) The missing proportionality constant is 1.219, as updated in resulting formula below. (2) For real tones we must account for the sidelobe interaction, which doesn't create addition "noise" in the frequency estimate but will create a predictable offset. Certainly windowing will significantly reduce this interaction and it's associated frequency offset at the expense of frequency resolution: a good trade to make if more than one single complex tone is present.
The formula as given is an accurate prediction for a single complex tone in the presence of white noise.
If the strategy is to select the strongest bin of a zero-padded DFT as the estimation of the frequency, the error estimation will depend on the signal to noise ratio of the captured sample, and the resolution bandwidth of the DFT. In summary for a single complex tone with white noise, I confirmed that the frequency accuracy (given as the standard deviation of the frequency measurement) would be:
$$f_{rms} = \frac{1.219}{\sqrt{T^3 S/No}}$$
Where:
$No$ is the single-sided noise density in W/Hz
$S$ is the power in the sinusoidal tone in W
$T$ is the duration of the captured signal in seconds.
Note 1: W above can be any unit of power since it cancels in the formula, and ultimately we are just interested in the ratio of the power in the noise relative to the signal.
Note 2: The FFT gives us the equivalent of a "quantized in frequency" result of the DTFT. If the predicted frequency accuracy (noise) above is less than the bin spacing, then it would be to our advantage given the selection strategy to zero pad the FFT to reduce the bin spacing to at least 10x less than that predicted accuracy given; otherwise our result in selecting the highest FFT bin would be limited to always selecting that "quantized" result and could have a peak error up to one half the bin spacing regardless of our noise (of course we could introduce optimized strategies which could use the information in each bin to improve the estimate, notably the adjacent bins).
Note 3: The side-lobe interaction between the two bins for a real tone (associated with the positive and negative frequencies we would see for a sinusoid in a continuous time Fourier Transform) will induce a known and predictable frequency offset when using a max absolute value criterion to select the closest frequency bin in the zero-padded FFT. This offset is not included in the noise prediction since it is a static offset, but if not accounted for will lead to additional errors in a frequency prediction for real tones. Similarly the effect of multiple tones will induce additional errors and reducing these offset errors specifically is a motivation for windowing.
If the noise is white (spread evenly over all frequencies), then the rectangular window (no further windowing) is the best strategy, leading the narrowest resolution and best output SNR of the FFT or DTFT. Any other window will both reduce the SNR and widen the resolution leading to lower accuracy (however would simplify the required offset compensation due to the sidelobes as described in the notes)- however this would make sense to do in conditions of interference that isn't white since it will result in significantly reduced side-lobes that would otherwise degrade accuracy in the estimate. The frequency resolution of the FFT if no windowing is done is $1/T$ where $T$ is the time duration of the capture. For the rectangular window and white noise, this is the "Equivalent Noise Bandwidth" (abbreviated ENBW), meaning it would have the same noise power at the output that we would see with a brick-wall filter (with white noise at the input).
As an example to demonstrate, assume we are trying to measure the frequency of an unknown stable single frequency tone that is at full scale at the input to our A/D converter, and our only noise is the quantization noise, and we'll assume a perfect 12 bit converter. Quantization noise is well modeled as a white noise and the total noise power is given as 6 dB/bit +1.8 dB below a full scale sine wave. So in this case the noise is 6*12 +1.8 =73.8 dB below the power of the sine wave. This noise is spread evenly across our sampling frequency, assume we sampled at 1 MHz (and that we know our sine wave is well below 500 KHz).
If our sine wave is stable enough (stationary) to measure for a 0.1 second data capture (100,000 samples at the 1 MSps sampling rate), then our equivalent noise bandwidth of our FFT would be $1/0.1 = 10$ Hz. This means out of that total quantization noise that is 73.8 dB below our signal, and spread evenly over the Nyquist bandwidth (+/- 500 KHz), 10 Hz out of 1000 Hz will be in the FFT bin for our signal of interest. Our real signal will be in two bins (similar to the positive and negative frequency components in the Fourier Transform of the sinusoid), so if we use both, our signal to noise ratio will increase from $73.8$ dB to $73.8 dB + 10Log(10/500,000) = 120.8$ dB.
If we use the zero-padded FFT to provide the resolution of the DTFT, then the accuracy of the peak of that result will be limited only by the 120.8 dB SNR of that sample and the 10 Hz resolution bandwidth of the 0.1 second data capture (assuming we compensate for the side-lobe interaction). The mainlobe centered on the actual signal is a Sinc function with the first null at 10 Hz. In a small signal noise model, every unit of noise (normalized to the signal power) in vicinity of the peak of that mainlobe would increase approximately by a factor of 10 when converting to frequency noise given the 10 Hz offset to the first frequency null of the Sinc response of one bin in the DFT (note: this factor was an initial approximation based on the Sinc response of a DFT bin, subsequently confirmed through simulation but not derived); for example if you had noise that was 0.1 rms compared to the signal rms level (which is a 20 dB SNR), the frequency accuracy would be on the order of 1 Hz rms. In this example given with the SNR at 120.8 dB with a 0.1 second data capture, the noise result would be approximately $\frac{1}{0.1}(10^{-120.8/20}) = 9.12E-6$ Hz rms.
To see how this matches the more compact formula I first introduced, we'll start with what $S/No$ would be:
$No$ is the single sided noise density, which is the power from $0$ to $500$ KHz counting the contributions from both the positive and negative frequencies we would get over the double sided span of $-500$ KHz to $+500$ KHz and then divided by the Nyquist frequency span to get power/Hz. So in this case we had total power relative to the power in the sine wave that was -73.8 dB lower. Given this is spread over 500 KHz (single-sided), $S/No$ would be $-73.8 dB - 10Log(500,000)= -73.8-57.0= 130.8$ dBc/Hz. Thus with $T$ of $0.1$ seconds, and including the proportionality constant from the formula, without yet including the addditional proportionality constant we would get the same result as above:
$$f_{rms} = 1 * 10^{-130.8/20}/\sqrt{0.1^3} = 9.12E-6$$
Through subsequent simulation I found the accurate result would be increased by a factor of 1.219 (given the difference of the linear fit approximation I used for small offsets in vicinity of the Sinc DFT response versus the actual effects of that Sinc response):
$$f_{rms} = 1.219 * 10^{-130.8/20}/\sqrt{0.1^3} = 1.11E-5$$
As noted earlier, for real tones the sidelobe interaction will create an additional but predictable frequency offset (so an offset we can compensate for to get an accurate prediction rather than the resulting frequency noise due to signal SNR).
