How large FFTs can pull signals out of the noise floor?

Question

I am trying to detect unknown RF tones around -140 dBm and my scan BW is 5 MHz, going through the Noise power calculations the signal is below the thermal noise based on the scan BW. I read that using large FFTs can help to pull signals out of the noise floor. My question is how large FFTs can accomplish this?

The receiver used is an RFIC (LNA, Filter and ADC), and I was thinking FFTs of the order of 2¹⁶ or higher running on a hardware accelerator.

The unknown RF signals are ERP emissions from electronic devices which have RF circuitry.

Answer 1

65,536 point FFT, assuming a 3 dB overall receiver noise figure and a 7.5 MHz FFT sampling rate of a digitally down-coverted 5 MHz bandpass signal.

Sub-band the 5 MHz bandwidth if you have the time margin for post processing, and the number of points needed goes down accordingly.

To meet your objective we basically need a receiver with a resolution bandwidth of 126 Hz or less (assuming the 3 dB noise figure stated), as long as the -140 dBm power level stated for the signal of interest is in an occupied bandwidth of less than this. Notice the point of interest to determine our ability to detect is not what the total power is, but what is the power per unit Hz; the power spectral density!

Let me explain...

Both what Marcus and Stanley said are quite good and this will add some practicality to what you are doing specific to your "below the noise assumptions". It is a bit of a misnomer to say that the signal is below the noise when we only consider total power in a given bandwidth. What we really want to look at is power spectral density (power per unit Hz) of the signal as compared to the noise and then decide if it is truly below the noise floor or not.(the power spectral density of the demodulated signal at the decision detector specifically when we "match filter", which from your description does not sound like doing any demodulation is feasible in your case). If it is truly below the noise floor, then we are challenged. So as long as the power spectral density of the signal is well above that of the noise, we can proceed with simple techniques of detection.

You mention a "scan BW" which can imply you are actually "scanning" by translating an input frequency from an observation of a 5 MHz bandwidth of interest through a narrow observation filter and then to a detector OR you are sampling a 5 MHz bandwidth of interest and taking a FFT (which is consistent with the rest of your description). Either approach can be used and you see both in the implementation of Spectrum Analyzer instruments. I assume the use of the word "scan" is figurative and you are sampling an analog BW of 5 MHz and then "scanning" by taking a FFT of that bandwidth. (Which is a very reasonable approach to take). However if you were actually scanning, the rate of scan would be important to limit. I don't think you are doing that, so I won't go into the further considerations of scan rate.

Consistent with Stanley's first point, you really need to understand the average occupied bandwidth of these "unknown tones" when specifying their expected power level. Just to clarify first any "matched filtering" techniques: If we knew anything further about the time varying characteristics of the "tones", we could use matched filtering techniques to optimize detection, consistent with what Marcus was pointing out with the FFT as just one example of doing that; which is a good choice for the detection of pure tones specifically up to the limits of our own oscillator phase noise and stability in our receiver--- Stanley is spot on in that pure tones don't exist, so reality sets a limit on how far we can take that, and further since you called these "unknown" and hinted this is EMI radiation noise I suspect that any other elaborate matched filtering techniques would not be easily done. (However! If you do know of particular patterns the electronics is doing, then that could indeed be exploited). Another example of using this technique to pull signals "out of the noise" is CDMA where for example Walsh codes could be used; here we know the signals are modulated by those specific codes, so we can "match" them in the receiver to provide the correlation gain described. It is interesting to compare the decoding matrix for the FFT (or OFDM specifically) to Walsh codes; in both cases we are correlating to code sequences; the Walsh codes are all two symbols (1, -1) while the FFT allows for doing the same thing with a full complex set on the unit circle.

However without knowing such detailed information about the transmitted signal, the first strategy would be simple bandpass filtering if the actual occupied power spectral density of the signal of interest is higher than our limit set by the thermal noise floor.

Let's go through a calculation to see what bandwidth you would need and what considerations you would make and how this "simple bandpass filtering" requirement translates to number of points needed in the FFT.

First we need to decide on a detection margin, typical numbers range from +6 dB to +12 dB, and are derived from probability of false alarm vs probability of detection which is the trade space. For simple computation I will use a reasonable number of +10 dB. This means if the noise in my detector is 10 dB below the signal of interest, I can make a reasonably confident detection on the presence of the signal.

Specifically for this case with AWGN noise the probability of false alarm would be $2 Q(3.16) = 0.16\%$ from $10^{10 \text{ dB}/20} = 3.16$ and the related area under the tails of the Gaussian distribution.

Given the signal level of -140 dBm, this implies a desired noise floor of -150 dBm.

To know what our noise floor is for a given filtering bandwidth, we need to know the receiver noise figure, which must account for all noise contributions up to the detector (including ADC quantization noise which can be included in a classical cascaded noise figure calculation). For purpose of proceeding with this example, I will assume a receiver noise figure of 3 dB. (You can change this to whatever your noise figure is, but given your objective, you would want to be sure you know what this is and that it is well designed).

Therefore the noise floor density is $-174 \text{ dBm/Hz} + 3 \text{ dB} = -171 \text{ dBm/Hz}$, where $-174 \text{ dBm/Hz}$ is the thermal noise floor at room temperature as computed from Boltzmann's constant and the temperature ($kT$).

For a total noise power of -150 dBm, which is 21 dB above the noise density, this is an implied bandwidth of $10^{21/10}= 126 \text{ Hz} $

If you know that the signal of interest occupies a bandwidth of 126 Hz or less, or more specifically in that bandwidth maintains a power level of -140 dBm or more, then you have every opportunity to detect that signal with simple digital filtering techniques (just filter the occupied bandwidth). You can change the numbers above based on your actual system, but this would be the first thing I would look at in terms of getting there simply.

If you want more information such as what specific frequency that signal may be in, or if you know the signal meets that constraint but from measurement to measurement the signal location may change, or it occupies several of these bandwidth "bins", then the FFT is an excellent choice for that since it is essentially a bank of such filters.

This drives the minimum number of points required in the FFT which sets the resolution bandwidth (RBW) given that $RBW = f_s/N$ where N is the number of non-padded FFT samples. You didn't mention the sampling rate of your ADC or more specifically the sampling rate at the FFT computation, but for practical reasons I will assume a minimum sampling rate of 7.5 MHz at the FFT computation (allowing for simple anti-alias filtering ahead of your ADC, and ADC 5 MHz bandpass sampling of a real signal. Therefore to achieve the desired resolution bandwidth of 126 Hz, the number of points required in the FFT must be:

$$7.5 \text{ MHz}/ 126 \text{ Hz} = 59,500$$

Taking advantage of radix-2 implementations, I would use the next higher power of 2 to get $2^{16} = 65,536$ points

At this point there are several strategies to achieve this if not directly (such as sub-banding if you have post-processing time but limited resources), so the main point is to know what is the spectral density of your signal, and what is the actual noise floor (noise figure) of your receiver. In terms of spectral density, not total power but power / Hz, as long as the signal is indeed NOT below your noise, then detecting the signal will be no issue.

Answer 2

If you look at the formula of a single DFT bin

$$X[k] = \sum_{n=0}^{N-1}x[n]e^{-j2\pi k\frac nN}\text,$$

you'll notice that his is essentially a correlation of $x$ with the complex sinusoid $e^{-j2\pi k\frac nN}$.

That means the DFT can just be understood as a filter bank of matched filters for single tones that fall in the DFT "raster".

Hence, you simply get FFT length-based processing gain: The length of the sum.

But: you probably don't have perfect knowledge of the exact frequency of the signal you're trying to detect¹! So, you can't put things into that perfect DFT raster.

Now, the larger you choose the FFT length $N$, the finer that raster will get, but also, the longer your observation has to be, and the more compute power you'll need.

At some point, the DFT stops being the best possible tone detector, and superresolution techniques become relevant. In this case (weak tone, you're sure that you've only got exactly one tone in your signal), the ESPRIT algorithm with a long observation period leading to the autocovariance matrix estimate that it takes as input, would probably work very nicely.

---

¹ There's inevitably frequency error in your receiver, and in your transmitter. Papers that start with We assume perfect synchronization typically skip the hard part of making a system work...

Answer 3

To say there are only 2 problems with DFT based detection of weak tones would be wrong but I’m going to ignore myself and say there are two problems.

Problem 1:

A pure tone is a rare thing. It has to exist forever and never deviate from its tonal form. Real tones turn on and off, fade, and drift, and are often a component of a more complex waveforms. A real tone has bandwidth. One cannot arbitrarily increase the length of a DFT. One can run out of signal. One can over resolve a tone. One’s local time reference isn’t perfect.

Problem 2

Neglecting things like heavy tailed nonGaussian correlated noise and the tendency of real situations to have other tonal components in proximity, like looking for a weak tone in the presence of a unknown number of other tones.

An M point DFT produces M outputs. You need to pick the correct bin. This generally comes down to the “correct” bin being a reliable local maximum. The detection statistics of the maximum of a set of correlated bins is a hard problem. One usually relies on heuristics.

The finite observation and over resolution problem can be mitigated to some extent by combining coherent and incoherent gain. That is, averaging a bin amplitude over some number of coherent DFTs.

The DFT is not a consistent estimator. It doesn’t reduce the noise variance as the length of the DFT increases but it does provide SNR gain providing the tone is essentially tonal over its time window.

Averaging the bin power or bin magnitude does reduce variance. The combination of coherent and incoherent processing is a consistent estimate.

In a real application, one needs to find that sweet spot.

Im not telling you not to use a covariance based frequency estimator. ESPIRT was originally developed to solve a direction finding problem in array processing. It works well but there are a different set of tradeoffs in array processing even if the mathematics are nearly the same.