4
$\begingroup$

Thanks in advance for the help. I don't post on here often, so I hope I can convey my question properly.

I think I understand what the Allan deviation (same as IEEE Allan variance?) does. You take a signal, divide it into sections, average the sections, then basically find the std of the differences between those averages. Scale by 1/2 because there are two samples in the difference, makes it the same as a std for white noise.

I've implemented the IEEE allan variance in MATLAB. I have an ADC signal I want to coadd (averaging filter with no overlap) to obtain the lowest noise. Here is the output.

enter image description here

If my noise was perfectly white it would follow the orange line, 1/sqrt(tau). But it has pink noise, so it flattens out. I've thought that means because of pink noise, coadding or averaging after a certain point won't make the signal less noisy. So using the above allan deviation, I think I should coadd 512 samples.

I decided to verify. I took my data (sampled over 15hrs to capture the 1/f better) and processed it with different rates of coadding. I expected to see 512 be the best. Here is what I found.

enter image description here

This makes it look like I can coadd forever and get lower and lower noise, in spite of the pink noise. This contradicts what I thought I knew, and makes me wonder if I should sample as fast as possible, and coadd as much as I can.

Why is this? Does it have something to do with having less samples left after coadding more?

Any help would be super appreciated.

UPDATE: Incredible answer by @DanBoschen. I've been studying his reply and I've learned several things. I have some follow-up questions and comments.

First, I tried to allude to this in my original post, but I think what the IEEE calls the "IEEE Allan Variance" is in fact an Allen Deviation. I might be totally wrong, but that's the impression I got from here. https://tf.nist.gov/phase/Properties/four.htm This is the page I used to write my script.

The first thing I learned from this answer is I was incorrectly assuming changing averaging time was changing capture time. It makes total sense that I am taking the same overall power and cutting down the bandwidth more and more. So for a certain capture time, it's always lower noise to average over the entire window. This is why I changed the x-axis of my plot to frequency instead of time. I was incorrectly correlating coadding and the allan deviation.

This does give me an interesting problem. By shrinking my capture window, I can achieve less noise power. However by growing my capture window I can average more, cutting my bandwidth also achieving less noise power. I imagine there is some sweet spot between capturing less and averaging more that gives the best performance. I wonder if there is a mathematical way of finding this point?

Here is my corrected IEEE Allan variance plot: enter image description here

$\endgroup$
3
  • $\begingroup$ Is the plot ADEV or AVAR (the vertical axis would be $\sigma^2$ is AVAR, and in either caes the horizontal axis is time in seconds ($\tau$) not frequency. If ADEV the line shown is going down at $1/\tau$ instead of $1/\sqrt{\tau}$ (The two are related as $ADEV = \sqrt{AVAR}$). When/if the noise is white the rms computation would match the ADEV computation (can you plot the noise plot on a log log scale? That should help with that comparison). Also the use of "Rate" in the horizontal axis for your averaging result is confusing as rate means frequency-- $\endgroup$ Commented Apr 6, 2023 at 0:07
  • $\begingroup$ ...It would help if you plot in the same units as time in seconds as well to allow for a clean comparison (Both plots as the standard deviation versus block averaging time in seconds, both in a log log scale). $\endgroup$ Commented Apr 6, 2023 at 0:13
  • $\begingroup$ I'm not an expert on Allen variance -- but search on "pink noise" and "Allen variance". Or "flicker noise", which is more or less synonymous with pink noise. $\endgroup$
    – TimWescott
    Commented Apr 6, 2023 at 0:51

1 Answer 1

5
$\begingroup$

What the OP is seeing in lower and lower noise as the averaging duration increases makes sense as the capture duration itself has not changed. It is the duration of the capture that will limit the effects of the lower frequency noise contributions. The averaging is a low pass filter (a frequency response in the shape of a Sinc), so given a single capture with total noise $P$, increasing the span of the moving average will effectively lower the bandwidth of the filter and therefore reduce how much of $P$ gets through the filter. Since the noise density is increasing as we approach DC, the benefit of the filtering will get progressively less, which is exactly what we are seeing.

What the ADEV curve is informing us is the total duration of the capture to minimize the total noise power $P$: with white noise, $P$ is constant regardless of capture duration, but with pink noise, with the spectrum increasing at 10 dB/decade as we approach DC, longer observations lead to an increase in total noise power. This is because the total duration of the capture sets the low frequency cutoff for the total noise in the signal: for a capture length of $T$ seconds, the process of the finite duration capture is effectively a high pass filter with a cutoff freq of $1/T$ (consider that it takes an infinite amount of time to truly observe DC).

I made a demonstration to help clarify this with an example noise signal that has a $1/f^2$, $1/f$ and white noise component as shown in the spectrum below. Here we see what I mentioned above with regards to noise increasing as we approach DC: and here with white noise clearly above 100KHz, if we only observed this for 10 us, we would not measure the elevated noise effects from the $1/f$ portions at 10 KHz and lower:

demo spectrum

The ADEV for this example noise was computed as follows, and the conclusion from this is we can operate on captures up to about 200 samples while assuming stationarity in the signal. Specifically for the durations where the ADEV curve is going down at $1/\sqrt{\tau}$ it means we have a white noise condition, a stationary signal with power independent of capture length:

ADEV for noise sample

The result for the moving average on this example noise data at different averaging times matches the OP's result and the intuition provided above, that with the moving average we are simply filtering the same noise process with a tighter and tighter low pass filter: the total noise as given by the standard deviation of the output after the moving average will always go down as averaging time is increased, but in the process we only have a narrower usable spectrum to work with.

Moving Avg Result

However below is a plot where I show the computed standard deviations vs block length within the same data. For shorter durations up through the time the ADEV plot is going down at $1/\sqrt{\tau}$ we expect the standard deviation to remain constant. It appears to go down at 10 seconds, but the data is 1 sample/sec so there are insignificant samples to get an accurate standard deviation at this time duration. What we do see is the gradual increase in standard deviation as we go past 100 seconds and beyond consistent with the floor in the ADEV, and then increasing significantly as we continue beyond that.

standard deviation vs block length

Consider the time series of the captured data directly as plotted below. Here we see the characteristics of a non-stationary signal where the longer we observe the further away we get from the starting position. We can take the average of the entire sequence to get the mean, but if we continue to observe longer, there will be a completely different mean. The averaging we choose to do is set by the bandwidth of the actual signal we are interested in, such that we minimize noise that is outside of this bandwidth. Also we see how the total noise (as given by the total deviations) will increase the longer we make the capture. A longer capture is necessary if we also want to observe low frequency signals, otherwise a high pass filter can be used to eliminate the noise elevation (and make the signal stationary)- this is what the ADEV computation does to provide consistent metrics for non-stationary signals. We see here that over much shorter capture durations the shorter capture itself has all the characteristics of a stationary process (within that capture the mean and standard deviation are relatively constant). With review of the ADEV we see that this assumption holds up pretty well out to 100 seconds. The plot below extends out to 4 million seconds.

time series of capture

Control Test with White Noise

Below is the experiment above repeated with a white noise test signal, First showing the ADEV:

White Noise ADEV

And then the standard deviation vs block averaging interval:

standard dev vs block duration - white noise

For completeness of intuition, I include a time series capture below for the white noise data used above, where we see the characteristics of a stationary process (mean and standard deviation do not change with capture duration).

time series capture - white noise

Conclusions with Regards to ADC Capture

The sampling rate sets the usable bandwidth according to Nyquist. The total noise power within that bandwidth will increase with capture duration if we have 1/f noise present or will remain constant if the noise is indeed white stationary noise over the duration of capture. If we subsequently perform low pass filtering (moving average) we are reducing the usable bandwidth (and can subsequently reduce the sampling rate - such as done with decimation), with a corresponding reduction in total noise. This is exactly what the OP is seeing, and the longer the average, the tighter that filter (on the same original process of a fixed total duration) and therefore the resulting total noise will keep going down.

The floor in the ADEV curve informs us of how long the signal can reasonably be assumed to be stationary, and with that, the total noise power for any capture duration up to that time will remain constant. Beyond that duration with longer captures we will start to see an increase in total noise power. The OP is averaging for longer and longer intervals, which is reducing the total noise but also reducing the bandwidth available for any signals of interest.

The upturn in ADEV is an indication that the signal as captured is no longer stationary. Not stationary meaning that if we were to continue to increase the duration, any estimate of the mean will be varying and standard deviation will be increasing as the capture length increases. This is in contrast to a captured signal that was stationary, where the error itself in our estimate of the mean and standard deviation would get progressively smaller as the capture duration increases (and converge to the "true" mean and standard deviation). Therefore the ADEV tells us the degree of stability over a given time duration for the variable of interest.

$\endgroup$
4
  • 1
    $\begingroup$ Incredible answer! Thanks for taking the time to write this out, I've learned a lot from looking at your examples. I've added an update to my post with some follow up comments and questions. $\endgroup$ Commented Apr 6, 2023 at 20:09
  • 1
    $\begingroup$ @nedflanders I'm impressed with your work and what you absorbed about this! Instead of adding new questions, can you close this one out and create a new question (or questions)? What is here is already quite long---assuming I answered your original question as posted that will be best. $\endgroup$ Commented Apr 7, 2023 at 0:04
  • $\begingroup$ @nedflanders Can you mvoe your additional questions to a new question? $\endgroup$ Commented Apr 8, 2023 at 0:47
  • $\begingroup$ dsp.stackexchange.com/questions/87466 $\endgroup$ Commented Apr 10, 2023 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.