0
$\begingroup$

We have the need to calculate the overlapping Allan deviation on possibly some billion data points (for long-running measurements). While alantools.oadev() is working for a certain workflow (up to 0.5 billion data points on a local computer with 16 GB RAM), it does not scale well. Are there any alternative implementations that would work with larger amounts of data, hopefully in a parallel way (using more than one CPU core)? There are a few research papers out there but actually no production-ready implementation.

$\endgroup$

1 Answer 1

0
$\begingroup$

The solution is to reduce the sample rate for long running measurements prior to using AllanTools. The trade is a wider confidence interval for any given tau, but it is likely given the description that the confidence interval is insignificant at the longer taus even after significantly decreasing the sampling rate: at what point do you consider the duration sufficient for a measurement at any given tau? If the measurement is sufficient to provide a reliable result with sufficiently small confidence interval, then continuing any further for longer taus can justifiably be done at a lower sampling rate (for example if 10 minutes was deemed to be sufficient for a 1 sec ADEV with samples captured at once/second (so 600 samples used in the averaging interval), then the same fidelity of a result is achieved for the ADEV at 1000 sec using a once every 1000 second update rate with a 600,000 sec averaging interval.

I recommend using the data at the highest rate as done now to compute the ADEV over several decades in the short-tau regions and then change over to progressively lower sampled data-sets as the longer time durations are determined. This will result in different runs over different ranges of taus, which can then be pieced together for the resulting ADEV plot for all taus.

It is important however that the data be properly resampled to avoid aliasing from down-sampling. This can be done with the scipy.resample command or a custom decimation filter if you are experienced with doing this. Thus any one sample at the lower rates will represent an average of all the samples over that duration either through digital filtering or proper analog filtering prior to being sampled.

$\endgroup$
1
  • $\begingroup$ I want to add that a simple moving average decimator would be optimum if your signal was white noise and assume (but don't know for certain) that would still be the case as you go beyond the flicker floor and into drift regions. The benefit of being able to use a moving average is that you can implement the resampling with a first order CIC structure which is so simple and efficient. I hope that part makes sense as it should really simplify what you are trying to do (create several data sets to use for each range of taus with appropriate sampling for that range). $\endgroup$ Jun 27 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.