0
$\begingroup$

I have a pressure signal (y) with 512000 samples and with a sampling frequency 5000 sample/sec. I am trying to find the number of statistically independent samples using autocorrelation. I used

autocorr(y)

I do not know if this is correct or how to get the number of independent samples

I found this question, but there is no valid answer

https://stats.stackexchange.com/questions/149517/autocorrelation-and-statistically-independent-samples

enter image description here

Thanks

$\endgroup$
  • 2
    $\begingroup$ What exactly do you mean by "the number of independent samples?" $\endgroup$ – AnonSubmitter85 Mar 3 '18 at 20:56
  • $\begingroup$ @ AnonSubmitter85. Thank you. I mean the signal after that will not have a correlation $\endgroup$ – Math Mar 3 '18 at 21:00
  • $\begingroup$ Do you mean that you want to the data to be critically sampled? The inverse of the bandwidth will tell you the time-domain resolution, if that is what you are after. $\endgroup$ – AnonSubmitter85 Mar 3 '18 at 21:25
  • $\begingroup$ Thanks. You can see at Lag 500 almost the correlation lost. I need to know how many sample at Lag 500. That what I meant $\endgroup$ – Math Mar 3 '18 at 21:50
  • $\begingroup$ If your timeseries is Normal, one can apply a linear filter to whiten the data so all the points will be independent. Your autocorrelation doesn’t look like Gaussian Noise so the concept of independent samples would appear to be misapplied. perhaps you could elaborate on your goal. $\endgroup$ – Stanley Pawlukiewicz Mar 3 '18 at 21:58
1
$\begingroup$

The autocorrelation you obtained shows that the signal becomes decorrelated with itself for specific time delays, which are the zero-crossings of the autocorrelation. The autocorrelation is not zero except at these time instants.

You can also define a threshold $\tau$ and say that, if the autocorrelation is less than $\tau$, then for your purposes the signal is decorrelated for any time shift $\Delta$ for which the autocorrelation is less than $\tau$.

When calculating the autocorrelation in Matlab (or similar), the lag is calculated in number of samples. For example, in your case it seems like the autocorrelation is zero for a lag of 200. Since your sampling interval is $T_s = 1/5000 = 200\,\mu$s, then your signal becomes decorrelated for a shift of 200 samples, or a time shift $\Delta = 200 T_s = 40\,$ms.

$\endgroup$
  • $\begingroup$ Thank you. You mean the number of independent samples is 200 samples then the signal will lose it correlation? $\endgroup$ – Math Mar 3 '18 at 22:50
  • $\begingroup$ No. I mean that if you time-shift the signal by 200 samples, it becomes decorrelated with the original signal. It's not clear (as was commented above) what it is that you mean by "independent samples". $\endgroup$ – MBaz Mar 3 '18 at 23:03
  • $\begingroup$ @Math: If the term you are using is what I think you are referring to, then, in statistics, the term you are referring to is usually (AFAIK) called the effective number of independent observations given a set of correlated observations. That concept along with formulae is provided at this link: degruyter.com/downloadpdf/j/mms.2010.xvii.issue-1/… $\endgroup$ – mark leeds Mar 4 '18 at 1:22
1
$\begingroup$

The "Effective" Number of Independent Observations in an Autocorrelated Time Series is a defined statistical term - https://www.jstor.org/stable/2983560?seq=1#page_scan_tab_contents

The number of independent observations n' of n observations with a constant variance but having a lag 1 autocorrelation $\rho$ equals

$n'=n\frac{1-\rho}{1+\rho}$

Also note this is an approximation valid for large n, [1] (reference provided by Ed V) equation 7 is more accurate for small n.

[1] N.F. Zhang, "Calculation of the uncertainty of the mean of autocorrelated measurements", Metrologia 43 (2006) S276-S281.

$\endgroup$
  • $\begingroup$ Nice answer and thanks for that reference: I upvoted both answers! An "effective number of observations" is defined in equation 3 in this reference: A. Zięba, P. Ramza, STANDARD DEVIATION OF THE MEAN OF AUTOCORRELATED OBSERVATIONS ESTIMATED WITH THE USE OF THE AUTOCORRELATION FUNCTION ESTIMATED FROM THE DATA, Metrol. Meas. Syst., Vol. XVIII (2011), No. 4, pp. 529–542. (The authors used all upper case in their paper's title, by the way.) Your equation also nicely ties into the variance reduction ratio (VFF): see "Unbiased estimation of standard deviation" in wikipedia. $\endgroup$ – Ed V Jul 22 at 2:03
  • 1
    $\begingroup$ Thanks @EdV, I got that equation from T. Agami Reddy and David E. Claridge "Uncertainty of “Measured” Energy Savings from Statistical Baseline Models" and the the reference above from following the reference in the Reddy paper to the Neter, J., W. Wasserman, and M.H. Kutner. 1989. Applied Linear Regression Models, and then somehow to the 1946 paper! Your reference is useful since I do want to understand how it generalised beyond AR(1) $\endgroup$ – David Waterworth Jul 22 at 4:33
  • $\begingroup$ You might be interested in this paper as well: N.F. Zhang, "Calculation of the uncertainty of the mean of autocorrelated measurements", Metrologia 43 (2006) S276-S281. As well, the cross validation exchange site has quite a bit on AR(p) processes and real experts on the topic: I have learned quite a bit there. Thanks again! $\endgroup$ – Ed V Jul 22 at 15:36
  • $\begingroup$ Thanks for that, it's quite useful. I spent a bit of time reading it to convince myself that the equation I posted was correct. I eventually worked out it's an approximation as n->inf $\endgroup$ – David Waterworth Jul 26 at 3:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.