For a certain experiment of which I have data, I want to calculate the autocorrelation of the measurements. For this I used the expression from Wikipedia, which reads

enter image description here

Here, $n$ is the dataset's size, $0 < k < n$, $\sigma$ is the standard deviation and $\mu$ is the mean.

Now, I've read the descriptions on the page, but I don't fully understand all of the terms in there. I understand why we divide by $(n-k$), because the number of datapoints used constantly changes (you have a finite signal, after all). However, I don't see why we divide by the variance. Moreover, I also don't know why we subtract the mean. Doesn't this lead to problems, if for example for n = 50 and k = 49, we want to multiply $(x_1-\mu)(x_{50}-\mu)$, and $x_1 > \mu$ but $x_{50} < \mu$? I don't really see what a negative correlation would entail.

For my data, which is only 0's and 1's, this is even more problematic. Using the above formula the correlations quickly decay to zero, but at the end they violently start to spike, because the contribution from the mean becomes so big, relatively. In my data the mean is 0.5 and the variance is 0.25, so for example, my final point $R(49)$ = $\frac{(0-0.5)(1-0.5)}{0.25}$ = -1

But -1 is huge, especially when 'previously' the correlations had died out to 0. How does one deal with this?

An illustration of this is enter image description here

  • $\begingroup$ My slightly unacademic guess would be that you just take the first, say, 2500 points as the signal has already died out by then? $\endgroup$ – user129412 Apr 15 '14 at 15:11

First: you should really read some basic theory of autocorrelation. For example.

I also don't know why we subtract the mean

First, to subtract the mean is the usual and right thing to do – it's so standard that often it is straightly assumed that the signal has zero mean. Recall that the covariance of two variables is $E[ (x-\mu_x)(y-\mu_y)]$ – of which the variance is a special case. Here we are assuming that the process is stationary, so the mean is constant.

I don't really see what a negative correlation would entail.

The correlation (as the covariance) can be positive, zero or negative. A negative correlation says that the values tend to have opposite values, that is: when X has a "big value" (relative to the mean), then it's more probable that Y has a "small value" (relative to the mean). If the correlation is positive, then's the reverse.

However, I don't see why we divide by the variance.

You don't have to. You divide by it if you want a normalized correlation (rather: a correlation coefficient), that will be in the $[-1, 1]$ range.

I understand why we divide by $(n−k)$...

Actually you have two alternatives here: if you divide by $(n-k)$ you get an unbiased estimator (good), but, you get large variance (the "noise" you see in the graph) for large values of $k$ (bad; however, one is often interested only in the correlation for small values of $k$). Alternatively, one can divide by $n$; this produce a biased estimator but better behaved (lower variance) for large $k$. This is also explained here or in any textbook about correlation estimators.

| improve this answer | |
  • $\begingroup$ Thank you for the detailed explanation. I'm indeed very new to the subject (my supervisor pointed me towards it today and I just tried to jump straight into it, rarely a good choice..) so I should definitely read up. $\endgroup$ – user129412 Apr 15 '14 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.