Autocorrelation function of a discrete signal

Question

For a certain experiment of which I have data, I want to calculate the autocorrelation of the measurements. For this I used the expression from Wikipedia, which reads

Here, $n$ is the dataset's size, $0 < k < n$, $\sigma$ is the standard deviation and $\mu$ is the mean.

Now, I've read the descriptions on the page, but I don't fully understand all of the terms in there. I understand why we divide by $(n-k$), because the number of datapoints used constantly changes (you have a finite signal, after all). However, I don't see why we divide by the variance. Moreover, I also don't know why we subtract the mean. Doesn't this lead to problems, if for example for n = 50 and k = 49, we want to multiply $(x_1-\mu)(x_{50}-\mu)$, and $x_1 > \mu$ but $x_{50} < \mu$? I don't really see what a negative correlation would entail.

For my data, which is only 0's and 1's, this is even more problematic. Using the above formula the correlations quickly decay to zero, but at the end they violently start to spike, because the contribution from the mean becomes so big, relatively. In my data the mean is 0.5 and the variance is 0.25, so for example, my final point $R(49)$ = $\frac{(0-0.5)(1-0.5)}{0.25}$ = -1

But -1 is huge, especially when 'previously' the correlations had died out to 0. How does one deal with this?

An illustration of this is

Answer 1

First: you should really read some basic theory of autocorrelation. For example.

I also don't know why we subtract the mean

First, to subtract the mean is the usual and right thing to do – it's so standard that often it is straightly assumed that the signal has zero mean. Recall that the covariance of two variables is $E[ (x-\mu_x)(y-\mu_y)]$ – of which the variance is a special case. Here we are assuming that the process is stationary, so the mean is constant.

I don't really see what a negative correlation would entail.

The correlation (as the covariance) can be positive, zero or negative. A negative correlation says that the values tend to have opposite values, that is: when X has a "big value" (relative to the mean), then it's more probable that Y has a "small value" (relative to the mean). If the correlation is positive, then's the reverse.

However, I don't see why we divide by the variance.

You don't have to. You divide by it if you want a normalized correlation (rather: a correlation coefficient), that will be in the $[-1, 1]$ range.

I understand why we divide by $(n−k)$...

Actually you have two alternatives here: if you divide by $(n-k)$ you get an unbiased estimator (good), but, you get large variance (the "noise" you see in the graph) for large values of $k$ (bad; however, one is often interested only in the correlation for small values of $k$). Alternatively, one can divide by $n$; this produce a biased estimator but better behaved (lower variance) for large $k$. This is also explained here or in any textbook about correlation estimators.