# Questions about autocorrelation, correlation, and randomness

I'm learning about autocorrelation functions and trying to calculate some from a time series. Conceptually, what is the difference between the following two autocorrelation functions?

$$g_{1}(\tau)=\frac{\big\langle (A(t)-\left \langle A \right\rangle)(A(t+\tau)-\left \langle A \right\rangle ) \big\rangle}{\big\langle (A(t) - \left \langle A \right \rangle )^{2} \big \rangle}$$

$$g_{2}(\tau)=\frac{\big\langle A(t)A(t+\tau)\big\rangle}{\big\langle A(t)^{2} \big\rangle}$$

My understanding is that the autocorrelation for the second equation will asymptotically go to zero if there is no correlation between the events in the time series. Is that the case? What about for the first equation? And why?

For the first equation, is the average value $$\left \langle A \right \rangle$$ the average over all of $$\tau$$, or do you need to take a running average and calculate it for each time $$t$$ when doing the calculation?

What happens if the time series is correlated? Will it never approach zero (in the case of the second equation)?

Is there any a priori way of determining how far to take the lag time $$\tau$$?

Finally, how does all of this relate to randomness?

1) g1 assumes that there is a non-zero mean and that always needs to be subtracted off for the correlation calc to make sense. g2 assumes that the mean is zero. g2 is incorrect if the mean is not zero.

2) the mean is taken over the whole sample because the autocorrelation calc has the underlying assumption that the process is stationary so the mean doesn't change during the process.

3) As the lag gets larger, then, if the process is stationary, then the autocorrelations should doe down approach zero. Note that there are "long memory" processes that don't have this characteristic but I don't think that they are considered stationary processes. Someone feel free to correct me if I'm wrong about long memory processes not being considered stationary.

And now for a completely different answer....

Your two functions $$g_1(\tau)$$ and $$g_2(\tau)$$ are sort of related to or minor variations (normalizations if you prefer) of what are commonly called the autocovariance function and the autocorrelation function respectively of wide-sense-stationary random processes on dsp.SE. Note, though, that apparently these terms have different meanings in the statistical circles in which @MarkLeeds travels and his answer reflects the different meanings in his natural habitat.

On dsp.SE and in signal processing circles in general, wide-sense-stationary random processes are generally assumed to be characterized by two properties:

1. Constant mean. All the random variables $$A(t)$$ comprising the random process have the same expected value $$\mu$$, i.e., $$\langle A(t) \rangle = \mu\tag{1a}~~\text{for all choices of } t.$$
2. Autocorrelation function $$\langle A(t)A(t^\prime) \rangle$$ that depends only on the difference between the two time instants $$t$$ and $$t^\prime$$ and not upon the individual values $$t$$ and $$t^\prime$$. Thus the autocorrelation function can written as $$R_A(\tau) = \langle A(t)A(t+\tau) \rangle ~~\text{for wide-sense-stationary processes}.\tag{1b}$$ Note that the difference between $$t^\prime = t+\tau$$ and $$t$$ is just $$\tau$$ and thus $$R(\cdot)$$ depends only on the time difference $$\tau$$ and not upon the individual values $$t$$ and $$t+\tau$$, as required.

In addition to the property that all random variables $$A(t)$$ comprising the wide-sense-stationary random process have the same mean $$\mu$$, all these random variables have the same variance $$\sigma^2$$. One commonly used formula for the variance of a random variable $$X$$ is $$\operatorname{var}(X) = \langle X^2\rangle - \left(\langle X\rangle\right)^2,$$ and thus $$\sigma^2 = \operatorname{var}(A(t)) = \langle (A(t))^2\rangle - \left(\langle A(t)\rangle\right)^2 = R_A(0) - \mu^2 ~~\text{for all choices of }t.\tag{1c}$$

A more general concept than wide-sense-stationarity is covariance-stationarity or autocovariance-stationarity for those who like to dot their eyes and cross their teas. A covariance-stationary process $$\{A(t)\}$$ has the property that the autocovariance function of the process, which is defined as $$C_A(t,t+\tau) = \big\langle [A(t)-\langle A(t)\rangle][A(t+\tau)-\langle A(t+\tau)\rangle]\big\rangle \tag{2}$$ and is generally a function of both $$t$$ and $$\tau$$, depends only on $$\tau$$, the time difference between the two time instants $$t$$ and $$t+\tau$$. All wide-sense-stationary processes have constant mean $$\mu$$ and thus $$(2)$$ simplifies to \begin{align}C_A(t,t+\tau) &= \big\langle [A(t)-\mu][A(t+\tau)-\mu]\big\rangle\\ &= \big\langle A(t)A(t+\tau) - \mu A(t) - \mu A(t+\tau) + \mu^2\big\rangle\\ &= \big\langle A(t)A(t+\tau)\big\rangle - \mu\big\langle A(t)\big\rangle - \mu \big\langle A(t+\tau)\big\rangle + \mu^2\\ &= R_A(\tau) - \mu^2 - \mu^2 + \mu^2 &\scriptstyle{\text{apply } (1a), (1b)}\\ &= R_A(\tau) - \mu^2 \end{align} that is, the autocovariance function of a wide-sense-stationary process also depends on just the time difference $$\tau$$ between $$t$$ and $$t+\tau$$ and so the process is covariance-stationary as well. In summary, wide-sense-stationary processes are also covariance-stationary processes. However, the converse is not necessarily true; a covariances-stationary process is not necessarily a wide-sense-stationary process. The key difference is that the mean of a covariance-stationary process need not be a constant as is required for wide-sense-stationary processes. Typically, a covariance-stationary process $$\{B(t)\}$$is the sum of a wide-sense-stationary process $$\{A(t)\}$$ and a deterministic time-varying signal $$s(t)$$, that is, $$B(t) = A(t) + s(t) ~~ \text{for all }t$$ and it is straightforward to show that $$C_B(\tau) = C_A(\tau)$$, that is, the two processes have the same autocovariance function. The notion of covariance-stationarity is obviously useful in treating topics such as noisy deterministic signals where the noise process is assumed to be wide-sense-stationary but the sum of the signal and the noise is not a wide-sense-stationary process; it is covariance-stationary.

## ------------

With that as prologue, let us consider the OP's

$$g_{1}(\tau)=\frac{\big\langle (A(t)-\left \langle A \right\rangle)(A(t+\tau)-\left \langle A \right\rangle ) \big\rangle}{\big\langle (A(t) - \left \langle A \right \rangle )^{2} \big \rangle}$$ $$g_{2}(\tau)=\frac{\big\langle A(t)A(t+\tau)\big\rangle}{\big\langle A(t)^{2} \big\rangle}$$

Assuming wide-sense-stationarity and interpreting $$\langle A\rangle$$ as $$\langle A(t)\rangle = \langle A(t+\tau)\rangle = \mu$$, the numerator of $$g_1(\tau)$$ is just the autocovariance function $$C_A(\tau)$$ while the denominator is just the variance $$\sigma^2$$ of the process. Since $$C_A(0) = \sigma^2$$,

$$g_1(\tau)$$ is just the correlation coefficient of the random variables $$A(t)$$ and $$A(t+\tau)$$. !!!

If one really wants to march to the beat of a different drummer and flout the conventions of this forum, one can call $$g_1(\tau)$$ the true autocorrelation function of the wide-sense-stationary process; it value is the correlation coefficient between variables spaced $$\tau$$ apart in time.

On the other hand, $$g_2(\tau)$$ is more problematic in that the numerator is the autocorrelation value $$R_A(\tau)$$ (in the usual dsp.SE sense) of the wide-sense-stationary random process and so has value $$C_A(\tau)+\mu^2$$. In particular, $$R_A(0) = \sigma^2+\mu^2$$. The denominator is just the mean-square value of the process (often called the power of the process) and has value $$R_A(0) = \sigma^2+\mu^2$$. Hence, $$g_2(\tau) = \frac{C_A(\tau)+\mu^2}{\sigma^2+\mu^2}$$ which has value $$1$$ at $$\tau=0$$. I don't remember seeing this formula previously but it may have its uses. There are more things in heaven and earth than are dreamt of in my philosophy.

Finally, turning to the OP's statement

My understanding is that the autocorrelation for the second equation will asymptotically go to zero if there is no correlation between the events in the time series

I don't think this is quite correct. As the above development shows, even if $$C_A(\tau)$$ converges to $$0$$ as $$\tau$$ increases without bound (not all processes have this property), $$g_2(\tau)$$ converges to $$\frac{\mu^2}{\sigma^2+\mu^2}$$ which equals $$0$$ only if $$\{A(t)\}$$ is a zero-mean wide-sense-stationary process. There exist, however, zero-mean wide-sense-stationary processes that have periodic autocorrelation functions and so $$C_A(\tau)$$ and $$R_A(\tau)$$ don't converge to a limit; they are periodic functions. See, for example, this answer of mine for a zero-mean wide-sense-stationary process with autocorrelation function $$\frac 12 \cos(\tau)$$.