explanation of correlation of stationary stochastic processes

Question

I have some doubts about correlation in stationary stochastic processes.

I know that the expectation of a random variable is $$E(x)=\int_{-\infty}^{+\infty} a f_x(a)da$$ and the mutual correlation of two variables is $E(x_1 x_2)$.

In two stationary stochastic processes the mutual correlation should be $E(x_1(t+\tau)x_2(t))$ now when and why I can translate this formula in: $$ \int_{-\infty}^{+\infty} x_1(t+\tau)x_2(\tau)d\tau $$ I have not understand where this last formula comes from.

Answer 1

You are mixing up two different notions.

Your random process is a collection of random variables $\{X(t)\colon -\infty < t < \infty\}$, one random variable for each time instant. The autocorrelation function of the process is $$R_X(t, \hat{t}) = E[X(t)X(\hat{t})], -\infty < t, \hat{t} < \infty$$ and is a two-dimensional function (meaning it has two arguments, and if you try to visualize it, you should imagine a surface above (and possibly below) the plane with coordinate axes $t$ and $\hat{t}$. Definitely above because $R_X(t,t) = E[(X(t))^2]\geq 0$ for all $t$ (and processes for which $R_X(t,t) = 0$ for all $t$ are kinda boring).

For a stationary process, $R(t,\hat{t})$ depends only on the difference $\hat{t}-t$ of the two arguments, and it is common practice to write it as a function of one variable $\tau = \hat{t}-t$. Thus, $$\mathcal R(\tau) = R(t,\hat{t}) = E[X(t)X(t+\tau)].$$

The other notion is that if the experiment is performed, then each random variable takes on a numerical value. Thus, $X(t)$ takes on a specific numerical value that we name as $x(t)$ while $X(\hat{t})$ takes on some other numerical value that we name as $x(\hat{t})$ and so on. Thus, we have a real-valued function (or signal) $x(t)$ (nothing random about that!) for which the autocorrelation function is defined as $$R_x(\tau) = \int_{-\infty}^\infty x(t)x(t+\tau)\,\mathrm dt.$$ Note that $x(t)$ is called a sample path or realization of the random process: it is what you observe on your oscilloscope if you lift up your head from the math a little bit and look at the real world.

It is every DSPer's fondest hope that $R_x(\tau)$ is the same function as $\mathcal R(\tau)$. Unfortunately, nothing in what has been said above justifies this hope. But, since people like to assume that it is indeed true that $R_x(\tau) = \mathcal R_X(\tau)$, mathematicians have come up with a notion called ergodicity which (for our purposes) says that for an ergodic process, the ensemble means (expectations) equal the corresponding time-averages. As a practical matter, all model-builders assume that the processes that they want to model are indeed both stationary and ergodic, and decide on what the mean and the autocorrelation function etc should be by making observations of actual sample paths etc.

For more than what you probably want to know about random processes, see this answer of mine, and for an example of a (boring) stationary random process that is not ergodic, see this other answer of mine.