# Understanding the definition of mean/autocorrelation

I was studying about the definitions of mean, expected value and autocorrelation. I wanted to verify my understanding the evaluation of mean, expected value and autocorrelation. At the same time to verify the definition of autocorrelation as the convolution of the signal with this delayed copy mathematically which I was not able to.

Here is my analysis:

Basically, I see mean as an average of all values in a population. For example, consider an experiment where we measure the outcome of the random variable that can go anywhere between $[1,100]$. By performing the above experiment $50$ times, we get $50$ outcomes:$a_1, a_2, \ldots, a_{50}$ which has values ranging from $1$ to $100$.

The mean of the above random variable is given by: $$\mu = \frac{a_1 + a_2 +\ldots + a_{50}}{50}$$

The expected value is given by: $$E(x)=(1*\textrm{probability of getting 1})+(2*\textrm{probability of getting 2})+\ldots+(100*\textrm{probability of getting n})$$

which can also be represented as: $$E(x)=\prod_{k=1}^{100} kP_k$$

where $P_k$ is the probability of getting $k$. Now taking the actual definition of an expected value which is: $$E(X)=\int_{-\infty}^{\infty} xF_x(X)dx$$

The above seems somewhat satisfying as we could relate the above with the example - taking $x$ as one of the outcome and $F_x(X)$ as the probabilities (actually it's a probability density function to be accurate).

For a random process, the above expands to (at time $t_1$): $$E(X_{t_1})=\int_{-\infty}^{\infty} x_{t_1}F_{X_{t_1}}(x_{t_1})dx$$ Similarly, expected value can be computed at several time instances since the random variable changes every time instant.

Now, lets take a Strict-Sense Stationary Random Process where expected value (mean) is constant and autocorrelation depends up on the time difference. So, the definition of Autocorrelation is given as per wikipedia:

$$R_x(\tau)=\frac{E\left[\left(X(t_1) - \mu\right)\left(X(t_1+\tau) - \mu\right)\right]}{\sigma_{t_1}\sigma_{t_2}}$$

The above is with Normalization. Taking out the normalization, we have:

$$R_x(\tau)=E\left[X(t_1)X(t_1+\tau)\right]$$

Till now it's pretty clear. Now applying the definition of an expected value in the above expression: $$R_x(\tau)=E\left[X(t_1)X(t_1+\tau)\right]=\int_{-\infty}^{\infty} x_{t_1}F_{x_{t_1}}(X_{t_1})x_{t_2}F_{x_{t_2}}(X_{t_2})dx\quad\text{where}\quad t_2 = t_1+\tau$$

But in Wikipedia, it's given as:

$$R_x(\tau)=E\left[X(t_1)X(t_1+\tau)\right]=\int_{-\infty}^{\infty} x(t)x^*(t-\tau)dt$$

This is where I am stuck at - I am not able to match the last two expressions. I am very well aware that autocorrelation of a random process is convolution with itself at a time lag of $\tau$ but I am not able to relate the expressions mathematically. I am stuck here for several days.

• Your definition of auto-correlation isn't correct, you have to use joint probability which in general isn't the product of marginal probabilities. Jun 4 '17 at 12:31
• @MohammadMohammadi So, you mean this definition?: latex.codecogs.com/… Jun 4 '17 at 12:39
• Yes, you have to use joint probability. But this correction still won't relate those two definitions. Jun 4 '17 at 12:42
• @MohammadMohammadi Yeah. Thats what I am wondering... But if we take the joint probability and even divide this definition latex.codecogs.com/… by twice the total time period T , then it would make some sense (like taking the values and averaging over a time period T - since there are two random variables - 2 times T). Jun 4 '17 at 12:45
• Still it has some problem, it has to be joint probability and also the integral would be the double integral over two random variables. But the more important problem is that you are trying to show that these two equations are equal, but they aren't. auto-correlation over time is an estimator of the ensemble auto-correlation for ergodic processes which converge to the auto-correlation in Mean Square sense or in other word expectation of squared difference of your estimator and the ensemble auto-correlation would goes to zero. Jun 4 '17 at 20:03

The definition of the autocorrelation function $R_x(\tau)$ depends on the nature of your $x$.

• If $x$ is a deterministic signal with finite energy then: $$R_x(\tau)=\int_{-\infty}^{+\infty}x(t)x^*(t-\tau)dt$$

• If $x$ is a deterministic signal with finite average power$^{(1)}$ then: $$R_x(\tau)=\lim_{T\to+\infty}\frac{1}{T}\int_{-T/2}^{+T/2}x(t)x^*(t-\tau)dt$$

• If $X$ is a (real-valued) random process then: $$R_X(t_1,t_2)=E[X(t_1)X(t_2)] = \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}u_1u_2p_X(u_1,u_2)du_1du_2,$$

where $X(t_1)$ and $X(t_2)$ are two random variables (the realizations of the random process $X$ at times $t_1$ and $t_2$, respectively), $u_1$, $u_2$ are their respective values and $p_X(u_1,u_2)$ is the joint probability density function of these two random variables. This statistical autocorrelation becomes only dependent on $\tau = t_1-t_2$ if the random process is stationary and we can write: $$R_X(\tau)=E[X(t)X(t-\tau)].$$

If, furthermore, the random process is ergodic then the statistical autocorrelation equals the temporal autocorrelation, i.e:

$$E[X(t)X(t-\tau)]= \lim_{T\to+\infty}\frac{1}{T}\int_{-T/2}^{+T/2}x(t)x(t-\tau)dt^{(2)}$$

$^{(1)}$: Signals with finite energy are also known as energy signals whereas signals with finite average power are also known as power signals. Moreover, the average power of an energy signal is zero and the energy of a power signal is infinite.

$^{(2)}$A random signal is by convention considered as a power signal.

• You define the autocorrelation in general for stochastic signals as an expectation (ensemble average), and say that this equals temporal average in the special case of ergodic processes. Can you tell me why for deterministic signals then you go straight to the temporal average for your definition, and not ensemble average? Apr 8 '20 at 12:25
• @teeeeee An ensemble average is taken, at some time instant say $t_1$, by averaging the outcomes of different experiments (or realizations) of a random process see page 175 here. By definition, a deterministic signal is nonrandom. So, you only get a single deterministic outcome. So, you can only take an average over $t$ for $x(t)$, i.e., what you called temporal average. Apr 10 '20 at 2:46
• I understand that, but since you have taken an average over time does that mean that the expression is only valid for ergodic signal? Or is it correct to say that all deterministic signals are ergodic anyway? Apr 10 '20 at 9:25
• @teeeeee, It doesn't actually make sense to talk about ergodicity for deterministic signals. Ergodicity basically says that assuming you have a random process ($X(t,\omega)$ where $t$ is time and $\omega$ outcomes) then taking a temporal average of a single outcome $\omega_k$ over variable $t$ (i.e., average of $X(t, \omega_k)$ over $t$) is equivalent to taking an ensemble average over the outcome variable $\omega$ of this random process considered at some time instant $t_k$ (i.e., average of $X(t_k, \omega)$ over $\omega$). How do you average outcomes when you don't have them? Apr 10 '20 at 13:17

In addition to stationarity, your process must be ergodic to relate these two definitions.

Ergodicity tells us joint probability of your signal's value at two instant of time (which only depend on their time difference) is equal to the number of instants of time which those combination with the same delay appeared in your signal divided by the total time.