# What is the distinction between ergodic and stationary?

I have trouble distinguishing between these two concepts. This is my understanding so far.

A stationary process is a stochastic process whose statistical properties do not change with time. For a strict-sense stationary process, this means that its joint probability distribution is constant; for a wide-sense stationary process, this means that its 1st and 2nd moments are constant.

An ergodic process is one where its statistical properties, like variance, can be deduced from a sufficiently long sample. E.g., the sample mean converges to the true mean of the signal, if you average long enough.

Now, it seems to me that a signal would have to be stationary, in order to be ergodic.

• And what kinds of signals could be stationary, but not ergodic?
• If a signal has the same variance for all time, for example, how could the time-averaged variance not converge to the true value?
• So, what is the real distinction between these two concepts?
• Can you give me an example of a process that is stationary without being ergodic, or ergodic without being stationary?
• You might want to look at this answer to a related question. – Dilip Sarwate Jan 17 '12 at 21:31
• This lecture spells it literally that ergodic is a subset of stationary. I just cannot understand what Stationary Ergodic Process article is doing in Wikipedia? Does it mean that there is non-stationary ergodic process? – Val May 17 '14 at 16:32
• @Val I won't defend what Wikipedia says but will point out that the last part of my answer below contains an example of a WSS process that is not stationary and yet is ergodic. – Dilip Sarwate Jun 2 at 2:35

A random process is a collection of random variables, one for each time instant under consideration. Typically this may be continuous time ($$-\infty < t < \infty$$) or discrete time (all integers $$n$$, or all time instants $$nT$$ where $$T$$ is the sample interval).

• Stationarity refers to the distributions of the random variables. Specifically, in a stationary process, all the random variables have the same distribution function, and more generally, for every positive integer $$n$$ and $$n$$ time instants $$t_1, t_2, \ldots, t_n$$, the joint distribution of the $$n$$ random variables $$X(t_1), X(t_2), \cdots, X(t_n)$$ is the same as the joint distribution of $$X(t_1+\tau), X(t_2+\tau), \cdots, X(t_n+\tau)$$. That is, if we shift all time instants by $$\tau$$, the statistical description of the process does not change at all: the process is stationary.
• Ergodicity, on the other hand, doesn't look at statistical properties of the random variables but at the sample paths, i.e. what you observe physically. Referring back to the random variables, recall that random variables are mappings from a sample space to the real numbers; each outcome is mapped onto a real number, and different random variables will typically map any given outcome to different numbers. So, imagine that some higher being as performed the experiment which has resulted in an outcome $$\omega$$ in the sample space, and this outcome has been mapped onto (typically different) real numbers by all the random variables in the process: specifically, the random variable $$X(t)$$ has mapped $$\omega$$ to a real number we shall denote as $$x(t)$$. The numbers $$x(t)$$, regarded as a waveform, are the sample path corresponding to $$\omega$$, and different outcomes will give us different sample paths. Ergodicity then deals with properties of the sample paths and how these properties relate to the properties of the random variables comprising the random process.

Now, for a sample path $$x(t)$$ from a stationary process, we can compute the time average $$\bar{x} = \frac{1}{2T} \int_{-T}^T x(t) \,\mathrm dt$$ but, what does $$\bar{x}$$ have to do with $$\mu = E[X(t)]$$, the mean of the random process? (Note that it doesn't matter which value of $$t$$ we use; all the random variables have the same distribution and so have the same mean (if the mean exists)). As the OP says, the average value or DC component of a sample path converges to the mean value of the process if the sample path is observed long enough, provided the process is ergodic and stationary, etc. That is, ergodicity is what enables us to connect the results of the two calculations and to assert that $$\lim_{T\to \infty}\bar{x} = \lim_{T\to \infty}\frac{1}{2T} \int_{-T}^T x(t) \,\mathrm dt$$ equals
$$\mu = E[X(t)] = \int_{-\infty}^\infty f_X(u) \,\mathrm du.$$ A process for which such equality holds is said to be mean-ergodic, and a process is mean-ergodic if its autocovariance function $$C_X(\tau)$$ has the property: $$\lim_{T\to\infty}\frac{1}{2T}\int_{-T}^T C_X(\tau) \mathrm d\tau = 0.$$

Thus, not all stationary processes need be mean-ergodic. But there are other forms of ergodicity too. For example, for an autocovariance-ergodic process, the autocovariance function of a finite segment (say for $$t\in (-T, T)$$ of the sample path $$x(t)$$ converges to the autocovariance function $$C_X(\tau)$$ of the process as $$T\to \infty$$. A blanket statement that a process is ergodic might mean any of the various forms or it might mean a specific form; one just can't tell,

As an example of the difference between the two concepts, suppose that $$X(t) = Y$$ for all $$t$$ under consideration. Here $$Y$$ is a random variable. This is a stationary process: each $$X(t)$$ has the same distribution (namely, the distribution of $$Y$$), same mean $$E[X(t)] = E[Y]$$, same variance etc.; each $$X(t_1)$$ and $$X(t_2)$$ have the same joint distribution (though it is degenerate) and so on. But the process is not ergodic because each sample path is a constant. Specifically, if a trial of the experiment (as performed by you, or by a superior being) results in $$Y$$ having value $$\alpha$$, then the sample path of the random process that corresponds to this experimental outcome has value $$\alpha$$ for all $$t$$, and the DC value of the sample path is $$\alpha$$, not $$E[X(t)] = E[Y]$$, no matter how long you observe the (rather boring) sample path. In a parallel universe, the trial would result in $$Y = \beta$$ and the sample path in that universe would have value $$\beta$$ for all $$t$$. It is not easy to write mathematical specifications to exclude such trivialities from the class of stationary processes, and so this is a very minimal example of a stationary random process that is not ergodic.

Can there be a random process that is not stationary but is ergodic? Well, N0, not if by ergodic we mean ergodic in every possible way one can think of: for example, if we measure the fraction of time during which a long segment of the sample path $$x(t)$$ has value at most $$\alpha$$, this is a good estimate of $$P(X(t) \leq \alpha) = F_X(\alpha)$$, the value of the (common) CDF $$F_X$$ of the $$X(t)$$'s at $$\alpha$$ if the process is asumeed to be ergodic with respect to the distribution functions. But, we can have random processes that are not stationary but are nonetheless mean-ergodic and autocovariance-ergodic. For example, consider the process $$\{X(t)\colon X(t)= \cos (t + \Theta), -\infty < t < \infty\}$$ where $$\Theta$$ takes on four equally likely values $$0, \pi/2, \pi$$ and $$3\pi/2$$. Note that each $$X(t)$$ is a discrete random variable that, in general, takes on four equally likely values $$\cos(t), \cos(t+\pi/2)=-\sin(t), \cos(t+\pi) = -\cos(t)$$ and $$\cos(t+3\pi/2)=\sin(t)$$, It is easy to see that in general $$X(t)$$ and $$X(s)$$ have different distributions, and so the process is not even first-order stationary. On the other hand, $$E[X(t)] = \frac 14\cos(t)+ \frac 14(-\sin(t)) + \frac 14(-\cos(t))+\frac 14 \sin(t) = 0$$ for every $$t$$ while \begin{align} E[X(t)X(s)]&= \left.\left.\frac 14\right[\cos(t)\cos(s) + (-\cos(t))(-\cos(s)) + \sin(t)\sin(s) + (-\sin(t))(-\sin(s))\right]\\ &= \left.\left.\frac 12\right[\cos(t)\cos(s) + \sin(t)\sin(s)\right]\\ &= \frac 12 \cos(t-s). \end{align} In short, the process has zero mean and its autocorrelation (and autocovariance) function depends only on the time difference $$t-s$$, and so the process is wide sense stationary. But it is not first-order stationary and so cannot be stationary to higher orders either. Now, when the experiment is performed and the value of $$\Theta$$ is known, we get the sample function which clearly must be one of $$\pm \cos(t)$$ and $$\pm \sin(t)$$ which have DC value $$0$$ which equals $$0$$, and whose autocorrelation function is $$\frac 12 \cos(\tau)$$, same as $$R_X(\tau)$$, and so this process is mean-ergodic and autocorrelation-ergodic even though it is not stationary at all. In closing, I remark that the process is not ergodic with respect to the distribution function, that is, it cannot be said to be ergodic in all respects.

• I couldn't understand the example. If you say Y is a constant, then any path of x(t) is a constant. The mean of a constant is itself, hence E[X(t)] = E[Y] = Y. Unless I missed something. – Royi Jan 18 '12 at 17:20
• I added a few words to clarify the meaning. $Y$ is a random variable, not a constant. Its value on any trial of the experiment need not be the same as $E[Y]$. – Dilip Sarwate Jan 18 '12 at 17:40
• Now I see your intention. I think it is important to note that usually the meaning of ergodic process is that by observing a single path in time one could deduce some properties of the RV as a whole. As you showed, in the above example, each path is a constant, yet the RV is a something with probability properties. Thanks for clarification. – Royi Jan 18 '12 at 17:48
• If a signal is ergodic, meaning the time average converges to the ensemble average, but the various $X$'s have different means because the process is not stationary, what is the definition of the ensemble average to which the time average is converging? – Dilip Sarwate Jan 23 '12 at 21:35
• @Matt In the solution of the book "communication systems" simon haykin writes that "for a random process to be ergodic it has to be stationary" – Roney Island Aug 17 '12 at 4:46

Let us consider a hypothetical random process where the sample functions are DC values and are different from each other:

X1(t) = constant= mean of X1(t)

X2(t) = constant= mean of X2(t)

The temporal mean of $X_1(t)$ and $X_2(t)$ are constant but not equal. if my process is stationary $X(t_1)$ and $X(t_2)$ are equal and RVs (refer Dilip's answer)

So ensemble mean of $X(t)$ is constant.

This ensemble mean is certainly not equal to the temporal mean of $X_1(t)$ and $X_2(t)$ (they themselves are not equal). This can be called a stationary but not an ergodic process.

In contrast , $X(t)=Acos(\omega t+\theta)$ where $\theta$ is an RV is ergodic.

I hope this video (from the Florida Institute of Technology. Titled "what is wide sense staionary, strict sense, ergodic signals" by Dr. Ivica Kostanic in his Communications Theory class) from 16:55 could clear your doubts

• Welcome to the DSP.SE! I would suggest you add the name and some descriptions on the video in case it is removed someday and the link is invalid. Thank you. – lennon310 Mar 7 '14 at 13:38

An ergodic process is a process for which you can substitute the ergodic mean for the temporal mean.

The real mean, variance, etc... are defined by following a process over time and averaging, etc... For instance, if you want to know the mean of my size, you would have to average it from when I am born to when I die. Obviously the later example is not a stationary process.

The ergodic mean would be if instead of following my size over time, you would freeze time, and take the mean over a sample of different individual humans. There is no reason for these two means to be the same, so the process of my size is not ergodic.

That is a bad example, but it get more important if you consider the simple case of a gas at equilibrium. For instance the mean squared velocity is noted $\bar{V^2}$ (mean over time), but is often calculated by taking the ensemble mean $\left< V^2 \right>$: the mean of the square velocity of all molecules of the gas at an instant $t$.

Most thermodynamics theorems requires to use $\bar{V^2}$, but it is more easy to compute and use $\left< V^2 \right>$. The ergodic hypothesis is the hypothesis stating that it is right to substitute one for the other. An ergodic process is a process for which the ergodic hypothesis is true.

The ergodic hypothesis is false in the general case.

• I don't understand this answer. The process of Jolow's size is neither stationary nor ergodic whereas the OP was wondering if there can be a stationary process that is not ergodic. Is the answer essentially that in general the ergodic hypothesis is false and it is (sort of) universally true that the sample mean is different from the ensemble mean, just get used to it and live with it? – Dilip Sarwate Jan 17 '12 at 19:09
• @DilipSarwate: after re-reading, it is a bad answer that does not answer the question, and I am considering deleting it. I was reminding my thermodynamics lectures, whereas the question was more about statistics... – Jean-Yves Jan 17 '12 at 20:00
• @DilipSarwate what is Jolow's size? – Roney Island Aug 17 '12 at 5:01
• @MichaelCorleone I do not remember what the reference to Jolow means. My guess is that Jean-Yves posted his answer under the nom-de-plume Jolow and I used that name in my answer, and that he has since then decided to use Jean-Yves as his user name on stackexchange. Such name changes are reflected on what is displayed on the screen but are not recorded as an edit of the answer. – Dilip Sarwate Aug 17 '12 at 13:29
• @DilipSarwate: you are right indeed. Jolow is just my nickname. – Jean-Yves Aug 18 '12 at 6:28

For an example of the opposite case (i.e., a random process that is ergodic but not stationary), consider a white noise process that is amplitude modulated by a deterministic square wave. The time average of of every sample function is equal to zero, as is the ensemble average over all time. So the process is ergodic. However, the variance of any individual sample function shows the original square wave dependence on time, so the process is not stationary.

This particular example is wide-sense stationary, but one can concoct related examples that are still ergodic but not even wide-sense stationary.

as I underestand, example below shows a ergodic and stationary process

 X1 X2 X3  | mean var ...
1  2  3   | 2    1
2  3  1   | 2
3  1  2   | 2
----------


mean 2 2 2 var 1

because the mean and variance of every column is constant along the time and mean and variance of every row is constant along the time