# Super basic questions on statistical process

Before starting: I am really a beginner in statistical process in time. I mainly do quantum information and while learning aspect of quantum noise I realized that I am actually too weak on basics of statistical process.

Thus, I would like to first tell you what I basically understand and that you confirm or correct me. Then I have some more specific questions.

## Allright, here is what (I think) I understood, and I would like to check

Let's consider a random process. I consider a random variable $$X$$. First, I can fix a given time $$t$$, and ask questions about the statistical properties of $$X(t)$$. For example its mean value $$$$, its standard deviation $$<(X(t)-)^2>$$, and I could imagine many other quantities. In principle without further assumptions I need a lot of information (an infinite one ?) to specify all the characteristics of the statistics of $$X(t)$$.

Furthermore: here it is what is happening at a given time $$t$$. Even if I fully described the statistics of $$X(t)$$ I would lack information. For example there might have some correlations between what happens at $$t$$ and what happens at $$t'$$. Thus I also need to specify some properties linking those different times. They are not given by what happens at a given time $$t$$.

An example I have in mind is a variable that verifies: $$X(t) \in [0,1]$$, uniformously distributed. But the correlations are such that $$X(t'>t)=X(t)$$. Then if I do a first experiment I will have $$X(t)=0.2$$ for example. And for all further times $$X(t')=0.2$$ as well because of correlations. If I re-run the experiment I can have $$X(t)=0.87$$ for example but again the values for further times will be the same as $$X(t)$$. Do you confirm my understanding ?

To specify what happens in time we thus need other information like autocorrelation functions: $$C(t,t')=$$. And probably many other quantities that I am not aware of.

## More specific questions

What is a rigorous definition of stationnary statistical process ? What I basically understand is that the statistical properties of $$X(t)$$ actually do not depend on $$t$$. And the statistical properties of $$X(t)X(t')$$ only depend on $$t-t'$$. But I guess there is a more precise definition than this.

In a paper I am reading, they assume a gaussian, stationnary process. They say that knowing the average value $$$$ and the autocorrelation function allows to fully specify the process. I do not understand this. I agree that it will fully describe the statistical behavior at a given time (because from autocorrelation function we have access to the variance in addition to the mean we already knew). But how is the autocorrelation function the only information we need to specify the statistical relationship between different times ?

Finally, basic question but... what is the name of the precise field that deals with all this ? It is not statistics (this is way too broad), I tried some keyword with "statistical noise" but I didn't find something good. Related: is there a good reference to really give the basics ? I don't think I need much.

• Some of your questions are answered here on dsp.SE. – Dilip Sarwate May 24 '20 at 14:20
• @DilipSarwate thank you very much, your answer is super great !! – StarBucK May 27 '20 at 9:37
• Thank you. I have edited the answer to add a few more details. If the revised version does not resolve all your questions, please consider editing your question to ask about only those matters that still remain unanswered. If everything you wanted to ask has been covered in my answer, please consider asking the moderators (click on the link flag below your question) to close your question. – Dilip Sarwate May 27 '20 at 16:30

For a Gaussian Random Process, Wide-Sense Stationarity implies Strict-Sense Stationarity as well.

That means, if mean $$\mu_X(t)$$ is independent of $$t$$ and is a constant and Auto-correlation function $$R_{X}(t, t+\tau)$$ is dependent only on time-shift $$\tau$$, then the Gaussian Process is actually Strict-Sense Stationary.

Why? Because a Gaussian Random Process, $$X(t)$$, is defined as that random process for which if we freeze time at $$n$$ instances to get random variables $$\{X(t_1), X(t_2), ..., X(t_n)\}$$, then these random variables are jointly Gaussian Distributed. So, we can say that if we have a Gaussian Random Process $$X(t)$$ and we freeze time at $$n$$ instances, $$t_1, t_2, ..., t_{n}$$, then we get a Jointly Gaussian Random vector $$\mathbf X$$ of $$n$$ dimension. And for a jointly Gaussian Random Vector, its joint Probability Density is fully determined by its Mean vector $$\mathbf \mu_{\mathbf X}$$ and Covariance Matrix $$\mathbf C$$, given by:

$$f_{\mathbf X}(\mathbf x) = \frac{1}{(2\pi)^{\frac{n}{2}}|C|^{\frac{1}{2}}}\cdot \exp(-\frac{1}{2}(\mathbf x - \mu_{\mathbf X})^TC^{-1}(\mathbf x - \mu_{\mathbf X}))$$

Also, note that the Auto-Correlation and Covariance Matrix $$\mathbf C$$ are related for a Wide-Sense Stationary Process, that, all the elements $$\mathbf C_{i,j}$$ can be computed from the Auto-Correlation function and constant mean of the WSS Random Process. $$\mathbf C_{i,j} = \mathbf E[(X(t_i) - \mu_X)(X(t_j)-\mu_X)] = \mathbf R_X(t_i-t_j) - \mu_X^2$$ Thus, if the mean of a Gaussian Random process is constant $$\mu_{X}$$ and Auto-Correlation is only dependent on time-shift $$\tau$$, then the Gaussian Random Process is Strict Sense Stationary, because we know all the joint Distributions completely for any $$n$$.This is not true for any other Random Process in general.

For detailed definition of Strict-Sense Stationary Random processes and Wide-Sense Stationary Random Process, see the rest of the explanation.

The way I visualize Random processes $$X(t, \zeta)$$ is: for every outcome $$\zeta$$ in a random experiment, the outcome $$\zeta$$ is mapped to a function $$x(t, \zeta)$$. This is like Random Variable concept. A random variable $$X$$ is a rule for assigning a value or set of values $$x(\zeta)$$ to each outcome $$\zeta$$ of a random experiment. Similarly, a Random process $$X(t)$$ is a rule of assigning a function $$x(t,\zeta)$$ to each outcome $$\zeta$$ of a random experiment. Three important intuitions are as follows:

1. If we freeze the outcome $$\zeta$$ of that random experiment, we will get a deterministic function of $$t$$, $$x(t)$$.

2. If we freeze $$t$$, then we get a Random Variable $$X(\zeta)$$.

3. If we freeze both $$t$$ and $$\zeta$$, we get a number $$x$$. $$X(t, \zeta) : ensemble \ of \ function \ of \ t \ which \ are \ uncertain$$

Consider a Random process $$X(t)$$ (dropped $$\zeta$$ to de-clutter) $$\forall \ t \in \mathbb R$$.

$$X(t)$$ will be called Strict-Sense Stationary(SSS) Process if its statistical properties are time-shift invariant. This means that the random processes $$X(t)$$ and $$X(t+\tau)$$ have the same statistics for all $$\tau \in \mathbb R$$.

What I mean by statistical properties is: If we freeze time at $$t$$, and get random variable $$x(t)$$ from the random process $$X(t)$$, then we can define $$f(x(t))$$ as the probability density function of $$x(t)$$. Similarly, we can define joint probability density function of $$n$$ such random variables $$x(t_1), x(t_2),..., x(t_n)$$ by fixing time at $$t_1, t_2,...,t_n$$, as $$f(x(t_1), x(t_2),...,x(t_n))$$. Then, Strict-Sense Stationary says that: $$f(x(t_1), x(t_2),...,x(t_n)) = f(x(t_1+\tau), x(t_2+\tau),...,x(t_n+\tau)), \ \forall \ \tau \in \mathbb R$$

We can put $$n=1$$ in the above expression to figure that, $$1^{st}$$ order probability density of SSS Random processes are independent of time $$t$$: $$f(x(t)) = f(x(t+\tau)) = f(x)$$

Similarly, if we put $$n=2$$, and look at the $$2^{nd}$$ order probability density $$f(x(t_1), x(t_2))$$ will also depend only on time-shift $$\tau$$ between $$t_2$$ and $$t_1$$. Since we can choose any value of $$\tau$$, consider the case that we choose $$\tau = -t_2$$ always, then: $$f(x(t_1), x(t_2)) = f(x(t_1-t_2), x(0))$$ The above expression says that joint density function of $$2$$ random variables $$x(t_1)$$ and $$x(t_2)$$ which we got by freezing time at $$t_1$$ and $$t_2$$, is always equal to the joint density function of $$2$$ random variables $$x(0)$$ and $$x(t_1-t_2)$$ which we get when we freeze time at $$0$$ and $$(t_1-t_2)$$. This means that $$2^{nd}$$ order density function is independent of time and only dependent on the time shift $$(t_1-t_2)$$.

Wide-Sense Stationary(WSS) Random Processes are those Random Processes which satify following $$2$$ requirements:

1. Mean of the Random Process, $$\mathbf E[X(t)]$$, is independent of time and is a constant. $$\mathbf E[X(t)] = \mu_X = c$$

2. Auto-correlation function of the Random Process, $$\mathbf R_X$$, is only dependent on time-difference $$\tau = t_1 - t_2$$ $$\mathbf E[x(t_1)x^*(t_2)] = \mathbf R_X(\tau)$$

For a WSS Process $$X(t)$$, the above requirement implies that the variance of the Random Process, $$\mathbf E[(X(t)-c)^2]$$, is also a constant. $$\mathbf E[(X(t)-c)^2] = \mathbf E[(X(t))^2] - c^2 = \mathbf R_X(0) - c^2$$

Note that there is no requirement on statistical properties of Wide Sense Stationary Random Processes.

The field of Stochastic Processes deals with these topics and a good starting reference would be Chapter: 9 of "Probability, Random Variables and Stochastic Processes, by A. Papoulis & S. Unnikrishna Pillai"

• This answer would be vastly improved if all the paragraphs except the last paragraph (with the reference to the Papoulis and Pillai book) were to be deleted. The earlier stuff is a mishmash of half-truths, imprecise or incorrect expressions (see, for example, the first displayed equations), and mis-understandings of the notions being discussed. – Dilip Sarwate May 25 '20 at 15:59
• @DilipSarwate Although your comment was not pointing specifically, what exactly is wrong with the expressions or explanations, I have tried improving my answer by adding more details, improving the language and giving more details on what I mean. Thanks for the review though. I hope I am more correct now than before. – DSP Rookie May 25 '20 at 17:05
• OK, the statement "Strict-Sense Stationary Random Processes are defined as those which are having time-invariant Joint PDF" is misleading if not downright wrong. In a strictly stationary process, the joint pdf $X(t_1), X(t_2)$ is not the same as the joint pdf of $X(t_3), X(t_4)$ unless it so happens that $t_1-t_2 = t_3-t_4$. You add the qualifier "Like all i.i.d. processes" which is a very special case. In your displayed equation which now has the $\,^2$ inserted (it didn't before you edited your answer), the quantity $\mu_X$ is undefined, and since the mean is $c$ (continued).... – Dilip Sarwate May 25 '20 at 19:28
• ...(continued) as per your first equation, the second expectation does not have value $\sigma_X^2$, the value is $\sigma_X^2 + (\mu_X-c)^2$. WSS processes need not be stationary to order $2$; an example of a WSS process which is not stationary to any order can be found in the answer linked to in my comment on the main question. In short, my recommendation is that you read that answer first, and then edit your own answer to address those and only those of the OP's questions that are not treated in that other answer. – Dilip Sarwate May 25 '20 at 19:36
• @DilipSarwate Yeah, that missing $^2$ was a typo which I corrected. And, $\mu_X = c$, which is clear from the first equation, because the point I am making is that mean is constant for WSS process. May be I went a little sloppy with the notations when I did not explicitly mention that $\mathbf EX[n] = \mu_X = c$. I will add this in my answer. But, thanks for the comments. I will go through the linked answer in your comment. – DSP Rookie May 25 '20 at 20:11