Super basic questions on statistical process

Question

Before starting: I am really a beginner in statistical process in time. I mainly do quantum information and while learning aspect of quantum noise I realized that I am actually too weak on basics of statistical process.

Thus, I would like to first tell you what I basically understand and that you confirm or correct me. Then I have some more specific questions.

Allright, here is what (I think) I understood, and I would like to check

Let's consider a random process. I consider a random variable $X$. First, I can fix a given time $t$, and ask questions about the statistical properties of $X(t)$. For example its mean value $<X(t)>$, its standard deviation $<(X(t)-<X(t)>)^2>$, and I could imagine many other quantities. In principle without further assumptions I need a lot of information (an infinite one ?) to specify all the characteristics of the statistics of $X(t)$.

Furthermore: here it is what is happening at a given time $t$. Even if I fully described the statistics of $X(t)$ I would lack information. For example there might have some correlations between what happens at $t$ and what happens at $t'$. Thus I also need to specify some properties linking those different times. They are not given by what happens at a given time $t$.

An example I have in mind is a variable that verifies: $X(t) \in [0,1]$, uniformously distributed. But the correlations are such that $X(t'>t)=X(t)$. Then if I do a first experiment I will have $X(t)=0.2$ for example. And for all further times $X(t')=0.2$ as well because of correlations. If I re-run the experiment I can have $X(t)=0.87$ for example but again the values for further times will be the same as $X(t)$. Do you confirm my understanding ?

To specify what happens in time we thus need other information like autocorrelation functions: $C(t,t')=<X(t)X(t')>$. And probably many other quantities that I am not aware of.

More specific questions

What is a rigorous definition of stationnary statistical process ? What I basically understand is that the statistical properties of $X(t)$ actually do not depend on $t$. And the statistical properties of $X(t)X(t')$ only depend on $t-t'$. But I guess there is a more precise definition than this.

In a paper I am reading, they assume a gaussian, stationnary process. They say that knowing the average value $<X>$ and the autocorrelation function allows to fully specify the process. I do not understand this. I agree that it will fully describe the statistical behavior at a given time (because from autocorrelation function we have access to the variance in addition to the mean we already knew). But how is the autocorrelation function the only information we need to specify the statistical relationship between different times ?

Finally, basic question but... what is the name of the precise field that deals with all this ? It is not statistics (this is way too broad), I tried some keyword with "statistical noise" but I didn't find something good. Related: is there a good reference to really give the basics ? I don't think I need much.

Answer 1

Answer to your question related to Gaussian Random Process :

For a Gaussian Random Process, Wide-Sense Stationarity implies Strict-Sense Stationarity as well.

That means, if mean $\mu_X(t)$ is independent of $t$ and is a constant and Auto-correlation function $R_{X}(t, t+\tau)$ is dependent only on time-shift $\tau$, then the Gaussian Process is actually Strict-Sense Stationary.

Why? Because a Gaussian Random Process, $X(t)$, is defined as that random process for which if we freeze time at $n$ instances to get random variables $\{X(t_1), X(t_2), ..., X(t_n)\}$, then these random variables are jointly Gaussian Distributed. So, we can say that if we have a Gaussian Random Process $X(t)$ and we freeze time at $n$ instances, $t_1, t_2, ..., t_{n}$, then we get a Jointly Gaussian Random vector $\mathbf X$ of $n$ dimension. And for a jointly Gaussian Random Vector, its joint Probability Density is fully determined by its Mean vector $\mathbf \mu_{\mathbf X}$ and Covariance Matrix $\mathbf C$, given by:

$$f_{\mathbf X}(\mathbf x) = \frac{1}{(2\pi)^{\frac{n}{2}}|C|^{\frac{1}{2}}}\cdot \exp(-\frac{1}{2}(\mathbf x - \mu_{\mathbf X})^TC^{-1}(\mathbf x - \mu_{\mathbf X}))$$

Also, note that the Auto-Correlation and Covariance Matrix $\mathbf C$ are related for a Wide-Sense Stationary Process, that, all the elements $\mathbf C_{i,j}$ can be computed from the Auto-Correlation function and constant mean of the WSS Random Process. $$\mathbf C_{i,j} = \mathbf E[(X(t_i) - \mu_X)(X(t_j)-\mu_X)] = \mathbf R_X(t_i-t_j) - \mu_X^2$$ Thus, if the mean of a Gaussian Random process is constant $\mu_{X}$ and Auto-Correlation is only dependent on time-shift $\tau$, then the Gaussian Random Process is Strict Sense Stationary, because we know all the joint Distributions completely for any $n$.This is not true for any other Random Process in general.

For detailed definition of Strict-Sense Stationary Random processes and Wide-Sense Stationary Random Process, see the rest of the explanation.

The way I visualize Random processes $X(t, \zeta)$ is: for every outcome $\zeta$ in a random experiment, the outcome $\zeta$ is mapped to a function $x(t, \zeta)$. This is like Random Variable concept. A random variable $X$ is a rule for assigning a value or set of values $x(\zeta)$ to each outcome $\zeta$ of a random experiment. Similarly, a Random process $X(t)$ is a rule of assigning a function $x(t,\zeta)$ to each outcome $\zeta$ of a random experiment. Three important intuitions are as follows:

1. If we freeze the outcome $\zeta$ of that random experiment, we will get a deterministic function of $t$, $x(t)$.

2. If we freeze $t$, then we get a Random Variable $X(\zeta)$.

3. If we freeze both $t$ and $\zeta$, we get a number $x$. $$X(t, \zeta) : ensemble \ of \ function \ of \ t \ which \ are \ uncertain$$

Consider a Random process $X(t)$ (dropped $\zeta$ to de-clutter) $\forall \ t \in \mathbb R$.

$X(t)$ will be called Strict-Sense Stationary(SSS) Process if its statistical properties are time-shift invariant. This means that the random processes $X(t)$ and $X(t+\tau)$ have the same statistics for all $\tau \in \mathbb R$.

What I mean by statistical properties is: If we freeze time at $t$, and get random variable $x(t)$ from the random process $X(t)$, then we can define $f(x(t))$ as the probability density function of $x(t)$. Similarly, we can define joint probability density function of $n$ such random variables $x(t_1), x(t_2),..., x(t_n)$ by fixing time at $t_1, t_2,...,t_n$, as $f(x(t_1), x(t_2),...,x(t_n))$. Then, Strict-Sense Stationary says that: $$f(x(t_1), x(t_2),...,x(t_n)) = f(x(t_1+\tau), x(t_2+\tau),...,x(t_n+\tau)), \ \forall \ \tau \in \mathbb R$$

We can put $n=1$ in the above expression to figure that, $1^{st}$ order probability density of SSS Random processes are independent of time $t$: $$f(x(t)) = f(x(t+\tau)) = f(x)$$

Similarly, if we put $n=2$, and look at the $2^{nd}$ order probability density $f(x(t_1), x(t_2))$ will also depend only on time-shift $\tau$ between $t_2$ and $t_1$. Since we can choose any value of $\tau$, consider the case that we choose $\tau = -t_2$ always, then: $$f(x(t_1), x(t_2)) = f(x(t_1-t_2), x(0))$$ The above expression says that joint density function of $2$ random variables $x(t_1)$ and $x(t_2)$ which we got by freezing time at $t_1$ and $t_2$, is always equal to the joint density function of $2$ random variables $x(0)$ and $x(t_1-t_2)$ which we get when we freeze time at $0$ and $(t_1-t_2)$. This means that $2^{nd}$ order density function is independent of time and only dependent on the time shift $(t_1-t_2)$.

Wide-Sense Stationary(WSS) Random Processes are those Random Processes which satify following $2$ requirements:

1. Mean of the Random Process, $\mathbf E[X(t)]$, is independent of time and is a constant. $$\mathbf E[X(t)] = \mu_X = c$$

2. Auto-correlation function of the Random Process, $\mathbf R_X$, is only dependent on time-difference $\tau = t_1 - t_2$ $$\mathbf E[x(t_1)x^*(t_2)] = \mathbf R_X(\tau)$$

For a WSS Process $X(t)$, the above requirement implies that the variance of the Random Process, $\mathbf E[(X(t)-c)^2]$, is also a constant. $$\mathbf E[(X(t)-c)^2] = \mathbf E[(X(t))^2] - c^2 = \mathbf R_X(0) - c^2$$

Note that there is no requirement on statistical properties of Wide Sense Stationary Random Processes.

The field of Stochastic Processes deals with these topics and a good starting reference would be Chapter: 9 of "Probability, Random Variables and Stochastic Processes, by A. Papoulis & S. Unnikrishna Pillai"