# Where does the delta function come from if we derive autocorrelation directly?

I am reading a book "Creating Noise" written by Hollos & Hollos and have a question about the autocorrelation function of the Gaussain white noise when reading the following passage:

From the theorem that the autocorrelation and psd are Fourier transform pair and the fact that psd of Gaussian white noise is $\sigma^2$, it is obvious that the autocorrelation of Gaussian white noise has a delta function $\delta(\tau)$ as formulated in (6). So at $\tau=0$, what we have is a pulse of inifinite amplitude. But if we derive the autocorrelation directly, as does the book in the passage, we only have $R(0)=\sigma^2$ which is a finite number, instead of an infinite pulse. I believe the autocorrelation result having a delata function is correct, but the direct derivation of it looks correct too. Being not able to find out where the mistake is, I am curious where the delta function $\delta(\tau)$ comes from if we derive the autocorrelation $R(\tau)$ directly from its definition. I wish you can give me some help with it. Thanks a lot.

• The auto-correlation of random variable with itself is its variance. However, for the whole white noise process the auto-correlation tends towards infinity as you are considering the autocorrelation of infinitely many samples(or whatever unit) with each other. In other words, it looks like one of the definitions is the auto-correlation of the random variable, and one is for the process. – Dole Jul 2 '16 at 21:17
• So, can anyone show me a correct definition of autocorrelation of a stochastic process that can leads to a delta function, please? – user5280911 Jul 3 '16 at 0:34
• I've added some information to my answer to address the question in your comment. – Matt L. Jul 3 '16 at 12:15

The book is indeed not consistent. Continuous-time (zero-mean) white Gaussian noise has an autocorrelation function

$$R(\tau)=\sigma^2\delta(\tau)\tag{1}$$

and a constant power spectral density (PSD)

$$S(\omega)=\mathcal{F}\{R(\tau)\}=\sigma^2\tag{2}$$

The power (or variance) of such a process is

$$R(0)=\int_{-\infty}^{\infty}S(\omega)d\omega=\infty\tag{3}$$

So in the book the variable $\sigma^2$ denotes two different things: the finite value of the (constant) PSD, and the (infinite) variance of the process. But it can't be both at the same time. So the equation $R(0)=\sigma^2$ doesn't make sense, if at the same time we have $R(\tau)=\sigma^2\delta(\tau)$.

EDIT: In reaction to your comment concerning the definition of the auto-correlation function: the definition given in the book is correct (for real-valued wide-sense stationary processes):

$$R(\tau)=E[x(t)x(t-\tau)]\tag{4}$$

Since we know that $x(t)$ is white and zero-mean, we know that

$$R(\tau)=E[x(t)x(t-\tau)]=0,\quad\tau\neq 0\tag{5}$$

For $\tau=0$ we get

$$R(0)=E[x^2(t)]\tag{6}$$

which is the power of $x(t)$, and, since $E[x(t)]=0$, it also equals its variance. And now comes the error in the book: the authors use $\sigma^2$ to denote the variance: $$R(0)=\sigma^2\tag{7}$$ But then they claim

The autocorrelation function of white noise must therefore be a delta function. $$R(\tau)=\sigma^2\delta(\tau)\tag{8}$$

But this is inconsistent with Eq. $(7)$ (of this answer). Note that both $(7)$ and $(8)$ are correct, but together they are wrong because they use the same symbol $\sigma^2$ with a different meaning. In $(7)$, $\sigma^2$ is the variance of $x(t)$, whereas in $(8)$, $\sigma^2$ is the value of the (constant) PDS of $x(t)$.

In the given example, $x(t)$ is white, so its power (variance) is infinite. This means that the constant $\sigma^2$ in $(7)$ cannot be finite. On the other hand, the constant $\sigma^2$ in ($8$) is finite, because it is simply the value of the constant PSD of $x(t)$.

The fact that the auto-correlation function of a zero-mean white wide-sense stationary process must be a weighted delta impulse at $\tau=0$ is obvious from the requirement that the PSD must be constant. The auto-correlation is the inverse Fourier transform of the PSD, and the inverse Fourier transform of a constant is a Dirac delta impulse.

• I know simple gratitude isn't usually allowed here, but thanks so much for this answer. So many people keep giving obscure and incorrect answers to this question. – Arturo don Juan Nov 28 '17 at 20:53

Delta functions are weird things. They're a mathematical construct with no real physical analog. One source says "The delta function only makes sense as a mathematical object when it appears inside an integral."

So the answer to your question really is: Don't think of the delta function as having infinite amplitude, because this is not reality. It's only true as a limit, and only in an abstract sense. Check out this for more detail: https://en.wikipedia.org/wiki/Dirac_delta_function

The definition of variance of a random variable:

$$Var(X)=E[(X-E(X))^2]$$ Since the variable was defined to have mean zero: $$E(X)=0$$ Inputting that into the formula, you get: $$E(X^2)$$

The autocovariance is defined as:

$$E(X_tX_s)-u_tu_s=E(X^2),\forall t=s$$

So the variance = expected auto-correlation of the random variable with itself is simply $\sigma^2$. However, for the whole process the result is sum of all the random variables involved in the process. Summing over ALL t=s (for uncorrelated process) produces this result, as described by so called Bienaymé formula. The window for an infinite white noise process is infinite, so the autocorrelation would be (at time zero=with itself):

$$\sum_{i=1}^{\infty} Var(X_i)$$

(More technically accurately take the lim and also include negative infinity). So the autocorrelation of white noise process is infinite when evaluated with itself (at time zero). Note that since the random process is independent, at any other instant the auto-correlation function is zero. Mathematically:

$$E(X_tX_{t+s})=0, s \neq 0$$

The "s" can be read as the time index of the auto-correlation function.

Note also that autocorrelation is defined differently in statistics and DSP: https://en.wikipedia.org/wiki/Autocorrelation. In statistics IIRC, the mean is subtracted and the result normalized, this I believe would produce auto-correlation of 1. Of course the answer should be only read as a guess of what possibly could have been conveyed by the book.

• You are the only one who understand my question. Could you please elaborate in more detail the statement "However, for the whole process ... ... involved in the process" because I believe this is the key -- the definition of autocorrelation of a stochastic process. I think the definition should be the variance of only one variable at some time t because this is how Var(X) is defined. Why do we need to sum over ALL Gaussian variables at ALL time in the process? Could you please show me a reference to a definition of autocorrelation of a stochastic process defined in such a way? Thanks a lot. – user5280911 Jul 3 '16 at 0:11
• I just realized that that sentence is for explaining the result of $\sigma^2$. Please let me clarify my comment a little bit. Why "so the autocorrelation would be (that infinite sum of variances)"? Could you please show me a reference to a definition of autocorrelation defined in this way? – user5280911 Jul 3 '16 at 0:17
• To be specific, the infinite sum of $Var(X_i)$ is only the numerator of the definition of R(0). The denominator, N, of R(0) approaches infinity as does the numerator. As a result, the final result of R(0) is still the finite value $\sigma^2$. So, returning to my original question, where does the infinite delta function come from? – user5280911 Jul 3 '16 at 0:29
• @user5280911 Sure. Autocorrelation defined in DSP is not the same as variance, no division by N. This is easy to show in a graph, you multiply the values of one function with the second function and sum over all the samples (or integrate), then slide the function (value t=2 of AC function) and repeat. The expected value thereof is the same as summing over all the individual variances of the stochastic process (when no covariance, and u=0), since that too is $\sum E(X^2)$. For the random variable the process consists of, the expected autocorrelation is the same as variance (when u=0)... – Dole Jul 3 '16 at 7:03
• , the expected autocorrelation of the stochastic variable (single value of the function ^2), can be viewed as the variance of a specific sample of the function (sampled over and over infinitely (again, only when E(u)=0)) . So the expected autocorrelation of the random variable used in the process and the whole process itself are very different concepts. – Dole Jul 3 '16 at 7:44

It turns out that the book is true only for discrete-time white noise. For continuous-time white noise, that's another story.

Let me start from a brief explanation in discrete-time white noise. In this case, at each isolated time instance, the value of the stochastic process can be regarded safely as a random variable. Note that for discrete signals, the delta function $\delta(0)=1$, not infinity, and its (discrete-time) Fourier transform is constant 1 as we all know.

Now we come to continuous-time white noise. The key here is that, at any isolated time instance, the value of the stochastic process CAN NOT be regarded as a random variable. As a result, it is incorrect to copy concepts in elementary probability theory and say, as in the book, that $R(0)=E(X^2)=\sigma^2$ where $\sigma^2$ is assumed to be a finite real number, e.g., variance of a Gaussian distribution, because there is no random variable and its pdf at all! If we must denote $\sigma^2=E(X^2)$ for simplicity, we must have in mind that this $\sigma^2$ is actually an inifinite value according to the definition of continuous-time white noise, that is, it itself contains delta function $\delta(t)$. I got to this point only after reading wikipedia for white noise. I suggest you take a careful read of the section on continuous-time white noise too, then you'll understand what I say above. The key to my question, to repeat, is that at isolated time instance, the value of the continuous-time white noise is not a random variable any more, so we can't (mistakenly) apply our knowledge of elementary probability theory, as did the author of the book.

I think it is just a handful way to say

$$R(s,t)= E[X(s)X(t)] = \sigma^2$$ if $$s=t$$, otherwise $$0$$. If we express $$R(s,t)$$ as $$R(\tau)$$, then if $$\tau=0$$, $$R(\tau) = R(0) = \sigma^2$$, else if $$\tau \ne 0$$, $$R(\tau) = 0$$.

The conclusion is that we can express the formula in a more compact way by using the delta function $$\delta(n)$$.

• @Carlos Danger why is it downvoted? I answered the wrong question or I provide the wrong answer, I would like to know the answer of the problem too by the way. – Ive Xu Nov 20 '18 at 9:43
• I didn't downvote it – Robert L. Nov 20 '18 at 17:09