Where does the delta function come from if we derive autocorrelation directly?

Question

I am reading a book "Creating Noise" written by Hollos & Hollos and have a question about the autocorrelation function of the Gaussain white noise when reading the following passage:

From the theorem that the autocorrelation and psd are Fourier transform pair and the fact that psd of Gaussian white noise is $\sigma^2$, it is obvious that the autocorrelation of Gaussian white noise has a delta function $\delta(\tau)$ as formulated in (6). So at $\tau=0$, what we have is a pulse of inifinite amplitude. But if we derive the autocorrelation directly, as does the book in the passage, we only have $R(0)=\sigma^2$ which is a finite number, instead of an infinite pulse. I believe the autocorrelation result having a delata function is correct, but the direct derivation of it looks correct too. Being not able to find out where the mistake is, I am curious where the delta function $\delta(\tau)$ comes from if we derive the autocorrelation $R(\tau)$ directly from its definition. I wish you can give me some help with it. Thanks a lot.

Answer 1

The book is indeed not consistent. Continuous-time (zero-mean) white Gaussian noise has an autocorrelation function

$$R(\tau)=\sigma^2\delta(\tau)\tag{1}$$

and a constant power spectral density (PSD)

$$S(\omega)=\mathcal{F}\{R(\tau)\}=\sigma^2\tag{2}$$

The power (or variance) of such a process is

$$R(0)=\int_{-\infty}^{\infty}S(\omega)d\omega=\infty\tag{3}$$

So in the book the variable $\sigma^2$ denotes two different things: the finite value of the (constant) PSD, and the (infinite) variance of the process. But it can't be both at the same time. So the equation $R(0)=\sigma^2$ doesn't make sense, if at the same time we have $R(\tau)=\sigma^2\delta(\tau)$.

---

EDIT: In reaction to your comment concerning the definition of the auto-correlation function: the definition given in the book is correct (for real-valued wide-sense stationary processes):

$$R(\tau)=E[x(t)x(t-\tau)]\tag{4}$$

Since we know that $x(t)$ is white and zero-mean, we know that

$$R(\tau)=E[x(t)x(t-\tau)]=0,\quad\tau\neq 0\tag{5}$$

For $\tau=0$ we get

$$R(0)=E[x^2(t)]\tag{6}$$

which is the power of $x(t)$, and, since $E[x(t)]=0$, it also equals its variance. And now comes the error in the book: the authors use $\sigma^2$ to denote the variance: $$R(0)=\sigma^2\tag{7}$$ But then they claim

The autocorrelation function of white noise must therefore be a delta function. $$R(\tau)=\sigma^2\delta(\tau)\tag{8}$$

But this is inconsistent with Eq. $(7)$ (of this answer). Note that both $(7)$ and $(8)$ are correct, but together they are wrong because they use the same symbol $\sigma^2$ with a different meaning. In $(7)$, $\sigma^2$ is the variance of $x(t)$, whereas in $(8)$, $\sigma^2$ is the value of the (constant) PDS of $x(t)$.

In the given example, $x(t)$ is white, so its power (variance) is infinite. This means that the constant $\sigma^2$ in $(7)$ cannot be finite. On the other hand, the constant $\sigma^2$ in ($8$) is finite, because it is simply the value of the constant PSD of $x(t)$.

The fact that the auto-correlation function of a zero-mean white wide-sense stationary process must be a weighted delta impulse at $\tau=0$ is obvious from the requirement that the PSD must be constant. The auto-correlation is the inverse Fourier transform of the PSD, and the inverse Fourier transform of a constant is a Dirac delta impulse.

Answer 2

Delta functions are weird things. They're a mathematical construct with no real physical analog. One source says "The delta function only makes sense as a mathematical object when it appears inside an integral."

So the answer to your question really is: Don't think of the delta function as having infinite amplitude, because this is not reality. It's only true as a limit, and only in an abstract sense. Check out this for more detail: https://en.wikipedia.org/wiki/Dirac_delta_function

Answer 3

The definition of variance of a random variable:

$$Var(X)=E[(X-E(X))^2]$$ Since the variable was defined to have mean zero: $$E(X)=0$$ Inputting that into the formula, you get: $$E(X^2)$$

The autocovariance is defined as:

$$E(X_tX_s)-u_tu_s=E(X^2),\forall t=s$$

So the variance = expected auto-correlation of the random variable with itself is simply $\sigma^2$. However, for the whole process the result is sum of all the random variables involved in the process. Summing over ALL t=s (for uncorrelated process) produces this result, as described by so called Bienaymé formula. The window for an infinite white noise process is infinite, so the autocorrelation would be (at time zero=with itself):

$$\sum_{i=1}^{\infty} Var(X_i) $$

(More technically accurately take the lim and also include negative infinity). So the autocorrelation of white noise process is infinite when evaluated with itself (at time zero). Note that since the random process is independent, at any other instant the auto-correlation function is zero. Mathematically:

$$E(X_tX_{t+s})=0, s \neq 0$$

The "s" can be read as the time index of the auto-correlation function.

Note also that autocorrelation is defined differently in statistics and DSP: https://en.wikipedia.org/wiki/Autocorrelation. In statistics IIRC, the mean is subtracted and the result normalized, this I believe would produce auto-correlation of 1. Of course the answer should be only read as a guess of what possibly could have been conveyed by the book.

Answer 4

It turns out that the book is true only for discrete-time white noise. For continuous-time white noise, that's another story.

Let me start from a brief explanation in discrete-time white noise. In this case, at each isolated time instance, the value of the stochastic process can be regarded safely as a random variable. Note that for discrete signals, the delta function $\delta(0)=1$, not infinity, and its (discrete-time) Fourier transform is constant 1 as we all know.

Now we come to continuous-time white noise. The key here is that, at any isolated time instance, the value of the stochastic process CAN NOT be regarded as a random variable. As a result, it is incorrect to copy concepts in elementary probability theory and say, as in the book, that $R(0)=E(X^2)=\sigma^2$ where $\sigma^2$ is assumed to be a finite real number, e.g., variance of a Gaussian distribution, because there is no random variable and its pdf at all! If we must denote $\sigma^2=E(X^2)$ for simplicity, we must have in mind that this $\sigma^2$ is actually an inifinite value according to the definition of continuous-time white noise, that is, it itself contains delta function $\delta(t)$. I got to this point only after reading wikipedia for white noise. I suggest you take a careful read of the section on continuous-time white noise too, then you'll understand what I say above. The key to my question, to repeat, is that at isolated time instance, the value of the continuous-time white noise is not a random variable any more, so we can't (mistakenly) apply our knowledge of elementary probability theory, as did the author of the book.

Answer 5

I think it is just a handful way to say

$R(s,t)= E[X(s)X(t)] = \sigma^2$ if $s=t$, otherwise $0$. If we express $R(s,t)$ as $R(\tau)$, then if $\tau=0$, $R(\tau) = R(0) = \sigma^2$, else if $\tau \ne 0$, $R(\tau) = 0$.

The conclusion is that we can express the formula in a more compact way by using the delta function $\delta(n)$.