Determining Loss of Information by Taking Average (Mean of Signal)

Question

So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am also curious if we also take the second moment (variance) of the same signal, how much information do we gain by using the variance along with the mean? In other words:

$$H_{\textrm{Mean}}(\Sigma_0^n x_iP_i)=\ ?$$ $$H_{\textrm{Variance}}(\Sigma_0^n (x_i-\mu )^2)=\ ?$$

Motivation: I just want to know how much information is dropped if we simply drop variace term and use just mean instead. I believe this concept cant be expanded to also consider information in third and fourth moments.

Answer 1

We can't say from the information you're giving alone. We need information on how the individual samples relate to each other!

Remember, the information of the sequence $(X)_{i=1}^N$ of $N$ random variables $X_i, \, i=1, \ldots, N$ is:

\begin{align} I(X=x) &= I((X_1=x_1, X_2=x_2, \ldots, X_N=x_N))\\ &\text{shorthand: }P(X_i=x_i)=:P(x_i)\\ &= I(x_1) + I((x_2, \ldots, x_N)|x_1)\\ &= I(x_1) + I(x_2|x_1)+I((x_3, \ldots, x_N)|x_1,x_2)\\ &= I(x_1) + \sum_{n=2}^N I(x_n|x_1,\ldots x_{n-1}) \tag 1\label{eq1}\\ &= -\log_2(P(x_1))- \underbrace{\sum_{n=2}^N\log_2(P(x_n|x_1,\ldots x_{n-1}))}_{:=v}\tag2\label{eq2} \end{align}

You'll notice when considiering $v$ that all these $P(x_n|x_1,\ldots x_{n-1})$ collapse to $P(x_n)$ if, and only if, the $X_n$ are independent.

In that case, $I(X)$ simply becomes $\sum I(X_n)$. For all other cases, that sum is the upper bound for what the overall information can be, but doesn't necessarily achieve that.

Let's construct an example where all $X_N$ are Gaussian:

$X_1\sim \mathcal N(0;\sigma^2)$ is drawn from a Gaussian source.

For all $N\ge i> 1$, let $X_i= i X_{i-1}$. Then, directly, $X_2\sim \mathcal N(0; 4\sigma^2)$, and generally $X_i\sim\mathcal N(0;i^2 \sigma^2)$.

All these variables are Gaussian, but they are not independent. Let's use that with $\eqref{eq1}$:

\begin{align} I(X=x) &= I((X_1=x_1, X_2=x_2, \ldots, X_N=x_N))\\ &= -\log_2(P(x_1))- \sum_{n=2}^N\log_2(P(x_n|x_1,\ldots x_{n-1}))\\ &= I(x_1) + \sum_{n=2}^N I(x_n|x_1,\ldots x_{n-1}) \label{eq3}\tag3\\ \end{align}

but what is e.g. $I(x_2|x_1)$? Well, it's 0, because there's no information in $x_2$ that we didn't have already through observing $x_1$! (You can also prove that formalistically by using the conditional entropy formula. But I found this more intuitive.)

The same is through for every other element of $v$.

What happens now if you take the mean of that series? Well, you get another Gaussian random variable, sure, but it has exactly the same information as the whole sequence: From the mean alone, you can trivially find the original $x_1$:

$$\bar X = \frac1N\sum_{n=1}^N X_n = \frac1N\sum_{n=1}^N n X_1=\frac1N X_1\sum_{n=1}^N n = C X_1,\;\; C\,\text{const.}$$

and therefore, the entropy of $\bar X$ is the entropy of $X$.

So, nothing's lost.

However, if all the $X_i$ are independent, but follow the same Gaussian distribution, then the mean $\bar X$ is a Gaussian variable of variance $\frac{\sigma^2}{N}$, and you lose the difference between that and $N$ times the entropy of a $\sigma^2$-variance random variable.

So, the answer to your question is in what you don't tell us:

Entropies of sources depend heavily on the conditional probabilities of their words; you need to describe a random variable in more than its instantaneous distribution.

Answer 2

Consider observing $y(n) = w(n)$ where $w(n)$ are IID gausian distributed with mean 0 and variance $\sigma^2$. The entropy of each other observations $y(n)$ is thus: $$\log_22\pi e\sigma^2$$.

Now if you average $y(n)$ the resultant variable is also gaussian distributed with mean 0 but variance $\sigma^2/N$. Therefore the entropy is $$\log_22\pi e\sigma^2/N$$ so the entropy difference is $$\log_2N$$ and this would be the entropy reduction.

The amount of information lost will depend on the distribution of the random variable. However, there are some interesting problems in signal processing where there would essentially be no loss.

For ex: consider a set of observations taken as $$y(n) = x + w(n)$$ where $w(n)$ is normally distributed IID random varibles and suppose we take $N$ observations of this kind.

Then the mean of $y(n)$ essentially represents what we refer to as a sufficient statistic for $x$ and infact represents the minimum variance unbiased estimator, i.e. the most efficient estimator. So in this case the sample mean encompasses all the necessary information and no information is lost.

However, if the noise $w(n)$ were not gaussian we couldn't claim this. So as you see the amount of information lost by sample mean depends on the distribution of the underlying variable.

Another example would be a trivial one of a deterministic random varaible. Here also no information would be lost by the sample mean.

Another way to look at it would be if the likelihood of posterior distribution (parameterized by the sample mean), after observing the sample mean renders the posterior useless, meaning encompasses no additional information about the random variable then there is no information loss, otherwise we would incur information loss.

Hope that helps

Answer 3

Let me try answering the question with only information provided in the question.

So, you are given n number of samples drawn from a Random Source with mean $\mu$ and variance $\sigma^2$. Without losing the idea which I am going to convey, we can assume that the Random Source is a Gaussian Source with mean $\mu$ and variance $\sigma^2$. Assuming that the Random Source is Gaussian will only maximize the expected information coming out of the source (Differential Entropy).

Now, Differential Entropy of a Gaussian Source depends only on the variance $\sigma^2$ and not on mean. Why is that? Intuitively speaking, variance of any random source is the measure of deviation from the central tendency (mean), which means variance tells us how much randomness is there in the consecutive samples drawn from that random source. And, measure of randomness or uncertainty is Entropy. More the randomness, more the entropy.

Differential Entropy of a Gaussian Source is given by :

$$h(X) = \frac{1}{2} ln(2\pi e \sigma^2)$$ So, more the variance, more is the variation in consecutive samples being thrown out of the source and hence more is the entropy.

Now, if you replace every N samples from this source by their sample mean $\hat{\mu}$, you will still have a Gaussian Random Source with mean $\mu$ but the variance of this sample mean becomes $\frac{\sigma^2}{N}$. There is an underlying assumption in making the preceding statement that the consecutive samples drawn from the Gaussian Random Source are uncorrelated. So going with the mentioned assumption, definitely some Entropy is lost. Which means some randomness is lost. How much? $$h(X_N) = \frac{1}{2}ln(2\pi e \frac{\sigma^2}{N}) = h(X) - \frac{1}{2}ln(N)$$ So, loss in Entropy is $\frac{1}{2}ln(N)$.

As you increase N, the loss increases and as $N \rightarrow \infty$, sample mean will approach true mean ($\hat{\mu} \rightarrow \mu$), and we will lose randomness completely.

Because, the variance of sample mean ($\hat{\mu}$) will approach 0, meaning subsequent sample means ($\hat{\mu[0]}, \hat{\mu[1]}, \hat{\mu[2]}, ....$) will be same and very very close to true mean ($\mu$). So, this Sample Mean Source will become a Deterministic Source spitting out values very very close to $\mu$ and not changing, this the entropy and information becomes 0.

This will not change even if you know $\sigma^2$ of the original Gaussian Random Source X. $Var(\hat{\mu}) \rightarrow 0$.