So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am also curious if we also take the second moment (variance) of the same signal, how much information do we gain by using the variance along with the mean? In other words:

$$H_{\textrm{Mean}}(\Sigma_0^n x_iP_i)=\ ?$$ $$H_{\textrm{Variance}}(\Sigma_0^n (x_i-\mu )^2)=\ ?$$

Motivation: I just want to know how much information is dropped if we simply drop variace term and use just mean instead. I believe this concept cant be expanded to also consider information in third and fourth moments.

  • 1
    $\begingroup$ So basically information is defined by Shannon's entropy. Strictly speaking, Shannon Entropy is defined by being the expectation of Shannon Information, not the other way around; that's kind of important here, because you're mixing means and expectations, which, depending on properties of your signal (correlation, ergodicity) has consequences for the answerability of your question! $\endgroup$ – Marcus Müller Apr 11 '20 at 21:39
  • 2
    $\begingroup$ Anyway, we can't tell you how much information is lost, because you don't specify what about your signal is unknown to begin with! This becomes answerable as soon as you write down a stochastic signal model :) $\endgroup$ – Marcus Müller Apr 11 '20 at 21:40
  • 1
    $\begingroup$ Mean is a property of an actual set of samples. Expected value is a property of a random process, or a random variable. The mean of a set of samples that are the output of a random process can be a good estimate of that process's expected value -- but it's not the expected value. $\endgroup$ – TimWescott Apr 12 '20 at 0:20
  • 1
    $\begingroup$ Your mathematical expression for the average is not a moving average. Do you mean the average of the whole signal, or do you mean a real moving average, i.e. $y_i = \frac{1}{N}\sum_{k=i-N+1}^i x_k$? $\endgroup$ – TimWescott Apr 12 '20 at 0:28
  • 1
    $\begingroup$ @GENIVI-LEARNER :) I didn't mean the definition of correlation, but how it applies to your signal. If the second sample is perfectly correlated to the first, there might be absolutely no information lost; if it's not, there might be information lost when averaged! $\endgroup$ – Marcus Müller Apr 12 '20 at 9:58

We can't say from the information you're giving alone. We need information on how the individual samples relate to each other!

Remember, the information of the sequence $(X)_{i=1}^N$ of $N$ random variables $X_i, \, i=1, \ldots, N$ is:

\begin{align} I(X=x) &= I((X_1=x_1, X_2=x_2, \ldots, X_N=x_N))\\ &\text{shorthand: }P(X_i=x_i)=:P(x_i)\\ &= I(x_1) + I((x_2, \ldots, x_N)|x_1)\\ &= I(x_1) + I(x_2|x_1)+I((x_3, \ldots, x_N)|x_1,x_2)\\ &= I(x_1) + \sum_{n=2}^N I(x_n|x_1,\ldots x_{n-1}) \tag 1\label{eq1}\\ &= -\log_2(P(x_1))- \underbrace{\sum_{n=2}^N\log_2(P(x_n|x_1,\ldots x_{n-1}))}_{:=v}\tag2\label{eq2} \end{align}

You'll notice when considiering $v$ that all these $P(x_n|x_1,\ldots x_{n-1})$ collapse to $P(x_n)$ if, and only if, the $X_n$ are independent.

In that case, $I(X)$ simply becomes $\sum I(X_n)$. For all other cases, that sum is the upper bound for what the overall information can be, but doesn't necessarily achieve that.

Let's construct an example where all $X_N$ are Gaussian:

$X_1\sim \mathcal N(0;\sigma^2)$ is drawn from a Gaussian source.

For all $N\ge i> 1$, let $X_i= i X_{i-1}$. Then, directly, $X_2\sim \mathcal N(0; 4\sigma^2)$, and generally $X_i\sim\mathcal N(0;i^2 \sigma^2)$.

All these variables are Gaussian, but they are not independent. Let's use that with $\eqref{eq1}$:

\begin{align} I(X=x) &= I((X_1=x_1, X_2=x_2, \ldots, X_N=x_N))\\ &= -\log_2(P(x_1))- \sum_{n=2}^N\log_2(P(x_n|x_1,\ldots x_{n-1}))\\ &= I(x_1) + \sum_{n=2}^N I(x_n|x_1,\ldots x_{n-1}) \label{eq3}\tag3\\ \end{align}

but what is e.g. $I(x_2|x_1)$? Well, it's 0, because there's no information in $x_2$ that we didn't have already through observing $x_1$! (You can also prove that formalistically by using the conditional entropy formula. But I found this more intuitive.)

The same is through for every other element of $v$.

What happens now if you take the mean of that series? Well, you get another Gaussian random variable, sure, but it has exactly the same information as the whole sequence: From the mean alone, you can trivially find the original $x_1$:

$$\bar X = \frac1N\sum_{n=1}^N X_n = \frac1N\sum_{n=1}^N n X_1=\frac1N X_1\sum_{n=1}^N n = C X_1,\;\; C\,\text{const.}$$

and therefore, the entropy of $\bar X$ is the entropy of $X$.

So, nothing's lost.

However, if all the $X_i$ are independent, but follow the same Gaussian distribution, then the mean $\bar X$ is a Gaussian variable of variance $\frac{\sigma^2}{N}$, and you lose the difference between that and $N$ times the entropy of a $\sigma^2$-variance random variable.

So, the answer to your question is in what you don't tell us:

Entropies of sources depend heavily on the conditional probabilities of their words; you need to describe a random variable in more than its instantaneous distribution.

  • $\begingroup$ What kind of correlation is this: $X_i= i X_{i-1}$?. Also in the case $I(x_2|x_1)=0$, are we implying mutual information between $x_2$ & $x_1$ is 1? $\endgroup$ – GENIVI-LEARNER Apr 12 '20 at 20:50
  • $\begingroup$ that is the maximum kind of correlation you can get :) $\endgroup$ – Marcus Müller Apr 12 '20 at 21:02
  • $\begingroup$ Reading your answer did give me bigger picture as you identified the mistake I made. I was thinking in terms of instantaneous distribution of the random variable, but as you mentioned for a signal we also have to look at the correlation between the RV's. The image in my head was for one RV :) $\endgroup$ – GENIVI-LEARNER Apr 12 '20 at 21:16
  • $\begingroup$ @GENIVI-LEARNER happy to hear that! $\endgroup$ – Marcus Müller Apr 12 '20 at 21:24
  • 1
    $\begingroup$ Thomas and Cover: Elements of Information Theory. $\endgroup$ – Marcus Müller Apr 13 '20 at 11:59

Consider observing $y(n) = w(n)$ where $w(n)$ are IID gausian distributed with mean 0 and variance $\sigma^2$. The entropy of each other observations $y(n)$ is thus: $$\log_22\pi e\sigma^2$$.

Now if you average $y(n)$ the resultant variable is also gaussian distributed with mean 0 but variance $\sigma^2/N$. Therefore the entropy is $$\log_22\pi e\sigma^2/N$$ so the entropy difference is $$\log_2N$$ and this would be the entropy reduction.

The amount of information lost will depend on the distribution of the random variable. However, there are some interesting problems in signal processing where there would essentially be no loss.

For ex: consider a set of observations taken as $$y(n) = x + w(n)$$ where $w(n)$ is normally distributed IID random varibles and suppose we take $N$ observations of this kind.

Then the mean of $y(n)$ essentially represents what we refer to as a sufficient statistic for $x$ and infact represents the minimum variance unbiased estimator, i.e. the most efficient estimator. So in this case the sample mean encompasses all the necessary information and no information is lost.

However, if the noise $w(n)$ were not gaussian we couldn't claim this. So as you see the amount of information lost by sample mean depends on the distribution of the underlying variable.

Another example would be a trivial one of a deterministic random varaible. Here also no information would be lost by the sample mean.

Another way to look at it would be if the likelihood of posterior distribution (parameterized by the sample mean), after observing the sample mean renders the posterior useless, meaning encompasses no additional information about the random variable then there is no information loss, otherwise we would incur information loss.

Hope that helps

  • $\begingroup$ What is your definition of information here? $\endgroup$ – Engineer Apr 12 '20 at 11:57
  • $\begingroup$ As defined by Shannon in terms of entropy $\endgroup$ – Dsp guy sam Apr 12 '20 at 12:20
  • 1
    $\begingroup$ You can add that to your answer. What is the information before averaging? What is it after? That was the question $\endgroup$ – Engineer Apr 12 '20 at 12:24
  • $\begingroup$ Ok...added.that $\endgroup$ – Dsp guy sam Apr 12 '20 at 14:01
  • 1
    $\begingroup$ For second and question I would kindly refer you to chapter 5 on sufficient statistic in Stephen M Kay's book on estimation theory. Too subtle a point to describe in comment $\endgroup$ – Dsp guy sam Apr 13 '20 at 7:17

Let me try answering the question with only information provided in the question.

So, you are given n number of samples drawn from a Random Source with mean $\mu$ and variance $\sigma^2$. Without losing the idea which I am going to convey, we can assume that the Random Source is a Gaussian Source with mean $\mu$ and variance $\sigma^2$. Assuming that the Random Source is Gaussian will only maximize the expected information coming out of the source (Differential Entropy).

Now, Differential Entropy of a Gaussian Source depends only on the variance $\sigma^2$ and not on mean. Why is that? Intuitively speaking, variance of any random source is the measure of deviation from the central tendency (mean), which means variance tells us how much randomness is there in the consecutive samples drawn from that random source. And, measure of randomness or uncertainty is Entropy. More the randomness, more the entropy.

Differential Entropy of a Gaussian Source is given by :

$$h(X) = \frac{1}{2} ln(2\pi e \sigma^2)$$ So, more the variance, more is the variation in consecutive samples being thrown out of the source and hence more is the entropy.

Now, if you replace every N samples from this source by their sample mean $\hat{\mu}$, you will still have a Gaussian Random Source with mean $\mu$ but the variance of this sample mean becomes $\frac{\sigma^2}{N}$. There is an underlying assumption in making the preceding statement that the consecutive samples drawn from the Gaussian Random Source are uncorrelated. So going with the mentioned assumption, definitely some Entropy is lost. Which means some randomness is lost. How much? $$h(X_N) = \frac{1}{2}ln(2\pi e \frac{\sigma^2}{N}) = h(X) - \frac{1}{2}ln(N)$$ So, loss in Entropy is $\frac{1}{2}ln(N)$.

As you increase N, the loss increases and as $N \rightarrow \infty$, sample mean will approach true mean ($\hat{\mu} \rightarrow \mu$), and we will lose randomness completely.

Because, the variance of sample mean ($\hat{\mu}$) will approach 0, meaning subsequent sample means ($\hat{\mu[0]}, \hat{\mu[1]}, \hat{\mu[2]}, ....$) will be same and very very close to true mean ($\mu$). So, this Sample Mean Source will become a Deterministic Source spitting out values very very close to $\mu$ and not changing, this the entropy and information becomes 0.

This will not change even if you know $\sigma^2$ of the original Gaussian Random Source X. $Var(\hat{\mu}) \rightarrow 0$.

  • $\begingroup$ Assuming that the Random Source is Gaussian will only maximize the expected information coming out of the source I didnt quite get what you meant here. Actually there is similarity to what this sentence say with D-Optamality criteria where you minimize the determinant of variance matrix to maximize information. $\endgroup$ – GENIVI-LEARNER Apr 12 '20 at 20:13
  • 1
    $\begingroup$ So, for a given variance the distribution that maximizes differential entropy is Gaussian Distribution. It can be proved. But including the proof would have made me digress from the actual point. $\endgroup$ – DSP Rookie Apr 12 '20 at 20:17
  • $\begingroup$ +1 for As you increase N, the loss increases and as 𝑁→∞, sample mean will approach true mean (𝜇̂ →𝜇), and we will lose randomness completely. This makes sense but unintuitive. If we assume the distribution is normal then mean and variance is considered sufficient statistic. Meaning we have complete information of the underlying system by having "both" mean and variance. Now Shannon's description of entropy is related to variance for Gaussian distribution so I know two distribution of same variance have same "information content" but to tell them apart we need information of their 𝜇's $\endgroup$ – GENIVI-LEARNER Apr 12 '20 at 20:21
  • 2
    $\begingroup$ Entropy or amount of information of a random source is defined in terms of what uncertainty it contains. Sure, to know about a Gaussian signal you need to know both mean and variance, but to convey certain amount of information using a Gaussian Source, only variance is enough to say how much information can be conveyed. Mean will just be a DC term, but you can only convey information in randomness, uncertainties, not with a constant. $\endgroup$ – DSP Rookie Apr 12 '20 at 20:41
  • 2
    $\begingroup$ If you have N people in a room ang you know average height. You sure have information about the population. But if you want to device some way to convey some information out of the room by sending one man at a time and only their height is observable. Then, it doesnot matter whether the average height of the people in the room was 170cm or 100cm. Only the variation in the height can be used to convey the information out. $\endgroup$ – DSP Rookie Apr 12 '20 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.