In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the information content of the desired signal. What is the relationship between entropy and SNR? With increase in signal to noise ration, the noise power decreases but this does not imply that the information content of the signal increases!! The information content may remain same, so does that mean entropy is unaffected?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
9
When you say that the "information content may remain the same," do you mean the information in the total signal, or the information of the desired signal? Hopefully this will answer both cases. I know Shannon entropy much better than Kolmogorov so I'll use that, but hopefully the logic will translate.
Let's say $X = S + N$ is your total signal ($X$), comprised of the sum of your desired signal $S$ and your noise component $N$. Let's call entropy $H$. As you said, noise adds entropy to a system by increasing its complexity. However, it's not necessarily only because we're more uncertain about the information content of the signal, but because there's more uncertainty in the signal overall. If SNR kind of measures how certain we are of what $S$ is, then $H(X)$ kind of measures how well we can predict future states of $X$ based on the current state of $X$. Entropy is concerned with how complex the whole signal is, regardless of the composition of noise vs. non-noise.
If you increase SNR by removing noise (attenuating $N$), you decrease the total signal $X$'s complexity and thus its entropy. You haven't lost any information carried by $S$, only (presumably meaningless) information carried by $N$. If $N$ is random noise, then obviously it doesn't carry meaningful information, but it takes a certain amount of information to describe $N$'s state, determined by the number of states that N can be in, and the probability of it being in each of those states. That's the entropy.
We can look at two Gaussian distributions with different variances, say one has a variance of $1$ and the other has a variance of $100$. Just looking at the equation for a Gaussian distribution, we see that the $Var=100$ distribution has a maximum probability that is only $\frac {1} {10}$th the value of the $var=1$ distr's probability. Conversely, this means that there is greater probability that the $Var=100$ distr will take values other than the mean, or that there is more certainty that the $Var=1$ distribution will take values near the mean. So, the $Var=1$ distribution has a lower entropy than the $Var=100$ distribution.
We established that higher variance implies higher entropy. Looking at error propagation, it is also true that $Var(X+Y) >= Var(X) + Var(Y)$ (equal for independent $X$, $Y$). If $X = S + N$, then for entropy $H$, $H(X) = H(S+N)$. Since $H$ is (indirectly) a function of variance, we can fudge things a little to say $H(Var[X]) = H(Var[S+N])$. To simplify, we say $S$ and $N$ are independent, so $H(Var[X]) = H(Var[S] + Var[N])$. Improved SNR often means attenuating noise power. This new signal with higher SNR will then be $X = S + (\frac {1} {k})N$, for $k>1$. Entropy then becomes $H(Var[X]) = H(Var[S] + (1/k)^2*Var[N])$. $k$ is greater than $1$, so $Var[N]$ will decrease when N is attenuated. If $Var[N]$ decreases, so does $Var[S+N]$, and therefore $Var[X]$, resulting in a decrease in $H(X)$.
Not very concise, sorry. In short, $X$'s entropy decreases if you increase SNR, but you've done nothing to $S$'s information. I can't find the sources right now, but there's a method to calculate SNR and mutual information (a bivariate measure similar to entropy) from each other. Maybe the main takeaway is that SNR and entropy don't measure the same thing.
-
$\begingroup$ Thank you for the details, it would have been really great if a reference was there for the little bitof analysis which you did since I need to provide this relationship between entropy and SNR in a paper and hence the citation. $\endgroup$ Feb 27, 2014 at 18:32
-
$\begingroup$ My analysis is pretty informal; it relies a bit too much on intuition/logic to claim any sort of rigor. The weak point I see immediately is the claim that increased SNR is equivalent to decreased overall variance. This statement holds if you increase SNR by attenuating noise, but not necessarily if you increase signal power (because that could increase signal variance==>overall variance==>entropy). There's likely another way to reach this conclusion, though. I think the relationship between MI and SNR came from Schloegl 2010 "Adaptive Methods in BCI Research - An Introductory Tutorial" $\endgroup$– dpbontFeb 28, 2014 at 1:29
-
$\begingroup$ :Sorry to again start this thread in continuation. I came across a problem when I was finding the error of the entropy of a model where error = desired_signal - estimated_signal. On increasing SNR, I found that entropy of error is increasing. But, when I calculate entropy of the desired signal $X$ with increasing SNR then X's entropy is decreasing. Can you please throw some insights on the former case where entropy of error increases with increasing SNR? $\endgroup$ Aug 14, 2014 at 2:37
-
$\begingroup$ Two questions. 1) When you say SNR increases, do you mean SNR of the estimated signal? (I assume so.) 2) What happens to your error as the error's entropy increases? Generally speaking, increase in entropy either means increase in variance/decrease in predictability. I could maybe envision a situation where your error variance increases but you remove an error bias (which could increase error entropy but decrease error). $\endgroup$– dpbontAug 15, 2014 at 3:56
-
$\begingroup$ Thank you for your prompt reply.(1)By SNR increase I mean increasing the SNR of the measurement noise in the model.So, in Eq $X=S+N$ I increase the SNR of $N$ and measure entropy H1(X) & H2(error).(2)The errors decreases and all values become more or less equal but not zero.The way I calculate entropy of error is as follows:Considering AR(2) model.At receiver's end,$z(t) = X(t) - (a1X(t-1) + b1X(t-2))$ where (a1,b1) are guessed parameters & $X(t)$ are observations at a particular SNR of $N$ say snr1. Let's say I have 5 pairs of guesses (a1,b1) & for each pair I get entropy of error. $\endgroup$ Aug 15, 2014 at 6:28
$\begingroup$
$\endgroup$
Here's a quote from [1, p. 186] to get you, OP or Googler, started:
Very roughly, $\mathcal{H} \approx \text{(number of active components in data)} \times \log \text{SNR}$
Here $\mathcal{H}$ is the negative entropy of the posterior distribution of the parameters of your model of the signal. Good luck!
[1] D. Sivia and J. Skilling, Data analysis: a Bayesian tutorial. OUP Oxford, 2006
