Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication process. Assuming Y to be the discretized output, what is meant by Shannon's source entropy for a given discretization bin $k$ : $H_{source} = lim_{k \rightarrow \infty} \frac{1}{k} H_k$ where $H_k$ stands for Shannons entropy. I found this in the paper. Now, problem_Set also mentions about source entropy but the formula is very different! Its the same as Shannon's entropy. So,what exactly is Source entropy? Is it the entropy of X or Y or is it the same as Shannon's entropy
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
So,what exactly is Source entropy?
When the paper talks about "source entropy" all they mean is the entropy of the information source. You can see that from the following passage in the paper- "Shannon showed that there must exist at least one encoding of the sequence generated by an information source which allows error-free transmission of the sequence when the channel capacity is larger than the source entropy."
Now, problem_Set also mentions about source entropy but the formula is very different!
Yes, and no. The formulas appear different, but they are not really. The first formula is $H_{source} = lim_{k→∞}\frac{1}{k}H_k$
$H_{source}$ is the total entropy of the source, and $H_k$ is the entropy of each possible discrete value that the random variables in $H$ can be. Thus, it is almost a trivial statement that the total entropy is the sum of the individual value entropies, while the number of discrete values can go to $\infty$.
The problem set equation is- $H(s) = \sum_i p_ilog_2\frac{1}{p_i}$
This is really the same equation, it just replaces the generic "$H_k$" with the implied $log_2\frac{1}{p_i}$. The $\frac{1}{k}$ in the first equation is equivalent to the $p_i$ in the second equation- i.e. they are assuming that all of the discrete values of H are equally likely. This is not generally true, but is true when the entropy is maximized.
-
$\begingroup$ Thank you for the eye opener. I have few more queries based on ur reply.(A)So, if Y is a discretized version, discretized into n bins, of X then source entropy would mean entropy of X or Y? (B)Also, when calculating & implementing entropy, do we consider the joint probability or the probability of each occurence of variable?(C)In the formula k would imply the number of discretization levels/bins or the total number of sample(ie data length)? $\endgroup$ Commented Jul 31, 2012 at 2:53
-
$\begingroup$ If $X$ is the source signal and Y is the signal that a receiver gets after $X$ travels through a channel, then $H_k$ is the source entropy, simply because $X$ is the source. When you ask about joint probabilities, are you talking about the joint probabilities of $X$ and $Y$. If so, that is not correct. When calculating entropy you just calculate it for $X$ or $Y$. You consider the joint probabilities when calculating the mutual information. $\endgroup$– Jim ClayCommented Jul 31, 2012 at 13:48
-
$\begingroup$ Thank you once again. So to sum up, formulae for source entropy and shannon's entropy is the same however, osurce entropy denotes the entropy of the sender's signal which is communicating through the channel. Whereas, Shannon's entropy in particulary does not imply either source or receive, it can be the entropy of either. Am I correct? How about part (C) in whichI ask about a confusion about the varaible k $\endgroup$ Commented Jul 31, 2012 at 17:27
-
$\begingroup$ Yes, that is correct. K implies the number of discrete values that the random variables can take, not the sample number. $\endgroup$– Jim ClayCommented Jul 31, 2012 at 17:52