1
$\begingroup$

This question already has an answer here:

By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high entropy preferred for processing information?

Then there is entropy rate, which I cannot understand due to heavy mathematics. All that I can follow is that entropy rate = entropy/ number of unique symbols.

Please forgive if these questions seem trivial, but it is quite hard to filter the answers from textbooks which are mainly focused on theoretical definition and not on how one can use entropy in real applications. Below are my confusions.

  • If the entropy of an information generating source is high, that is considered to be good. But what is the role of entropy rate?

  • Does entropy have a lower and upper bound or a range? Based on equi-probable symbols, we say then 0.5 is the max entropy as equi-probable symbols convey maximum uncertainty about the source. Is the range of entropy is 0--1?

  • What information is gained or loss when entropy is discussed from information theory viewpoint?

$\endgroup$

marked as duplicate by MBaz, Peter K. Oct 12 '17 at 18:55

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ The entropy of a source is calculated with Shannon's formula. The "entropy of a system" is not defined in this context. Forget about 'information processing" as used in computing -- that's only tangentially related to information theory. $\endgroup$ – MBaz Oct 12 '17 at 2:20
  • $\begingroup$ I don't see any reference to the difference between the different entropies in your original question. In any case, "entropy" is an overloaded term, used in many different concepts. Shannon entropy is relevant for data communications and information theory; other definitions are used in other fields. By the way, there is no such thing as "Shannon's entropy rate". It's just "entropy". $\endgroup$ – MBaz Oct 12 '17 at 13:00
  • $\begingroup$ I had never heard of it; thanks for the link. That book's approach is too mathematical for my taste, though. I'd recommend reading Information Theory, Inference, and Learning Algorithms by MacKay. It's available free from the author, online. $\endgroup$ – MBaz Oct 12 '17 at 16:01
  • $\begingroup$ @SrishtiM the only part of this question that isn't a duplicate of an existing question is the part about entropy rate. You can think of entropy rate as a generalization of entropy for a source with some temporal correlations (eg. a Markov source where future symbol probabilities depend on the previous symbols). If the symbols $X_i$ were all i.i.d we have $\frac{1}{n}H(X_1,\ldots,X_n) = \frac{1}{n} n H(X_1)=H(X_1)$ and entropy rate reduces to the usual definition of entropy of a random variable. For some intuition please see section 4.2 of Elements of Information Theory by Cover and Thomas. $\endgroup$ – Atul Ingle Oct 12 '17 at 20:25