# What is the difference between entropy and entropy rate and which one is important in information theory? [duplicate]

By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high entropy preferred for processing information?

Then there is entropy rate, which I cannot understand due to heavy mathematics. All that I can follow is that entropy rate = entropy/ number of unique symbols.

Please forgive if these questions seem trivial, but it is quite hard to filter the answers from textbooks which are mainly focused on theoretical definition and not on how one can use entropy in real applications. Below are my confusions.

• If the entropy of an information generating source is high, that is considered to be good. But what is the role of entropy rate?

• Does entropy have a lower and upper bound or a range? Based on equi-probable symbols, we say then 0.5 is the max entropy as equi-probable symbols convey maximum uncertainty about the source. Is the range of entropy is 0--1?

• What information is gained or loss when entropy is discussed from information theory viewpoint?

## marked as duplicate by MBaz, Peter K.♦Oct 12 '17 at 18:55

• @SrishtiM the only part of this question that isn't a duplicate of an existing question is the part about entropy rate. You can think of entropy rate as a generalization of entropy for a source with some temporal correlations (eg. a Markov source where future symbol probabilities depend on the previous symbols). If the symbols $X_i$ were all i.i.d we have $\frac{1}{n}H(X_1,\ldots,X_n) = \frac{1}{n} n H(X_1)=H(X_1)$ and entropy rate reduces to the usual definition of entropy of a random variable. For some intuition please see section 4.2 of Elements of Information Theory by Cover and Thomas. – Atul Ingle Oct 12 '17 at 20:25