# Issues related to entropy

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the following questions

1. Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!! Then if an original raw signal is quantized and the quantized information's entropy is calculated and found to be lesser than the original entropy would imply loss of information. So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.
2. What are the differences between Shannon's entropy, topological entropy and source entropy?
3. What exactly is the significane of Kolgomorov complexity or Kolgomorov entropy. How is it related to Shannon's entropy?
4. What information does mutual information between two vectors convey?
• I would strongly recommend this (free online) book. Chapter 2 introduces entropy in the right context. The whole book is excellent and accompanies a lecture series by the author (David Mackay) – Henry Gomersall Jul 18 '12 at 10:13
• I'll just observe that entropy is one of those topics where it's very difficult (but not impossible) to obtain an "intuitive" understanding of it. You are struggling over some difficult concepts. – Daniel R Hicks Jul 18 '12 at 17:15

I will try to tackle questions 1 and 4.

1) Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!!

No, the values of the logarithms themselves are negative, so the negative sign makes them positive. All probabilities are a real number from 0 to 1, inclusive. The log of 1 is zero, and the log of anything less than 1 is negative. This may seem problematic since the log of 0 is $-\infty$, but we are really trying to maximize the expected value of these logs, so when we multiply by the probability itself the entire value approaches 0, not $\infty$. Entropy peaks when the probability is $1/2$.

So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.

No, in communicating information we absolutely do NOT want to be certain about what the next symbol will be. If we are certain, what information is gained by receiving it? None. It is only through uncertainty about what the transmitter will send that we can receive any information.

4) What information does mutual information between two vectors convey?

When there is mutual information between two vectors knowing something about one tells you something about the other. Mathematically this equates to the following- knowledge of one vector affects the probabilities of the other vector. If they were independent, this would not be the case.

An example of mutual information is digital walkie-talkies. One vector is the bit stream that the first walkie-talkie sends. The second vector is the signal that the second walkie-talkie receives. The two are obviously related, but due to noise and unknown channel conditions the second walkie-talkie cannot know for certain what the first one sent. It can make some really good guesses based on the signal, but it cannot know for sure.

• Thank you for the wonderful insights and examples. however, the answers to rest of the questions are unattempted.Do you suggest that I put thoseQs separately? – user1214586 Jul 19 '12 at 19:25
• @user1214586 You could try again with the other two. You are covering a lot of territory with those questions, so it's a lot of work to do a decent job answering them all. Even worse, you have a hard time finding one person who knows the answer to all of the questions. You have a better shot, for both reasons, if you keep the question more focused. – Jim Clay Jul 20 '12 at 19:28