Issues related to entropy

Question

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the following questions

1. Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!! Then if an original raw signal is quantized and the quantized information's entropy is calculated and found to be lesser than the original entropy would imply loss of information. So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.
2. What are the differences between Shannon's entropy, topological entropy and source entropy?
3. What exactly is the significane of Kolgomorov complexity or Kolgomorov entropy. How is it related to Shannon's entropy?
4. What information does mutual information between two vectors convey?

Answer 1

I will try to tackle questions 1 and 4.

1) Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!!

No, the values of the logarithms themselves are negative, so the negative sign makes them positive. All probabilities are a real number from 0 to 1, inclusive. The log of 1 is zero, and the log of anything less than 1 is negative. This may seem problematic since the log of 0 is $-\infty$, but we are really trying to maximize the expected value of these logs, so when we multiply by the probability itself the entire value approaches 0, not $\infty$. Entropy peaks when the probability is $1/2$.

So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.

No, in communicating information we absolutely do NOT want to be certain about what the next symbol will be. If we are certain, what information is gained by receiving it? None. It is only through uncertainty about what the transmitter will send that we can receive any information.

4) What information does mutual information between two vectors convey?

When there is mutual information between two vectors knowing something about one tells you something about the other. Mathematically this equates to the following- knowledge of one vector affects the probabilities of the other vector. If they were independent, this would not be the case.

An example of mutual information is digital walkie-talkies. One vector is the bit stream that the first walkie-talkie sends. The second vector is the signal that the second walkie-talkie receives. The two are obviously related, but due to noise and unknown channel conditions the second walkie-talkie cannot know for certain what the first one sent. It can make some really good guesses based on the signal, but it cannot know for sure.