I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the following questions
- Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are maximizing a negative value!! Then if an original raw signal is quantized and the quantized information's entropy is calculated and found to be lesser than the original entropy would imply loss of information. So,why do we want to maximise entropy since it would mean that we are maximizing the uncertainty of the next symbol whereas we want to be certain about waht the next occurence of the symbol would be.
- What are the differences between Shannon's entropy, topological entropy and source entropy?
- What exactly is the significane of Kolgomorov complexity or Kolgomorov entropy. How is it related to Shannon's entropy?
- What information does mutual information between two vectors convey?