I have come know of Huffman and Arithmatic encoding that try to curtail redundant data and thus compress data. Why are they called Entropy encoding? What does that mean? Why not just call them compression encoding?


The term entropy comes from the field of information theory. Information theory posits that unexpected messages bear more information than expected ones. For instance, saying that there will be a solar eclipse tomorrow has much more information (assuming it's true) than saying the sun will rise tomorrow. Saying the sun will rise tomorrow, while true (and important!), has very little information for most recipients because they have pretty much 100% expectation of it.

Anyway, you can prove mathematically that information is maximized when all of the possible messages have equal probability. This maximizes the randomness or "entropy".

A perfect, lossless compression scheme could theoretically reduce the number of bits of a message to $ceil(log_2(entropy))$ but no further. Lossy compression schemes can reduce further, but information will be lost.

I personally am not fond of the term "entropy" because I associate it more with the 2nd law of thermodynamics. I believe that that association was intentional though, in the sense that entropy (in the thermodynamics sense) implies randomness and chaos, while low entropy implies orderliness.

| improve this answer | |
  • $\begingroup$ Actually I learnt entropy in thermodynamics too and that is why it was not clear why entropy which is associated with disorder is being used in the term "entropy encoding" $\endgroup$ – quantum231 Oct 10 '14 at 22:29
  • $\begingroup$ Because unpredictability, as quantified by entropy, is what makes a signal informative. A signal you can predict is a signal you need not sample. It contains no information. Thus the connection to information theory. $\endgroup$ – Emre Oct 12 '14 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.