What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about Kolmogorov entropy and why it is needed and its significance. Thank you.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Let me give you a quick answer, focused on ways you can improve and research for a deeper knowledge by yourself.
1 - Kolmogorov and Shannon Entropies are related to measures of information and strongly related to one another. (See the wikipedia page for Kolmogorov complexity for further info.)
2 - Boltzmann entropy, which is an older concept, is a completely differenty and unrelated concept from the theory of Thermodynamics. (Again, wikiepdia...)
3 - The formal definition of kolomogorov entropy uses the Shannon entropy in the formula.
If you are interested in the information entropy concept, i suggest you read the Book by Cover &Thomas, which has an entire chapter on Kolmogorov Complexity.
-
$\begingroup$ Thank you. In link press.princeton.edu/chapters/s3_9634.pdf Eq(3.8) it is mentioned that Entropy,S=log of phase space volume. Now, boltzmann states (en.wikipedia.org/wiki/…) that entropy = logarithm of number of microstates. Can I say that Kolmogorov entropy = logarithm of phase space volume as given in Eq(3.8)? Is the thermodynamics entropy be related to entropy used in inforamtion theory? $\endgroup$ Commented Jan 21, 2014 at 1:43
-
$\begingroup$ Here's the entropy you actually asked about: scholarpedia.org/article/Kolmogorov-Sinai_entropy A lot of mathematical notions, i don't even follow it that much. but basically, mathematicall the 3 definitions are very similar, but they are used in different scenarios for different purposes. kolmogorov, for algorithm complexity, Shannon for information content of a random variable, and Boltzmann for arrengements of different states of matter. $\endgroup$– boneCommented Jan 21, 2014 at 1:51
-
$\begingroup$ So, what exactly is kolmogorov entropy in laymans term and how is it differnet from Shannons entropy?\ $\endgroup$ Commented Jan 21, 2014 at 2:05
$\begingroup$
$\endgroup$
0
Boltzmann entropy shows uncertainty, with which a macrostate of a thermodynamic system describes its possible microstate. Except its application to thermodynamic systems (and multiplication by a physical constant), Boltzmann entropy (or more concretely, its generalization, namely, Gibbs entropy) is absolutely the same as Shannon entropy. They have similar forms and nearly the same meaning. As far as I remember, Shannon even took "H" for his entropy from Boltzmann's H-theorem.
Kolmogorov entropy... if you mean Kolmogorov-Sinai entropy than it's entropy of dynamic systems. It has the same nature as Boltzmann and Shannon entropy, but being applied to specific domain.
However, Kolmogorov complexity, which is also a measure of information quantity, is conceptually very different. While Shannon entropy is introduced based on knowledge of the stochastic model of the data source, Kolmogorov complexity doesn't suppose this knowledge meaning that one should account for the complexity of the model to get appropriate information measure.
In Shannon theory, the notion of information is derived from the notion of probability. At the same time, in Kolmogorov theory, the opposite way is taken - the notion of probability is derived from the notion of information, which is introduced basing on universal Turing machine. This results in very deep methodological differences, e.g., for machine learning.
