# Conceptual question on entropy and its relation to information

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-parametric models" download link ( http://dl.acm.org/citation.cfm?id=1195853). The rationale provided is that minimization of error entropy is equivalent to maximization of likelihood. I am new to this area and find it hard to understand the intuition behind why entropy of the error minimization will yield the parameters. What happens when entropy is minimized?

1. What happens when Shannon Entropy is maximized? Entropy (Shannon's) is the uncertainty = average information or uncertainty (unsure).

2. And what happens when entropy is minimized and

3. What is the meaning of minimizing entropy of error?

• +1. Great question. A key to learn for most who want to learn info. theory. Jul 15 '15 at 4:39

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article.

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

To find the density that maximizes the entropy we have to use the calculus of variations, and that involves determining several coefficients called Lagrange multipliers. This is where the moments come in: they're the constraints that determine the multipliers. We set the $n$'th absolute moment for n=1..N to constants and we get the corresponding maximum entropy distribution. For example, the maximum entropy distribution whose first two moments are $\mu$ and $\sigma^2$ is the Gaussian distribution $N(\mu, \sigma^2)$. For details of the derivation please refer to a textbook.

I haven't read the paper but I'm guessing the rationale is that if you're trying to fit a model, you want to capture as much information of the signal as possible. Equivalently, you want to leave as little residual information as possible on the table; i.e., in the error. For details see 3.4 Minimum error entropy algorithm in Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives.

• Thank you for your kind attention, but my basic question are 1. Relation between entropy and information and then applying entropy in estimation of parameters which leads to the second question 2. why to minimize entropy of error and idea behind this Jul 26 '14 at 19:13
• When entropy of error is minimized, the papers mention that all moments not only the second order moments are minimized. What does this mean? Jul 26 '14 at 19:19
• Thank you for the link but I could not follow the part which says that if entropy is to be minimized then the likelihood function needs to be maximized.Secondly,when you say about distribution for maximum entropy,it means a uniform distribtuion.What is the distribution if we want to minimize entropy and what can we say about the information then?Since Entropy is related to information,what can we say about information when entropy of error is maximized and minimized And when just the entropy is maximized and minimized? Jul 26 '14 at 21:22
• Lastly,how can you say that deterministic signal has no information content?Sine wave is a deterministic signal & has a distribution.Could you kindly explain these further,thank you for your effort Jul 26 '14 at 21:24
• Once you know the parameters of the signal, including its initial conditions, sampling yields no additional information because you can already predict the signal.
– Emre
Jul 26 '14 at 21:36