Return to Answer

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article: http://en.wikipedia.org/wiki/Maximum_Entropy

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

To find the density that maximizes the entropy we have to use the calculus of variations, and that involves determining several coefficients called Lagrange multipliers. This is where the moments come in: they're the constraints that determine the multipliers. We set the $n$'th absolute moment for n=1..N to constants and we get the corresponding maximum entropy distribution. For example, the maximum entropy distribution whose first two moments are $\mu$ and $\sigma^2$ is the Gaussian distribution $N(\mu, \sigma^2)$. For details of the derivation please refer to a textbook.

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article.

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

To find the density that maximizes the entropy we have to use the calculus of variations, and that involves determining several coefficients called Lagrange multipliers. This is where the moments come in: they're the constraints that determine the multipliers. We set the $n$'th absolute moment for n=1..N to constants and we get the corresponding maximum entropy distribution. For example, the maximum entropy distribution whose first two moments are $\mu$ and $\sigma^2$ is the Gaussian distribution $N(\mu, \sigma^2)$. For details of the derivation please refer to a textbook.

I haven't read the paper but I'm guessing the rationale is that if you're trying to fit a model, you want to capture as much information of the signal as possible. Equivalently, you want to leave as little residual information as possible on the table; i.e., in the error. For details see 3.4 Minimum error entropy algorithm in Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives.

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article: http://en.wikipedia.org/wiki/Maximum_Entropy

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

The $n$'th moment of the error is $\mathrm E[ |x-\hat x|^n]$. We want to minimize this, of course. Typically one only considers the second moment, denoted the variance, leading to the posterior mean estimator. If you consider other moments too you get the maximum entropy estimator because the moment minimization is the criterion used to determine the Lagrange multipliers in the calculus of variations derivation of the maximum entropy distribution. (Recall that we are seeking a function; that's why we invoke the calculus of variations.) For details of the derivation please refer to a textbook.

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article: http://en.wikipedia.org/wiki/Maximum_Entropy

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

To find the density that maximizes the entropy we have to use the calculus of variations, and that involves determining several coefficients called Lagrange multipliers. This is where the moments come in: they're the constraints that determine the multipliers. We set the $n$'th absolute moment for n=1..N to constants and we get the corresponding maximum entropy distribution. For example, the maximum entropy distribution whose first two moments are $\mu$ and $\sigma^2$ is the Gaussian distribution $N(\mu, \sigma^2)$. For details of the derivation please refer to a textbook.

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article: http://en.wikipedia.org/wiki/Maximum_Entropy

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

The $n$'th moment of the error is $\mathrm E[ (x-\hat x)^n]$. We want to minimize this, of course. Typically one only considers the second moment, denoted the variance, leading to the posterior mean estimator. If you consider other moments too you get the maximum entropy estimator because the moment minimization is the criterion used to determine the Lagrange multipliers in the calculus of variations derivation of the maximum entropy distribution. (Recall that we are seeking a function; that's why we invoke the calculus of variations.) For details of the derivation please refer to a textbook.

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article: http://en.wikipedia.org/wiki/Maximum_Entropy

Information and unpredictability are closely related. If you think of the signal as a stochastic process, its information content is determined by unpredictable it is. At one extreme, if the signal tends towards being deterministic, you can tell what it will be any an arbitrary time so there is no value in sampling the signal; it has no information content. Entropy formalizes this notion.

The $n$'th moment of the error is $\mathrm E[ |x-\hat x|^n]$. We want to minimize this, of course. Typically one only considers the second moment, denoted the variance, leading to the posterior mean estimator. If you consider other moments too you get the maximum entropy estimator because the moment minimization is the criterion used to determine the Lagrange multipliers in the calculus of variations derivation of the maximum entropy distribution. (Recall that we are seeking a function; that's why we invoke the calculus of variations.) For details of the derivation please refer to a textbook.