1
$\begingroup$

Solving for ill-posed linear models, I saw that Maximal entropy is also parsimonious and in that regards similar to L1-sparsity promoting regularization. How is it different and are they interchangeable ? is it the entropy maximization always convex?

In the L1 case I use PBDM to minimize the combined terms looking for a sparse solution:

$$ ||Ax-y||_2^2 + \lambda ||x||_1$$

This is convex, and stright forward to solve, even though I'm not sure how to get the $\lambda$ parameters yet.

For maximal entropy, I read that I need to maximize :

$$ \sum_i x_i \log(x_i) $$

such that $Ax = y, F*x <= g $

so what is F here, and is g similar to $\lambda$?

$\endgroup$

1 Answer 1

1
$\begingroup$

Apparently your problem is only $A x = y$,

$F \circ x \preceq g$ it could make your problem infeasible, I wouldn't artificially add it. If you don't know what $F$ or $g$ to use, you could start with $F=0$, and $g=0$, so that accepts any $x$, in other words don't use that constraint.

About the entropy maximization, you drop a negative sign in your expression, you must maximize $\sum -x \log(x)$

The regularization effect is related to convexity, the derivative of $-x \log(x)$ is $-(1 + \log(x))$. As $x$ increase the derivative increase, so if you have degrees of freedom that let you vary $x_i$ and $x_j$, the entropy maximization will pull those two values closer to each other, i.e. you can increase entropy by increasing the smaller and decreasing the greater of them.

In particular, a well known result is that under the constraint $\sum x=1$, the uniform distribution maximizes entropy.

Summarizing

The L1 regularization will pull values towards zero. Maximum entropy regularization will pull the values closer to each other.

Edit

How to use it for positive and negative values.

If you want to use this regularization for positive and negative values you could think of some transformation $\mathbb{R} \to \mathbb{R}^+$.

If you use $\exp(x)$ as you mentioned in the comment, you end up maximizing $ \sum x \exp(x)$. Negative values will have negligible contributions, while positive values may have huge contributions to the sum, so it will probably move the negative values to as a function of the positive ones.

Also,the minimum is $0 = x \exp(x) + \exp(x) = (x+1) \exp(x)$, what $x=1", has in special.

Maybe it would be preferrable to use $\exp(x^2)$, then the positive and negative side will look the same. The critical points are $(2x \cdot x^2 + 2x)\exp(x^2) =2\, x \,(x^2+1) \exp(x^2) = 0$, i.e. it has a minimum in $x=0$.

The other point is that $\exp(x)$ or $\exp(x^2)$ is prone to overflow. If you want to avoid problems due to the exponential growth of the exponential function you could try $x^2$, but this is not convex any more $x + 2x \log(x^2)=0$, has three solutions $x=0$, $x=\pm e^{-1/4}$.

Then the things starts to get too complicated, and this is not entropy maximization anymore. If you are open to other regularizations I would give L2 regularization a chance.

$\endgroup$
3
  • $\begingroup$ thank you for the answer! It seems that using this method I'm restricted to only positive values in x (because of the log(x)), is there a way to circumvent that and use ME for any both negative and positive x values? (take an exp of everything?) $\endgroup$
    – yourds
    Dec 18, 2022 at 2:49
  • $\begingroup$ Exp could be something, however for floating point you could have numeric problems. $\endgroup$
    – Bob
    Dec 18, 2022 at 11:48
  • $\begingroup$ thank you again for the detailed answer! $\endgroup$
    – yourds
    Dec 19, 2022 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.