# L1 regularization vs maximal entropy?

Solving for ill-posed linear models, I saw that Maximal entropy is also parsimonious and in that regards similar to L1-sparsity promoting regularization. How is it different and are they interchangeable ? is it the entropy maximization always convex?

In the L1 case I use PBDM to minimize the combined terms looking for a sparse solution:

$$||Ax-y||_2^2 + \lambda ||x||_1$$

This is convex, and stright forward to solve, even though I'm not sure how to get the $$\lambda$$ parameters yet.

For maximal entropy, I read that I need to maximize :

$$\sum_i x_i \log(x_i)$$

such that $$Ax = y, F*x <= g$$

so what is F here, and is g similar to $$\lambda$$?

Apparently your problem is only $$A x = y$$,

$$F \circ x \preceq g$$ it could make your problem infeasible, I wouldn't artificially add it. If you don't know what $$F$$ or $$g$$ to use, you could start with $$F=0$$, and $$g=0$$, so that accepts any $$x$$, in other words don't use that constraint.

About the entropy maximization, you drop a negative sign in your expression, you must maximize $$\sum -x \log(x)$$

The regularization effect is related to convexity, the derivative of $$-x \log(x)$$ is $$-(1 + \log(x))$$. As $$x$$ increase the derivative increase, so if you have degrees of freedom that let you vary $$x_i$$ and $$x_j$$, the entropy maximization will pull those two values closer to each other, i.e. you can increase entropy by increasing the smaller and decreasing the greater of them.

In particular, a well known result is that under the constraint $$\sum x=1$$, the uniform distribution maximizes entropy.

Summarizing

The L1 regularization will pull values towards zero. Maximum entropy regularization will pull the values closer to each other.

## Edit

How to use it for positive and negative values.

If you want to use this regularization for positive and negative values you could think of some transformation $$\mathbb{R} \to \mathbb{R}^+$$.

If you use $$\exp(x)$$ as you mentioned in the comment, you end up maximizing $$\sum x \exp(x)$$. Negative values will have negligible contributions, while positive values may have huge contributions to the sum, so it will probably move the negative values to as a function of the positive ones.

Also,the minimum is $$0 = x \exp(x) + \exp(x) = (x+1) \exp(x)$$, what \$x=1", has in special.

Maybe it would be preferrable to use $$\exp(x^2)$$, then the positive and negative side will look the same. The critical points are $$(2x \cdot x^2 + 2x)\exp(x^2) =2\, x \,(x^2+1) \exp(x^2) = 0$$, i.e. it has a minimum in $$x=0$$.

The other point is that $$\exp(x)$$ or $$\exp(x^2)$$ is prone to overflow. If you want to avoid problems due to the exponential growth of the exponential function you could try $$x^2$$, but this is not convex any more $$x + 2x \log(x^2)=0$$, has three solutions $$x=0$$, $$x=\pm e^{-1/4}$$.

Then the things starts to get too complicated, and this is not entropy maximization anymore. If you are open to other regularizations I would give L2 regularization a chance.

• thank you for the answer! It seems that using this method I'm restricted to only positive values in x (because of the log(x)), is there a way to circumvent that and use ME for any both negative and positive x values? (take an exp of everything?) Dec 18, 2022 at 2:49
• Exp could be something, however for floating point you could have numeric problems.
– Bob
Dec 18, 2022 at 11:48
• thank you again for the detailed answer! Dec 19, 2022 at 18:38