1
$\begingroup$

Often in variational methods (and not only) we have an energy that is of the form: $$E(u) = \frac{1}{2}\|f-u\|^2_2 + \frac{\alpha}{2}\|\psi(u)\|^2_2,$$

where the first term is referred to as the data term, and the second as the smoothness term. I understand that the squared $L_2$ norm is especially appealing since it results in a simple expression when we are looking for the minimum. I am looking for the formal motivation for designing such an energy. More specifically, I would like to see it as resulting from some bound upon the actual error. Let $f$ be a degraded/noisy image of the original image $g$, and $u$ be the image that we are looking for. Is it possible for the above to be interpreted as a bound of $\|g-f\|^2_2$? Let us consider an even simpler problem (where $h$ is a sufficiently smoothed version of $f$): $$E(u) = \frac{1}{2}\|f-u\|^2_2 + \frac{\alpha}{2}\|u-h\|^2_2$$ If I start from $\|g-f\|_2$ I can get the following bound: $$\|u-g\| = \|u-f+f-g\| \leq \|u-f\| + \|f-g\|$$ $$\|u-g\| = \|u-h+h-g\| \leq \|u-h\| + \|h-g\|$$ Combining the two inequalities with some weight $\lambda \in [0,1]$ I get: $$\|u-g\|\leq (1-\lambda)\|u-f\| + \lambda\|u-h\| + (1-\lambda)\|f-g\| + \lambda\|h-g\|$$ Since $f,g,h$ are fixed, the minimization is only over the first two terms: $$\min_u (1-\lambda)\|u-f\| + \lambda\|u-h\|$$ And this can be rewritten through $\alpha$ by setting $\alpha = \frac{\lambda}{1-\lambda}$. My issue is that these are still not squared norms. If I try to square both sides, this results in a different minimization energy. The other way to get the squares involves writing out something of the form: $$\|u-g\|^2 = \|(u-f) + (f-g)\|^2 = \|u-f\|^2 + \|f-g\|^2 - 2(u-f)\cdot(g-f) \\ \leq \|u-f\|^2 + \|f-g\|^2$$ The last inequality holds only when $(u-f)\cdot(g-f) \geq 0$ though. Said otherwise, the angle between $(u-f)$ and $(g-f)$ needs to be smaller than 90 degrees which is not something that can be guaranteed I believe. A looser bound is: $$\|u-g\|^2 \leq \|u-f\|^2 + \|f-g\|^2 + 2\|u-f\|\|f-g\|,$$ but it also involves non-squared terms.

Here's something else I noticed about the non-squared energy:

$$\alpha\|u-f\| + \beta\|u-h\|, \alpha, \beta > 0$$

will always have as solutions $u=f$ or $u=h$ for $\alpha<\beta$ respectively $\alpha>\beta$. When $\alpha=\beta$ any solution of the form $(1-\lambda)f + \lambda h$ is admissible. It is thus clear that the above is not very interesting as a minimization energy.

Cross-posted to: https://math.stackexchange.com/q/3692290/463794 I expect to get a different perspective there.

$\endgroup$
1
$\begingroup$

One of the motivations to use the $ {L}_{2} $ norm comes from the Maximum a Posteriori Estimation (MAP) framework.

If you model $ \psi \left( u \right) \sim \mathcal{N} \left( 0, \alpha \right) $ then if you derive the MAP Estimator in case the added noise is Gaussian you'd get the exact model you posted above.

An example of the derivation of MAP model to the above can be seen in my answer to Estimating the Signal by Deconvolution with a Prior on the Filter Coefficients and the Signal Samples.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Could you expand upon that? I also noticed that the non-squared energy produces only trivial solutions for non-equal weights (see the edit above), hence not very useful. $\endgroup$ – lightxbulb May 27 at 12:53
  • 1
    $\begingroup$ I added a link to a derivation I made for a similar model. $\endgroup$ – Royi May 27 at 13:42
  • $\begingroup$ Are you aware of any different interpretations? Or maybe a resource on the topic I could refer to? In general I do not want to assume Gaussian noise, and would prefer if I could produce the quadratic energy from a bound. In that regard I guess I could use the looser bound that I have above and drop terms that involve $g$ because it is unknown, but that seems like a rather handwavy justification. $\endgroup$ – lightxbulb May 27 at 14:05
  • 1
    $\begingroup$ The regularization term isn't about the noise (That's the fidelity term). The regularization, in MAP framework, is all about a prior. Namely the model of the data. In Image Processing the most popular model is the Piece Wise Smooth. Usually called Total Variation which is equivalent of having Laplace Distribution prior for the gradient of the image. Have a look on other posts of mine. I wrote about it frequently. $\endgroup$ – Royi May 27 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.