# What is energy in image processing?

There are several web articles on the topic, but, after having read them, I still don't know what "energy" is, in the context of computer vision (CV) and image processing (IP). It is related to optimisation, given that, apparently, there are the so-called "energy minimisation problems".

Why is it called "energy" (that is, why the word "energy")? I read it is because it is related to the concept of "energy" in physics, but assume that I have no idea of what energy in physics is. How would you explain what energy is in IP and CV? What is the intuition behind it?

Why do we care about energy? Given that there are the energy minimisation problems, we want to minimise energy, in such cases. I have also read that there are energy maximisation problems, so sometimes we also want to maximise energy.

I have read that it has different meanings depending on the context, but I just want to know its meaning in image processing and computer vision, that is, in the context of images.

• Votes and best answer validation are useful for Stack Exchange – Laurent Duval Oct 10 at 20:24
• @LaurentDuval Sorry. It's been a while. I will try to look at the question and answers again ;) – nbro Oct 11 at 0:08

The usage of the term "energy" in image processing context has historical reasons. It can be mapped to "brightness" or "intensity".

Imagine you have a (for matters of simplicity) greyscale image, that you want to transmit. Historically, that was done via some kind of analog modulation, where, simply speaking, dark "pixels", low intensity meant "low voltage" and bright "pixels", high intensity meant "high voltage". When looking at a CRT television, you could take the word "energy" quite literally, as an all black image will produce significantly lower energy consumption than an all white image.

The term "engergy minimisation" is usually employed, when trying to optimize some kind of algorithm or codec by comparing the difference of original signal and encoded-decoded signal. The codec would be perfect, if this difference was all zeros, so the target is "to minimise the energy" of the difference signal. Usage of "energy maximisation" is analog.

So, think of "energy" as "intensity" or "brightness" in the context of image processing.

• Can you give a concrete example of an "energy function", in the context of image processing, that needs to be minimised or maximised? – nbro Apr 8 at 13:16
• Take a bmp image. Convert it to JPEG, then back to bmp. Do the difference of the images and take a look at it. The higher you set the quality of the JPEG encoder, the nearer the difference will be to an all black image. – Max Apr 8 at 13:21

For computer vision, or any engineering problem, energy is used because it can be relatable to physical quantities.

In simple circuits, we know that power is current times the voltage: $$P(t) = I(t)V(t)\tag{1}$$ and energy is the integral of this over time: $$E = \int_{-\infty}^{+\infty} P(t) dt$$ If we assume Ohm's law $$V(t) = I(t) R$$ holds then (1) can be rewritten as $$P(t) = I^2(t)R$$ so the energy is then $$E = \int_{-\infty}^{+\infty} I^2(t) R dt$$ This form looks very much like the squared error in some optimization problems.

The nice thing about squared error problems is they often have provably optimal solutions. As a result, there are many good algorithms around for solving them.

In image processing, as in many parts of engineering, people know this background and so often apply least squares techniques to the problems that they come across: designing filters, template matching, etc.

Because they use least squares techniques, and because of the relation of those techniques to the idea of energy, they use the word energy when they talk about such problems - even if it's really just an artifact of the solution mechanism they're using.

Energy is a useful, and powerful, proxy In many tasks, when one is uncertain about the data, it is tempting to average them to reduce noise. What most people have forgotten is that "averaging values", ie summing values and dividing the sum by the cardinal, is an algorithm, not a definition. A proper optimization-like definition to the average is:

the quantity that minimizes the squared distance to observed values.

In other words, the average is:

$$\hat{m} =\arg \min \sum_k (x_k-m)^2$$

If you add weights ($$\sum_k w_k(x_k-m)^2$$), you recover all linear filters, that bear a close relationship to Gaussian statistics, and of course orthogonality. Minimizing an energy is relatively tractable, as the differentiation of a square may turn into linear equations.

In image processing, one feels that sum approximations (with parameters $$\theta$$) $$A_\theta(I)$$ to an image $$I$$ can be appropriate: approximating the image by its mere averages, or by a limited sum of weighted functions. This can be cast into an energy minimization problem:

$$\hat{I} =\arg \min (I-A_\theta(I))^2$$

As this cost function is minimized by the image itself, it is not sufficient to account for image degradation: noise, smearing, blur, motion, etc. Thus, many works use a least energy fidelity term, and often add a penalty that accounts for image specificity: total variation, sparsity, positivity, etc.