# AutoExposure with nonlinear camera: Non linear proportional PID?

Now I', wondering how to best implement a control algorithm for it. I tried using a PID controler but started with a simple P-control first. The problem is, that when using the equation delta_exposure = k*e whereas e = mean_grey_is - mean_grey_want = mean_grey_is - 127 and with k as a fixed value it is hard to define a good value for k because it depends on how big e is and on how big the actual mean_grey_is is. Assuming mean_grey_is is something like 150 (e=23) I'd only need a certain small change of my current ExposureTime. But when mean_grey_is something like 173 (e is twice as big) and I'm using the same k-value, the change of the ExposureTime obviously would also be twice as big, but to get a fast and quick controlling response, the change of ExposureTime would need to be a lot larger as you could see in the matlab figure above :(

How to do this?

EDIT2 @Jazzmaniac: believe me, I don't want you to get frustrated and I really tried and try to describe all the necessary information with completely detailed. Sorry if it wasn't enough so far... The camera has NO auto-exposure or auto-parameter calibration itself, thus does NOT change any parameters itself. I MYSELF did all the measurement above to measure the camera response depending on the different built-in camera parameters. I changed the parameters to a fixed set (v1_0,v2_0,t1_0,t2_0), then varied the ExposureTime while capturing a fixed/constant scene at constant illumination. After that I changed the camera parameters again to a different set (v1_1,v2_1,t1_1,t2_1) and again varied the ExposureTime while capturing the same constant scene at same illumination. That's how I got the different camera responses plotted above (you can clearly see that they heavily depend on the 4 parameters v1,v2,t1,t2 (I plotted for t1=t2=constant)). And still I believe that the camera response varies even for a constant set of parameters (v1,v2,t1,t2)=const with the illumination. And that's exactly the point I was asking about: When I capture the frames with varying ExposureTime in a rather dark room, clearly I need (starting at small exposuretimes) high ExposureTimes to reach the saturation at the mean greylevel 255 in the image. When I keep everything constant, turn on the lights in the room and do the SAME measurement again, then I clearly need much smaller ExposureTimes then before to reach the saturation in the image. Therefore, as stated and asked above already, the two camera responses are different concerning the x-axis-scaling. I know you did already state: "Regarding your question with the different x-axis scalings. That doesn't matter,[...]" -> That's what I'm constantly asking about: WHY doesn't it matter? When using the camera response model and calculating the inverse F^(-1)(Vmn) I get complete different values. Was this description detailed enough? I hope so because I tried my hardest to make my question understandable...

EDIT3: @Jazzmaniac: I tried to solve the optimization but I simply couldn't get it working. First of all: I neglected the coefficient $\frac{4}{MN}$ just to simplify it. Shouldn't be a problem I suppose. Therefore I can take the gradient as:

$$\frac{\partial P[V]}{\partial T} = 2 \sum_{m,n} \left(V_{m,n}-\frac{1}{2} \right)\cdot\left(F'(F^{-1}(V_{m,n}))\cdot F^{-1}(V_{m,n})\frac{1}{T} \right)$$

I'm stuck with different things about it:

1. Is the value of $V_{m,n}$ really only the actual value (because actually the value does iteratively depend on the exposuretime itself via $F(T*A_{m,n})$

2. Is the gradient meant as a function of $T$ as $\frac{\partial P[V]}{\partial T} = g(T)$ or is the $T$ in the gradient the current ExposureTime?

3. And how to optimize this now? Sorry I'm not that fit about numerical optimizations. I first thought usually (symbolic maths) you would try to calculate the minimum by solving the equation $\frac{\partial P[V]}{\partial T} = 0 \Leftrightarrow T = ...$ but that's obviously not simply possible is it?

Currently I have the following MATLAB Code:

obj = videoinput('gige', 1, 'Mono8');
src = getselectedsource(obj);
src.set('ExposureTime', 5000)

% Functionhandles (camera response f, its inverse f_inv and derivative f_diff)
f = @(x) polyval(fit.coeff, x);
f_inv = @(x) polyval(fit_inv.coeff, x);
f_diff = @(x) (f(x+2)-f(x-2))/4;

% Get image & normalize greyvalues to [0, 1]
IMG = getsnapshotfcn(obj);
IMGnorm = double(IMG(:))/255;

% The total penalty:
P = sum((IMGnorm-0.5).^2);

% The gradient of P in dependance of T as a function handle dPdT = g(T)
dPdT = @(T) 2*sum((IMGnorm-0.5).*f_diff(f_inv(IMGnorm)).*1/T.*f_inv(IMGnorm));

• Regarding your edit: It's not quite clear to me why the curves would saturate at gray levels that are not at or near 255. How exactly do you produce these curves? – Jazzmaniac Nov 25 '13 at 21:23
• well I have the same frustration because I get the feeling you're not really telling me all the details. So does the camera automatically change its parameters if the lighting conditions change? If not then your response doesn't change, you only move to a different part of the same response. I still think you should try to understand the internal intensity mapping of the camera and create a mathematical description for it with all the parameters. Then you don't have to calibrate at all. – Jazzmaniac Nov 27 '13 at 9:31
• Ok, so that clears things up and confirms the situation I was assuming. The variation in exposure time scaling is to be expected, because you're not really measuring against exposure time but intensity. But the model I gave you does not need an absolute measure of intensity, any scale proportional to intensity gives exactly the same result, because the scaling factors from the transfer function and its inverse cancel with the derivatives. Also note that nowhere in the final equation are there any references to the intensities, all is just pixel values, which is all you measure. – Jazzmaniac Nov 27 '13 at 12:46
• So you can either trust me on that assertion or you do the math yourself by introducing an arbitrary scaling factor to the intensities $A_{m,n}$ and propagate that change through the inverse and the derivatives and see them cancel in the end result. That scaling really only sets the unit of the intensity measurement, and we never made any use of that. So you don't have any problem. – Jazzmaniac Nov 27 '13 at 12:48
• have you seen that I have added some additional details to the answer? I thought you might find it helpful to understand how the answer relates to your previous approach. – Jazzmaniac Nov 27 '13 at 18:59

If you know the exact response of the camera, you can convert the brightness samples of each pixel to a linear intensity scale and perform the averaging there. That will make your whole problem intensity linear and should solve all your problems.

However, I would strongly recommend using a more advanced exposure algorithm. For example you could introduce a penalty function that gets large for very low intensities and very high intensities. Summing up the penalty of all pixels gives you a total penalty that you can minimize by changing the exposure time. If the penalty is nicely behaved (smooth, etc) then you can use fast converging optimization algorithms.

As requested I will briefly describe a possible implementation of the penalty method.

Your variables are the exposure time $T$, the pixel values $V_{m,n}\in [0,1]$ for $m\in 1.. M$ and $n \in 1..N$. You also have the invertible transfer function $F$ of the camera that maps real physical intensities to pixel values. This function is also differentiable and its derivative is $F'$.

With this we can express the pixel values in terms of the physical intensities $A_{m,n}$ and the exposure time $T$ as: $$V_{m,n} = F\left(T A_{m,n} \right)$$

Next we introduce a penalty functional that assigns high scores to very dark and very bright regions and low scores to well lit regions. Note that you can have more than just one free parameter (here exposure time) and a more complicated penalty model. I'll just go for the simplest sensible one. The penalty functional is

$$P[V] = \frac{4}{MN}\sum_{m,n} \left(V_{m,n}-\frac{1}{2}\right)^2$$

Now we can plug in our exposure model

$$P[V] = \frac{4}{MN}\sum_{m,n} \left(F(TA_{m,n})-\frac{1}{2}\right)^2$$

and see how the penalty changes with the exposure time

$$\frac{\partial P[V]}{\partial T} = \frac{4}{MN} \sum_{m,n} 2\left(F(TA_{m,n})-\frac{1}{2} \right)\cdot\left(F'(TA_{m,n})\cdot A_{m,n} \right)$$

This expression unfortunately depends on the unknown physical intensities $A$. But we can use the exposure model again, but inverted, to estimate those from the pixel values we have. That means with $T A_{m,n} = F^{-1}(V_{m,n})$ and therefore also $A_{m,n}=F^{-1}(V_{m,n})/T$ we get

$$\frac{\partial P[V]}{\partial T} = \frac{8}{MN} \sum_{m,n} \left(V_{m,n}-\frac{1}{2} \right)\cdot\left(F'(F^{-1}(V_{m,n}))\cdot F^{-1}(V_{m,n})\frac{1}{T} \right)$$

Summing up, you can use the pixel values $V_{m,n}$ from just one measurement to calculate both the penalty and its gradient, using even just an approximate model of your camera. With value and slope you can then start to iteratively search for the minimum penalty by descending the gradient in some clever way.

How exactly you do that is a matter of experimentation and taste. But with the square based model here I would suggest you use a parabolic approximation of the penalty function to guess the minimum.

Also note that I chose the penalty so that it is normalized between 0 and 1. So you can use that scale to decide if a found minimum is well exposed absolutely or not. This could be useful as a quality indicator for your images.

Addendum: For sake of completeness let me quickly show how this model generalizes your approach as described in the question.

Assume that the camera is in fact linear, and that the response function is therefore $$F(A)=s A$$ for some proportionality factor $s$. The derivative is then $F'(A)=s$ and the inverse $F^{-1}(V)=\frac{V}{s}$. With this special case, the gradient of the penalty becomes $$\frac{\partial P[V]}{\partial T}=\frac{8}{MN} \sum_{m,n}\left(V_{m,n}-\frac{1}{2} \right)\cdot \left( \frac{V_{m,n}}{T}\right)$$ and the intensity scaling factor $s$ cancels. Finding an extremum of this penalty function corresponds to choosing $T$ so that the gradient vanishes. $$\sum_{m,n}\left(V_{m,n}-\frac{1}{2} \right)\cdot V_{m,n}=0$$

Now let us quickly consider the trivial case where all $V_{m,n}=V$ are constant and not equal to zero. Then we can divide by $V$ and get that $V-1/2=0$ or $V=1/2$. With our convention of $V$ being in $[0,1]$ that means the solution is a constant flat image of 50% gray.

Of course all $V_{m,n}$ being identical is a very non practical assumption. So let's look at the more general case and expand the product in the sum. $$\sum_{m,n}V_{m,n}^2-\frac{1}{2} V_{m,n}=0$$ or written slightly different $$\sum_{m,n} \left( V_{m,n}-\frac{1}{2} \right)^2 + V_{m,n} - \frac{1}{4} - \frac{1}{2} V_{m,n}=0$$ and after collecting and scaling $$\sum_{m,n} 2\left( V_{m,n}-\frac{1}{2} \right)^2 + V_{m,n} = \frac{1}{2}$$ We can express that with the mean brightness $\langle V \rangle$ and the quadratic deviation from the 50% brightness $\Delta V$ as $$2 \Delta V + \langle V \rangle = \frac{1}{2}$$

For very small deviations $\Delta V$ we get the solution you proposed in your question, namely $\langle V \rangle = \frac{1}{2}$. But the more brightness variations you have in the picture the bigger the (strictly positive) deviation becomes, and the mean brightness must drop below $\frac{1}{2}$.

So this model generalizes your proposal for greater brightness variations, and that is very important. Because a picture that is black almost everywhere but has some small area with very high intensity would still be over exposed if you just make the mean brightness equal to 1/2. The huge variation will however make sure that the mean brightness will be chosen to be significantly lower, and save your result from over exposure!

• @bjoern, I'm a mathematical physicist, so I'm exactly on your page. Optimizing the penalty means you look for a global minimum of the penalty function (if the penalty is well designed then there's only one local minimum, or several very close ones that all work well). How exactly you look for that minimum depends on the optimization algorithm you employ. A smooth penalty function would suggest a gradient based algorithm. If your math skills are sufficient, you can derive the gradient analytically, if not you can just evaluate the penalty function at two nearby points and find the slope. – Jazzmaniac Nov 20 '13 at 20:17
• From there you can estimate a new exposure parameter that lies in descending direction, and depends on the value of the gradient. There are many optimization strategies, most with known convergence requirements and speeds. – Jazzmaniac Nov 20 '13 at 20:19
• @bjoern, No, you just know the theoretical connection between the penalty as a function of pixel values and the actual penalty value at your image sample. The rest is blind optimization. That means you move to where you expect the penalty to be lower and take another picture. Then iterate from the start. – Jazzmaniac Nov 21 '13 at 10:02
• @bjoern, you also know the direction if you find the gradient of the penalty function. You can either evaluate the gradient analytically using just one measurement (and known properties of the penalty function) or two nearby measurements and taking their numerical difference. There is absolutely no disadvantage compared to your method. – Jazzmaniac Nov 21 '13 at 14:33
• @bjoern, the camera response function is not all that important. The gradient will still point in the right direction if you get it wrong. However, you should be able to measure it by just increasing the exposure time for a constant motive, because the exposure time enters the total intensity linearly. So the graph you posted above IS already your transfer function! – Jazzmaniac Nov 21 '13 at 20:24