What Is the Correct Way to Apply Richarson Lucy Deconvolution to Luminance Data?

Question

My question concerns the Richardson–Lucy deconvolution algorithm, which is described in Richardson's original paper - William Hadley Richardson - Bayesian Based Iterative Method of Image Restoration. I am interested in applying it in the context of a raw image converter for digital photography. The RawTherapee discussion that motivated this post can be found in PIXLS.US forum: Quick question on RT Richardson Lucy implementation.

By using a Poisson noise model, the original paper assumes the algorithm is being applied to the linear data recovered from the sensor, which in particular is not gamma corrected any does not have any additional nonlinear tone mapping applied. A Poisson random variable is no longer Poisson after a nonlinear transformation, so the model no longer applies to such data.

My understanding is that if we simply demosaic the data from the Bayer array on the sensor, then apply deconvolution to the resulting RGB data, we are in good shape because it is linear.

However, it seems traditional to apply the deconvolution only to the luminance data. (I'm not totally sure why, or whether it is advisable, so please consider this a sub-question.) RawTherapee does this, for example, by converting the image to a LAB color space and deconvolving the L channel.

But there's the issue. It doesn't seem like L is linear in the number of photons captured. I found some data on Jim Kasson's Blog - The Zone System and digital cameras that suggests reflectance percentages of (12.5, 25, 50, 100) correspond to L values of (42.4, 57.0, 75.8, 100). Possibly this conversion depends on the illuminant being used; I am not well-versed enough in color science to say. However, the point is that the mapping is definitely non-linear.

So, is there a practical way to apply the Richardson–Lucy algorithm to a "linearized" version of the luminance data? Is this even desirable? I am not sure what the best practices are here, so any references you could provide would be greatly appreciated. I could not find any mention of this issue in the references I looked at.

Answer 1

I think the answer is in Jim Kasson's Post (The Zone System and Digital Cameras) you linked to:

If you are only moderately familiar with the Zone System, you may be asking yourself about the utility of recording light brighter than 100% reflectance. The 100% reflectance calibration assumes a matte subject with perfectly Lambertian reflectance. The higher light levels occur when the subject has some specularity or gloss to it. Water is a classic, and photographically-important example. Metals, glass, plastics, skin, stone and many paints are also often far from Lambertian. So we often need the headroom to accommodate those materials.

All of the above assumes that the camera is calibrated using UniWB, and that the UniWB hack works perfectly to predict raw file values. However, UniWB never achieved much penetration among digital camera users, and I think the number of users is waning today as cameras get more and more dynamic range, so it’s worthwhile looking at what happens with widely-used camera setups.

So the model to of Reflectance and the actual number of photons the sensors is exposed to are linear only if the camera is behaving according to UniWB Model (See also Guillermo Luijk - UniWB Tutorial) and the material used is indeed perfectly Lambertian.

Both probably doesn't hold and create some non linear effects.

CMOS / CCD Sensors are indeed linear in their response (At least before they are clipped).

So, given a RAW data with Linear Demosaicing (Pay attention that some algorithms may be non linear in this step) you can indeed apply the Lucy Richardson Convolution on the data. Pay attention that LAB Color Space is a non linear transformation of the RGB colors (even without the Gamma Correction).
So I'd apply the deconvolution in the following order:

1. Demosaic RAW Data.
2. Convert data into YCbCr Color Space (Or similar) which decompose the colors into luminosity and color data with Linear Transformation.
3. Apply the deconvolution on the Y channel.
4. Convert back to RGB based color space.

Answer 2

Richardson Lucy does not need to necessarily work in linear space.

It works by minimizing a log-likelihood function, so as far as it is concerned it does not matter whether the data is an array of photoelectrons e, xe, (xe)^y, or similar, with x and y being constants: minimizing the log of any of those will result in the 'same' solution in e-, ADU (ADU = e- times gain x) or DN after gamma has been applied (ADU^y with y = gamma).

Issues arise when the array is in a domain that results in a different minimum being found. For instance L* and sRGB would work fine except for the linear toe portion: with L*, y would be 1/3, with sRGB, y = 1/2.4 - except in the linear toe. All bets are off of course once you start doing non-exponential tone mapping. Some more detail in the blog post Strolls with my Dog - ELEMENTS OF RICHARDSON-LUCY DECONVOLUTION by Jack of Alma Photos.