# What is the state-of-the-art method for dealing with illumination issues in background subtraction?

I have been struggling to find a well-cited approach to this. Essentially, I am experimenting with different background subtraction algorithms, but none seem to perform well when there are significant lighting changes (say, from a window in the scene getting significant glare at specific times).

The question is simple (relatively): what are the state-of-the-art methods for background subtraction with sudden illumination changes? My scenario is surveillance cameras.

The standard approach is to use a mixture of Gaussians to model the various laws that command the intensity of a pixel. In this approach, a pixel can be assigned several normal laws (with different mean and variance), each of them will model its value under different conditions. Of course, only 1 Gaussian needs to be active at a time.

While the maximum number of Gaussians in the mixture is a (fixed) parameter, the parameters of each Gaussian are learned online. You can find an example implementation in the OpenCV library.

A few years ago, someone (sorry, i can't find the name anymore) proposed an interesting alternative approach: using variations of the image gradient orientation instead of variations of the light intensities. The gradient orientation has the advantage of being contrast invariant, making it more robust to illumination changes. Intuitively, it works because the gradient orientation is tied to the shapes in the image, not to their color or luminosity.

This is not a "well-cited" approach or state of the art for background subtraction in any case. However, your second sentence said that you were experimenting with different approaches, so I think what I have to say will still have some value.

## General on the approach:

What I'm proposing is a contrast-invariant image representation, called the Tree of Shapes, or Level Line Tree (there were more names, unfortunately, they still did not settle on a commonly accepted name).

The general idea in that proposed application is to build up a contrast-invariant-ish representations of a couple of images $I_x$ and $I_y$, and then look for differences. The advantage is that you can get, as an output, parts present in $I_y$ but not in $I_x$ (or vice versa) as opposed to most simple image comparisons, where the result is all the differences, i.e. everything present in one of the images and not the other.

## Possible way to use with security cameras:

If you application is security camera, then I guess you can get a few ground-truth images in good conditions (uniformish lighting, low occlusions). Then, you could build a Tree of Shapes from your ground-truth image, and then use it to detect new objects present in current images. The trees should be similar, because they are not based on global gray levels of pixels, but rather on "local contrast": the main question driving the tree building process is is it brighter/darker than the surroundings, and not how bright/dark is it.

## Literature:

Relevant for your particular application would be the last couple of pages from this paper:

You can find some relevant references in my answer here, but it seems to me now that the short overview I did for that answer is far from complete.

The authors of that particular paper also published a booklet in LNM series about that particular tree:

And finally, those approaches were more theoretical than practical until just a few days ago, because the worst-case construction time was quadratic in the number of image pixels. Just yesterday, a quasi-linear algorithm was presented which finally makes the Tree of Shapes usable in various applications: