Object detection using several images

how do people combine different information (e.g., RGB and depth image or RGB and thermal image) to make their object detection algorithms better?

If the single modality images A and B are perfectly registered, I can think of:

1. Write one object detector working on A, write one working on B and then write one regressor that takes the outputs from A and B and learns a final output.
2. Instead of working with feature vector f to describe a certain region (e.g. a bounding box) in the image A one could learn a prediction model based on a feature vector F = [f;g] where g is the corresponding feature for image B.

These approaches do not make as much sense if the images are not nicely registered, but there is a certain amount of misalignment.

So again: how do people combine different modalities for object detection and in particular deal with misregistration?

• This is an uneasy question in its whole generality. There is no single approach for "object detection" in monomodal images and combining information will probably be very ad-hoc. Regarding registration, one obviously have to use features from both modalities that can be unambiguously put in correspondence. A non-trivial task. (Think for instance that there is little relation between the distance to an object and its temperature). – Yves Daoust Oct 8 '15 at 9:38