Object detection using several images

Question

how do people combine different information (e.g., RGB and depth image or RGB and thermal image) to make their object detection algorithms better?

If the single modality images A and B are perfectly registered, I can think of:

1. Write one object detector working on A, write one working on B and then write one regressor that takes the outputs from A and B and learns a final output.
2. Instead of working with feature vector f to describe a certain region (e.g. a bounding box) in the image A one could learn a prediction model based on a feature vector F = [f;g] where g is the corresponding feature for image B.

These approaches do not make as much sense if the images are not nicely registered, but there is a certain amount of misalignment.

So again: how do people combine different modalities for object detection and in particular deal with misregistration?

Answer 1

Usually, people tend to go with the second option, since its a joint approach. In either case, the methods fusing these two data modalities (RGB+Depth etc) together are termed multi-modal methods.

Also, another 3rd option is not to fuse on the feature level but to jointly minimize an energy function or jointly voting for the correct hypothesis.

One successful algorithm which works on RGB+Depth is LineMOD. The publications are here and here. (Modified Line). LineMOD takes the second path, and fuses the depth and RGB features. It also has an OpenCV implementation (a basic one).