how do people combine different information (e.g., RGB and depth image or RGB and thermal image) to make their object detection algorithms better?

If the single modality images A and B are perfectly registered, I can think of:

  1. Write one object detector working on A, write one working on B and then write one regressor that takes the outputs from A and B and learns a final output.
  2. Instead of working with feature vector f to describe a certain region (e.g. a bounding box) in the image A one could learn a prediction model based on a feature vector F = [f;g] where g is the corresponding feature for image B.

These approaches do not make as much sense if the images are not nicely registered, but there is a certain amount of misalignment.

So again: how do people combine different modalities for object detection and in particular deal with misregistration?

  • $\begingroup$ This is an uneasy question in its whole generality. There is no single approach for "object detection" in monomodal images and combining information will probably be very ad-hoc. Regarding registration, one obviously have to use features from both modalities that can be unambiguously put in correspondence. A non-trivial task. (Think for instance that there is little relation between the distance to an object and its temperature). $\endgroup$
    – user7657
    Oct 8, 2015 at 9:38

1 Answer 1


Usually, people tend to go with the second option, since its a joint approach. In either case, the methods fusing these two data modalities (RGB+Depth etc) together are termed multi-modal methods.

Also, another 3rd option is not to fuse on the feature level but to jointly minimize an energy function or jointly voting for the correct hypothesis.

One successful algorithm which works on RGB+Depth is LineMOD. The publications are here and here. (Modified Line). LineMOD takes the second path, and fuses the depth and RGB features. It also has an OpenCV implementation (a basic one).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.