how do people combine different information (e.g., RGB and depth image or RGB and thermal image) to make their object detection algorithms better?
If the single modality images A and B are perfectly registered, I can think of:
- Write one object detector working on A, write one working on B and then write one regressor that takes the outputs from A and B and learns a final output.
- Instead of working with feature vector
fto describe a certain region (e.g. a bounding box) in the image A one could learn a prediction model based on a feature vector
F = [f;g]where
gis the corresponding feature for image B.
These approaches do not make as much sense if the images are not nicely registered, but there is a certain amount of misalignment.
So again: how do people combine different modalities for object detection and in particular deal with misregistration?