Suggestions for creating 3D Bounding Boxes of Books on a Shelf from RGB-D Images

Question

I am new to image processing, so I beg your pardon if this is a trivial question. I am trying to use an RGB-D camera to detect 3D bounding boxes of books on a shelf. Here are some examples of possible scenarios: books on a shelf 1 or books on a shelf 2. I would like to infer the dimensions (length, width and height) and the pose of each book and of the shelf. So approximate each of the element in the scene with a 3D bounding box.

I would like to get suggestions on which computer vision or image processing techniques are best suited to implement a solution to my problem. I am thinking of YOLO 5 or the following paper:

Deng, Z. and Jan Latecki, L., 2017. Amodal detection of 3d objects: Inferring 3d bounding boxes from 2d ones in rgb-depth images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5762-5770).

Are there any better or easier alternatives or already implemented packages?

Thank you all very much!

Answer 1

Exploiting the 3D geometry is important here. From the depth image, you could get the 3D point cloud by backprojection. Once the point cloud (set of 3D points) is given, you might benefit from VoteNet:

Deep Hough Voting for 3D Object Detection in Point Clouds, Charles R. Qi, Or Litany, Kaiming He, Leonidas J. Guibas, ICCV 2019 https://arxiv.org/abs/1904.09664

If, in addition, the color (RGB) information is available, we could utilize within the same framework, this time maybe using ImVoteNet:

ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes, Charles R. Qi, Xinlei Chen, Or Litany, Leonidas J. Guibas, CVPR 2020 https://arxiv.org/abs/2001.10692

Both of the papers provide open source implementations. See here for VoteNet and this repository for ImVoteNet. Note that both of these are methods that require heavy supervision, i.e. large amounts of annotated training data. If such data is not available at your disposal, I would start looking into hand-crafted algorithms, e.g. descriptors or point pair feature detectors.

Hope this helps.