Making a 3D point cloud from multiple RGB-D images

Question

I have multiple (4 kinect) cameras that give RGB-D (color and depth) information of the same scene from different points of view. I want to make a 3D point cloud out of these depth maps. I can get a 3D point cloud for each camera using a function available via the SDK. but my question is how can I merge these points in the same coordinate system to get a single point cloud? what are the available methods? are there any libraries available that perform this in python ?

My best guess is to use one of the cameras' coordinate system as reference and transform every point into that coordinate system. but how can I figure out the representation of one camera's coordinate system in another camera's coordinates so I can perform the rotation and translation ?

I am new to computer vision so please give some references I can use for the methods you mention.thank you.

Answer 1

Simple answer: Use LiveScan3D. Should be easy enough.

Longer answer: You are looking for a way to 'register' multiple point clouds. This is an extremely well studied problem in computer vision. In your case I would consider two scenarios:

1. Target-based (photogrammetry-like): Place some targets in the world such as markers (see OpenCV Aruco markers). Estimate the marker pose in each camera image (the camera center is set to be the origin). Once the marker pose is computed in all views, you could transform each 3D cloud into the coordinate frame of one of the cameras (choice is arbitrary).

2. Markerless: The 3D geometry itself is usually rich enough to relate the views together. This is called registration. You are indeed looking into a multiview registration problem. The typical pipelines follow two main steps: (i) pairwise registration; you can use for example FGR for this. Then these individual pairwise transformation estimates have to be linked together. If your view-graph in which there exist an edge for each pair of cameras seeing common structure, is simple, you could compute the minimum spanning tree and chain the transformations along the tree to compute all the coordinate frames with respect to the first one (free to choose).

Note that, with the advances in deep learning we can now solve this problem in an end-to-end manner, efficiently:

Gojcic, Z., Zhou, C., Wegner, J. D., Guibas, L. J., & Birdal, T. (2020). Learning multiview 3D point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1759-1769). https://arxiv.org/pdf/2001.05119.pdf

The first option requires you to physically place fiducial tags in the scene but is easier to implement. The second option is more automatic, but might be harder to get to work.