# Multiple-image dense point cloud reconstruction with camera extrinsics/intrinsics

I have the following data available:

• Images from different angles
• X, Y, Z, pitch, yaw, and roll for each camera/image, together with the timestamp
• Lens distortion parameters and intrinsic camera parameters such as focal length and CCD width

On my dataset, OpenSfM produced poor reconstructions and I realized that it tries to figure out camera extrinsics, which I already know.

My question: I am trying to produce a dense point cloud. Is there a name for this problem when I have such prior information? Which publicly available libraries can help me accomplish this task? Can someone point me in the right direction?

Thanks!

First, a warm welcome to SE! Basically, you have a calibrated 3D reconstruction problem. The typical approach follows a 5-stage pipeline:

1. Identify 2D features in each image along with the associated descriptors. Algorithms such as SURF, SIFT or AKAZE are heavily used and are available in many vision libraries such as OpenCV.
2. Match the extracted keypoints across the whole dataset by using their descriptors. Once again, this is a well studied topic. The keypoints and their multiview association will be called feature matches and will form the basis for 3D reconstruction.
3. Extend the pairwise matches to the multiview case by linking the cameras to one another, associating the ones that have enough common points. In MATLAB, the relevant function would be findTracks.
4. Use an N-view triangulation algorithm (typically a multi-ray intersection) to find an initial estimate of the 3D points. In MATLAB one could use the triangulateMultiview to solve for the multiview intersection. Normally, prior to this stage, one would solve for the camera poses, but since you have it already, I'm skipping this part.
5. Now use bundle adjustment to refine the 3D structure. I am not sure how you obtain the poses, but it is strongly recommended to also adjust the camera extrinsics additionally to the points. Libraries such as Ceres Solver are capable of solving for both structure and motion, simultaneously.

Optionally, a post-processing step consisting of densification can be run. PMVS could be of help here:

Accurate, dense, and robust multi-view stereopsis. Yasutaka Furukawa and Jean Ponce. IEEE Trans. on Pattern Analysis and Machine Intelligence, 32(8):1362-1376, 2010.

It is another possibility to benefit from existing multi-stage SfM pipelines, such as OpenMVG. You would simply replace the OpenMVG poses by your camera poses and keep the rest of the workflow the same. This way, you get some outlier treatment and post-processing steps for free. MATLAB also has an SfM toolbox that you could easily test.