How can we combine 2D image with 3D image for the same sense?

Question

We have 3D point cloud data captured by a laser scanner and need to register it with 2D images to get further information. In our case, the 2D image and 3D points are not overlapping and therefore we need to preprocess them so as to extrinsically calibrate the 3D scanner with the 2D image sensor (camera).

For the same modality, such as 2D-2D, this is done with good results using standard techniques, but for different data space, we still face challenges in this task.

These kind of images can be used in auto-driving cars and other applications.

Answer 1

Formally speaking, you would like to extrinsically calibrate the laser scanner to the 2D image. I have taken the liberty to edit your question to reflect that.

Here is how my initial approach would be:

1. Calibrate the intrinsics of the 2D camera. For that, just use OpenCV. You should store the intrinsic parameters: focal lengths, principal point and distortion coefficients.

2. Take an object which you could easily locate in 3D and 2D. Such an object could be any 3D model for instance if you can perform pose estimation both in 3D and 2D. Because it would be difficult to do this for free-form objects, I would propose to use a sphere-object which appears to be ellipse-like in the camera, and a sphere in the laser scanner. For instance, take an unused pilates-ball.

3. Place the pilates ball in the overlapping area of the point cloud and the 2D image. Then capture multitudes of images (at least about 30) by moving the ball in the overlapping region to various locations. At each location, make sure that the ball occupies about $1/4^{th}$ of the image and record the 2D and 3D frames.

4. Detect the ellipses in all the images, i.e. for each image, one ellipse. Use any open source tool, for instance 1D-hough transform to do that. I strongly suggest to refine the ellipse to subpixel accuracy, for instance via this method (source code). Record the center of the sphere (ellipse) in 2D $\{(x_i,y_i)\}$.

5. Detect the spheres in all of the laser point clouds. Again, many packages do exist. Even MATLAB has one method. Record the detected centers of the 3D sphere $\{(X_i,Y_i,Z_i)\}$. Again, a non-linear refinement of the sphere in 3D can help you in getting more accuracy. Check this one out for a complete pipeline on this if you like more sophisticated approaches.

6. Now you have an easy problem to solve: You have established 2D-3D coorespondences in the form $\{(x_i,y_i)\}\sim\{(X_i,Y_i,Z_i)\}$, have the calibration information of 2D camera and thus, all you need to do is to solve the PnP problem. Again many open and closed source software exists such as OpenCV's solvePnP or MATLAB's estimateWorldCameraPose.

The result will be a rotation matrix $R$ and a translation vector $t$ that maps the 3D scanner's coordinate frame to the image domain, thereby solving the registration problem.

I hope instructions are clear and this helps.

Note that, many things can be done to make this procedure easier, more robust, more accurate and etc. This created quite a big interest from academia, and if you dig a bit, you will come across a lot of sources. Nevertheless, for the beginning, the aforementioned procedure is pretty easy to implement.

Answer 2

If the geometry of the two cameras is known, you can solve this problem relatively easily.

You have your 3D scanner set up at a known location, producing triplets of points $x_s,y_s,z_s$. These points could be anything, from converted $x,y,z$ to rotation, elevation, distance. But, at the end of the day, each one of these points bears information about a block of space infront of the 3d scanner.

The same block of space is now also viewed by a typical camera, set up at a known (different) location which only produces $x_c,y_c, brightness$ points.

So, the point now is to transform the $x_s,y_s,z_s$ so that they match their $x_c,y_c$. This is relatively easy, if you already know the camera's transformation matrix that takes points from the world space (your scanner's space) to image space (your camera's output).

For more information, please see this link, this link, this link, this link or more generally look for anything related to "camera calibration".

Hope this helps.

EDIT: I am using the term "Geometry" here to denote that at the very least, the two cameras should be imaging the same space and at the very least, you should know the orientation of one with respect to the other and their relative positions.

If you don't know any of this information, you could still try to co-register the two but with considerably much more effort.

The very least your 3D scanner should support in this case would be to return actual colour for every point in 3D space.

The basic idea would be to calibrate the optical camera to recover its transformation matrix. So, now you have a way of transforming 3D point clouds to 2D images that look almost the same as if they were shot by the optical camera.

But here comes the difficult part: Because you don't know the two cameras relative positions and orientations, you would have to try a huge amount of combinations to find the one that would match your original camera configuration.

Furthermore, there will be combinations of locations / orientations that cannot be investigated because of shadowing. So, co-registration only makes sense if the optical camera's position and orientation is WITHIN the spherical space that the 3D scanner is imaging. Even then, there might be limits to how many equivalent 2D views you could produce depending on the geometry of the objects being imaged.

Happy to ammend this response with more specific information if more details are added to the question. What would be really useful would be and extract of the actual data.

Hope this helps.