I want to solve for the extrinsics by using direct linear transformation on corresponding 3D LIDAR points and 2D camera points. I already have the intrinsics.

Problem is, points behind the camera gets re-projected as well (see picture below).

enter image description here

So I constrain to only points "in front of the camera", i.e z > 0. The problem is, on different trials where different sets of points are used, the produced extrinsic matrix produces differing axes. Sometimes, constraining z > 0 gives the right results (centre part of image), whereas other times I need z < 0. So the question is, how do I constrain the Z axes of the camera to be sticking out of the camera?

def with_intrinsic(points2d, points3d, intrinsic):
    cam1_K_inverse = np.linalg.inv(intrinsic)

    #direct linear transformation calibration, assumes no intrinsic matrix
    assert points2d.shape[0] >= 3
    assert points3d.shape[0] == points2d.shape[0]

    A = []

    points2d_homo = []
    for u,v in points2d:
        points2d_homo.append([u, v, 1])

    points2d_homo = np.array(points2d_homo).T #columns to be data points

    points2d_inv = np.dot(cam1_K_inverse, points2d_homo).T
    assert points2d_inv.shape == (points2d.shape[0], 3)
    assert points2d_inv[0, 2] == 1

    for idx in range(points2d.shape[0]):
        x3d, y3d, z3d = points3d[idx]
        u, v, _ = points2d_inv[idx]

        A.append([x3d, y3d, z3d, 1, 0, 0, 0, 0, -u * x3d, -u * y3d, -u * z3d, -u])
        A.append([0, 0, 0, 0, x3d, y3d, z3d, 1, -v * x3d, -v * y3d, -v * z3d, -v])

    A = np.array(A)

    U, D, VT = np.linalg.svd(A)
    M = VT.T[:, -1].reshape((3, 4))

    error = get_reprojection_error(points2d, points3d, intrinsic, M)
    logging.debug("error with_intrinsic: %s", error)

    return M
  • 1
    $\begingroup$ It's a polar transformation. So, why don't you constrain the angle, $\theta$? You have only 180 degrees of rotation as far as I understand. $\endgroup$ Mar 15, 2016 at 17:45

1 Answer 1


The question is a bit unclear (and am sorry to say, the code even more so). Assuming your 3d-2d point correspondences are correct, all you'd need to do after solving the DLT is to rearrange the orientation of the axes while leaving their directions and the parity of the frame unchanged (i.e. ensure that the determinant of the rotation matrix is +1).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.