Camera calibration/pin hole camera model and working out 3d position

Question

I have a calibrated camera and have the intrinsic parameters. I also have the extrinsic parameters relative to a point (the world origin) on a planar surface in the real world. This point I have set as the origin in the real world coordinates [0,0,0] with a normal of [0,0,1].

From these extrinsic parameters I can work out the camera position and rotation in the world plane 3d coordinates using this here: http://en.wikipedia.org/wiki/Camera_resectioning

Now I have a second point which I have extracted the image coordinates for [x, y]. How do I now get the 3d position of this point in the world coordinate system?

I think the intuition here is that I have to trace a ray that goes from the optical center of the camera (which I now have the 3D position for as described above), through the image plane [x,y] of the camera and then through my real world plane which I defined at the top.

Now I can intersect a world coordinate 3d ray with a plane as I know normal and point on that plane. What I don't get is how I find out the 3d position and direction when it leaves the image plane through a pixel. It's the transformation through different coordinate systems that is confusing me.

Answer 1

If you have the extrinsics then it is very easy. Having extrinsics is the same as having "camera pose" and the same as having the homography. Check this post in stackoverflow.

You have extrinsics, also called camera pose, which is described as a translation and a rotation:

$\displaystyle Pose =\begin{bmatrix}R|t \end{bmatrix} = \begin{bmatrix}R_{11} &R_{12}&R_{13}&t_x\\R_{21}&R_{22}&R_{23}&t_y\\R_{31}&R_{32}&R_{33}&t_z \end{bmatrix} $

You can get Homography from Pose this way:

$\displaystyle H = \frac{1}{t_z}\begin{bmatrix}{R_{1x}}&{R_{2x}}&{t_x}\\{R_{1y}}&{R_{2y}}&{t_y}\\{R_{1z}}&{R_{2z}}&{t_z}\end{bmatrix}$

Then you can project your 2D points into the corresponding 3D points by multiplying the Homography by the points:

$p_{2D}=\begin{bmatrix}x &y &1\end{bmatrix}\quad$ add $\quad z=1\quad$ to make them homogeneous

$p_{3D}=H*p_{2D} $

$p= p / p(z)\quad$ Normalize the points

Answer 2

You have two options, use back projection or projection between two planes (homography).

With back projection you take a pseudo inverse of you camera matrix $P$ and multiply the result with your homogenous presentation of image point:

$$ P = K\begin{bmatrix}R & -R\textbf{C}\end{bmatrix} \\ \textbf{X}_{reprojected} = P^+\textbf{x} $$

Now you have a 3D line which travels trough the camera center $\textbf{C}$ and point $\textbf{X}$. If you want, you can convert this to some more easily dealt with presentation. For example with one point and direction vector (remember to normalize the homogenous coordinates $\textbf{V} = \omega\begin{bmatrix}X & Y & Z & 1\end{bmatrix}^T$ such that scale factor $\omega=1$ before the actual calculations):

$$ \textbf{u} = \textbf{X}_{reprojected}-\textbf{C} \\ \textbf{v} = \frac{ \textbf{u} }{\|\textbf{u}\|} \\ \textbf{L}(t) = \textbf{C} + t\textbf{v} $$

If you have plane $\Pi = \begin{bmatrix}\pi_1 & \pi_2 & \pi_3 & \pi_4\end{bmatrix}^T, \pi_1X + \pi_2Y + \pi_3Z + \pi_4 = 0$ you can solve the equation $\textbf{L}(t) = \Pi$ for $t$.

If you decide to use homography, you need to compute the $3\times3$ homography matrix $H$ which is defined as projection between the imaged plane and the plane of camera sensor:

$$ \textbf{X}_{plane} = \begin{bmatrix}X & Y & 0 & 1\end{bmatrix}^T \\ \textbf{x} = P\textbf{X}_{plane} = H\begin{bmatrix}X & Y & 1\end{bmatrix}^T $$

Now if you know $\textbf{x}$: $$ \textbf{X}_{plane} = H^{-1}\textbf{x} $$

If you did not compute the $H$ while calibrating the camera (probably using direct linear transformation, DLT), you can use following formulation:

$$ H = R + \frac{1}{d}\textbf{T}\textbf{N}^T $$

Where $d$ is the dinstance of the camera from the plane and $\textbf{T} = -R\textbf{C}$. (Ma, Soatto, Kosecká, Sastry - An invitation to 3-D Vision From Images to Geometric Models, p. 132)

Answer 3

You can't know the 3d position of the second point. It can be any of the points on the ray from your center of the camera until infinity.

You can do the following:

• Create a predefined 3d space which resembles the real life scene
• Get more points of images from a different angle, using the intersection of the rays from different angles, you can get an approximation of the 3d point.