I have a calibrated camera and have the intrinsic parameters. I also have the extrinsic parameters relative to a point (the world origin) on a planar surface in the real world. This point I have set as the origin in the real world coordinates [0,0,0] with a normal of [0,0,1].

From these extrinsic parameters I can work out the camera position and rotation in the world plane 3d coordinates using this here: http://en.wikipedia.org/wiki/Camera_resectioning

Now I have a second point which I have extracted the image coordinates for [x, y]. How do I now get the 3d position of this point in the world coordinate system?

I think the intuition here is that I have to trace a ray that goes from the optical center of the camera (which I now have the 3D position for as described above), through the image plane [x,y] of the camera and then through my real world plane which I defined at the top.

Now I can intersect a world coordinate 3d ray with a plane as I know normal and point on that plane. What I don't get is how I find out the 3d position and direction when it leaves the image plane through a pixel. It's the transformation through different coordinate systems that is confusing me.


3 Answers 3


If you have the extrinsics then it is very easy. Having extrinsics is the same as having "camera pose" and the same as having the homography. Check this post in stackoverflow.

You have extrinsics, also called camera pose, which is described as a translation and a rotation:

$\displaystyle Pose =\begin{bmatrix}R|t \end{bmatrix} = \begin{bmatrix}R_{11} &R_{12}&R_{13}&t_x\\R_{21}&R_{22}&R_{23}&t_y\\R_{31}&R_{32}&R_{33}&t_z \end{bmatrix} $

You can get Homography from Pose this way:

$\displaystyle H = \frac{1}{t_z}\begin{bmatrix}{R_{1x}}&{R_{2x}}&{t_x}\\{R_{1y}}&{R_{2y}}&{t_y}\\{R_{1z}}&{R_{2z}}&{t_z}\end{bmatrix}$

Then you can project your 2D points into the corresponding 3D points by multiplying the Homography by the points:

$p_{2D}=\begin{bmatrix}x &y &1\end{bmatrix}\quad$ add $\quad z=1\quad$ to make them homogeneous

$p_{3D}=H*p_{2D} $

$p= p / p(z)\quad$ Normalize the points

  • $\begingroup$ what do you mean by p(z) here ? $\endgroup$
    – Bilal
    Mar 30, 2020 at 9:03

You have two options, use back projection or projection between two planes (homography).

With back projection you take a pseudo inverse of you camera matrix $P$ and multiply the result with your homogenous presentation of image point:

$$ P = K\begin{bmatrix}R & -R\textbf{C}\end{bmatrix} \\ \textbf{X}_{reprojected} = P^+\textbf{x} $$

Now you have a 3D line which travels trough the camera center $\textbf{C}$ and point $\textbf{X}$. If you want, you can convert this to some more easily dealt with presentation. For example with one point and direction vector (remember to normalize the homogenous coordinates $\textbf{V} = \omega\begin{bmatrix}X & Y & Z & 1\end{bmatrix}^T$ such that scale factor $\omega=1$ before the actual calculations):

$$ \textbf{u} = \textbf{X}_{reprojected}-\textbf{C} \\ \textbf{v} = \frac{ \textbf{u} }{\|\textbf{u}\|} \\ \textbf{L}(t) = \textbf{C} + t\textbf{v} $$

If you have plane $\Pi = \begin{bmatrix}\pi_1 & \pi_2 & \pi_3 & \pi_4\end{bmatrix}^T, \pi_1X + \pi_2Y + \pi_3Z + \pi_4 = 0$ you can solve the equation $\textbf{L}(t) = \Pi$ for $t$.

If you decide to use homography, you need to compute the $3\times3$ homography matrix $H$ which is defined as projection between the imaged plane and the plane of camera sensor:

$$ \textbf{X}_{plane} = \begin{bmatrix}X & Y & 0 & 1\end{bmatrix}^T \\ \textbf{x} = P\textbf{X}_{plane} = H\begin{bmatrix}X & Y & 1\end{bmatrix}^T $$

Now if you know $\textbf{x}$: $$ \textbf{X}_{plane} = H^{-1}\textbf{x} $$

If you did not compute the $H$ while calibrating the camera (probably using direct linear transformation, DLT), you can use following formulation:

$$ H = R + \frac{1}{d}\textbf{T}\textbf{N}^T $$

Where $d$ is the dinstance of the camera from the plane and $\textbf{T} = -R\textbf{C}$. (Ma, Soatto, Kosecká, Sastry - An invitation to 3-D Vision From Images to Geometric Models, p. 132)


You can't know the 3d position of the second point. It can be any of the points on the ray from your center of the camera until infinity.

You can do the following:

  • Create a predefined 3d space which resembles the real life scene
  • Get more points of images from a different angle, using the intersection of the rays from different angles, you can get an approximation of the 3d point.
  • $\begingroup$ Hang on. I can find the 3d world point that the 3d ray intersects the planar surface, surely? As I know a 3d world coordinate and a 3d world normal of that plane.....the 3d point I am trying to find is the point it intersects that planar surface!! (Sorry I feel my explanation wasn't good enough) $\endgroup$
    – Cheetah
    Mar 30, 2012 at 9:39
  • $\begingroup$ What do you mean with planar surface? The image plane, or the plane of the zero world coordinates? In the case of the latter, you can compute the intersection, but that means your 3d scene is not 3d, but 2d :) (because it's a plane). $\endgroup$
    – Geerten
    Mar 30, 2012 at 9:45
  • $\begingroup$ Yeah sorry, that just didn't occur to me. I understand what you are saying, it just doesn't really make sense to me visually. So, yes my scene is actually "2d" because I have the image plane and I have the real world plane, which the real world origin [0,0,0] lies on and has a real world normal of [0,0,1], therefore every point that lies on this real world plane is in the form of [x, y, 0]. I know I can compute the intersection though ax+by+cz+d=0, but this is what I am having trouble with. (To be continued in next comment) $\endgroup$
    – Cheetah
    Mar 30, 2012 at 10:09
  • $\begingroup$ I have a ray which starts at my camera centre/origin which I have the real world [x, y, z] for and a real world normal [nx, ny, nz]. I need to fire a ray from this point that intersects the image plane at [u, v] and then intersects the real world plane at [x, y, 0] (it is this x, y that I want to get). What I am having trouble with is the first bit, the intersection with the image plane. I can't see how I do that? $\endgroup$
    – Cheetah
    Mar 30, 2012 at 10:15
  • $\begingroup$ You might want to look at: en.wikipedia.org/wiki/Line-plane_intersection $\endgroup$
    – Geerten
    Mar 30, 2012 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.