# Step by Step Camera Pose Estimation for Visual Tracking and Planar Markers

I have been working on the topic of camera pose estimation for augmented reality and visual tracking applications for a while and I think that although there is a lot of detailed information on the task, there are still a lot of confussions and missunderstandings.

I think next questions deserve a detailed step by step answer.

• What are camera intrinsics?
• What are camera extrinsics?
• How do I compute homography from a planar marker?
• If I have homography how can I get the camera pose?
• I'm fuzzy on the renormalization you do: 1. H is the homography found from the data using some procedure (say SVD). 2. inv(K)*H=A is the thing you work with here. Then you make q1 = a1/norm(a1) and q2 = a2/norm(a2) as orthonormal columns of a rotation matrix, and make q3=q1xq2... Then you take t/(something) to get the translation vector. How is it you can just divide q1 and q2 by possibly different things, and how do you pick what to divide t by? Or is the idea that the SVD procedure and multiplication by inv(K) give something close but not quite orthogonal/orthonormal rotation matrix, so th – user2600616 Aug 17 '13 at 3:53
• But how could I get 3D point (X,Y,1)? – waschbaer Dec 7 '16 at 10:25

It is important to understand that the only problem here is to obtain the extrinsic parameters. Camera intrinsics can be measured off-line and there are lots of applications for that purpose.

What are camera intrinsics?

Camera intrinsic parameters is usually called the camera calibration matrix, $K$. We can write

$$K = \begin{bmatrix}\alpha_u&s&u_0\\0&\alpha_v&v_0\\0&0&1\end{bmatrix}$$

where

• $\alpha_u$ and $\alpha_v$ are the scale factor in the $u$ and $v$ coordinate directions, and are proportional to the focal length $f$ of the camera: $\alpha_u = k_u f$ and $\alpha_v = k_v f$. $k_u$ and $k_v$ are the number of pixels per unit distance in $u$ and $v$ directions.

• $c=[u_0,v_0]^T$ is called the principal point, usually the coordinates of the image center.

• $s$ is the skew, only non-zero if $u$ and $v$ are non-perpendicular.

A camera is calibrated when intrinsics are known. This can be done easily so it is not consider a goal in computer-vision, but an off-line trivial step.

What are camera extrinsics?

Camera extrinsics or External Parameters $[R|t]$ is a $3\times4$ matrix that corresponds to the euclidean transformation from a world coordinate system to the camera coordinate system. $R$ represents a $3\times3$ rotation matrix and $t$ a translation.

Computer-vision applications focus on estimating this matrix.

$$[R|t] = \begin{bmatrix} R_{11}&R_{12}&R_{13}&T_x\\R_{21}&R_{22}&R_{23}&T_y\\R_{31}&R_{32}&R_{33}&T_z \end{bmatrix}$$

How do I compute homography from a planar marker?

Homography is an homogeneaous $3\times3$ matrix that relates a 3D plane and its image projection. If we have a plane $Z=0$ the homography $H$ that maps a point $M=(X,Y,0)^T$ on to this plane and its corresponding 2D point $m$ under the projection $P=K[R|t]$ is

$$\tilde m = K \begin{bmatrix} R^1 & R^2 & R^3 & t \end{bmatrix} \begin{bmatrix} X \\ Y \\ 0 \\ 1 \end{bmatrix}$$

$$= K \begin{bmatrix}R^1&R^2&t\end{bmatrix} \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix}$$

$$H = K \begin{bmatrix}R^1 & R^2 & t \end{bmatrix}$$

In order to compute homography we need point pairs world-camera. If we have a planar marker, we can process an image of it to extract features and then detect those features in the scene to obtain matches.

We just need 4 pairs to compute homography using Direct Linear Transform.

If I have homography how can I get the camera pose?

The homography $H$ and the camera pose $K[R|t]$ contain the same information and it is easy to pass from one to another. The last column of both is the translation vector. Column one $H^1$ and two $H^2$ of homography are also column one $R^1$ and two $R^2$ of camera pose matrix. It is only left column three $R^3$ of $[R|t]$, and as it has to be orthogonal it can be computed as the crossproduct of columns one and two:

$$R^3 = R^1 \otimes R^2$$

Due to redundancy it is necessary to normalize $[R|t]$ dividing by, for example, element [3,4] of the matrix.

• I think it is misleading to say that calibration is "easy and not the goal of CV". In usual case we also need to estimate the distortion parameters. Instead of self calibration I would recommend planar calibration (Zhang - A Flexible New Technique for Camera Calibration) as it is more flexible if separated calibration procedure can be done. You also have small error in "If I have homography how can I get the camera pose?" as you don't take into the account the calibration (H_{calib} = K^-1H). – buq2 Jul 1 '12 at 5:34
• camera pose from homography is wrong. There are several way to do it' some of them are highly non-trivial. – mirror2image Jul 5 '12 at 11:52
• I don'r see why it is wrong. I compute it this way and works. Why do you say it is wrong? – Jav_Rock Jul 5 '12 at 12:29
• You wrote in the last section that H^1 and R^1 and equal, but in the 3rd section you state that H=K[R T] which would mean that R^1 is actually K^-1H^1. But this is not strictly true as there is infinite number of H which will satisfy the equations and will cause problems when solving R^1, R^2 and T (the unknown scale). Your answer disregards robust intrinsic and distortion calibration and some of the equations are wrong for which reason this is not a good answer for the question. – buq2 Jul 5 '12 at 22:23
• Yes, I was missing the kalibration matrix in step three as I took this from my code and I multiply by K in a different function of the codes. – Jav_Rock Jul 6 '12 at 7:58

While explaining the two-dimensional case very well, the answer proposed by Jav_Rock does not provide a valid solution for camera poses in three-dimensional space. Note that for this problem multiple possible solutions exist.

This paper provides closed formulas for decomposing the homography, but the formulas are somewhat complex.

OpenCV 3 already implements exactly this decomposition (decomposeHomographyMat). Given an homography and a correctly scaled intrinsics matrix, the function provides a set of four possible rotations and translations.

The intrinsics matrix in this case needs to be given in pixel units, that means your principal point is usually (imageWidth / 2, imageHeight / 2) and your focal length is usually focalLengthInMM / sensorWidthInMM * imageHeight.

• What's a correctly scaled intrinsics matrix ? – Guig Jun 23 '17 at 5:38
• I have updated my answer. Please see above. – Emiswelt Jun 23 '17 at 8:33
• Hey @Emiswelt, isn't the focal length focalLengthInMM / sensorWidthInMM * imageWidth? Why you choose the height instead? – El Marce May 22 '18 at 2:32