# How to normalise data for iterative computation of extrinsic camera matrix?

Hartley & Zisserman defend that data normalisation is an essential step prior to the estimation of geometric image transformations. The reasoning behind this, as stated in their book [1, Section 4.4.4] and in the accepted answer to this question, is added numerical stability, and accuracy when the data is noisy.

Suppose we are interested in computing the the projection matrix $$\mathtt{P} = \mathtt{K}\cdot[\mathtt{R} \mid \mathtt{t}]$$ from $$N$$ correspondences of 2D image points and 3D world points $$\{\mathbf{x}_i\leftrightarrow\mathbf{X}_i\}$$ by minimising the geometric error, i.e. a non-linear least squares iterative scheme, where $$\mathtt{K}$$ is the $$3\times 3$$ intrinsic camera matrix and $$[\mathtt{R} \mid \mathtt{t}]$$ is the $$3\times 4$$ extrinsic camera matrix (the camera pose). In this case [1, Section 7.1]:

each set of points should be translated so that their centroid is at the origin, and $$\mathbf{x}_i$$ (resp. $$\mathbf{X}_i$$) should be scaled so that their root-mean-squared distance from the origin is $$\sqrt{2}$$ (resp. $$\sqrt{3}$$).

The normalised points are thus obtained as \begin{align} \mathbf{\tilde{x}_i} &= \mathtt{T} \cdot \mathbf{x}_i\\ \mathbf{\tilde{X}_i} &= \mathtt{U} \cdot \mathbf{X}_i, \end{align}

where homogeneous coordinates are assumed. The transformation matrices are of the form \begin{align} \mathtt{T} &= \begin{bmatrix} s_1 & 0 & u_1\\ 0 & s_1 & u_2\\ 0 & 0 & 1 \end{bmatrix}\\ \mathtt{U} &= \begin{bmatrix} s_2 & 0 & 0&v_1\\ 0 & s_2 & 0&v_2\\ 0 & 0 & s_2 & v_3\\ 0 & 0 & 0 & 1 \end{bmatrix}. \end{align}

After the minimisation procedure, a normalised $$\mathtt{\tilde{P}}$$ is obtained, which is then used to get the camera matrix for the original (unnormalised) coordinates: $$$$\mathtt{P} = \mathtt{T}^{-1} \cdot \mathtt{\tilde{P}} \cdot \mathtt{U}.$$$$

Now, suppose we have the common case where $$\mathtt{K}$$ is known and hence the transformation to estimate is the pose matrix $$[\mathtt{R} \mid \mathtt{t}]$$. A simple manipulation of the perspective projection equation (dropping the subscripts for succinctness) yields the following: \begin{align} \mathbf{x} &= \mathtt{K} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathtt{K}^{-1} \cdot \mathbf{x} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathbf{y} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \rightarrow \mathtt{T}^{-1} \cdot \mathbf{\tilde{y}} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1} \cdot \mathbf{\tilde{X}}\\ \mathbf{\tilde{y}} &= [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathbf{\tilde{X}}, \quad \text{with} \quad [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}]=\mathtt{T} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1}.\\ \end{align}

This suggests that the same data normalisation scheme can be applied, and the problem has an analogous solution to the first case by minimising the geometric error in terms of $$\mathbf{y} = \mathtt{K}^{-1}\cdot \mathbf{x}$$, therefore removing $$\mathtt{K}$$ from the iteration.

The problem, however, lies in the relationship $$[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$$: when multiplying $$\mathtt{R}$$ by either $$\mathtt{T}$$ or $$\mathtt{U}$$, or by both, it ceases being a valid rotation matrix, and thus $$[\mathtt{R} \mid \mathtt{t}]$$ is not a valid pose.

Does this imply that the problem can only be solved in terms of estimating $$\mathtt{P}$$ if one desires to normalise the dataset? Or are there specific $$\mathtt{T}, \mathtt{U}$$ such that $$[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$$ produces a valid pose? In that case, does it still have the benefits of H&Z's isotropic normalisation?

[ 1 ] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision. Cambridge University Press.