0
$\begingroup$

Hartley & Zisserman defend that data normalisation is an essential step prior to the estimation of geometric image transformations. The reasoning behind this, as stated in their book [1, Section 4.4.4] and in the accepted answer to this question, is added numerical stability, and accuracy when the data is noisy.

Suppose we are interested in computing the the projection matrix $\mathtt{P} = \mathtt{K}\cdot[\mathtt{R} \mid \mathtt{t}]$ from $N$ correspondences of 2D image points and 3D world points $\{\mathbf{x}_i\leftrightarrow\mathbf{X}_i\}$ by minimising the geometric error, i.e. a non-linear least squares iterative scheme, where $\mathtt{K}$ is the $3\times 3$ intrinsic camera matrix and $[\mathtt{R} \mid \mathtt{t}]$ is the $3\times 4$ extrinsic camera matrix (the camera pose). In this case [1, Section 7.1]:

each set of points should be translated so that their centroid is at the origin, and $\mathbf{x}_i$ (resp. $\mathbf{X}_i$) should be scaled so that their root-mean-squared distance from the origin is $\sqrt{2}$ (resp. $\sqrt{3}$).

The normalised points are thus obtained as \begin{align} \mathbf{\tilde{x}_i} &= \mathtt{T} \cdot \mathbf{x}_i\\ \mathbf{\tilde{X}_i} &= \mathtt{U} \cdot \mathbf{X}_i, \end{align}

where homogeneous coordinates are assumed. The transformation matrices are of the form \begin{align} \mathtt{T} &= \begin{bmatrix} s_1 & 0 & u_1\\ 0 & s_1 & u_2\\ 0 & 0 & 1 \end{bmatrix}\\ \mathtt{U} &= \begin{bmatrix} s_2 & 0 & 0&v_1\\ 0 & s_2 & 0&v_2\\ 0 & 0 & s_2 & v_3\\ 0 & 0 & 0 & 1 \end{bmatrix}. \end{align}

After the minimisation procedure, a normalised $\mathtt{\tilde{P}}$ is obtained, which is then used to get the camera matrix for the original (unnormalised) coordinates: \begin{equation} \mathtt{P} = \mathtt{T}^{-1} \cdot \mathtt{\tilde{P}} \cdot \mathtt{U}. \end{equation}

Now, suppose we have the common case where $\mathtt{K}$ is known and hence the transformation to estimate is the pose matrix $[\mathtt{R} \mid \mathtt{t}]$. A simple manipulation of the perspective projection equation (dropping the subscripts for succinctness) yields the following: \begin{align} \mathbf{x} &= \mathtt{K} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathtt{K}^{-1} \cdot \mathbf{x} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathbf{y} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \rightarrow \mathtt{T}^{-1} \cdot \mathbf{\tilde{y}} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1} \cdot \mathbf{\tilde{X}}\\ \mathbf{\tilde{y}} &= [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathbf{\tilde{X}}, \quad \text{with} \quad [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}]=\mathtt{T} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1}.\\ \end{align}

This suggests that the same data normalisation scheme can be applied, and the problem has an analogous solution to the first case by minimising the geometric error in terms of $\mathbf{y} = \mathtt{K}^{-1}\cdot \mathbf{x}$, therefore removing $\mathtt{K}$ from the iteration.

The problem, however, lies in the relationship $[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$: when multiplying $\mathtt{R}$ by either $\mathtt{T}$ or $\mathtt{U}$, or by both, it ceases being a valid rotation matrix, and thus $[\mathtt{R} \mid \mathtt{t}]$ is not a valid pose.

Does this imply that the problem can only be solved in terms of estimating $\mathtt{P}$ if one desires to normalise the dataset? Or are there specific $\mathtt{T}, \mathtt{U}$ such that $[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$ produces a valid pose? In that case, does it still have the benefits of H&Z's isotropic normalisation?

[ 1 ] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision. Cambridge University Press.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.