0
$\begingroup$

Hartley & Zisserman defend that data normalisation is an essential step prior to the estimation of geometric image transformations. The reasoning behind this, as stated in their book [1, Section 4.4.4] and in the accepted answer to this question, is added numerical stability, and accuracy when the data is noisy.

Suppose we are interested in computing the the projection matrix $\mathtt{P} = \mathtt{K}\cdot[\mathtt{R} \mid \mathtt{t}]$ from $N$ correspondences of 2D image points and 3D world points $\{\mathbf{x}_i\leftrightarrow\mathbf{X}_i\}$ by minimising the geometric error, i.e. a non-linear least squares iterative scheme, where $\mathtt{K}$ is the $3\times 3$ intrinsic camera matrix and $[\mathtt{R} \mid \mathtt{t}]$ is the $3\times 4$ extrinsic camera matrix (the camera pose). In this case [1, Section 7.1]:

each set of points should be translated so that their centroid is at the origin, and $\mathbf{x}_i$ (resp. $\mathbf{X}_i$) should be scaled so that their root-mean-squared distance from the origin is $\sqrt{2}$ (resp. $\sqrt{3}$).

The normalised points are thus obtained as \begin{align} \mathbf{\tilde{x}_i} &= \mathtt{T} \cdot \mathbf{x}_i\\ \mathbf{\tilde{X}_i} &= \mathtt{U} \cdot \mathbf{X}_i, \end{align}

where homogeneous coordinates are assumed. The transformation matrices are of the form \begin{align} \mathtt{T} &= \begin{bmatrix} s_1 & 0 & u_1\\ 0 & s_1 & u_2\\ 0 & 0 & 1 \end{bmatrix}\\ \mathtt{U} &= \begin{bmatrix} s_2 & 0 & 0&v_1\\ 0 & s_2 & 0&v_2\\ 0 & 0 & s_2 & v_3\\ 0 & 0 & 0 & 1 \end{bmatrix}. \end{align}

After the minimisation procedure, a normalised $\mathtt{\tilde{P}}$ is obtained, which is then used to get the camera matrix for the original (unnormalised) coordinates: \begin{equation} \mathtt{P} = \mathtt{T}^{-1} \cdot \mathtt{\tilde{P}} \cdot \mathtt{U}. \end{equation}

Now, suppose we have the common case where $\mathtt{K}$ is known and hence the transformation to estimate is the pose matrix $[\mathtt{R} \mid \mathtt{t}]$. A simple manipulation of the perspective projection equation (dropping the subscripts for succinctness) yields the following: \begin{align} \mathbf{x} &= \mathtt{K} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathtt{K}^{-1} \cdot \mathbf{x} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \mathbf{y} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathbf{X}\\ \rightarrow \mathtt{T}^{-1} \cdot \mathbf{\tilde{y}} &= [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1} \cdot \mathbf{\tilde{X}}\\ \mathbf{\tilde{y}} &= [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathbf{\tilde{X}}, \quad \text{with} \quad [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}]=\mathtt{T} \cdot [\mathtt{R} \mid \mathtt{t}] \cdot \mathtt{U}^{-1}.\\ \end{align}

This suggests that the same data normalisation scheme can be applied, and the problem has an analogous solution to the first case by minimising the geometric error in terms of $\mathbf{y} = \mathtt{K}^{-1}\cdot \mathbf{x}$, therefore removing $\mathtt{K}$ from the iteration.

The problem, however, lies in the relationship $[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$: when multiplying $\mathtt{R}$ by either $\mathtt{T}$ or $\mathtt{U}$, or by both, it ceases being a valid rotation matrix, and thus $[\mathtt{R} \mid \mathtt{t}]$ is not a valid pose.

Does this imply that the problem can only be solved in terms of estimating $\mathtt{P}$ if one desires to normalise the dataset? Or are there specific $\mathtt{T}, \mathtt{U}$ such that $[\mathtt{R} \mid \mathtt{t}] = \mathtt{T}^{-1} \cdot [\mathtt{\tilde{R}} \mid \mathtt{\tilde{t}}] \cdot \mathtt{U}$ produces a valid pose? In that case, does it still have the benefits of H&Z's isotropic normalisation?

[ 1 ] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision. Cambridge University Press.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.