What is the purpose and advantage of using a checkerboard than a 3D calibration object for camera calibration ?
With a 3D calibration object (e.g. a transparent cuboid like a table), I can take one image with a camera, then find the corners of the cuboid, which gives me 8 2D-3D correspondences.
Since a projective camera matrix $P$ has 11 DoF, I have enough equations to obtain $P$.
Now with a checkboard given one image, I obtain the internal camera matrix and external kamera matrix.
However, the external camera matrix describes the change of basis from the checkerboard coordinate system to the camera coordinate system, and thus changes from image to image. Thus is the idea of the checkerboard to take multiple images to obtain a more accurate estimation of the internal camera parameters, which should be same across all taken images ?
So for image i, we have 2D points $p_{i}$ with corresponding 3D points $P_{i}$ (on the checkerboard). Let $K_{i}$ be the internal camera matrix, $E_{i}$ be the external camera matrix and $M = \begin{pmatrix}1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0\end{pmatrix}$ the projective camera model.
So I am assuming that the optimization problem that is being solved is
$$\min_{K,E_1,\ldots,E_m} \sum_{j=1}^{m}\sum_{i=1}^{n} || (K M E_{j}) P_{i}- p_{i,j}||^2$$
so that we obtain one internal camera matrix that fits to all images ?
Accordingly, am I right that it is necessary to do both, the calibration with the checkerboard to get an accurate estimate of $K$ and then the calibration with the 3D calibration object to obtain the extrinsic camera matrix, so that the 3D calibration object defines the world coordinate system ?