How should calibration points be distributed?

Question

I do camera calibration for a single camera with a chessboard pattern in opencv for the first time. The focus is fixed. I'm not sure how to do it, there seems to be many questions:

1. How many images should be used for calibration?
2. What pattern should be used? number of points, size of squares
3. What transformations should be applied to the calibration target? rotated, tilted, shifted...

I came to the conclusion that none of the questions above can be answered on their own, because they all depend on each other. Say for example taking a lot of images that are very similar to each other are intuitively not as good as having less images that are better distributed in the frame.

I took a step back and thought about what's going on. (I tried) Basically, there's a mathematical model of the camera and its aberrations (focal length, distortion, etc...) and the calibration is the process of collecting data to fit the model (its values) to the hardware at hand. This appears to be mostly statistics. In order to fit the model to the entire image well, sample data should be collected from the entire image.

I conclude that all questions above boil down to how should the detectable points of the pattern be distributed in the 3D space.

I'd like to know if this thought makes any sense.

I'm tired of reading handwavy suggestions like "the more images, the better", "at least arbitrary number of images should be used", etc. that do not provide a quantifiable measure for the quality of a calibration. Sure, there's a reprojection error, that tells how good the calibration fits to the provided calibration data, but not how good that data is suited for calibration in the first place.

I came to the conclusion that it makes more sense to evaluate the quality of a calibration by the distribution of the sample data used to create it. Makes sense?

---

Questions 1 and 2 above are thus combined to : What should be the overall density (or density distribution) of detected calibration points among all calibration images?

Say I take all detected points from all calibration images and place them in one image, I could see how many calibration points there are per pixel area $\text{px}^{2}$.If I have a lot of images with the pattern in the center of the image, I get a higher density there.

What would be a good minimum (baseline) value for the density of calibration points per pixel area? $0.1 \ \text{px}^{-2}$ ? $0.5 \ \text{px}^{-2}$ ?

Should the distribution of calibration points per pixel area be constant over the entire image? Or do I need a higher density (read: more sample values) at the edges for example?

---

Question 3 is asking for the distribution in the 3rd dimension. The calibration target with the pattern can be moved back and forth (along the optical axis) or tilted. Both transformations lead to calibration points in front or behind the plane that's in focus¹. Depending on the depth of field, corner points can be detected up to some distance from the focal plane. This adds a 3rd dimension to the space in which calibration points can be detected. How should calibration points be distributed with respect to distance from the plane of focus¹? Should most points be around the plane of focus¹ given that detection of points further away from it is not as good?

---

tl, dr;

In front of a camera, there's a 3D space enclosed by a frustum of a pyramid/cone in which calibration points can be detected. How should calibration points be distributed in that space in order to get a good calibration?

---

¹ With that I mean the plane of points that will be "in focus" in the image.

Answer 1

For that I have some guidelines, which I'd like to share. Even though some of these might not be valid for certain calibration software, for typical cases, they hold:

1) The entire pattern should be visible in all sequences.
2) At least 1/4th of the image should be occupied by the pattern.
3) Preferably, the lighting should be uniform everywhere on the pattern. 
(As little shadows as possible)
4) The light should reflect from the pattern to the camera (as much as possible). 
Thus, objects made up of metal are not suggested.
5) There should be minimal blur. If focus/depth-of-field is an issue, 
use a smaller plate or less tilts.
6) If you can't fix the camera, it shouldn't move at all. 
For extrinsic calibration, moving the camera after calibrating will ruin the entire 
calibration. Be careful about this as even small shakes or movements distort the accuracy.
7) Don't pre-process the calibration images. 
Even if you do very smart things, you might perturb the accuracy.
8) Ideally, have 15-50 images fitting the above criterion. 
This should give you 450-1500 points to optimize in total.
9) If you are printing the pattern on a piece of paper, then make sure that 
contrast between black-white squares is significant and no strange shape is appears on the paper. 
10) Don't bend the calibration pattern. Make sure that it is as rigid (and planar) as possible
11) If you are printing on an A4 paper, make sure you have some white space around the pattern, 
remember you will want to hold it somehow.
12) If you have distortion, use small circles, they are less affected from distortion. 
If you have low resolution but less distortion use larger circles. 
13) Start with low tilts and fronto-parallel images. 
Move to the tilts and rotations in all axes, gradually, covering the entire visible space.
If you have bias in the space coverage or distribution of points, 
then it means that some regions of the space will be better calibrated.
Keep this as equal as possible. Some variation can be tolerated.

I hope these help.