This is a conceptual question I am having difficulty to understand due to my limited knowledge of computer vision: If a pinhole calibration or any calibration is a mapping from world coordinates to image coordinate, is it possible to invert this mapping? I understand most of the times camera matrix is 3x4 so non invertible, but is it possible to have a square camera matrix. Thanks
If you have a single calibrated camera, and you have its 3x4 camera matrix, then you can map image points to the world points on the z=0 plane. This assumes that you know that your image point does indeed correspond to a world point on that plane.
So, you cannot map any image point to its corresponding world point using a single image from calibrated camera. To do that, you would need two views from different viewpoints, and you would need to know the 3D rotation and translation between the views.
It is possible to invert the mapping, but projecting a 2D image point into 3D world won't give you a single point but a ray, which is the locus of all 3D world points that map to the same point in the image plane.
This is why the info of depth is being lost during the process of image formation. As Dima said you can get back this info using 2 or more views and doing triangulation which consists in getting the intersection of the 2 rays.
Calibration can be done with different types of targets for every camera, the most important thing is to ensure a right recognition: the more precise is your calibration (so the knowledge of your camera parameters) the more precise will be 3D reconstruction.