Suppose I have two cameras, c1 and c2 that are securely installed and do not move. What I want is a matrix T, that takes a pixel of an image from camera c1 and calculates the according pixel in an image from camera c2.
I now wonder:
Do I need to calibrate both c1 and c2 to get their intrinsic and extrinsic parameters and additionally find a relationship between the coordinate system of camera c1 and c2
or
Can I calculate a registration between an image of c1 and an image of c2 using interest points (e.g., by MATLABs cpselect and fitgeotrans functions) and use this registration matrix also for future images?
I believe that a generally suitable transformation matrix T between the two cameras' images is only possible to get if you first calibrate both cameras (in the sense that you calculate the extrinsic and intrinsic parameters). In my feeling it is otherwise impossible to reflect the fact that we are dealing with 3D objects. T will only be able to transform the 2D images. Is this correct? If so, can you elaborate more on why this is the case?