# How to warp a pixel patch to another camera

I have two cameras with rotation and translation starting from the world origin and a patch in the first camera's reference coordinate system. I want to warp this patch into the second camera's reference coordinate system. My idea is to transform every pixel like this:

$$T=T_2-T_1$$

$$R=R_1^{T}R_2$$

$$x' = \left[(x K_{inv} R) + T\right]\;K$$

Where $R$ and $T$ are the transformation matrices between the cameras and $K$ is the affine transformation matrix that describes the camera properties.

Is this correct? I know that when you go from 3D to 2D you need to project, but that matrix is not invertible, so you cannot really do it here? Am I missing something really big?

• I don't think you can do what you are trying to do. You would need 3D coordinates for each pixel, and as far as I can tell you have 3D coordinates for each camera but not for the pixels. Getting 3D coordinates for the pixels requires you to know how far each is from the camera, and you can't measure that with just one camera. You would need to identify the same patch on each camera and use the angles and the 3D positions of the cameras to determine the 3D position of each pixel. With that you could warp the 3D pixel coordinates from one camera to the other.
– JRE
Sep 10, 2014 at 10:07
• Unless the patch is flat, you can't perform the mapping, by lack of depth information. Is it flat ?
– user7657
Apr 23, 2018 at 20:22

$x K_{inv}$, with $x(3) = 1$, produces the pixels as 3D points "stuck on the screen" of the first camera. After transforming you get the same 3D points, but in the coordinate frame of the second camera, and then you project in that camera. So, overall, what you are doing is looking at the first camera from the point of view of the second one.
• The homography they use is: $K_{1}R[n^{T}x_{p}I-tn^{T}]K^{-1}_{0}u_{0}$ Jun 12, 2014 at 16:28