# How to warp a pixel patch to another camera

I have two cameras with rotation and translation starting from the world origin and a patch in the first camera's reference coordinate system. I want to warp this patch into the second camera's reference coordinate system. My idea is to transform every pixel like this:

$$T=T_2-T_1$$

$$R=R_1^{T}R_2$$

$$x' = \left[(x K_{inv} R) + T\right]\;K$$

Where $R$ and $T$ are the transformation matrices between the cameras and $K$ is the affine transformation matrix that describes the camera properties.

Is this correct? I know that when you go from 3D to 2D you need to project, but that matrix is not invertible, so you cannot really do it here? Am I missing something really big?

• I don't think you can do what you are trying to do. You would need 3D coordinates for each pixel, and as far as I can tell you have 3D coordinates for each camera but not for the pixels. Getting 3D coordinates for the pixels requires you to know how far each is from the camera, and you can't measure that with just one camera. You would need to identify the same patch on each camera and use the angles and the 3D positions of the cameras to determine the 3D position of each pixel. With that you could warp the 3D pixel coordinates from one camera to the other.
– JRE
Sep 10 '14 at 10:07
• Unless the patch is flat, you can't perform the mapping, by lack of depth information. Is it flat ?
– user7657
Apr 23 '18 at 20:22

$x K_{inv}$, with $x(3) = 1$, produces the pixels as 3D points "stuck on the screen" of the first camera. After transforming you get the same 3D points, but in the coordinate frame of the second camera, and then you project in that camera. So, overall, what you are doing is looking at the first camera from the point of view of the second one.

I suspect that is not what you want....

• Yeah, that's what I was wondering. I cannot find resources on how warping really works. I think I'm taking it from the wrong angle: I have the 3D points, the patches associated to them and the initial camera where the patches where taken from, maybe I can use the normal of the patch and the rigid transformation together to reproject the patch in the new camera? Jun 11 '14 at 17:24
• You need to clarify what you mean by "patch": a portion of a plane? A quadrangle on a plane? In either case the transformation between the two images of it is a homography. Jun 12 '14 at 3:10
• In this case a patch is a 8x8 pixels feature descriptor. I found the homography from this paper: robots.ox.ac.uk/~lav/Papers/molton_etal_bmvc2004/… but I don't know how to find the normal. Supposedly it is some translation of the Z axis? Jun 12 '14 at 16:16
• The homography they use is: $K_{1}R[n^{T}x_{p}I-tn^{T}]K^{-1}_{0}u_{0}$ Jun 12 '14 at 16:28
• Afraid I have no idea of what it means to "warp" a feature descriptor. Jun 12 '14 at 23:54