Suppose, I've two images taken by a camera at two different locations. The two locations are related by a Rotation and Translation. So, points $X1$ in first frame will be at $M \cdot X1$ where
$$ M = \begin{bmatrix} R & T \\ 0 & 1 \end{bmatrix} $$
Now suppose, I have image1 and I want to warp it to image2 using $R, T$. I have ground truth depth map corresponding to image1. To do this, I will represent each pixel in image1 as a vector. First I'll convert it to 3D point by multiplying Depth (in homogeneous coordinates) and inverse of camera matrix. Then I apply the transformation $M$ and then project it back to image space by multiplying by camera matrix and then normalizing the 3rd coordinate to 1.
But how and where do I fix the origin in the image1?
