Where to place origin when warping an image using 3D rotation and translation?

Suppose, I've two images taken by a camera at two different locations. The two locations are related by a Rotation and Translation. So, points $$X1$$ in first frame will be at $$M \cdot X1$$ where

$$M = \begin{bmatrix} R & T \\ 0 & 1 \end{bmatrix}$$

Now suppose, I have image1 and I want to warp it to image2 using $$R, T$$. I have ground truth depth map corresponding to image1. To do this, I will represent each pixel in image1 as a vector. First I'll convert it to 3D point by multiplying Depth (in homogeneous coordinates) and inverse of camera matrix. Then I apply the transformation $$M$$ and then project it back to image space by multiplying by camera matrix and then normalizing the 3rd coordinate to 1.

But how and where do I fix the origin in the image1?