How does transformation matrix and camera matrix change when indexing pixels from -1 to 1 instead of 0 to N-1

Question

I'm trying to warp frames for view synthesis. Specifically, given a frame1 and poses of frame1 and frame2, I'm trying to warp frame1 to frame2's pose/view. For this, I'm indexing pixel locations as $[0,M-1]$ and $[0,N-1]$ for y and x directions respectively. My code is working perfectly (except occluded regions, which lead to holes).

I'm looking at this View Synthesis paper. The code released by authors instead index pixels in the range $[-1,1]$. Directly using transformation and camera matrices is giving invalid transformed coordinates. Can you please explain how I should change the transformation and camera matrices so that it works for pixel indices in the range $[-1,1]$?

More Details:
I've implemented the warping in python using the below equations. Given a pixel location $p_1=[x,y]^T$

$$\hat{p}_1 = [x,y,1]^T$$ $$ P_1 = Z K^{-1} \hat{p}_1$$ $$ \hat{P}_1 = [P_1^T,1]^T $$ $$ \hat{P}_2 = T \hat{P}_1 $$ $$ P_2 = \hat{P}_2[0:3] $$ $$ \hat{p}_2 = K P_2 $$ $$ p_2 = \hat{p}_2[0:2] / \hat{p}_2[2] $$

$p_1$ is the point in frame1 and $p_2$ is the corresponding point in frame2. $P_1$ is 3D world point in the view of frame1 and $P_2$ is the same 3D point in the view of frame2. $T$ is the transformation between views of frame1 and frame2. $Z$ is true depth of point $p_1$ in view of frame1. K is the camera intrinsic matrix. hat ($\hat{p})$ is used for projective coordinates.

I compute the $4 \times 4$ transformation matrix $T$ as follows: $$T = \begin{bmatrix} R & t \\ 0 & 1 \\ \end{bmatrix}$$ Where $R$ is the rotation matrix and $t$ is the translation.

The $3 \times 3$ camera matrix $K$ is given by $$K = \begin{bmatrix} \frac{W/2}{tan(hfov/2)} & 0 & W/2 \\ 0 & \frac{H/2}{tan(vfov/2)} & H/2 \\ 0 & 0 & 1 \\ \end{bmatrix}$$ Where $H=W=240$ are the height and width of the frames, $hfov=60,\ vfov=45$

In short, what changes should I make to matrices $T$ & $K$?

Answer 1

Before attempting to answer this, let's first see what camera transformation matrix does. Consider the 3d point

$$ P_1 = [X_1, Y_1, Z_1]^T $$

Pre-multiplying by camera matrix gives

$$ K \cdot P_1 = \begin{bmatrix} \frac{W/2}{tan(hfov/2)} & 0 & W/2 \\ 0 & \frac{H/2}{tan(vfov/2)} & H/2 \\ 0 & 0 & 1 \\ \end{bmatrix} \begin{bmatrix} X_1 \\ Y_1 \\ Z_1 \end{bmatrix} $$

$$ = \begin{bmatrix} \frac{W/2}{tan(hfov/2)} X_1 + W/2 \\ \frac{H/2}{tan(vfov/2)} Y_1 + H/2 \\ Z_1 \end{bmatrix} $$

As we can see, $W/2$ at $K[1,1]$ scales $x$ value and the $W/2$ at $K[1,3]$ adds an offset. Thus, if $x \in [-1,1]$ then by this scaling and offsetting, we get $x \in [0,W]$.

Conversely, when $x \in [0,W]$, we should use the camera matrix as defined earlier. Hence when $x \in [-1,1]$, we shouldn't add the scaling or offsetting. Thus, only the camera matrix needs to be changed to

$$K = \begin{bmatrix} \frac{1}{tan(hfov/2)} & 0 & 0 \\ 0 & \frac{1}{tan(vfov/2)} & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}$$