# Math for projecting 3d world point to VR image coordinates

Suppose, I've a 3d world point $$P$$ and I want to project it onto the image plane of camera, I can do it as follows $$M = \begin{bmatrix} R & t \\ 0 & 1 \\ \end{bmatrix}$$

$$p = K \cdot M \cdot P$$ where $$K$$ is the camera intrinsic matrix and $$p$$ gives my image point in homogeneous coordinates.

Now, I want to do the same thing for VR images (360° or 180° degree). Can you provide me any links for the math involved here?

I've attached two sample images here where the camera moves forward. As you can see the change in the frames is not the same when camera moved forward in normal videos. I believe it uses some other coordinate system (may be spherical) and also the images are such that when viewed closely on a headset, it gives 3d feeling.

Your images look like being shot with an angular fisheye projection . Also, to account for epipolar geometry of stereo images, you should include the fundamental matrix into the transform engine of your geometry pipeline.

To render a 3D scene in OpenCV with a fisheye camera, one uses a fisheye camera model. Regardless of a graphics software you are using, you may find useful a description of this model's math in an OpenCV documentation.

EDIT:

When using an application like Blender, you need not construct matrices and call API methods. You just select values for dialog controls. For example, to select a fisheye lens, you do it in accord with the following ref.: https://docs.blender.org/manual/en/2.79/render/cycles/camera.html :

CAMERA

. Lens

. Type : Fisheye

• Thank you so much for the references to start with. I've the blend file that generated the images in question. Do you know where I can check in blender the above information? – Nagabhushan S N Jul 15 at 11:13
• docs.blender.org/manual/en/2.79/render/cycles/camera.html : CAMERA Lens Type : Fisheye – V.V.T Jul 15 at 11:52
• Thanks. It is Panoramic Equirectangular in mine. I found it under Properties > Object Data Properties > Camera > Lens > Type. – Nagabhushan S N Jul 15 at 12:40