A teammate and I were discussing how best to get a 3D coordinate point (X, Y, Z) in world units from an RGB-D depth image. I'm using an intel real sense D435.
From how I have researched it using the camera intrinsics, and the depth of pixel I want I can get the corresponding (X, Y) coordinates by mapping the 2D point to 3D world coordinate accounting for camera distortion.
Could this mapping from 2D to 3D also be completed by using the camera's known field of view and the depth of a pixel? His argument is that since if we are looking at a flat wall from a known depth then we should know based on the camera's FOV what the length and height of the image in the world coordinate system. Will this always be the case?