Beyond cross-eyed 3D (VR application)

Question

Making static 3D image is very simple - we just take image pair with something like this:

then we can see 3D image by putting the left and right eye image side by side and crossing eyes or overlapping the images and using color filters for each eye. So far so good.

VR headsets (e.g. Oculus Rift) now allows seeing the 3D image directly.

However, I would like to incorporate slight movements and tilting of the head. Of course, this would immediately spoil the 3D effect.

First idea for better capture is using more cameras (e.g. 2x2 or 3x3 array) to acquire necessary viewpoints (e.g. top and bottom camera is necessary to reconstruct 3D when viewed with head leaned on one side). But how to interpolate between these views based on shifted and rotated head pose? Is depth/3D reconstruction necessary or are there any computational photography techniques to generate appropriate left/right eye images (for a given viewpoint/pose)?

I found various articles regarding depth reconstruction, 3D reconstruction and light fields, but nothing leads me toward the goal of interpolating the views.

Ultimately, I am planning to create a panorama viewer for VR glasses (something like krpano's web viewer supporting Oculus Rift). However, the viewer can account for limited head movements (pan, tilt) and the 3D impression does not hold for leaning, slight head shifts and correct for different eye-to-eye distance.

Any pointing to appropriate articles will be appreciated, thanks!

Answer 1

I have finally found a great overview video on the topic from Richard Szeliski: "Reflections of Image-Based Modelling and Rendering"

The research started with stereo and depth estimation. The depth map can be used to "warp" image in 3D to give it some perceived depth. This does not create parallax effects though.

Another approach called Layered Depth Images used segmentation to several depth layers each having its own depth map. This allows parallax effect to occur, but extra care must be taken to matte foreground and background layers during rendering.

Fast 3D reconstruction of the sorrounding is possible using primitives (boxes, cylinders) matched to scene points. The 3D reconstruction is tedious and requires many images and extensive walk around.

Then, more image-based (rather than geometry-based) approach are presented, which are Lumigraphs and Light Fields. These allow some level of reconstruction of the plenoptic function and solve for rays crossing given viewing plane.

Finally, Concentric Mosaics is the no. 1 solution I have been looking for. A camera travelling around circle (tangential direction) is able to capture 3D plenoptic function of the scene. Each point thus can be viewed from different angles and the viewer can move inside the circle, see parallax, reflections, transparency and occlussion.

Extension is possible by adding support for rays in vertical direction - this will lead to capturing 4D plenoptic function from a camera moving on sphere (tangetial direction) or by using a spherical rig of many cameras.

Further advances in the field are expected with the introduction of plenoptic cameras (e.g. Lytro) that can capture the 4D plenoptic function directly. Some work on making plenoptic mosaics is already there (e.g. "Panorama Light Field Imaging").