# Backprojection of 2D image on 3D model

I am trying to implement the paper "Face Reconstruction in the Wild". In that I use a template 3D model and rotate it to align it with the orientation of the face in the 2D image. Now I want to project the 2D image onto the rotated 3D model. Could you please help me regarding how to do this. The paper gives no references for this so I think there might be an obvious solution I'm missing.

In section 2 of the paper it is mentioned that "Our image formation model assumes weak perspective projection, relatively distant illumination, and Lambertian reflectance" (emphasis is mine).

All three assumptions are making things much easier in various points of the algorithm further down the line but "weak perspective projection" makes the point of reprojecting the images on the model particularly easy.

A three dimensional projection is some $f: R^3 \rightarrow R^2$. There are essentially two types of it, orthographic and perspective. In an orthographic projection, points are projected vertically to the viewing plane. In a perspective transformation, there still is a viewing plane, but the actual rendered space shape is now the viewing frustum. Its shape (which maps to characteristics of a "camera") defines the final position of a 3D point on the 2D viewing plane.

A weak perspective transformation is essentially an orthographic but it involves a scaling too. What you scale with is the "average" distance of the object from the camera. So, given $x,y,z$ a point on some 3D mesh, its projection is roughly $P_x = \frac{x}{z}$. The bigger the $z$, the larger the scaling, the closer the point appears to the "ground line" (or horizon). When projecting "objects" (i.e collections of points linked by lines forming a mesh), you substitute the $z$ for the $z_{object}$ which is the average of all $z$ coordinates of all the points that make up the mesh. In this way, you still project vertically to the viewing plane but a far away object has a large $z_{object}$ which results in larger scaling of its $P_x, P_y$ coordinates which results in the object appearing smaller and closer to the horizon. Obviously, here, the object has to be somewhat coherent. A ball is ideal. A cube too. If you tried to render the star destroyer using this technique along other objects, then its perspective would look odd as its shape is such that it includes parts that extend very close to the camera but would have to be divided by the average $z_{object}$ of the whole mesh which is far, far away. Therefore, the "nose" of the destroyer would have to appear much bigger than what it will be rendered as.

This might seem like a digression but hopefully the spaceship example was illuminating on the characteristics of the weak perspective transformation because they are useful in pose matching.

Let's return to your case: The authors are assuming "weak perspective transformation". Essentially, they are saying that they assume that all portrait pictures have been acquired from far away enough to diminish the perspective effect of the camera lens because of the "relief" of the face (nose closer than eye socket). And that's great, because it means that you can use your fiducial points to pick up the orientation and scale of the face. You can then rotate (and scale) the 3D model and then project orthographically the image on to the 3D template. "Orthographically" as in, put the image on a plane, rotate it to the estimate pose around the 3D model and then extend lines vertically from the image / viewing plane towards the model. Once the line "touches" (intersects) the mesh, note down the $z$ position of that point (this now gives you the "depth" dimension you lost during taking the picture) and assign the image's pixel value to that point on the mesh (thus, projecting the image on the mesh).

Hope this helps.

• Hey. Just one more question. In section 2.4 of the same paper, the author uses S_t which contains the albedo and the surface normals of the template. Since the template was a 3D point cloud, I was able to get the surface normals using MATLAB but I did not understand how to get the albedo. Could you please help me with this. – malreddysid Oct 31 '16 at 23:36