# Why is the backprojection of the Middlesbury depth map "cone shaped"?

I wrote some come to backproject depth maps. Testing it with the Middlebury cone set, I get:

First of all it has noise, which I shouldn't get from the ground truth, but more importantly it seems like the points are going further from each other with the depth.

Can anybody explain this phenomenon?

My code:

  float baseline = 0.3;
float focalLength = 303.0;
float numerator = baseline * focalLength;

for (int i=10; i<groundTruthDisparity.rows-10; ++i) {
for (int j=10; j<groundTruthDisparity.cols-10; ++j) {
float distance = groundTruthDisparity.at<uchar>(i,j) / 4.0;

if (distance != 0 && !isnan(distance)) {
depthMap.at<float>(i,j) = numerator / distance;
}
}
}

for(int y=1; y<height-1; y++) {
for(int x=1; x<width-1; x++) {
float depth = depthMap.at<float>(y,x);

// backproject to world
cv::Matx21d homogeneous = cv::normalize((cv::Matx21d(x,y) - principalPoint) * (1.0 / focalDistance));
cv::Matx31d transformedPoint = cv::Matx21d(homogeneous(0), homogeneous(1), 1) * depth;

cloudPoints.push_back(transformedPoint);
}
}


The depth map looks like this (colored in HSV):

## 1 Answer

The reason that pixels get wider with depth is that the depth map works in projection space, so each pixel is basically shaped like a frustum, not like a box. Not sure where the noise is coming from but the depth map doesn't look perfectly clean either.

• I changed "wider" to "further from each other". The scanned scene was not cone shaped, so I suppose my reconstruction is wrong? I would expect to see the same width in the point cloud from front to back, don't know if I am explaining it well. Apr 10, 2015 at 23:36
• Did you render with orthographic projection? That'd be the only way that pixels wouldn't get wider (or separate!) With distance. I think the issue is that you need to make your pixels larger as they go into the distance. The exact wideness would depend on your original projection parameters. Your reprojection definitely does not look like orthographic projection, so the issue seems related to this sort of thing (mixing projection modes?) Apr 10, 2015 at 23:39
• PS this question will attract a lot of attention on the game Dev stack exchange site if you don't get enough info from here. Apr 10, 2015 at 23:42
• To see the point cloud I just use a viewer (Meshlab), which is using a 60 deg projection. I'm still not sure if the problem is in the visualization or in the creation of the depth values. This is the output I would want to have: youtu.be/36TkuvwlfR4?t=47s Apr 10, 2015 at 23:45
• Oh interesting. My diagnosis was wrong then. I think the noise is just the cracks between the pixels then. Reprojecting depth like this is not my forte at all but I think what you need to do is find a way to weld the corners of the pixels together. A naive solution would be to average the depth of the 4 pixels that meet at a corner, but I bet there are way better techniques that preserve the original structure better! Apr 11, 2015 at 0:05