Why is the backprojection of the Middlesbury depth map "cone shaped"?

Question

I wrote some come to backproject depth maps. Testing it with the Middlebury cone set, I get:

First of all it has noise, which I shouldn't get from the ground truth, but more importantly it seems like the points are going further from each other with the depth.

Can anybody explain this phenomenon?

My code:

  float baseline = 0.3;
  float focalLength = 303.0;
  float numerator = baseline * focalLength;    

  for (int i=10; i<groundTruthDisparity.rows-10; ++i) {
    for (int j=10; j<groundTruthDisparity.cols-10; ++j) {
      float distance = groundTruthDisparity.at<uchar>(i,j) / 4.0;

      if (distance != 0 && !isnan(distance)) {
        depthMap.at<float>(i,j) = numerator / distance;
      }
    }
  }

  for(int y=1; y<height-1; y++) {
    for(int x=1; x<width-1; x++) {
      float depth = depthMap.at<float>(y,x);

      // backproject to world
      cv::Matx21d homogeneous = cv::normalize((cv::Matx21d(x,y) - principalPoint) * (1.0 / focalDistance));
      cv::Matx31d transformedPoint = cv::Matx21d(homogeneous(0), homogeneous(1), 1) * depth;

      cloudPoints.push_back(transformedPoint);
    }
  }

The depth map looks like this (colored in HSV):

Answer 1

The reason that pixels get wider with depth is that the depth map works in projection space, so each pixel is basically shaped like a frustum, not like a box. Not sure where the noise is coming from but the depth map doesn't look perfectly clean either.