What transformations to go from top-down fisheye to occupancy (ground texture)

Question

My goal is to warp a top-down fisheye view to end up with a flat square 'texture' that could be applied to a flat plane and would represent the 'real' floor eg a 1024x1024 texture would map to a 4 meter by 4 meter of floor in the room (ignore the other 'items'). Or even 2 meter by 2 meter around the center.

In other words I just want to flatten the floor in the image and project it 'orthographically' so that each one of those checkerboard squares is exactly the same size. If the whole floor was covered in checkers, with no other items, my output would be flat checkerboard crop of 4 meters by 4 meters.

camera is imx 219 200 FOV about 1.3 meters off the ground pointing straight down. (well if it were pointing exactly straight down the red down would be on the green dot)

I am looking for python opencv implementation, but if someone at least knows what it is that I need to do/measure to achieve it, it will be super helpful.

I have not seen any other examples of fisheye pointing downward, and more than just the 'straight lines' I need the 'orthographic' projection where all squares are same size regardless of distance from camera.

My concern is that after fisheye correction, the squares toward the center will be 'larger' than at the side, even if the lines are vertically and horizontally parallel. eg, if fisheye 'fixes' this image, will all the squares be same size, or will just lines be parallel?

for example the following would not be acceptible, even if all the lines are parallel:

After calibration, I end up with this:

In the image above, I copy-pasted the closest black square and pasted it along each row so you can see how the squares get smaller as they get farther away. I need to know how to correct that.

Answer 1

Allright well, @TimeWescott was essentially correct about what I had to do, though the theory took some investigation and thought before I realized what the parts to the puzzle are, and the inherent limitation.

The key point is that you CANNOT map a >180 FOV image to a flat texture, of course! This is because after 180, you cross the horizon and are now mapping the 'ceiling'.

Also my first attempt at calibration with printed board was not as accurate as I needed. I re calibrated using my 27 in monitor as the checkerboard and moving the camera around instead.

The reality is that OpenCV does do the right thing. Essentially it will just crop at some FOV like 90 but it will give an image that is not only 'straight lines' but lines 'equally spaced'. It will stretch the out to be the same size.

So with this code I get the map I need:

DIM=(1232,1232)
K=np.array([
  [418.06635, 0.0,     621.82551], 
  [0.0,     418.06635, 610.67209], 
  [0.0,       0.0,       1.0    ]])
D=np.array([[-0.02120], [-0.00141], [-0.00050], [0.00000]])

map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, np.eye(3), K, DIM, cv2.CV_16SC2)
img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

So the most one could 'hope' for as far as mapping would be FOV 179 because FOV180 would map to an 'infinitely large image'. So OpenCV just chops it off at whatever FOV allows a full image to show, without adding 'black pixels' to handle stretch. So basically whatever is inside a circle representing about about FOV 90 gets unwarped and undistrorted and mapped to flat 'floor texture' as I need.

You could theoretically do some kind of separate unwarp of the top half to get a ceiling texture with a big empty hole in the middle, but bottom line, you cannot map more than 179 FOV to a ground texture and reality is closer to 90 fov with quite a bit of blurriness toward the edges already (meaning it's not worth playing around with warping settings since that's already the largest 'useful' floor texture you'll get)