1
$\begingroup$

My goal is to warp a top-down fisheye view to end up with a flat square 'texture' that could be applied to a flat plane and would represent the 'real' floor eg a 1024x1024 texture would map to a 4 meter by 4 meter of floor in the room (ignore the other 'items'). Or even 2 meter by 2 meter around the center.

In other words I just want to flatten the floor in the image and project it 'orthographically' so that each one of those checkerboard squares is exactly the same size. If the whole floor was covered in checkers, with no other items, my output would be flat checkerboard crop of 4 meters by 4 meters.

camera is imx 219 200 FOV about 1.3 meters off the ground pointing straight down. (well if it were pointing exactly straight down the red down would be on the green dot)

I am looking for python opencv implementation, but if someone at least knows what it is that I need to do/measure to achieve it, it will be super helpful.

I have not seen any other examples of fisheye pointing downward, and more than just the 'straight lines' I need the 'orthographic' projection where all squares are same size regardless of distance from camera.

enter image description here

My concern is that after fisheye correction, the squares toward the center will be 'larger' than at the side, even if the lines are vertically and horizontally parallel. eg, if fisheye 'fixes' this image, will all the squares be same size, or will just lines be parallel?

enter image description here

for example the following would not be acceptible, even if all the lines are parallel:

enter image description here

After calibration, I end up with this: enter image description here

In the image above, I copy-pasted the closest black square and pasted it along each row so you can see how the squares get smaller as they get farther away. I need to know how to correct that.

$\endgroup$
13
  • $\begingroup$ See the OpenCV page on lens calibration. $\endgroup$
    – TimWescott
    Nov 15, 2021 at 2:55
  • $\begingroup$ @TimWescott is regular camera calibration same as fisheye calibration? if so why does cv have fisheye calibration module? what kind of transform will make not only the lines straight but equally spaced? $\endgroup$ Nov 15, 2021 at 3:38
  • $\begingroup$ The OpenCV method should also work for a fisheye lens, but you may need more terms. That method works just fine for the 110-degree FOV cameras we're using, which is pretty close to what you're using. And yes -- for a surface that's parallel to the camera's focal plane, the whole point of getting $x$ and $y$ correct is that straight lines will be straight, etc. $\endgroup$
    – TimWescott
    Nov 15, 2021 at 4:06
  • $\begingroup$ @TimWescott i understand straight lines straight, but what i need is strait lines spaced out the same. ie the image to 'expand' so checkerboard not only has straight lines, but all of them are eg 30 pixels square, whether they are close to center or far at side. This is not a usual fisheye which retains 'converging lines'. it's orthographic, for 'mapping'. $\endgroup$ Nov 15, 2021 at 16:46
  • $\begingroup$ I guess I thought squares further away would be smaller, but maybe it's not true as long as plane is parallel to camera (camera pointing straight down) $\endgroup$ Nov 15, 2021 at 16:54

1 Answer 1

1
$\begingroup$

Allright well, @TimeWescott was essentially correct about what I had to do, though the theory took some investigation and thought before I realized what the parts to the puzzle are, and the inherent limitation.

The key point is that you CANNOT map a >180 FOV image to a flat texture, of course! This is because after 180, you cross the horizon and are now mapping the 'ceiling'.

Also my first attempt at calibration with printed board was not as accurate as I needed. I re calibrated using my 27 in monitor as the checkerboard and moving the camera around instead.

The reality is that OpenCV does do the right thing. Essentially it will just crop at some FOV like 90 but it will give an image that is not only 'straight lines' but lines 'equally spaced'. It will stretch the out to be the same size.

So with this code I get the map I need:

DIM=(1232,1232)
K=np.array([
  [418.06635, 0.0,     621.82551], 
  [0.0,     418.06635, 610.67209], 
  [0.0,       0.0,       1.0    ]])
D=np.array([[-0.02120], [-0.00141], [-0.00050], [0.00000]])

map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, np.eye(3), K, DIM, cv2.CV_16SC2)
img = cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

enter image description here

So the most one could 'hope' for as far as mapping would be FOV 179 because FOV180 would map to an 'infinitely large image'. So OpenCV just chops it off at whatever FOV allows a full image to show, without adding 'black pixels' to handle stretch. So basically whatever is inside a circle representing about about FOV 90 gets unwarped and undistrorted and mapped to flat 'floor texture' as I need.

You could theoretically do some kind of separate unwarp of the top half to get a ceiling texture with a big empty hole in the middle, but bottom line, you cannot map more than 179 FOV to a ground texture and reality is closer to 90 fov with quite a bit of blurriness toward the edges already (meaning it's not worth playing around with warping settings since that's already the largest 'useful' floor texture you'll get)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.