# Skewed rectified aerial images

I have a series of aerial images from which I would like to get disparity maps for every consequent pair. In order to do that I do a stereo rectification.

The problem is that sometimes rectified images are tilted at a large angle. For example:

My questions are:

1. What is the reason behind this?
2. What can I do in order to get better results?

Here is what I expect my rectified images to be:

What I tried:

1. Using different feature detection methods of , like ORB, SIFT, SURF.
SURF is giving the best results of them, but still some images get skewed.

2. Played with different models' parameters.
Some tweaking had positive effect, like decreasing hessianThreshold of SURF, but nothing really provided 100% of good results.

3. Implemented an algorithm that would select keypoints scattered over the original image. This mostly had a good effect. Thire was a significant improvement for ORB and a small one for SIFT and SURF. But results are still not ideal. I asked a separate question about the algorithm on Code Review Stack Exchange: Efficiently selecting spatially distributed weighted points

Edit:
I asked a related question on SO: How to prevent rectified images to be cropped?

• Not sure if this is relevant - I'm just familiar with mono camera calibration, but openCV's getOptimalNewCameraMatrix gives you the ROI of the image, which you can then use to crop your image – Florent May 17 '18 at 5:46
• @Florent Thanks for your suggestion. Unfortunately, cropping the images is not what I am looking for. – Georgy May 17 '18 at 8:43