# Sum of Squared Intensity Difference for block matching with ORB features, c++

I am trying to implement feature matching algorithm between image reference and image current.

The pipeline is as follows:

1. Gray scale Images (intensities = [0-255]) are imported.

2. Extract ORB keypoints in both reference and current images:

Ptr<FeatureDetector> detector = ORB::create();

1. Obtain blocks surrounding all keyPoints within specified window_size in both reference and current images. NOTE: If the blocks exceed the image upper or lower bounds, the block intensities would obviously be zero (Example: KP_A in image reference).

4.Instead of using existing feature matching algorithms in opencv, I am trying to utilize sum of squared of intensity differences (SSD) in the blocks acquired across ORB keypoints in reference and current images.

1. Extracting correct features demands implementing crossCheckedMatching() to ensure features are chosen correctly.

2. Visualize matched features.

Complete code: here.

Problem 1: My final result looks something like this:

Whereas, I have read in few lecture notes and articles, SSD and block matching incorporated with cross checked matching method should provide decent matches. Does these matched features look any good in someone's eyes or awful after cross check matching procedure?

Problem 2: I do not know how to find the correct threshold for evaluating SSD values between two images (Empirical tuning vs. Existing logical approaches?).

Final goal, adapted from the textbook:

1. use these matched features in RANSAC to filter outliers.

2. Find Homography (Theory says, 4 pairs of matched points needed!)

3. Camera Pose Estimation

4. 3D reconstruction of environment

Apologies for such a long question, but I tried to squeeze it as much as I could.