Obtaining Correct (Least Squares Sense) Affine Transform Parameters Between Two Images

I have two images that I want to compute the affine motion model parameters. The model that I use is $$x' = a_1x+a_2y+a_3$$ $$y' = a_4x+a_5y+a_6$$ To calculate those 6 parameters, I picked 6 points (overdetermined system) between two images, and calculated those parameters by using MATLAB (A = X\X_primes). However, when I apply the transformation, even the points that I picked (X) do not translate into the specified locations (X_prime). I inferred that my least square solution should be wrong, and I am looking for how to make it better, i.e. what kind of points should I choose on the image to obtain better parameters for affine motion model?

• One issue is in the pixels you pick. Imagine they all come from a flat surface, they are not sufficient to do the job. Interest keypoints, like contours and edges, might be more efficient – Laurent Duval Mar 25 '18 at 14:44
• @LaurentDuval By trial and error I obtained a good result by picking some feature points (corners) on the image as you stated. Thanks for your comment:) – Canberk Mar 26 '18 at 8:55