Affine Transformation for matching 2D point sets

Question

I have two sets of 18 corresponding 2D points. These coordinates represent the main joint points of a human body, describing the "human pose".

I want to decide if these two poses are similar/have the same shape. In order to do so, I find the affine transformation (matrix) that transforms the second pose to the first pose (using least-squares). This gives me the image of the second pose pictured on the first one.

Now, my question is, what is the best way to conclude if the image of the second pose is similar to the first one? I'm thinking about using a distance metric like mean square or SAD, but I'm not sure. + What about normalizing/standardization?

Answer 1

There are two ways by which you can approach this problem:

1. Match the point clouds directly with RANSAC.

2. Consider matching the resulting graphs (closely related to 1, still)

What you have done already is closest to #1. To complete it, you need to repeatedly obtain subsamples, find the transformation and then check its "error". The "error" is how many projected points are within a distance threshold (which you are going to have to estimate). For more information, please see this thesis. It does a great job at defining the problem in the beginning (which is very important) and then talking about the approaches.

But, since you are already dealing with pose estimation, you can incorporate its constraints into the model and go for estimation via graph matching. In this case, you would be looking at extracting a graph from your images and then compare it with the pose graph, OR, compare the extent to which two graphs coincide (the graph obtained from the left image to the graph obtained from the right image). A key part of this is the derivation of image skeletons. Gonzalez' book has a very good section on skeletons (chapter 11) and their applications.

The existence of a graph imposes certain constraints on the relationships between the points. For example, the hand "leafs" are expected to be no more than 5-7 "hops" from the root node (e.g. the head node) and this can improve the detection rate. For more information please see this paper. Or more generally look for "graph based pose estimation" and "graph recognition". Be aware that "graph matching" as a term is associated with something entirely different in mathematics.

Hope this helps.