# Gesture recognition using only simple techniques

I'm following a computer vision course and I have this exercise: write a program that, given an hand image, it can recognize if the hand is open, closed, in a punch, or holding an "ok" posture, using only the techniques provided till now (pixel 4/8 connected, connected region, contour finding, holes finding, blob property like centroid, area, perimeter, eccentricity, image moments, image transformation like invert/power/log/gamma correction/contrast stretching, histogram computation and equalization).

I have done it with some basic blob properties(closed hand has a low eccentricity, "ok" has a hole, open hand has a big difference between the area of the inscribed ellipse in the blob and the blob area itself with a low eccentricity).. It seems to work but the first image is a little bit problematic.

I think there could be something more to make a more robust algorithm. Maybe some kind of moment property? Could some blob axis/orientation/extreme points help?

PS test images:

• Are you allowed to use skeletonization? See the relevant Wikipedia article for more ideas.
– Emre
Jan 9, 2013 at 23:03
• no is not allowed because it is the content of a future classes! Jan 11, 2013 at 21:46
• You could try applying Symbolic Aggregate approXimation to the outer contour of each object. Basically reducing a complex shape to a time-serie and then cluster these in some way. Jan 12, 2013 at 13:55
• i think i don't have to use any kind of machine learning, just a smart way to combine isses written in the question.. Jan 12, 2013 at 16:00
• maybe some morphological feature? Jan 12, 2013 at 16:45

You can try looking at Hu invariant moments. They can be constructed from basic moments, and are rotation, scale, reflection, and translation invariant.

Calculate them for a set of training contours first, and then apply them to the test contour.

There are implementations in Matlab and OpenCV, as far as I remember.

According to the book Programming computer vision with Python an interesting approach is to use dense SIFT (a.k.a. HoG) features on your images, and feed these features to a classifier.

I didn't try it myself, but it seems quite sound as an approach. Furthermore, the inventor of the HoG feature proposes the Flutter app that worked quite well in my tests, and it would be very weird if the inventor didn't use his own features or a close derivative.

One of my friends did this for his undergraduate thesis. What he basically did was encode properties of each gesture. For example,in the first figure, take a rectangular mask over theportions of the hand. The parts where the skin meets the rectangular mask edge should be noted and marked. Then the relative positions of the larger edge and the smaller edge can be compared.

So for pointing upwards, you will have a smaller edge upwards and larger edge at the wrist.

For pointing sideways, you have a smaller edge at one side and larger edge at the bottom.

The directions at the very least can be covered in this way.