# Machine Vision Techniques for Sign Language Recognition and Interpretation(using Matlab)

I'm currently working on applying Machine Vision methods to sign language interpretation and I'm really overwhelmed with the body of knowledge out there. I've been reading up on object recognition from various resources (Peter Corke's Robotics, Vision and Control, Richard Szeliski's Computer Vision, Algorithms and Application), but finding it hard to decide on what methods to go with, hence my question: For simplicity's sake, let's assume I just want to process an image of a hand holding up 1 finger, get my program to process the image and out "one". Do same for 2...up till five. (I know this is really simple, but will give me the direction I need before going on to more complex things). (1)What method do I need to store the original image of the hand holding up 1 finger to put in my database, which other images will be compared to? (2)What method do I to compare an inptu to what's stored in my database (Please, give me the most simple methods you know of).

I am afraid that the question is trying to probe a very wide range of issues to receive a truly meaningful answer.

In general, this would be a classification problem. As such, you would need to decide on a number of features which would reflect the key changing characteristics of your images and a classifier which, after being trained, would be able to discriminate between your different gestures. For more information please see: http://en.wikipedia.org/wiki/Statistical_classification

The issues of how many training images you need per gesture and which features and classifier, all depend on your environment and specifics of your application.

The simplest thing you can do is template matching, where you would simply compare an acquired image of a gesture with stored images of gestures. For more information please see: http://en.wikipedia.org/wiki/Template_matching

But of course, this approach is a tiny step forward. What about changing backgrounds, different pairs of hands, scaling of the objects because of different distances to the camera, slight rotations, occlusions... ? And then, once the static recognition is done, you will have to consider the version that works on uninterrupted video.

There are things you can do to deal with these issues but for robust recognition, it is likely that you will need to move to 3D data representing the movement of hands in space. (For example: http://www.sciencedirect.com/science/article/pii/S1077314210001748 ).

You can acquire 3D data in a number of ways, e.g. virtual reality gloves (please see: http://dx.doi.org/10.1109/IJCNN.1999.832699), devices such as kinect (please see: http://dx.doi.org/10.1109/ICSESS.2012.6269439) and plain simple accelerometers (please see: http://dx.doi.org/10.1109/JSEN.2011.2166953).

The simplest method to start with 3D data is the one based on accelerometers. In this case you would be working with time-series data (e.g. timestamp, acceleration vector).

Happy to amend the response if you provide more specific data about your application.