I am looking for a framework for extracting hand region from the video feed from the webcam(analogous to Viola Jones HAAR detection of face). This region of interest will serve as input for a CNN for hand gesture recognition, which I have already coded using python and CAFFE. There are way too many frameworks and I am confused on which one to use, priority would be to minimise the processing time. I already tried HAAR cascades for hand detection but they are pretty unreliable. It would be most helpful if implementation is readily available in python.

Thanks in advance

  • $\begingroup$ Can you please try and make the question a little bit more specific? Are you after the "centroid" of the hand? Are you after a full model of the hand including digits, etc? What are your NN inputs? In the meantime, please note that as far as I am aware, the most accurate hand recognition I have ever experienced is Leap Motion's. The device uses stereo vision and fits a rather complex model of a human hand to the video feed with unbelievable accuracy. $\endgroup$ – A_A Mar 2 '17 at 13:48
  • $\begingroup$ @A_A I am not allowed to use stereo vision. In a way I am after centroid(analogous to viola-jones HAAR cascade detection of face), I am hoping to detect the hand region as my CNN model predicts the gesture accurately when the input is hand region alone $\endgroup$ – Amal Vincent Mar 2 '17 at 13:57
  • $\begingroup$ That's fine, thank you for clarifying. Can I please ask you to define a bit better the "hand region"? You are "lucky" with the face because you only have to detect it in one projection. I can hold a palm up to a web cam and it looks like a paddle, or I can turn it to its side and it looks like a stick. Similar centroids possibly (especially if you work on the thresholded image) but completely different hand orientations. How do we go about that? Are you interested in a "paddle" moving left right or telling between rock, paper, scissors from any angle? $\endgroup$ – A_A Mar 2 '17 at 14:19
  • $\begingroup$ @A_A you read my mind kind of like rock paper scissors, and I need to detect both hands simultaneously as a matter of fact and detect a gesture in which both hands are touching $\endgroup$ – Amal Vincent Mar 2 '17 at 14:42

The requirement for being able to detect the "profile" of hand gestures is not a straightforward one to satisfy. There are issues of form and rotation / scale independency which are not "easy".

There are two main problems here:

  1. Segmenting the hands
  2. Figuring out the gesture.

To segment the hands, let's assume that the hands are imaged over a constant single colour background. This means that the overall hand region can be isolated with a "simple" threshold.

The second problem is one of recognising form. In other words, what are the "shape" differences between "rock", "paper" and "scissors"?

Therefore, whatever is used should be "able" to discriminate between "holding two fingers extended" versus "all fingers retracted in the shape of a fist", etc.

The "easy" way out is to use template matching. Very "simply", take pictures of the hand gestures for each hand and for various orientations and use them to "search" for their existence in an image using cross correlation (or other similar concept in some feature space).

This however, done in the "simple" way is very costly (computationaly) and also scale dependent. So, if the hands are near the camera they are larger, if they are further away they are smaller and therefore very difficult to "catch" from static images that were acquired at some standard distance.

So, an alternative would be to use Morphological Operators and specifically Skeletonisation.

The main idea here is to apply skeletonisation on the extracted hand regions, quite a few times to end up with a set of "lines" describing the shape you are trying to recognise. In this way, in one pass you effect the skeletonisation and in another pass you collect the (very few, compared to the original) pixel locations that compose the gesture. To then figure out the gesture itself, you work with features extracted from the pixel locations.

The added benefit of this type of a solution is scale invariance. The relative positions of the "finger segments" of the "scissors" gesture are the same whether you hold it near the camera or far away from the camera (Two lines forming a V ending up in a big blob). With careful selection of the features and search process you can even use the same algorithm and training dataset for the left and right hands separately (for example, using graph search).

Hope this helps.

  • $\begingroup$ cool response, I thought of the colour matching myself, I could use HAAR to detect the face and pick colour from the face, however I wanted know if there is more straightforward implementation already available, since CNN for gesture detection is the central theme of my work, I did not want spend too much time on the hand detection $\endgroup$ – Amal Vincent Mar 2 '17 at 17:39
  • $\begingroup$ @AmalVincent Thanks for your comment. If you found the response helpful you can upvote or accept it through the controls on the left. You might want to see this link on slightly more involved methods to adapt to segment "skin colour". This is always assuming that your hand images are imaged against a constant colour background. $\endgroup$ – A_A Mar 2 '17 at 18:25
  • $\begingroup$ I already upvoted, but since I am new here I would need 15 reps before it takes effect, I will accept the answer in a couple of days, if I do not receive a better reply $\endgroup$ – Amal Vincent Mar 3 '17 at 3:00
  • $\begingroup$ @AmalVincent Thanks for letting me know, I tend to remind "new" users who might not yet have got into the way the system works. Good luck. $\endgroup$ – A_A Mar 3 '17 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.