8
$\begingroup$

I am currently working on a gesture recognition system (for an Android Application). I think that I have completed the Image processing stage, where I am able to extract the contour of the hand (I am wearing a glove to avoid background subtraction for now).

I am also getting the bounding ellipse/rectangle, the centroid as 'important' shape features of the hand.

My problem is that not a lot of literature indicate what the next stage is before the classification of the image through the hidden markov model. I am finding this 'Feature Extraction' stage very ambiguous.

Currently I am getting a list of angles of the contour (which is approximated so as to obtain a limited number of angles)

The problem is that I am clueless as what to do next. When I search for literature as 'Shape classification using HMMs' I still find it hard to what should I do next.

As a tool for HMM I plan to use JaHMM. But I don't know how to experiment with the tool at this stage since I do not know what will be the input to this classification stage!

This shows the information I can obtain from my hand

I have encountered a list of functions I found in some literature, such as Fast Fourier Descriptors, Curvature Descriptors, B Spline; yet I am clueless as to how to apply these functions to my current data (i.e. list of angles, e.g.: -63, 154, 3, 23, 54, ....)

UPDATE 1:

Thank you for your information. @Peter K.

Regarding the poses: I was going to go for a set of words in American Sign Language which are fingerspelled, e.g 'dog' -> 'd' 'o' 'g' (3 states); but the moment I haven't decided what to go for.

I am currently reading some more papers to see what type of information I should extract, such as :

  • centroid of hand
  • angle of motion
  • distance from a particular point to different points of the hand contour (etc..)

Now I have encountered a paper which seems to show what I want to do, I'm not sure:

http://espace.library.uq.edu.au/eserv.php?pid=UQ:10700&dsID=n0273.pdf

I am currently reading section 5 - Vector Quantization (I heard of this term before but do not actually know what it involves, and in figure 5 there seems to be a complex algorithm which, if I understood correcty, converts the set of values I achieve from the hand (just mentioned) into 1 digit which I can use as the Observation sequence to train an HMM for 1 particular sign. Do you think I'm moving on the right track? (I'm working on Android, (NDK), I found JaHMM as an HMM tool, and using OpenCV for image processing.

@Peter K. Thank you for your answer. Regarding the data generation I was planning to follow the steps of this paper, after I produce my personal dataset: (section 4 and 5) http://www.i.ci.ritsumei.ac.jp/~shimada/papers/vi02-tanibata.pdf

UPDATE 2: keeping in mind that a gesture is formed of {posture a, posture b, posture c}

I am now thinking that I must make use of some form of classification algorithm. That is, I currently have a set of feature vectors:

Posture A: [angle of ellipse surrounding it, height:width ratio]

  • 0.802985 33.909615
  • 0.722824 31.209663
  • 0.734535 30.206722
  • 0.68397 31.838253
  • 0.713706 34.29641
  • 0.688798 30.603661
  • 0.721395 34.880161

Posture B: [structured the same as posture A]

  • 0.474164 16.077467
    • 0.483104 14.526289
    • 0.478904 14.800572
    • 0.483134 14.523611
    • 0.480608 14.41159
    • 0.481552 15.563665
    • 0.497951 15.563585

etc..

and I would like that when I feed a feature vector I obtain a simple symbol, e.g. 'A', 'B', etc.

Is this possible? I also migrated the question here: https://stackoverflow.com/questions/15602963/vector-quantization-algorithms-used-to-provide-observation-sequences-for-hidden

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ You need to define what you mean by "gesture". Do you mean the pose of the hand? Or the motion of the hand? Or the motion of the fingers of the hand? There are lots of ways to skin this cat... $\endgroup$
    – Peter K. ♦
    Mar 23, 2013 at 1:03
  • $\begingroup$ In this case, I am aiming for both movement and postures, basic symbols of the american sign langauge starting from finger spelling of letters to basic (at the point in time: one handed) gestures $\endgroup$
    – test
    Mar 23, 2013 at 10:57
  • 1
    $\begingroup$ Will have a read and see what I can come up with. Might take a while to respond. Watch this space. $\endgroup$
    – Peter K. ♦
    Mar 23, 2013 at 17:22
  • $\begingroup$ Hello I am trying to do hand gesture recognition with Android Open CV , I am a beginner with both tools , any help with steps and ideas from your experience. Thank you so much. $\endgroup$
    – user2253346
    Apr 5, 2014 at 21:12
  • $\begingroup$ Hello if I were you I would start by following the tutorials on the android open cv page. Prior to that the first step would be to make sure you have your native development set up approriately so that you can execute C code in your android application (if you will be using C instead of java for more efficient results!) $\endgroup$
    – test
    Apr 6, 2014 at 13:08

2 Answers 2

Reset to default
5
$\begingroup$

I have used HMM for gesture recognition (not pose recognition). What I did was: tracking the hand and recognize the gesture the hand was drawing in the air, you can image it as a trail.

You can use HMM as sequence recognizer, so first of all you need to transform your image into a discrete number sequence.

For each gesture you want to recognize, you need to train an HMM for that.

So you have a dictionary with some known word. Each one is a trained HMM. If you have a new word (unknown obdervation) you can compute the probability for each word of the dictionary to be likely the unknown one.

Some pseudo-code:

##################### training phase

examples = [112233, 11233, 123, 1122223333]
word1 = train(example)

examples = [222333, 22222223333, 2222333333]
word2 = train(example)

examples = [124555, 1122445, 1111111222224444555]
word3 = train(example)

dictionary = [word1, word2, word3]


##################### recognition phase

#let's say you have a new unkown word: 12245, you want to know what word of the dictionary it is more likly

unkown = 12245
probabilities = []
for w in word:
    probabilities.append( calculate_prob(unkown) )

Now, see what the is the maximum value in probabilities and you get the most likely word of the dictionary!

See here:

http://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf

http://www.codeproject.com/Articles/69647/Hidden-Markov-Models-in-C

http://www.creativedistraction.com/demos/gesture-recognition-kinect-with-hidden-markov-models-hmms/

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you for such a great response. I did understand everything you said. My current problem is how I will obtain the observations i.e. [112223333,222333, etc...] i.e. How can I convert my current data (e.g. length and width of hand + motion at the same time + angle of motion, etc,) to these types of numbers? From my edit I mentioned that I will probably be looking at clustering and K Means to obtain a 'code' vector. What's your feedback? Thank you very much again! $\endgroup$
    – test
    Mar 23, 2013 at 18:52
  • 1
    $\begingroup$ yes this is a problem. i used kmeans but it didn't fit in my problem so i simply used centroids and "discretized" the gesture with eculidean distance from centroids.. for sure you can make it smart ad add more data like velocity and kmeans could fit but i don't know sorry.. make some trials! sometimes it happened in machinelarning that some empirical solutions works better for some particular data.. try different solutions! $\endgroup$
    – nkint
    Mar 24, 2013 at 14:33
  • $\begingroup$ (if you need some example how to use kmeans with opencv just ask) $\endgroup$
    – nkint
    Mar 24, 2013 at 14:34
  • $\begingroup$ probabilly not a good solution but you can try to "skeletonize" the hand.. en.wikipedia.org/wiki/Morphological_skeleton and here some code: felix.abecassis.me/2011/09/opencv-morphological-skeleton $\endgroup$
    – nkint
    Mar 24, 2013 at 14:35
  • $\begingroup$ if you resolve it just tell me how did you manage to do it, it is a nice task : ) $\endgroup$
    – nkint
    Mar 24, 2013 at 14:36
3
$\begingroup$

Let's start with pose recognition. This paper traces the boundary of the hand, and counts the number of finger tip detections from that boundary. One thing to note in that paper is that there is no "state" information required. For pose / position estimation, HMMs are probably not a good fit.

The gesture information fits better into the HMM gamut for problem-solving. However, I'd need to see a bit more of the sort of data you are going to use for gestures. Can you explain a bit more about the algorithm that generates the data you have?

The problem is that selecting the right structure of the hidden Markov model has quite a bit of bearing on the achievable accuracy... Warning: PDF link!

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I've created an update for the question! Thanks a lot for your information $\endgroup$
    – test
    Mar 23, 2013 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.