Steps for recognizing hand written signs

Question

What are the steps to built a system which recognizes hand written signs in an image and returns the most similar sign in Python? I know that depends on the image and many variables, but isn't there a sequence of steps, a main path to follow for recognizing images with at most 2 or 3 signs per image?

I thought the steps should be:

1. Take an image and convert to binary

2. Apply find_contour to binary image

3. Extract scale- and rotation-invariant properties

4. Create a feature vector with scale/rotation-invariant properties (if there is much more than 1 sign, like for example a star and a square not overlapped, is that a problem?)

5. Repeat steps 1 to 4 for every image

6. Put them into a k-NN or SVM

These next steps, I do not know how to implement:

1. I choose an image

2. System spits out the result (most similar image)

Answer 1

That's what you would do with images, not so much with handwriting, which needs more preprocessing and robust features because people don't write exactly the same way twice. See for example:

• Improving Offline Handwritten Text Recognition with Hybrid HMM/ANN Models
• Adaptive Membership Functions for Handwritten Character Recognition by Voronoi-Based Image Zoning
• A visual approach to sketched symbol recognition
• Feature extraction and classifier combination for image-based sketch recognition
• Sketched symbol recognition with auto-completion
• HBF49 feature set: A first unified baseline for online symbol recognition
• Neural network-based symbol recognition using a few labeled samples

Answer 2

In addition to Emre's suggestions, I'll add my favorite textbook on the subject:

Character Recognition Systems by Cheriet, et al.

http://www.goodreads.com/author/show/1112467.Mohamed_Cheriet

An earlier but still useful work is that of Bunke, but used copies can be very expensive (check www.addall.com) so you might want to find one in an engineering library. http://textbooks.findthelisting.com/l/2266645/Handbook-of-character-recognition-and-document-image-analysis-981022270X

Handwriting recognition is an extraordinarily difficult problem in the general case. It's necessary to distinguish between analysis of handwriting on the fly ("online") and analysis/recognition of handwriting on paper or in a scanned image ("offline").

Try to write the most detailed specification you can about what it is you want to achieve, including the following:

• script & language (e.g. English, Hindi, traditional Chinese, Hangul,...)
• image acquisition method (e.g. scanned paper, smart phone picture,...)
• size of dictionary of symbols to be matched (10? 500? 7000?)
• OCR character database used to measure performance
• accuracy
• means to train new symbols - can the user do it?
• target platform (phone? desktop? custom device? multiple devices?)
• amount of time you want to spend on development

This problem has tortured researchers for decades, and although great progress has been made on many fronts it's exceedingly difficult to do well.

For OCR problems in particular I recommend spending a few days trying to design the best algorithm you can based only on what you already know. Implement the algorithm, test it, and struggle with it. Find out what doesn't work. Rewrite your specification, and then dig up a good textbook on OCR and read it cover to cover.

Also, read this book to learn more about how humans can read characters and recognize symbols:

Reading in the Brain by Stanislas Dehaene. http://www.unicog.org/biblio/Author/DEHAENE-S.html

If you can define a very narrow scope for your recognition algorithm, set modest goals for accuracy, read up on standard methods, and then give the problem at least a few months then I'm sure you'll learn quite a bit.