Document image vanishing point detection and evaluation

Question

It is a natural idea to substitute our scanner with mobile phone, and perform OCR with the phone at the same time. If this can come true, two problems must be tackled: one is related to document image binarization, and the other is related to geometric distortion correction. Among all the distortions, projective distortion is most common, and in order to correct this kind of distortion usually the horizontal vanishing point and vertical vanishing point must be identified. There are many different ways to obtain the vanishing points, and different methods can lead to different vanishing points. Then my question is given several candidate vanishing points (horizontal vanishing points, for example) how can we select the best one? Of course we can get the projective transform model based on each vanishing point, and then correct the distorted image. Visual evaluation the corrected image can tell which vanishing point is better. But could I find the best vanishing point just before the correction? Thanks!

Answer 1

Maybe I'll carry the discussion to another level, but I disagree with the fact that you need to explicitly correct for perspective distortions or even binarize the image. Many state-of-the-art techniques, which are based on deep learning, are rather successful in performing correction-free OCR. These are generally termed "end-to-end" OCR. They already outperform their processing-based counterparts with a significant margin. A quick google search yields:

• https://www.robots.ox.ac.uk/~vgg/publications/2016/Jaderberg16/jaderberg16.pdf
• http://www.whuang.org/papers/phe2016_aaai.pdf
• https://crypto.stanford.edu/~dwu4/papers/HonorThesis.pdf
• https://arxiv.org/abs/1603.03101
• https://arxiv.org/abs/1511.06841
• https://arxiv.org/pdf/1604.03286.pdf

Answer 2

why use OCR techniques that are sensitive to vanishing points?

projective distortion just skews the letters- right? So, why not just model those expected skews into the OCR templates for each letter? Also, the nearby letters would skew similarly, so you could probably use correlated parallel lines among neighboring letters to estimate the amount of local/global skew (and avoid dealing with VPs).

also, I assume you mean the phone will take a pic of the document, so don't you also have to worry about the paper not being flat and having curve linear distortions too, which seems to me would completely confound, and supersede any vanishing point issue you are worried about.