13
$\begingroup$

I am trying, for my own learning purposes, to develop an implementation of an algorithm that would list books, given an image of a bookshelf like the following:

enter image description here

The first step is to slice the image into individual books.

My algorithm, in Mathematica, is:

    img = ColorConvert[Import["https://i.stack.imgur.com/IaLQk.jpg"], "GrayScale"]
  • do a basic edge detection &

  • remove the text and try to keep the long lines

    edge = DeleteSmallComponents[EdgeDetect[img, 3],Last[ImageDimensions[img]]/5]

  • then remove unwanted horizontal lines

    lines = Sort[ImageLines[img5] /. {{0., _}, {_, _}} -> Sequence[]]
Show[img, Graphics[{Thick, Orange, Line /@ lines}]]

The results, though, are less than great:

enter image description here

My questions are:

  1. How can I improve this to yield better results?
  2. Is there a smarter way to do this?
  3. how further should I process the images to increase accuracy at the (later) OCR phase?
  4. How to use the color information to improve the segmentation?
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ @OrenPinsky I don't see the problem with the segmentation: the number of "false book areas" is not high (I can see only one in the sample you provided), and if you plan to do some kind of text recognition as the next step, that should be enough to discriminate between books (area has text) and not-books (no text) $\endgroup$
    – penelope
    Nov 15, 2012 at 16:02
  • 3
    $\begingroup$ @OrenPinsky, I would say your results are pretty great. :) $\endgroup$
    – Dima
    Nov 15, 2012 at 22:15

5 Answers 5

Reset to default
7
$\begingroup$

Here is the link to a research paper that tries to do the same thing as you wanted. It might help you.using image features Also a cool video on the youtube

Improve this answer
$\endgroup$
1
  • $\begingroup$ Unfortunately, the first link is broken. $\endgroup$
    – Youngjae
    Nov 12, 2015 at 14:42
4
$\begingroup$

Which method are you using to detect the lines? Have you tried experimenting with LSD?

Here are the results of a quick test I did using LSD:

In this first image I have displayed only the vertical line segments with an angle between 75 and 105 degrees and the length greater than $0.1 * height$ of the image: pic1

The second image are the results with the same angle constraint but disregarding the lengths of the segments: pic2

You can try playing with this a bit, figure out how to chose the best line segments, extend them to lines and maybe get slightly better results than the ones you posted.

Improve this answer
$\endgroup$
3
  • 6
    $\begingroup$ "Have you tried experimenting with LSD?" Nice try, FBI ;) $\endgroup$
    – Lorem Ipsum
    Nov 18, 2012 at 20:51
  • $\begingroup$ Mathematica's ImageLine is based on the Hough transform, and I am now convinced (from the feedback here, mostly) that it works pretty well. It bothers me, though, that I am loosing relevant data when I transform into grayscale, and that in this application color data could be (intuitively) help a robust edge detector.Will try LSD and see how it goes! (it worked amazingly well for Steve Jobs! ;-) $\endgroup$
    – Oren Pinsky
    Nov 19, 2012 at 2:16
  • $\begingroup$ I've seen a friend use LSD for door detection, I think he was pleased with the results in the end. I'd say it's worth a shot :D $\endgroup$
    – penelope
    Nov 19, 2012 at 9:55
1
$\begingroup$

You can try doing edge detection on individual color domains and then merge them, using your method of choice for edge detection.

Compared to edge detection directly on the color image, it might produce better results.

Improve this answer
$\endgroup$
0
$\begingroup$

The paper from the broken link provided by isrish might be found Combining image and text features: a hybrid approach to mobile book spine recognition , Proc. 19th ACM international conference on Multimedia, 2011. One can also check out other papers from David Chen et al., for example Low-Cost Asset Tracking using Location-Aware Camera Phones , Proc. SPIE 2010.

Improve this answer
$\endgroup$
0
$\begingroup$

12 years after the original question, AI is here to solve everything:

https://simonwillison.net/2024/Feb/21/gemini-pro-video/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.