# Slicing and Recognition of book title and author from bookshelf image

I am trying, for my own learning purposes, to develop an implementation of an algorithm that would list books, given an image of a bookshelf like the following:

The first step is to slice the image into individual books.

My algorithm, in Mathematica, is:

    img = ColorConvert[Import["http://i.stack.imgur.com/IaLQk.jpg"], "GrayScale"]

• do a basic edge detection &
• remove the text and try to keep the long lines

edge = DeleteSmallComponents[EdgeDetect[img, 3],Last[ImageDimensions[img]]/5]

• then remove unwanted horizontal lines

lines = Sort[ImageLines[img5] /. {{0., _}, {_, _}} -> Sequence[]]
Show[img, Graphics[{Thick, Orange, Line /@ lines}]]


The results, though, are less than great:

My questions are:

1. How can I improve this to yield better results?
2. Is there a smarter way to do this?
3. how further should I process the images to increase accuracy at the (later) OCR phase?
4. How to use the color information to improve the segmentation?
• @OrenPinsky I don't see the problem with the segmentation: the number of "false book areas" is not high (I can see only one in the sample you provided), and if you plan to do some kind of text recognition as the next step, that should be enough to discriminate between books (area has text) and not-books (no text) Nov 15, 2012 at 16:02
• @OrenPinsky, I would say your results are pretty great. :)
– Dima
Nov 15, 2012 at 22:15

Here is the link to a research paper that tries to do the same thing as you wanted. It might help you.using image features Also a cool video on the youtube

• Unfortunately, the first link is broken. Nov 12, 2015 at 14:42

Which method are you using to detect the lines? Have you tried experimenting with LSD?

Here are the results of a quick test I did using LSD:

In this first image I have displayed only the vertical line segments with an angle between 75 and 105 degrees and the length greater than $0.1 * height$ of the image:

The second image are the results with the same angle constraint but disregarding the lengths of the segments:

You can try playing with this a bit, figure out how to chose the best line segments, extend them to lines and maybe get slightly better results than the ones you posted.

• "Have you tried experimenting with LSD?" Nice try, FBI ;) Nov 18, 2012 at 20:51
• Mathematica's ImageLine is based on the Hough transform, and I am now convinced (from the feedback here, mostly) that it works pretty well. It bothers me, though, that I am loosing relevant data when I transform into grayscale, and that in this application color data could be (intuitively) help a robust edge detector.Will try LSD and see how it goes! (it worked amazingly well for Steve Jobs! ;-) Nov 19, 2012 at 2:16
• I've seen a friend use LSD for door detection, I think he was pleased with the results in the end. I'd say it's worth a shot :D Nov 19, 2012 at 9:55

You can try doing edge detection on individual color domains and then merge them, using your method of choice for edge detection.

Compared to edge detection directly on the color image, it might produce better results.

The paper from the broken link provided by isrish might be found Combining image and text features: a hybrid approach to mobile book spine recognition , Proc. 19th ACM international conference on Multimedia, 2011. One can also check out other papers from David Chen et al., for example Low-Cost Asset Tracking using Location-Aware Camera Phones , Proc. SPIE 2010.