Detecting Individual Photos on a Scanned Page

Question

I'm working with a collection of old photographs. They were scanned on brown paper in various configurations. Some scans have a single photo, others have three or four. Some of the photos are mounted on white cards, others are not:

I developed an algorithm to find the individual photographs on each scan, which is described here. The gist is to:

1. Find the median color (independently for each RGB channel, 0-255)
2. Blur the image.
3. Binarize according to whether each pixel's RGB values have an RMSE of <20 from the median.

This leaves a mostly-black image with a few large white rectangles on it, like so:

To find the rectangle coordinates, I use the following procedure:

1. Pick a random white pixel, (x, y) (statistically, this is likely to be in a photo)
2. Call this a 1×1 rectangle.
3. Extend the rectangle out in all directions, so long as you keep adding new white pixels.
4. If this rectangle is larger than 100×100, record it as a photo. (The originals are much larger than the photos I've uploaded for this question.)
5. Color the rectangle black.
6. If <90% of the image is black, go back to step 1.

This has worked on most (90+%) of the scans that I've tested it on, but it is quite ad-hoc, particularly the RMSE 20 threshold in the first portion and the entire procedure to find the rectangle coordinates.

Is there a more principled way to find these photo rectangles? How would someone with experience in image processing solve this problem?

Answer 1

Most of the image processing/computer vision algorithms are ad hoc. Your algorithm is good. However, I think that a few points can be improved:

1. Instead of finding median color, you could do use the geometry of your images, which is more robust. For example, you know that the background part (The one that touches the boundaries is black, and the one inside it has the color you are looking for.

2. In order to find the rectangle coordinates, you should use a blob analysis procedure. In Matlab it is done by using regionprops command. It will give you properties like

   • area of blob (To remove the small ones)
   • solidity (Can be used to separate the rectangles from non-rectangles)
   • coordinates of pixels

Also, you could try to use morphological pre-processing like opening, to remove small blobs.

Answer 2

I have approached a similar problem using line Hough transform. The a rectangle can be found by searching four local maxima separated by 90°.

Once a rectangle is computed from line intersections, it can be further tested by comparing color/texture at the corners (within image and outside the image).

The detection can be more restrictive if you know aspect ratio of the photos, background color or other information beforehand.

Finally a rule removing intersecting rectangles or rectangles within rectangles is convenient.

Filtering prior to detection as you did enhances the repeatability. Try looking for a segmentation method that will separate foreground and background. I can think of Watershed transform that would work if the background is not too noisy.