It seemed to work relatively well where there was a strong lightness difference at the edge of the poster, compared to background.
This matches what
squares.cpp seems to do: scale the image, do a canny edge detection, simplify the resulting contours and look for squares.
There's two things that, looking at your example images, go wrong here:
- The contour detection / simplification doesn't seem to deal well with reflections crossing the edge (see Cpt America poster in last photo, and the batman vs superman poster in the 4th image)
- The canny edge detector will fail (and so does my brain) on posters when the frame/background difference isn't clear. I had to look twice myself at the second picture to make sure where the chrome frame ends and where the poster begins
So, this points to it being helpful to pre-process your images in different way, try to detect the rectangle, change your preprocessing, repeat, and so on.
A few observations:
- maybe you can just reduce the aggressiveness of the contour simplification
- Try different edge detectors! Canny is not always the best choice.
- the weaker reflections just seem to be relatively white light – maybe convert your image to a brightness/hue/saturation colorspace, and reduce the variance of the brightness channel
As a side note: this approach (pyramid scaling, edge detection, contour extraction, contour simplification, rectangle detection) sounds more complex than just a normal Feature extraction by means of a transform of the image. For example, there are several transforms in OpenCV, some of which are very usable to detect dominant straight lines in the picture. If you can find parallelograms in those, you should be one step ahead, especially if you can really assume the poster is the biggest rectangle.
I think the Hough transform is quite exactly what you need.