Improving Canny Edge Detection and Contours Image Segmentation

Question

I am working on an image segmentation project to identify playing cards on varying backgrounds. For my use case I care most that I accurately extract all the cards in the image; false-positives are inconvenient but not fatal. I also care that I can run the segmentation without user interaction (i.e., an algorithm like GrabCut does not work for me).

So far I have the following simple algorithm based on Canny edge detection and contour selection:

# Pre-processing: Convert frame to standard size, 1024x768
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
gray = cv2.medianBlur(gray, 9)
edges = cv2.Canny(gray, 10, 25)
_, contours, _ = cv2.findContours(edges, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
for c in contours:
    rect = cv2.minAreaRect(c)
    box = cv2.boxPoints(rect)
    area = cv2.contourArea(box)
    ratio = area / size
    if ratio < 0.015: # Any contour large enough is a candidate
        continue
    # Mark this box as possible card

For some (many) inputs it works well enough, or even very well:

However for other inputs the contour detection does not successfully disambiguate cards. The contours are grouped in ways that link cards or result in undesirable bounding boxes:

For the record, here are some other things I have considered:

• Using a bilateral filter instead of median blur. Might have improved results a bit, e.g., in the bad example above the first column third row card is now correctly segmented.
• Using histogram equalization to enhance contrast. Generally obscured some necessary edges.

• Attempting to automatically determine Canny thresholds for each image based on the median. Generally this approach missed many of the noisy card edges that we need to consider.

  def auto_canny(image, sigma=0.5):
      v = np.median(image)
      lower = int(max(0, (1.0 - sigma) * v))
      upper = int(min(255, (1.0 + sigma) * v))
      return cv2.Canny(image, lower, upper)

• Attempting to use watershed segmentation. The fundamental problem is automatically determining the initial segmentation markers, which turns out to be roughly equivalent to identifying the cards in the first place. Finding "white" is particularly hard because its color varies with the ambient lighting conditions.

• Attempting to use mean shift filtering / segmentation. I expected it might denoise the edge detection process by simplifying the image structure but it seemed to have more of a "blurring" effect, especially around reflections.

For reference (or, better yet, if you want to play along at home) I have a set of example inputs and the outputs of my current algorithm: album.

Answer 1

You might try a Hough Transform after your canny-Detection instead of contour detection. (Given that you dont have too much perspective distortion, i.e. the cards are rectangular and not trapezoidal).

Have a look at this related question and the links therein.