Object detection in binary image

Question

This task comes from tracking object on a steady background. So far I was able to remove the background and obtain binary masks like this: I need to get bounding rectangle of the toy without the cord attached to it (on some frames it's quite thick). Sounds easy enough but I can't think of a good way to do it.

I tried cross-correlation with template of the toy I cropped. It wasn't really nice, plus I needed rectangle to be able to change it's size. And the cord added some mess.

meanShift() also didn't show good results because of the cord and occasional noice.

For now I stopped on findContours() with some tweaks, but it still produces some weird boundaries on what seems like okay image.

I keep thinking there must be some more simple and effective solution.

Answer 1

First I would recommend filling in the contour of the toy - in case it looks like the one in the second image. You could do this by analyzing the hierarchy output from findContours: make white all regions having a parent or by using an iterative morphological operations (not directly implemented in OpenCV).

Once the toy is nice and fat (not just the edge), you can remove the cord. I would use here some erosion with a flat kernel (horizontal line).

See the code below (this is in Python, but you'll have a clear overview):

import cv2

lemBGR = cv2.imread("lem.png")
lem = cv2.cvtColor(lemBGR,cv2.COLOR_BGR2GRAY)

# Dilate the image in order to close any external contour of the leming
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))
lem = cv2.dilate(lem,kernel)

# Identify holes in the leming contour
# This could be done by iterative morphological operations,
# but this is not directly implemented in OpenCV
contour,hier = cv2.findContours(lem.copy(),cv2.RETR_CCOMP,cv2.CHAIN_APPROX_SIMPLE)
# And fill them
for c,h in zip(contour, hier[0]):
    if h[3]!=-1:
        cv2.drawContours(lem,[c],0,255,-1)

# Now bring the leming back to its original size
lem = cv2.erode(lem,kernel)

# Remove the cord by wiping-out all vertical lines
kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(15,1))
#lem = cv2.erode(lem,kernel) # first wipe-out
#lem = cv2.dilate(lem,kernel) # then bring back to original size
# erode and then dilate is the same as opening
lem = cv2.morphologyEx(lem,cv2.MORPH_OPEN,kernel)

# Find the contour of the leming
contour,_ = cv2.findContours(lem.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)

# And draw it on the original image
for c in contour:
    # enter your filtering here
    x,y,w,h = cv2.boundingRect(c)
    cv2.rectangle(lemBGR,(x,y),(x+w,y+h),(0,255,0),2)

# Display the result
cv2.imshow("lem",lemBGR)
cv2.waitKey()

cv2.imwrite("lem-res.png",lemBGR)

And below are my results.

Answer 2

Assumptions

1. The string is thin compared to the toy hanging from it.

2. The string is oriented in up-down-direction in all frames.

Proposed solution

1. Calculate the row-sum of the binary image. This gives you a column vector that holds the number of foreground pixels of each row.

2. Analyze this vector, e.g. by simple thresholding, for a steep increase of foreground pixels.

3. This position is the position where the toy starts. Hence, you can neglect the corresponding upper part of the image, because it mainly contains the string.