Hough lines and Convex Hull methods give jagged lines

Question

I need to do a perspective transformation on an image section from newspaper, here the girl's image. The section's coordinates are given as rough estimates. I need to find the boundary of the section and do a perspective transformation. I had been referring the following link.

http://opencv-code.com/tutorials/automatic-perspective-correction-for-quadrilateral-objects/

However, Hough Transform and Convex Hull method on contour finding give jagged lines. This is how the edge detection on the section looks like.

edges = cv2.Canny(gray, 50, 150)

How can I improve the Canny Edge lines to make them straighter so jagged lines are removed? The Convex Hull finds the boundary but not with a number of smaller lines and the boundary is not complete. Here is how the Convex Hull method's boundary looks like.

contours, hierarchy = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    for i, cnt in enumerate(contours):
        if hierarchy[0,i,3] == -1 and cv2.contourArea(cnt)>5000:
            hull = cv2.convexHull(cnt, returnPoints = True)
            print "hull", hull
            for tempodd, itemodd in enumerate(hull):
                if tempodd % 2 == 0:
                    tempeven = tempodd
                    itemeven = itemodd
                    continue
                cv2.line(img, (itemeven[0][0], itemeven[0][1]), (itemodd[0][0], itemodd[0][1]), (255,0,0), 4)
                print "itemodd", itemodd, "itemeven", itemeven

Here since the Canny is getting jagged lines, the convex hull method gives multiple lines too. Point me where I am going wrong. Thank you.

Answer 1

Let's go step by step.

1. First of all you can remove perspective distortion without camera calibration. e.g.

   Robust Radial Distortion from a Single Image, Faisal Bukhari and Matthew N. Dailey

   Robust Line Based Calibration of Radial Distortion from a Single View, Thorsten Thormahlen, Hellward Broszio, Ingolf Wassermann

2. For any given image, Hough transform might find multiple lines, many of those would be an indicator of the same edge. But luckily you could easily merge the lines which overlap. To detect the overlap you could well use the angle between the lines and the distance of lines. This way, you obtain a clustering of the lines. In any case, finding a short line on each edge would be sufficient for you to recover the projective quad.

3. If you are always sure that the orientation of this quad is more or less fronto parallel, then you could as well use directed gradients (X or Y) to emphasize the vertical or horizontal lines, during the edge detection step.

I agree with the other comments on Gaussian smoothing and anti aliasing. I would not recommend the use of convex hulls though, as they are very noise sensitive.

Answer 2

You could solve the problem in several ways:

1. Image pre-filtering (smoothing) with some kind of Gaussian filter, before doing edge detection, will yield smoother edges.
2. Morphological operations, such as dilate or close on edge image will close the small holes.
3. Find 4 most dominant lines using Hough method on edges image and then find their points of intersection

Answer 3

So, basically what you want is the bounds of your picture. I think, although doing edge detection will yield corner feature after finding the intersection of the bounding lines, but that is more work.

I would suggest you of finding most dominant four corners by 'corner feature detection'(Using Harris or fast method). Then you have the bounds of your picture. Now you can have the sub image then apply the perspective transform to the raw image or to the corner vertices and then interpolate.