Calculating a homography based on detected lines

Question

I know you can calculate homographies from image to camera plane using correspondence points between a "perfect model" and the image points.

I'm doing it for a football pitch/field, and have used edge detection to find the white lines in the pitch.

But the camera does not (always) cover all of the pitch, so I can't see all the corners... and I only the corners are 100% known points in the model (no other distinguished points).

So the problem is that unless the line intersects with another line and forms a corner, I only know the image points of the line, not it's corresponding "perfect/real-world" coordinates in the model.

Is there some way I can use the detected lines to calculate a homography, or even just a set of candidate homographies, even if the detected lines do not intersect with each other and create a corner?

Example image, showing the pitch, our field of view, and the points of the pitch where I can know the corresponding realworld/model coordinates(green circles), and an example of 2 lines that might be completely useless since in our field of view, I have no clue exactly at which point they start or stop in the corresponding realworld /model of the pitch:

The red lines are examples of lines which I would like to use, but I don't know their realworld coordinates, and it's kind of hard to estimate them because depending on the camera pose, the correspondent points could be "anywhere".

Answer 1

I will explain two approaches for this:

1) One approach would require a line matching algorithm. After matching the lines, you could simply use the end points of the lines in order to compute homography. To achieve that EDLine or LSD based descriptors are recently proposed in OpenCV. Also, hashing and fast matching of them are also implemented. Check out the videos here:

http://www.youtube.com/watch?v=MqMjvSkM39k

http://www.youtube.com/watch?v=naSWTlbg3To

The recent opencv_contrib repository contains the source code to these methods.

In the case that the line end points are noisy, you could then directly utilize the lines to compute the homographies. Such papers would then read:

Internal Report: 2005-V04 Computing Homographies from Three Lines or Points in an Image Pair G. Lopez-Nicolas, J.J. Guerrero, O.A. Pellejero, C. Sagues

Internal Report: 2003-V01 Robust line matching and estimate of homographies simultaneously G. Lopez-Nicolas

Probabilistic Matching of Lines for Their Homography Taemin Kim, Jihwan Woo, and In So Kweon

2) There is one method specific to the fields are given here:

"Using line and ellipse features for rectification of broadcast hockey video.", Gupta, Ankur, James J. Little, and Robert J. Woodham Computer and Robot Vision (CRV), 2011 Canadian Conference on. IEEE, 2011.

and

"Combining line and point correspondences for homography estimation.", Dubrofsky, Elan, and Robert J. Woodham. International Symposium on Visual Computing. Springer Berlin Heidelberg, 2008.

The idea is as follows: Any line, parametrized by its coefficients $\mathbf{l}_i=(u,v,1)^T$ maps to $\mathbf{l}_i^{'}=(x,y,1)^T$ in the other image using:

$$ \mathbf{l}_i^{'} = \mathbf{H}^T \mathbf{l}_i $$

In this form the equation can be directly plugged into the DLT method:

$$ A_i = \begin{bmatrix} -u & 0 & ux & -v & 0 & vx & -1 & 0 & x \\ 0 & -u & uy & 0 & -v & vy & 0 & -1 & y \end{bmatrix} $$

The only difference is the normalization, which you will find in the references above.

Adding Ellipses: Any point $x$ lies on the conic section $C$ if $x^TCx=0$. This gives rise to the transformation relation:

$$ C' = H^{-T}CH^{-1} $$

The references above also explain how to insert this constraint to the DLT algorithm.

Using ellipses and lines, it is possible to derive a robust projective relationship.

Answer 2

If the lines are not parallel, you can calculate the point of their intersection and use it as a point of reference. In your painting, you can use the purple points as well:

By the way, the intersection of the lines need not to be in the image. As long as the lines are parallel

If the lines are parallel, you can use them to get additional constraints. For instance if you have N<4 points and K lines you might be able to estimate the transformation

Recall that the equation of projective transformation is:

$x' = \frac{\left ( a_{11}x+a_{12}y+a_{13} \right ) } {\left ( a_{31}x+a_{32}y+1 \right )} \\ y' = \frac{\left ( a_{21}x+a_{22}y+a_{23} \right ) } {\left ( a_{31}x+a_{32}y+1 \right )} $

Your goal is to find the coefficients $a_{11},a_{12},a_{13},a_{21},a_{22},a_{23},a_{31},a_{32} \\$

Thus, if there is a line $ax+by+c=0$ that maps to $Ax'+By'+C$, then:

$ Ax'+By'+C = 0 \implies \\ A(a_{21}x+a_{22}y+a_{23}) + B(a_{21}x+a_{22}y+a_{23}) + C( a_{31}x+a_{32}y+1) = 0 $

It can be re-written as :

$ \left( \begin{array}{ccc} Ax & Ay & A & Bx & By & B & Cx & Cy \\ \end{array} \right) \left( \begin{array}{ccc} a_{11} \\ a_{12} \\ a_{13} \\ a_{21} \\ a_{22} \\ a_{23} \\ a_{31} \\ a_{32} \\ \end{array} \right) = -C $

$A,B,C$ are known values, because you calculated the lines equations. You can input any point $(x,y)$ for which $ ax+by+c = 0 $, and get additional constraint. Combine them together with the constraints that you get from points, and you might get additional information. Note that you will not get more information from more than two points, since any third point will add linear dependent lines to the matrix of constraints.

Additional references "Homography estimation by Elan Dubrovsky" - See part 2.3.1, estimation of homography from lines.