I know about Harris corner detector, and I understand the basic idea of its second moment matrix, $$M = \left[ \begin{array}{cc} I_x^2 & I_xI_y \\ I_xI_y & I_y^2 \end{array} \right]$$, edges and other unstable points can be removed via $M$.

But about Hessian detector, it uses Hessian matrix to detect key points and remove edges, $$\mathcal{H} = \left[ \begin{array}{cc} I_{xx} & I_{xy} \\ I_{xy} & I_{yy} \end{array} \right]$$, and I don't understand how could $\mathcal{H}$ remove edge and detect stable points? What's the intuitive basic idea behind it?


2 Answers 2


I will try to avoid math, because math and "how to do it" tutorials can be easily found.

So, I start by pointing out one VERY important thing: One does not compute Harris for a single pixel, but for a vicinity (a patch of image) around that pixel! Let $I(i)_{xx}, I(i)_{xy} ...$ be your derivatives for a point $i_0$, then,

$H = \left[ \begin{array}{cc} \sum_{i\in V}I(i)_{xx} w (i-i_0) & \sum_{i\in V}I(i)_{xy}w (i-i_0) \\ \sum_{i\in V}I(i)_{xy} w (i-i_0)& \sum_{i\in V}I(i)_{yy} w (i-i_0)\\ \end{array} \right] $

The $w(t)$ is a Gaussian kernel. The previous eq tells you to integrate the derivative values over vicinity $V$ around current pixel. Each value of the neighbors is multiplied with a value that shrinks as the distance increases. The law of decreasing follows a Gaussian, because $w(t)$ is Gaussian centred at $i_0$. And that's it with math.

Now, back to the empirical observations. If you use solely the derivatives, and that pixel is part of a linear structure (edge), then, you get a strong response for the derivatives. On the other hand, if the pixel is at a corner (an intersection of two edges) then, the derivative responses will cancel themselves off.

Saying that, the the Hessian is able to capture the local structure in that vicinity without "cancelling" effect. BUT very important, you have to integrate in order to get a proper Hessian.

Having a Hessian, obtained using Harris method or by other means, one might want to extract information about the vicinity. There are methods to get numerical values on how likely is to have an edge at current pixel, a corner, etc. Check the corner detection theory.

Now, about "stable points" or salient points. Picture that you are in a foreign town with no GPS and only with a good map. If you are "teleported" in a middle of a street, you might locate the street on the map, but you cannot tell where exactly are you on that street or in what direction you should go to move left or right (wrt to the map). Imagine now that you are at a intersection. Then, you precisely can point your position on the map!. (Of course, assume that two streets doesn't intersect more than once).

Imagine now that you must match two images. One acts as a map, and the other as the city. You must find pixels that can be uniquely described, so you can do the matching. Check images on this post for example of matching. These points are called salient points. Moreover, the corner points tend not to change their 'cornerness' properties when the image is scaled, translated, rotated, skewed, etc. (affine transforms) This is why they are called "stable".

Some points in the image allows you to uniquely identify them. These pixels are located at corners or at intersection of lines. Imagine that your vicinity $V$ is on a line. Except for the orientation of the line, you cannot find anything else from that vicinity. But if $V$ is on a corner, than, you can find out the directions of the lines that intersect, maybe the angle, etc.

Not all corner points are salient, but only corner points have great chances of being salient.

Hope it helps!

p.s. How to find if a point is corner or not, take a look at Harris paper.

p.p.s. More on matching, search for SIFT or SURF.

p.p.p.s. There is a "generalization" of the Harris method, called Structure Tensor. Check Knutsson seminal work!

  • $\begingroup$ Small correction: integrating in the vicinity of each pixel is important for Harris and structure tensor approaches. For the Hessian it doesn't make much sense: It's equivalent to using a larger filter size for the 2nd order derivative filters. $\endgroup$ Oct 23, 2013 at 15:03
  • $\begingroup$ @nikie you are correct, if you have an analytic signal. For digital images (where you have discrete values on a grid) the reasoning does not apply. You need to integrate the Hessian, otherwise you get a degenerate matrix. There is a form of structure tensor that does not need integration and can work with derivatives computed only for current pixel. See Knutsson link in my post. $\endgroup$
    – visoft
    Oct 23, 2013 at 15:20
  • 1
    $\begingroup$ @visoft: I'm not sure what you mean. Strictly speaking, the Hessian is only defined for differentiable functions, not for images. You usually approximate it by convolution with a derivative filter kernel (e.g. Gaussian derivative). If you do that, you get a Hessian matrix for each pixel that isn't degenerate. And integrating over these filter results has the same effect as using a larger kernel. It just takes more CPU cycles. $\endgroup$ Oct 23, 2013 at 17:38
  • $\begingroup$ Ok, I'll check the literature and get back to you. $\endgroup$
    – visoft
    Oct 23, 2013 at 18:37
  • $\begingroup$ @nikie I edited the post, removing Hessian references. $\endgroup$
    – visoft
    Oct 24, 2013 at 16:12

For another perspective on using $\mathcal{H}$, remember that the first derivatives ($I_{x}, I_{y}$) are the slope at a point, and tell how strongly an image is changing in the $x$ or $y$ direction.

The second derivatives are the curvature- they tell if the area around the point concave up or down. To remember how this is used, recall the second derivative test in multivariable calc for finding local max/minima:


The test is $D = I_{xx}I_{yy} - I_{xy}^2$, and this is exactly what's used in the feature detector of SURF (as well as other interest point detectors). It's called the "determinant of the Hessian", $\det \mathcal{H}$ because it uses the determinant of that 2x2 matrix to summarize the curvature at a point.

When $D>0$, the point looks like either a bowl upward or downward, while $D<0$ looks like a saddle point. Since an edge looks more like a saddle than a bowl, this is one way that you can separate good, stable interest points from edges using the Hessian (but there are other trick you can do with $\mathcal{H}$ as well):

Bowl vs saddle


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.