What is the story behind the story about SIFT descriptor?

Question

The following is from Lowe 2004 paper ( http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf ).

One obvious approach would be to sample the local image intensities around the keypoint at the appropriate scale, and to match these using a normalized correlation measure. However, simple correlation of image patches is highly sensitive to changes that cause misregistration of samples, such as affine or 3D viewpoint change or non-rigid deformations. A better approach has been demonstrated by Edelman, Intrator, and Poggio (1997). Their proposed representation was based upon a model of biological vision, in particular of complex neurons in primary visual cortex. These complex neurons respond to a gradient at a particular orientation and spatial frequency, but the location of the gradient on the retina is allowed to shift over a small receptive field rather than being precisely localized. Edelman et al. hypothesized that the function of these complex neurons was to allow for matching and recognition of 3D objects from a range of viewpoints.

I am trying to understand SIFT descriptor. I understand the previous stage (keypoint detector).

I don't know why it is implemented that way. I want to know the story behind the story.

Answer 1

The descriptor obtained from a $64\times 64$ neighborhood of interest point at the obtained scale.

It will divide this $64\times 64$ region to $16\times 16$ patches which lead to 16 patches.

For each patch we calculate the gradients and then find the dominant direction of gradients(which has some details), then taking the dominant direction as the reference direction we will divide the 360 degrees to 8 angular region each has 45 degrees, then sum over the magnitude of each gradients which lie in each angular region.

We could consider this as distribution or 8 bin histogram of gradient direction (considering strong gradients has more information we have to use them with higher weight in calculation of distribution so we use their magnitude as their weight which leads to sum over their magnitude). Then we will normalize these histograms.

At the end for each patch we have an 8 bin histogram and we have 16 patches which leads to 128 number descriptor.

By finding dominant direction our descriptor also becomes rotation invariant. By using gradients our descriptor becomes invariant regarding to baseline illumination and by normalizing obtained histograms our descriptor becomes invariant to the contrast of image.