Using Convolution as Feature Extraction

Question

I am now studying image processing in my spare time. My understanding of convolution is about 'response to a specific filter':

When we have a raw image, or raw signal; and a filter, aka kernel; we apply a 'moving dot product' between the image and the filter.

The filter/kernel represents our domain-knowledge about the raw signal, which is usually some kind of features or patterns. For image processing, such features are often edge, corner, SIFT, etc.

When we apply the "moving dot product", we practically divide the image into "patches" and ask each patch of the image: "Do you look like the feature I am looking for?", high response value means "yes, I am the feature (edge) you are looking for"; while low response value means "no, I am not"

My first question is : is my understanding correct?

My second question is: If my understanding is correct, then we should be able to design much more fancy kernels which represents some very specific features: such as eye (dark in the middle, surrounded by white areas), or nose (triangular shape). But I haven't found a lot of literatures in the area of defining your own kernel.

Can anyone provide any insight here?

Thanks!

Answer 1

Your understanding of the convolution process is correct. However, note that in the convolution the kernel is first mirrored before the dot product. Though, for a symmetric kernel this does not matter.

Understanding the convolution in your way, of finding specific features, is a similar interpretation as that of a Matched filter. Furthermore, using the convolution to find a specific part of the image (e.g. eye), is sometimes called Template matching (but here, usually correlation is used, which is equal to convolution except the kernel is not mirrored).

Another interpretation of the convolution is done via the convolution theorem, which says that in the frequency domain, the frequency response of the filter and the image are multiplied. Essentially, you can then understand the filter to enhance / reduce higher or lower frequencies, (or do more fancy stuff in the frequency domain). With this interpretation, it sometimes becomes more understandable why a certain kernel has a certain effect (e.g. the Gaussian filter is easier to understand as a lowpass than a filter to look for Gaussian shapes in the image).

Answer 2

It is possible using something like Multirate filter bank and multidimensional directional filter banks. The issue with such approach is that you need do it with different resolution using sorf of Pyramid. Also, performance might not be great as processing time is spent on resampling and applying filters that is large in size. You need to be careful to use separable filters, otherwise complexity will be N*M (image size product filter size) for every pixel (unless you use FFT, that is cumbersome with the filters that are not powers of 2 in size).