3
$\begingroup$

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A is a matrix). I have been taught that null space is a set of vectors that are squished to 0 when transformation matrix A is applied. Then I came across SVM where kernel functions are used. I read that null spaces are also called kernels of a matrix. My questions are as follows.

  1. Are both kernel functions and null spaces same? If they are related, how are they related?

  2. What is the relation between kernel functions used in SVM and null space of matrix? If yes, how? Is this derived from the null space of transformations used in SVM?

  3. What is the reasoning behind kernels used in kernel convolution? For example: Is the Gaussian kernel used a representation of the transformation? How are they related to null spaces? If they are related, how are they related?

Could you please answer these questions? I am very confused. Thank you in advance.

$\endgroup$
1
$\begingroup$

A kernel in the context of digital signal processing refers to the impulse response $h[n]$ of a filter. Particularly for finite length FIR kernels, filtering is carried out by the convolution operator; hence the kernel of the filter is the kernel of the convolution sum. This kernel maps an input vector $x[n]$ into an output vector $y[n]$ i.e., $$ y[n] = h[n] \star x[n] = \sum_{k=-\infty}^{\infty} h[k]x[n-k] ~,$$the sum truncated to a finite length for practical FIR convolutions.

As you have described, a null space of a linear mapping is the set of vectors $x$ whose image is the null vector in the range space of the mapping given by the matrix A. As the name implies it is a set of vectors and not a kernel function (sequence). The only kernel in this context is the matrix A which maps input vectors to output vectors, just as in the context of a convolution operator. $$y = A x$$

$\endgroup$
  • $\begingroup$ So, is it safe to summarise that kernels in linear algebra are not related to kernel functions used in digital signal processing? $\endgroup$ – VaM999 Jul 12 '18 at 10:46
  • $\begingroup$ No. Since a convolution operation can also be expressed as a matrix multiplication (just as a linear mapping), the matrix kernel of the linear transform can be related to the impulse response kernel of the convolution... $\endgroup$ – Fat32 Jul 12 '18 at 10:58
  • $\begingroup$ Thank you. I just updated the question, I would like to know how they are related. Could you please help me clear my confusion? $\endgroup$ – VaM999 Jul 12 '18 at 12:03
1
$\begingroup$

Kernel is a polysemic term, even only in Mathematics. In algebra, it can be the for instance:

and some more. For integral operators, it often denotes a "constant" object around which variable objects are integrated, like the $K$ in:

$$\int f(x) K(x) dx \,.$$

As @Fat32 said, it can refer to the mask used for convolution (although I have heard the name more for image processing than for signals), which is quite similar to the above integral expression.

In statistical learning, as in operator theory, one often mentions positive definite kernels that extend the notions for positive-definite functions or matrices, that can cast high dimensional problems to lower ones.

Of course, a finite difference kernel in image processing is not positive definite. But to me, kernel is being used as an umbrella concept for "stuff that are somehow constant".

Yet, I shall think about it deeper, there might be stronger links.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.