What is the relation between kernel functions, kernels used in convolution and null spaces of a matrix?

Question

I have recently started learning about machine learning and have come across kernels and null spaces. I understand that null space is the set of all vectors that satisfy the equation A.v = 0 (Where A is a matrix). I have been taught that null space is a set of vectors that are squished to 0 when transformation matrix A is applied. Then I came across SVM where kernel functions are used. I read that null spaces are also called kernels of a matrix. My questions are as follows.

1. Are both kernel functions and null spaces same? If they are related, how are they related?

2. What is the relation between kernel functions used in SVM and null space of matrix? If yes, how? Is this derived from the null space of transformations used in SVM?

3. What is the reasoning behind kernels used in kernel convolution? For example: Is the Gaussian kernel used a representation of the transformation? How are they related to null spaces? If they are related, how are they related?

Could you please answer these questions? I am very confused. Thank you in advance.

Answer 1

A kernel in the context of digital signal processing refers to the impulse response $h[n]$ of a filter. Particularly for finite length FIR kernels, filtering is carried out by the convolution operator; hence the kernel of the filter is the kernel of the convolution sum. This kernel maps an input vector $x[n]$ into an output vector $y[n]$ i.e., $$ y[n] = h[n] \star x[n] = \sum_{k=-\infty}^{\infty} h[k]x[n-k] ~,$$the sum truncated to a finite length for practical FIR convolutions.

As you have described, a null space of a linear mapping is the set of vectors $x$ whose image is the null vector in the range space of the mapping given by the matrix A. As the name implies it is a set of vectors and not a kernel function (sequence). The only kernel in this context is the matrix A which maps input vectors to output vectors, just as in the context of a convolution operator. $$y = A x$$

Answer 2

Kernel is a polysemic term, even only in Mathematics. In algebra, it can be the for instance:

• the set of vectors which map onto zero (linear algebra)
• pair of elements that maps to a common value (set theory)

and some more. For integral operators, it often denotes a "constant" object around which variable objects are integrated, like the $K$ in:

$$\int f(x) K(x) dx \,.$$

As @Fat32 said, it can refer to the mask used for convolution (although I have heard the name more for image processing than for signals), which is quite similar to the above integral expression.

In statistical learning, as in operator theory, one often mentions positive definite kernels that extend the notions for positive-definite functions or matrices, that can cast high dimensional problems to lower ones.

Of course, a finite difference kernel in image processing is not positive definite. But to me, kernel is being used as an umbrella concept for "stuff that are somehow constant".

Yet, I shall think about it deeper, there might be stronger links.