Convolution of an input signal with a fixed impulse response is a linear operation. However, if the input-output relation of a system is

$$y(t)=(x*x)(t)\tag{1}$$

then the system is non-linear, which is straightforward to show.

On the other hand, a system with input-output relation

$$y(t)=(x*h)(t)\tag{2}$$

is linear (and time-invariant) because it convolves any input signal $x(t)$ with a fixed impulse response $h(t)$, which is independent of the input signal.

Convolution of an input signal with a fixed impulse response is a linear operation. However, if the input-output relation of a system is

$$y(t)=(x*x)(t)\tag{1}$$

then the system is non-linear, which is straightforward to show. Similarly, any convolution with a kernel that depends on the input signal is a non-linear operation.

On the other hand, a system with input-output relation

$$y(t)=(x*h)(t)\tag{2}$$

is linear (and time-invariant) because it convolves any input signal $x(t)$ with a fixed impulse response $h(t)$, which is independent of the input signal.