In the book I'm studying from "Applied digital signal processing - Manolakis" it says:
The problem with FIR notch filters is that the bandwidth of the notches is large. A simple way to create sharper notches is to place the zeros on the unit circle and two complex conjugate poles at the same angle with the zeros, close to the zeros but inside the unit circle.
But why is that? How it changes the thing? I tested it in matlab and in fact I get a way better filter but I don't understand the reason.
Consider the frequency response for a simple FIR notch filter given as a single pole and single zero:
$$H(z) = \frac{(z-z_0)}{(z-z_p)}$$
When the poles and zeros are plotted on the z-plane, the unit circle represents the frequency axis. So if we sweep z over all possible values where the magnitude of z = 1 (the unit circle) the resulting response will be the frequency response of the filter, both in magnitude and phase. Thus since the unit circle is $e^{j\omega}$ for $\omega$ in the range of 0 to $2\pi$, the frequency response is expressed as:
$$H(e^{j\omega})$$
That said we can easily now answer your question graphically (and thereby intuitively) if we consider the numerator and denominator of $H(e^{j\omega})$ as two vectors in the 2d space of the z-plane as the value for z makes its way around the unit circle. This is shown in the graphic below for the simple DC notch filter with a single zero at $z=1$ (which is DC), showing the numerator and denominator vectors when $\omega$ is little less than $\pi/4$: ($z=e^{j\pi/4}$). The pole at the origin is "trivial" as in all cases the magnitude will be 1, so has no effect on the overall magnitude $|H(e^{j\omega})|$, but it does modify the phase of this transfer function. Therefore the overall transfer function will start with a null at DC and then grow rather slowly as z moves its way around the unit circle (all due to the magnitude of the numerator only).
The plot shows the pole and zero on the z-plane to the left, and then the vectors for the numerator and denominator are repeated on the right. (When subtracting 2 points on a 2D plane to describe a vector, the second point becomes the origin for the vector described).
Now for comparison consider what happens when we move the pole from the origin to be closer to the zero for the same position of z as in the graphic below. Notice in this case how the denominator is much smaller and quickly approaches the magnitude of the numerator, thus causing the DC notch to be much tighter.
The overall responses for both cases are shown below. The closer we make the pole to the zero, the tighter the response, but also the more precision is needed in a fixed point design.
This was explained simplest by using a DC notch filter. I have extended this to arbitrary notch filters for any frequency using the exact technique described by the OP (complex conjugate poles close to the complex conjugate zeros with the same angle) at this post:
