We don't ignore poles on the contour. As mentioned in a comment, poles are avoided by modifying the contour as shown in the figure below, where a contour appropriate for a pole at $s=0$ is shown.
Fig. 1: Nyquist contour for a pole at $s=0$ (from "Modern Control Engineering" by K. Ogata).
The contour moves around the pole along a semi-circle centered at the location of the pole. The radius of that semi-circle approaches zero, such that the whole right half-plane is enclosed by the resulting contour. Note that by choosing the contour in this way, a pole on the imaginary axis is outside the contour, and will not add to the encirclements of the origin in the Nyquist plot.
Of course we could also move along a semi-circle in the left half-plane to avoid a pole on the imaginary axis. In that case, the pole would be inside the contour.
As an example, consider the function
It has a double zero in the left half-plane, one pole in the left half-plane, and one pole on the imaginary axis at $s=0$. If we use the contour shown in Fig.1, we get the Nyquist plot shown in the right-hand side figure below (the corresponding contour is shown on the left).
There is no encirclement of the origin, in agreement with the fact that there are no poles and zeros inside the contour. Note that due to our choice of the contour, the pole at $s=0$ is outside the contour.
If we choose a different contour with a small semi-circle in the left half-plane to avoid the pole at $s=0$ (left-hand side figure below), the pole at $s=0$ is inside the contour, and, consequently, the Nyquist plot shows one counter-clockwise encirclement of the origin (right-hand side figure below), corresponding to one pole and no zeros inside the contour.
In sum, poles on the imaginary axis are avoided by moving along semi-circles of infinitesimal radius, and, depending on whether the semi-circle is in the right or the left half-plane, the poles on the imaginary axis are either outside or inside the contour, which is reflected in the Nyquist plot by the number of encirclements of the origin.