Say we have a linear system with unity feedback, with loop transfer function $L(j\omega)$. The closed-loop transfer function from reference to output is $T(j\omega) = \frac{Y(j\omega)}{R(\omega)}=\frac{L(j\omega)}{1+L(j\omega)}$.
At frequencies for which $L(j\omega)$ approaches $-1$, clearly $|T(j\omega)| \rightarrow \infty$, so the system is unstable - if excited at this frequency, the output is unbounded.
But systems can be unstable even if $L(j\omega)$ never equals $-1$. From the Nyquist plot, we can show that $T(j\omega)$ can have poles in the right half plane if the contour encircles $-1$, even if $L(j\omega)$ never exactly reaches it. (However, I have very little intuition for why this is true - I just see it as a theorem from complex analysis that happens to be useful here).
Alternatively, from the Bode plot, we say a system is unstable if there are any frequencies for which $|L(j\omega)|$ > 1 and $\angle L(j\omega)< -\pi$ (i.e. phase margin is negative, or gain margin is less than unity). However, I'm not sure why these two conditions result in instability, since these don't result in $|T(j\omega)|$ going to $\infty$.
Two questions:
(1) If $|T(j\omega)|$ never goes to infinity (which is the case when $L(j\omega)$ is never exactly $-1$), how can a system possibly be unstable? Is $|T(j\omega)| \rightarrow \infty$ not the right criterion for deciding whether a system is unstable?
(2) Intuitively, why is a system unstable if there are any frequencies for which $|L(j\omega)|$ > 1 and $\angle L(j\omega)< -\pi$? I understand that you can see it from the Nyquist plot because these two conditions tend to result in encirclements of $-1$ in the $L(s)$ plane, but I'm looking for the intuition.