What are the disadvantages of having too many poles?
Poles in a filter come from recursions. Consider a discrete filter
$$y(k) = x(k) + \alpha y(k-1)$$
where $y(k)$ describes the output of the system, $x(k)$ the input and $y(k-1)$ the output of the system the sample before.
Now what the recursive part $y(k-1)$ does is that it feeds the system the last output, which can cause problems depending on the $\alpha$ you choose. If $\alpha \geq 1$, the system reacts to any given input with an ever increasing output, which makes the system unstable.
Now the more complex our system gets, the more things you have to factor in for its design, as pointed out in Stanleys' answer. Problems related to poles do not stem from their count however.
Also, poles generally increase the magnitude of the systems' transfer function, causing certain frequency components of signals to be amplified by the system.