Is this because of poles?
Yes. A steep drop or rise in a filter's frequency-domain response can only be achieved with a filter that has a long memory. In a FIR filter, this long memory can only come from, well, being long. In IIR filter, this long memory can come from having poles that are close to the unit circle.
Your assumption why IIR filters can have steeper transitions from passbands to stopbands compared to FIR filters of the same order is correct: IIR filter have poles away from the origin of the complex plane, and poles inside the unit circle close to zeros on the unit circle cause the corresponding frequency response to change rapidly with frequency.
The FIR ...