An IIR filter has poles, which means it has feedback from the system output that factor into its output computations. The poles of a discrete time system must have an absolute magnitude smaller than 1 for the system to be stable. This equates to having the poles fall inside a unit circle in the complex plane (generally referring to the z plane associated with the z domain transfer function of the system).
The analogous situation for "real world" systems (systems that can be modeled by linear differential equations with constant coefficients - thus can be represented by a transfer function in the Laplace domain or S domain), is that the poles of the system transfer function must be on the left hand side of the S plane.
For discrete time systems, if poles are outside the unit circle, values represented internally as well as the system output can grow without bound. If poles are located on the unit circle, values internal to the system as well as the output may oscillate.
For a stable system, internal values and the system output are expected to be a function of the system input. This will not be the case if the system is oscillatory or has values that exceed the size of the numbers used to represent internal values (register overflow).
If poles are too close to the unit circle, the system may be marginally stable. Is such cases, the system may behave for some limited set of input conditions, but may become uncontrolled for other conditions. The reason for this is that DSP systems are inherently non-linear. Internal values are often represented using fixed point arithmetic and are always stored in finite sized registers, so if the maximum values that can be represented are exceeded, the system experiences a non linearity. Another feature of DSP systems is that signals are quantized. Signal quantization adds low level non-linear effects to the system. Quantization error is often modeled as noise, but it can become correlated with system values and result in oscillations called limit cycles.
Care must be taken to avoid saturating (hitting absolute maximum values) in fixed point representations. Generally it is considered better, if absolute values are exceeded, that the representation be held at the maximum value rather than cause a sign inversion of the value. This is called saturation limiting and it does a better job of preserving the system behavior that allowing sign inversions.
In general an unstable DSP system will saturate to a fixed value or oscillate in a chaotic way due to internal nonliteraries.