for a filter:
$$ H(z)=\frac{1+0.1z^{-1}}{1+0.1z^{-1}+0.9998z^{-2}} $$
Which precautions could be taken to ensure that the filter does not become unstable because of coefficient quantization? I've think that it would depend of the implementation chosen (I've read there are implementations pretty robust against quantization), but not really sure. Thanks in advance.
your filter is in the form of a standard biquad with 5 independent coefficients ($b_2=0$ in your case):
$$ H(z) = \frac{b_0 + b_1 z^{-1} + b_2 z^{-2}}{ 1 + a_1 z^{-1} + a_2 z^{-2}} $$
your two poles are the value of $z$ that make the denominator equal to zero
$$ 1 + a_1 z_p^{-1} + a_2 z_p^{-2} = 0 $$
or (multiplying both sides by $z_p^2$),
$$ z_p^2 + a_1 z_p + a_2 = 0 \tag{1} $$
and stability depends entirely on whether your poles (after coefficient quantization) are contained in the unit circle:
$$ |z_p| < 1 $$
or
$$ |z_p|^2 < 1 $$
the solution to Eq. 1 is (quadratic formula):
$$ z_p = -\frac{a_1}{2} \pm \sqrt{\left(\frac{a_1}{2}\right)^2 - a_2} $$
and, if the poles are complex conjugate, then $\left(\frac{a_1}{2}\right)^2 < a_2$ and your complex conjugate poles are
$$ z_p = -\frac{a_1}{2} \pm j \sqrt{a_2 - \left(\frac{a_1}{2}\right)^2} $$
in the case of two real poles, when $\left(\frac{a_1}{2}\right)^2 \ge a_2$, you must guarantee (to ensure stability):
$$ -1 < -\frac{a_1}{2} \pm \sqrt{\left(\frac{a_1}{2}\right)^2 - a_2} < +1 $$
that means both of the following must be guaranteed (after quantization):
$$ -\frac{a_1}{2} + \sqrt{\left(\frac{a_1}{2}\right)^2 - a_2} < +1 $$
$$ -1 < -\frac{a_1}{2} - \sqrt{\left(\frac{a_1}{2}\right)^2 - a_2} $$
what is necessary (but not sufficient) is that $|a_1| < 2$ and what also must be true is (getting rid of the $\sqrt{\cdot}$)
$$ \left(\frac{a_1}{2}\right)^2 - a_2 < \left(1 + \frac{a_1}{2}\right)^2 = 1 + a_1 + \left(\frac{a_1}{2}\right)^2 $$
and
$$ \left(\frac{a_1}{2}\right)^2 - a_2 < \left(1 - \frac{a_1}{2}\right)^2 = 1 - a_1 + \left(\frac{a_1}{2}\right)^2 $$
or
$$ a_2 > -1 - a_1 $$
$$ a_2 > -1 + a_1 $$
so for real poles ($\left(\frac{a_1}{2}\right)^2 \ge a_2$) you must make sure, after quantization that
$$|a_1| < 2$$ and $$-1 + |a_1| < a_2 \le \left(\frac{a_1}{2}\right)^2$$
for complex conjugate poles ($\left(\frac{a_1}{2}\right)^2 < a_2$), it's much, much easier. i'll leave this for an exercise for the reader but you can easily show that (for complex conjugate):
$$ |z_p|^2 = a_2 $$
which means
$$ 0 < |z_p|^2 = a_2 < 1 $$
and you still will see that $|a_1| < 2$ is necessary because $|a_1| < 2 a_2 < 2$. it turns out that $-\frac{a_1}{2}$ is the point on the $\mathfrak{Re}(z)$ axis that is midway between the two poles, whether they be real or complex conjugate.
now, if your digital filter is slewing from one stable complex pole case to another stable complex pole place, then $a_2$ is slewing from one value less than 1 to another value less than one and all filters in between will have $a_2$ less than one and will be stable, at least in a "static" sense. rapid coefficient slewing can create instabilities in and of itself, but if the slewing stops (at a stable condition), any instability also stops.
so in both real and complex conjugate cases, to have stability in the above biquad you must have
$$ -2 < a_1 < 2 $$
and
$$ 0 < a_2 < 1 $$
for the complex conjugate case $a_1^2 < 4 a_2$
or
$$-1 + |a_1| < a_2 \le \left(\frac{a_1}{2}\right)^2 = \frac{|a_1|^2}{4}$$
for the real pair case $a_1^2 \ge 4 a_2$.
note that, in the real pole pair case, we know that $$-1 + |a_1| < \frac{|a_1|^2}{4}$$ because $$-1 + |a_1| - \frac{|a_1|^2}{4} = -\frac{1}{4}(|a_1|-2)^2 < 0$$. so there is always a gap for $a_2$ to fit into.
The poles of that filter are extremely close to the unit circle (pole radius $=0.9999$). You will run into problems with any fixed-point implementation of that filter (with any reasonable word length), because it is not only coefficient quantization that will give you problems but also round-off noise (due to signal quantization). If you only consider coefficient quantization (not signal quantization), you need at least 14 bits for theoretical stability (for a direct-form structure). There are other filter structures, but they won't help much for such a critical case as given in your example.
What you need to do is ensure a certain stability margin, i.e. a maximum pole radius $r\le r_0$, where $r_0<1$. The appropriate value for the limit $r_0$ on the maximum radius depends on the available coefficient word length and on the chosen filter structure. Note that the shift of the pole locations due to coefficient quantization also depends on the pole angle (unless you use a coupled-form structure), so unfortunately there is no general rule for determining $r_0$.
EDIT:
I'll add some explanation in reaction to RBJ's comments. RBJ correctly claims that even when the coefficients are quantized, then, regardless of the number of bits, a second order section can always be made stable. This is obviously true, because we can always make sure that the (direct-form) coefficients $a_1$ and $a_2$ of the denominator polynomial $A(z)=1+a_1z^{-1}+a_2z^{-2}$ are inside the stability triangle:
$$\begin{align}a_2&\ge -a_1-1\\ a_2&\ge a_1-1\\ a_2&\le 1\end{align}$$
What I was talking about was the straightforward (or naive) quantization, where you just choose the quantized coefficient that is closest to the original coefficient given the number of available bits and the desired range of values. If you do this for the filter given in your question then you indeed need 14 bits in order to obtain a (theoretically) stable filter. As mentioned before, due to signal quantization that filter will still not be useful in practice.
So the approach I was talking about is to change the (unquantized) pole locations such that the maximum pole radius is less or equal to some chosen $r_0<1$, and then apply (naive) quantization (and of course check the result). RBJ's equivalent approach would be to do this in one step, i.e. quantize but at the same time guarantee that the quantized coefficients end up inside the stability triangle. Either way, you end up with a stable filter that deviates from the ideal filter. The magnitude of that deviation depends on how much the actual pole locations have to be changed in order to obtain a stable filter for the given number of bits spent on the coefficients.
To answer the direct question: the easiest test is the following:
This will also you allow to assess whether the quantization has changed the shape of the transfer function in an unacceptable way, even if it's still stable.
However, this specific filter will be very difficult to implement, it may be stable, but it will incredibly noise. The quantization during calculation induces the noise and this noise is amplified by the pole transfer function. This particular example has a maximum noise gain of 72 dB, so you will need a LOT of bits depending on what our signal to noise requirements are or consider implementing the fixed point math in double precision.
