It has been some time since I've looked at Chaos theory. Since Chaos theory is mainly examining non-linear equations, the idea of BIBO isn't sufficient. One of the ways to characterize a non-linaer system is via Lyapunov exponents. It is a way of characterizing the predictability of the system.
This is definitely not the usual thing that's covered in DSP, since DSP is mostly concerned with Linear Systems or systems that have been linearized.
Another version of stability comes from Compressive Sensing (CS) or Sparse Reconstruction. In CS you are looking at solving $\mathbf{Ax}=\mathbf{b}$ where there are more equations than unknowns but we want an vector $\mathbf{x}$ to be sparse i.e. only have a few non-zero elements (the other elements should be zero or small in comparison to the few large elements). Note that because the system is under-determined there are multiple solutions for $\mathbf{Ax}=\mathbf{b}$.
A concern in CS is the stability of the answer i.e. if we recover a sparse vector we want other sparse solutions to be nearby, so if we don't have an exact answer we are at least within a known error bound of the correct answer. If we can find the largest real number $\alpha$ and the smallest real number $\beta$ such that
$$\alpha||\mathbf{x}_1-\mathbf{x}_2|| \leq A(\mathbf{x}_1-\mathbf{x}_2) \leq \beta||\mathbf{x}_1-\mathbf{x}_2|| $$
for all vectors $\mathbf{x}_1$ and $\mathbf{x}_2$. Then we can bound the error on the recovered vector in cases of: noise-free, noisy, and modelling error i.e. error in the $\mathbf{A}$ matrix. The above equation is referred to as the bi-Lipchitz condition and leads directly to idea of the Restricted Isometry Constraint (RIC) - which is calculating the values of $\alpha$ and $\beta$ when the vectors $\mathbf{x}$ are constrained to be s-sparse i.e. have s elements or fewer that are non-zero. Note that RIC is a sufficient condition but not a necessary one. Other characterizations include: the Null Space Property, the Spark, and Mutual Coherence etc. Good overviews of the theory are:
Blumensath, and
Davenport
Again this is a highly specialized area and beyond the usual scope of DSP, but you asked the question :)