Getting started with wavelet-based real-time anomaly detection for temporal signals using Python

Question

I have an anomaly detection problem I wanted to consider addressing in software (python) with wavelets. I’m new to wavelets as well as the python’s spicy.signal wavelet package, let alone with the numerous additional packages floating around out there (e.g., PyWavelets).

Here’s the problem formulation: Consider a desired low-pass signal subject to various anomalies and a small amount of additive white Gaussian noise (AWGN):

$$ r(t) = d(t)x(t) + n(t) + a(t) $$

where $x(t)$ is the desired low-pass signal, $n(t)$ is the AWGN, and $a(t)$ are the additive anomalies and $d(t)$ are the multiplicative anomalies. The anomalies are roughly modeled as a linear combination of scaled and shifted versions of two types of effects: impulsive spikes, exponentially decaying transients. Multiplicative square pulse-type signal drop-outs directly affect the desired signal:

$$ a(t) = s(t) + e(t). $$

The spikes can be written as: $$ s(t) = \sum^{N_s-1}_{k=0} s_k (t)\\ s_k(t) = S_k \delta(t-t_k) $$ where $N_s$ is the number of spikes, $\delta(t)$ is the Dirac delta function, and $S_k$ and $t_k$ are the amplitude and delay associated with the $k$th spike

The exponential transients are written as: $$ \begin{align*} e(t) & = \sum^{N_e-1}_{k=0}e_k(t)\qquad e_k(t) = \left \{ \begin{array}{lll} E_k \cdot \exp({-[t-t_k]/\tau_k} )&& t\ge t_k\\ 0 && t< t_k \end{array} \right . \end{align*} $$ where it is assumed that the transients are well-separated in time and $\tau_k$ are known not to exceed some maximum.

Dropouts are crudely modeled as product of scaled and shifted square pulses: \begin{align*} d(t) & =\prod^{N_d-1}_{k=0}d_k(t)\\ d_k(t) & = \left \{ \begin{array}{lll} \epsilon_k && t\in[t_k, t_k+T_{d_k}]\\ 1 && t\notin[t_k, t_k+T_{d_k}] \end{array} \right . \qquad \epsilon_k\in[0,1) \end{align*} where it is assumed that the drop-outs do not overlap in time and that $T_{d_k}$ are less than some maximum.

The goal is to flag the time intervals affected by the anomalies which are further assumed to arise one at a time (e.g., a spike will not occur in the middle of a dropout). The exponential transients are considered to be non-negligible for a limited period of time (say up to 90% attenuation from the start of the transient).

I don’t want to rely heavily on a precise parametric model for detecting the anomalies since the above anomaly modeling is only a crude approximation. It seems like wavelets may be a good method for localizing these effects in the received signal. The desired signal’s bandwidth is far lower than that of the anomalies which overlap and extend beyond the desired signal bandwidth. It is necessary for the detection procedure to be at least near real-time.

I don’t know much about wavelets aside from a few very basic tutorials and I haven’t played with any python-based wavelet packages.

My questions are as follows:

1) Are wavelets indeed well-suited to this type of detection problem? If not, what other approaches would make sense to try?

2) If wavelets do make sense here, what type of wavelet transform should be considered (e.g., continuous or discrete)?

3) What type of mother wavelets should be tried here?

4) Are the wavelet routines in scipy.signal adequate for a preliminary, meaningful study? If so, which are the relevant functions? If not, which package would be preferable? PyWavelets?

5) Can anyone point me to resources that have addressed directly similar problems?

Answer 1

I suggest that you look at the Wikipedia article:

https://en.wikipedia.org/wiki/Change_detection

and first focus on the statistical aspects of the problem. The references cited reflect most of the foundational material of the topic. The article is of high quality. You should also think about the systems aspect of your problem like, the architecture of the system in the sense of issues like having a set of essentially special purpose detectors, or something that allows some interactions. Your problem statement says that only a single anomaly is present at a time, and this should be reflected in the organization of system. Questions such as Do you need automatic gain control (AGC) or do you need a constant false alarm (CFAR) behavior do have indirect effects.

The major departure from the usual change detection problem, is $a(t)$. Typically, it is in $a(t)$ that one wishes to find changes. I don't think that this is necessarily a problem but $a(t)$ will propagate through your detectors.

Change detectors work on changes from something typical and $a(t)$ seems like a good place to look for "typical". This is probably where your likelihood ratio test or score metric is going to reside.

I'm guessing that your essential problem is that $a(t)$ is from a remote sensor, and your anomalies are associated with a transmission channel. If so, transmission glitches are not typically subtle low SNR events that necessitate near optimal techniques needed to find them. They are obvious and are often managed by simple heuristics. Much that follows will be irrelevant if this is the case.

Finally, Wavelets. The answer is, you are going to determine this for yourself, by trial and error and thinking through what makes sense. One heuristic is that time-bandwith matching has a small loss relative to something like a matched filter.

The choice of basis functions is specific to your application. People use Wavelets. People use STFT, People use Wigner. People use Cyclostationarity. People use running histograms.

It all depends on how parsimonious a basis set represents your signals of interest, but you can't neglect $a(t)$. Those basis function have a direct bearing on characterizing "typical", which is why I strongly recommend you think about the statistical organization of your system, and then work on selecting your basis functions.