How to discard deviating values from standard deviation and running average

Question

I'm building a inductive vehicle loop detector. By measuring the frequency of a LC-oscillator where the cars will pass over the inductor of the LC-Oscillator.

If the cars pass over the inductor the frequency will change over time according to one of the following profiles.

The frequency will also slightly change / drift based on environmental variables: like temperature, humidity or other variances in the environment of the loop. To filter these last variations out, I used running average and std_variation based on: This post/comment on stack exchange. And was planning to detect vehicles when the frequency would go too many standard deviations away from the average.

The measurements made when there is a vehicle present shouldn't be added to the average calculation. This is easy to do when a certain threshold is passed.

But the problem is when a car is approaching the frequency will start to change but hasn't yet passed the threshold so these samples are added to the statistics. When a car approaches very slowly the standard deviation will be changed so much that the car will never be detected.

What solutions / alternatives / suggestions do you have to detect approaching cars and discard the values from the statistics for the reference empty loop.

Thanks in advance

---

Update: because I didn't get any response in the DSP section, I asked the question in the electronics section.

Answer 1

Let's ignore the issue that you already have a discrimination problem in your hands (bus/car/truck/van) there is an initial discrimination problem that has to be solved for the data to be considered valid for any moving object.

The issue is one of rate of change, so the problem should probably be moved into derivative space. That is, ignore the mean frequency and only use the derivative of the curve. Extracting derivatives of real data is a tricky issue as a true derivative amplifies the noise, so you end up with a bandpass filter with a single zero at DC that carries out a first derivative only for the bandwidth of interest and attenuates the noise. (That's a field all in itself, but this should get you started).

You are left with a curve that represents the rate of change.

Ignoring the sign and the additional gains in information that you could extract by using the statistics of the signal, the problem is basically reduced to that of a binary classifier, which in many fields is described by the receiver operating characteristics curve (ROC). You can see these in the following figure. Depending on the location of the threshold you get a different mix of false positives and negatives.

Clearly in this image from wikipedia the signal and noise are represented by Normal distributions, but that is seldom the case. In your case it might be a safe assumption for the noise, but not for the signal.

Given that you have additional statistics that you can use for your vehicle classifier, you can allow for a higher proportion of false positives to get through.

A very similar problem to this is in the biological field with the detection of spikes in neural action potentials. The way this is commonly solved is by using the standard deviation (i.e., power) of a section of the input that is "known" to be free of signal and is therefore noise and using that standard deviation to set the threshold (in these applications the voltage threshold is set between 4 to 6 standard deviations of the noise). Of course, this has not solved the problem, as you still have to figure out how to get the signal out of your noise calculations.

A clever way to do this, is to rely on the statistics of the noise being Gaussian, and only use the tail of the total input distribution where you are almost assured that all you will see is noise. And infer the noise standard deviation only from that tail. A family of algorithms can be derived from this, in neural spike detection these are known as the ADAband family of algorithms.

The idea is:

1. calculate the power (squared magnitude) of the input
2. calculate a short-term estimate of the standard deviation using a small number of samples.
3. store a number of estimates, let's say 100 (a longer time segment) in an array.
4. sort the stored values (this is in a sense similar to a median filter).
5. use only the bottom bins (e.g., the first 15 represent the bottom 15% of the corrected standard deviation estimates.
6. weigh each bin appropriately according to the chi-squared distribution it comes from and combine them to produce a better standard deviation estimate.
7. appropriately filter/combine successive estimates to reduce its variability.

Do note that this is just to obtain a reasonable estimate for the threshold. Any classification algorithm (including CUMSUM) would require a threshold to be defined. The advantage here is that the threshold adapts to the conditions of the input.

Answer 2

Perhaps you could try a CUSUM-like algorithm. This algorithm works by detecting the change in a statistic, typically the mean. In your case, you want to detect a change in the "mean " frequency.

https://en.wikipedia.org/wiki/CUSUM

There are other change-detection algorithms, such as the filtered-derivative algorithm which could work well for you. But for the filtered-derivative, you need to specify a window length of say 10 seconds. So if the change occurs in more than 10 seconds, you could miss that event.

https://hal.archives-ouvertes.fr/hal-00461214v1/document

Just to clarify, the CUSUM algorithm is more complex but more flexible as you don't need to specify a window length which can increase latency, in your case a CUSUM-like algorithm might be more appropriate as you likely need low latency.