Computing the derivative from a set of real-time data?

Question

So I have an array of say 25 samples and I would want to be able to note the trends of whether it's decreasing n or increasing from those 25 sample time interval(basically 25 samples array is my buffer that is being filled by every say 1 ms). Note that it is general trend that I am looking for, not the individual derivative(as I would have obtained using finite difference or other numerical differentiation techniques). Basically I expect my data to be noisy so there might be ups and downs even after doing filtering and so on. But it's the general trend of increasing or decreasing behaviour that I am looking for. I want to integrate the increasing/decreasing behaviour in every ms to trigger some event which is more of a user interface event (blinking a LED) so it does not have to very delay of processing as long as I can detect the general trend.

Thanks in advance!

Answer 1

Depending on the level of noise you expect to encounter on your signal, you might actually be able to use the finite differencing you suggest, originally. If you want to know the general trend of your data upward or downward, sum together the, in your case, 24, differences. So, lets say that you have some metric, $D$, for your signal. Lets say that for a given signal, $x[n]$, the first order difference is denoted as $x\prime[n]$ (and is, of course, one entry shorter than $x[n]$). Define $D$ as

$$ D = \sum_{n=0}^{N-2} x\prime[n] .$$

Then, classify your signal as uptrending or downtrending via

$$ D > 0 ~\therefore~\uparrow$$ $$ D < 0 ~\therefore~\downarrow$$

I made some basic assumptions about the nature of the signals in question. I just did the most basic analysis and thought, well, maybe just use a ramp, $r[n]$, corrupted by some noise, $z[n] \sim \mathcal{N}(0,\sigma^2)$. This may or may not be the situation you're looking at, so you'll need to adjust accordingly and see what works for you. But my final model was just the simple $$x[n] = r[n] + z[n]$$ for the length you gave, $N = 25$.

I went ahead and ran some experiments in MATLAB. I used the following code:

experiments = 1000;
N = 25;

slope_min = -2;
slope_max = 2;
slope_range = slope_max - slope_min;
noise_level = 2;


for i=1:experiments

    slope(i) = slope_range*rand(1)+slope_min;
    r = slope(i)*(1:N);
    z = noise_level.*randn(1,N);

    x = r + z;

    diffsum(i) = sum(diff(x));
end

First, let me show what some of the signals look like...

So, I varied the variance of the noise signal, $z[n]$ and recorded the following correlations between the true slope (up or down trend) of $r[n]$ in comparison with the metric we defined above, $D$. If $D$ does a good job determining the trend of the signal, we should see a strong correlation between downtrends (negative slopes) and negative $D$ and uptrends (positive slopes) and positive $D$.

We can see from the above that $D$ does seem to do a decent job. As the noise level increases (or the signal moves further away from our model), the correlations are not as strong. Try it out on your signals, see if does anything for you.

Answer 2

A more principled way of solving this problem is to apply signal detection theory (a.k.a hypothesis testing). I will outline here an easier case where we are trying to decide if the data has a positive slope trend vs. no trend. Setting up the hypotheses, $$ H_0: y(n) = w(n) $$ $$ H_1: y(n) = A n + w(n), \; \; A>0. $$

Here $\{y(n)\}_{n=0}^{24}$ is the data you get, $A$ is an unknown positive slope, $\{w(n)\}_{n=0}^{24}$ is i.i.d. Gaussian noise $N(0,\sigma^2)$ with known variance. Working out the log-likelihood ratio, your test is of the form $$ t = \sum_{n=0}^{24} n \; y(n) \underset{H_0}{\overset{H_1}{\gtrless}} \gamma $$ where $\gamma$ is set in order to obtain a user specified false alarm probability. For instance, if you want your false alarm rate to be 0.05 (i.e. a 5 percent chance of announcing positive slope when there is no trend in reality), then the parameter $\gamma = (\sigma \sqrt{\sum_{n=1}^{24} y(n)^2}) \Phi^{-1}(0.05)$, where $\Phi$ is the c.d.f. of a Gaussian $N(0,\sigma^2)$.

Practically, you would calculate $t$ and if it exceeds this $\gamma$, conclude that there is positive trend. Note that there is no finite differencing in the calculation of the test statistic $t$, hence the procedure will be computationally quite stable.

If you want to avoid doing this math, I am sure you will find hypothesis testing routines for trend detection in commercial statistical software suites like R or Minitab.

Answer 3

Just adding to what Eric said, the same can be done in Python with scipy.signal.detrend

Answer 4

If you need to smooth it, convoluting with the derivative of a gaussian is a standard approach if your data is noisy. This is effectively equivalent to using gaussian smoothing then applying the finite difference method suggested. The reason it is nice to know about this equivalence is because one may have developed a good intuition for whether, and how much, guassian smoothing is appropriate for a signal, and that then carries over to finding a derivative of that signal.

What will work depends a lot on your data set. In image processing slightly smoothed derivatives are often used, but for HOG filters on 200ish pixel wide images finite differences work better for person detection than smoothed derivatives.

Answer 5

You need a model, or potential models, of the underlying cause of the data and the noise source, and the computed result will depend on which models you choose or guess.

For instance, if you think the data source can be modeled by a linear, quadratic, or other polynomial equation in Gaussian noise, then you might try a linear regression or some other polynomial regression fits. Then check the regressions to see if or which model was a good match, and if close enough, use the (least squares) estimated polynomial coefficients to calculate the derivatives. If you predict a sinusoidal model, then perhaps something similar using a DFT/FFT to find and extract the sine waves and phases, and then compute some derivatives from the sinusoidal estimates.