0
$\begingroup$

I have a signal (x-axis: Time, y-axis: Data) which has more than 10000 samples per second. The signal itself represents some physical data (e.g. speed, acceleration, etc.) and has some noise on it.

Details about the considered signal:

  • The signal increases/decreases over the considered capture interval with a small gradient
  • Let's consider the case where the signal decreases:
    • Due to the arbitrary noise (which has a very small amplitude) it regularly happens that one (or more consecutive) samples are not smaller than their predecessor.
    • However due to the high sampling rate those deviations are just an issue for the evaluation routine (Python) trying to decide whether the signals is continously decreasing in the considered time interval.
    • For a human being, with optical inspection of the plot it is very clear that the signal is decreasing

What is the most robust way to determine, whether the signal is increasing or decreasing over time?

I thought about using a moving average filter but I am curious to know whether there is a more proper way to reach my goal.

Any answer is highly appreciated.

|improve this question
$\endgroup$
  • $\begingroup$ Are you sure you have "jitter", or is it noise? Jitter is disturbances in the sampling times. $\endgroup$ – Juancho Mar 18 '16 at 18:20
  • $\begingroup$ Any chance of including a plot of the data? We probably need to know a bit more about your problem in order to really nail down a solution. $\endgroup$ – Peter K.♦ Mar 18 '16 at 18:43
  • $\begingroup$ @Juancho Yes, you are right. It is noise. The unwanted deviations are in the captured data not in the time, which is equally spaced (equidistant). $\endgroup$ – lR8n6i Mar 19 '16 at 9:07
  • 1
    $\begingroup$ @Peter K. I will try to get some example data next week and add a plot. Until then, I added some details about the considered signal in my original question. $\endgroup$ – lR8n6i Mar 19 '16 at 9:31
1
$\begingroup$

You can simply average N samples to know if 100 samples was larger than the previous 100 samples, either in steps of 100 or in steps of 10 or whatever is ideal for your platform. it allows you to achieve average scale assessment of any number of values and at any intervals.

It's the most easy, simple, and customiseable way to do it. your eye is also just assessing the averages, so you just have to tell the pc to do the same as the eye, using a similar number of samples.

|improve this answer
$\endgroup$
  • $\begingroup$ I solved my problem using this method. It proved to be accurate enough for my use case. $\endgroup$ – lR8n6i Mar 21 '16 at 15:59
1
$\begingroup$

You may use a bandlimited differentiator (i.e. the cascading of a low-pass filter to remove noise, and a differentiator to get the value of the slope).

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks! I will give this a try. Hope there are some robust ways to implement a differentiator in Python... $\endgroup$ – lR8n6i Mar 19 '16 at 9:33
1
$\begingroup$

Here some answers for the case where you're processing your data in a single batch (as opposed to online/in real time). These methods will take all of your data into account, so local sample-to-sample fluctuations aren't a problem. These are very well established methods and it should be easy to find existing code.

If your signal is increasing/decreasing linearly with respect to time, then you can fit a line to the amplitude vs. time curve using linear regression. The slope of the line tells you the amount of the increase or decrease per unit time. Alternatively, you can calculate the Pearson correlation coefficient. Its value ranges from -1 for perfectly decreasing to 1 for perfectly increasing, with 0 meaning no dependence. Unlike slope, correlation would give you a measure of the degree of linear dependence between signal amplitude and time, but not the magnitude of the increase/decrease.

If your signal is increasing/decreasing monotonically with time but the relationship isn't necessarily linear, then you can calculate the Spearman rank correlation or Kendall's tau. Similar to the Pearson correlation, these statistics will measure how much your signal depends on time, ranging from -1 (perfectly decreasing) to 1 (perfectly increasing), with 0 meaning no dependence. Unlike Pearson correlation, the nature of the increase/decrease can be any monotonic function, not just a straight line (e.g. an exponential or logarithmic curve).

For any of these methods, you can do statistical tests to make more well-founded statements about the increase/decrease. The issue is that you're trying to estimate something about the underlying signal from a finite, noisy set of data points. For a given true/underlying signal, any statistic you compute will be different each time you run the measurement because the noise is different. So, say your statistic says the signal is increasing; is this because it's really increasing, or could it be because noise just made it look that way? Statistical tests will help answer that question. Things to look into here are 'confidence intervals' and 'statistical hypothesis testing' (i.e. 'p values'). Depending on your application and level of noise, statistical testing may or may not be necessary. There may be nuances if your noise itself has temporal correlations.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your feedback. If I will need some more robust algorithm, I am going to implement one of your suggestions. $\endgroup$ – lR8n6i Mar 21 '16 at 16:02
  • $\begingroup$ Well, I must say you got me with this Pearson correlation. I am playing around with it right now... I found and interesting Python command, pearsonr(x, y). Am I right that x must be the time values and y the data values of the signal to be evaluated (increasing or decreasing)? Or should x be a linear regression of y? What Pearson number would be accurate enough to decide whether a signal is increasing? Is f.e. 0.9 enough or should it be 0.999...? $\endgroup$ – lR8n6i Mar 21 '16 at 17:05
  • $\begingroup$ A Pearson coefficient of 0.99 seems to be just right for my use case. So I combined your suggestion with the one of @predatflaps which now represents my final solution. $\endgroup$ – lR8n6i Mar 22 '16 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.