2
$\begingroup$

I used the CUSUM algorithm to detect steps in data. Basically the data looks like this, the data has a constant amplitude and then there is a rapid variation or a step. For example, the signal has a value of 1000 and rapidly drops to 950 or 900. With the CUSUM algorithm, I can detect easily these changes and the time at which they occur. But now I would also like to automate the amplitude estimations of these steps. Eventually, I'll need to implement this in real-time ( a small delay is tolerable for the amplitude estimation)

One solution I would like to try is this one :

  • take N samples before the step and take N samples after the step and subtract the respective averages.

Is there a better solution?

Edit : - The transition from 1000 to 900, for example, might take 10 samples in all.

$\endgroup$
  • $\begingroup$ If you also calculate the standard deviations, you could calculate a T statistic $\endgroup$ – Stanley Pawlukiewicz Mar 26 '18 at 20:24
1
$\begingroup$

Your straightforward idea of using moving averages might actually work quite well. Since you ask "is there a better solution", I must ask what you mean by "better."

Change point detection is a well studied problem in statistics and signal processing. The problem has many variations depending on your optimality criteria, and prior knowledge about the data. For example: do you care about catching the change point as quickly as possible? Or do you care about the precise location of the change? Or do you care about the exact amplitudes before and after the change, or just the difference in amplitudes?

Since you have a real-time requirement, I would search for "online Bayesian change point detection" which will show you lots of papers, and many packages in R to play with.

https://www.r-bloggers.com/a-simple-intro-to-bayesian-change-point-analysis/

https://www.jstatsoft.org/v23/i03/paper

https://hips.seas.harvard.edu/files/adams-changepoint-tr-2007.pdf

Detection of Abrupt Changes: Theory and Application, Basseville and Nikiforov

$\endgroup$
  • $\begingroup$ I am mostly interested in the amplitude estimation. I don't need to find the exact point, though it might help. In my real-life application the transition from 1000 to 900 will probably last a few samples because of pre-filtering that I need to filter out some weird harmonics. Thanks for the info! $\endgroup$ – Ben Mar 27 '18 at 0:01
  • $\begingroup$ If you make your question more specific with these additional details I'll try to update my answer with some algorithm ideas. Some sample data and plot might help too. $\endgroup$ – Atul Ingle Mar 27 '18 at 3:18
0
$\begingroup$

You may also want to try applying a "median filter" to your data before doing an average. I don't know is that is already mentioned in the references Alul Ingle gave. It should give you an improvement on the averages of your levels without impacting the drop size.

Hope this helps.

Ced

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.