3
$\begingroup$

I would like a simplified measure to quantify spikes in my time series, I have one series that has many and I would like to compare it to another that has very few. I have thought through several options for example fitting a sine wave and looking at the R values as a measure of smoothness etc but I thought that a measure of the Coefficient of variation of the differences in the series may be suitable. I was wondering if this was a good approach and also as the mean values of the differences are very small I was wondering if theres a common alternative for the CV which of course would become meaningless.

Also just to mention that there is no proportionality between the means and standard deviations of the data.

Another way to ask this question would be if anyone had suggestions for the inverse, a successful measurement of smoothness? or a way to summarise jerking in the data

|improve this question
$\endgroup$
  • $\begingroup$ What do you consider a spike? Just one anomalous reading or can it go for a while? $\endgroup$ – geometrikal Apr 20 '16 at 0:37
  • $\begingroup$ It's just a sudden drop in signal (so a spike in the negative direction) it takes place over just one or two data points, hence why the std of the differences show this up. $\endgroup$ – BranH Apr 20 '16 at 1:03
  • $\begingroup$ How would you compare one signal with one spike of unit amplitude, and one with two spikes with $0.5$? $\endgroup$ – Laurent Duval Apr 20 '16 at 1:10
  • $\begingroup$ It would depend really on where we set the threshold to determine a 'spike' however, at the moment as these relate to specific events we would want a measure which weighted the signal with spikes of higher amplitude as greater $\endgroup$ – BranH Apr 20 '16 at 1:31
  • $\begingroup$ @BranH Bigger spikes, more important than smaller spikes, ok. But proportion? Would you prefer one spike or two-half spikes? It could be important to know $\endgroup$ – Laurent Duval Apr 20 '16 at 21:01
5
$\begingroup$

I would try a median filter.

  1. Let your original signal be $f[n]$.

  2. Median filter $f[n]$ using $N$ pixels, where $N > 2 \times S + 1$, where $S$ is the maximum number of samples in the spike. The resulting signal, lets call it $g[n]$ should have all the spikes removed.

  3. Find the absolute of the difference between the two signals, $h[n] = |f[n] - g[n]|$. This signal represents the spikes.

  4. Count the number of positive transitions in $h[n]$ that are above a threshold. This is the number of spikes.

Example:

Original signal $f[n]$

enter image description here

Median filtered signal $g[n]$

enter image description here

Absolute difference $h[n]$

enter image description here

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you for this reply I like the idea of filtering the data especially to visualize it. I wondered if there was a metric I could give related to this 'spikiness' as well as the number of spikes as this may be hard to compare across different time series as they have slightly different lengths. $\endgroup$ – BranH Apr 20 '16 at 1:07
  • $\begingroup$ @geometrikal On such a spiky residue, do you have hints to determine an appropriate (automatic?) threshold based on its statistics? $\endgroup$ – Laurent Duval Apr 20 '16 at 1:08
  • $\begingroup$ @LaurentDuval No, I just used a hard threshold in data processing scripts I used the above method for. Maybe a rolling threshold based on the average of the previous x samples? It would be good to know something more principled. $\endgroup$ – geometrikal Apr 20 '16 at 2:20
  • $\begingroup$ @geometrical I am currently trying to find potential outliers in regression, I am currently investigating such rules of thumb to detect when the outlier start... $\endgroup$ – Laurent Duval Apr 20 '16 at 2:29
  • $\begingroup$ @LaurentDuval In some mechanical signal processing work I did, we looked at using kurtosis (the fourth order moment) as a measure of spikiness. $\endgroup$ – Peter K.♦ Apr 20 '16 at 20:00

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.