First, I would like to point out that I am a complete outsider to the field of signal processing, so please bear with me and go easy on the jargon :) I nevertheless decided to post here, because I believe your field has developed the best methods for the kind of problem I have.

I want to do a statistical analysis on a dataset. I have a list of subjects, and a measurement was taken for each subject multiple times over a time period. To do my analysis, I want to first transform my data from longitudinal, time-series data to one single number per subject that somehow describes the "peak" in the data, if there was such a peak. I would like your advice on algorithms that can be used to calculate such a number.

Example data

I am including some example data in a picture. In my layperson terminology, I would say that:

  • example A has a single peak around day 23, but it is not very "large"
  • example B has no real peaks, it is mostly a flat line with a bit of randomness
  • example C has one "large" peak on day 33 and one "smaller" peak on day 50-51

I would like the algorithm to be able to produce a number that maps the "largeness" of the peak to a rational number and goes towards 0 when there is not really a peak. I don't have a perfect definition of my "largeness" measurement, but it should combine

  1. The steepness of the peak start. If this were an ideal mathematical function and not a noisy measurement, I would have taken the function slope
  2. The absolute height of the peak (the first peak in example C is higher than the peak in A because it reaches 20)
  3. The relative height of the peak. It has to be relative to the "typical" height of the measurement for that example (I'm afraid in my picture all the data has roughly the same "typical" height between 0 and 5, but in reality, I have examples where it differs between subjects) - I am not yet sure if I want to relate it to an average over all measurements or to an average over the measurements which are not part of a recognized peak, the second is probably more informative but the first easier to calculate
  4. The duration of the peak - the first peak in C is a very short one, but the second one has a kind of a plateau.

If there is more than one peak per subject, I am fine with taking the highest peak. A stretch goal would be to be able to recognize multiple peaks per subject, but I would have trouble defining a threshold for "real peak" and "noise". Another interesting extension would be to be able to repeat the inverse of this and recognize the largest dip per subject, defined by the same criteria as above but going under the typical height instead of upwards.

I suppose these four criteria won't be doable in a single calculation - that's OK, I can imagine taking a weighted sum of them and being able to play with adjusting my own weights. But if you know of a better way, I am open to it.

To make the task a bit harder (well, it's real-world data), the measurements are not taken at regular intervals, the intervals vary even within a single subject, and the duration of measurement varies considerably between subjects. Also, I expect that there are no underlying patterns in the noise, cyclic or otherwise. On the bright side, I am not doing any real-time calculations and I expect no more than 500 subjects, so I can afford computationally intensive algorithms.

I don't expect a full mathematical explanation of the workings of the algorithm(s) here within an answer. But as I don't have an overview over the field, I would be very grateful for sketching the kind of approach I can take, with names of algorithms and ideally pointers to books or publications that describe these algorithms. Also, if there are other important factors which determine something interesting about a peak, I am interested to hear about them.

  • $\begingroup$ Do the "peaks" occur at random? In any case a pointer/link to the real data would go a long way to prompting people (probably not me) to implement an algorithm. But... it looks to me that you need more resolution for real analysis; i.e. the intervals between points on your graph are to crude. Having said that some form of Wavelet Analysis might be usefull. It's interesting and effective but does require some "on-ramp" time. I could explain a little more but there "isn't enough room in the margin" :) :) Try searching, on this board as well, for that. $\endgroup$ – rrogers Aug 6 at 20:07
  • $\begingroup$ Is this similar to what you want? ncbi.nlm.nih.gov/pmc/articles/PMC4667093 what sort of processing power to you have? ML might be appropriate. $\endgroup$ – rrogers Aug 6 at 20:10
  • $\begingroup$ You could look at en.wikipedia.org/wiki/Median_filter and: en.wikipedia.org/wiki/Nonlinear_filter And there are other things depending upon what you call "noise". $\endgroup$ – rrogers Aug 6 at 20:15
  • $\begingroup$ @rrogers yes, you can assume that the peaks occur at random. And it is impossible to obtain more resolution. If it means that there is uncertainty in what we call a "peak", so be it. $\endgroup$ – Rumi P. Aug 8 at 6:42
  • $\begingroup$ Well for myself I would run a median filter and look at what got tossed. You should post a link to some original data streams; the more the better, although anything above 1000 data points will do. Then some analysis can be done by somebody else. I found this paper that seems pretty thorough in coverage and has sections "12.3 Median Filters " (look at the tossed/residues) and "12.4.1 Impulsive Noise Detection " in: dsp-book.narod.ru/305.pdf There are also statistical tests that can be done to see if the spikes are exceptional or just part of a noise pattern/distribution. $\endgroup$ – rrogers Aug 9 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.