4
$\begingroup$

I have data that'ss not too noisy and I am trying to detect a pattern where it gradually increases then decreases in a short period of time (20 ticks? it should be roughly similar per session but can vary slightly between different sessions). This pattern usually happens in pairs but not always.

enter image description here

So looking at the top plot of the data over time, the pattern I am after is at time 100 and 300. you can see the data is roughly stable, then gradually increases and decreases and stabilizes, then after a while again increases, decreases and stabilizes.

As you can see there are also instances where it suddenly changes (550 & 800), or it increases then decreases over a longer period of time (700-800) but I need to detect my short pattern only.

What I have done sofar:

I have taken the variance of the last 10 values which is the middle plot. It gives me a relatively good idea when the data starts to deviate but, variance ignores the sign so I cannot tell if the data is increasing or decreasing and it doesn't differentiate between sudden and gradual changes.

So I decided to take the squared difference between each value and the previous value and keep the sign. I took the square to exaggerate the difference to make it more distinguishable from smaller variations and kept the sign to help identify when it increases then decreases.

I can't say if diff[i] > diff[i-1] for 5 ticks, and then diff[i] < diff[i-1] for 5 ticks because diff is noisy & changes its sign to positive midway at 130 even though overall it was decreasing.

I feel like there would be a better way than trying to filter the noise and then applying that if statement above. But I'm drawing blanks at the moment.

Whats the best way to programatically detect a short but gradual increase then decrease in noisy numbers?

I've added a csv with the data at this pastebin: csv_pastebin

$\endgroup$
5
  • $\begingroup$ Could you please mark my answer? $\endgroup$
    – Royi
    May 4 '21 at 16:52
  • $\begingroup$ @Royi I appreciate you taking the time to answer but it doesn't solve the question of finding gradual rise and fall of the data. $\endgroup$
    – Ta946
    May 5 '21 at 15:18
  • $\begingroup$ Have you applied the filter I wrote and then applied your approach? It would work. Share a CSV of data and I will show you. $\endgroup$
    – Royi
    May 5 '21 at 17:02
  • $\begingroup$ @Royi sorry couldnt find my data so had to recreate it somehow. editing the question to add the csv $\endgroup$
    – Ta946
    May 7 '21 at 3:10
  • $\begingroup$ Could you please review my answer? $\endgroup$
    – Royi
    Nov 18 '21 at 12:15
1
$\begingroup$

I would suggest downsampling your data instead of filtering, since filters can have lots of unwanted effects on your data - if you have enough samples. To downsample, take the mean of ~10 samples, then the mean of the next 10, etc.

If you don't have enough samples for this, use the downsampled signal only for detection of your incidence, it will most probably have a more useful derivative.

I found this paper very helpful regarding filters - Alainde Cheveigne, Israel Nelken - Filters: When, Why, and How (Not) to Use Them.

$\endgroup$
1
  • 1
    $\begingroup$ That link was a nice read. It certainly highlighted alot of problems ive seen with using filters. Didn't think downsampling would ever work, downsampling to every 10 points made me lose the sudden edges at 550 & 800, but downsampling to every 3 did the job perfectly $\endgroup$
    – Ta946
    Nov 19 '21 at 8:48
4
$\begingroup$

What you need to do is find a feature which separate the phenomenon you're after from the noise.

If the jump is high relative to the noise STD it means it has good SNR when you compare the instantaneous derivative to the mean derivative (Caused by noise).

As you figured it out, in the case of great SNR as you shown above it works pretty easily and your approach did the work.

What about cases with bad SNR?
Then you need to apply some pre processing to increase the SNR.
You can apply Low Pass Filter but it means it will blur the exact location of the "Jump" you're after.
A nice trick I uses once was borrowing from Image Processing a class of filters called Edge Preserving Filters.
For instance it is really easy to apply the Bilateral Filter in 1D.
While it will remove noise and improve the SNR on flat regions it won't hurt the "Jumps" you're after.
So after using it even signals with worse SNR then the example above will start look like the case above and make your current method viable.

Here is an example in your given data how the bilateral filter works:

enter image description here

As you can see in red, the smoothed signal preserves the form of the input signal yet smoother. If you had some noise it would have been more prominent.

The code is available at my StackExchange Signal Processing Q51264 GitHub Repository. It includes an implementation of 1D Bilateral Filter.

$\endgroup$
2
  • $\begingroup$ I've abandonded this project but coming back to it to answer your request to review. Bilateral filtering was abit too "good" at what it does that it preserved not just the vertical jump at 550 & 800, but also the horizontal "edge" at 130 that was causing me the issue $\endgroup$
    – Ta946
    Nov 19 '21 at 8:45
  • 1
    $\begingroup$ @Ta946, As long as you found an answer and marked it, it is perfect. I am happy to see the downsampling idea by Etienne Etincelle worked. $\endgroup$
    – Royi
    Nov 19 '21 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.