Detect Gradual Increase Before a Decrease in Noisy Data

I have data that'ss not too noisy and I am trying to detect a pattern where it gradually increases then decreases in a short period of time (20 ticks? it should be roughly similar per session but can vary slightly between different sessions). This pattern usually happens in pairs but not always.

So looking at the top plot of the data over time, the pattern I am after is at time 100 and 300. you can see the data is roughly stable, then gradually increases and decreases and stabilizes, then after a while again increases, decreases and stabilizes.

As you can see there are also instances where it suddenly changes (550 & 800), or it increases then decreases over a longer period of time (700-800) but I need to detect my short pattern only.

What I have done sofar:

I have taken the variance of the last 10 values which is the middle plot. It gives me a relatively good idea when the data starts to deviate but, variance ignores the sign so I cannot tell if the data is increasing or decreasing and it doesn't differentiate between sudden and gradual changes.

So I decided to take the squared difference between each value and the previous value and keep the sign. I took the square to exaggerate the difference to make it more distinguishable from smaller variations and kept the sign to help identify when it increases then decreases.

I can't say if diff[i] > diff[i-1] for 5 ticks, and then diff[i] < diff[i-1] for 5 ticks because diff is noisy & changes its sign to positive midway at 130 even though overall it was decreasing.

I feel like there would be a better way than trying to filter the noise and then applying that if statement above. But I'm drawing blanks at the moment.

Whats the best way to programatically detect a short but gradual increase then decrease in noisy numbers?

I've added a csv with the data at this pastebin: csv_pastebin

– Royi
May 4 '21 at 16:52
• @Royi I appreciate you taking the time to answer but it doesn't solve the question of finding gradual rise and fall of the data. May 5 '21 at 15:18
• Have you applied the filter I wrote and then applied your approach? It would work. Share a CSV of data and I will show you.
– Royi
May 5 '21 at 17:02
• @Royi sorry couldnt find my data so had to recreate it somehow. editing the question to add the csv May 7 '21 at 3:10
– Royi
Nov 18 '21 at 12:15

I would suggest downsampling your data instead of filtering, since filters can have lots of unwanted effects on your data - if you have enough samples. To downsample, take the mean of ~10 samples, then the mean of the next 10, etc.

If you don't have enough samples for this, use the downsampled signal only for detection of your incidence, it will most probably have a more useful derivative.

I found this paper very helpful regarding filters - Alainde Cheveigne, Israel Nelken - Filters: When, Why, and How (Not) to Use Them.

• That link was a nice read. It certainly highlighted alot of problems ive seen with using filters. Didn't think downsampling would ever work, downsampling to every 10 points made me lose the sudden edges at 550 & 800, but downsampling to every 3 did the job perfectly Nov 19 '21 at 8:48

What you need to do is find a feature which separate the phenomenon you're after from the noise.

If the jump is high relative to the noise STD it means it has good SNR when you compare the instantaneous derivative to the mean derivative (Caused by noise).

As you figured it out, in the case of great SNR as you shown above it works pretty easily and your approach did the work.

Then you need to apply some pre processing to increase the SNR.
You can apply Low Pass Filter but it means it will blur the exact location of the "Jump" you're after.
A nice trick I uses once was borrowing from Image Processing a class of filters called Edge Preserving Filters.
For instance it is really easy to apply the Bilateral Filter in 1D.
While it will remove noise and improve the SNR on flat regions it won't hurt the "Jumps" you're after.
So after using it even signals with worse SNR then the example above will start look like the case above and make your current method viable.

Here is an example in your given data how the bilateral filter works:

As you can see in red, the smoothed signal preserves the form of the input signal yet smoother. If you had some noise it would have been more prominent.

The code is available at my StackExchange Signal Processing Q51264 GitHub Repository. It includes an implementation of 1D Bilateral Filter.

• I've abandonded this project but coming back to it to answer your request to review. Bilateral filtering was abit too "good" at what it does that it preserved not just the vertical jump at 550 & 800, but also the horizontal "edge" at 130 that was causing me the issue Nov 19 '21 at 8:45
• @Ta946, As long as you found an answer and marked it, it is perfect. I am happy to see the downsampling idea by Etienne Etincelle worked.
– Royi
Nov 19 '21 at 11:47