0
$\begingroup$

There are instances when neural networks, after being trained continually on a subset of data, tend to drastically lose their performance. I've attached an image below depicting the same. I am looking for an algorithm to detect this sudden fall in performance. enter image description here I'm quite new to this area and therefore don't have any idea where to start. My initial guess would be to first smooth the plot, so as to reduce the noise.

Edit 1: I highly appreciate the good advice that I have received in the comments and answers. I have added some code that generates live time-series, that'll help better understand my problem. The code is designed to run in Jupyter Notebook cells.

import matplotlib.pyplot as plt
from IPython.display import clear_output
import time
import numpy as np
def test():
    accuracy = []
    for _ in range(1000000000):
        time.sleep(1)
        accuracy.append(np.random.randint(30))
        clear_output(wait=True)
        plt.figure(figsize=(20,5))
        plt.plot(accuracy)
        plt.show()
test()
$\endgroup$
8
  • $\begingroup$ It's not clear to me where the "sudden change" that you're looking to identify appears in the plot. I'm not familiar with the details of your end application. $\endgroup$
    – Jason R
    Commented Jul 12, 2022 at 17:36
  • $\begingroup$ @JasonR Thanks for your comment. I have modified the image. I'd like this question to be domain independent and just get a signal processing perspective: How does one identify a sudden drop in a rising plot? $\endgroup$ Commented Jul 12, 2022 at 17:46
  • $\begingroup$ Would simple thresholding work here? Either absolute (drops below 20) or relative (drops to less than 10% of the average of the last 10 values). $\endgroup$
    – Hilmar
    Commented Jul 12, 2022 at 18:14
  • $\begingroup$ @Hilmar Thank you! I think thresholding would probably work. Could you please elaborate? Is there a package or library that does this? $\endgroup$ Commented Jul 12, 2022 at 18:38
  • 2
    $\begingroup$ ah, so it's python. Yes, calculating the average over ten numbers in python is trivial, pretty sure: it's quite literally sum(values[-10:]) / 10. That's it. That's the code. $\endgroup$ Commented Jul 12, 2022 at 19:30

2 Answers 2

2
$\begingroup$

The change in any signal with respect to another variable is defined by it's derivative with respect to that variable. In order to compute the "derivative" when you have a discrete signal (like in your case), (by asuming that the samples are spaced by a unity) you should use the finite difference in the following way

 s_change = s - s_old
 s_old = s

"s" is the current sample, "s_old" is inizialized to zero the first time then and it is updated to contain the value of the previous sample.

if "s_change" is huge it means that the signal suddenly changed, if its positive it means it's a sudden increased, if it's negative it means that it's a sudden drop.The bigger it's value the bigger the sudden change.

You can average each N samples and also compute the s_change each N averaged samples if you wanna filter out some noise.

I hope this is what you were looking for.

$\endgroup$
1
1
$\begingroup$

The concept of quickest detection might be of use to you. The general idea is to form a test statistic that accumulates over time, and to declare changes when the test statistic exceeds some threshold. This is a good in-depth analysis of the method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.