# Any better way to detect Signal with ADC Overflow problem?

I am trying to create an algorithm to detect signal with "ADC overflow" as shown in picture on lower right. It happened because someone had incorrectly down converted an int16 signal into int8 signal. The sampling rate of the signal was 1000Hz, about 2 minutes in duration and has bandwidth up to 300Hz due to low pass filtering. I have few thousands of them and the "ADC overflow" could happen at any place. I need to quickly identify those signal and their overflow locations.

So far I use this technique but it is not robust enough, because I have to keep lowering the threshold=0.8 for higher frequency signal. It's also very hard to differentiate random noise from ADC overflow. Does anyone has better way to do this? Thanks

# Python
# y = int8 signal with ADC overflow as shown on the right
diff = np.abs( np.diff( y.astype(np.int64) ) ) / 255
overflow_loc = (diff >= 0.8)   # <-- 0.8 may not work for high freq signal
overflow_loc += np.roll(overflow_loc, 1)
loc = np.where(overflow_loc == 1)[0]


Ok I tried the Histogram method, but I still can not tell the different. Here is my Python code and output. Please enlighten...

freq = 10
x = np.linspace(0, 1.0 / freq, 256)
y = 1.1 * 127 * np.sin(2 * np.pi * freq * x)
yo = y.astype(np.int8)

fig, axs = plt.subplots(2, 2)

axs[0, 0].plot(x, y)
axs[0, 0].set_yticks([-128, 127])
axs[0, 0].plot(x, [-128] * len(x), '-.')
axs[0, 0].plot(x, [127] * len(x), '--')
axs[0, 0].set_title('ideal signal (np.float64)')

axs[0, 1].hist(y, bins=200)
axs[0, 1].set_title('amplitude histogram (ideal signal)')

axs[1, 0].plot(x, yo)
axs[1, 0].set_yticks([-128, 127])
axs[1, 0].plot(x, [-128] * len(x), '-.')
axs[1, 0].plot(x, [127] * len(x), '--')
axs[1, 0].set_title('overflow signal (np.int8)')

axs[1, 1].hist(yo, bins=200)
axs[1, 1].set_title('amplitude histogram (overflow signal)')

plt.show()


• Have you tried an amplitude histogram? Clipping puts horns at the extremes – Stanley Pawlukiewicz Apr 21 '18 at 1:18
• not yet, but how does it works? please enlighten – Scoodood C Apr 21 '18 at 1:23
• for example, both [10, 266] are interpreted as [10,10] in the final int8 signal – Scoodood C Apr 21 '18 at 1:33
• How does a histogram work? or how does the pdf of clipped Gaussian noise look like? – Stanley Pawlukiewicz Apr 21 '18 at 1:33
• do you have some sort of PSD of your signal of interest? If possible, one without your numeric wraparound and one with? – Marcus Müller Apr 21 '18 at 20:24

How do you transform overflowed signal back to normal? Just by adding or subtracting multiplies of 256. So with every new sample you just determine whether overflow occured - you need to define some threshold and compare it with first derivative of your signal, e.g. x[i+1]-x[i] > 200. If this happened, you know that you need to subtract 256 from x[i+1] to get continuous signal. Opposite case x[i+1]-x[1] < -200 means that you need to add 256. So the pseudocode would be:

base = 0
thresh = 200
for each i in x
d = x[i] - x[i-1]
if d > thresh then base = base - 256
if d < -thresh then base = base + 256
y[i] = x[i] + base
next

• But your image reminds me well known problem of incorrect handling signed/unsigned values. These overflows happen when you save the recording as unsigned ints and then read them as signed (or vice versa) – gabonator Apr 22 '18 at 8:31
• But your image reminds me well known problem of incorrect handling signed/unsigned values. These overflows happen when you save the recording as unsigned ints and then read them as signed (or vice versa) – gabonator Apr 22 '18 at 8:31
• Take a look here, you are just not handling the samples correctly: stackoverflow.com/questions/28521857/… – gabonator Apr 22 '18 at 8:41