# How do you filter a signal, split in batches, using an IIR filter?

I have a signal that is measured by an optical sensor that is connected to an ESP-32. I am trying to apply a band pass filter "in real time" on this microcontroller. The signal is comming in batches of a certain size from a buffer.

What I am trying to do is apply a Butterworth filter on these batches in a sequential manner. I am currently testing things out in Python using scipy. I am using a band pass filter with pass-band [0.35, 10]Hz.

sos = butter(2, [0.35, 10], btype='band', fs=1000, output='sos')
zi = sosfilt_zi(sos)
batch_size = [2048] # normally testing for various batch sizes
for i in batch_size:
stream_filtered = []
# afe1_led1 is a complete list of measurements, to be broken up into batches
for j in range(0, len(afe1_led1)//i):
batch = afe1_led1[j*i:(j+1)*i]
filtered_data, zi = sosfilt(sos, batch, zi=zi)
stream_filtered.extend(filtered_data/1000)
# the division by 1000 here is due to unexpected gain added by the filter


The output of this filtering process is clearly flawed. The output signal is scaled by 3rd order of magnitude (~ 1000) and is also not resembling the expected waveform. The output signal looks more like a 10Hz sinusoidal waveform when using a 4th order Butterworth. When using a 2nd order Butterworth, the output has even higher frequency components, adding detail to the 10Hz oscilation. As seen in the image, the blue signal is the expected waveform (filtered by a Chebyshev filter normally, as 1 big batch). The orange image is the output of the scipy code snippet.

My question is: what is causing this behaviour? What is the part I am missing in relation to using an IIR filter with a batched input, that is causing this aparent instability?

Edit 1:

Following the suggestion given by @Hilmar, I tested the filter using simple sinusoids. It does in fact work. However, when testing with the actual data, it still does not. By @Jdip's suggestion, you can access a segment of the actual data in a numpy.ndarray here: unfiltered data. Each element is float64.

Some additional information about the blue signal (expected). It is the filtered output of a Chebyshev filter using signal.sosfiltfilt(). Note this is different to signal.sosfilt() used in the code provided above. Although I do not believe this makes a big difference.

• Can you share the data somehow?
– Jdip
Commented May 8 at 20:57
• The initialization of your state is wrong, but otherwise your code looks fine. What type is your original data? Is this fixed point? " # the division by 1000 here is due to unexpected gain added by the filter" That should NOT be the case and I don't see that when running it with a sine wave. Try a sine wave first Commented May 8 at 21:36