Filtering with scipy sosfiltfilt generates flat edge artifact

Question

I need to filter some array of data before differentiating it, I do this so I don't drown the signal in noise. It's also important that no delay is applied to the signal, that's why I decided to use a forward-backward filter like scipy's sosfiltfilt and implemented the following:

order = 4
freq = 0.00025

sos = sc.signal.butter(order, freq, output='sos', fs = 1)

t = np.linspace(t0, tf, points)
data_filtered = sc.signal.sosfiltfilt(sos, data, padtype=None)

plt.figure(figsize = (10, 5))
plt.plot(data, 'k', label='Raw')
plt.plot(t, data_filtered, 'r', label='Filtered')
plt.legend(loc='best')
plt.show()

Which generates the following data:

Both the original data and the filtered data can be found there.

An artifact appears where the end is flattened, this is a problem since the derivative will artificialy fall to 0 for those values.

I tried playing around with sosfiltfilt parameters, padding doesn't help and higher orders flatten more data.

Why does this happen and how can I fix it? Is there a different solution for my needs?

PS: Don't really know how to provide the data if you want to try it yourself but I'm happy to do so, thanks!

Answer 1

The padding length was too short, tried this:

import numpy as np
import scipy.signal as sp
import matplotlib.pyplot as plt

order = 4
freq = 0.00025
[b,a] = sp.butter(order, freq, fs = 1)
fpad = sp.filtfilt(b, a, imported_data, padlen=10000)

plt.figure()
plt.plot(imported_data, 'k-', label='input')
plt.plot(fpad, 'c-', linewidth=1.5, label='pad')
plt.show()

And it worked.