# Rolling sum of dataframe with custom function gets really slow on large averaging windows. Can I fix with convolve?

I am using the following function to estimate the Gaussian window rolling average of my timeseries. Though it works great from small size averaging windows, it crushes (or gets extremely slow) for larger averaging windows.

def norm_factor_Gauss_window(s, dt):

numer         = np.arange(-3*s, 3*s+dt, dt)
multiplic_fac = np.exp(-(numer)**2/(2*s**2))
norm_factor   = np.sum(multiplic_fac)
window        = len(multiplic_fac)

return window,  multiplic_fac, norm_factor

# Create dataframe for MRE
aa = np.sin(np.linspace(0,2*np.pi,1000000))+0.15*np.random.rand(1000000)
df = pd.DataFrame({'x':aa})

hmany  = 10
dt     = 1      # ['seconds']
s      = hmany*dt  # Define averaging window size ['s']

# Estimate multip factor, normalizatoon factor etc
window, multiplic_fac, norm_factor= norm_factor_Gauss_window(s, dt)

# averaged timeseries
res2 =(1/norm_factor)*df.x.rolling(window, center=True).apply(lambda x: (x * multiplic_fac).sum(), raw=True, engine='numba', engine_kwargs= {'nopython': True,  'parallel': True} , args=None, kwargs=None)

#Plot
plt.plot(df.x[0:2000])
plt.plot(res2[0:2000])


I am aware that people usually speed up moving average operations using convolve(e.g., https://stackoverflow.com/questions/14313510/how-to-calculate-rolling-moving-average-using-python-numpy-scipy)

Would it be possible to use convolve here somehow to fix this issue? Also, are there any other suggestion that would help me speed up the operation for large averaging windows?

• yeah, you should just use convolve instead of implementing a convolution yourself as you seem to be doing in res2=…. Sep 8, 2022 at 13:11
• This looks mostly like a simple linear filter operation with a long-ish impulse response. By far the most effective way to implement this is Overlap Add or Overlap Save algorithms (leveraging the FFT). I would also take a look at the overhead of using DataFrame objects compared to simple arrays. Sep 8, 2022 at 13:29
• @MarcusMüller would you mind adding an answer implementing this to the code. I don't think I really understnad how convolve works to implement it myself. Thank you for your time! Sep 8, 2022 at 15:35
• @Hilmar would scipy.signal work for such a job? Sep 8, 2022 at 15:36
• docs.scipy.org/doc/scipy/reference/generated/… should work, also according to signal;.convolve() will automatically select this if it's better Sep 8, 2022 at 16:25

With the help of @Hilmar an @Marcus I was able to drastically improve the speed of this code using the following:

from scipy import signal
def norm_factor_Gauss_window(s, dt):

numer         = np.arange(-3*s, 3*s+dt, dt)
multiplic_fac = np.exp(-(numer)**2/(2*s**2))
norm_factor   = np.sum(multiplic_fac)
window        = len(multiplic_fac)

return window,  multiplic_fac, norm_factor

# Create dataframe for MRE
aa = np.sin(np.linspace(0,2*np.pi,1000000))+0.15*np.random.rand(1000000)
df = pd.DataFrame({'x':aa})

hmany  = 10
dt     = 1      # ['seconds']
s      = hmany*dt  # Define averaging window size ['s']

# Estimate multip factor, normalizatoon factor etc
window, multiplic_fac, norm_factor= norm_factor_Gauss_window(s, dt)

# averaged timeseries

res2 = (1/norm_factor)*signal.fftconvolve(df.x.values, multiplic_fac[::-1], 'same')

#Plot
plt.plot(df.x[0:2000])
plt.plot(res2[0:2000])

• You should be able to give yourself the check mark now, so please do so!
– Peter K.
Oct 8, 2022 at 19:32