# filter long data streams with python

I am generating a time-stream data set that needs to be filtered. The data-set is pretty large (i.e. too large to be filtered in one go). I'm trying to split the filtering operation up into chunks but I obviously get a spike at each chunk start. Is there a correct way to split a large data set up into smaller ones and have a continuous result - i.e (pseudcode).

timestream # variable with say 1e9 data points
b,a = sig.butter(1, 0.16, 'lowpass')


Split the timestream up in to say 1000 "chunks"

f_t = []
for c in chunks:
t_data = sig.lfilter(b,a,c)
f_t.append(t_data)

• Use numpy.memmap: "a memory-map to an array stored in a binary file on disk" Oct 18 '15 at 9:11

The key is to use the "state" argument of the filtering commands. The idea is that filter commands such as lfilter can accept a vector specifying the inital state of the filter, and can also output their state at the end of the filter computation. In pseudocode:
let state = vector of zeros