filter long data streams with python

Question

I am generating a time-stream data set that needs to be filtered. The data-set is pretty large (i.e. too large to be filtered in one go). I'm trying to split the filtering operation up into chunks but I obviously get a spike at each chunk start. Is there a correct way to split a large data set up into smaller ones and have a continuous result - i.e (pseudcode).

timestream # variable with say 1e9 data points
b,a = sig.butter(1, 0.16, 'lowpass')

Split the timestream up in to say 1000 "chunks"

f_t = []
for c in chunks:
   t_data = sig.lfilter(b,a,c)
   f_t.append(t_data)

Answer 1

The key is to use the "state" argument of the filtering commands. The idea is that filter commands such as lfilter can accept a vector specifying the inital state of the filter, and can also output their state at the end of the filter computation. In pseudocode:

let state = vector of zeros
let filtered_data = 0
for c in chunks
    y, state_new = filter(b, a, c, state)
    append y to filtered_data
    let state = state_new
end

See the docs for more details.