I am generating a time-stream data set that needs to be filtered. The data-set is pretty large (i.e. too large to be filtered in one go). I'm trying to split the filtering operation up into chunks but I obviously get a spike at each chunk start. Is there a correct way to split a large data set up into smaller ones and have a continuous result - i.e (pseudcode).
timestream # variable with say 1e9 data points b,a = sig.butter(1, 0.16, 'lowpass')
Split the timestream up in to say 1000 "chunks"
f_t =  for c in chunks: t_data = sig.lfilter(b,a,c) f_t.append(t_data)