I am generating a time-stream data set that needs to be filtered. The data-set is pretty large (i.e. too large to be filtered in one go). I'm trying to split the filtering operation up into chunks but I obviously get a spike at each chunk start. Is there a correct way to split a large data set up into smaller ones and have a continuous result - i.e (pseudcode).

timestream # variable with say 1e9 data points
b,a = sig.butter(1, 0.16, 'lowpass')

Split the timestream up in to say 1000 "chunks"

f_t = []
for c in chunks:
   t_data = sig.lfilter(b,a,c)
  • $\begingroup$ Use numpy.memmap: "a memory-map to an array stored in a binary file on disk" $\endgroup$
    – denis
    Oct 18 '15 at 9:11

The key is to use the "state" argument of the filtering commands. The idea is that filter commands such as lfilter can accept a vector specifying the inital state of the filter, and can also output their state at the end of the filter computation. In pseudocode:

let state = vector of zeros
let filtered_data = 0
for c in chunks
    y, state_new = filter(b, a, c, state)
    append y to filtered_data
    let state = state_new

See the docs for more details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.