I have several EEG channels that need filtering, and one channel may reach a recording length of 12 hours or potentially even more. This results in billions of data points stored which, when loaded, is represented as a NumPy array (by default as 16 bit long integer). In order not to load all data into memory at once, I make use of Python's
mmap. This solution works until I call
scipy.sosfiltfilt(). As I observed, this method creates copies of the entire signal which gets eventually filtered and is returned. The problem arises with really large arrays when making copies becomes simply unfeasible in terms of memory usage.
I have done some research, however, I have not come across a solution. Therefore, my question is whether there is any solution to filter such lengthy signals efficiently. My goal is to achieve this by for example, using an in place updating approach instead of creating copies without having to reimplement the method. Or does
mmap in any way? Or should I just create smaller chunks of the data? Or is there any other feasible way to do this?