I'm working with accelerometer data that is sampled at a non-uniform rate. There are major gaps in the data. Below is a scatter plot of the data
I can also give a sense of the frquencies at which the data is sampled. Below is a plot of the frequency distribution.
As can be seen, the majority of the data comes in at 128Hz, with some at 100Hz, and then a range of other values. Below I show a histogram that is created by collecting the time difference between samples. The x-axis shows the different time steps, and the y-axis shows the incident number.
What I want to do is to decimate this data down to 1Hz. What is best practice for this? My understanding is that I would:
1) Interpolate up to say 128Hz first using cubic splines. 2) Apply a Hemming or Butterworth low pass filter. 3) Downsample by keeping every 128th point.
Issues: When I interpolate using cubicsplines, I get enormous values during the stretches in which there is a major gap in the data. I could potentially mask for these gaps.
I'm working in python and have been looking at the scipy library to handle this. I know Pandas has .resample().interpolate(), but it seems too memory intensive and slow. The data takes up about 40gigs of memory. Any insight or thoughts would be super appreciated. Thanks friends!!