# Decimating Non-Uniform large time-series data

I'm working with accelerometer data that is sampled at a non-uniform rate. There are major gaps in the data. Below is a scatter plot of the data

I can also give a sense of the frquencies at which the data is sampled. Below is a plot of the frequency distribution.

As can be seen, the majority of the data comes in at 128Hz, with some at 100Hz, and then a range of other values. Below I show a histogram that is created by collecting the time difference between samples. The x-axis shows the different time steps, and the y-axis shows the incident number.

What I want to do is to decimate this data down to 1Hz. What is best practice for this? My understanding is that I would:

1) Interpolate up to say 128Hz first using cubic splines. 2) Apply a Hemming or Butterworth low pass filter. 3) Downsample by keeping every 128th point.

Issues: When I interpolate using cubicsplines, I get enormous values during the stretches in which there is a major gap in the data. I could potentially mask for these gaps.

I'm working in python and have been looking at the scipy library to handle this. I know Pandas has .resample().interpolate(), but it seems too memory intensive and slow. The data takes up about 40gigs of memory. Any insight or thoughts would be super appreciated. Thanks friends!!

• Can you talk a little bit more about what is depicted in this diagram and how was it created? – A_A Jan 29 at 21:13
• Yes. The data shown is a histogram of the number of samples collected per second. The x-axis is the number of samples per second, and the y-axis shows the incident number for the different frequencies. All i did was group by second and count the number of rows I had for each second grouping. – neezi Jan 30 at 15:51
• Do you have time stamps for each sample? – A_A Jan 30 at 15:54
• Yes I have timestamps for each sample – neezi Jan 30 at 19:37
• OK, then, could you please do a histogram of the sample-to-sample intervals? This will give us the distribution of the sampling period. You can then decide for a better $Fs$. I would suggest that you treat "long gaps" as separate signals (possibly refering to the same "session", but still separate time series). – A_A Jan 30 at 19:44