I'm working with accelerometer data that is sampled at a non-uniform rate. There are major gaps in the data. Below is a scatter plot of the data

enter image description here

I can also give a sense of the frquencies at which the data is sampled. Below is a plot of the frequency distribution. Frequency Histogram

As can be seen, the majority of the data comes in at 128Hz, with some at 100Hz, and then a range of other values. Below I show a histogram that is created by collecting the time difference between samples. The x-axis shows the different time steps, and the y-axis shows the incident number.

enter image description here

What I want to do is to decimate this data down to 1Hz. What is best practice for this? My understanding is that I would:

1) Interpolate up to say 128Hz first using cubic splines. 2) Apply a Hemming or Butterworth low pass filter. 3) Downsample by keeping every 128th point.

Issues: When I interpolate using cubicsplines, I get enormous values during the stretches in which there is a major gap in the data. I could potentially mask for these gaps.

I'm working in python and have been looking at the scipy library to handle this. I know Pandas has .resample().interpolate(), but it seems too memory intensive and slow. The data takes up about 40gigs of memory. Any insight or thoughts would be super appreciated. Thanks friends!!

  • $\begingroup$ Can you talk a little bit more about what is depicted in this diagram and how was it created? $\endgroup$ – A_A Jan 29 '20 at 21:13
  • $\begingroup$ Yes. The data shown is a histogram of the number of samples collected per second. The x-axis is the number of samples per second, and the y-axis shows the incident number for the different frequencies. All i did was group by second and count the number of rows I had for each second grouping. $\endgroup$ – neezi Jan 30 '20 at 15:51
  • $\begingroup$ Do you have time stamps for each sample? $\endgroup$ – A_A Jan 30 '20 at 15:54
  • $\begingroup$ Yes I have timestamps for each sample $\endgroup$ – neezi Jan 30 '20 at 19:37
  • $\begingroup$ OK, then, could you please do a histogram of the sample-to-sample intervals? This will give us the distribution of the sampling period. You can then decide for a better $Fs$. I would suggest that you treat "long gaps" as separate signals (possibly refering to the same "session", but still separate time series). $\endgroup$ – A_A Jan 30 '20 at 19:44

I probably wouldn't bother interpolating at all. You want to downsample by factor 128 anyway, so from that point of view, your raw data is already a good approximation of being uniformly sampled at 128Hz. Just go with low pass filtering and downsampling and you should be good. Try it on a small fraction of the data, to get an idea of the error you introduce by this approximation, my gut feeling says, it will be acceptable, i.e. small compared to the accelerometer error. If actual time span does matter, calculate the average sampling rate and use this value instead of 128Hz.

For handling large data sets with limited RAM, I would opt for generators in python (yield instead of return) This way, there is no need to keep all the data in memory, you just "stream" the data from file to file in (overlapping and windowed) chunks of appropriate size.

  • $\begingroup$ Thank you. May I ask what to do with large gaps of NaN's in my data? I'm fine with interpolating away small gaps, but my data contains several larger chunks (in time space). Do I need to filter each chunk individually? Can a simply scipy lfilter handle NaNs? Thanks :) $\endgroup$ – neezi Jan 30 '20 at 15:53
  • $\begingroup$ Just set the NaNs to Zero or the last valid value, your call. $\endgroup$ – Max Jan 30 '20 at 16:12
  • $\begingroup$ Will that not create artifacts in my data? What I imagine is as follows: 1) Create a mask for these gaps 2) Set the NaNs to zero 3) Filter 4) Apply the mask to reset those values back into NaNs Is that what you imagine? $\endgroup$ – neezi Jan 30 '20 at 19:42
  • $\begingroup$ Yes, something like that. $\endgroup$ – Max Jan 31 '20 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.