2
$\begingroup$

I'm working with accelerometer data that is sampled at a non-uniform rate. There are major gaps in the data. Below is a scatter plot of the data

enter image description here

I can also give a sense of the frquencies at which the data is sampled. Below is a plot of the frequency distribution. Frequency Histogram

As can be seen, the majority of the data comes in at 128Hz, with some at 100Hz, and then a range of other values. Below I show a histogram that is created by collecting the time difference between samples. The x-axis shows the different time steps, and the y-axis shows the incident number.

enter image description here

What I want to do is to decimate this data down to 1Hz. What is best practice for this? My understanding is that I would:

1) Interpolate up to say 128Hz first using cubic splines. 2) Apply a Hemming or Butterworth low pass filter. 3) Downsample by keeping every 128th point.

Issues: When I interpolate using cubicsplines, I get enormous values during the stretches in which there is a major gap in the data. I could potentially mask for these gaps.

I'm working in python and have been looking at the scipy library to handle this. I know Pandas has .resample().interpolate(), but it seems too memory intensive and slow. The data takes up about 40gigs of memory. Any insight or thoughts would be super appreciated. Thanks friends!!

Improve this question
$\endgroup$
10
  • $\begingroup$ Can you talk a little bit more about what is depicted in this diagram and how was it created? $\endgroup$
    – A_A
    Commented Jan 29, 2020 at 21:13
  • $\begingroup$ Yes. The data shown is a histogram of the number of samples collected per second. The x-axis is the number of samples per second, and the y-axis shows the incident number for the different frequencies. All i did was group by second and count the number of rows I had for each second grouping. $\endgroup$
    – neezi
    Commented Jan 30, 2020 at 15:51
  • $\begingroup$ Do you have time stamps for each sample? $\endgroup$
    – A_A
    Commented Jan 30, 2020 at 15:54
  • $\begingroup$ Yes I have timestamps for each sample $\endgroup$
    – neezi
    Commented Jan 30, 2020 at 19:37
  • $\begingroup$ OK, then, could you please do a histogram of the sample-to-sample intervals? This will give us the distribution of the sampling period. You can then decide for a better $Fs$. I would suggest that you treat "long gaps" as separate signals (possibly refering to the same "session", but still separate time series). $\endgroup$
    – A_A
    Commented Jan 30, 2020 at 19:44

2 Answers 2

Reset to default
2
$\begingroup$

I probably wouldn't bother interpolating at all. You want to downsample by factor 128 anyway, so from that point of view, your raw data is already a good approximation of being uniformly sampled at 128Hz. Just go with low pass filtering and downsampling and you should be good. Try it on a small fraction of the data, to get an idea of the error you introduce by this approximation, my gut feeling says, it will be acceptable, i.e. small compared to the accelerometer error. If actual time span does matter, calculate the average sampling rate and use this value instead of 128Hz.

For handling large data sets with limited RAM, I would opt for generators in python (yield instead of return) This way, there is no need to keep all the data in memory, you just "stream" the data from file to file in (overlapping and windowed) chunks of appropriate size.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. May I ask what to do with large gaps of NaN's in my data? I'm fine with interpolating away small gaps, but my data contains several larger chunks (in time space). Do I need to filter each chunk individually? Can a simply scipy lfilter handle NaNs? Thanks :) $\endgroup$
    – neezi
    Commented Jan 30, 2020 at 15:53
  • $\begingroup$ Just set the NaNs to Zero or the last valid value, your call. $\endgroup$
    – Max
    Commented Jan 30, 2020 at 16:12
  • $\begingroup$ Will that not create artifacts in my data? What I imagine is as follows: 1) Create a mask for these gaps 2) Set the NaNs to zero 3) Filter 4) Apply the mask to reset those values back into NaNs Is that what you imagine? $\endgroup$
    – neezi
    Commented Jan 30, 2020 at 19:42
  • $\begingroup$ Yes, something like that. $\endgroup$
    – Max
    Commented Jan 31, 2020 at 6:41
1
$\begingroup$

Generally resampling data that is non-uniformly sampled and/or contains non-valid samples is an interesting task.

If the downsampling ratio is large compared to the maximum (or average) sampling interval, you can possibly use a simple method. Take some average of a neighbourhood.

For a filtering approach, designing a continuous domain filter with some cutoff frequency that you sample at the offsets of input sample times vs desired output sample, normalize with number of input samples might work.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.