Smoothing 3D Data for the Second Derivative

Question

I am not a signal processing expert and my feeble attempts at solving this problem have come up short.

I have a C++ application which is being fed regularly-spaced (in time) 3D position samples. The samples are generally accurate and, if plotted, appear to be quite smooth. However, when the second derivative is computed (to look at accelerations), there are occasional/rare spikes in the data which are unreasonable. (I know the general range of possible accelerations. And by "rare", I mean that over about two ours of data, I generally see less than a dozen spikes which are out of the realm of possibility.) I want to do as little to the data as possible, but smooth it enough to remove the spikes.

My first attempt was to use a Kalman filter in our math library. This removed the spikes, but changed the data far more than what I would really want and seems heavy-handed for what I need.

There seem to be a lot of options out there for 2D data, but I don't have enough signal processing background to understand the trade-offs or necessarily what might translate well to 3D data. Any education or suggestions would be appreciated!

Answer 1

I have always found that a "Median Filter", https://en.wikipedia.org/wiki/Median_filter to be effective on streaming data.
A brief intuitive description is below; but you must take steps to insure you restrict the bandwidth to the real data source you expect. While you might not know the sources/spectrum/timing of noise and spikes (in which case the median filter is actually pretty good) you hopefully know the signals and characteristics of the signals you are looking for. For extensive analysis, still iffy IMHO, along the lines of PDF (probability distribution function) and spectral analysis: there are some IEEE articles; but my analysis of most of the articles were that they weren't really useful in the face of unknown interference. Aside from accommodating only the real signal bandwidth I wouldn't try z-s domain filtering without a really good reason.

------------------------ Median filter abbreviated description Basically you pass the data into a shift register and pick the middle value. Examples always show a sort process; but since you lose only one value and gain only one value during a shift, you can bypass doing full sorts each step Having said that: if the spikes are in the input data then that is where this should be applied; but if the spikes are real "jerks" (i.e. occurring in the second derivative) then that is where it should be applied. Since the derivative process amplifies higher order noises one has to careful. If I was doing what you describe I would impose the median filter on the source data, first differences, and then on second differences (acceleration). And then examine the results; look inside the algorithm and understand what data is tossed! There is an alternate form: you examine a shift register of the data for min's and max's and just toss the extremes.
Both techniques incur some data loss that has to be taken into account. You can interpolate, which is a little expensive, or just accommodate a lower sampling rate; say one in ten samples will always be tossed.