4
$\begingroup$

I am not a signal processing expert and my feeble attempts at solving this problem have come up short.

I have a C++ application which is being fed regularly-spaced (in time) 3D position samples. The samples are generally accurate and, if plotted, appear to be quite smooth. However, when the second derivative is computed (to look at accelerations), there are occasional/rare spikes in the data which are unreasonable. (I know the general range of possible accelerations. And by "rare", I mean that over about two ours of data, I generally see less than a dozen spikes which are out of the realm of possibility.) I want to do as little to the data as possible, but smooth it enough to remove the spikes.

My first attempt was to use a Kalman filter in our math library. This removed the spikes, but changed the data far more than what I would really want and seems heavy-handed for what I need.

There seem to be a lot of options out there for 2D data, but I don't have enough signal processing background to understand the trade-offs or necessarily what might translate well to 3D data. Any education or suggestions would be appreciated!

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to DSP.SE! Can you share some example data (real or simulated)? That might help us answer you. The Kalman Filter should do what you want. Have you tried changing the noise variances (process and measurement) ? Sometimes thinking of the Kalman Filter as a tunable box with two knobs: how much the signal should change (process noise) and how well you can measure it (measurement noise) is better than making those parameters equal the real system. Also, what options are there for 2D? Please add that to your question, we might be able to point to extensions that work for 3D data. $\endgroup$
    – Peter K.
    Nov 13, 2015 at 14:32
  • $\begingroup$ Unfortunately, I cannot share any of the data. For the Kalman filter, I have tried tuning it, but what I see is that it either doesn't do enough or does too much. And since the data is "generally" very well behaved, I don't want to modify the data any more than necessary. Looking around, I considered a least-squares fit, but that seems to be a 2D solution. And I am not sure that best fits the problem. Ideally, I think I just want to adjust points around the outliers and leave the rest of the data untouched. $\endgroup$
    – DiB
    Nov 13, 2015 at 15:05
  • $\begingroup$ And for more clarification, I would be happy with some suggestions on algorithms to consider. I am just not sure what is out there that is reasonably applicable to this type of smoothing. $\endgroup$
    – DiB
    Nov 13, 2015 at 15:34

1 Answer 1

1
$\begingroup$

I have always found that a "Median Filter", https://en.wikipedia.org/wiki/Median_filter to be effective on streaming data.
A brief intuitive description is below; but you must take steps to insure you restrict the bandwidth to the real data source you expect. While you might not know the sources/spectrum/timing of noise and spikes (in which case the median filter is actually pretty good) you hopefully know the signals and characteristics of the signals you are looking for. For extensive analysis, still iffy IMHO, along the lines of PDF (probability distribution function) and spectral analysis: there are some IEEE articles; but my analysis of most of the articles were that they weren't really useful in the face of unknown interference. Aside from accommodating only the real signal bandwidth I wouldn't try z-s domain filtering without a really good reason.

------------------------ Median filter abbreviated description Basically you pass the data into a shift register and pick the middle value. Examples always show a sort process; but since you lose only one value and gain only one value during a shift, you can bypass doing full sorts each step Having said that: if the spikes are in the input data then that is where this should be applied; but if the spikes are real "jerks" (i.e. occurring in the second derivative) then that is where it should be applied. Since the derivative process amplifies higher order noises one has to careful. If I was doing what you describe I would impose the median filter on the source data, first differences, and then on second differences (acceleration). And then examine the results; look inside the algorithm and understand what data is tossed! There is an alternate form: you examine a shift register of the data for min's and max's and just toss the extremes.
Both techniques incur some data loss that has to be taken into account. You can interpolate, which is a little expensive, or just accommodate a lower sampling rate; say one in ten samples will always be tossed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.