4
$\begingroup$

An accelerometer is used on a high vibration environment (a bicycle) to measure gravity. The signal clips, so data higher than a clipping value is clamped to that value. So, simply filtering the signal gives the wrong median mean value when the vibration is high.

I want the mean value over a time much longer than the vibration frequency, so a few second filter is fine. I do not care about the vibration waveform itself. Just want the average value. How to remove the error caused by the clipping?

Bad idea one:
I had suggested that the signal's noise distribution be modeled, the amount of clipping predicted based on measured vibration amplitude, and that used to correct the data. Maybe the distribution is Rayliegh, maybe Gaussian, maybe... So maybe that first idea is not a good one.

Possibly brilliant idea two:
But how about just measuring the distribution on the spot? Make a histogram with several bins, accumulating a count in a bin each time the sampled data is in a bin's range.

Then, fit a curve over the top 3 or 5 highest bins, and the position of the peak is the signal median the modal mean, where the signal mean value would be, if it were not clipped.

Is this crazy? Good? Done before and has a name?

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you post some data so we can judge the severity of the problem? $\endgroup$
    – Emre
    Commented May 23, 2012 at 5:59
  • 3
    $\begingroup$ Please clarify what kind of processing you do on your data. You mention median - but median is actually robust to clipping (a gaussian with mean $\mu$ has median $\mu$, even if clipped to range $[a, b]$ as long as $a \lt \mu \lt b$ - the mean is not robust but the median is). You mention filtering. What kind of filter do you apply to your data? $\endgroup$
    – pichenettes
    Commented May 23, 2012 at 9:33
  • $\begingroup$ The question is sparked by another person's question on stack exchange. I was piqued by how to solve the problem. Is it polite to reference their post? Ya, ok: stackoverflow.com/questions/10703932/… $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 13:31
  • $\begingroup$ @pichenettes, as for filtering, there is a filter applied in the hardware. I have no idea how that responds to overload, but likely there is an effect for some time after clipping. So data points near the clip max ought to be untrusted. $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 13:52

3 Answers 3

Reset to default
5
$\begingroup$

Is this crazy? Good? Done before and has a name?

I wouldn't say that it is "Good" because clipping is not noise that can be modeled as an additive component with some distribution.

fit a curve over the top 3 or 5 highest bins, and the position of the peak is the signal median

The position of that peak would be the modal mean, not the median

The effect of clipping on the signal's distribution would be to a) truncate the real distribution (whatever that is) at the maximum (which depends on the sensor & measurement (or "environment" as you put it) and b) distort the distribution towards this maximum depending on how much clipping was present in the signal.

b) means that whatever you do, your median / mean will be biased by the amount of clipping.

Ideally, you would have to change the sensor to one that can respond to higher accelerations.

If you absolutely must use the current sensor and deal with the clipping then prior to any further processing of its output you could try removing the clipped values (with a simple thresholding operation) and fitting a curve to the data. If you were to fit an spline for example, then the curve would depart at the beginning of the clipped part with the slope of the usable signal prior the clipping and arrive at the end of the clipped part with a slope of the usable signal after the clipping. In this way, all clipped parts will be converted to some "soft" peak (because of the splines "smoothing" effect).

Of course, this is a compromise. It could well be that the clipping is so much that very little "usable" signal remains on either side to fit a "good" spline. Furthermore, depending on your sampling frequency ($Fs$), the signal might be full of impulse-like data which would lead to "very tall" splines as the algorithm would try to fit the sharp slopes. More information would be needed to take a decision on that but in general a higher $Fs$ would help.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I disagree / agree (smile). Of course the noise can be modeled. How well, that is indeed an issue, and may make the first, rejected, plan unworkable. As for the clipping, it would truncate the 'tail' of the distribution, actually piling up all of the tail area at the point of the clip (values above the clip level would all be under-reported as 'max' instead of max+overload). The result is distortion -away- from the max. $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 13:43
  • $\begingroup$ Thanks a lot for setting me straight on mean,median & mode. $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 14:13
  • $\begingroup$ "he position of that peak would be the modal mean, not the median" Exactly! So, my histogram method has found an estimate of the mean of the original (before clipping) data! Yes? $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 14:41
  • $\begingroup$ I am afraid not. When clipping, the sensor has run out of dynamic range and does not return meaningful values. Anything inside the clipping region is like it was never recorded. It has been lost. In fact, i would be suspicious of values near the clipping points (pos, neg) depending on the linearity of the sensor. Think about this:What is the mean of a heavily distorted g signal so that it looks like a square wave with an almost equal number of clipped pos, neg values?. That's how detrimental clipping is. $\endgroup$
    – A_A
    Commented May 24, 2012 at 8:57
  • $\begingroup$ Re: your first comment: I agree about truncating "the tail" but it's actually, "the tails". The clipping is caused by a sensor with an S-Curve response like en.wikipedia.org/wiki/Sigmoid_function . Anything meaningful is in the linear part of that curve. The sensor begins to saturate both towards high pos values as well as neg (hopefully symmetrical). Therefore, it truncates the "original" distribution at those maxima. Perhaps your sensor returns only pos values (?) in which case, the truncation is still there but shifted "to the right" all the way into the pos part of the axis. $\endgroup$
    – A_A
    Commented May 24, 2012 at 9:03
1
$\begingroup$

Just to clarify (or perhaps muddy) the issue, I can see that you might have this problem if you were attempting something along the lines of using the accelerometer to calculate speed/distance over a relatively long period ("relatively long" here being seconds to hours, depending).

You want the average (and I'd have to sit and think about it a couple of hours to say which average) acceleration over time to determine speed and distance, but short clipped transients can disrupt that average.

So the problem is interpolating the missing data. This involves first recognizing that the data is missing (possibly simple in this case since the value returned is at/near the maximum for the sensor) and then somehow figuring out what's missing.

You may be aided to an extent by the fact that, with an accelerometer, a positive transient is very likely to be closely followed by a corresponding negative transient, though the negative will probably be more spread out and less "spikey". (So it occurs to me that one solution may be to simply ignore both the positive and accompanying negative transients, on the assumption that they cancel out.) At the very least, analyzing the negative transient may give you insight into the magnitude of the positive transient -- the area-under-the-curve thing, etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The acceleration of the bicycle (dv/dt) is probably too subtle to see. For now, they are trying to get a good value for gravity. re interpolating, that would be nice, if I wanted the shape of the (clipped) vibrations, say, if it were an audio recording that was clipped. But for this, I just want the average, filtered over a few seconds. The vibration is just noise. $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 14:34
  • $\begingroup$ Then, as I suggested, skipping the positive spike and it's associated negative spike may be the best approach. $\endgroup$
    – Daniel R Hicks
    Commented May 23, 2012 at 16:02
1
$\begingroup$

If clipping occurred for any finite span of time and the vibration peak is unbounded, then the actual mean value could be anything (large). However the peak might actually be bounded, under the assumption that you were not killed by the peak acceleration, nor the bike destroyed. That assumption about the maxima of the peaks would limit the possible range of the mean.

If data is missing, then one possibility is to model the data or data distribution, then do a best fit of the model parameters to the valid portion of the data, and take the mean of the model. For instance you could try fitting the the data to suspected statistical distributions (a clipped Gaussian, etc.) with a known mean. However, whether this works depends on whether the model describes the actual accelerations before clipping.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Isn't the histogram actually measuring the distribution?? So it is model independent, up to the point where the actual distribution is bi-modal. Which could happen, I suppose, for very weird vibrations. $\endgroup$
    – Bobbi Bennett
    Commented May 23, 2012 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.