2
$\begingroup$

We have a position sensor that under some conditions receives some high frequency noise. We can eliminate that very well with a simple mean filtering.

Tracking data sampling and n=30 mean filtering

Unfortunately this causes too much lag when there is not any noise (at the beginning of the graph at s=90 to 95). Thus we have been experimenting with Savitzky-Golay filtering:

Savitzky-Golay grade 3, f=0.9

However, the results are not that great. The filtering under heavy noise is worse than mean filtering (Or the latency reduction is not that good). What else can we do?

Some more background:

  • The problem is an online problem, so we cannot retrospectively analyze the data and for instance shift the mean by half a period.
  • The 'regular' movement is performed by a human moving an object, the 'noise' is from a high frequency vibration.
Improve this question
$\endgroup$
9
  • $\begingroup$ You mentioned "high frequency noise". Is it possible that you are affected by a well described interference? You can consider capturing a longer sequence of the background noise process (without changing position) and then doing an FFT on the result; if the frequency characteristics of the interference are consistent, then it is feasible you can design an optimized filter based on frequency domain parameters to minimize it. $\endgroup$ – Dan Boschen Sep 12 '18 at 11:15
  • $\begingroup$ @Dan: This is the route we are likely to go down: Use FFT/DCT to detect the noise and switch filters based on that. The noise is not uniform enough for a general filter, I believe. $\endgroup$ – Christopher Oezbek Sep 12 '18 at 20:33
  • $\begingroup$ Using that approach, what intelligence do you use to distinguish between noise and signal of interest? $\endgroup$ – Dan Boschen Sep 12 '18 at 20:47
  • $\begingroup$ Our signal (human movement) is very low frequency, the noise (vibrations) has a high but unknown frequency. That's why mean filtering works great, but the delay in signal cannot be accepted. $\endgroup$ – Christopher Oezbek Sep 12 '18 at 20:50
  • 1
    $\begingroup$ Yes filter delay is directly proportional to the roll-off factor of the frequency specifically; so best strategy with a fixed filter is as low as a cutoff as possible with the smallest roll-off you can get away with to suppress the higher frequency signal which would provide the minimum delay possible (and better than a simple moving average which is a very poor low pass filter) $\endgroup$ – Dan Boschen Sep 12 '18 at 20:54
2
$\begingroup$

Although this is an old entry and probably the author of the question has already solved the issue, I leave here my solution, just in case, it could help someone else.

As explained in the question, you need a kind of real-time "smoother" for the trajectory. You are going to have trade-offs: speed, latency, energy efficiency, etc.

I propose to use Alpha-Beta filters, Moving Average Filters or some kind of Adaptative Average filter. They should be faster, easy to implement and with less latency than Kalman filter or similar. I have made a couple of tests, trying to reproduce the type of data used in the question, and testing those filters with quick(and nasty) own implementations. The results are shown below in the next figures:

enter image description here

enter image description here

Cheers.

P.D. Probably, the vibration noise commented in the original question is not absolutely uncorrelated. The data I have generated is using random noise.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The noise was uncorrelated in our case and yes we did use an adaptive mean filtering based on FFT to detect high frequency noise components. $\endgroup$ – Christopher Oezbek Dec 1 '20 at 19:49
  • $\begingroup$ Thanks for sharing your solution @ChristopherOezbek. I already imagined that you found something because this was very old. :) $\endgroup$ – Arritmic Dec 1 '20 at 19:52
1
$\begingroup$

There are two possible methods that you can try.

  • Polynomial approximation (Spline fitting)
  • Total variation denoising

In polynomial approximation, you can try to fit polynomial approximation of the entire data vector. In total variation denoising, you can regulate the amount of sparsity seen in the first-order difference term using the regularization parameter $\lambda$. In other words, under heavy noise, the first order difference of the input data is sparse.

Improve this answer
$\endgroup$
11
  • $\begingroup$ @Matrox: These are both offline approaches once all the data is collected, right? Our particular problem is realtime/online in that we must calculate the next 'denoised' output directly after receiving the next input data point. Global optimizations are not available. $\endgroup$ – Christopher Oezbek Sep 12 '18 at 11:02
  • $\begingroup$ You can use a sliding window and do batch-processing. In the first frame, wait until you get N samples, process the data, then when you get N+1 th sample, shift the window by 1. $\endgroup$ – Maxtron Sep 12 '18 at 14:39
  • $\begingroup$ Sliding window causes a delay of commonly N / 2. We don't want that. $\endgroup$ – Christopher Oezbek Sep 12 '18 at 20:34
  • $\begingroup$ Have you tried using fast implementations of total variation denosing (TVD). AFAIK, it takes roughly 3-4 milliseconds to run TVD on an input vector of length $N < 1000$. I generally use MATLAB. $\endgroup$ – Maxtron Sep 12 '18 at 20:52
  • $\begingroup$ But it is an online / real-time problem: I receive a new input data point (let's say at x = 52.5 mm), I have to give an answer immediately (not after collecting 999 other data points) to the actual x = 52.9 mm that I should pass to the user/application. This is not a smoothing problem, it is a filtering one. $\endgroup$ – Christopher Oezbek Sep 12 '18 at 20:56
0
$\begingroup$

Saying that your signal is «really low frequency» while you need to have low delay/smearing in order to capture it seems like a contradiction.

If you can describe what (aspects of) the inputsignal that you need to preserve are, and what the noise is like, perhaps a better solution presents itself.

The easiest solution will be some kind of lowpass filter, and they come with a set of trade-offs.

-k

Improve this answer
$\endgroup$
3
  • $\begingroup$ I don't think that is really a contradiction: think about a human moving his hand up and down at a frequency of once every 4 seconds (low frequency). If I want to trace this in real-time I wouldn't want to use a filter that has such a long delay. $\endgroup$ – Christopher Oezbek Dec 1 '20 at 19:46
  • $\begingroup$ Say that your signal really is «DC». How would delay affect it? $\endgroup$ – Knut Inge Dec 1 '20 at 20:18
  • $\begingroup$ EVen though it is batch processing, perhaps this nature article on eye movement filtering could have some useful clues? nature.com/articles/s41598-017-17983-x $\endgroup$ – Knut Inge Dec 2 '20 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.