10
$\begingroup$

I'm writing a program that averages the user's weight across different days. I'm planning to use a 5-point moving-average (current day, two before and two after). Sometimes, a data point is missing for 1-2 days. How are these cases usually handled?

(if there's a better low-pass filter I could use, I'd love suggestions)

$\endgroup$
  • $\begingroup$ first thing that comes to mind is to interpolate the points before using the moving-average filter $\endgroup$ – someguy Jun 24 '12 at 14:06
  • 3
    $\begingroup$ Really more of a statistical question than a signal processing question, at least in the absence of more context. But you can simply skip recomputing the average, use the current average as the replacement value, or wait for subsequent measurements and attempt interpolation, linear or otherwise. $\endgroup$ – Daniel R Hicks Jun 24 '12 at 14:28
  • $\begingroup$ As the others noted, this will typically be an application-specific decision based upon your consideration of how you want the filtered output to behave. Most signal processing theory is based upon uniformly-spaced samples, so you're not going to get something that can objectively be called the "right answer." $\endgroup$ – Jason R Jun 24 '12 at 15:20
  • $\begingroup$ @JasonR I filter in order to give a more reasonable estimate of the user's weight at that point. The data is uniformly sampled (sampling frequency = 1/day), except some data points are missing. $\endgroup$ – Anna Jun 24 '12 at 15:31
  • $\begingroup$ @Anna: Right, I understand why you're filtering the data. However, your data is not uniformly sampled since you have missing data points. Therefore, as I noted, you're not likely to find a satisfying theoretical answer to your problem. An ad-hoc solution that you deem to "make sense" for your particular application is probably going to be the answer. $\endgroup$ – Jason R Jun 25 '12 at 1:32
4
$\begingroup$

As a general impression, regression would work better in automatically fitting the missing points rather than a moving average filter you have chosen.

If you use an AR (auto regressive filter) or ARMA filter - you can have a predicted value of a sample output based on past inputs.

$$ \hat X[i] = \sum { \omega_{k}*x[i-1-k]} + \eta $$

Where $\hat X[i]$ is the predicted value.

Specifically in your case, say you know the weight of the person has a specific range $X_{max}, X_{min}$. Now if you don't have $x[i-1]$ value - apply two different substitutions - one with Min and one with Max and based on the available model you will have two extreme case results for $\hat X[i]$ and you can choose something between them.

There are various other alternatives - you can keep

$$\hat X[i] = X[i-1]$$ or $$\hat X[i] = \text {Long term sample average of X }$$

Essentially it is a game of prediction of that said value and continue using it as a signal. Of course, prediction won't be same as an original sample but that't the price you pay for not having data.

$\endgroup$
  • 2
    $\begingroup$ Why do you say that regression would work better in fitting? Thanks $\endgroup$ – Spacey Jun 25 '12 at 3:22
3
$\begingroup$

A simple and general method for filling in missing data, if you have runs of complete data, is to use
Linear regression. Say you have 1000 runs of 5 in a row with none missing.
Set up the 1000 x 1 vector y and 1000 x 4 matrix X:

y       X
wt[0]   wt[-2] wt[-1] wt[1] wt[2]
---------------------------------
68      67     70     70    68
...

Regression will give you 4 numbers a b c d that give a best match

wt[0] ~= a * wt[-2]  + b * wt[-1]  + c * wt[1]  + d * wt[2]

for your 1000 rows of data — different data, different a b c d.
Then you use these a b c d to estimate (predict, interpolate) missing wt[0].
(For human weights, I'd expect a b c d to be all around 1/4.)

In python, see numpy.linalg.lstsq .

(There are zillions of books and papers on regression, at all levels. For the connection with interpolation, though, I don't know of a good introduction; anyone ?)

$\endgroup$
1
$\begingroup$

If you don't know some of the data, your best bet in not to average over it at all. Guessing it with linear regression and the like may help, but it also may introduce extra complexity and unintended bias to your data. I would say that if you're averaging over these five data points: $[a, b, c, ?, e]$, your answer should be

$$\frac{a+b+c+e}{4}$$

$\endgroup$
1
$\begingroup$

i think the simplest way would be to "predict" the date for the "whole" in the time series using the data that came before. then you can use this timeseries for parameter estimation. (you could then proceed and repredict the missing values using your estimated parameters from the whole (completed) timeseries and repeat this until they converge). you should derive the confidence bounds from the number of real datapoints you have, though, and not from the length of the completed dataseries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.