# minimizing the mean squared error?

I'm reading this book "Applied Optimal Estimation" by A. Gelb. In the example 1.0-1, there are two measurements $z_{1}$ and $z_{2}$ with some noise for measuring $x$ as following $$z_{1} = x + v_{1} \\ z_{2} = x + v_{2}$$ Now to design estimator for $x$, we need the estimated and the actual value of $x$, so we define the estimating error $\tilde{x}$ to be $$\tilde{x} = \hat{x} - x$$ where $\hat{x} = k_{1}z_{1} + k_{2}z_{2}$. My question is that why $\hat{x}$ is the summation of the two measurements? Shouldn't it be one of the measurements or at least the summation divided by 2 so that we can make the subtraction between the ideal $x$ and observed $\hat{x}$ measurements.

• @DilipSarwate, my point here is that $\hat{x}$ is $x$ doubled. If we assume there is no noise in both measurements $z_{1} \ , z_{2}$, $\tilde{x} = x$ instead of $\tilde{x} = 0$. Taking the average of the two estimates make sense to me not the doubled value. – CroCo Jun 1 '14 at 19:12
• @CroCo: I think what you fail to see is that $\hat{x} = k_{1}z_{1} + k_{2}z_{2}$ can be simply the average of the two measurements if one chooses $k_1=k_2=1/2$. – Matt L. Jun 1 '14 at 20:15
• @MattL., what about choosing $k_{1}$ or $k_{2}$ other than half? I'm still not convinced. – CroCo Jun 1 '14 at 22:37
• @DilipSarwate, I'm still confused with your answers. Based on what you're saying, there is no rule of determining $\hat{x}$ or I can do whatever calculations with these measurements then assign them to $\hat{x}$. It is not a matter of what I want, it is what is the correct way. – CroCo Jun 1 '14 at 22:38
• @CroCo: He added the weighted measurements together, where the weights are the same if the variances are the same. If the variances are not the same, greater importance are given to the measures with smaller error variance, i.e., the ones which are likely to be more correct. Note that $k_1+k_2=1$ or in your case with three measurements $k_1+k_2+k_3=1$, and, hence, your first estimate is at least valid (and optimal given that the measurements have the same error variance), while for the second you have $k_1+k_2+k_3=3$. – Oscar Jun 3 '14 at 18:47

Given measurements \begin{align} Z_1 &= x + N_1\\ Z_2 &= x + N_2 \end{align} where $N_1$ and $N_2$ are independent zero-mean Gaussian random variables with variances $\sigma_1^2$ and $\sigma_2^2$ respectively, it can be shown that the minimum-mean-square-error (MMSE) estimate of $x$ in terms of $Z_1$ and $Z_2$ is a linear function of $Z_1$ and $Z_2$. Note that this linearity is a consequence of the Gaussian assumption; for other kinds of distributions that might be assumed for the noise variables $N_1$ and $N_2$, the MMSE estimate is usually not a linear function.
So, let us consider the estimate $\hat{X} = k_0 + k_1Z_1 + k_2Z_2$ for the unknown parameter $x$. The mean-square error is thus \begin{align} E[(\hat{X}-x)^2] &= E[(k_0 + k_1Z_1 + k_2Z_2 - x)^2]\\ &= E[(k_0 + (k_1+k_2-1)x + k_1N_1+k_2N_2)^2]\\ &= \operatorname{var}(k_1N_1+k_2N_2) + (k_0 + (k_1+k_2-1)x)^2 &{\scriptstyle{\text{because}~E[(Y+a)^2=\operatorname{var}(Y)+(\mu_Y+a)^2}}\\ &\geq \operatorname{var}(k_1N_1+k_2N_2) \end{align} where equality holds in that last inequality if we choose $k_0=0$ and $k_1+k_2=1$. With a slight change of notation to incorporate these constraints into our calculations, let us write \begin{align} E[(\hat{X}-x)^2] &= \operatorname{var}(\alpha N_1+(1-\alpha)N_2)\\ &= \alpha^2\sigma_1^2 + (1-\alpha)^2 \sigma_2^2 &{\scriptstyle{\text{because}~N_1, N_2 ~\text{are independent random variables}}}\\ &= \alpha^2(\sigma_1^2 + \sigma_2^2) -2\alpha\sigma_2^2 + \sigma_2^2 \end{align} where it is easily shown that the minimum value of that quadratic function of $\alpha$ occurs when we choose $$\alpha = \frac{\sigma_2^2}{\sigma_1^2 + \sigma_2^2} = k_1, \qquad 1-\alpha = \frac{\sigma_1^2}{\sigma_1^2 + \sigma_2^2} = k_2,$$ that is, when the two measurements are afflicted with Gaussian noise of different variances (say with $\sigma_2^2 > \sigma_1^2$), then the measurement with the smaller variance ($N_1$ in this example) is given greater weight $k_1=\frac{\sigma_2^2}{\sigma_1^2 + \sigma_2^2}$ than the measurement with the larger variance ($N_2$ in this case) which has smaller weight $k_2 = \frac{\sigma_1^2}{\sigma_1^2 + \sigma_2^2}$.
When $\sigma_2^2 = \sigma_1^2$, the measurements are given equal weight $\frac 12$ which is, according to the OP's insistence, the weight that must be assigned in all circumstances so as to minimize the mean-square error because that is the only weighting that makes sense to the OP.
• Judging from a comment in reaction to my comment, the OP seems to start considering weights other than $1/2$, but he is "still not convinced". Let's hope that this answer will bring some light to the darkness. – Matt L. Jun 2 '14 at 8:43