I have two unbiased estimators of one parameter, $\tau$. The first estimator, $r_1$, is the better estimator with lower variance than the second estimator, $r_2$.

I also have: $ \mathbb{E} \left[ {r}_{1} \right] = 0.8 \tau $ , and $ \mathbb{E} \left[ {r}_{2} \right] = 0.2 \tau $.

I want to use both estimators to achieve lower variance. Is there a systematic way to find a linear combination of $r_1$ and $r_2$ that results in a lower variance and unbiased estimate, without a knowledge of the variance and covariance of the estimators?

  • 3
    $\begingroup$ the linear combination of the two estimators , say $w_1 \times $ the first estimate + $w_2 \times $ the second estimate, will be unbiased as long as $w_1 + w_2 = 1$. The resulting variance of the linear combination depends on the variance of $\hat{\tau}$ and the weights one uses. You can let $w_2 = 1- w_{1}$ and take the derivative of the variance expression and minimize it. This will let you solve for the weights that minimize it.. $\endgroup$
    – mark leeds
    Oct 17 '19 at 5:29
  • $\begingroup$ @markleeds, it is only true if both estimators are in biased to begin with. Regarding taking the derivative, well he wrote given the variances of the estimators aren’t known. $\endgroup$
    – Royi
    Oct 18 '19 at 0:45
  • $\begingroup$ @Royi: 1) He said right at the beginning that they were unbiased. 2) He can't achieve a lower variance if he doesn't know the variance of the $\tau$ estimate to begin with ? In the posting, OP says that he doesn't know the covariance ? There is no covariance. He's using the same estimator in both of his estimates so covariance has nothing to with the problem. But yes, he does need to know the variance of his $\tau$ estimate, namely the variance of $\hat\tau$. If he doesn't know that, he can't find weights that will minimize the var of the new unbiased estimator of what ever he's estimating. $\endgroup$
    – mark leeds
    Oct 18 '19 at 2:49
  • $\begingroup$ @Royi: Here's a similar problem posed in a different way. One has two stocks, A and B, whose return ( over the next year say ) are normally distributed with mean $\mu_1$ and $\mu_2$ and variance $\sigma^2$. Investor has a fixed amount of money that he needs to split amongt these two stocks. He has to spend it all. What weights should be attached to the two stocks so that his portfolio has minimal variance. Investor doesn't care about return but he has to spend all his money. There's a specific answer that can be calculated. It's essentially the same problem as what the OP asked. $\endgroup$
    – mark leeds
    Oct 18 '19 at 2:57
  • $\begingroup$ @Royi: Sometimes I write these things and never see them again ( and don't bother searching ). So, for posterity's sake, the stock problem I described turns into minimize $w_1^2 \times \sigma^2 + (1 - w_1)^2 \times \sigma^2$. The $w_1$ that minimizes that expression which is the variane of his portfolio, ( take derivative and set to zero ) is $w_1 = 1/2$. This means that the investor should put half his dollars in stock A and half his dollars in stock B if all he wants to do is minimize the variance of his portfolio. $\endgroup$
    – mark leeds
    Oct 18 '19 at 3:38

Basically I'd do something like:

$$ \frac{ {\sigma}_{2}^{2} }{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2}} {r}_{1} + \frac{ {\sigma}_{1}^{2} }{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2}} {r}_{2} $$

This is the optimal weighing given knowledge of the Variance only (Well, linear).
Basically it assumes the cross correlation is 0.


There are many ways to derive this.
The easiest would be intuition of knowing that Variance is the Linear Measure of the quality of an estimator. But we're leaving intuition our for this moment.

So easiest formal derivation could be using the ideas @Mark Leeds talked about.

Our model is:

$$ {r}_{3} = {w}_{1} {r}_{1} + {w}_{2} {r}_{2} $$

We want to guarantee the mean of $ {r}_{3} $ is un biased so:

$$ \mathbb{E} \left[ {r}_{3} \right] = \mathbb{E} \left[ {r}_{1} \right] = \mathbb{E} \left[ {r}_{2} \right] = {w}_{1} + \mathbb{E} \left[ {r}_{1} \right] + {w}_{2} + \mathbb{E} \left[ {r}_{2} \right] \Rightarrow {w}_{1} + {w}_{2} = 1 $$

So we'll use $ {w}_{2} = 1 - {w}_{1} $. Now we want to minimize its variance. Since it is not biased we're after minimizing its second moment:

$$\begin{align*} \mathbb{E} \left[ {r}_{3}^{2} \right] & = \mathbb{E} \left[ { \left( {w}_{1} {r}_{1} + \left( 1 - {w}_{1} \right) {r}_{2} \right) }^{2} \right] \\ & = {w}_{1}^{2} \mathbb{E} \left[ {r}_{1}^{2} \right] + { \left( 1 - {w}_{1} \right) }^{2} \mathbb{E} \left[ {r}_{2}^{2} \right] + 2 {w}_{1} { \left( 1 - {w}_{1} \right) } \mathbb{E} \left[ {r}_{1} {r}_{2} \right] \\ & = {w}_{1}^{2} {\sigma}_{1}^{2} + {\left( 1- {w}_{1} \right)}^{2} {\sigma}_{2}^{2} \end{align*}$$

We assumed $ \mathbb{E} \left[ {r}_{1} {r}_{2} \right] = 0 $.

Now, this is a quadratic form so to minimize just derive it and find what vanishes it:

$$ {w}_{1} {\sigma}_{1}^{2} - {\sigma}_{2}^{2} + {w}_{1} {\sigma}_{2}^{2} = 0 \Rightarrow {w}_{1} = \frac{ {\sigma}_{2}^{2} }{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2} }, \; {w}_{2} = \frac{ {\sigma}_{1}^{2} }{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2} } $$

Pay attention that the solution doesn't require us to know the exact variances of the estimators being fused. Only their ratio.

So in the case above where $ {\sigma}_{1}^{2} : {\sigma}_{2}^{2} = 0.8 : 0.2 $ the solution is $ {w}_{1} = 0.2 $ and $ {w}_{2} = 0.8 $. This is the best you can do without the knowledge of the co variance.

This fused veritable will have variance $ {\sigma}_{3}^{2} = {0.2}^{2} {\sigma}_{1}^{2} + {0.8}^{2} {\sigma}_{2}^{2} = 0.16 \hat{r} $

Wikipedia Solution

You may, as a reference, look at Wikipedia - Sensor Fusion.
They wrote the same equation in a real clumsy manner:

enter image description here

Yet they are equivalent (Just multiply their result by $ \frac{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2} }{ {\sigma}_{1}^{2} + {\sigma}_{2}^{2}} $)

Pay attention that the sensor model assumes that 2 sensors are not correlated (Well, it assumes noise is property of the measurement equipment, hence this assumption is made).

  • $\begingroup$ in my case there is correlation. $\endgroup$
    – Elnaz
    Oct 18 '19 at 10:20
  • $\begingroup$ In case the there is correlation but it is not known the weights won't be optimal but it will work. This is the best you can do unless you us known information about the correlation. $\endgroup$
    – Royi
    Oct 18 '19 at 10:40
  • $\begingroup$ @ Royi: That's the idea but the variance of $.2 \hat\tau$ is $.4 \times $ var($\hat\tau$) and the variance of $.8 \hat\tau $ is $.64 \times $ var($\hat\tau$) so you don't need $\sigma^2_1$ and $\sigma^2_2$. $\endgroup$
    – mark leeds
    Oct 18 '19 at 12:53
  • $\begingroup$ What? I didn't write a specific values. The equation above is correct to nay variance ratio between the 2. $\endgroup$
    – Royi
    Oct 18 '19 at 12:59
  • $\begingroup$ You don't need time to calculate $ \frac{0.2}{1} $ and $ \frac{0.8}{1} $. As it doesn't matter in the equation above ot know the variances, only the ratio. So here you go $ {w}_{1} = 0.2 $ and $ {w}_{2} = 0.8 $. $\endgroup$
    – Royi
    Oct 18 '19 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.