In the book Fundamentals of Statistical Processing, Volume I: Estimation Theory by Steven M. Kay on page 19, it say that the mean square error of an estimator, $\hat{\theta}$, given the true value, $\theta$, is defined as

\begin{align} \text{mse}(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2] \\ &= E \left[ \left[\left( \hat{\theta} - E[\hat{\theta}] \right) + \left( E[\hat{\theta}] - \theta \right) \right]^2 \right]\\ &= \text{var}(\hat{\theta}) + b^2(\theta) \end{align} where $b(\theta) = E[\hat{\theta}] - \theta$ is the bias.

But if I write it out, it doesn't come out to be the same

\begin{align} \text{mse}(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2] \\ &= E \left[ \left[\left( \hat{\theta} - E[\hat{\theta}] \right) + \left( E[\hat{\theta}] - \theta \right) \right]^2 \right]\\ &= E \left[ \left( \hat{\theta} - E[\hat{\theta}] \right)^2 + \left( E[\hat{\theta}] - \theta \right)^2 + 2 \left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right]\\ &= \text{var}(\hat{\theta}) + b^2(\theta) + 2 E \left[\left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right] \end{align}

What am I missing here?



$$ E\left[\hat \theta - E\left(\hat \theta \right)\right] = E\left(\hat \theta\right) - E\left(\hat \theta\right) $$

  • $\begingroup$ well now i feel stupid :) thanks anyways $\endgroup$
    – Zero
    Dec 30 '20 at 16:32
  • $\begingroup$ It's important that $\theta$ is modeled as a deterministic parameter rather than a random variable. That's why we can take the other factor out of the expectation. $\endgroup$
    – Matt L.
    Dec 30 '20 at 17:13
  • $\begingroup$ @MattL. yeah...i just didn't even think about expanding further and stopped like in the question...thanks :) $\endgroup$
    – Zero
    Jan 1 at 10:50
  • $\begingroup$ @MattL. It is indeed true that for classical estimation (contrary to Bayesian) $\theta$ is deterministic. $\endgroup$
    – Gilles
    Jan 1 at 14:13

It's important to point out here that Kay is talking about classical estimation, which means that the parameters to be estimated are unknown but deterministic. Hence, the term $E[\hat{\theta}]-\theta$ is deterministic, and, consequently, the last term in your equation becomes

$$\begin{align} E \left[\left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right]&= \left(E[\hat{\theta}]-\theta\right)E\left[\hat{\theta}-E[\hat{\theta}]\right]\\&=\left(E[\hat{\theta}]-\theta\right)\left(E[\hat{\theta}]-E[\hat{\theta}]\right) \\&=0\end{align}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.