In the book Fundamentals of Statistical Processing, Volume I: Estimation Theory by Steven M. Kay on page 19, it say that the mean square error of an estimator, $\hat{\theta}$, given the true value, $\theta$, is defined as

\begin{align} \text{mse}(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2] \\ &= E \left[ \left[\left( \hat{\theta} - E[\hat{\theta}] \right) + \left( E[\hat{\theta}] - \theta \right) \right]^2 \right]\\ &= \text{var}(\hat{\theta}) + b^2(\theta) \end{align} where $b(\theta) = E[\hat{\theta}] - \theta$ is the bias.

But if I write it out, it doesn't come out to be the same

\begin{align} \text{mse}(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2] \\ &= E \left[ \left[\left( \hat{\theta} - E[\hat{\theta}] \right) + \left( E[\hat{\theta}] - \theta \right) \right]^2 \right]\\ &= E \left[ \left( \hat{\theta} - E[\hat{\theta}] \right)^2 + \left( E[\hat{\theta}] - \theta \right)^2 + 2 \left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right]\\ &= \text{var}(\hat{\theta}) + b^2(\theta) + 2 E \left[\left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right] \end{align}

What am I missing here?


2 Answers 2



$$ E\left[\hat \theta - E\left(\hat \theta \right)\right] = E\left(\hat \theta\right) - E\left(\hat \theta\right) $$

  • $\begingroup$ well now i feel stupid :) thanks anyways $\endgroup$
    – Zero
    Dec 30, 2020 at 16:32
  • $\begingroup$ It's important that $\theta$ is modeled as a deterministic parameter rather than a random variable. That's why we can take the other factor out of the expectation. $\endgroup$
    – Matt L.
    Dec 30, 2020 at 17:13
  • $\begingroup$ @MattL. yeah...i just didn't even think about expanding further and stopped like in the question...thanks :) $\endgroup$
    – Zero
    Jan 1, 2021 at 10:50
  • $\begingroup$ @MattL. It is indeed true that for classical estimation (contrary to Bayesian) $\theta$ is deterministic. $\endgroup$
    – Gilles
    Jan 1, 2021 at 14:13

It's important to point out here that Kay is talking about classical estimation, which means that the parameters to be estimated are unknown but deterministic. Hence, the term $E[\hat{\theta}]-\theta$ is deterministic, and, consequently, the last term in your equation becomes

$$\begin{align} E \left[\left( \hat{\theta} - E[\hat{\theta}] \right) \left( E[\hat{\theta}] - \theta \right) \right]&= \left(E[\hat{\theta}]-\theta\right)E\left[\hat{\theta}-E[\hat{\theta}]\right]\\&=\left(E[\hat{\theta}]-\theta\right)\left(E[\hat{\theta}]-E[\hat{\theta}]\right) \\&=0\end{align}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.