Given $ M \in \mathbb{R}^{N \times N} $ which is a Positive Definite Matrix.
Let $ \hat{x} $ the MMSE of $ x $ given $ z $, namely $ \hat{x} = \mathbb{E} \left[ x \mid z \right] $.

Prove the equivenalce of the following expressions:

  1. $ \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} \left( \hat{x} - x \right) \mid z \right] $.

  2. $ \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} M \left( \hat{x} - x \right) \mid z \right] $.

  3. $ \arg \min_{\hat{x}} \operatorname{Tr} \left( M \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} \left( \hat{x} - x \right) \mid z \right] \right) $

Equivalence means all are actually minimized by $ \hat{x} = \mathbb{E} \left[ x \mid z \right] $.

The equivalence of 2 and 3 is easy using the Cyclic Property of the $ \operatorname{Tr} \left( \cdot \right) $ operator.
Yet showing the invertible transformation keeps the solution (Equivalence of 1 and 2) isn't trivial.

  • 1
    $\begingroup$ This question has been asked simultaneously on stats.SE where it seems a more natural fit. I recommend closing it here and migrating it to stats.SE $\endgroup$ Mar 25, 2014 at 3:29
  • $\begingroup$ @DilipSarwate, As you can see, It was answered here and not there. Since both in beta and have low exposure with almost no overlap in people I see no harm in it. Where I'll get the answer I will post it on the other one so they both will have this knowledge. $\endgroup$
    – Royi
    Mar 25, 2014 at 4:54

2 Answers 2


Use the Cholesky decomposition of $M$ as a change of coordinates.

$tr(ABC) = tr(CAB)$ - trace is invariant under cyclic permutations, and $tr(scalar)=scalar$. Thus, $tr(M E[ x x^T]) = tr(E[M x x^T]) = E[tr(M x x^T) ] = E[tr( x^T M x ) ] = E[x^T M x]$. Now, replace $x$ with $\hat{x}-x$.

  • $\begingroup$ The property still holds in this case. As for the first comment, use the first line of the answer. $\endgroup$
    – Batman
    Mar 25, 2014 at 13:48
  • 1
    $\begingroup$ PD by definition requires hermitian/symmetric, so cholesky exists. Try it on your own, noting that $\hat{x} = E[X|Z] = E[X|\text{invertible function of }z]$ and using the cholesky decomposition to find the invertible function. $\endgroup$
    – Batman
    Mar 25, 2014 at 17:17
  • $\begingroup$ I don't think the function should be applied on $ z $. Unless the answe I added is wrong. Though the property of the Conditional Expectation you mentioned is intuitive, I would be happy to see its derivation. Is there a place to see a full derivation of it? $\endgroup$
    – Royi
    Jul 1, 2018 at 15:46
  • $\begingroup$ @Royi: I loved your proof. In Batman's statement regarding $invertible function$, I think that he is using the fact that $E[X | z, g(z)] = E[X | g(z)]$. This can be proven using the tower property of conditional expectation. ( condition on g(z) again and then the inside z drops out ). But I'm still not clear how one can use that result to prove the original statement. If you do, the knowledge is appreciated. $\endgroup$
    – mark leeds
    Jul 2, 2018 at 0:17
  • $\begingroup$ @markleeds, I also don't see how to apply a function on $ z $ in order to show the equivalence of 1 and 2. Hopefully Batman will explain. $\endgroup$
    – Royi
    Jul 2, 2018 at 4:14

Since $ M \in \mathbb{S}^{N}_{++} $ (In other convention $ M \succ 0 $) by Cholesky Decomposition there is a Triangular Matrix $ R \in \mathbb{R}^{N \times N} $ such that $ M = {R}^{T} R $.

Using this fact one could prove $ 1 \iff 2 $ as following:

$$\begin{align*} \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} M \left( \hat{x} - x \right) \mid z \right] & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} {R}^{T} R \left( \hat{x} - x \right) \mid z \right] && \text{$M = {R}^{T} R $} \\ & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( R \left( \hat{x} - x \right) \right)}^{T} \left( R \left( \hat{x} - x \right) \right) \mid z \right] && \text{} \\ & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( R \hat{x} - y \right)}^{T} \left( R \hat{x} - y \right) \mid z \right] && \text{Defining $ y = R x $} \end{align*}$$

Clearly, the above, as the classic MMSE problem, with respect to $ y $ given $ z $ which is minimized when:

$$ R \hat{x} = \mathbb{E} \left[ y \mid z \right] \Rightarrow \hat{x} = {R}^{-1} \mathbb{E} \left[ y \mid z \right] $$

Since $ M \succ 0 $ then $ {R}^{-1} $ is defined and the above is valid.
Moreover by linearity of the Expectation Operator:

$$ \hat{x} = {R}^{-1} \mathbb{E} \left[ y \mid z \right] = {R}^{-1} \mathbb{E} \left[ R x \mid z \right] = {R}^{-1} R \mathbb{E} \left[ x \mid z \right] = \mathbb{E} \left[ x \mid z \right] $$

Hence the minimizer of 1 indeed minimizes 2.

Using the cyclic property of the Trace Operator and the linearity of the Expectation Operator (Hence one could change the order of the $ \operatorname{Tr} \left( \cdot \right) $ and $ \mathbb{E} \left[ \cdot \right] $) one could easily show $ 2 \iff 3 $ hence the problem is solved.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.