In Chapter 5 of the Optimum Array Processing by Harry van Trees [1], the formula of the complex Gaussian distribution appears very different from the formula that is used elsewhere in the same book series, in Detection, Estimation, and Filtering Theory [2], in Chapter 3. While (3.36) in Chapter 3 is $$ p_{{\bf r}}({\bf R}) = \frac{1}{\pi^N|{\bf K}|} \exp\left( -{\bf R}^H{\bf K_r}^{-1}{\bf R} \right) $$ the formula (5.34) in Chapter 5 is $$ p_{{\bf x}}({\bf X}) = \frac{1}{\pi^N|{\bf S}|} \exp\left( -{\bf X}{\bf S_x}^{-1}{\bf X}^H \right) $$

This seems confusing, because the Hermitian transpose has shifted from the first factor in the exponent to the last, apparently making the exponent a square matrix, whereas probability densities are required to be scalars. However, the change in the notation is clearly intentional because the author uses it consistently throughout the chapter. What explains the distinction between these two definitions of the complex Gaussian distribution?


[1] H. L. van Trees, Optimum Array Processing. New York, NY, USA: John Wiley & Sons, Inc., 2002.

[2] H. L. van Trees, Detection, Estimation, and Modulation Theory, Part I: Detection, Estimation, and Filtering Theory. John Wiley & Sons., 2nd ed. 2013

  • 2
    $\begingroup$ Isn't it simply $\mathbf{X}$ is a row vector whereas $\mathbf{R}$ is a column one? $\endgroup$
    – AlexTP
    Feb 21 at 14:01
  • $\begingroup$ Welcome to SE.SP! I can't see equation (5.34) in my version of Part I because it's a much older version (equations in it are not numbered including the chapter number). Chapter 5's equation (34) in my version is $K_n(t,u) = \frac{N_o}{2} \delta(t-u)$. $\endgroup$
    – Peter K.
    Feb 21 at 14:11
  • $\begingroup$ I have now discussed the topic with various academic experts. The overall understanding seems to be that (5.34) is merely a misprint and the Hermitian transpose should be in the first factor in both cases. In other words, (3.36) is the correct formula. $\endgroup$ Mar 13 at 9:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.