0
$\begingroup$

The KL divergence between two real-valued Gaussian distributions with means $\mu_1$ and $\mu_2$ and common variance $\sigma^2$ is well known to be:

$$ D_{\text{KL}}\left(\mathcal{N}(\mu_1, \sigma^2) \parallel \mathcal{N}(\mu_2, \sigma^2)\right) = \frac{\lVert \mu_2 - \mu_1 \rVert_2^2}{2\sigma^2}. $$

Assume now two circularly symmetric complex Gaussian distributions with means $\nu_1$ and $\nu_2$ and common variance $\gamma^2$, where we denote the total variance as $\gamma^2$, i.e. the real and imaginary components of each of the distributions have variance $\gamma^2 \over 2$ each.

One would expect the complex KL-divergence to also be:

$$ D_{\text{KL}}\left(\mathcal{N}_{\mathbb{C}}(\nu_1, \gamma^2) \parallel \mathcal{N}_{\mathbb{C}}(\nu_2, \gamma^2)\right) = \frac{\lVert \nu_2 - \nu_1 \rVert_2^2}{2 \gamma^2}. $$

To compute the KL-divergence, I express $\mathcal{N}_{\mathbb{C}}(\nu_1, \gamma^2)$ as a product distribution: $$ p(x, y) = \frac{1}{\pi\gamma^2}\exp\left(-\frac{(x-\Re(\nu_1))^2(y-\Im(\nu_1))^2}{\gamma^2} \right), $$ and proceed similarly for $\mathcal{N}_{\mathbb{C}}(\nu_2, \gamma^2)$ to obtain $q(x, y)$. We observe that the $2$ in the denominators is gone because it cancels with the $\gamma^2\over 2$. Now, computing the KL-divergence amounts to computing the integral: $$ \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} p(x,y)\log\left(\frac{p(x,y)}{q(x,y)}\right) \; \mathrm{d}x \,\mathrm{d}y. $$

Unsurprisingly, this yields: $$ \frac{\lVert \nu_2 - \nu_1 \rVert_2^2}{\gamma^2} $$

However, this quantity is double the expected one. How is this possible? Is there a special definition of the KL divergence for complex distributions or am I making another error? My intuition would dictate that two "equivalent" distributions (real and complex Gaussian with a given mean and standard deviation) would have the same KL divergence.

$\endgroup$

1 Answer 1

2
$\begingroup$

The circularly-symmetric complex normal distribution does not generalize the real normal distribution because the complex one consists of two jointly (real) normal ones, by definition.

Another way to see it is that the only way to make a complex normal distribution real is to force the imaginary part to have zero mean and zero variance whose PDF turns into a Dirac delta function that cannot be treated as a conventional function.

Therefore, there is no unified way to treat the two distributions in your context. The results are different because they are as they are.

$\endgroup$
3
  • $\begingroup$ That is a valid view, but the whole thing still seems "off" right? If the closed-form solution for the KL divergence depends only on mean separation and variance, it totally contradicts intuition that the divergence would be different for two distributions which are specified in the exact same way. So many other things are nearly identical between the circularly symmetric complex Gaussian and a regular Gaussian, why not this? $\endgroup$
    – Sami
    Aug 21, 2022 at 17:08
  • $\begingroup$ @Sami OK, if you want to see it that way, you can resort to the additive property of independent distributions: the KL divergence is doubled. However, it is not rigorous to say that the complex normal generalizes the real normal. $\endgroup$
    – AlexTP
    Aug 21, 2022 at 17:29
  • $\begingroup$ I agree and edited the question. $\endgroup$
    – Sami
    Aug 21, 2022 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.