I need to prove that mutual information given by

$$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$

is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am proceeding like this

$$I(X;Y)=\int_{x,y} f(x) \frac{ f\left(x,y \right)}{f(x)} \log_2 (f(Y|X)) \, dx \, dy - \int_{x,y}f(x,y) \log_2(f(y)) \, dx \, dy$$

where first term gives me $H(X|Y)$ and second term gives me

$$\int_{x,y}f(y) f(x|y) \log_2(f(y)) \, dx \, dy $$

and I am taking

$$\int_{x} f\left (x|y \right) dx =1$$

I mean to say if this equation is right what does this physically symbolize that given $y$ there must be some $x$ that means given output there must be some input with probability $1$? Am I doing the derivation right ?


1 Answer 1


Your derivation is correct. Your second integral can also be written as

$$ \int_{x,y}f(x,y)\log_2(f(y))dxdy=\int_y\log_2(f(y))\left(\int_xf(x,y)dx\right) dy $$

where the integral in parenthesis is equal to

$$ \int_xf(x,y)dx=\int_xf(x|y)f(y)dx=f(y)\int_xf(x|y)dx $$

You can understand this integral in two ways:

  1. Like your idea: $\int_xf(x|y)dx=1$, which means that regardless of what value $y$ has, $x$ must have some value, so the integral over all possibilities is 1. You can understand that also by seeing $f(x|y)$ as some distribution for $x$ (which is parametrized by $y$). And, the integral over the support of a distribution needs to be $1$.

  2. $\int_xf(x,y)dx=f(y)$ is the marginalization of $f(x,y)$ regarding the variable $x$. With the integration, you remove the influence (i.e. marginalize out) of the variable $x$ from the probability density.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.