How to prove relationship between mutual information and differential entropy?

Question

I need to prove that mutual information given by

$$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$

is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am proceeding like this

$$I(X;Y)=\int_{x,y} f(x) \frac{ f\left(x,y \right)}{f(x)} \log_2 (f(Y|X)) \, dx \, dy - \int_{x,y}f(x,y) \log_2(f(y)) \, dx \, dy$$

where first term gives me $H(X|Y)$ and second term gives me

$$\int_{x,y}f(y) f(x|y) \log_2(f(y)) \, dx \, dy $$

and I am taking

$$\int_{x} f\left (x|y \right) dx =1$$

I mean to say if this equation is right what does this physically symbolize that given $y$ there must be some $x$ that means given output there must be some input with probability $1$? Am I doing the derivation right ?

Answer 1

Your derivation is correct. Your second integral can also be written as

$$ \int_{x,y}f(x,y)\log_2(f(y))dxdy=\int_y\log_2(f(y))\left(\int_xf(x,y)dx\right) dy $$

where the integral in parenthesis is equal to

$$ \int_xf(x,y)dx=\int_xf(x|y)f(y)dx=f(y)\int_xf(x|y)dx $$

You can understand this integral in two ways:

1. Like your idea: $\int_xf(x|y)dx=1$, which means that regardless of what value $y$ has, $x$ must have some value, so the integral over all possibilities is 1. You can understand that also by seeing $f(x|y)$ as some distribution for $x$ (which is parametrized by $y$). And, the integral over the support of a distribution needs to be $1$.

2. $\int_xf(x,y)dx=f(y)$ is the marginalization of $f(x,y)$ regarding the variable $x$. With the integration, you remove the influence (i.e. marginalize out) of the variable $x$ from the probability density.