5
$\begingroup$

Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information about channel input $X$ can be obtained observing channel output $Y$.

Doesn't this mean that $H(X|Y)$ and $I(X,Y)$ complement each other or tell us the same thing? If so, then since mutual information is symmetric $I(X,Y)=I(Y,X)$, shouldn't conditional entropy be symmetric as well?

$\endgroup$
  • 2
    $\begingroup$ Consider something more basic. Shouldn't the conditional probability $P(A\mid B)$ have the same value as the conditional probability $P(B\mid A)$? After all, they are dependent on $P(A\cap B)$ and $P(B\cap A)$ respectively and we know that $P(A\cap B) = P(B\cap A) ~ \ldots$ $\endgroup$ – Dilip Sarwate Jan 17 '14 at 15:02
3
$\begingroup$

No,because while mutual information tells you how much uncertainty can a random variable $X$ remove from another random variable $Y$, the conditional entropy of $X$ given $Y$ tells you how much uncertainty remained in X after using the information (given by $I(X;Y)$) that $Y$ gave it.

$H(X|Y)=H(X)-I(X;Y)$

That's why the conditional entropy depends on the value of the entropy before the observation and the mutual information isn't, because it is only the difference ($\delta$) between two entropy states, before and after the observation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.