Background information

According to Cover's text book on information theory Mutual Information is calculated as:

$$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)}) $$

Now if we were interested in a single Word W_i we would just calculate this with one sum:

$$(2) I(W_i;C) = \Sigma_k P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)}) $$

OR we can rewrite this as

$$ (3) I(W_i;C) = \Sigma_k P(C_k,W_i)log(\frac{P(W_i|C_k)}{P(W_i)}) $$

because $$ (4) P(W_i|C_k) = \frac{P(W_i,C_k)}{P(C_k)}$$ (unless i am incorrect)

Question While reading a paper on feature selection using information theory measurements I came across A segment of the text:

enter image description here

Should the equation be more like (3), and I should contact the Author, or have I mis-understood/interpreted mutual information?

  • $\begingroup$ I don't see any difference between Prof. Cover's equation $(3)$ and the Author's equation $(2)$ from the paper. $\endgroup$
    – Sudarsan
    Sep 12, 2013 at 23:46

1 Answer 1


No there is nothing wrong with the formulas in the paper.

  • MI(S) is the average mutual information of set S over all classes c in C. It is NOT a reformulation of the formula for MI(W,C).

  • MI(w) is identical to your formula for I(Wi,C) (except it uses slightly different notation).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.