Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this community
Beyond the raw equation for calculating mutual information, what does it mean in physical terms? For example: From Information Theory, we know that entropy is the smallest loss-less compression scheme that we can use on a alphabet with a specific probability distribution.
What would that mean in terms of Mutual Information?
Background: I'm trying to calculate the mutual information of uni-gram words and determine which of two books did they come from.
essential $$ I(book; word)$$
Mutual Information by definition relates two random variables (RV) and it measures the dependence between the two RVs from the information content perspective i.e. the measure of amount of information contained by one RV about the other RV. And Mutual information is a symmetric quantity, i.e., $I(X;Y) = I(Y; X)$.
In case of a communication channel, the maximum achievable capacity for the channel is the maximum of the mutual information between the channel input and the output $C = \max_{p(x)} I(X; Y)$.
In your case, the two RVs $X$ and $Y$ would correspond to books and words. The mutual information would measure the amount of information common between a (book, word) pair. Obviously you'd associate the word to the book with which you have the maximum mutual information. This is the maximum mutual information approach.
Two additional intuitive takes on mutual information:
When two random variables are independent the joint distribution $p(x, y)$ and the product of the marginal distributions $p(x)$ and $p(y)$ are identical. One could thus assess the degree of independent between two random variables by computing a probabilistic distance between $p(x) \times p(y)$ and $p(x, y)$ - this distance being 0 when the two variables are independent. A common probabilistic distance between variable is the Kullback-Leibler divergence. If you take the Kullback-Leibler divergence between the joint distribution and the product of the marginals of two random variables, you end up with... mutual information.
From a compression / coding perspective, imagine you are given a sequence of $N$ pairs of observations $(x, y)$. You want to compress them into a file. Two strategies: storing all the (x) in one compressed file then independently all the (y) in another compressed file ; vs compressing the pairs. Using an optimal coder, the file size in the first case is $N \times H(X) + N \times H(Y)$, while in the second case the file size is $N \times H(X, Y)$. The second approach is more efficient if there's a relationship between the two observed variables! How many bits have we saved per observation? $\frac{N \times H(X) + N \times H(Y) - N \times H(X, Y)}{N} = I(X,Y)$! So mutual information tells us how many bit per observation do we save by coding two data streams jointly rather than independently.
I am not sure about your example, though... Mutual information is computed between two random variables (distributions). I can see how "book" can represent the distribution of words in a book ; but I am not sure what "word" means here. Mutual information also requires "paired" observations to be computed.
