Beyond the raw equation for calculating mutual information, what does it mean in physical terms? For example: From Information Theory, we know that entropy is the smallest loss-less compression scheme that we can use on a alphabet with a specific probability distribution.
What would that mean in terms of Mutual Information?
Background: I'm trying to calculate the mutual information of uni-gram words and determine which of two books did they come from.
essential $$ I(book; word)$$