- What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and entropy. What is the interpretation if MI of error is decreasing or getting minimized?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
What does mutual information (MI) convey?
It indicates that there is a relationship between the two signals- i.e. that they are not independent. It could be that they are correlated, but the relationship does not have to be that simple. The higher the mutual information, the stronger the relationship (i.e. the more one of the signals can "tell" you about the other).
What does this imply and mean in the case of mutual information of error, and in general definition of MI and entropy.
If the error has mutual information with the observation this implies that the receiver did not use all of the information in the observation- i.e. the receiver is not ideal. If the receiver was ideal it would have used all of the information in the observation, so what was left in the error signal would be independent of the observation.
What is the interpretation if MI of error is decreasing or getting minimized?
Decreasing MI of error implies that the receiver is doing a better job. Once the MI is zero it is impossible to do any better.
-
$\begingroup$ Thank you for your intuitive reply which has created some more doubts, if you may kindly clarify them.(A) When you say that signal is dependent if MI is non-zero. So, is correlation and dependency the same?A signal that is independent is also uncorrelated? (B)If MI of error reduces then how does it imply that we are reaching the true estimates?(C)SInce,formula of MI contains entropy, can I say that increase of entropy increases information content or just the information?(I am not clear about relation between entropy and information).What is the relation between entropy & information? $\endgroup$ Commented Jul 27, 2014 at 19:57
-
$\begingroup$ Lastly, does an ideal receiver means no noise? Thank you $\endgroup$ Commented Jul 27, 2014 at 22:02
-
$\begingroup$ No, an ideal receiver does not mean "no noise". There is always noise. $\endgroup$– Jim ClayCommented Jul 28, 2014 at 15:05
-
$\begingroup$ No, correlation and dependency are not the same. Signals that are correlated are not independent, but dependent signals are not necessarily correlated. $\endgroup$– Jim ClayCommented Jul 28, 2014 at 15:06