# Can somebody help me with some very basic questions about the fundamentals of information theory?

I just started to study information content of the source and there are some very fundamental topics I am not really clear on. These might be a long question or a series of questions. Please help me.

1. Relationship between uncertainty and information: Information is related to uncertainty or surprise. What is the nature of this relationship? Is there any difference between uncertainty and information? Before the information reaches the destination, there is an uncertainty at the destination about which signal would appear. When the information reaches the destination, there is no uncertainty left, right? So, information is literally killing or eliminating the uncertainty. Is my concept about information correct?
2. Information content of an improbable or unlikely message: The relationship between the probability of a message and information content is $I=\log_2(1/P)$, right? So If the probability of occurrence of a symbol is one, then the destination is always sure that that definite symbol will occur. So there is no uncertainty about the message. In other way, when the message reaches the server, there is no surprise. So the information content of that symbol is zero. Mathematically it is zero too. That is, the source didn't really need to transmit that message as the destination knew what message was coming beforehand, correct?

But what would be the information content of a message with probability zero? I know mathematically it is infinity. But what would be the physical interpretation? And the probability of the message to occur is zero, meaning that symbol is never going to be generated. So the destination is never waiting for that symbol to occur. So in that case, isn't the information content of that message supposed to be zero?

That is going to be my first question. I might need to post a few questions later. Thanks in advance.

• If an angel/demon appears and gives you all the stock market closing prices for the next year, you don't expect that, the probability of that message happening is (close enough to) zero, but if (magic happens) the informational value of that message would be extremely high, not zero. – hotpaw2 Oct 7 '15 at 3:09
• Realistically, if you receive a message that has a probability of zero, then there's a 100% probability that the message has been corrupted in transmission. – Simon B Oct 7 '15 at 16:01

1. What is the nature of this relationship? There are many possibilities. Shannon used a logarithm because he wanted information to be additive, that is, $I(xy)=I(x)+I(y)$ for independent events $x$ and $y$.
• the information metric is additive only if message $x$ and message $y$ are independent. the measure of information of message $xy$ is less than the sum of measures of information of the two messages separately, if the two messages are somewhat dependent statistically or probabilistically. – robert bristow-johnson Nov 6 '15 at 3:58
• @robertbristow-johnson, you make a good point: $I(xy)\leq I(x)+I(y)$, with equality iff $x$ and $y$ are independent. – MBaz Nov 6 '15 at 14:11