1
$\begingroup$

I just started to study information content of the source and there are some very fundamental topics I am not really clear on. These might be a long question or a series of questions. Please help me.

  1. Relationship between uncertainty and information: Information is related to uncertainty or surprise. What is the nature of this relationship? Is there any difference between uncertainty and information? Before the information reaches the destination, there is an uncertainty at the destination about which signal would appear. When the information reaches the destination, there is no uncertainty left, right? So, information is literally killing or eliminating the uncertainty. Is my concept about information correct?

  2. Information content of an improbable or unlikely message: The relationship between the probability of a message and information content is $I=\log_2(1/P)$, right? So If the probability of occurrence of a symbol is one, then the destination is always sure that that definite symbol will occur. So there is no uncertainty about the message. In other way, when the message reaches the server, there is no surprise. So the information content of that symbol is zero. Mathematically it is zero too. That is, the source didn't really need to transmit that message as the destination knew what message was coming beforehand, correct?

    But what would be the information content of a message with probability zero? I know mathematically it is infinity. But what would be the physical interpretation? And the probability of the message to occur is zero, meaning that symbol is never going to be generated. So the destination is never waiting for that symbol to occur. So in that case, isn't the information content of that message supposed to be zero?

That is going to be my first question. I might need to post a few questions later. Thanks in advance.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If an angel/demon appears and gives you all the stock market closing prices for the next year, you don't expect that, the probability of that message happening is (close enough to) zero, but if (magic happens) the informational value of that message would be extremely high, not zero. $\endgroup$
    – hotpaw2
    Commented Oct 7, 2015 at 3:09
  • 1
    $\begingroup$ Realistically, if you receive a message that has a probability of zero, then there's a 100% probability that the message has been corrupted in transmission. $\endgroup$
    – Simon B
    Commented Oct 7, 2015 at 16:01

1 Answer 1

Reset to default
1
$\begingroup$

  1. What is the nature of this relationship? There are many possibilities. Shannon used a logarithm because he wanted information to be additive, that is, $I(xy)=I(x)+I(y)$ for independent events $x$ and $y$.

    You're correct that the arrival of a message at the receiver decreases its uncertainty. It may not be decreased to zero, because there may be an error in the communication. Also, a given channel can only support a certain information rate, which is its capacity.

  2. It does not make sense to ask about messages of probability zero. The source alphabet is defined so that all symbols have probability greater than zero.

Improve this answer
$\endgroup$
2
  • $\begingroup$ the information metric is additive only if message $x$ and message $y$ are independent. the measure of information of message $xy$ is less than the sum of measures of information of the two messages separately, if the two messages are somewhat dependent statistically or probabilistically. $\endgroup$
    – robert bristow-johnson
    Commented Nov 6, 2015 at 3:58
  • $\begingroup$ @robertbristow-johnson, you make a good point: $I(xy)\leq I(x)+I(y)$, with equality iff $x$ and $y$ are independent. $\endgroup$
    – MBaz
    Commented Nov 6, 2015 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.