1
$\begingroup$

Why is the value of $p_i\log(p_i)$ in entropy formula considered $0$ when $p_i =0$? I get that it is the limiting value, but does that mean $p$ is never equals to zero, but only tends to?

$\endgroup$

2 Answers 2

3
$\begingroup$

Two possible answers:

  • If a symbol has probability zero, then it does not influence the calculation and there is no need to include it, so you never actually calculate $0\log(0)$.

  • If you insist in including symbols with zero probability, then $0\log(0)=0$ by convention.

$\endgroup$
3
  • $\begingroup$ it doesn't have to be "by convention". the limit $$ \lim_{p \to 0} p \log(p) = 0 $$ $\endgroup$ Commented Mar 26, 2021 at 16:33
  • 1
    $\begingroup$ @robertbristow-johnson I would agree only if you write the limit with $p \rightarrow 0^+$. But the limit itself (the "two-sided limit") does not exist, as far as I understand these things. $\endgroup$
    – MBaz
    Commented Mar 26, 2021 at 17:12
  • $\begingroup$ yeah you're right. too late to change it. oops. $\endgroup$ Commented Mar 26, 2021 at 19:03
2
$\begingroup$

I am not adding a lot to MBaz, mostly graphical hints. It can be interesting to look at the elementary function behind Shannon entropy: $H: p\mapsto -p \log p -(1-p) \log (1-p)$, displayed below:

Shannon Entropy for bernoulli probability

While it seems not defined at $p=0$ or $p=1$, the function $H$ is very symmetric and behaves quite well at $0$ and $1$ for a Bernoulli trial. Here, we talk about a binary event ($X$ happens with probability $p=P(X)$. The converse event is "not $X$". While entropy is often described as a measure of information, it can be seen as a measure of uncertainty. If $X$ is always equal to 1, it is certain. If $X$ never occurs, its converse is certain as well. In both cases, we have not surprise: the uncertainty is zero, and the "definitions" $-p \log p = 0$ for $p=0$ or $-(1-p) \log (1-p) =0$ for $p=1$ make sense.

More details can be found in Entropy is a measure of uncertainty.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.