Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that $$ b=\begin{cases} 0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\ 1, & \text{if} \,\, a=01 \quad \text{or} \quad a=10 \end{cases} $$ Now assume we transmit $(a_i,bj)$ where $a_i$ is a symbol in $A$ and $b_j$ is the parity bit associated with it. I calculated the entropy for $A$ as follows: $$ H(A)=4 [4 \log_2(4)]=2 $$ and to calculate the entropy of the new symbols, call it alphabet $C=\{000,011,101,110\}$, we need the probabilities: $$ p_c(0)=\mathbb{P}(a=00 \, \text{and} \, b=0)=\mathbb{P}(a=00)\mathbb{P}(b=0 \mid a=00)=\mathbb{P}(a=00)=1/4 $$ Similarly, $p_c(1)=p_c(2)=p_c(3)=p_c(0)=1/4$, the entropy of $C$ is $$ H(C)=4 [4 \log_2(4)]=2=H(A) $$ How is this the case? We have more bits per symbols in $C$.


1 Answer 1


Because the entropy represents information quantity, or if being measured in bit, the smallest number of bits per symbol we need to represent a source.

The source $A$ contains $4$ equiprobable symbols hence it is obvious that we need $\log_2(4) = 2$ bits per symbol to represent the source.

The source $C$ is simply $A$ with its parity bits hence no new information is added. We have more bits per symbols, but the number of information bits is unchanged.

In communications, these additional bits are added to scope with error of transmissions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.