I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it.

I am given the following:

  • $X$ is a uniformly distributed random variable between 1-8 (so with probability $\frac{1}{8}$)

  • $Y$ is distributed as such: $\textrm{Pr}(Y = k) = 2^{-k}$, where $k = 1,2,3.....$

I need to find $H(X)$, $H(Y)$, and $H(X +Y, X-Y)$.

I'm able to find the first two:

\begin{align} H(X) &= -\sum_{k=1}^8\left(\frac{1}{8}\cdot \log_2\left(\frac{1}{8}\right)\right)=3\\ H(Y) &= -\sum_{k=1}^\infty \left(2^{-k}\cdot \log_2\left(2^{-k}\right)\right) = \sum k2^{-k} = \frac{\frac{1}{2}}{\left(1-\frac{1}{2}\right)^2}=2 \end{align}

The part I'm not really sure how to do is the $H(X+Y, X-Y)$. I'm just not sure how you could combine these two. Could anyone point me in the right direction?


Using the chain rule: $$H(A,B)=H(A)+H(B|A)$$ we have $$H(X+Y,X-Y)=H(X+Y)+H(X-Y|X+Y)$$

Now let's assume $X+Y$. Since $X\in[1,8]$ and $Y\in(0,0.5]$, for any given $X+Y$, one can find out the actual $X$ and $Y$ and therefore, the value of $X-Y$. For this reason, we have $$H(X-Y|X+Y)=0$$



Again, based on the above explanation there is a one-one relationship between $X+Y$and $(X,Y)$ and we conclude $$H(X+Y,X-Y)=H(X,Y)=H(X)+H(Y|X)$$

You didn't mention if $X$ and $Y$ are independet or not. If so, then $H(Y|X)=H(Y)$ and $$H(X+Y,X-Y)=H(X)+H(Y)=5$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.