# Questions tagged [information-theory]

The tag has no usage guidance.

90 questions
Filter by
Sorted by
Tagged with
59 views

### Discrete entropies

I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it. I am given the following: $X$ is a uniformly distributed random ...
204 views

### Mechanics of a matrix Interleaver

I am having trouble understanding exactly how the matrix interleaver. I have read the following page from MathWorks. In it, it gives the following example where "123456" is interleaved as "142536." ...
2k views

146 views

### Does any error correction code still work in such situation?

I'm looking for a kind of error correction code or solution that can correct my codeword in this case: My message holds $k$ bits, and $2k$ bits codeword (rate is $1/2$) is produced by the generator ...
7k views

### What are some typical lossless compression ratios?

A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I ...
1k views

### What is Shannon's source entropy

Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication ...
170 views

### Issues related to entropy

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the ...
Background information According to Cover's text book on information theory Mutual Information is calculated as: $$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)})$$ ...