# Questions tagged [information-theory]

The tag has no usage guidance.

100 questions
Filter by
Sorted by
Tagged with
4k views

### Relationship between entropy and SNR

In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
931 views

### Capacity of AWGN channel

I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is: $$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$ and it is ...
4k views

### What is the meaning of Mutual Information beyond the numerical calculation?

Beyond the raw equation for calculating mutual information, what does it mean in physical terms? For example: From Information Theory, we know that entropy is the smallest loss-less compression scheme ...
7k views

### What are some typical lossless compression ratios?

A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I ...
671 views

### Is concept of “bit” in computer programming similar to the concept of “bit” in information theory?

until today I knew that one bit is a variable, or a space in memory that can hold a value of either One (high) or Zero (low). This is the concept I learned from studying computer programming, ...
181 views

### Issues related to entropy

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the ...
2k views

### Why is mutual information symmetric but conditional entropy isn't?

Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information ...
704 views

### What is the meaning of channel capacity?

If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is? Is there a connection between the modulation kind and ...
170 views

### Does any error correction code still work in such situation?

I'm looking for a kind of error correction code or solution that can correct my codeword in this case: My message holds $k$ bits, and $2k$ bits codeword (rate is $1/2$) is produced by the generator ...
160 views

### Mutual Information, is this calculation correct or should I contact the author about fixing it?

Background information According to Cover's text book on information theory Mutual Information is calculated as: $$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)})$$ ...
369 views

### How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle

I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right)$$ Suppose I have a bitstream encoded as a signal that ...
133 views

213 views

### Difficulties in understanding mutual information concept

What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and ...
265 views

### Mutual information of $\infty$-PAM

I'm trying to compute using matlab the mutual information for an $\infty$-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck. I'm working with a real-valued ...
115 views

### Conceptual question on entropy and its relation to information

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-...
221 views

### Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?

A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
421 views

### Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation

I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
460 views

### Conceptual definition of entropy

What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about ...
81 views

### Determining Loss of Information by Taking Average (Mean of Signal)

So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am ...
3k views

678 views

### What is the effect of noise on Shannon entropy?

Since entropy is a measure of uncertainty or randomness, intuitively we would suppose that adding noise to an image would increase its entropy since we are now more uncertain about the information of ...
1k views

### What is Shannon's source entropy

Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication ...
559 views

### Can the Entropy Be Used as a Measure of the Contrast of an Image?

I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/...
1k views

### Choosing samples per symbol for modem pulse

One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem. The smaller the pulse width,...
Do you have any idea about how we can find the principal eigenvectors of an unknown matrix ${H}$? The elements of $H$ are unknown in general. If you are familiar with channel estimation procedure in ...