# Questions tagged [information-theory]

The tag has no usage guidance.

98 questions
Filter by
Sorted by
Tagged with
1k views

### Entropy : do we prefer higher or lower entropy?

The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ...
355 views

### How to prove relationship between mutual information and differential entropy?

I need to prove that mutual information given by $$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$ is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am ...
333 views

54 views

### Entropy of a dynamical system and source code length

The entropy of a source $H(S)$, gives you the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the ...
245 views

201 views

### Difficulties in understanding mutual information concept

What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and ...
113 views

### Conceptual question on entropy and its relation to information

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-...
136 views

### How to Detect a Inhomogeneity Region in Image

I have an image that includes object and background. However, the object appears some inhomogeneity region due to illumination. My work is that how to detect inhomogeneity region. Which is feature can ...
560 views

### Confusion about shannon's entropy of the grayscale image after lossy compression

I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB ...
95 views

### Entropy when having symbols with same probability

I am trying to work out this problem: Assume that we are transmitting data from a dictionary of 15 symbols. Let the probability of sending one symbol be the same as that of sending any other symbol....
381 views

### Extracting features from a signal: a 21st century approach?

I have a discrete signal of random length (let's say between 500 and 800 in time). This signal has an important information which I want to collect and to put it together in a feature vector to train ...
44 views

### Capacity computation in Costa sheme

I can not understand the following sentences, could you please anyone guide me ? "Comparing with the embedding in audio signals, the usable bandwidth is greatly reduced, from 22.05 kHz to 4 kHz. ...
112 views

### Transmission Line Information Loss

Suppose we have a communication channel that has a bit-error rate of 50% and the bits are flipped at random such that the receiver has no efficient way to tell which bits have been flipped and there ...
3k views

### Relationship between entropy and SNR

In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
1k views

### Meaning of channel capacity in the AWGN channel

I understand the concept of channel capacity as the maximal rate of the channel code I can apply without making a mistake in the receiver, in that sense the capacity is between 0 and 1. What I don't ...
421 views

### Conceptual definition of entropy

What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about ...
2k views

### Why is mutual information symmetric but conditional entropy isn't?

Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information ...
368 views

### Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation

I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
96 views

### transmitting data using phase modulation of two sine waves passed through distortion

I have two sine signals with frequencies f1 and f2, and phase p1 and p2. I modulate the phase of these signals to transmit two symbols simultaneously, chosen from the set [0,1,2,3]. For example, I ...