The Stack Overflow podcast is back! Listen to an interview with our new CEO.

# Questions tagged [information-theory]

The tag has no usage guidance.

85 questions
Filter by
Sorted by
Tagged with
3k views

### Relationship between entropy and SNR

In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
658 views

### Capacity of AWGN channel

I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is: $$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$ and it is ...
4k views

### What is the meaning of Mutual Information beyond the numerical calculation?

Beyond the raw equation for calculating mutual information, what does it mean in physical terms? For example: From Information Theory, we know that entropy is the smallest loss-less compression scheme ...
7k views

### What are some typical lossless compression ratios?

A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I ...
613 views

### Is concept of “bit” in computer programming similar to the concept of “bit” in information theory?

until today I knew that one bit is a variable, or a space in memory that can hold a value of either One (high) or Zero (low). This is the concept I learned from studying computer programming, ...
170 views

### Issues related to entropy

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the ...
143 views

### Does any error correction code still work in such situation?

I'm looking for a kind of error correction code or solution that can correct my codeword in this case: My message holds $k$ bits, and $2k$ bits codeword (rate is $1/2$) is produced by the generator ...
2k views

### Why is mutual information symmetric but conditional entropy isn't?

Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information ...
151 views

### Mutual Information, is this calculation correct or should I contact the author about fixing it?

Background information According to Cover's text book on information theory Mutual Information is calculated as: $$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)})$$ ...
535 views

### What is the meaning of channel capacity?

If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is? Is there a connection between the modulation kind and ...
271 views

### How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle

I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right)$$ Suppose I have a bitstream encoded as a signal that ...
263 views

### Image registration / fusion optimal similarity metrics

I have a theoretical question about optimal similarity metrics for comparing data sets. In reading this linked paper, pp. 488-489 [1], I read the following 2 interesting statements. On page 488: ...
1k views

### Meaning of channel capacity in the AWGN channel

I understand the concept of channel capacity as the maximal rate of the channel code I can apply without making a mistake in the receiver, in that sense the capacity is between 0 and 1. What I don't ...
464 views

### What is the entropy for these cases?

This question stems from an article, "Entropy estimation of symbol sequences" download link where the abstract mentions the need for using symbols in information theory. Random chains of ...
192 views

### Difficulties in understanding mutual information concept

What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and ...
112 views

### Conceptual question on entropy and its relation to information

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-...
207 views

### Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?

A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
326 views

### Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation

I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
386 views

### Conceptual definition of entropy

What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about ...
256 views

### Is there a proof that equal bandwidths have equal information-carrying capacity?

Consider a continuous channel, band-limited to the $[f_{\text{min}},\,f_{\text{max}}]$ frequency window. A well-known statement says that, as far as the dependence on the frequencies, the information-...
225 views

61 views

### Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth. $C = 2\times BW \times log_{2}(L)$ bits/sec The way it was explained to me ...
42 views

185 views

### Don't understand what is meant by signal dimension

I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...
252 views

### How to calculate the entropy of random normally distributed samples?

I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers. I'm trying to ...
149 views

### Why is bandwidth always limited in a real (physical) channel?

I'm talking about a continuous analog channel. Why can't it support infinite bandwidth? Is there a physics reasons for it say for electrical signals?
301 views

### Source and channel encoding

In information theory, there are two concepts that are widely used: source encoding and channel encoding. I'm afraid I may not be capable of expressing my doubts clearly, but the thing is that I don'...
59 views

### Discrete entropies

I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it. I am given the following: $X$ is a uniformly distributed random ...
208 views