Questions tagged [information-theory]
The information-theory tag has no usage guidance.
100
questions
14
votes
2answers
4k views
Relationship between entropy and SNR
In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
9
votes
3answers
931 views
Capacity of AWGN channel
I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is:
$$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$
and it is ...
8
votes
2answers
4k views
What is the meaning of Mutual Information beyond the numerical calculation?
Beyond the raw equation for calculating mutual information, what does it mean in physical terms?
For example:
From Information Theory, we know that entropy is the smallest loss-less compression scheme ...
8
votes
2answers
7k views
What are some typical lossless compression ratios?
A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I ...
7
votes
3answers
671 views
Is concept of “bit” in computer programming similar to the concept of “bit” in information theory?
until today I knew that one bit is a variable, or a space in memory that can hold a value of either One (high) or Zero (low). This is the concept I learned from studying computer programming, ...
7
votes
1answer
181 views
Issues related to entropy
I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the ...
6
votes
1answer
2k views
Why is mutual information symmetric but conditional entropy isn't?
Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information ...
5
votes
2answers
704 views
What is the meaning of channel capacity?
If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is?
Is there a connection between the modulation kind and ...
5
votes
1answer
170 views
Does any error correction code still work in such situation?
I'm looking for a kind of error correction code or solution that can correct my codeword in this case:
My message holds $k$ bits, and $2k$ bits codeword (rate is $1/2$) is produced by the generator ...
5
votes
1answer
160 views
Mutual Information, is this calculation correct or should I contact the author about fixing it?
Background information
According to Cover's text book on information theory Mutual Information is calculated as:
$$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)})
$$
...
4
votes
3answers
369 views
How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle
I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right) $$
Suppose I have a bitstream encoded as a signal that ...
4
votes
3answers
133 views
Encoding sequence of unfair coin flips
The question: Consider transmitting the results of $1000$ flips of an unfair coin where the probability of heads is given by $p_H$. The information contained in an unfair coin flip can be computed:
$...
4
votes
1answer
47 views
Can the Nyquist sample rate be extended to stochastic sampling?
It appears there are lots of questions here about Nyquist, and a few questions about stochastic sampling here. But I haven't found any that address quite what I'm after. This is the closest I've ...
4
votes
1answer
266 views
Image registration / fusion optimal similarity metrics
I have a theoretical question about optimal similarity metrics for comparing data sets.
In reading this linked paper, pp. 488-489 [1], I read the following 2 interesting statements. On page 488:
...
3
votes
2answers
1k views
Meaning of channel capacity in the AWGN channel
I understand the concept of channel capacity as the maximal rate of the channel code I can apply without making a mistake in the receiver, in that sense the capacity is between 0 and 1.
What I don't ...
3
votes
2answers
514 views
What is the entropy for these cases?
This question stems from an article,
"Entropy estimation of symbol sequences" download link
where the abstract mentions the need for using symbols in information theory.
Random chains of ...
3
votes
2answers
597 views
Will the capacity of a channel becomes unbounded if i increase its signal-to-noise ratio $S/N$ without limit?
According to the Shannon-Hartley theorem the capacity $C$ of a channel which has a signal-to-noise ratio of $S/N$ and a bandwidth $B$ can be calculated to be $C = B \log_2 \left( 1 + \frac{S}{N}\right)...
3
votes
1answer
213 views
Difficulties in understanding mutual information concept
What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and ...
3
votes
1answer
265 views
Mutual information of $ \infty $-PAM
I'm trying to compute using matlab the mutual information for an $ \infty $-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck.
I'm working with a real-valued ...
3
votes
1answer
115 views
Conceptual question on entropy and its relation to information
Learning
Informative
Statistics:
A
Nonparametric
Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-...
3
votes
1answer
221 views
Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?
A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
3
votes
1answer
421 views
Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation
I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
2
votes
2answers
460 views
Conceptual definition of entropy
What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about ...
2
votes
3answers
81 views
Determining Loss of Information by Taking Average (Mean of Signal)
So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am ...
2
votes
3answers
3k views
What is the theoretical probability of error for OOK transmission?
I am trying to simulate an optical wireless communication channel which uses OOK modulation.
Looking at equation 14 in this literature, I found that:
For electrical SNR at the receiver
$$\textrm{...
2
votes
3answers
288 views
Is there a proof that equal bandwidths have equal information-carrying capacity?
Consider a continuous channel, band-limited to the $[f_{\text{min}},\,f_{\text{max}}]$ frequency window. A well-known statement says that, as far as the dependence on the frequencies, the information-...
2
votes
2answers
254 views
Entropy Of A Symbol
Does the entropy of a symbol represent the most optimal average number of bits that can be used to represent a symbol? For example take the example of tossing a coin:
$$
H_{source} = -p_{head} ...
2
votes
1answer
678 views
What is the effect of noise on Shannon entropy?
Since entropy is a measure of uncertainty or randomness, intuitively we would suppose that adding noise to an image would increase its entropy since we are now more uncertain about the information of ...
2
votes
1answer
1k views
What is Shannon's source entropy
Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication ...
2
votes
1answer
559 views
Can the Entropy Be Used as a Measure of the Contrast of an Image?
I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/...
2
votes
2answers
1k views
Choosing samples per symbol for modem pulse
One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem.
The smaller the pulse width,...
2
votes
1answer
307 views
Dominant eigenvectors of an unknown matrix
Do you have any idea about how we can find the principal eigenvectors of an unknown matrix ${H}$?
The elements of $H$ are unknown in general. If you are familiar with channel estimation procedure in ...
2
votes
1answer
252 views
Effect of noise and entropy
Relation of Entropy and SNR : Based on this question and answer, I had another question that struck me and I am curious to know, if somebody can shed some light, on the following situation: $y= ...
2
votes
0answers
54 views
Is there a theoretical limit for how much information hiding can be done through watermarking?
There is a classic channel-capacity notion that exist based on the work of Shannon that gives gives measure of amount of information that can be passed through the channel given its properties.
In ...
1
vote
2answers
379 views
Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?
I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth.
$C = 2\times BW \times log_{2}(L)$ bits/sec
The way it was explained to me ...
1
vote
1answer
44 views
Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?
Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that
$$
b=\begin{cases}
0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\
1, &...
1
vote
2answers
301 views
Channel Capacity - is this only for discrete/digital signals?
The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission?
1
vote
2answers
632 views
Confusion about shannon's entropy of the grayscale image after lossy compression
I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB ...
1
vote
1answer
153 views
Concatenated Deletion & Sticky Channel Capacity
Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
1
vote
1answer
91 views
Why are some LDPC codes more immune to noise than others?
When one analyzes an LDPC code, it is usual to plot the BER versus the $E_b / N_0$. My questions are:
Why are some LDPC codes better than others in terms of noise immunity? After all, they are just ...
1
vote
1answer
47 views
Can channel capcity be explained in simple terms? [closed]
Can channel capacity be explained in simple terms - without mutual information and such probabilistic concepts? In its most general form, what are all the parameters that it depends on? What is its ...
1
vote
1answer
138 views
How to Detect a Inhomogeneity Region in Image
I have an image that includes object and background. However, the object appears some inhomogeneity region due to illumination. My work is that how to detect inhomogeneity region. Which is feature can ...
1
vote
1answer
21 views
Why the requirement of the GCD of the lengths of all circuits in the graph being one?
I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine):
All the examples of artificial languages given above are ergodic. This
...
1
vote
1answer
91 views
Maximizing entropy on a channel
In Shannon's paper "A Mathematical Theory of Communication", in Theorem 8 he states:
Theorem 8: Let the system of constraints considered as channel have a capacity $C = \log W$. If we assign
$$p^{(s)}...
1
vote
2answers
425 views
Don't understand what is meant by signal dimension
I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...
1
vote
1answer
334 views
How to calculate the entropy of random normally distributed samples?
I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers.
I'm trying to ...
1
vote
2answers
207 views
Why is bandwidth always limited in a real (physical) channel?
I'm talking about a continuous analog channel.
Why can't it support infinite bandwidth?
Is there a physics reasons for it say for electrical signals?
1
vote
1answer
343 views
Source and channel encoding
In information theory, there are two concepts that are widely used: source encoding and channel encoding.
I'm afraid I may not be capable of expressing my doubts clearly, but the thing is that I don'...
1
vote
1answer
59 views
Discrete entropies
I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it.
I am given the following:
$X$ is a uniformly distributed random ...
1
vote
1answer
303 views
Why is $x$ not considered a primitive polynomial while being considered an irreducible polynomial?
In "Shu Lin, Daniel J. Costello-Error Control Coding (2nd Edition), Prentice Hall 2004" it is given that in $GF(2)$, if $f(x)$ is an irreducible polynomial of degree $m$ it divides $x^n+1$ where $n = ...
