Questions tagged [information-theory]
The information-theory tag has no usage guidance.
99
questions
0
votes
0answers
10 views
Mutual information smaller on data subsets compared to entire data set
I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al [1].
Lets consider two features $X$ and $Y$, and the ...
2
votes
3answers
72 views
Determining Loss of Information by Taking Average (Mean of Signal)
So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am ...
3
votes
1answer
220 views
Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?
A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
0
votes
1answer
60 views
Proof that DFT does not require more than N points
I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
0
votes
0answers
14 views
About convergence of KL divergence: if the two probability distributions are type, does the law of large number work?
If I pick $N$ samples from $P_X$ and $P_Y$, they are two independent discrete distributions.
$X_1,X_2,\ldots,X_N$ are drawn i.i.d from $P_X$, and $Y_1,\ldots,Y_N$ are drawn i.i.d from $P_Y$.
I got $...
1
vote
1answer
20 views
Why the requirement of the GCD of the lengths of all circuits in the graph being one?
I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine):
All the examples of artificial languages given above are ergodic. This
...
0
votes
0answers
20 views
Channel Impulse Response is zero mean Gaussian random variable?
In the Paper "Key Generation From Wireless Channels" the channel estimation is given as:
$\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||S_B||^2}$,
$\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||...
4
votes
1answer
47 views
Can the Nyquist sample rate be extended to stochastic sampling?
It appears there are lots of questions here about Nyquist, and a few questions about stochastic sampling here. But I haven't found any that address quite what I'm after. This is the closest I've ...
0
votes
0answers
35 views
Information entropy of Poisson noise
I would like to calculate the information entropy of the channel with additive Poisson noise.
The model consists of an input signal, an optical image, an image acquisition device, and the output ...
-1
votes
1answer
44 views
Bandwidth and probability of continuous signal
I'm a little bit confused as to how to calculate the bandwidth of a signal. A question has me believing that it is correlated with the probability distribution. I am however not sure of this.
...
0
votes
1answer
37 views
Information theory vs system theory ( commented in Determining linearity, causality, memory, and time invariability from a picture of the signal)
I have seen this question :
At the comment asked about :
Is this "information theory" or "systems theory"? – The Photon
So i liked to know why this comment asked? Why the system theory vs ...
1
vote
1answer
290 views
Why is $x$ not considered a primitive polynomial while being considered an irreducible polynomial?
In "Shu Lin, Daniel J. Costello-Error Control Coding (2nd Edition), Prentice Hall 2004" it is given that in $GF(2)$, if $f(x)$ is an irreducible polynomial of degree $m$ it divides $x^n+1$ where $n = ...
0
votes
1answer
29 views
Example of Entropy and Channel Capacity Computation
Can you help me on verifying if this computation of entropy is correct and on understanding its meaning?
I am not sure of the result especially because it is equal to 0: it means that we cannot ...
14
votes
2answers
4k views
Relationship between entropy and SNR
In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
0
votes
1answer
32 views
Wireless communications outage
For communication systems, there is usually a target minimum rate $R$ bits per second. An outage occurs if the actual rate ever falls below $R$. Since $\frac{1}{2}\text{log}_2(1+\text{SNR})$, ...
0
votes
1answer
480 views
How do I measure the amount of information within an image?
I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
1
vote
1answer
44 views
Understanding Asymptotic Equipartition Property
I have some problems in understanding the precise meaning of the Asymptotic Equipartition Property, related to a large number n of independent and identically distributed random variables (X1, X2, ...,...
0
votes
1answer
42 views
What is the Asymptotic Equipartition Property (AEP)?
I am currently studying about Polar Codes in 5G standard and while reading my paper I found something called AEP which is required for channel coding. I surfed the web but didn't found a satisfying ...
0
votes
2answers
52 views
When channels are in a cascade and are independent is the capacity the sum?
Just wondering if two channels are independent then is the overall capacity, the sum of the individual capacities?
For example suppose I have two BEC where the probability of erasure is $\varepsilon$...
0
votes
2answers
46 views
Definition of Perfect Code (and example of Hamming Code)
Let's consider the following sentences about coding a message before transmitting it in a communication system (here you find the complete pdf)
First question: can you explain me these two ...
0
votes
1answer
33 views
Band-limited signal recovery with finite bit depth
The sampling theorem tells us that a signal with no frequencies above $f$ can be completely described by sampling it a rate of $2f$. However, the theorem makes no reference to quantization, and so I ...
1
vote
0answers
25 views
Pearson correlation coefficient vs mutual information
I recently computed both the pearson coefficient and the mutual information of two time series as well as the corresponding error bars, and what I find out in the end is a significant level of ...
2
votes
1answer
500 views
Can the Entropy Be Used as a Measure of the Contrast of an Image?
I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/...
1
vote
2answers
312 views
Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?
I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth.
$C = 2\times BW \times log_{2}(L)$ bits/sec
The way it was explained to me ...
0
votes
0answers
43 views
Capacity achieving codes for BPSK at low SNR
I don't know a lot about comm system design.
I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
0
votes
1answer
19 views
How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?
Anyone know where to find graphs showing how well it works vs other schemes? thanks
1
vote
1answer
91 views
Maximizing entropy on a channel
In Shannon's paper "A Mathematical Theory of Communication", in Theorem 8 he states:
Theorem 8: Let the system of constraints considered as channel have a capacity $C = \log W$. If we assign
$$p^{(s)}...
1
vote
2answers
285 views
Channel Capacity - is this only for discrete/digital signals?
The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission?
robert bristow-johnson
13.3k3 gold badges23 silver badges56 bronze badges
1
vote
1answer
138 views
How to Detect a Inhomogeneity Region in Image
I have an image that includes object and background. However, the object appears some inhomogeneity region due to illumination. My work is that how to detect inhomogeneity region. Which is feature can ...
0
votes
1answer
49 views
Relationship between spectral efficiency and PAR
Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
1
vote
1answer
152 views
Concatenated Deletion & Sticky Channel Capacity
Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
0
votes
1answer
605 views
White noise and entropy confusion about concepts
The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
1
vote
1answer
44 views
Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?
Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that
$$
b=\begin{cases}
0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\
1, &...
1
vote
0answers
26 views
Does Mutual information utilize spatial information to calculate image similarity?
To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ...
4
votes
1answer
266 views
Image registration / fusion optimal similarity metrics
I have a theoretical question about optimal similarity metrics for comparing data sets.
In reading this linked paper, pp. 488-489 [1], I read the following 2 interesting statements. On page 488:
...
0
votes
2answers
41 views
Rate in CDMA setup
I am working on a CDMA system with only 4 chips/symbol. I want to use $R=\frac{1}{2}log_2(1+SNR)$ chips/channel use to calculate what $SNR$ I require to successfully receive a given transmission. In ...
0
votes
1answer
34 views
Is there some literature to understand finite state ISI channel?
I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
2
votes
2answers
1k views
Choosing samples per symbol for modem pulse
One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem.
The smaller the pulse width,...
3
votes
1answer
404 views
Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation
I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
4
votes
3answers
354 views
How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle
I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right) $$
Suppose I have a bitstream encoded as a signal that ...
1
vote
2answers
385 views
Don't understand what is meant by signal dimension
I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...
0
votes
1answer
262 views
Mechanics of a matrix Interleaver
I am having trouble understanding exactly how the matrix interleaver. I have read the following page from MathWorks. In it, it gives the following example where "123456" is interleaved as "142536." ...
1
vote
0answers
40 views
Capacity region of 'symbol asynchronous' Gaussian multiple access channel
I was interested to learn what theoretical limits have been determined so far for the 'multiple access channel' with additive Gaussian noise. This brought me to the 1989 paper by Sergio Verdu, named "...
5
votes
2answers
690 views
What is the meaning of channel capacity?
If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is?
Is there a connection between the modulation kind and ...
1
vote
0answers
155 views
What is the difference between entropy and entropy rate and which one is important in information theory? [duplicate]
By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high ...
3
votes
2answers
568 views
Will the capacity of a channel becomes unbounded if i increase its signal-to-noise ratio $S/N$ without limit?
According to the Shannon-Hartley theorem the capacity $C$ of a channel which has a signal-to-noise ratio of $S/N$ and a bandwidth $B$ can be calculated to be $C = B \log_2 \left( 1 + \frac{S}{N}\right)...
1
vote
1answer
331 views
How to calculate the entropy of random normally distributed samples?
I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers.
I'm trying to ...
2
votes
3answers
2k views
What is the theoretical probability of error for OOK transmission?
I am trying to simulate an optical wireless communication channel which uses OOK modulation.
Looking at equation 14 in this literature, I found that:
For electrical SNR at the receiver
$$\textrm{...
0
votes
1answer
269 views
coding gain and shaping gain in SCMA
It is said that sparse code multiple access (SCMA) has the advantage of higher shaping and coding gain over LDS-CDMA due to the multidimensional constellation. What exactly is the shaping gain and how ...
1
vote
0answers
249 views
SNR for channel after equalization
Suppose I have a channel described by $h[n]$ or $H(z)$ over which I send a simple PAM2 signal. After the channel AWGN is added.
What is the best possible SNR with equalization?
As suggested in the ...
