Skip to main content

Questions tagged [information-theory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

What is the correct interpretation of the capacity formula with modulation?

Someone sent me the image below, and I think I'm confused as well. Is the second formula (modulation) correct? What does it mean? From the symbol, I think it represents capacity, but it doesn't make ...
internet's user avatar
  • 131
0 votes
0 answers
28 views

Peak prediction

In my block, I am changing the gain of Amplifier (PGA) in step of +/- 1dB depending on the control voltage derived from the AGC (Automatic Gain Control). But can I change the gain of PGA in one shot ...
JHKKHJ's user avatar
  • 11
2 votes
2 answers
105 views

What exactly is transmitted in the channel in lossy compression?

Consider the problem of lossily compressing a memoryless binary source $X \sim \operatorname{Bernoulli}\left(\frac{1}{2}\right)$ at target Hamming distortion $D < \frac{1}{2}$. According to the ...
nalzok's user avatar
  • 125
4 votes
0 answers
96 views

Distortion-Rate function for standard Gaussian signals quantized with a uniform alphabet

Let's say $X \sim N(0, 1)$ is a standard Gaussian input signal and $Y \sim U\{y_1, y_2, \cdots, y_{16}\}$ follows a discrete uniform distribution (in fact, $\{y_1, y_2, \cdots, y_{16}\}$ is a ...
nalzok's user avatar
  • 125
2 votes
0 answers
21 views

How to prove the mutual infomation expression in paper "Optimized Degree Distributions for Binary and Non-Binary LDPC Codes in Flash Memory"

Recently I met a question when I read the paper Optimized Degree Distributions for Binary and Non-Binary LDPC Codes in Flash Memory and Soft Information for LDPC Decoding in Flash: Mutual-Information ...
Milin's user avatar
  • 141
1 vote
0 answers
31 views

Applying Blahut-Arimoto Algorithm for AWGN Channels Capacity Calculation with Cost

I am exploring the Blahut-Arimoto algorithm (or any numerical method) for channel capacity calculations, particularly for continuous channels. I understand its application in discrete scenarios, but I ...
Alireza Ghazavi's user avatar
1 vote
1 answer
114 views

How to calculate the following mutual infomation between $X$ and $Y$ in matlab or python?

Recently I meet the following question: Let $X\in {\mu_1,\mu_2,\cdots,\mu_m}$, $Y \in {R_1,R_2,\cdots,R_n}$, where $\mu_i, i = 1,2,\cdots,m$ is means of $m$ gauss probablity density function, $Y$ is ...
Milin's user avatar
  • 141
1 vote
1 answer
143 views

Lower bound on information or entropy?

Inspired by this question: Does a simple photograph contain more information than a complex painting? Say that I have a discrete representation of some physical object - eg an image file. What is the ...
Knut Inge's user avatar
  • 3,462
2 votes
1 answer
1k views

How to code Rician fading channel gains from k-factor?

I know that MATLAB provides some built-in functions but I am not interested in those. I want to know how can we code to obtain the Rician fading channel gains from normal or Gaussian distributions. ...
ZZ1's user avatar
  • 23
0 votes
1 answer
586 views

Advantages/disadvantages of impulse excitation vs step excitation

i'm currently working on a circuit transient simulation, and I'm wondering how the type of input excitation can affect the final result. In particular the focus is on impulse input vs step input, to ...
Spaetzle22's user avatar
1 vote
2 answers
2k views

What is the intuition explaining the Shannon-Hartley theorem?

In the digital domain, the signal to noise ratio (SNR) may be interpreted loosely but usefully as the number of discrete or quantized levels one can transmit reliably. For instance, loosely speaking, ...
kbakshi314's user avatar
0 votes
0 answers
61 views

What are the differences between GPS and GLONASS?

I'm studying the localization systems, especially the GPS one. I inquired about other systems and wanted to understand what are the main differences between GPS and GLONASS, why has the first type ...
Riccardo Barbo's user avatar
1 vote
0 answers
95 views

Entropy Loss in Linear Filters

I've been trying to figure out the result for the entropy loss/entropy gain in linear systems derived in "The Mathematical Theory of Communication" by Claude Shannon. Claude Shannon states ...
GuntherOnFire's user avatar
2 votes
0 answers
62 views

Proof that block of independent Gaussian symbols has higher mutual information than correlated Gaussian symbols in band- & power-limited AWGN channel

Given a discrete-time AWGN channel defined by: $$ Y = X + N $$ where $N$ is a zero-mean Gaussian RV with $E[N^2] = P_N$ representing additive noise, $X$ is the transmitted symbol with power ...
Robert L.'s user avatar
  • 2,222
5 votes
2 answers
997 views

Shannon capacity limit & FEC comparisons

In this figure from Fundamentals of Digital Communications by Madhow , pp 328 there is a comparison of a rate 1/3 turbo code with the shannon limit for rate 1/3 BPSK: Trying to understand the limit ...
user67081's user avatar
  • 733
1 vote
0 answers
120 views

Soft-output decoder

I am trying to implement the soft-output for my soft-decoder, however, the resulting log-likelihood ratios are not corresponding to the corrected hard bits. The soft-output regards the LLR-values for ...
Cooltafel's user avatar
1 vote
2 answers
3k views

value of 0 log0 in entropy formula

Why is the value of $p_i\log(p_i)$ in entropy formula considered $0$ when $p_i =0$? I get that it is the limiting value, but does that mean $p$ is never equals to zero, but only tends to?
Elin Das's user avatar
0 votes
0 answers
61 views

Is there a way to describe the pattern of a signal as a numeric value?

I've been digging a little into signal processing and I just started a networks and telecommunications class at college. We saw there are different encode techniques Amplitude, Frequency and Phase ...
Armando Canales Lima's user avatar
1 vote
0 answers
52 views

Secret Sharing & Information Theory

I'm reading a paper -- Common Randomness in Information Theory and Cryptograph-secret. And I do have trouble in understanding formula (2.6): $$ \frac {1}{n} H(K)> H-\epsilon \tag{2.6}\label{2.6}, $$...
Chen Niu's user avatar
4 votes
3 answers
563 views

Encoding sequence of unfair coin flips

The question: Consider transmitting the results of $1000$ flips of an unfair coin where the probability of heads is given by $p_H$. The information contained in an unfair coin flip can be computed: $...
Certusic's user avatar
2 votes
3 answers
514 views

Determining Loss of Information by Taking Average (Mean of Signal)

So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am ...
GENIVI-LEARNER's user avatar
0 votes
0 answers
144 views

About convergence of KL divergence: if the two probability distributions are type, does the law of large number work?

If I pick $N$ samples from $P_X$ and $P_Y$, they are two independent discrete distributions. $X_1,X_2,\ldots,X_N$ are drawn i.i.d from $P_X$, and $Y_1,\ldots,Y_N$ are drawn i.i.d from $P_Y$. I got $...
jerry's user avatar
  • 1
1 vote
1 answer
70 views

Why the requirement of the GCD of the lengths of all circuits in the graph being one?

I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine): All the examples of artificial languages given above are ergodic. This ...
nalzok's user avatar
  • 125
0 votes
0 answers
40 views

Channel Impulse Response is zero mean Gaussian random variable?

In the Paper "Key Generation From Wireless Channels" the channel estimation is given as: $\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||S_B||^2}$, $\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||...
user674907's user avatar
4 votes
1 answer
105 views

Can the Nyquist sample rate be extended to stochastic sampling?

It appears there are lots of questions here about Nyquist, and a few questions about stochastic sampling here. But I haven't found any that address quite what I'm after. This is the closest I've ...
kdbanman's user avatar
  • 143
-1 votes
1 answer
133 views

Bandwidth and probability of continuous signal

I'm a little bit confused as to how to calculate the bandwidth of a signal. A question has me believing that it is correlated with the probability distribution. I am however not sure of this. ...
Augustin Roegiers's user avatar
0 votes
1 answer
48 views

Information theory vs system theory ( commented in Determining linearity, causality, memory, and time invariability from a picture of the signal)

I have seen this question : At the comment asked about : Is this "information theory" or "systems theory"? – The Photon So i liked to know why this comment asked? Why the system theory vs ...
Soheil Paper's user avatar
0 votes
1 answer
72 views

Wireless communications outage

For communication systems, there is usually a target minimum rate $R$ bits per second. An outage occurs if the actual rate ever falls below $R$. Since $\frac{1}{2}\text{log}_2(1+\text{SNR})$, ...
Engineer's user avatar
  • 3,042
0 votes
1 answer
278 views

Example of Entropy and Channel Capacity Computation

Can you help me on verifying if this computation of entropy is correct and on understanding its meaning? I am not sure of the result especially because it is equal to 0: it means that we cannot ...
Kinka-Byo's user avatar
  • 334
3 votes
1 answer
127 views

Understanding Asymptotic Equipartition Property

I have some problems in understanding the precise meaning of the Asymptotic Equipartition Property, related to a large number n of independent and identically distributed random variables (X1, X2, ...,...
Kinka-Byo's user avatar
  • 334
0 votes
2 answers
523 views

When channels are in a cascade and are independent is the capacity the sum?

Just wondering if two channels are independent then is the overall capacity, the sum of the individual capacities? For example suppose I have two BEC where the probability of erasure is $\varepsilon$...
Dead_Ling0's user avatar
0 votes
2 answers
1k views

Definition of Perfect Code (and example of Hamming Code)

Let's consider the following sentences about coding a message before transmitting it in a communication system (here you find the complete pdf) First question: can you explain me these two ...
Kinka-Byo's user avatar
  • 334
0 votes
1 answer
62 views

Band-limited signal recovery with finite bit depth

The sampling theorem tells us that a signal with no frequencies above $f$ can be completely described by sampling it a rate of $2f$. However, the theorem makes no reference to quantization, and so I ...
daturkel's user avatar
1 vote
0 answers
104 views

Pearson correlation coefficient vs mutual information

I recently computed both the pearson coefficient and the mutual information of two time series as well as the corresponding error bars, and what I find out in the end is a significant level of ...
Johncowk's user avatar
  • 141
2 votes
1 answer
3k views

Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth. $C = 2\times BW \times log_{2}(L)$ bits/sec The way it was explained to me ...
Frank's user avatar
  • 169
1 vote
0 answers
304 views

Capacity achieving codes for BPSK at low SNR

I don't know a lot about comm system design. I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
rhz's user avatar
  • 489
1 vote
1 answer
43 views

How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?

Anyone know where to find graphs showing how well it works vs other schemes? thanks
Christian Chapman's user avatar
0 votes
2 answers
3k views

How do I measure the amount of information within an image?

I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
Mark.F's user avatar
  • 113
-1 votes
1 answer
266 views

Proof that DFT does not require more than N points

I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
MarcinKonowalczyk's user avatar
1 vote
1 answer
103 views

Maximizing entropy on a channel

In Shannon's paper "A Mathematical Theory of Communication", in Theorem 8 he states: Theorem 8: Let the system of constraints considered as channel have a capacity $C = \log W$. If we assign $$p^{(s)}...
Giorgian Borca-Tasciuc's user avatar
2 votes
1 answer
2k views

Can the Entropy Be Used as a Measure of the Contrast of an Image?

I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/...
Shadr's user avatar
  • 123
1 vote
1 answer
112 views

What is the Asymptotic Equipartition Property (AEP)?

I am currently studying about Polar Codes in 5G standard and while reading my paper I found something called AEP which is required for channel coding. I surfed the web but didn't found a satisfying ...
Himanshu Sharma's user avatar
0 votes
1 answer
125 views

Relationship between spectral efficiency and PAR

Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
divB's user avatar
  • 473
1 vote
1 answer
188 views

Concatenated Deletion & Sticky Channel Capacity

Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
William Hird's user avatar
0 votes
1 answer
2k views

White noise and entropy confusion about concepts

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
SKM's user avatar
  • 621
2 votes
1 answer
54 views

Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?

Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that $$ b=\begin{cases} 0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\ 1, &...
Lod's user avatar
  • 167
1 vote
0 answers
35 views

Does Mutual information utilize spatial information to calculate image similarity?

To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ...
Mohammad nagdawi's user avatar
0 votes
2 answers
83 views

Rate in CDMA setup

I am working on a CDMA system with only 4 chips/symbol. I want to use $R=\frac{1}{2}log_2(1+SNR)$ chips/channel use to calculate what $SNR$ I require to successfully receive a given transmission. In ...
Engineer's user avatar
  • 3,042
0 votes
1 answer
38 views

Is there some literature to understand finite state ISI channel?

I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
Userhanu's user avatar
  • 181
3 votes
2 answers
2k views

Choosing samples per symbol for modem pulse

One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem. The smaller the pulse width,...
user827822's user avatar