# Questions tagged [information-theory]

The tag has no usage guidance.

117 questions
Filter by
Sorted by
Tagged with
29 views

### What is the correct interpretation of the capacity formula with modulation?

Someone sent me the image below, and I think I'm confused as well. Is the second formula (modulation) correct? What does it mean? From the symbol, I think it represents capacity, but it doesn't make ...
• 131
28 views

### Peak prediction

In my block, I am changing the gain of Amplifier (PGA) in step of +/- 1dB depending on the control voltage derived from the AGC (Automatic Gain Control). But can I change the gain of PGA in one shot ...
• 11
105 views

### What exactly is transmitted in the channel in lossy compression?

Consider the problem of lossily compressing a memoryless binary source $X \sim \operatorname{Bernoulli}\left(\frac{1}{2}\right)$ at target Hamming distortion $D < \frac{1}{2}$. According to the ...
• 125
96 views

### Distortion-Rate function for standard Gaussian signals quantized with a uniform alphabet

Let's say $X \sim N(0, 1)$ is a standard Gaussian input signal and $Y \sim U\{y_1, y_2, \cdots, y_{16}\}$ follows a discrete uniform distribution (in fact, $\{y_1, y_2, \cdots, y_{16}\}$ is a ...
• 125
21 views

### How to prove the mutual infomation expression in paper "Optimized Degree Distributions for Binary and Non-Binary LDPC Codes in Flash Memory"

Recently I met a question when I read the paper Optimized Degree Distributions for Binary and Non-Binary LDPC Codes in Flash Memory and Soft Information for LDPC Decoding in Flash: Mutual-Information ...
• 141
1 vote
31 views

### Applying Blahut-Arimoto Algorithm for AWGN Channels Capacity Calculation with Cost

I am exploring the Blahut-Arimoto algorithm (or any numerical method) for channel capacity calculations, particularly for continuous channels. I understand its application in discrete scenarios, but I ...
1 vote
114 views

### How to calculate the following mutual infomation between $X$ and $Y$ in matlab or python?

Recently I meet the following question: Let $X\in {\mu_1,\mu_2,\cdots,\mu_m}$, $Y \in {R_1,R_2,\cdots,R_n}$, where $\mu_i, i = 1,2,\cdots,m$ is means of $m$ gauss probablity density function, $Y$ is ...
• 141
1 vote
143 views

### Lower bound on information or entropy?

Inspired by this question: Does a simple photograph contain more information than a complex painting? Say that I have a discrete representation of some physical object - eg an image file. What is the ...
• 3,462
1k views

### How to code Rician fading channel gains from k-factor?

I know that MATLAB provides some built-in functions but I am not interested in those. I want to know how can we code to obtain the Rician fading channel gains from normal or Gaussian distributions. ...
• 23
586 views

i'm currently working on a circuit transient simulation, and I'm wondering how the type of input excitation can affect the final result. In particular the focus is on impulse input vs step input, to ...
1 vote
2k views

### What is the intuition explaining the Shannon-Hartley theorem?

In the digital domain, the signal to noise ratio (SNR) may be interpreted loosely but usefully as the number of discrete or quantized levels one can transmit reliably. For instance, loosely speaking, ...
• 331
61 views

### What are the differences between GPS and GLONASS?

I'm studying the localization systems, especially the GPS one. I inquired about other systems and wanted to understand what are the main differences between GPS and GLONASS, why has the first type ...
1 vote
95 views

### Entropy Loss in Linear Filters

I've been trying to figure out the result for the entropy loss/entropy gain in linear systems derived in "The Mathematical Theory of Communication" by Claude Shannon. Claude Shannon states ...
62 views

### Proof that block of independent Gaussian symbols has higher mutual information than correlated Gaussian symbols in band- & power-limited AWGN channel

Given a discrete-time AWGN channel defined by: $$Y = X + N$$ where $N$ is a zero-mean Gaussian RV with $E[N^2] = P_N$ representing additive noise, $X$ is the transmitted symbol with power ...
• 2,222
997 views

### Shannon capacity limit & FEC comparisons

In this figure from Fundamentals of Digital Communications by Madhow , pp 328 there is a comparison of a rate 1/3 turbo code with the shannon limit for rate 1/3 BPSK: Trying to understand the limit ...
• 733
1 vote
120 views

### Soft-output decoder

I am trying to implement the soft-output for my soft-decoder, however, the resulting log-likelihood ratios are not corresponding to the corrected hard bits. The soft-output regards the LLR-values for ...
1 vote
3k views

### value of 0 log0 in entropy formula

Why is the value of $p_i\log(p_i)$ in entropy formula considered $0$ when $p_i =0$? I get that it is the limiting value, but does that mean $p$ is never equals to zero, but only tends to?
• 37
61 views

### Is there a way to describe the pattern of a signal as a numeric value?

I've been digging a little into signal processing and I just started a networks and telecommunications class at college. We saw there are different encode techniques Amplitude, Frequency and Phase ...
1 vote
52 views

### Secret Sharing & Information Theory

I'm reading a paper -- Common Randomness in Information Theory and Cryptograph-secret. And I do have trouble in understanding formula (2.6): $$\frac {1}{n} H(K)> H-\epsilon \tag{2.6}\label{2.6},$$...
• 11
563 views

1 vote
70 views

### Why the requirement of the GCD of the lengths of all circuits in the graph being one?

I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine): All the examples of artificial languages given above are ergodic. This ...
• 125
40 views

• 3,042
38 views

### Is there some literature to understand finite state ISI channel?

I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
• 181