# Questions tagged [information-theory]

The tag has no usage guidance.

112 questions
Filter by
Sorted by
Tagged with
1 vote
24 views

### Applying Blahut-Arimoto Algorithm for AWGN Channels Capacity Calculation with Cost

I am exploring the Blahut-Arimoto algorithm (or any numerical method) for channel capacity calculations, particularly for continuous channels. I understand its application in discrete scenarios, but I ...
1 vote
91 views

### How to calculate the following mutual infomation between $X$ and $Y$ in matlab or python?

Recently I meet the following question: Let $X\in {\mu_1,\mu_2,\cdots,\mu_m}$, $Y \in {R_1,R_2,\cdots,R_n}$, where $\mu_i, i = 1,2,\cdots,m$ is means of $m$ gauss probablity density function, $Y$ is ...
1 vote
105 views

### Lower bound on information or entropy?

Inspired by this question: Does a simple photograph contain more information than a complex painting? Say that I have a discrete representation of some physical object - eg an image file. What is the ...
651 views

### How to code Rician fading channel gains from k-factor?

I know that MATLAB provides some built-in functions but I am not interested in those. I want to know how can we code to obtain the Rician fading channel gains from normal or Gaussian distributions. ...
454 views

i'm currently working on a circuit transient simulation, and I'm wondering how the type of input excitation can affect the final result. In particular the focus is on impulse input vs step input, to ...
1 vote
1k views

### What is the intuition explaining the Shannon-Hartley theorem?

In the digital domain, the signal to noise ratio (SNR) may be interpreted loosely but usefully as the number of discrete or quantized levels one can transmit reliably. For instance, loosely speaking, ...
53 views

### What are the differences between GPS and GLONASS?

I'm studying the localization systems, especially the GPS one. I inquired about other systems and wanted to understand what are the main differences between GPS and GLONASS, why has the first type ...
1 vote
91 views

### Entropy Loss in Linear Filters

I've been trying to figure out the result for the entropy loss/entropy gain in linear systems derived in "The Mathematical Theory of Communication" by Claude Shannon. Claude Shannon states ...
48 views

### Proof that block of independent Gaussian symbols has higher mutual information than correlated Gaussian symbols in band- & power-limited AWGN channel

Given a discrete-time AWGN channel defined by: $$Y = X + N$$ where $N$ is a zero-mean Gaussian RV with $E[N^2] = P_N$ representing additive noise, $X$ is the transmitted symbol with power ...
794 views

### Shannon capacity limit & FEC comparisons

In this figure from Fundamentals of Digital Communications by Madhow , pp 328 there is a comparison of a rate 1/3 turbo code with the shannon limit for rate 1/3 BPSK: Trying to understand the limit ...
1 vote
117 views

### Soft-output decoder

I am trying to implement the soft-output for my soft-decoder, however, the resulting log-likelihood ratios are not corresponding to the corrected hard bits. The soft-output regards the LLR-values for ...
1 vote
2k views

### value of 0 log0 in entropy formula

Why is the value of $p_i\log(p_i)$ in entropy formula considered $0$ when $p_i =0$? I get that it is the limiting value, but does that mean $p$ is never equals to zero, but only tends to?
45 views

### Is there a way to describe the pattern of a signal as a numeric value?

I've been digging a little into signal processing and I just started a networks and telecommunications class at college. We saw there are different encode techniques Amplitude, Frequency and Phase ...
1 vote
51 views

### Secret Sharing & Information Theory

I'm reading a paper -- Common Randomness in Information Theory and Cryptograph-secret. And I do have trouble in understanding formula (2.6): $$\frac {1}{n} H(K)> H-\epsilon \tag{2.6}\label{2.6},$$...
518 views

1 vote
65 views

### Why the requirement of the GCD of the lengths of all circuits in the graph being one?

I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine): All the examples of artificial languages given above are ergodic. This ...
39 views

802 views

### Channel Capacity - is this only for discrete/digital signals?

The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission?
1 vote
1k views

### Don't understand what is meant by signal dimension

I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...