# Questions tagged [information-theory]

The tag has no usage guidance.

108 questions
Filter by
Sorted by
Tagged with
41 views

i'm currently working on a circuit transient simulation, and I'm wondering how the type of input excitation can affect the final result. In particular the focus is on impulse input vs step input, to ...
1 vote
184 views

### What is the intuition explaining the Shannon-Hartley theorem?

In the digital domain, the signal to noise ratio (SNR) may be interpreted loosely but usefully as the number of discrete or quantized levels one can transmit reliably. For instance, loosely speaking, ...
• 301
47 views

### What are the differences between GPS and GLONASS?

I'm studying the localization systems, especially the GPS one. I inquired about other systems and wanted to understand what are the main differences between GPS and GLONASS, why has the first type ...
1 vote
84 views

### Entropy Loss in Linear Filters

I've been trying to figure out the result for the entropy loss/entropy gain in linear systems derived in "The Mathematical Theory of Communication" by Claude Shannon. Claude Shannon states ...
34 views

### Proof that block of independent Gaussian symbols has higher mutual information than correlated Gaussian symbols in band- & power-limited AWGN channel

Given a discrete-time AWGN channel defined by: $$Y = X + N$$ where $N$ is a zero-mean Gaussian RV with $E[N^2] = P_N$ representing additive noise, $X$ is the transmitted symbol with power ...
• 2,142
285 views

### Shannon capacity limit & FEC comparisons

In this figure from Fundamentals of Digital Communications by Madhow , pp 328 there is a comparison of a rate 1/3 turbo code with the shannon limit for rate 1/3 BPSK: Trying to understand the limit ...
• 653
1 vote
106 views

### Soft-output decoder

I am trying to implement the soft-output for my soft-decoder, however, the resulting log-likelihood ratios are not corresponding to the corrected hard bits. The soft-output regards the LLR-values for ...
1 vote
719 views

### value of 0 log0 in entropy formula

Why is the value of $p_i\log(p_i)$ in entropy formula considered $0$ when $p_i =0$? I get that it is the limiting value, but does that mean $p$ is never equals to zero, but only tends to?
• 15
29 views

### Is there a way to describe the pattern of a signal as a numeric value?

I've been digging a little into signal processing and I just started a networks and telecommunications class at college. We saw there are different encode techniques Amplitude, Frequency and Phase ...
1 vote
49 views

### Secret Sharing & Information Theory

I'm reading a paper -- Common Randomness in Information Theory and Cryptograph-secret. And I do have trouble in understanding formula (2.6): $$\frac {1}{n} H(K)> H-\epsilon \tag{2.6}\label{2.6},$$...
• 11
353 views

1 vote
45 views

### Why the requirement of the GCD of the lengths of all circuits in the graph being one?

I am reading A Mathematical Theory of Communication. The second requirement of an ergodic process confuses me (emphasis mine): All the examples of artificial languages given above are ergodic. This ...
• 113
35 views

• 568
1 vote
419 views

### How to calculate the entropy of random normally distributed samples?

I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers. I'm trying to ...
• 392