# Questions tagged [information-theory]

The tag has no usage guidance.

93 questions
Filter by
Sorted by
Tagged with
28 views

### Wireless communications outage

For communication systems, there is usually a target minimum rate $R$ bits per second. An outage occurs if the actual rate ever falls below $R$. Since $\frac{1}{2}\text{log}_2(1+\text{SNR})$, ...
19 views

### Example of Entropy and Channel Capacity Computation

Can you help me on verifying if this computation of entropy is correct and on understanding its meaning? I am not sure of the result especially because it is equal to 0: it means that we cannot ...
33 views

### Understanding Asymptotic Equipartition Property

I have some problems in understanding the precise meaning of the Asymptotic Equipartition Property, related to a large number n of independent and identically distributed random variables (X1, X2, ...,...
41 views

### When channels are in a cascade and are independent is the capacity the sum?

Just wondering if two channels are independent then is the overall capacity, the sum of the individual capacities? For example suppose I have two BEC where the probability of erasure is $\varepsilon$...
41 views

### Definition of Perfect Code (and example of Hamming Code)

Let's consider the following sentences about coding a message before transmitting it in a communication system (here you find the complete pdf) First question: can you explain me these two ...
28 views

### Band-limited signal recovery with finite bit depth

The sampling theorem tells us that a signal with no frequencies above $f$ can be completely described by sampling it a rate of $2f$. However, the theorem makes no reference to quantization, and so I ...
12 views

### Pearson correlation coefficient vs mutual information

I recently computed both the pearson coefficient and the mutual information of two time series as well as the corresponding error bars, and what I find out in the end is a significant level of ...
108 views

### Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth. $C = 2\times BW \times log_{2}(L)$ bits/sec The way it was explained to me ...
34 views

### Capacity achieving codes for BPSK at low SNR

I don't know a lot about comm system design. I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
16 views

### How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?

Anyone know where to find graphs showing how well it works vs other schemes? thanks
156 views

### How do I measure the amount of information within an image?

I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
46 views

### Proof that DFT does not require more than N points

I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
89 views

20 views

### Punctured Turbo Code Decoding: Convergence Issue

I have been trying to create a basic BCJR decoder for a rate (1/2) punctured turbo code. The puncturing is simple, the encoder alternatively selects between the parity bits generated by the two ...
30 views

### What is the Asymptotic Equipartition Property (AEP)?

I am currently studying about Polar Codes in 5G standard and while reading my paper I found something called AEP which is required for channel coding. I surfed the web but didn't found a satisfying ...
23 views

### Two inputs channel capacity

Assume that we have an AWGN channel with two inputs (A,B) and the o/p is written as Y = A + g B + Z where Z is an independent Gaussian noise with zero mean and unit variance and g is a constant ...
35 views

### Relationship between spectral efficiency and PAR

Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
144 views

### Concatenated Deletion & Sticky Channel Capacity

Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
425 views

### White noise and entropy confusion about concepts

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
42 views

Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that $$b=\begin{cases} 0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\ 1, &... 0answers 22 views ### Does Mutual information utilize spatial information to calculate image similarity? To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ... 2answers 36 views ### Rate in CDMA setup I am working on a CDMA system with only 4 chips/symbol. I want to use R=\frac{1}{2}log_2(1+SNR) chips/channel use to calculate what SNR I require to successfully receive a given transmission. In ... 1answer 32 views ### Is there some literature to understand finite state ISI channel? I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ... 2answers 1k views ### Choosing samples per symbol for modem pulse One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem. The smaller the pulse width,... 3answers 300 views ### How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle I'm trying to understand something about channel "bandwidth" B in the Shannon-Hartley Theorem:$$C = B \log_2 \left( 1 + \dfrac{S}{N} \right) $$Suppose I have a bitstream encoded as a signal that ... 2answers 231 views ### Channel Capacity - is this only for discrete/digital signals? The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission? 2answers 241 views ### Don't understand what is meant by signal dimension I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ... 0answers 39 views ### Capacity region of 'symbol asynchronous' Gaussian multiple access channel I was interested to learn what theoretical limits have been determined so far for the 'multiple access channel' with additive Gaussian noise. This brought me to the 1989 paper by Sergio Verdu, named "... 2answers 591 views ### What is the meaning of channel capacity? If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is? Is there a connection between the modulation kind and ... 0answers 116 views ### What is the difference between entropy and entropy rate and which one is important in information theory? [duplicate] By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high ... 2answers 395 views ### Will the capacity of a channel becomes unbounded if i increase its signal-to-noise ratio S/N without limit? According to the Shannon-Hartley theorem the capacity C of a channel which has a signal-to-noise ratio of S/N and a bandwidth B can be calculated to be C = B \log_2 \left( 1 + \frac{S}{N}\right)... 1answer 290 views ### How to calculate the entropy of random normally distributed samples? I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers. I'm trying to ... 1answer 238 views ### coding gain and shaping gain in SCMA It is said that sparse code multiple access (SCMA) has the advantage of higher shaping and coding gain over LDS-CDMA due to the multidimensional constellation. What exactly is the shaping gain and how ... 0answers 211 views ### SNR for channel after equalization Suppose I have a channel described by h[n] or H(z) over which I send a simple PAM2 signal. After the channel AWGN is added. What is the best possible SNR with equalization? As suggested in the ... 0answers 54 views ### Is there a theoretical limit for how much information hiding can be done through watermarking? There is a classic channel-capacity notion that exist based on the work of Shannon that gives gives measure of amount of information that can be passed through the channel given its properties. In ... 0answers 95 views ### Intuitive interpretation of transform coding I am currently studying source coding of waveform in a course of Information Theory and Coding. More precisely, I am trying to understand, both mathematically and intuitively, transform coding and its ... 3answers 724 views ### Capacity of AWGN channel I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is:$$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$and it is ... 1answer 154 views ### Given a noise-free channel with limited bandwidth, why is the channel capacity infinite? [duplicate] In the Shannon Hartley theorem, if N = 0 but B is limited, the theorem tells us that C is infinite. Why is this intuitively true? If bandwidth is limited, I can only post so many pulses per second ... 2answers 155 views ### Why is bandwidth always limited in a real (physical) channel? I'm talking about a continuous analog channel. Why can't it support infinite bandwidth? Is there a physics reasons for it say for electrical signals? 1answer 534 views ### What is the effect of noise on Shannon entropy? Since entropy is a measure of uncertainty or randomness, intuitively we would suppose that adding noise to an image would increase its entropy since we are now more uncertain about the information of ... 1answer 70 views ### Word length for time series This question is based on , "A Symbolic Representation of Time Series, with Implications for Streaming Algorithms". Let, N = w\cdot l where N is the number of data points in a time series ... 1answer 211 views ### Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size? A sequence of data of length N can be subdivided into equal sixed blocks each of length (size) l. For each block, w, we can calculate the entropy known as the block entropy. Considering, entropy ... 1answer 234 views ### Shannon's entropy for non-equiprobable symbols I am trying to understand the concept of Shannon's entropy and deciding the codelength. In first case, b is an array of 5 symbols. The symbol set is \{1,2,...,8\}.... 1answer 1k views ### Entropy : do we prefer higher or lower entropy? The entropy of a source H(S) gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ... 1answer 342 views ### How to prove relationship between mutual information and differential entropy? I need to prove that mutual information given by$$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$is equivalent to I(X;Y)=H(Y) - H(Y|X) I am ... 1answer 326 views ### How to solve this entropy integral? I am having the entropy integral below where \mathbf x is a N dimensional Gaussian vector having variance as \mathbf P and mean zero,$$ -\int\frac {\exp\left(\frac{1}{2}\mathbf x^T\mathbf P^{-...
When one analyzes an LDPC code, it is usual to plot the BER versus the $E_b / N_0$. My questions are: Why are some LDPC codes better than others in terms of noise immunity? After all, they are just ...
I wrote a script in MATLAB to simulate an LDPC created using an algorithm of a colleague of mine. The parity check-matrix is regular and has dimensions $1012 \times 1518$ and the code rate is $R=1/3$. ...