# Questions tagged [information-theory]

The tag has no usage guidance.

99 questions
Filter by
Sorted by
Tagged with
10 views

### Mutual information smaller on data subsets compared to entire data set

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al . Lets consider two features $X$ and $Y$, and the ...
77 views

### Determining Loss of Information by Taking Average (Mean of Signal)

So basically information is defined by expected value of Shannon's information i.e. Entropy. I am curious how much information is lost if we simply take the average of the sample given to us. I am ...
14 views

47 views

### Can the Nyquist sample rate be extended to stochastic sampling?

It appears there are lots of questions here about Nyquist, and a few questions about stochastic sampling here. But I haven't found any that address quite what I'm after. This is the closest I've ...
35 views

### Information entropy of Poisson noise

I would like to calculate the information entropy of the channel with additive Poisson noise. The model consists of an input signal, an optical image, an image acquisition device, and the output ...
44 views

### Bandwidth and probability of continuous signal

I'm a little bit confused as to how to calculate the bandwidth of a signal. A question has me believing that it is correlated with the probability distribution. I am however not sure of this. ...
37 views

### Information theory vs system theory ( commented in Determining linearity, causality, memory, and time invariability from a picture of the signal)

I have seen this question : At the comment asked about : Is this "information theory" or "systems theory"? – The Photon So i liked to know why this comment asked? Why the system theory vs ...
32 views

### Wireless communications outage

For communication systems, there is usually a target minimum rate $R$ bits per second. An outage occurs if the actual rate ever falls below $R$. Since $\frac{1}{2}\text{log}_2(1+\text{SNR})$, ...
30 views

### Example of Entropy and Channel Capacity Computation

Can you help me on verifying if this computation of entropy is correct and on understanding its meaning? I am not sure of the result especially because it is equal to 0: it means that we cannot ...
45 views

### Understanding Asymptotic Equipartition Property

I have some problems in understanding the precise meaning of the Asymptotic Equipartition Property, related to a large number n of independent and identically distributed random variables (X1, X2, ...,...
53 views

### When channels are in a cascade and are independent is the capacity the sum?

Just wondering if two channels are independent then is the overall capacity, the sum of the individual capacities? For example suppose I have two BEC where the probability of erasure is $\varepsilon$...
47 views

### Definition of Perfect Code (and example of Hamming Code)

Let's consider the following sentences about coding a message before transmitting it in a communication system (here you find the complete pdf) First question: can you explain me these two ...
33 views

### Band-limited signal recovery with finite bit depth

The sampling theorem tells us that a signal with no frequencies above $f$ can be completely described by sampling it a rate of $2f$. However, the theorem makes no reference to quantization, and so I ...
27 views

### Pearson correlation coefficient vs mutual information

I recently computed both the pearson coefficient and the mutual information of two time series as well as the corresponding error bars, and what I find out in the end is a significant level of ...
354 views

### Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth. $C = 2\times BW \times log_{2}(L)$ bits/sec The way it was explained to me ...
43 views

### Capacity achieving codes for BPSK at low SNR

I don't know a lot about comm system design. I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
19 views

### How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?

Anyone know where to find graphs showing how well it works vs other schemes? thanks
512 views

### How do I measure the amount of information within an image?

I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
61 views

### Proof that DFT does not require more than N points

I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
91 views

27 views

### Does Mutual information utilize spatial information to calculate image similarity?

To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ...
42 views

### Rate in CDMA setup

I am working on a CDMA system with only 4 chips/symbol. I want to use $R=\frac{1}{2}log_2(1+SNR)$ chips/channel use to calculate what $SNR$ I require to successfully receive a given transmission. In ...
34 views

### Is there some literature to understand finite state ISI channel?

I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
1k views

### Choosing samples per symbol for modem pulse

One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem. The smaller the pulse width,...
362 views

### How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle

I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right)$$ Suppose I have a bitstream encoded as a signal that ...
293 views

### Channel Capacity - is this only for discrete/digital signals?

The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission?
410 views

### Don't understand what is meant by signal dimension

I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...
40 views

### Capacity region of 'symbol asynchronous' Gaussian multiple access channel

I was interested to learn what theoretical limits have been determined so far for the 'multiple access channel' with additive Gaussian noise. This brought me to the 1989 paper by Sergio Verdu, named "...
701 views

### What is the meaning of channel capacity?

If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is? Is there a connection between the modulation kind and ...