Questions tagged [information-theory]

The tag has no usage guidance.

85 questions
Filter by
Sorted by
Tagged with
60 views

Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I'm confused by the Nyquist channel capacity formula. How can channel maximum capacity approach double the bandwidth. $C = 2\times BW \times log_{2}(L)$ bits/sec The way it was explained to me ...
28 views

Capacity achieving codes for BPSK at low SNR

I don't know a lot about comm system design. I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
16 views

How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?

Anyone know where to find graphs showing how well it works vs other schemes? thanks
65 views

How do I measure the amount of information within an image?

I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
207 views

Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?

A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
39 views

Proof that DFT does not require more than N points

I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
89 views

19 views

Punctured Turbo Code Decoding: Convergence Issue

I have been trying to create a basic BCJR decoder for a rate (1/2) punctured turbo code. The puncturing is simple, the encoder alternatively selects between the parity bits generated by the two ...
135 views

How to Detect a Inhomogeneity Region in Image

I have an image that includes object and background. However, the object appears some inhomogeneity region due to illumination. My work is that how to detect inhomogeneity region. Which is feature can ...
23 views

Two inputs channel capacity

Assume that we have an AWGN channel with two inputs (A,B) and the o/p is written as Y = A + g B + Z where Z is an independent Gaussian noise with zero mean and unit variance and g is a constant ...
31 views

Relationship between spectral efficiency and PAR

Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
142 views

Concatenated Deletion & Sticky Channel Capacity

Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
328 views

White noise and entropy confusion about concepts

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
42 views

220 views

coding gain and shaping gain in SCMA

It is said that sparse code multiple access (SCMA) has the advantage of higher shaping and coding gain over LDS-CDMA due to the multidimensional constellation. What exactly is the shaping gain and how ...
180 views

SNR for channel after equalization

Suppose I have a channel described by $h[n]$ or $H(z)$ over which I send a simple PAM2 signal. After the channel AWGN is added. What is the best possible SNR with equalization? As suggested in the ...
53 views

Is there a theoretical limit for how much information hiding can be done through watermarking?

There is a classic channel-capacity notion that exist based on the work of Shannon that gives gives measure of amount of information that can be passed through the channel given its properties. In ...
89 views

Intuitive interpretation of transform coding

I am currently studying source coding of waveform in a course of Information Theory and Coding. More precisely, I am trying to understand, both mathematically and intuitively, transform coding and its ...
654 views

Capacity of AWGN channel

I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is: $$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$ and it is ...
231 views

Shannon's entropy for non-equiprobable symbols

I am trying to understand the concept of Shannon's entropy and deciding the codelength. In first case, b is an array of 5 symbols. The symbol set is $\{1,2,...,8\}$....
149 views

Why is bandwidth always limited in a real (physical) channel?

I'm talking about a continuous analog channel. Why can't it support infinite bandwidth? Is there a physics reasons for it say for electrical signals?
145 views

Given a noise-free channel with limited bandwidth, why is the channel capacity infinite? [duplicate]

In the Shannon Hartley theorem, if N = 0 but B is limited, the theorem tells us that C is infinite. Why is this intuitively true? If bandwidth is limited, I can only post so many pulses per second ...
505 views

What is the effect of noise on Shannon entropy?

Since entropy is a measure of uncertainty or randomness, intuitively we would suppose that adding noise to an image would increase its entropy since we are now more uncertain about the information of ...
70 views

Word length for time series

This question is based on , "A Symbolic Representation of Time Series, with Implications for Streaming Algorithms". Let, $N = w\cdot l$ where $N$ is the number of data points in a time series ...
935 views

Entropy : do we prefer higher or lower entropy?

The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ...
321 views

How to prove relationship between mutual information and differential entropy?

I need to prove that mutual information given by $$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$ is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am ...
319 views

How to solve this entropy integral?

I am having the entropy integral below where $\mathbf x$ is a $N$ dimensional Gaussian vector having variance as $\mathbf P$ and mean zero,  -\int\frac {\exp\left(\frac{1}{2}\mathbf x^T\mathbf P^{-...
71 views

Why are some LDPC codes more immune to noise than others?

When one analyzes an LDPC code, it is usual to plot the BER versus the $E_b / N_0$. My questions are: Why are some LDPC codes better than others in terms of noise immunity? After all, they are just ...
1k views

How to explain this BER vs SNR behaviour?

I wrote a script in MATLAB to simulate an LDPC created using an algorithm of a colleague of mine. The parity check-matrix is regular and has dimensions $1012 \times 1518$ and the code rate is $R=1/3$. ...
Is it true:$C = C_1 * C_2$ where $C_1$ and $C_2$ are the two cascaded BSC which makes $C$.