Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [information-theory]

The tag has no usage guidance.

0
votes
0answers
9 views

Symmetric Capacity of a Binary Discrete Memoryless Channel

While reading this paper https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5075875, I found that the the symmetric capacity of BDMC $W$ is given by $$I(W) = \sum_{y\in Y}\sum_{x\in X}\...
0
votes
0answers
10 views

Punctured Turbo Code Decoding: Convergence Issue

I have been trying to create a basic BCJR decoder for a rate (1/2) punctured turbo code. The puncturing is simple, the encoder alternatively selects between the parity bits generated by the two ...
0
votes
0answers
19 views

Two inputs channel capacity

Assume that we have an AWGN channel with two inputs (A,B) and the o/p is written as Y = A + g B + Z where Z is an independent Gaussian noise with zero mean and unit variance and g is a constant ...
0
votes
1answer
19 views

Relationship between spectral efficiency and PAR

Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
1
vote
1answer
134 views

Concatenated Deletion & Sticky Channel Capacity

Suppose you are sending information through a deletion channel with a deletion rate of .5 and then the output of this channel is sent through a sticky channel with a repeat rate of .5 also. No error ...
0
votes
1answer
141 views

White noise and entropy confusion about concepts

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
1
vote
1answer
37 views

Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?

Let $A=\{00,01,10,11\}$ with equal probabilities for each symbol, and $B=\{0, 1\}$ be a parity generator such that $$ b=\begin{cases} 0, & \text{if} \,\, a=00 \quad \text{or} \quad a=11 \\ 1, &...
1
vote
0answers
20 views

Does Mutual information utilize spatial information to calculate image similarity?

To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ...
0
votes
0answers
37 views

Modulation to achieve Shannon capacity in AWGN channels

If my understanding is correct, to achieve the Shannon capacity in AWGN channels, we need a modulation that codewords follow Normal distribution, or uniform on surface of a sphere (sphere hardening ...
0
votes
2answers
35 views

Rate in CDMA setup

I am working on a CDMA system with only 4 chips/symbol. I want to use $R=\frac{1}{2}log_2(1+SNR)$ chips/channel use to calculate what $SNR$ I require to successfully receive a given transmission. In ...
0
votes
1answer
29 views

Is there some literature to understand finite state ISI channel?

I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
2
votes
2answers
493 views

Choosing samples per symbol for modem pulse

One of the parameters in my DSP library for pulse design is "samples per symbol", and I would like to get some advice about choosing this parameter when designing a modem. The smaller the pulse width,...
0
votes
0answers
35 views

why we use positive side bandwidh when we calculate spectral efficiency

when we calculate spectral efficiency we devide bit rate by positive side bandwidth but signal possesses not only positive side band but also negative side band that is R bps : 2W Hz => R/W bps : ...
5
votes
3answers
200 views

How to reconcile “bandwidth” in the Shannon-Hartley Theorem with a spread-spectrum principle

I'm trying to understand something about channel "bandwidth" $B$ in the Shannon-Hartley Theorem: $$C = B \log_2 \left( 1 + \dfrac{S}{N} \right) $$ Suppose I have a bitstream encoded as a signal that ...
1
vote
1answer
79 views

Channel Capacity - is this only for discrete/digital signals?

The units of channel capacity is bits/second. Does this mean this only refers to discrete/digital signals? Is channel capacity analogous to bandwidth in case of an analog signal transmission?
1
vote
2answers
112 views

Don't understand what is meant by signal dimension

I don't understand the concept of dimension of a signal. I ran into it in an explanation of Shannon Capacity, and in a paper on spread spectrum. I was hoping somebody could explain with an example. ...
1
vote
0answers
34 views

Capacity region of 'symbol asynchronous' Gaussian multiple access channel

I was interested to learn what theoretical limits have been determined so far for the 'multiple access channel' with additive Gaussian noise. This brought me to the 1989 paper by Sergio Verdu, named "...
4
votes
2answers
444 views

What is the meaning of channel capacity?

If the transmitted information signal takes values from a modulation constellation, then is there a way to know what the channel capacity is? Is there a connection between the modulation kind and ...
1
vote
0answers
81 views

What is the difference between entropy and entropy rate and which one is important in information theory? [duplicate]

By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high ...
2
votes
2answers
235 views

Will the capacity of a channel becomes unbounded if i increase its signal-to-noise ratio $S/N$ without limit?

According to the Shannon-Hartley theorem the capacity $C$ of a channel which has a signal-to-noise ratio of $S/N$ and a bandwidth $B$ can be calculated to be $C = B \log_2 \left( 1 + \frac{S}{N}\right)...
1
vote
1answer
151 views

How to calculate the entropy of random normally distributed samples?

I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers. I'm trying to ...
0
votes
1answer
176 views

coding gain and shaping gain in SCMA

It is said that sparse code multiple access (SCMA) has the advantage of higher shaping and coding gain over LDS-CDMA due to the multidimensional constellation. What exactly is the shaping gain and how ...
1
vote
0answers
139 views

SNR for channel after equalization

Suppose I have a channel described by $h[n]$ or $H(z)$ over which I send a simple PAM2 signal. After the channel AWGN is added. What is the best possible SNR with equalization? As suggested in the ...
3
votes
0answers
53 views

Is there a theoretical limit for how much information hiding can be done through watermarking?

There is a classic channel-capacity notion that exist based on the work of Shannon that gives gives measure of amount of information that can be passed through the channel given its properties. In ...
1
vote
0answers
89 views

Intuitive interpretation of transform coding

I am currently studying source coding of waveform in a course of Information Theory and Coding. More precisely, I am trying to understand, both mathematically and intuitively, transform coding and its ...
8
votes
3answers
553 views

Capacity of AWGN channel

I am confused understanding basic concepts of communication over AWGN channels. I know the capacity of a discrete time AWGN channel is: $$C=\frac{1}{2}\log_2\left(1+\frac{S}{N}\right)$$ and it is ...
0
votes
1answer
125 views

Given a noise-free channel with limited bandwidth, why is the channel capacity infinite? [duplicate]

In the Shannon Hartley theorem, if N = 0 but B is limited, the theorem tells us that C is infinite. Why is this intuitively true? If bandwidth is limited, I can only post so many pulses per second ...
1
vote
2answers
131 views

Why is bandwidth always limited in a real (physical) channel?

I'm talking about a continuous analog channel. Why can't it support infinite bandwidth? Is there a physics reasons for it say for electrical signals?
1
vote
1answer
427 views

What is the effect of noise on Shannon entropy?

Since entropy is a measure of uncertainty or randomness, intuitively we would suppose that adding noise to an image would increase its entropy since we are now more uncertain about the information of ...
0
votes
1answer
65 views

Word length for time series

This question is based on , "A Symbolic Representation of Time Series, with Implications for Streaming Algorithms". Let, $N = w\cdot l$ where $N$ is the number of data points in a time series ...
3
votes
1answer
203 views

Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?

A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
0
votes
1answer
196 views

Shannon's entropy for non-equiprobable symbols

I am trying to understand the concept of Shannon's entropy and deciding the codelength. In first case, b is an array of 5 symbols. The symbol set is $\{1,2,...,8\}$....
0
votes
1answer
754 views

Entropy : do we prefer higher or lower entropy?

The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ...
0
votes
1answer
248 views

How to prove relationship between mutual information and differential entropy?

I need to prove that mutual information given by $$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$ is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am ...
0
votes
1answer
272 views

How to solve this entropy integral?

I am having the entropy integral below where $\mathbf x$ is a $N$ dimensional Gaussian vector having variance as $\mathbf P$ and mean zero, $$ -\int\frac {\exp\left(\frac{1}{2}\mathbf x^T\mathbf P^{-...
1
vote
1answer
61 views

Why are some LDPC codes more immune to noise than others?

When one analyzes an LDPC code, it is usual to plot the BER versus the $E_b / N_0$. My questions are: Why are some LDPC codes better than others in terms of noise immunity? After all, they are just ...
0
votes
1answer
1k views

How to explain this BER vs SNR behaviour?

I wrote a script in MATLAB to simulate an LDPC created using an algorithm of a colleague of mine. The parity check-matrix is regular and has dimensions $1012 \times 1518$ and the code rate is $R=1/3$. ...
0
votes
1answer
495 views

What is the overall capacity of two cascaded BSC channel?

Is it true:$C = C_1 * C_2$ where $C_1$ and $C_2$ are the two cascaded BSC which makes $C$.
0
votes
0answers
312 views

Differences between $\tt wentropy$ in MATLAB and entropy definition

MATLAB offer's a Shannon entropy calculation in its wavelet package: $$ H(X) = -\sum_{i=0}^{N-1} x_i^2\log(x_i^2) $$ I'm wondering what this formula means, since it has no relation with the ...
1
vote
1answer
265 views

Source and channel encoding

In information theory, there are two concepts that are widely used: source encoding and channel encoding. I'm afraid I may not be capable of expressing my doubts clearly, but the thing is that I don'...
1
vote
1answer
57 views

Discrete entropies

I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it. I am given the following: $X$ is a uniformly distributed random ...
0
votes
1answer
152 views

Mechanics of a matrix Interleaver

I am having trouble understanding exactly how the matrix interleaver. I have read the following page from MathWorks. In it, it gives the following example where "123456" is interleaved as "142536." ...
1
vote
3answers
2k views

What is the theoretical probability of error for OOK transmission?

I am trying to simulate an optical wireless communication channel which uses OOK modulation. Looking at equation 14 in this literature, I found that: For electrical SNR at the receiver $$\textrm{...
0
votes
1answer
52 views

Entropy of a dynamical system and source code length

The entropy of a source $H(S)$, gives you the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the ...
1
vote
1answer
151 views

Why is $x$ not considered a primitive polynomial while being considered an irreducible polynomial?

In "Shu Lin, Daniel J. Costello-Error Control Coding (2nd Edition), Prentice Hall 2004" it is given that in $GF(2)$, if $f(x)$ is an irreducible polynomial of degree $m$ it divides $x^n+1$ where $n = ...
0
votes
1answer
65 views

What are bits in the context of channel capacity units?

'bits' are used with 2 different meanings. One can use 'bits' to mean the binary digits i.e. 1's and 0's. Bits are also the units of information in a event of discrete source. I think when channel ...
1
vote
1answer
45 views

Can channel capcity be explained in simple terms? [closed]

Can channel capacity be explained in simple terms - without mutual information and such probabilistic concepts? In its most general form, what are all the parameters that it depends on? What is its ...
1
vote
0answers
527 views

Is there any formula for coding gain of Turbo code?

I want to compare throughput under different MCS( modulation and coding scheme). We know Shannon capacity $C = B \log(1+SNR)$ can be used to calculate throughput by SNR. However, there is no MCS ...
2
votes
3answers
248 views

Is there a proof that equal bandwidths have equal information-carrying capacity?

Consider a continuous channel, band-limited to the $[f_{\text{min}},\,f_{\text{max}}]$ frequency window. A well-known statement says that, as far as the dependence on the frequencies, the information-...
1
vote
2answers
122 views

What are the fundamental reasons that suggest us to minimise transmission power for energy-efficient radio link design?

My question in short: Under which scenarios minimising transmission power is beneficial in terms of transmitter energy-efficiency? is there any fundamental reason to keep a low-power oriented design ...