14 votes

Capacity of AWGN channel

Assuming a channel whose input at each time is a continuous random variable $X$ and its output is $Y=X+Z$, where $Z\sim\mathcal{N}(0,N)$ and $Z$ is independent of $X$, then $$C_{\text{CI-AWGN}}=\frac{...
user avatar
  • 4,115
10 votes
Accepted

Is concept of "bit" in computer programming similar to the concept of "bit" in information theory?

They are not the same, but they're related. In particular, if you look at a computer memory holding $M$ "computer" bits, where each bit can be considered random and independent of all other bits, and ...
user avatar
  • 13.8k
9 votes
Accepted

Entropy : do we prefer higher or lower entropy?

You seem to have a number of misunderstandings, which I'll try to clarify while also trying to help with your questions. The entropy of a source $H(S)$ gives the average codeword length to encode a ...
user avatar
  • 13.8k
7 votes

Capacity of AWGN channel

The capacity formula $$C = 0.5 \log (1+\frac{S}{N}) \tag{1}$$ is for discrete time channel. Assuming you have a sequence of data $\left\lbrace a_n \right\rbrace$ to send out, you need an orthonormal ...
user avatar
  • 5,780
6 votes

How to Detect a Inhomogeneity Region in Image

There are many properties of inhomogeneity: Local Variance / STD. Local Histogram. The Gradient Function Histogram of the Gradient. Mean versus the Median / Mode.
user avatar
  • 41.3k
6 votes
Accepted

What is the meaning of channel capacity?

1) Is there a connection between the modulation kind and the channel capacity? The capacity of a channel indicates the upper limit of how many bits can be transmitted per second over the channel with ...
user avatar
  • 11.8k
6 votes

What is the meaning of channel capacity?

1) Is there a connection between the modulation kind and the channel capacity? Channel capacity is usually defined as the number of information (usually measured by the number of bits) can be sent ...
user avatar
  • 5,780
5 votes
Accepted

Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?

I think you're confusing two different (but related) terms. Nyquist says that in a channel of bandwidth $B$ you can transmit up to $2B$ orthogonal pulses per second. So, $R_p \leq 2B$, where $R_p$ is ...
user avatar
  • 13.8k
4 votes
Accepted

Difficulties in understanding mutual information concept

What does mutual information (MI) convey? It indicates that there is a relationship between the two signals- i.e. that they are not independent. It could be that they are correlated, but the ...
user avatar
  • 11.8k
4 votes
Accepted

What is the theoretical probability of error for OOK transmission?

It is not clear whether you want a derivation of the formula for the error probability of a wireless OOK system, or ideas about how the formula might apply to an optical communication system. For a ...
user avatar
4 votes

How to reconcile "bandwidth" in the Shannon-Hartley Theorem with a spread-spectrum principle

If I'm only using bandwidth B1, doesn't that mean I can filter some of the noise out? And if so, would I be able to get S/N down? Wouldn't that be an alternative to occupying the full channel ...
user avatar
4 votes
Accepted

Shannon capacity limit & FEC comparisons

I start with BPSK achieving 1 bits/sec/Hz over passband AWGN. Factoring in 1/3 rate coding this becomes 0.333 bits/sec/Hz. This is not correct. The Shannon noisy channel coding theorem states that ...
user avatar
  • 5,780
4 votes

What is the intuition explaining the Shannon-Hartley theorem?

Echoing what already answered: you are approaching this backwards. SNR is a concept that's very fundamental and applicable to way more things that just channel capacity. If you have a signal $y(t)$ ...
user avatar
  • 32.6k
3 votes

Is there a proof that equal bandwidths have equal information-carrying capacity?

I'm not a mathematician so I won't pretend to claim that this is anything like a proof, but at an intuitive level I think that the fact that you can deterministically transform baseband to passband ...
user avatar
  • 11.8k
3 votes
Accepted

What is the overall capacity of two cascaded BSC channel?

No, this is not correct. Consider the chain of two BSCs with error probabilities $p_1$, $p_2$ as a single BSC with unknown error probability $p$. Now, we know that in overall no error occurs, when: ...
user avatar
3 votes
Accepted

What is the effect of noise on Shannon entropy?

The grayscale image is actually a discrete image. entropy calculates a histogram and from that extracts "empiric" probabilities, to be used in the common ...
user avatar
3 votes
Accepted

Why is bandwidth always limited in a real (physical) channel?

Information bandwidth is dependent on signal to noise ratios. At absolute zero, quantum level signal quantization and quantum noise will limit the lower bound on the noise floor. At higher ...
user avatar
  • 34k
3 votes

Capacity of AWGN channel

To say that the input signal has a Gaussian distribution means that it is distributed as a Gaussian random variable. In practice, one relies on coding over multiple instances of the channel (in time) ...
user avatar
  • 1,372
3 votes
Accepted

Channel Capacity - is this only for discrete/digital signals?

The definition of channel capacity can be applied to either digital or analog cases. The meaning depends on how you calculate it. The definition is: $$C=\max_{p_X(x)}\ I(X;Y)$$ In the digital world, ...
user avatar
  • 4,872
3 votes
Accepted

How to reconcile "bandwidth" in the Shannon-Hartley Theorem with a spread-spectrum principle

You are mixing up two different notions that have little to do with each other. The use of spread-spectrum signaling is not in an effort to achieve (or even approach) the capacity of the (wideband) ...
user avatar
3 votes
Accepted

Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?

Because the entropy represents information quantity, or if being measured in bit, the smallest number of bits per symbol we need to represent a source. The source $A$ contains $4$ equiprobable ...
user avatar
  • 5,780
3 votes
Accepted

value of 0 log0 in entropy formula

Two possible answers: If a symbol has probability zero, then it does not influence the calculation and there is no need to include it, so you never actually calculate $0\log(0)$. If you insist in ...
user avatar
  • 13.8k
2 votes

Why is mutual information symmetric but conditional entropy isn't?

There's a cool Venn diagram from here. This shows clearly that $I(X:Y)$ is independent of the order of $X$ and $Y$.
user avatar
2 votes
Accepted

What are some typical lossless compression ratios?

My survey paper on compression, "A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems", shows that most practical techniques on general benchmarks achieve ...
user avatar
2 votes
Accepted

Conceptual question on entropy and its relation to information

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as ...
user avatar
  • 2,827
2 votes
Accepted

Relation between SER and the channel capacity

Yes, there is a relationship between SER and channel capacity. The channel capacity equation is- $$ C = B\log_2(1 + \frac{S}{N}) $$ where $C$ is the channel capacity in bits/s, $B$ is the bandwidth ...
user avatar
  • 11.8k
2 votes

What is the entropy for these cases?

Adding a bit of detail to Marcus' answer: Your question is about the "entropy of symbols" and the "entropy of real numbers". In information theory, only sources have entropy. A source has an alphabet,...
user avatar
  • 13.8k
2 votes
Accepted

What is the entropy for these cases?

The entropy $H(X)$ of a continuous random variable $X$ is infinite. Proof is trivial (note that we can, without loss of generality, use the natural logarithm, since any other logarithm is the same but ...
user avatar
2 votes
Accepted

Mutual information of $ \infty $-PAM

I got an answer here, https://math.stackexchange.com/questions/1554659/mutual-information-for-a-continuous-uniform-distribution The $ \text{erf}(x) $ function approximates to 1 with an error less ...
user avatar
  • 171
2 votes

Is concept of "bit" in computer programming similar to the concept of "bit" in information theory?

Bit is a unit of measurement and multiple quantities are measured in bits. It's not that bit in programming and information theory mean different things. It's that memory and information content ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible