14
votes
Capacity of AWGN channel
Assuming a channel whose input at each time is a continuous random variable $X$ and its output is $Y=X+Z$, where $Z\sim\mathcal{N}(0,N)$ and $Z$ is independent of $X$, then $$C_{\text{CI-AWGN}}=\frac{...
- 4,115
10
votes
Accepted
Is concept of "bit" in computer programming similar to the concept of "bit" in information theory?
They are not the same, but they're related. In particular, if you look at a computer memory holding $M$ "computer" bits, where each bit can be considered random and independent of all other bits, and ...
- 13.8k
9
votes
Accepted
Entropy : do we prefer higher or lower entropy?
You seem to have a number of misunderstandings, which I'll try to clarify while also trying to help with your questions.
The entropy of a source $H(S)$ gives the average codeword length to encode a ...
- 13.8k
7
votes
Capacity of AWGN channel
The capacity formula
$$C = 0.5 \log (1+\frac{S}{N}) \tag{1}$$
is for discrete time channel.
Assuming you have a sequence of data $\left\lbrace a_n \right\rbrace$ to send out, you need an orthonormal ...
- 5,780
6
votes
How to Detect a Inhomogeneity Region in Image
There are many properties of inhomogeneity:
Local Variance / STD.
Local Histogram.
The Gradient Function
Histogram of the Gradient.
Mean versus the Median / Mode.
- 41.3k
6
votes
Accepted
What is the meaning of channel capacity?
1) Is there a connection between the modulation kind and the channel capacity?
The capacity of a channel indicates the upper limit of how many bits can be transmitted per second over the channel with ...
- 11.8k
6
votes
What is the meaning of channel capacity?
1) Is there a connection between the modulation kind and the channel
capacity?
Channel capacity is usually defined as the number of information (usually measured by the number of bits) can be sent ...
- 5,780
5
votes
Accepted
Nyquist noiseless channel capacity; how can bit-rate be two times the bandwidth?
I think you're confusing two different (but related) terms.
Nyquist says that in a channel of bandwidth $B$ you can transmit up to $2B$ orthogonal pulses per second. So, $R_p \leq 2B$, where $R_p$ is ...
- 13.8k
4
votes
Accepted
Difficulties in understanding mutual information concept
What does mutual information (MI) convey?
It indicates that there is a relationship between the two signals- i.e. that they are not independent. It could be that they are correlated, but the ...
- 11.8k
4
votes
Accepted
What is the theoretical probability of error for OOK transmission?
It is not clear whether you want a derivation of the formula for the error probability of a wireless OOK system, or ideas about how the formula might apply to an optical communication system.
For a ...
- 18.9k
4
votes
How to reconcile "bandwidth" in the Shannon-Hartley Theorem with a spread-spectrum principle
If I'm only using bandwidth B1, doesn't that mean I can filter some of the noise out? And if so, would I be able to get S/N down? Wouldn't that be an alternative to occupying the full channel ...
- 26.6k
4
votes
Accepted
Shannon capacity limit & FEC comparisons
I start with BPSK achieving 1 bits/sec/Hz over passband AWGN. Factoring in 1/3 rate coding this becomes 0.333 bits/sec/Hz.
This is not correct.
The Shannon noisy channel coding theorem states that ...
- 5,780
4
votes
What is the intuition explaining the Shannon-Hartley theorem?
Echoing what already answered: you are approaching this backwards. SNR is a concept that's very fundamental and applicable to way more things that just channel capacity.
If you have a signal $y(t)$ ...
- 32.6k
3
votes
Is there a proof that equal bandwidths have equal information-carrying capacity?
I'm not a mathematician so I won't pretend to claim that this is anything like a proof, but at an intuitive level I think that the fact that you can deterministically transform baseband to passband ...
- 11.8k
3
votes
Accepted
What is the overall capacity of two cascaded BSC channel?
No, this is not correct. Consider the chain of two BSCs with error probabilities $p_1$, $p_2$ as a single BSC with unknown error probability $p$.
Now, we know that in overall no error occurs, when:
...
- 6,078
3
votes
Accepted
What is the effect of noise on Shannon entropy?
The grayscale image is actually a discrete image. entropy calculates a histogram and from that extracts "empiric" probabilities, to be used in the common ...
- 26.6k
3
votes
Accepted
Why is bandwidth always limited in a real (physical) channel?
Information bandwidth is dependent on signal to noise ratios. At absolute zero, quantum level signal quantization and quantum noise will limit the lower bound on the noise floor. At higher ...
- 34k
3
votes
Capacity of AWGN channel
To say that the input signal has a Gaussian distribution means that it is distributed as a Gaussian random variable. In practice, one relies on coding over multiple instances of the channel (in time) ...
- 1,372
3
votes
Accepted
Channel Capacity - is this only for discrete/digital signals?
The definition of channel capacity can be applied to either digital or analog cases. The meaning depends on how you calculate it. The definition is:
$$C=\max_{p_X(x)}\ I(X;Y)$$
In the digital world, ...
- 4,872
3
votes
Accepted
How to reconcile "bandwidth" in the Shannon-Hartley Theorem with a spread-spectrum principle
You are mixing up two different notions that have little to do with each other.
The use of spread-spectrum signaling is not in an effort to achieve (or even approach) the capacity of the (wideband) ...
- 18.9k
3
votes
Accepted
Why $H(A)=H(C)$ where $C$ is $A$ with an additional parity bit?
Because the entropy represents information quantity, or if being measured in bit, the smallest number of bits per symbol we need to represent a source.
The source $A$ contains $4$ equiprobable ...
- 5,780
3
votes
Accepted
value of 0 log0 in entropy formula
Two possible answers:
If a symbol has probability zero, then it does not influence the calculation and there is no need to include it, so you never actually calculate $0\log(0)$.
If you insist in ...
- 13.8k
2
votes
Why is mutual information symmetric but conditional entropy isn't?
There's a cool Venn diagram from here. This shows clearly that $I(X:Y)$ is independent of the order of $X$ and $Y$.
- 121
2
votes
Accepted
What are some typical lossless compression ratios?
My survey paper on compression, "A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems", shows that most practical techniques on general benchmarks achieve ...
- 136
2
votes
Accepted
Conceptual question on entropy and its relation to information
The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as ...
- 2,827
2
votes
Accepted
Relation between SER and the channel capacity
Yes, there is a relationship between SER and channel capacity. The channel capacity equation is-
$$
C = B\log_2(1 + \frac{S}{N})
$$
where $C$ is the channel capacity in bits/s, $B$ is the bandwidth ...
- 11.8k
2
votes
What is the entropy for these cases?
Adding a bit of detail to Marcus' answer:
Your question is about the "entropy of symbols" and the "entropy of real numbers". In information theory, only sources have entropy. A source has an alphabet,...
- 13.8k
2
votes
Accepted
What is the entropy for these cases?
The entropy $H(X)$ of a continuous random variable $X$ is infinite. Proof is trivial (note that we can, without loss of generality, use the natural logarithm, since any other logarithm is the same but ...
- 26.6k
2
votes
Accepted
Mutual information of $ \infty $-PAM
I got an answer here,
https://math.stackexchange.com/questions/1554659/mutual-information-for-a-continuous-uniform-distribution
The $ \text{erf}(x) $ function approximates to 1 with an error less ...
- 171
2
votes
Is concept of "bit" in computer programming similar to the concept of "bit" in information theory?
Bit is a unit of measurement and multiple quantities are measured in bits. It's not that bit in programming and information theory mean different things. It's that memory and information content ...
- 121
Only top scored, non community-wiki answers of a minimum length are eligible