# Tag Info

13

Assuming a channel whose input at each time is a continuous random variable $X$ and its output is $Y=X+Z$, where $Z\sim\mathcal{N}(0,N)$ and $Z$ is independent of $X$, then $$C_{\text{CI-AWGN}}=\frac{1}{2}\log_2\left(1+\frac{P}{N}\right)$$ is the capacity of the continuous-input channel under the power constraint $$\mathsf{E}X^2\le P$$ The mutual information ...

9

You seem to have a number of misunderstandings, which I'll try to clarify while also trying to help with your questions. The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. While this is true, I think it's not ...

8

They are not the same, but they're related. In particular, if you look at a computer memory holding $M$ "computer" bits, where each bit can be considered random and independent of all other bits, and there are roughly 50% of zeros, then the memory also holds roughly $M$ "information theory" bits. Of course, this is often not the case: computer bits are ...

8

When you say that the "information content may remain the same," do you mean the information in the total signal, or the information of the desired signal? Hopefully this will answer both cases. I know Shannon entropy much better than Kolmogorov so I'll use that, but hopefully the logic will translate. Let's say $X = S + N$ is your total signal ($X$), ...

7

The capacity formula $$C = 0.5 \log (1+\frac{S}{N}) \tag{1}$$ is for discrete time channel. Assuming you have a sequence of data $\left\lbrace a_n \right\rbrace$ to send out, you need an orthonormal waveform set $\left\lbrace \phi_n(t) \right\rbrace$ for modulation. In linear modulation, whom M-ary modution belongs to, $\phi_n(t) = \phi(t - nT)$ where $T$ ...

6

No,because while mutual information tells you how much uncertainty can a random variable $X$ remove from another random variable $Y$, the conditional entropy of $X$ given $Y$ tells you how much uncertainty remained in X after using the information (given by $I(X;Y)$) that $Y$ gave it. $H(X|Y)=H(X)-I(X;Y)$ That's why the conditional entropy depends on the ...

6

1) Is there a connection between the modulation kind and the channel capacity? The capacity of a channel indicates the upper limit of how many bits can be transmitted per second over the channel with no errors (okay, technically it is "arbitrarily low number of errors", but it's basically the same thing). We do various things to try to get as close to that ...

6

1) Is there a connection between the modulation kind and the channel capacity? Channel capacity is usually defined as the number of information (usually measured by the number of bits) can be sent per channel use to get arbitrarily low number of errors, but we don't know exactly how (random coding). A channel use can be thought as a modulation symbol ...

5

Let me give you a quick answer, focused on ways you can improve and research for a deeper knowledge by yourself. 1 - Kolmogorov and Shannon Entropies are related to measures of information and strongly related to one another. (See the wikipedia page for Kolmogorov complexity for further info.) 2 - Boltzmann entropy, which is an older concept, is a ...

5

There are two different kinds of channel models that are being confused here. In the binary symmetric channel, the inputs and the outputs are constrained to be $0$ or $1$ and the key parameter is the transition probability: the probability that an input $0$ is changed to an output $1$ or vice versa. The channel capacity $C$ is a number between $0$ and $1$ ...

5

There are two main factors when figuring out how many bits are transmitted per symbol (or "channel use"): the modulation and the error correction encoding. For instance, BPSK modulation with no encoding transmits 1 bit/symbol, while QPSK with no encoding transmits 2 bits/symbol. Higher order modulation schemes (e.g. 8-PSK, 16-QAM, 32-QAM, etc.) can ...

4

What does mutual information (MI) convey? It indicates that there is a relationship between the two signals- i.e. that they are not independent. It could be that they are correlated, but the relationship does not have to be that simple. The higher the mutual information, the stronger the relationship (i.e. the more one of the signals can "tell" you about ...

4

Shannon's Lossless Source Coding theorem states that the expected length of any code is lower bounded by the entropy of the probability distribution over the symbols for that code: $$H\left(X\right) \leq \mathbb{E}_{P_{X}}\left(l \circ f\right)$$ where $\circ$ denotes the function composition operation, and $l$ is the length of the codeword assigned by ...

4

There are many properties of inhomogeneity: Local Variance / STD. Local Histogram. The Gradient Function Histogram of the Gradient. Mean versus the Median / Mode.

4

It is not clear whether you want a derivation of the formula for the error probability of a wireless OOK system, or ideas about how the formula might apply to an optical communication system. For a wireless OOK system, there are two main types of receivers called coherent receivers and noncoherent receivers and these have different error probabilities. ...

4

If I'm only using bandwidth B1, doesn't that mean I can filter some of the noise out? And if so, would I be able to get S/N down? Wouldn't that be an alternative to occupying the full channel bandwidth? Capacity increases linearly with bandwidth, but only logarithmically with SNR. So, increasing SNR by decreasing bandwidth is a bad idea if your noise isn't ...

4

I start with BPSK achieving 1 bits/sec/Hz over passband AWGN. Factoring in 1/3 rate coding this becomes 0.333 bits/sec/Hz. This is not correct. The Shannon noisy channel coding theorem states that the reliable discrete-time rate $r$ (whose unit is bits per symbol, or bits per channel-use, or bpcu) is upper-bounded $$r \lt \frac{1}{2}\log_2\left(1 + \frac{S}{... 3 A binary input - binary output channel which flips bits with equal probabilities is known as a Binary Symmetric Channel. (Image Credit: Wikipedia - Binary Symmetric Channel) The channel capacity C is the maximum (actually, a supremum) mutual information between the channel inputs and the channel outputs over input distributions. You can't get ... 3 HINT: Compute the entropy H of your source and note that in your case it represents the number of bits required to represent each symbol. Then from \quad H [bits/symbol] \times x [symbols/second] = 5 [bits/second] compute x. 3 I'm not a mathematician so I won't pretend to claim that this is anything like a proof, but at an intuitive level I think that the fact that you can deterministically transform baseband to passband and back again by multiplying with complex sinusoids implies that the two are equivalent. 3 No, this is not correct. Consider the chain of two BSCs with error probabilities p_1, p_2 as a single BSC with unknown error probability p. Now, we know that in overall no error occurs, when: neither BSC1 nor BSC2 create an error: (1-p_1)(1-p_2) both channels make an error: p_1p_2. Hence the overall probability of an error is$$ p= 1-[(1-p_1)(...

3

The grayscale image is actually a discrete image. entropy calculates a histogram and from that extracts "empiric" probabilities, to be used in the common information formula $$I= -\log_2 (P(g))$$ with $P(g)$ being the (empiric) probability of an intensity value $g$. Now, entropy $H$ is the expectation of Information. For your whole image $V$ that means $... 3 To say that the input signal has a Gaussian distribution means that it is distributed as a Gaussian random variable. In practice, one relies on coding over multiple instances of the channel (in time) instead of relying on a Gaussian input distribution. There is a beautiful theory full of proofs that is beyond the scope of this answer (Information Theory). ... 3 The definition of channel capacity can be applied to either digital or analog cases. The meaning depends on how you calculate it. The definition is: $$C=\max_{p_X(x)}\ I(X;Y)$$ In the digital world,$p_X(x)$is a mass probability function and the mutual information is calculated as$H(Y)-H(Y|X)$, where$H(\cdot)$denotes the entropy. In the analog case, ... 3 You are mixing up two different notions that have little to do with each other. The use of spread-spectrum signaling is not in an effort to achieve (or even approach) the capacity of the (wideband) channel. Indeed, the spread signal uses only a small fraction of the capacity of the wideband channel, and the rest of the capacity is available for use by ... 3 Because the entropy represents information quantity, or if being measured in bit, the smallest number of bits per symbol we need to represent a source. The source$A$contains$4$equiprobable symbols hence it is obvious that we need$\log_2(4) = 2$bits per symbol to represent the source. The source$C$is simply$A$with its parity bits hence no new ... 3 I think you're confusing two different (but related) terms. Nyquist says that in a channel of bandwidth$B$you can transmit up to$2B$orthogonal pulses per second. So,$R_p \leq 2B$, where$R_p$is the pulse rate. To achieve$R_p = 2B\$, the pulses need to be sinc-shaped. Other, more practical pulses achieve slightly less than that. For example, raised ...

2

Boltzmann entropy shows uncertainty, with which a macrostate of a thermodynamic system describes its possible microstate. Except its application to thermodynamic systems (and multiplication by a physical constant), Boltzmann entropy (or more concretely, its generalization, namely, Gibbs entropy) is absolutely the same as Shannon entropy. They have similar ...

2

My survey paper on compression, "A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems", shows that most practical techniques on general benchmarks achieve compression ratio ~2X and some upto 4X, although higher potential (e.g. ~16X in some cases) exists (see Section 2.2). The reason for not achieving full potential is ...

2

The basic idea behind maximum entropy models is that you want to make the least assumption about the data. This is considered equivalent to retaining as much unpredictability as possible, as quantified by entropy. For more information I refer you to this Wikipedia article. Information and unpredictability are closely related. If you think of the signal as a ...

Only top voted, non community-wiki answers of a minimum length are eligible