# Tag Info

## Hot answers tagged information-theory

11

Assuming a channel whose input at each time is a continuous random variable $X$ and its output is $Y=X+Z$, where $Z\sim\mathcal{N}(0,N)$ and $Z$ is independent of $X$, then $$C_{\text{CI-AWGN}}=\frac{1}{2}\log_2\left(1+\frac{P}{N}\right)$$ is the capacity of the continuous-input channel under the power constraint $$\mathsf{E}X^2\le P$$ The mutual information ...

8

It really depends on how much redundancy there is in the data. If all 250GB was just '0', then you could get fabulous levels of compression. This page shows some results for compression of English text. It compresses 2,988,578 bytes worth of text using various techniques. The top three are: 330,571 (88.94%), 333,759 (88.83%), and 352,722 (88.20%). Relating ...

7

When you say that the "information content may remain the same," do you mean the information in the total signal, or the information of the desired signal? Hopefully this will answer both cases. I know Shannon entropy much better than Kolmogorov so I'll use that, but hopefully the logic will translate. Let's say $X = S + N$ is your total signal ($X$), ...

7

I will try to tackle questions 1 and 4. 1) Entropy- It is desired that the entropy of the system be maximized. Maximizing entropy means no symbol is better than the others or we do not know what the next symbol / outcome would be. However, the formula states a negative sign before the summation of the probability logarithms. Thus, it means we are ...

7

They are not the same, but they're related. In particular, if you look at a computer memory holding $M$ "computer" bits, where each bit can be considered random and independent of all other bits, and there are roughly 50% of zeros, then the memory also holds roughly $M$ "information theory" bits. Of course, this is often not the case: computer bits are ...

7

You seem to have a number of misunderstandings, which I'll try to clarify while also trying to help with your questions. The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. While this is true, I think it's not ...

7

The capacity formula $$C = 0.5 \log (1+\frac{S}{N}) \tag{1}$$ is for discrete time channel. Assuming you have a sequence of data $\left\lbrace a_n \right\rbrace$ to send out, you need an orthonormal waveform set $\left\lbrace \phi_n(t) \right\rbrace$ for modulation. In linear modulation, whom M-ary modution belongs to, $\phi_n(t) = \phi(t - nT)$ where $T$ ...

6

Two additional intuitive takes on mutual information: When two random variables are independent the joint distribution $p(x, y)$ and the product of the marginal distributions $p(x)$ and $p(y)$ are identical. One could thus assess the degree of independent between two random variables by computing a probabilistic distance between $p(x) \times p(y)$ and $p(x, ... 6 Mutual Information by definition relates two random variables (RV) and it measures the dependence between the two RVs from the information content perspective i.e. the measure of amount of information contained by one RV about the other RV. And Mutual information is a symmetric quantity, i.e.,$I(X;Y) = I(Y; X)$. In case of a communication channel, the ... 6 1) Is there a connection between the modulation kind and the channel capacity? Channel capacity is usually defined as the number of information (usually measured by the number of bits) can be sent per channel use to get arbitrarily low number of errors, but we don't know exactly how (random coding). A channel use can be thought as a modulation symbol ... 5 I'm afraid that this will be an incomplete answer, but I'm giving it in the hopes that it is better than nothing. It is impossible to recover any information with a BER of 50% because, as information theory tells us, no information is getting through such a channel. You can see this intuitively if you realize that no matter what bit you send, 0 or 1, it ... 5 Let me give you a quick answer, focused on ways you can improve and research for a deeper knowledge by yourself. 1 - Kolmogorov and Shannon Entropies are related to measures of information and strongly related to one another. (See the wikipedia page for Kolmogorov complexity for further info.) 2 - Boltzmann entropy, which is an older concept, is a ... 5 1) Is there a connection between the modulation kind and the channel capacity? The capacity of a channel indicates the upper limit of how many bits can be transmitted per second over the channel with no errors (okay, technically it is "arbitrarily low number of errors", but it's basically the same thing). We do various things to try to get as close to that ... 4 There are two different kinds of channel models that are being confused here. In the binary symmetric channel, the inputs and the outputs are constrained to be$0$or$1$and the key parameter is the transition probability: the probability that an input$0$is changed to an output$1$or vice versa. The channel capacity$C$is a number between$0$and$1$... 4 No,because while mutual information tells you how much uncertainty can a random variable$X$remove from another random variable$Y$, the conditional entropy of$X$given$Y$tells you how much uncertainty remained in X after using the information (given by$I(X;Y)$) that$Y$gave it.$H(X|Y)=H(X)-I(X;Y)$That's why the conditional entropy depends on the ... 4 Shannon's Lossless Source Coding theorem states that the expected length of any code is lower bounded by the entropy of the probability distribution over the symbols for that code: $$H\left(X\right) \leq \mathbb{E}_{P_{X}}\left(l \circ f\right)$$ where$\circ$denotes the function composition operation, and$l$is the length of the codeword assigned by ... 3 A binary input - binary output channel which flips bits with equal probabilities is known as a Binary Symmetric Channel. (Image Credit: Wikipedia - Binary Symmetric Channel) The channel capacity$C$is the maximum (actually, a supremum) mutual information between the channel inputs and the channel outputs over input distributions. You can't get ... 3 What does mutual information (MI) convey? It indicates that there is a relationship between the two signals- i.e. that they are not independent. It could be that they are correlated, but the relationship does not have to be that simple. The higher the mutual information, the stronger the relationship (i.e. the more one of the signals can "tell" you about ... 3 My survey paper on compression, "A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems", shows that most practical techniques on general benchmarks achieve compression ratio ~2X and some upto 4X, although higher potential (e.g. ~16X in some cases) exists (see Section 2.2). The reason for not achieving full potential is ... 3 So,what exactly is Source entropy? When the paper talks about "source entropy" all they mean is the entropy of the information source. You can see that from the following passage in the paper- "Shannon showed that there must exist at least one encoding of the sequence generated by an information source which allows error-free transmission of the sequence ... 3 HINT: Compute the entropy$H$of your source and note that in your case it represents the number of bits required to represent each symbol. Then from$\quad H$[bits/symbol]$\timesx$[symbols/second] =$5$[bits/second] compute$x$. 3 There are two main factors when figuring out how many bits are transmitted per symbol (or "channel use"): the modulation and the error correction encoding. For instance, BPSK modulation with no encoding transmits 1 bit/symbol, while QPSK with no encoding transmits 2 bits/symbol. Higher order modulation schemes (e.g. 8-PSK, 16-QAM, 32-QAM, etc.) can ... 3 I'm not a mathematician so I won't pretend to claim that this is anything like a proof, but at an intuitive level I think that the fact that you can deterministically transform baseband to passband and back again by multiplying with complex sinusoids implies that the two are equivalent. 3 No, this is not correct. Consider the chain of two BSCs with error probabilities$p_1$,$p_2$as a single BSC with unknown error probability$p$. Now, we know that in overall no error occurs, when: neither BSC1 nor BSC2 create an error:$(1-p_1)(1-p_2)$both channels make an error:$p_1p_2$. Hence the overall probability of an error is $$p= 1-[(1-p_1)(... 3 To say that the input signal has a Gaussian distribution means that it is distributed as a Gaussian random variable. In practice, one relies on coding over multiple instances of the channel (in time) instead of relying on a Gaussian input distribution. There is a beautiful theory full of proofs that is beyond the scope of this answer (Information Theory). ... 3 The definition of channel capacity can be applied to either digital or analog cases. The meaning depends on how you calculate it. The definition is:$$C=\max_{p_X(x)}\ I(X;Y)$$In the digital world,$p_X(x)$is a mass probability function and the mutual information is calculated as$H(Y)-H(Y|X)$, where$H(\cdot)$denotes the entropy. In the analog case, ... 3 If I'm only using bandwidth B1, doesn't that mean I can filter some of the noise out? And if so, would I be able to get S/N down? Wouldn't that be an alternative to occupying the full channel bandwidth? Capacity increases linearly with bandwidth, but only logarithmically with SNR. So, increasing SNR by decreasing bandwidth is a bad idea if your noise isn't ... 3 You are mixing up two different notions that have little to do with each other. The use of spread-spectrum signaling is not in an effort to achieve (or even approach) the capacity of the (wideband) channel. Indeed, the spread signal uses only a small fraction of the capacity of the wideband channel, and the rest of the capacity is available for use by ... 3 Because the entropy represents information quantity, or if being measured in bit, the smallest number of bits per symbol we need to represent a source. The source$A$contains$4$equiprobable symbols hence it is obvious that we need$\log_2(4) = 2$bits per symbol to represent the source. The source$C$is simply$A$with its parity bits hence no new ... 3 I think you're confusing two different (but related) terms. Nyquist says that in a channel of bandwidth$B$you can transmit up to$2B$orthogonal pulses per second. So,$R_p \leq 2B$, where$R_p$is the pulse rate. To achieve$R_p = 2B\$, the pulses need to be sinc-shaped. Other, more practical pulses achieve slightly less than that. For example, raised ...

Only top voted, non community-wiki answers of a minimum length are eligible