Questions tagged [information-theory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
79 views

Word length for time series

This question is based on , "A Symbolic Representation of Time Series, with Implications for Streaming Algorithms". Let, $N = w\cdot l$ where $N$ is the number of data points in a time series ...
3
votes
1answer
239 views

Conceptual problem : numberof symbols for nonuniform distribution using entropy : how to determine block size?

A sequence of data of length $N$ can be subdivided into equal sixed blocks each of length (size) $l$. For each block, $w$, we can calculate the entropy known as the block entropy. Considering, entropy ...
0
votes
1answer
259 views

Shannon's entropy for non-equiprobable symbols

I am trying to understand the concept of Shannon's entropy and deciding the codelength. In first case, b is an array of 5 symbols. The symbol set is $\{1,2,...,8\}$....
1
vote
1answer
2k views

Entropy : do we prefer higher or lower entropy?

The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ...
0
votes
1answer
392 views

How to prove relationship between mutual information and differential entropy?

I need to prove that mutual information given by $$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$ is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am ...
0
votes
1answer
347 views

How to solve this entropy integral?

I am having the entropy integral below where $\mathbf x$ is a $N$ dimensional Gaussian vector having variance as $\mathbf P$ and mean zero, $$ -\int\frac {\exp\left(\frac{1}{2}\mathbf x^T\mathbf P^{-...
1
vote
1answer
96 views

Why are some LDPC codes more immune to noise than others?

When one analyzes an LDPC code, it is usual to plot the BER versus the $E_b / N_0$. My questions are: Why are some LDPC codes better than others in terms of noise immunity? After all, they are just ...
0
votes
1answer
2k views

How to explain this BER vs SNR behaviour?

I wrote a script in MATLAB to simulate an LDPC created using an algorithm of a colleague of mine. The parity check-matrix is regular and has dimensions $1012 \times 1518$ and the code rate is $R=1/3$. ...
0
votes
1answer
757 views

What is the overall capacity of two cascaded BSC channel?

Is it true:$C = C_1 * C_2$ where $C_1$ and $C_2$ are the two cascaded BSC which makes $C$.
0
votes
0answers
422 views

Differences between $\tt wentropy$ in MATLAB and entropy definition

MATLAB offer's a Shannon entropy calculation in its wavelet package: $$ H(X) = -\sum_{i=0}^{N-1} x_i^2\log(x_i^2) $$ I'm wondering what this formula means, since it has no relation with the ...
1
vote
1answer
358 views

Source and channel encoding

In information theory, there are two concepts that are widely used: source encoding and channel encoding. I'm afraid I may not be capable of expressing my doubts clearly, but the thing is that I don'...
1
vote
1answer
61 views

Discrete entropies

I've been given a problem where I need to find the entropy of two random variables. I can find part of the answer, but not all of it. I am given the following: $X$ is a uniformly distributed random ...
0
votes
1answer
304 views

Mechanics of a matrix Interleaver

I am having trouble understanding exactly how the matrix interleaver. I have read the following page from MathWorks. In it, it gives the following example where "123456" is interleaved as "142536." ...
2
votes
3answers
3k views

What is the theoretical probability of error for OOK transmission?

I am trying to simulate an optical wireless communication channel which uses OOK modulation. Looking at equation 14 in this literature, I found that: For electrical SNR at the receiver $$\textrm{...
0
votes
1answer
56 views

Entropy of a dynamical system and source code length

The entropy of a source $H(S)$, gives you the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the ...
1
vote
1answer
368 views

Why is $x$ not considered a primitive polynomial while being considered an irreducible polynomial?

In "Shu Lin, Daniel J. Costello-Error Control Coding (2nd Edition), Prentice Hall 2004" it is given that in $GF(2)$, if $f(x)$ is an irreducible polynomial of degree $m$ it divides $x^n+1$ where $n = ...
0
votes
1answer
97 views

What are bits in the context of channel capacity units?

'bits' are used with 2 different meanings. One can use 'bits' to mean the binary digits i.e. 1's and 0's. Bits are also the units of information in a event of discrete source. I think when channel ...
1
vote
1answer
49 views

Can channel capcity be explained in simple terms? [closed]

Can channel capacity be explained in simple terms - without mutual information and such probabilistic concepts? In its most general form, what are all the parameters that it depends on? What is its ...
1
vote
0answers
726 views

Is there any formula for coding gain of Turbo code?

I want to compare throughput under different MCS( modulation and coding scheme). We know Shannon capacity $C = B \log(1+SNR)$ can be used to calculate throughput by SNR. However, there is no MCS ...
2
votes
3answers
301 views

Is there a proof that equal bandwidths have equal information-carrying capacity?

Consider a continuous channel, band-limited to the $[f_{\text{min}},\,f_{\text{max}}]$ frequency window. A well-known statement says that, as far as the dependence on the frequencies, the information-...
1
vote
2answers
142 views

What are the fundamental reasons that suggest us to minimise transmission power for energy-efficient radio link design?

My question in short: Under which scenarios minimising transmission power is beneficial in terms of transmitter energy-efficiency? is there any fundamental reason to keep a low-power oriented design ...
3
votes
1answer
294 views

Mutual information of $ \infty $-PAM

I'm trying to compute using matlab the mutual information for an $ \infty $-PAM input (the limit of a very dense PAM constellation) for a range of snr and I got stuck. I'm working with a real-valued ...
7
votes
3answers
699 views

Is concept of “bit” in computer programming similar to the concept of “bit” in information theory?

until today I knew that one bit is a variable, or a space in memory that can hold a value of either One (high) or Zero (low). This is the concept I learned from studying computer programming, ...
1
vote
1answer
76 views

Can somebody help me with some very basic questions about the fundamentals of information theory?

I just started to study information content of the source and there are some very fundamental topics I am not really clear on. These might be a long question or a series of questions. Please help me. ...
3
votes
2answers
535 views

What is the entropy for these cases?

This question stems from an article, "Entropy estimation of symbol sequences" download link where the abstract mentions the need for using symbols in information theory. Random chains of ...
4
votes
1answer
270 views

Image registration / fusion optimal similarity metrics

I have a theoretical question about optimal similarity metrics for comparing data sets. In reading this linked paper, pp. 488-489 [1], I read the following 2 interesting statements. On page 488: ...
1
vote
1answer
202 views

Filters performance analysis

I am working on some experimental data which, at some point, need to be time-integrated and then high-pass filtered (to remove low frequency disturbancies introduced by integration and unwanted DC ...
0
votes
1answer
430 views

Relation between SER and the channel capacity

Is there a relation between Symbol Error Rate (SER) and the channel capacity? What is {Modulation Order x (1-SER) / symbol duration [bps]} called?
2
votes
1answer
310 views

Dominant eigenvectors of an unknown matrix

Do you have any idea about how we can find the principal eigenvectors of an unknown matrix ${H}$? The elements of $H$ are unknown in general. If you are familiar with channel estimation procedure in ...
2
votes
1answer
266 views

Effect of noise and entropy

Relation of Entropy and SNR : Based on this question and answer, I had another question that struck me and I am curious to know, if somebody can shed some light, on the following situation: $y= ...
3
votes
1answer
218 views

Difficulties in understanding mutual information concept

What does mutual information (MI) convey? If 2 signals are independent then MI is zero; What does this imply and mean in the case of mutual information of error, and in general definition of MI and ...
3
votes
1answer
117 views

Conceptual question on entropy and its relation to information

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in semi-...
2
votes
1answer
144 views

How to Detect a Inhomogeneity Region in Image

I have an image that includes object and background. However, the object appears some inhomogeneity region due to illumination. My work is that how to detect inhomogeneity region. Which is feature can ...
1
vote
2answers
707 views

Confusion about shannon's entropy of the grayscale image after lossy compression

I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB ...
0
votes
1answer
112 views

Entropy when having symbols with same probability

I am trying to work out this problem: Assume that we are transmitting data from a dictionary of 15 symbols. Let the probability of sending one symbol be the same as that of sending any other symbol....
1
vote
0answers
385 views

Extracting features from a signal: a 21st century approach?

I have a discrete signal of random length (let's say between 500 and 800 in time). This signal has an important information which I want to collect and to put it together in a feature vector to train ...
0
votes
1answer
44 views

Capacity computation in Costa sheme

I can not understand the following sentences, could you please anyone guide me ? "Comparing with the embedding in audio signals, the usable bandwidth is greatly reduced, from 22.05 kHz to 4 kHz. ...
0
votes
2answers
116 views

Transmission Line Information Loss

Suppose we have a communication channel that has a bit-error rate of 50% and the bits are flipped at random such that the receiver has no efficient way to tell which bits have been flipped and there ...
14
votes
2answers
4k views

Relationship between entropy and SNR

In general any form of enropy is defined as uncertainty or randomness. In a noisy environment, with increase in noise, I believe that entropy increases since we are more uncertain about the ...
3
votes
2answers
1k views

Meaning of channel capacity in the AWGN channel

I understand the concept of channel capacity as the maximal rate of the channel code I can apply without making a mistake in the receiver, in that sense the capacity is between 0 and 1. What I don't ...
2
votes
2answers
498 views

Conceptual definition of entropy

What is the conceptual difference between Kolmogorov-Sinai entropy, Shannon entropy and Boltzmann entropy? Are they interchangeable and mean the same? Where can I find a good lucid explanation about ...
6
votes
1answer
2k views

Why is mutual information symmetric but conditional entropy isn't?

Conditional entropy $H(X|Y)$ tells us how much the average uncertainty about a channel input $X$ is after observing channel output $Y$, and mutual information $I(X,Y)$ measures how much information ...
3
votes
1answer
445 views

Mathematical model of equivalent-time sampling, the resulting unevenly spaced periodic signal, and its interpretation

I sample a continuous signal $s(T)$ over time. This leads to $s[t]$, which depends on several factors of which some are (pseudo-) periodic. I am interested in the effect of one such periodic factors ...
1
vote
0answers
96 views

transmitting data using phase modulation of two sine waves passed through distortion

I have two sine signals with frequencies f1 and f2, and phase p1 and p2. I modulate the phase of these signals to transmit two symbols simultaneously, chosen from the set [0,1,2,3]. For example, I ...
2
votes
2answers
277 views

Entropy Of A Symbol

Does the entropy of a symbol represent the most optimal average number of bits that can be used to represent a symbol? For example take the example of tossing a coin: $$ H_{source} = -p_{head} ...
5
votes
1answer
182 views

Does any error correction code still work in such situation?

I'm looking for a kind of error correction code or solution that can correct my codeword in this case: My message holds $k$ bits, and $2k$ bits codeword (rate is $1/2$) is produced by the generator ...
8
votes
2answers
8k views

What are some typical lossless compression ratios?

A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I ...
2
votes
1answer
1k views

What is Shannon's source entropy

Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication ...
7
votes
1answer
182 views

Issues related to entropy

I have long been faced with the confusion regarding entropy and would be obliged if the following are answered in less technical jargon. Following the link Different kinds of entropy raises the ...
5
votes
1answer
161 views

Mutual Information, is this calculation correct or should I contact the author about fixing it?

Background information According to Cover's text book on information theory Mutual Information is calculated as: $$(1) I(W;C) = \Sigma_k\Sigma_i P(C_k,W_i)log(\frac{P(C_k,W_i)}{P(C_k)P(W_i)}) $$ ...