Questions tagged [information-theory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
2answers
140 views

What are the fundamental reasons that suggest us to minimise transmission power for energy-efficient radio link design?

My question in short: Under which scenarios minimising transmission power is beneficial in terms of transmitter energy-efficiency? is there any fundamental reason to keep a low-power oriented design ...
1
vote
1answer
74 views

Can somebody help me with some very basic questions about the fundamentals of information theory?

I just started to study information content of the source and there are some very fundamental topics I am not really clear on. These might be a long question or a series of questions. Please help me. ...
1
vote
1answer
200 views

Filters performance analysis

I am working on some experimental data which, at some point, need to be time-integrated and then high-pass filtered (to remove low frequency disturbancies introduced by integration and unwanted DC ...
1
vote
1answer
47 views

Understanding Asymptotic Equipartition Property

I have some problems in understanding the precise meaning of the Asymptotic Equipartition Property, related to a large number n of independent and identically distributed random variables (X1, X2, ...,...
1
vote
0answers
28 views

Pearson correlation coefficient vs mutual information

I recently computed both the pearson coefficient and the mutual information of two time series as well as the corresponding error bars, and what I find out in the end is a significant level of ...
1
vote
0answers
29 views

Does Mutual information utilize spatial information to calculate image similarity?

To calculate images similarity based on Mutual Information (MI), histogram and join histogram is required. In the literature, there are 2 contradict opinions about MI operation. 1. MI does not utilize ...
1
vote
0answers
40 views

Capacity region of 'symbol asynchronous' Gaussian multiple access channel

I was interested to learn what theoretical limits have been determined so far for the 'multiple access channel' with additive Gaussian noise. This brought me to the 1989 paper by Sergio Verdu, named "...
1
vote
0answers
164 views

What is the difference between entropy and entropy rate and which one is important in information theory? [duplicate]

By entropy I understand the uncertainty or randomness. But if the uncertainty is high then what is its implication and advantage in transmitting information or as an information source or is high ...
1
vote
0answers
267 views

SNR for channel after equalization

Suppose I have a channel described by $h[n]$ or $H(z)$ over which I send a simple PAM2 signal. After the channel AWGN is added. What is the best possible SNR with equalization? As suggested in the ...
1
vote
0answers
107 views

Intuitive interpretation of transform coding

I am currently studying source coding of waveform in a course of Information Theory and Coding. More precisely, I am trying to understand, both mathematically and intuitively, transform coding and its ...
1
vote
0answers
694 views

Is there any formula for coding gain of Turbo code?

I want to compare throughput under different MCS( modulation and coding scheme). We know Shannon capacity $C = B \log(1+SNR)$ can be used to calculate throughput by SNR. However, there is no MCS ...
1
vote
0answers
384 views

Extracting features from a signal: a 21st century approach?

I have a discrete signal of random length (let's say between 500 and 800 in time). This signal has an important information which I want to collect and to put it together in a feature vector to train ...
1
vote
0answers
96 views

transmitting data using phase modulation of two sine waves passed through distortion

I have two sine signals with frequencies f1 and f2, and phase p1 and p2. I modulate the phase of these signals to transmit two symbols simultaneously, chosen from the set [0,1,2,3]. For example, I ...
0
votes
1answer
1k views

Entropy : do we prefer higher or lower entropy?

The entropy of a source $H(S)$ gives the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the source. ...
0
votes
1answer
714 views

What is the overall capacity of two cascaded BSC channel?

Is it true:$C = C_1 * C_2$ where $C_1$ and $C_2$ are the two cascaded BSC which makes $C$.
0
votes
1answer
101 views

Entropy when having symbols with same probability

I am trying to work out this problem: Assume that we are transmitting data from a dictionary of 15 symbols. Let the probability of sending one symbol be the same as that of sending any other symbol....
0
votes
1answer
32 views

Wireless communications outage

For communication systems, there is usually a target minimum rate $R$ bits per second. An outage occurs if the actual rate ever falls below $R$. Since $\frac{1}{2}\text{log}_2(1+\text{SNR})$, ...
0
votes
1answer
282 views

coding gain and shaping gain in SCMA

It is said that sparse code multiple access (SCMA) has the advantage of higher shaping and coding gain over LDS-CDMA due to the multidimensional constellation. What exactly is the shaping gain and how ...
0
votes
1answer
276 views

Mechanics of a matrix Interleaver

I am having trouble understanding exactly how the matrix interleaver. I have read the following page from MathWorks. In it, it gives the following example where "123456" is interleaved as "142536." ...
0
votes
1answer
56 views

Entropy of a dynamical system and source code length

The entropy of a source $H(S)$, gives you the average codeword length to encode a given source alphabet. i.e. it is the average number of bits per symbol required to encode the information in the ...
0
votes
1answer
38 views

Information theory vs system theory ( commented in Determining linearity, causality, memory, and time invariability from a picture of the signal)

I have seen this question : At the comment asked about : Is this "information theory" or "systems theory"? – The Photon So i liked to know why this comment asked? Why the system theory vs ...
0
votes
1answer
30 views

Example of Entropy and Channel Capacity Computation

Can you help me on verifying if this computation of entropy is correct and on understanding its meaning? I am not sure of the result especially because it is equal to 0: it means that we cannot ...
0
votes
2answers
51 views

Definition of Perfect Code (and example of Hamming Code)

Let's consider the following sentences about coding a message before transmitting it in a communication system (here you find the complete pdf) First question: can you explain me these two ...
0
votes
1answer
19 views

How good is the Leech lattice for AWGN coding or MSE quantization of Gaussian sources?

Anyone know where to find graphs showing how well it works vs other schemes? thanks
0
votes
1answer
583 views

How do I measure the amount of information within an image?

I'm trying to find some information threshold that is required for performing detection of objects in images. However, I'm not sure how exactly to quantify the amount of information contained within ...
0
votes
1answer
677 views

White noise and entropy confusion about concepts

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the ...
0
votes
2answers
42 views

Rate in CDMA setup

I am working on a CDMA system with only 4 chips/symbol. I want to use $R=\frac{1}{2}log_2(1+SNR)$ chips/channel use to calculate what $SNR$ I require to successfully receive a given transmission. In ...
0
votes
1answer
34 views

Is there some literature to understand finite state ISI channel?

I am trying to understand channels with memory, where the state of the channel gives the past input information. I want to delve further into capacity for channels with ISI. I came across a ...
0
votes
1answer
175 views

Given a noise-free channel with limited bandwidth, why is the channel capacity infinite? [duplicate]

In the Shannon Hartley theorem, if N = 0 but B is limited, the theorem tells us that C is infinite. Why is this intuitively true? If bandwidth is limited, I can only post so many pulses per second ...
0
votes
1answer
249 views

Shannon's entropy for non-equiprobable symbols

I am trying to understand the concept of Shannon's entropy and deciding the codelength. In first case, b is an array of 5 symbols. The symbol set is $\{1,2,...,8\}$....
0
votes
1answer
2k views

How to explain this BER vs SNR behaviour?

I wrote a script in MATLAB to simulate an LDPC created using an algorithm of a colleague of mine. The parity check-matrix is regular and has dimensions $1012 \times 1518$ and the code rate is $R=1/3$. ...
0
votes
1answer
88 views

What are bits in the context of channel capacity units?

'bits' are used with 2 different meanings. One can use 'bits' to mean the binary digits i.e. 1's and 0's. Bits are also the units of information in a event of discrete source. I think when channel ...
0
votes
1answer
380 views

Relation between SER and the channel capacity

Is there a relation between Symbol Error Rate (SER) and the channel capacity? What is {Modulation Order x (1-SER) / symbol duration [bps]} called?
0
votes
1answer
44 views

Capacity computation in Costa sheme

I can not understand the following sentences, could you please anyone guide me ? "Comparing with the embedding in audio signals, the usable bandwidth is greatly reduced, from 22.05 kHz to 4 kHz. ...
0
votes
2answers
115 views

Transmission Line Information Loss

Suppose we have a communication channel that has a bit-error rate of 50% and the bits are flipped at random such that the receiver has no efficient way to tell which bits have been flipped and there ...
0
votes
0answers
10 views

Mutual information smaller on data subsets compared to entire data set

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al [1]. Lets consider two features $X$ and $Y$, and the ...
0
votes
0answers
15 views

About convergence of KL divergence: if the two probability distributions are type, does the law of large number work?

If I pick $N$ samples from $P_X$ and $P_Y$, they are two independent discrete distributions. $X_1,X_2,\ldots,X_N$ are drawn i.i.d from $P_X$, and $Y_1,\ldots,Y_N$ are drawn i.i.d from $P_Y$. I got $...
0
votes
0answers
20 views

Channel Impulse Response is zero mean Gaussian random variable?

In the Paper "Key Generation From Wireless Channels" the channel estimation is given as: $\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||S_B||^2}$, $\tilde{h}_{1,A} = \sigma_1^2 + \frac{\sigma^2}{||...
0
votes
0answers
35 views

Information entropy of Poisson noise

I would like to calculate the information entropy of the channel with additive Poisson noise. The model consists of an input signal, an optical image, an image acquisition device, and the output ...
0
votes
2answers
53 views

When channels are in a cascade and are independent is the capacity the sum?

Just wondering if two channels are independent then is the overall capacity, the sum of the individual capacities? For example suppose I have two BEC where the probability of erasure is $\varepsilon$...
0
votes
1answer
34 views

Band-limited signal recovery with finite bit depth

The sampling theorem tells us that a signal with no frequencies above $f$ can be completely described by sampling it a rate of $2f$. However, the theorem makes no reference to quantization, and so I ...
0
votes
0answers
45 views

Capacity achieving codes for BPSK at low SNR

I don't know a lot about comm system design. I have an AWGN channel and low SNR (-3dB). Bandwidth is constrained to some maximum, B. I would like to know good choices of modulation and coding to ...
0
votes
1answer
66 views

Proof that DFT does not require more than N points

I'm trying to show how the discrete Fourier transform (DFT) arises from the equation for the continuous-time Fourier Transform. I've run into an interesting caveat which I can't seem to find an ...
0
votes
1answer
46 views

What is the Asymptotic Equipartition Property (AEP)?

I am currently studying about Polar Codes in 5G standard and while reading my paper I found something called AEP which is required for channel coding. I surfed the web but didn't found a satisfying ...
0
votes
1answer
49 views

Relationship between spectral efficiency and PAR

Low PAR means less fluctiations in the magnitude of a signal and hence analog signal processing (e.g., power amplifier) is tremendously reduced. For example, constant envelope modulation schemes like ...
0
votes
1answer
74 views

Word length for time series

This question is based on , "A Symbolic Representation of Time Series, with Implications for Streaming Algorithms". Let, $N = w\cdot l$ where $N$ is the number of data points in a time series ...
0
votes
1answer
375 views

How to prove relationship between mutual information and differential entropy?

I need to prove that mutual information given by $$I(X;Y)=\int_{x,y}f(x,y) \log_2 \left( \frac{\left (f(x,y)\right)}{f(x) f(y)}\right) \, dx \, dy$$ is equivalent to $I(X;Y)=H(Y) - H(Y|X)$ I am ...
0
votes
1answer
343 views

How to solve this entropy integral?

I am having the entropy integral below where $\mathbf x$ is a $N$ dimensional Gaussian vector having variance as $\mathbf P$ and mean zero, $$ -\int\frac {\exp\left(\frac{1}{2}\mathbf x^T\mathbf P^{-...
0
votes
0answers
399 views

Differences between $\tt wentropy$ in MATLAB and entropy definition

MATLAB offer's a Shannon entropy calculation in its wavelet package: $$ H(X) = -\sum_{i=0}^{N-1} x_i^2\log(x_i^2) $$ I'm wondering what this formula means, since it has no relation with the ...
-1
votes
1answer
50 views

Bandwidth and probability of continuous signal

I'm a little bit confused as to how to calculate the bandwidth of a signal. A question has me believing that it is correlated with the probability distribution. I am however not sure of this. ...

1
2