Timeline for Entropy : do we prefer higher or lower entropy?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 30, 2017 at 20:39 | vote | accept | SKM | ||
| Jan 30, 2017 at 20:14 | comment | added | MBaz | @SKM Sigh... that paper addresses a different problem, which is estimating the entropy of correlated sources by looking at finite output sequences. | |
| Jan 30, 2017 at 20:04 | comment | added | SKM | If entropy of a dynamical system emitting messages is $H$ and the sequence length of the message is $N$ then the code length is $n (<N)$ because entropy converges to $H$ for $n$. But calculation of the probability to be plugged into Shannon's entropy equation is based on the entire sequence length, $N$ and not small $n$. So, is it safe to say that $n$ is selected based on the bit length for which entropy reaches the entropy of the dynamical system? | |
| Jan 30, 2017 at 20:02 | comment | added | SKM | perso.ens-lyon.fr/pierre.borgnat/MASTER2/… In this paper Fig.1 shows the plot of entropy vs. length of sequence $N$ for a dynamical system called the Henon map. The entropy of this dynamical system is known = log(2) where $k=2$ denoting number of unique symbols / alphabet. The graph can tell us the block size $n$ which always is close to the entropy of the system. My confusion and this question arises from this approach in the paper. | |
| Jan 30, 2017 at 19:21 | comment | added | MBaz | @SKM (A) You need to specify a criterion for selection. If you want to an efficient encoder, in general you need to jointly encode many messages. (B) If I'm understanding correctly, for variable length encoding, you need Kraft's inequality (en.wikipedia.org/wiki/Kraft_inequality). In any case, I suggest studying the Shannon-Fano algorithm, and then Huffman's. I think you'll find them very useful to understand these concepts. | |
| Jan 30, 2017 at 18:37 | comment | added | SKM | (B) If entropy of either source is $H$, I want to know if there is a way to determine what should be $n$ so that I can transmit few bits $n < N$ or block of bits from a sequence of message of length $N$ based on knowledge of $H$ (something like varaible length encoding)? Could you please clarify these points? | |
| Jan 30, 2017 at 18:36 | comment | added | SKM | @MBaz: Thank you for your reply. I would like to reframe two points as it may not be clear earlier. (A) Considering two sources source 1 and source 2. The entropy of source 1 is higher than source 2. Which one do we prefer to use as the source to generate messages? Based on your answer, I should select source 2 since that would give better encoders. Is my understanding correct? | |
| Jan 30, 2017 at 15:46 | comment | added | MBaz | @Tendero You're right: it is the entropy of the source that increases. Thanks for catching that -- fixed! | |
| Jan 30, 2017 at 15:44 | history | edited | MBaz | CC BY-SA 3.0 |
Fix mistake
|
| S Jan 30, 2017 at 15:42 | history | suggested | Tendero | CC BY-SA 3.0 |
Improved formatting inside quotes
|
| Jan 30, 2017 at 15:28 | comment | added | Tendero | This answer was awesome to read, great job. I have one doubt, however. You state that "Shannon says that we need to group source messages together to achieve larger encoding efficiency. This process will also increase the entropy of each message", but above you said that "$S_1$ is a message, so it cannot have an entropy". Is it me or is there a contradiction there? I didn't understand the first sentence, where you refer to the entropy of each message. | |
| Jan 30, 2017 at 15:21 | review | Suggested edits | |||
| S Jan 30, 2017 at 15:42 | |||||
| Jan 30, 2017 at 14:56 | history | answered | MBaz | CC BY-SA 3.0 |