1
$\begingroup$

I have a hardware device that outputs a normal distribution of readings. It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers.

I'm trying to determine a theoretical rate of entropy generation per reading, so for example 3.4 bits /reading or the like. Is there some formula I can make use of?

PS. As entropy is actually generated by the observer rather than by the process itself, I'm particularly interested in the significance of the bit depth of the individual readings. I.E. what happens to the entropy rate as I alter the sample depth? I feel that I need to elucidate this point with a (facile) example...

Imagine measuring men's foot sizes, and you're using whole inches. They would therefore range from 1 inch to 12 inches. That's 12 values and thus contain 3.6 bits of entropy /sample if feet were uniformly distributed in size. Now if you change to a metric rule ranging from 1mm to 305mm, you'll be acquiring entropy at the rate of 8.3 bits /sample. This demonstrates that whilst God determines the size of men's feet, the entropy generated from measuring them is actually determined by the type of rule (or analogue to digital converter sample depth).

This would all be very simple if my actual case was uniformly distributed, but unfortunately it's a normal distribution.

Improve this question
$\endgroup$
1
$\begingroup$

It's (fitted) parameters are mean = 35 units, std.dev. = 8 units and I read the values as 8 bit integers.

The choice of parameters hint at a normal distribution. Is this the case? You might want to obtain a histogram and perhaps substitute mean and standard deviation with modal mean and range. This will also help with an estimation of entropy.

I'm trying to determine a theoretical rate of entropy generation per reading, so for example 3.4 bits /reading or the like.

A first step would be to estimate the Shannon Entropy. In this case, your alphabet is the range of 256 symbols that are possible to be formed with an 8-bit integer and you would be estimating the entropy of a source that generates these symbols with probabilities dictated by the histogram of a long enough sequence of symbols.

If these 256 symbols appeared with equal probabilities in your sequence then you would need all 8-bits to represent them. However, if a subset of symbols appeared with higher probabilities than others then, on average, it would be possible to represent this sequence with less bits per symbol. To determine Shannon's Entropy, obtain the histogram of a long enough sequence, divide each entry by the sum of all entries so that it is now a probability density function (PDF), i.e. it has a total sum of 1.0, and then calculate:

$$H(X) = -\sum_{k=0}^{N} x_i \log_2(x_i)$$

Where $X$ is your PDF (derived above) and $N$ is the length of the histogram.

PS. As entropy is actually generated by the observer rather than by the process itself, I'm particularly interested in the significance of the bit depth of the individual readings.

Entropy is a characteristic of the source. This particular way of deriving entropy applies to a particular kind of source called ergodic and this is why I am also emphasising the "long enough sequence" (For more information please see this and specifically section 5.)

I am not sure what you mean by "significance of the bit depth". The idea is this: Suppose that you have an alphabet with 256 symbols. Obviously, you need 8-bits to represent them. But, if you were to discover that your source is actually generating numbers in the range $0..32$ for 98% of the time and a single 256 value for the rest 2% of the total length of a given sequence, then, for 98% of the time you would be wasting 3 bits, because $2^{5} = 32$ and you would need all 8 bits for only 2% of the time. Imagine these bits on a sequence:

00000011 00000100 00001110 00001001 00001100 10001101 00011001 00001110

If you could somehow squeeze those 3 bits out of your stream, you could save time of transmission (therefore fit more data through a channel) or storage space.

This is why figures such as 2.5 bits make sense. You can't define half a bit BUT over two symbols (i.e. 3 bits + 3 bits = 6 bits) you would be saving 1 bit (because 2.5 bits + 2.5 bits = 5bits).

For more information on encodings that allow you to effect exactly that "squeezing", please see variable length coding.

Hope this helps.

EDIT: If you are absolutely sure that the source is normally distributed, then substitute $X$ for an expression of the normal PDF and the sum for an integral over the sufficiently probable figures

Improve this answer
$\endgroup$
6
  • $\begingroup$ Yes it's a normal distribution. $\endgroup$ – Paul Uszak Sep 27 '17 at 12:17
  • $\begingroup$ I was hoping to obtain the theoretical entropy rate from the mean and std.dev. I don't want to measure it empirically - that's easy! $\endgroup$ – Paul Uszak Sep 27 '17 at 12:19
  • $\begingroup$ Entropy is definitely not a characteristic of the source. The observer creates the entropy. Since the distribution is continuous, I could measure it to 16 bit accuracy which immediately increases the entropy rate. Or 32 bit. And the rate will always be a decimal number rather than integer. There are many more decimals than integers and 12.88 bits /sample is easily possible. The bit depth is the crux of my question, hence the PS. $\endgroup$ – Paul Uszak Sep 27 '17 at 12:23
  • $\begingroup$ @PaulUszak I would not call that "easy" because there are lots of things that come into play. As far as entropy is concerned, I think that you are highlighting a matter that has to do with the calculation, not the property of entropy which is inherent to the source, because, as we see here, it depends on the statistics of the source itself, not on the way this source is perceived by an observer. This is also why we say that variable length encoding can compress an i.i.d sequence arbitrarily close to the entropy of its source. $\endgroup$ – A_A Sep 27 '17 at 12:58
  • $\begingroup$ You're telling me :-) I've been trying to get an answer to this for 18 months across 4 or 5 SE sites, (I forget exactly how many). I have to admit that I'm at a loss as to where to go next with this... $\endgroup$ – Paul Uszak Sep 27 '17 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.