Why entropy value can be used as measure of contrast of an image?

I was looking for value that can measure a contrast of an image. And I found a couple of answers like this one https://stackoverflow.com/questions/13397394/how-to-measure-contrast-in-opencv-visual-c/13398126.

But I can't find any explanations WHY is that. No articles or links. So I wrote this code:

def shannon_entropy(img):

histogram = img.histogram()
histogram_length = sum(histogram)

samples_probability = [float(h) / histogram_length for h in histogram]

return -sum([p * math.log(p, 2) for p in samples_probability if p != 0])

And found the Shannon entropy for four images: - original image "image_1". Entropy is 7.82426730799067 - more contrast image "contrast". Entropy is 5.682661230050924 - much more contrast image "contrast2". Entropy is 2.1507540571698587 - less contrast image "contrastLow". Entropy is 6.565885023104716

I thought, that for the last image entropy will be the highest, but no. And it completely baffled me.

Can somebody explain:

1. Why entropy is a measure of contrast?
2. What's wrong with my images or calculations?
• Why did you think that for the last image entropy would be the highest? To me, all this is "as expected"; aside from your purely black and white image, which pretty much by definition must have an entropy < 1 bit/pixel. Your entropy calculation is wrong! – Marcus Müller May 16 at 21:00

Hard facts first.

Why entropy value can be used as measure of contrast of an image?

It can't. Entropy is not a function of contrast.

I'll show why. Time for a bit of basic Information Theory¹:

What entropy is, and what happens to it under transformations

Definitions

Let's define $$H(·)$$ as the entropy of a source; imagine $$X$$ is your source picture, $$Y$$ is the picture after you subjected it to some operation $$F:\, F(X)=Y$$ (e.g. contrast spreading, or compression, or adding random noise).

Intuition of Entropy

$$H(X)$$ is a measure for the amount of information in the source image. It's hence a measure for the "uncertainty" of the value of each pixel; if that picture was just one intensity all over, the uncertainty would be low, and so the information, and thus the information.

Let's assume your input image has enough contrast to at least make use of 8 bits of dynamic range. Then, the highest entropy you could get would be 8 bit per pixel – in case all pixel values are present equally likely. That's how I know your entropy calculation is broken – a black and white image only has two values, and the discrete uniform distribution over 2 values has exactly 1 bit of entropy, and any other distribution has less. You'll need to fix your entropy calculation!

Contrast

However, contrast doesn't care about the full distribution – there's different contrast definitions/measures, but they typically only consider the first and second statistical moment. Many distributions can have the same variance – but only one is the uniform distribution that maximizes entropy. From that alone you can see that while there is a relation between contrast and entropy, it's not a direct relation:

You just can't have high entropy with very low contrast. However, high contrast doesn't say much about entropy.

Applying image transformations

Consider your contrast enhancements:

$$H(X|Y)$$ is the amount of the uncertainty that remains about the original image when you apply the operation $$Y=F(X)$$.

When your operation $$F$$ is reversible (e.g. if you just invert the pixel values, or if you scale them by 2 without clipping because an original value was larger than half the maximum pixel value), then there's nothing unclear about the original image (just apply the "reverse" operation), and thus $$H(X|F(X)=Y)=0$$. If you think about it, it's pretty intuitive: A revertible operation doesn't change the probabilities, it just changes the values that have these probabilities.

Information in the face of change

We call the information we get about the source when looking at our transformed image the mutual information between $$X$$ and $$Y$$, $$I(X;Y)$$. It's the difference between the source $$X$$'s entropy and the conditional entropy $$H(X|Y)$$, i.e. $$I(X;Y) = H(Y) - H(Y|X)$$! In the case that the operation is revertible, as you can see, $$H(X|Y)=0\implies I(X;Y)=H(X)$$.

And, since we know the deterministic operation $$F$$, the uncertainty about what $$Y$$ looks like if we know $$X$$ is $$H(Y|X)=0$$, too, so that $$H(Y)=I(X;Y)$$, and conclusively, $$H(Y)=H(X)$$. In other words, when you do an invertible operation, you can't increase or reduce entropy.

Now, there's the thing: Your contrast operations aren't revertible, simply because

1. your image has finite bit depth, so some rounding happens and different values in $$X$$ get mapped to the same value in $$Y$$,
2. your image has finite bit depth, so there is some clipping when you reach the extreme values.

That means that you get a $$H(X|Y) > 0$$; from that we can see that $$I(X;Y). However, since the operation you apply is still deterministic (i.e. you can calculate $$Y=F(X)$$ based on $$X$$, it's always the same), the uncertainty about $$Y$$ knowing $$X$$ is zero, $$H(Y|X) = 0$$, and therefore

$$H(Y)=I(X;Y)+H(Y|X)=I(X;Y) + 0 = I(X;Y) < H(X),$$

which is exactly what you see: all your contrast-stretching and -reducing operations decrease the entropy compared to the original image.

What we've learned today

1. Having high contrast is a necessary, not a sufficient condition for high entropy
2. Entropy is limited by the number of values occupied
3. Entropy is invariant to revertible (that implies deterministic) (image) operations
4. Entropy is reduced by deterministic non-revertible operations

Corollary:

1. if you want to increase an image's entropy, you'll need to do something non-deterministic, like adding noise.
2. Information theory is fun and you should definitely study it.

PS.

Here's an entropy function that makes sense; I don't know what kind of data type img is, but I assume you can just go through all its values (it's iterable).

from collections import Counter

def shannon_entropy(img):
ld = lambda x: math.log(x, 2)
value_counter = Counter(img)
N = len(img)
H = 0
for count in value_counter.values(): #this gives the counts of values
p = float(count)/N
H = H - p*ld(p)
return H

¹ yes, that is a pun.

• First fo all, there was a pretty funny pun. I want to say, that it was a realy good introduction to image entoropy, thanks a lot. Now I know alot of interesting things. And thanks for your code example too. But now when I know that entropy can't estimate an image contrast, I have another question, that led me to question about entropy: how can I measure an Image contrast? If you got any clues where to start, please, let me know. P.S. I can't add points to your answer, so no offence, i just don't got enough rep :) – Shadr May 17 at 9:23
• :) I'd simply go with the root mean square (RMS) of the pixel intensities: $$C(X)=\sqrt{\frac1{\#X}\sum\limits_{x\in X}{(x - \bar x)^2}}\text,$$ where $X$ is the image, $\#X$ is the number of pixels in that image, and $\bar x$ is the average pixel value. – Marcus Müller May 17 at 11:17
• Thanks a lot again – Shadr May 17 at 12:42