Confusion about shannon's entropy of the grayscale image after lossy compression

I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB using a Matlab function which I wrote. Now about the entropy. The entropy of the original file was 6.93 and of the compressed file was 7.17. Actually the entropy of the compressed version must be lesser, right? I am confused. Can someone give me a intuition based explanation. And are the cases similar for lossy and lossless compression?

Shannon's entropy only makes sense when you define it over a probability distribution. You're using a probability distribution where each pixel is sampled independently from the histogram of pixel intensities. Clearly, real pictures don't come from such a distribution. If they did, they would all look like white noise.

Also note that individual elements don't have entropy. Only distributions have entropy. So if you have two images it doesn't make sense to compare their entropy. It only makes sense to compare the entropy of the random distributions that you sampled the images from.

• any digital image has a probability mass function, right? Here its a gray scale - 256 level image. So 0 to 255 has corresponding frequencies. – dexterdev May 16 '14 at 6:33
• I'll edit my response to make it more clear – Aaron May 16 '14 at 6:36
• I cannot digest your answer. Still image files have a distribution. Then why do we find entropy of image? You may be right, sometimes. So where can I get some references or so to understand this concept. – dexterdev May 17 '14 at 6:08
• If you ask what the entropy of a fair coin flip is the answer is 1 bit because there is a fifty percent chance of getting heads and a fifty percent chance of getting tails. – Aaron May 17 '14 at 6:40
• OK I know that definition. OK you are right, but I want some links or references to read on this. If you know something, please suggest me. – dexterdev May 17 '14 at 7:07

There is great confusion about the application of Shannon's original information entropy to signals and images, rather than the original's (one dimensional) text files.

Recently there has been a method presented to reconcile the different applications. Shannon's original objective was to find the minimally redundant representation of a text string. The entropy formula coincided with Gibbs formula for thermodynamic entropy. Unfortunately the coincidence has meant more than 50 years of confusion between the two concepts.

For image and signal processing we are nearly always interested in compression and minimum redundancy rather than thermodynamics or statistical physics.

Returning to the question, it is possible to estimate the entropy of an image. The entropy is normally give in terms of bits per pixel (bpp). A random noise 256 gray level image has near 8 bpp entropy. Measuring the entropy of a file is rather different because compression is achieved by reducing the file size. But we could say that the file size is S, and the image it represents has NxM pixels hence its average entropy is S/(NxM). However, the file may be further compressed using Huffman coding to a size S', hence its corresponding average image entropy is S'/(NxM).

If you wish to know more about Shannon information entropy of an image check out the arXiv preprint:

https://arxiv.org/abs/1609.01117