I have a .pgm format 480 * 640 gray scale image named 'columns.pgm'. Using PCA (principal component analysis), by preserving 40 principal components I compressed original image with 1218 KB to 301 KB using a Matlab function which I wrote. Now about the entropy. The entropy of the original file was 6.93 and of the compressed file was 7.17. Actually the entropy of the compressed version must be lesser, right? I am confused. Can someone give me a intuition based explanation. And are the cases similar for lossy and lossless compression?
Shannon's entropy only makes sense when you define it over a probability distribution. You're using a probability distribution where each pixel is sampled independently from the histogram of pixel intensities. Clearly, real pictures don't come from such a distribution. If they did, they would all look like white noise.
Also note that individual elements don't have entropy. Only distributions have entropy. So if you have two images it doesn't make sense to compare their entropy. It only makes sense to compare the entropy of the random distributions that you sampled the images from.
There is great confusion about the application of Shannon's original information entropy to signals and images, rather than the original's (one dimensional) text files.
Recently there has been a method presented to reconcile the different applications. Shannon's original objective was to find the minimally redundant representation of a text string. The entropy formula coincided with Gibbs formula for thermodynamic entropy. Unfortunately the coincidence has meant more than 50 years of confusion between the two concepts.
For image and signal processing we are nearly always interested in compression and minimum redundancy rather than thermodynamics or statistical physics.
Returning to the question, it is possible to estimate the entropy of an image. The entropy is normally give in terms of bits per pixel (bpp). A random noise 256 gray level image has near 8 bpp entropy. Measuring the entropy of a file is rather different because compression is achieved by reducing the file size. But we could say that the file size is S, and the image it represents has NxM pixels hence its average entropy is S/(NxM). However, the file may be further compressed using Huffman coding to a size S', hence its corresponding average image entropy is S'/(NxM).
If you wish to know more about Shannon information entropy of an image check out the arXiv preprint: