Noise problem when transforming and quantizing difference between images

Question

I have a program which does forward and inverse DCT and quantization and dequantization using JPEG matrix. If I apply it to a single image it works fine, but if I try to apply it to a difference between current and previous frame it gets extremely noisy. Album with examples ("high frequency components" line in descriptions means any component starting from anti-diagonal and beyond it).

What is the correct way to compress the difference between two images?

uncompressed

Compressed, high frequency components removed

Difference between this frame and previous one was coded. High frequency components kept

Difference between this frame and previous one was coded. High frequency components removed

Answer 1

JPEG was developed for photographic content, not for difference-of-pictures content. Therefore, the fact that it's not overly applicable to the difference between successive frames isn't surprising.

In your specific quantization approach, where you eradicate high frequencies, that's actually even pretty intuitive: The difference between successive frames are probably small, and fine-grained. Of course, when you erase high frequency content, you lose most of the difference information and get a large error. So, I'd call this an indication that your high-frequency deletion works, and little more.

So, wrong approach to video coding; you could use JPEG for I-frames and code the differences to previous frames with some sort of linear predictor. Or, look at existing video codecs, and adapt some of the methods used there - in the case of grass, which presumably is blowing in the wind in a pretty uniform direction with smaller local variations, motion prediction coding might be of special interest to you.