# Extremely computationally intensive image/video compression schemes which offer better compression than industry standard JPEG/H.264

I am trying to search for image/video compression schemes which offer superior compression performance (holding image quality as constant) when compared to widely adopted industry standards (i.e. JPEG/H.264), but for reasons of them being either too computationally or resource intensive, implementing them on CPU-based workstations is infeasible and therefore unadopted in common standards.

I could only think of fractal compression as a good example. Does the community know of any more of such examples?

• As far as images go, there are more sophisticated algorithms than JPEG, such as JPEG2000. JPEG2000 provides improved compression ratios, but I'm not sure that its lack of adoption is due to computational complexity. More so, JPEG is considered to be "good enough" for most applications, so there's no real motivation to use anything better (especially since disk storage is so plentiful in modern systems). Those who care about utmost image quality, like professional photographers, often shun compression altogether, relying upon raw captures of image sensor outputs. Jan 15, 2012 at 17:52
• I think the quality of JPEG depends more on how much data you throw away than on how much time you spend computing it. Jan 15, 2012 at 21:19
• Principal Component Analysis / KLT/ Hotelling transforms can do a much better job than the DCT at representing a signal with a small number of basis functions, but are quite computationally expensive. I'm not sure if anyone's actually done full up compression schemes with them though. Jan 16, 2012 at 22:25
• @eglaser: Care to elaborate on PCA/KLT/Hotelling? It would be good if you could post your reply as an answer. Jan 18, 2012 at 14:26
• @eglaser KLT is better than DCT in terms of energy compactness. However, it doesn't necessarily means it is a better codec; this is because KLT for each image/video piece is unique and hence, not only you have to transmit resultant co-efficients, you also have to send the kernel as well. Jan 19, 2012 at 18:12

There are various aspects of compression - whether you are referring to image or video. (May be audio is totally different so i am not referring this here).

If you really look at the history of compression standard, when they were first formed - MPEG1 real time encoders were rare. CPU at that time itself was not enough to make it real time; however, it was not the processing complexity (alone) that stopped more complexity algorithms to be used under codecs. (Probably the primary factor that was responsible for inclusion/exclusion of certain algorithms were a. patents, b. feasibility of implementation).

However, i would like to re-phrase the question a bit more simpler:

What other research in compression is available which is still to come to market (or didn't come at all but might be worth for some applications)?

Here are some points to look at:

1. Better transform?

DCT by far has been universal transform across all codecs. However, there are many research papers which suggest that Wavelets could be better. Indeed the only JPEG2000 included wavelets - but that standard itself wasn't big hit. Probably it was better than DCT in some way; specially there are claims that Wavelets outperformce DCT [See: Ref 1]

I think wavelet is computationally expensive over DCT (partly because 8x8 DCT implementations are highly optimized). But i guess i didn't really flew that much because from the market perspective because DCT by the time was far more wide spread and the difference in improvement was not so substantial for people to switch.

2. Multi hypothesis motion compensation
While MPEG 2 was happy enough with $B$ pictures, extensive research from Bern Girod [See Ref 2] proved that - the concept can be extended way beyond this. Indeed H.264 actually has made great generalization on multiple target based motion compensaion - basically one can store past 16 pictures (compare to 2 in MPEG2) and indeed the prediction can be extremely high.

While, this is already part of the standard in H.264 - this is still not fully exploited. There is a need to make such extensively large search for motion estimation practically viable. Most H.264 encoder are still nascent in exploiting full potential of this.

3. Scalable Image/Video compression
Scalable Image and video compression is again a holy grail of how multiple description encoding can be done for image and video. Basically the same Image or video can be encoded at multiple Resolution, quantization (quality), or frame rate. These produces different layers of quality vs. bit rate right within the same stream - and the streaming server can actually adapt the bitrate and quality dynamically to best fit the video in the quality. See [Ref 3], [Ref 4], [Ref 5]

Of this, only real success is progressive JPEG which is practically useful for websites and indeed useful. Again while the promise of dynamically adapting bit stream is great but it works only if there are practical systems where estimating such bit rate is possible and type of adaptation can serve some real purpose. However, in video domain, in spite the support of this in the standards (MPEG2, MPEG4 and H.264) - there is hardly any usage of this.

4. Object Video
This is perhaps ultimate holy grail of Compression. Object video [Ref 6] as initiated by MPEG standard body itself however, this was also backed by extensive research.

The basic idea is to segment the video in terms of distinct objects and later on each such object can be treated as separate plane. each object and be applied with different bit rate, quantization, frame rate and much efficient prediction can happen. Also, one can manipulate such objects like the way you can do stuff in photoshop/gimp.

However, as it turns out, the segmentation of real-world image and video with perceptual relevant objects - is a hard problem! So even if i do have encoder/decoders - i need robus and real time segmentation system to make things possible. This is by far a really open problem as to how do you apply video in the form of such layered video to make it work.

5. More dimensions
Apart from this - there are codecs for stereo video (a.k.a multiview) and 3D video (almost similar concept) which additionally try to reduce the redundancy in the additional dimension. While this is still research going on here, H.264 does have a profile for this now. Same is true for the Cinema encoding.

There are many research projects still in the area of this - however, video compression is quite a commodity market by now. And research which doesn't substantially generate new application or if there are feasibility issues in implementation wont survive practically in spite it's ingenuity.

Hope this helps.

References: