There are various aspects of compression - whether you are referring to image or video. (May be audio is totally different so i am not referring this here).
If you really look at the history of compression standard, when they were first formed - MPEG1 real time encoders were rare. CPU at that time itself was not enough to make it real time; however, it was not the processing complexity (alone) that stopped more complexity algorithms to be used under codecs. (Probably the primary factor that was responsible for inclusion/exclusion of certain algorithms were a. patents, b. feasibility of implementation).
However, i would like to re-phrase the question a bit more simpler:
What other research in compression is available which is still to come
to market (or didn't come at all but might be worth for some
applications)?
Here are some points to look at:
1. Better transform?
DCT by far has been universal transform across all codecs. However, there are many research papers which suggest that Wavelets could be better. Indeed the only JPEG2000 included wavelets - but that standard itself wasn't big hit. Probably it was better than DCT in some way; specially there are claims that Wavelets outperformce DCT [See: Ref 1]
I think wavelet is computationally expensive over DCT (partly because 8x8 DCT implementations are highly optimized). But i guess i didn't really flew that much because from the market perspective because DCT by the time was far more wide spread and the difference in improvement was not so substantial for people to switch.
2. Multi hypothesis motion compensation
While MPEG 2 was happy enough with $B$ pictures, extensive research from Bern Girod [See Ref 2] proved that - the concept can be extended way beyond this. Indeed H.264 actually has made great generalization on multiple target based motion compensaion - basically one can store past 16 pictures (compare to 2 in MPEG2) and indeed the prediction can be extremely high.
While, this is already part of the standard in H.264 - this is still not fully exploited. There is a need to make such extensively large search for motion estimation practically viable. Most H.264 encoder are still nascent in exploiting full potential of this.
3. Scalable Image/Video compression
Scalable Image and video compression is again a holy grail of how multiple description encoding can be done for image and video. Basically the same Image or video can be encoded at multiple Resolution, quantization (quality), or frame rate. These produces different layers of quality vs. bit rate right within the same stream - and the streaming server can actually adapt the bitrate and quality dynamically to best fit the video in the quality.
See [Ref 3], [Ref 4], [Ref 5]
Of this, only real success is progressive JPEG which is practically useful for websites and indeed useful. Again while the promise of dynamically adapting bit stream is great but it works only if there are practical systems where estimating such bit rate is possible and type of adaptation can serve some real purpose. However, in video domain, in spite the support of this in the standards (MPEG2, MPEG4 and H.264) - there is hardly any usage of this.
4. Object Video
This is perhaps ultimate holy grail of Compression. Object video [Ref 6] as initiated by MPEG standard body itself however, this was also backed by extensive research.
The basic idea is to segment the video in terms of distinct objects and later on each such object can be treated as separate plane. each object and be applied with different bit rate, quantization, frame rate and much efficient prediction can happen. Also, one can manipulate such objects like the way you can do stuff in photoshop/gimp.
However, as it turns out, the segmentation of real-world image and video with perceptual relevant objects - is a hard problem! So even if i do have encoder/decoders - i need robus and real time segmentation system to make things possible. This is by far a really open problem as to how do you apply video in the form of such layered video to make it work.
5. More dimensions
Apart from this - there are codecs for stereo video (a.k.a multiview) and 3D video (almost similar concept) which additionally try to reduce the redundancy in the additional dimension. While this is still research going on here, H.264 does have a profile for this now. Same is true for the Cinema encoding.
There are many research projects still in the area of this - however, video compression is quite a commodity market by now. And research which doesn't substantially generate new application or if there are feasibility issues in implementation wont survive practically in spite it's ingenuity.
Hope this helps.
References:
A Comparative Study of DCT- and Wavelet-Based Image Coding by Zixiang Xiong, et. al. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 9, NO. 5, AUGUST 1999 pp. 692
Efficiency Analysis of Multihypothesis Motion-Compensated Prediction for Video Coding
Bernd Girod IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 2, FEBRUARY 2000 173
Spatial-domain resolution-scalable video coding by Barry G. Haskell, Hsueh-Ming Hang. Proc. SPIE 2094, Visual Communications and Image Processing November 718 (1993);
The MPEG-4 Fine-Grained Scalable Video Coding Method for Multimedia Streaming Over IP
by Hayder M. Radha, et. al. IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 3, NO. 1, MARCH 2001 53
Advances in Scalable Video Coding by Ohm, J.-R.; Proceedings of the IEEE Jan. 2005 Volume: 93 Issue:1 pp. 42 - 56
Object-Based Texture Coding of Moving Video in MPEG-4, by Andr ́ Kaup, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 9, NO. 1, FEBRUARY 1999 pp. 5
