H.265 is based on incremental gains over time. The paradigms have largely remained the same. What are the odds of deep learning providing huge performance gains? Here is the main metric:

  1. Perceptual quality remains the same at 2x improvement in bit-rate.
  2. Assume unlimited compute power at encoder.
  3. No need for speed.

Of course, I understand this is a really ill-posed problem because there are so many factors involved. But if it can be done, what would give the performance gain? Is it the following?

  1. Larger CTUs instead of the 64x64 one in H.265. Perhaps even go towards object-based coding.
  2. Better block matching algorithms that map longer dependencies.
  3. Perceptual loss for block matching instead of MAD or MSE.
  4. Etc...
  • 2
    $\begingroup$ Hi! Your question is pretty unclear. What do you mean with "huge performance gains"? What is your measure for performance? compression ratio? Quality vs bitstream size? Speed of decoding? Power efficiency? Error resilience? What is the thing about video encoding that you want to improve? Like this, you're basically asking "can machine learning be applied to fruit salad?", to which the answer is "it's likely.". $\endgroup$ Dec 1, 2016 at 14:52
  • 1
    $\begingroup$ @MarcusMüller great point, made edits! $\endgroup$
    – jkschin
    Dec 1, 2016 at 14:58
  • 1
    $\begingroup$ Sure it can be applied to anything : )) $\endgroup$ Dec 1, 2016 at 15:05
  • $\begingroup$ @TolgaBirdal would love more elaboration on that and the intuitions on why it works and the challenges in implementing it :) $\endgroup$
    – jkschin
    Dec 1, 2016 at 15:06

3 Answers 3


[EDIT: addition of a March 2017 preprint]

Deep learning already has many applications in video, like enhancement (Deep Convolutional Neural Network for Decompressed Video Enhancement) or semantic analysis.

Recently, there have been some announcements related to video compression, for instance:

Traditional image and video compression algorithms rely on hand-crafted encoder/decoder pairs (codecs) that lack adaptability and are agnostic to the data being compressed. Here we describe the concept of generative compression, the compression of data using generative models, and show its potential to produce more accurate and visually pleasing reconstructions at much deeper compression levels for both image and video data. We also demonstrate that generative compression is orders-of-magnitude more resilient to bit error rates (e.g. from noisy wireless channels) than traditional variable-length entropy coding schemes.

How important is deep learning here, and what performances are obtained is not clear to me yet.

  • 1
    $\begingroup$ Notice that the face examples in arxiv.org/pdf/1703.01467.pdf look like completely different people after decompression. :D $\endgroup$
    – endolith
    May 27, 2017 at 1:31
  • 1
    $\begingroup$ I saw that happen already in compressed video surveillance, with strong issues in forensics identification $\endgroup$ May 27, 2017 at 5:34
  • 1
    $\begingroup$ Great answer, especially with the edit on adding the March 2017 paper. If anyone comes across any other papers, it'd be great to update this answer too. $\endgroup$
    – jkschin
    Jun 16, 2017 at 8:01
  • 1
    $\begingroup$ Its the area of deep autoencoders, youtube.com/watch?v=z5ZYm_wJ37c $\endgroup$
    – Peter
    Feb 25, 2018 at 11:18

Well video compression is essentially changing the representation of the video into an intermediate one, such that this representation could be used to recover the original video to the best possible extent. What you store is a much shorter representation instead of the full video.

Well, putting it that way immediately reminds me of the (Convolutional) Auto Encoders. Let's begin with an image: Given an input image, the auto-encoder tries to recover the original with an encoder-decoder scheme, where the middle layer consists of a short vector of weights, which can act as the features of the image. They are proven to be pretty good function approximators.

Video is basically a volumetric 3D image, where each slide corresponds to an image frame. One could then think about designing an Auto Encoder to recover the slices simultaneously, using once again a much shorter representation. One could slide this volume over the video frame and compress the video.

The compressor = Encoder, Decompressor = Decoder. That simple. More advanced networks, explicitly using the time information might benefit from LSTMs on top.

This method is unsupervised. So all you need is a bunch of videos.

Having said that, I would not argue about how good the performance would be, as it is really upon the network design.


It is very difficult for end-to-end compression using Deep Learning (DL compared to conventional video compression like HEVC.

The HEVC design goal is for efficient, hardware friendly video coding for beyond HD solution. We have unlimited power at encoder but not the decoder, and the limitation of hardware video encoder comes from memory assessment. HEVC process CTU (64x64) as one processing unit and several reference pictures of HD or beyond HD size (1920 x 1080). All the parameters, transform matrix is written directly in the source code. Nowe compared to DL based coding, you can amazing how much complexity, memory access DL required if process multiple frames and have to load the learned weight.

In addition, DN can perform dimension reduction like autoencoder but it is totally missed "coding" part. We cannot transmit features in floating point. We need to break it down by quantization, further, remove statistical redundancy with entropy coding and high-level syntax to form the bitstream. There are few pieces of research focus on compress the network itself.

Furthermore, conventional video standard performs the exhaustive search. They try every possible case which gains in terms of rate and distortion. Rate distortion optimization giving significant gain for HEVC.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.