Can deep learning be applied to video compression?

Question

H.265 is based on incremental gains over time. The paradigms have largely remained the same. What are the odds of deep learning providing huge performance gains? Here is the main metric:

1. Perceptual quality remains the same at 2x improvement in bit-rate.
2. Assume unlimited compute power at encoder.
3. No need for speed.

Of course, I understand this is a really ill-posed problem because there are so many factors involved. But if it can be done, what would give the performance gain? Is it the following?

1. Larger CTUs instead of the 64x64 one in H.265. Perhaps even go towards object-based coding.
2. Better block matching algorithms that map longer dependencies.
3. Perceptual loss for block matching instead of MAD or MSE.
4. Etc...

Answer 1

[EDIT: addition of a March 2017 preprint]

Deep learning already has many applications in video, like enhancement (Deep Convolutional Neural Network for Decompressed Video Enhancement) or semantic analysis.

Recently, there have been some announcements related to video compression, for instance:

• Generative Compression (Preprint, March 2017)

Traditional image and video compression algorithms rely on hand-crafted encoder/decoder pairs (codecs) that lack adaptability and are agnostic to the data being compressed. Here we describe the concept of generative compression, the compression of data using generative models, and show its potential to produce more accurate and visually pleasing reconstructions at much deeper compression levels for both image and video data. We also demonstrate that generative compression is orders-of-magnitude more resilient to bit error rates (e.g. from noisy wireless channels) than traditional variable-length entropy coding schemes.

• Video: Deep Learning reconstructing Blade Runner for new video compression techniques
• Wyner-Ziv Video Coding using Hadamard Transform and Deep Learning

How important is deep learning here, and what performances are obtained is not clear to me yet.

Answer 2

Well video compression is essentially changing the representation of the video into an intermediate one, such that this representation could be used to recover the original video to the best possible extent. What you store is a much shorter representation instead of the full video.

Well, putting it that way immediately reminds me of the (Convolutional) Auto Encoders. Let's begin with an image: Given an input image, the auto-encoder tries to recover the original with an encoder-decoder scheme, where the middle layer consists of a short vector of weights, which can act as the features of the image. They are proven to be pretty good function approximators.

Video is basically a volumetric 3D image, where each slide corresponds to an image frame. One could then think about designing an Auto Encoder to recover the slices simultaneously, using once again a much shorter representation. One could slide this volume over the video frame and compress the video.

The compressor = Encoder, Decompressor = Decoder. That simple. More advanced networks, explicitly using the time information might benefit from LSTMs on top.

This method is unsupervised. So all you need is a bunch of videos.

Having said that, I would not argue about how good the performance would be, as it is really upon the network design.

Answer 3

It is very difficult for end-to-end compression using Deep Learning (DL compared to conventional video compression like HEVC.

The HEVC design goal is for efficient, hardware friendly video coding for beyond HD solution. We have unlimited power at encoder but not the decoder, and the limitation of hardware video encoder comes from memory assessment. HEVC process CTU (64x64) as one processing unit and several reference pictures of HD or beyond HD size (1920 x 1080). All the parameters, transform matrix is written directly in the source code. Nowe compared to DL based coding, you can amazing how much complexity, memory access DL required if process multiple frames and have to load the learned weight.

In addition, DN can perform dimension reduction like autoencoder but it is totally missed "coding" part. We cannot transmit features in floating point. We need to break it down by quantization, further, remove statistical redundancy with entropy coding and high-level syntax to form the bitstream. There are few pieces of research focus on compress the network itself.

Furthermore, conventional video standard performs the exhaustive search. They try every possible case which gains in terms of rate and distortion. Rate distortion optimization giving significant gain for HEVC.