Using a vector quantizer (VQ), groups of image samples (or motion compensated and/or transformed data) can be quantized as vectors. I'm curious about why VQs are not preferred in the modern, highly popular video codecs such as H.264/AVC and H.265/HEVC. A quick Wikipedia search showed that there are some codecs (such as Daala) using VQs. However, the best codecs, in terms of compression efficiency, do not use them. Is that because of the difficulty of designing a good codebook and efficient searching of the codebook?
Theoretically, VQ is always better than scalar quantization (SQ). However, as you already mentioned, the complexity of VQ is higher than the complexity of SQ. It increases exponentially together with the dimensionality of vectors. Therefore, the question is how to design a practical coding system which can provide a best rate-distortion performance under a given complexity constraint.
A designing of a codebook is another very important issue in VQ. The main practical concerns about using a codebook in VQ are:
- How to generate a universal codebook for a large number of images.
- How to scale the codebook for various rate-distortion requirements.
- How much storage space is reasonable for storing the codebook in memory.
All this practical constraints make it difficult for VQ to force through standardization process.
Sparse dictionaries, dictionary learning or dictionary coding has been popular (and also shown to be quite high-performing) for noise reduction (denoising) and it is basically a kind of vector quantizer. Building the code-book (dictionary) takes time and I don't know exactly how large it typically becomes for denoising applications, but if it is calculated off-line once-and-for-all and was known to both coder and decoder then it would not need to be sent as part of the video stream and then codebook size would not impact stream size or bit rate. If codebook is too large then decoding on RAM-constrainted devices would maybe be infeasible. Also if too large we could start getting cache misses which could impact performance.
Also almost all of todays high performing video coders are hybrid coders combining transform coding and motion coding. One would probably need to find a good way to combine the vector quantization with hybrid coding concept. It is not obvious to me how to do that. Should the VQ only be used for I frame image coding or also in some way to code the motion fields?