Extracting DCT coefficients from H.264/AVC bitstream

Question

Is it possible to extract DCT coefficients from H.264/AVC bitstream without doing a full-decoding of the video?

There are a lot of papers on compressed domain video processing which are based on the assumption that DCT coefficients can be extracted through partial decoding and thus a lower computational complexity and faster than real time decoding can be achieved by using the DCT coefficients for various computer vision tasks (e.g. tracking, action recognition). However, nobody really describes the exact process of such partial decoding, and I seriously doubt it can be achieved with the recent video codecs such as H.264/AVC or H.265/HEVC. Does anybody have an insight on this one?

Answer 1

It is possible to extract the transform coefficients without fully decoding the entire picture, but you would still have to find your way through the bit stream, parsing whatever is necessary in order to arrive at every transform unit, which might also not be present for every coding unit (if some macroblock matches exactly one in a reference picture then you don't need to encode any residual, for example). How you would use only the transform information to do your computer vision tasks, I have no idea, but it is entirely possible to only extract the transform coefficients. What i suggest you do is grab whatever standard you want (H.264 or H.265 or some other) and go through all of the tables with the syntax elements, and the parsing process for arriving at the transform coefficients would be to ignore every fixed width syntax element not useful for decoding the transform (anything related to motion vectors or reference pictures) and parse whatever is necesary to obtain all the transform units in the coded picture. I don't know if there is any tool out there that does this, perhaps you can go into ffmpeg or one of the libraries like x264 or x265 and modify them to extract what you want. It is a difficult task, but definitely possible.

Hope it helps