1
$\begingroup$

I'm trying to understand how video compression through dense optical flow works (I'm a newbie with video compression).

I easily understood what optical flow is, the thing I can't understand is why a simple video compression could not just send the parts of a frame marked with a "motion vector" (as you would send only the parts that have moved to or from a position from frame to frame).

I mean: MPEG is way more complex than that, what's the catch with sending parts of a frame that have changed to "stick them over it to get the next one"? Is this a technical flaw (i.e. too much data to be transferred) or is there something else involved that invalidates this simple reasoning?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ I don't understand what you're referring to. "Just transmitting changes" is one of the many things that MPEG does; motion vectors play a large role. But first, you'd have to detect changes. You assume that's easy. It's not. And, you might be overestimating how much of a scene actually stays unchanged. Notice that you'll have to take lighting, camera imperfections, and everything else that might change a scene into account – across the JPEG-typical block boundaries, too! $\endgroup$
    – Marcus Müller
    Feb 25, 2017 at 17:25
  • $\begingroup$ @MarcusMüller Thanks for answering! What I really meant is: what stops me from doing an optical flow calculation in frame A and just re-transmit all of the parts that my optical flow detected as "changed" and then put them as a mask on the old frame A? $\endgroup$
    – Dean
    Feb 25, 2017 at 17:58
  • 1
    $\begingroup$ nothing. It's, as I said, among the things MPEG does. $\endgroup$
    – Marcus Müller
    Feb 25, 2017 at 18:16

2 Answers 2

Reset to default
1
$\begingroup$

You could do that, in theory, if you knew "what are the parts" (or objects) frame-to-frame in a video. But:

  • objects move, change scale, rotate, have color changes, they even can be occluded.
  • the segmentation is a complicated task in general.

MPEG 4, at least, allows to manipulate objects instead of pixels, see MPEG-4 Natural Video Coding - An overview. If objects and backgrounds are already decoupled, like filmed with a green screen:

green screen

then the object contents can be processed separately. Otherwise, what is more important is that, visually, the representation is accurate enough. Thus, a block from some object in a frame can be cast to another block of a different object in a different frame, as long as the result is satisfactory. Thus, in a lot of cases, the block matching is sufficient, and more importantly, relatively fast to compute.

Improve this answer
$\endgroup$
1
$\begingroup$

Video codecs tries to find apparent movement (akin to optical flow) first. Bits have to be allocated to describe that motion. Then the residual is coded using a lossy transform/quantazation that tends to leave mainly a few coefficients close to "DC", and run-length coding or similar of those.

-k

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.