Video Compression - When would an average of the previous and next I or P frame be useful

Question

I'm doing practice exam questions for my multimedia systems exam one tuesday and a topic which seems to come up frequently is the appropriate use of encoding B-frames based on:

1) the previous I or P frame

2) the next I or P frame

3) an average of the previous I or P frame

I'm having a bit of trouble thinking up decent examples of these, and I'm not entirely sure when an average would be most appropriate. Would someone mind checking my examples or maybe helping me think of some more concise or realistic ones?

1) A dropped ball about to hit the floor. The previous frame the ball is the same shape but in the next frame it gets deformed because of the impact of hitting the ground, so the previous frame is better.

2) A man running along the pavement and a car moves between him and the camera temporarily in the previous frame. Due to object occlusion the image of the man in the previous frame is not complete so is not a good predictive frame, but the next frame the image of the man is complete so it is better to predict from this.

3) I'm not sure at all what to put on this one. I was thinking about a car moving with constant velocity across the screen because then it's position in the target would be the average of the previous and next - but this seems quite contrived!

Anyway, any help would be appreciated so if you have ANY thoughts let me know!

Answer 1

Before elaborating anything, i would urge you to go through my answer here that simplifies some of the common confusions prevailing about motion vectors and estimations.

---

Your essential question is - when or how does encoder decides to put B frame or bidirectional motion vectors. I will break your questions into few as follows:

1. When does encoder decides to use B frames?

Almost always the number of B frames between I or P is always fixed. This is usually 2 or 3 pictures. The distance between I frame (called GoP size) may keep varying, but number of B frames will always remain fixed. From the point of view of MPEG, this is not really a restriction, but as one of your other question discusses, if you keep changing B frames, there will be serious problems of rhythms between the transmission and receptions.

Also, even before estimating whether a given frame will be a good choice to act as a B frame or a new I or P frame, you have to actually decide the forward reference and past reference for from where the prediction. Iterating over whether given frame will be perfect to work as P against frame already gone or should you choose some distance frame as P and current as B is too much overhead to get encoding done, and in the process you are fluctuating delay heavily.

So in essence, typically encoders decides number of B frames as 1,2 or maximum 3 and corresponding pictures are always fixed to be B frames.

2. So then what if i have a huge black/uncorrelated picture between last I and future P?

Ok - now that you have chosen to see that given frame is B, but almost nothing matches of that frame between either past or future; or sometimes only either side of the frame? This doesn't really put encoder in trouble though.

In MPEG, every frame is divided in terms of macro blocks. Each macroblcok has motion prediction from forward or reference blocks from respective past or future pictures.

• I frame can ONLY have Intra blocks - which has no motion compensation, so it doesn't need any other frame ready for decoding itself.

• P frame can have INTRA as well as FORWARD-PREDICTED macro-blocks. So even in case of P frame, when current block doesn't match with anyone in the past frame, P frame can simply use INTRA block.

• B frame allows all type of macroblock including INTRA (as in I), FORWARD-PREDICTED mb (where motion vector depends only on past picture) , BACKWARD-PREDICTED mb (where motion vector depends only on future picture) and BIDIRECTIONAL macroblock which has two motion vectors from past and future and they are averaged out to form current block.

Hence, in case, the B picture or part of it happens to be very independent it is quite possible to put INTRA blocks for those MBs, and if it matches only either side of the picture encoder can use only FORWARD or BACKWARD prediction either way.

3. So does all MB's of a given frame have to be of same type?

Not at all, suppose a new small object just entered in a frame which is encoded as B frames, only part of the frame needs INTRA Macroblocks but majority of others can predict their motion from other blocks.

Understand, that all MPEG cares about is, if encoder can find a block which is almost as good as a current one, use to minimize the error, if not bad luck and encode this as a fresh new information. Even this ability to find itself varies from encoder to encoder.

4. So how does encoder make so many choices?

Usually, for the B pictures, encoder starts a usual business by searching the most appropriate blocks from past and followed by future picture.

a. if both frame produce some reasonable motion vector, this is most preferred to minimize the energy and book the block as BIDIRECTIONAL MB.

b. if either past frame or future frame doesn't produce motion vector - i.e. the minimum energy (SAD) is higher than a useful number, then correspondingly only 1 motion vector is encoded making MB FORWARD or BACKWARD prediction only.

c. if none of the reference frames have good enough motion vector - the MB is encoded as INTRA.

5. So why don't I simply encode everything as INTRA block and save from all the hassel?

You can absolutely do that. It will still be perfectly legal MPEG stream. Just that INTRA coding will take more symbols to encode and hence it will have to be quantized badly to fit under a given bit budget. Putting everything as INTRA has significant loss of efficiency because typically video always has huge temporal redundancy that INTRA blocks don't exploit; but they do give random access freedom.

Answer 2

I'm assuming what you want to know is something like "Hey, I hear that H.264/AVC/MPEG-4 Part 10 allows each B macroblocks to either copy from a single past I or P frames, copy from a single future I or P frame, code as an INTRA block without reference to the past or future, or "bipredict" and copy from some kind of average of two different I or P frames. Why would an encoder ever choose biprediction?"

Two examples I like to use:

A gradual sunset

• A gradual sunset, where the sun-lit objects slowly dim to black, but in the gaps between the leaves you start to notice a campfire and other lights.

Say you have a large macroblock in the current B frame that includes part of a foreground leaf that is dimming from sun-lit bright green to black, a background dim campfire that is more-or-less constant brightness, and the ragged edge between them.

The encoder could try to estimate that macroblock from only the past or only the future I or P frame. We'll end up with an estimate where the leaf is too bright or too dim, or else (if we use weighted prediction) the campfire is too bright or too dim. In any of those four cases, the ragged edge makes it take a lot of bits to describe exactly which pixels need to be fixed-up.

However, averaging the corresponding location in the past I or P frame with the "next" "future" I or P frame, we get an estimate that the leaf will be some intermediate brightness, the campfire will be the same constant brightness, and the ragged edge between them is the same. If the encoder chooses that "biprediction" approach, then in this case the estimate is closer to the actual snapshot we're trying to encode as a B frame. So biprediction lets us get better quality or use fewer bits or both (in this case).

A big van

• A big van moving across the screen, partially blocking some highly detailed unchanging background

As the van moves forward, some highly-detailed background behind its back edge becomes exposed. That information cannot be found in the previous I or P frame (it's still blocked by the van), but it can be found in the "next" "future" I or P frame.

As the van moves forward, the region in front of the van will soon be occluded. That highly-detailed background immediately in front of the van in the current B frame cannot be found in the "future" I or P frame (where it is blocked by the van), but it can be copied from the previous I or P frame.

Doing any sort of biprediction "averaging" in these two cases would only make things worse.