HEVC Deblocking filter algorithm

Question

I'm doing research on block noise caused by video compression (especially in newer codecs) and I've read the IEEE article about the HEVC deblocking filter (here).

However, when reading the algorithm I'm confused because the formula seems wrong. The first formula to determine if the deblocking filter will affect a block is:

$ |p_2,_0 - 2p_1,_0+p_0,_0|+|p_2,_3 - 2p_1,_3+p_0,_3|+|q_2,_0 - 2q_1,_0+q_0,_0|+|q_2,_3 - 2q_1,_3+q_0,_3|>\beta \ (1)$

It is also written (and shown in a figure) that $\beta $ increases with the quantification parameter QP.

Moreover, later it is written that to choose between normal and strong deblocking, the following equation is used (with $i$ going from $0$ to $3$):

$ |p_2,_i - 2p_1,_i+p_0,_i|+|q_2,_i - 2q_1,_i+q_0,_i|<\beta/8 \ (2) $

Since IEEE is a respectable source and the paper is a couple years old, it would surprise me that there is a typo. However, I do not understand how this formula can work. It does seem to me that the second inequation would always be wrong at least for some $i$ if the first one is true.

Am I misunderstanding how the filter works or is it a typo in the paper?

Answer 1

The ffmpeg implementation of the HEVC decoding follows this formula instead:

$ |p_2,_0 - 2p_1,_0+p_0,_0|+|p_2,_3 - 2p_1,_3+p_0,_3|+|q_2,_0 - 2q_1,_0+q_0,_0|+|q_2,_3 - 2q_1,_3+q_0,_3|<\beta \ (1)$

This appears in the hevc_deblock.asm file.

;compare
    pcmpgtw         m15, m13, m14
    movmskps        r13, m15 ;filtering mask 0d0 + 0d3 < beta0 (bit 2 or 3) , 1d0 + 1d3 < beta1 (bit 0 or 1)
    test            r13, r13
    je              .bypassluma

So I conclude that there is a mistake in the paper and the correct formula is that one (since it also makes more sense).

edit: As the comment suggested, I checked in the reference implementation for the algorithm and the interesting part is here in the file TComLoopFilter.cpp:

for (UInt iBlkIdx = 0; iBlkIdx<uiBlocksInPart; iBlkIdx ++)
      {
        Int dp0 = xCalcDP( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0), iOffset);
        Int dq0 = xCalcDQ( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0), iOffset);
        Int dp3 = xCalcDP( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3), iOffset);
        Int dq3 = xCalcDQ( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3), iOffset);
        Int d0 = dp0 + dq0;
        Int d3 = dp3 + dq3;

        Int dp = dp0 + dp3;
        Int dq = dq0 + dq3;
        Int d =  d0 + d3;

//...

        if (d < iBeta) //<-not a >Beta as the paper said
        {
          Bool bFilterP = (dp < iSideThreshold);
          Bool bFilterQ = (dq < iSideThreshold);

          Bool sw =  xUseStrongFiltering( iOffset, 2*d0, iBeta, iTc, piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0))
          && xUseStrongFiltering( iOffset, 2*d3, iBeta, iTc, piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3));

          for ( Int i = 0; i < DEBLOCK_SMALLEST_BLOCK/2; i++)
          {
            xPelFilterLuma( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+i), iOffset, iTc, sw, bPartPNoFilter, bPartQNoFilter, iThrCut, bFilterP, bFilterQ, bitDepthLuma);
          }
        }