# HEVC Deblocking filter algorithm

I'm doing research on block noise caused by video compression (especially in newer codecs) and I've read the IEEE article about the HEVC deblocking filter (here).

However, when reading the algorithm I'm confused because the formula seems wrong. The first formula to determine if the deblocking filter will affect a block is:

$|p_2,_0 - 2p_1,_0+p_0,_0|+|p_2,_3 - 2p_1,_3+p_0,_3|+|q_2,_0 - 2q_1,_0+q_0,_0|+|q_2,_3 - 2q_1,_3+q_0,_3|>\beta \ (1)$

It is also written (and shown in a figure) that $\beta$ increases with the quantification parameter QP.

Moreover, later it is written that to choose between normal and strong deblocking, the following equation is used (with $i$ going from $0$ to $3$):

$|p_2,_i - 2p_1,_i+p_0,_i|+|q_2,_i - 2q_1,_i+q_0,_i|<\beta/8 \ (2)$

Since IEEE is a respectable source and the paper is a couple years old, it would surprise me that there is a typo. However, I do not understand how this formula can work. It does seem to me that the second inequation would always be wrong at least for some $i$ if the first one is true.

Am I misunderstanding how the filter works or is it a typo in the paper?

The ffmpeg implementation of the HEVC decoding follows this formula instead:

$|p_2,_0 - 2p_1,_0+p_0,_0|+|p_2,_3 - 2p_1,_3+p_0,_3|+|q_2,_0 - 2q_1,_0+q_0,_0|+|q_2,_3 - 2q_1,_3+q_0,_3|<\beta \ (1)$

This appears in the hevc_deblock.asm file.

;compare
pcmpgtw         m15, m13, m14
movmskps        r13, m15 ;filtering mask 0d0 + 0d3 < beta0 (bit 2 or 3) , 1d0 + 1d3 < beta1 (bit 0 or 1)
test            r13, r13
je              .bypassluma

So I conclude that there is a mistake in the paper and the correct formula is that one (since it also makes more sense).

edit: As the comment suggested, I checked in the reference implementation for the algorithm and the interesting part is here in the file TComLoopFilter.cpp:

for (UInt iBlkIdx = 0; iBlkIdx<uiBlocksInPart; iBlkIdx ++)
{
Int dp0 = xCalcDP( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0), iOffset);
Int dq0 = xCalcDQ( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0), iOffset);
Int dp3 = xCalcDP( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3), iOffset);
Int dq3 = xCalcDQ( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3), iOffset);
Int d0 = dp0 + dq0;
Int d3 = dp3 + dq3;

Int dp = dp0 + dp3;
Int dq = dq0 + dq3;
Int d =  d0 + d3;

//...

if (d < iBeta) //<-not a >Beta as the paper said
{
Bool bFilterP = (dp < iSideThreshold);
Bool bFilterQ = (dq < iSideThreshold);

Bool sw =  xUseStrongFiltering( iOffset, 2*d0, iBeta, iTc, piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+0))
&& xUseStrongFiltering( iOffset, 2*d3, iBeta, iTc, piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+3));

for ( Int i = 0; i < DEBLOCK_SMALLEST_BLOCK/2; i++)
{
xPelFilterLuma( piTmpSrc+iSrcStep*(iIdx*uiPelsInPart+iBlkIdx*4+i), iOffset, iTc, sw, bPartPNoFilter, bPartQNoFilter, iThrCut, bFilterP, bFilterQ, bitDepthLuma);
}
}
• Welcome to DSP.SE! +1 for your question as well as answer for adding to an important subject. However, I wish you could find a bit more sources from published work be it tutorials, or may be the standard itself than a code like ffmpeg itself. You should always take software implementation by pinch of salt. Commented Jul 15, 2015 at 3:54
• Well I have also looked into the reference implementation so I added it as well. I don't have access to the standard specification (since my university doesn't have access and my lab would rather avoid spending the money if I don't really need it) so that's the only other source I could get. Commented Jul 15, 2015 at 6:36