3
$\begingroup$

Wanted to understand how google meet is removing halo effect when composing their image. They describe their method briefly in their blog as:

animation "The blur shader simulates a bokeh effect by adjusting the blur strength at each pixel proportionally to the segmentation mask values, similar to the circle-of-confusion (CoC) in optics. Pixels are weighted by their CoC radii, so that foreground pixels will not bleed into the background. We implemented separable filters for the weighted blur, instead of the popular Gaussian pyramid, as it removes halo artifacts surrounding the person. The blur is performed at a low resolution for efficiency, and blended with the input frame at the original resolution."

Am facing a similar issue where background color from original image is coming into the new composition as shown below creating a halo effect:

Halo

Am familar with guassian pyramids but not sure how it helping in this case and how seperable filters have an impact. Can anyone explain how to solve this problem?

$\endgroup$

1 Answer 1

1
$\begingroup$

Based on what I read in your reference I would say that

The key part for this solution is the model to compute the mask, if you have one that is a good starting point.

The second important feature is that it uses a position dependent filter, and the filter is parameterized with the length of the value of the mask. In their word avoiding the foreground to bleed in the foreground. They don't know how exactly, but I would say that from the mask a parameter $r_{i,j}$, that is large for the foreground and small for the background.

The possible realization of separable filter would be

$$ v_{i,j} = \frac{ \sum_k f(k\cdot r_{i,j}) x_{i-k,j}}{\sum_k f(k\cdot r_{i,j})} $$

$$ y_{i,j} = \frac{ \sum_k f(k\cdot r_{i,j}) v_{i,j-k}}{\sum_k f(k\cdot r_{i,j})} $$

And you are free to play with the normalization factors.

Using separable filter means that you use two 1D filters rather than small 2D filters for the image in multiple resolutions. This may be better in terms of memory. It can be implemented with two copies of the image, $x$ is written to $v$, and the second equation update the image $x_{i,j}$ in place.

Other possible advantage, since the filter is computed for every pixel separately. A separable filter of size $N \times M$, would require $N + M$ evaluations of the filter, while a 2D filter would require $M \times N$ evaluations, and in the realization above since the horizontal and vertical filters are the same it will require only $N$ evaluations.

Having the mask dependent filter is important to pass the foreground (the faces) with little distortion.

$\endgroup$
1
  • $\begingroup$ I think the idea of separable filters was in the context of Gaussian Pyramid. It is known the Gaussian Pyramid creates halos when applying effects on pixels in the lower levels. $\endgroup$
    – Royi
    Nov 22, 2021 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.