After advice about detecting focus quality of objects in a photo detected using YoloV3

Question

I've spent the last couple of days playing with YoloV3, and have had very good results. My use case is sports photography, and the object detection for people/bikes etc is very very good, I'm very impressed. In the future I think I'll train it with my own dataset to improve it further, but out of the box it does a fantastic job already.

What I want to improve:

Once an object has been detected, how can I generate some sort of metric to quantify how well-focused it is?

Past/current approaches

• 1) My first thought/Google, was "variance". First I turn the image grayscale, then use a Laplacian convolution to highlight the edges. Then simply compute the variance of the pixels in the bounding box. High numbers "probably" mean high contrast i.e. high focus, whereas lower numbers would tend to mean low contrast and probably poor focus.

It works pretty well, but not 100% reliable. Imagine you have a person stood up with their arms spread wide, against a skilled bokeh background. Most of the bounding box is blurred background, so the variance ends up being low.

• 2) To improve upon this, I came to the conclusion that there will almost always there be a significant portion of the object somewhere around the center of the bounding box. I set my criteria to center square, 20% of the width of the BB and 20% of the height.

Unfortunately this throws up scenarios where that square happens to be 10% background, between someone's arm and body when running around a tight corner facing the camera, etc.

• 3) "OK, a cross". Thickness equal to 20% of the BB, up and down through the middle and left and right through the middle.

Not bad, not bad. Still getting a lot of background on some images though, as the edges of the box are where background is going to tend to reside.

• 4) "OK, a reduced cross". Same as above, but only extending from the center 2/3rds of the way out to the edges.

Almost fantastic. With the caveat that sometimes you end up with just a competitor's chest, and if they're wearing a single-colour top.... the variance isn't all that.

Examples:

In this photo, the motorbike (close enough...) apparently has great focus, while the person isn't so good. Mainly due to the near-uniformity of his central cross.

Here's a more troubling example. Look at that variance, 5 FFS!

So I think that's the end of that approach.

The future...

I could go on and on with this, and I'm ALWAYS going to end up with some photos that it just doesn't work well for.

I think a different approach is needed.

One thought is just to take the largest variance over a small region, say a 10% width/height square that roams across the bounding box.

But then if the foreground is completely out of focus, and the background is sharp, we'll get a false positive from the background.

Anyone cleverer/more experienced than me have a fantastic solution for this?

It's clearly possible, not least because http://remove.bg and PhotoShop already do a fanastic job of separating foreground from background. But how?

EDIT: I completely neglected to mention that I'm using a Laplacian convolution on a grayscale version of the photos before computing the variance, to detect the edges.

Answer 1

For your application, image segmentation would be more useful than bounding boxes that contain also background. Other useful keywords: instance-aware image segmentation, instance segmentation.

Figure 1. Instance segmentation example image from Mask R-CNN, by Karol Majek. Bounding boxes are also shown.

Examples of implementations using some version of Yolo:

• Mask-YOLO by Jianing Sun
• Instance Segmentation by Wenzhen

Other implementation examples:

• Mask R-CNN for Object Detection and Segmentation by Karol Majek
• LightNet++ by Huijun Liu

Answer 2

Here is what I would try on the source image:

Split your image into 5x5 pixel blocks (maybe 3 maybe 7, who knows?)

Create output image one fifth (third, seventh, ??) size

For each block
  For each color channel

    Find best fit plane
    Measure RMS of (pixel value-plane value)

  Next

  Set output pixel to  RMS(R,G,B)

Next

In blurry/plain areas the RMS should be low. With sharp lines it should be high.

Ced

---

So, I thought I'd give this a go with 5x5 block.

Here are some results.

Clearly (pun intended), not only is this algorithm an edge detector, but it also a focus metric at those locations.

---

Those are kind of dim, so I multiplied by 5.

If nothing else, they are kind of cool looking, but I think you can tell where the well focused areas are.

---

The 5x5 takes a while to process, so I decided to try a 3x3 fit on a simple 4 point gradient estimate (Down from a 16 point gradient estimate).

Still (pun intended again), the blurry lady pic proves this is a focus metric.

Here is the relevant code called for each color channel:

'==================================================================
Public Sub FindFocus(ArgV As Float[]) As Float[]

        Dim w, h, x, y As Integer

        w = ArgV.Bounds[0]
        h = ArgV.Bounds[1]

        Dim theFocus As New Float[w, h]

        For x = 1 To w - 2
          For y = 1 To h - 2
            GoSub CalculateFocusAtPoint
          Next            
        Next

        Return theFocus

'-------------------------------------------------------------------
CalculateFocusAtPoint:

        Dim dx, dy As Integer
        Dim a, b, c As Float

'---- Plane Estimate:  z = ax + by + c

        a = (ArgV[x + 1, y] - ArgV[x - 1, y]) * 0.5
        b = (ArgV[x, y + 1] - ArgV[x, y - 1]) * 0.5
        c = ArgV[x, y]

'---- Calculate the RMS of the NonPlanar

        Dim v, e, s2 As Float

        s2 = 0

        For dx = -1 To 1
          For dy = -1 To 1
            v = a * dx + b * dy + c  
            e = ArgV[x, y] - v
            s2 += e * e
          Next
        Next

'---- Set the Value

        theFocus[x, y] = Sqr(s2 / 9)

        Return
End
'==================================================================

---

Finally, just because I could, I turned your pic into a "drawing".

Answer 3

Could you please share the part of your code where you include Laplacian inside bounding boxes? Many thanks.

Regarding to your question, there is a paper reviewing focus methods:

Pertuz, Said, Domenec Puig, and Miguel Angel Garcia. "Analysis of focus measure operators for shape-from-focus." Pattern Recognition 46.5 (2013): 1415-1432.

Maybe in your case is better to use other method, instead of Laplacian.

Answer 4

For a metric of the blurredeness, you are using a laplacian which gives you usable limits of the object which are not included in most of the edge of the objects square. If you walk lines of pixels inwards from the edges of the detected zone, omitting the photo edges, when you cross a black zone from your laplacian, it means that you have transitioned inside the object that you want to measure... it is an approximative boundary, so to find ideal object boundary you may have to improvise with some tests to find if it can be better than your current other process.

You have to only sample from a certain type of pixel line found using the scan.

scan lines that correspond to "light,dark, light", running from the edge, the second "light area" would most often be of the actual object. You'd have to define areas as pixel lengths, it's all a major bit of programming.

You can them run a variance algo on all those pixels until the moment that you reach the nex black zone from the laplacian, which means that your pixel sample line is away from the object that you want to measure. For the threshold black level, you can try something like "within the 3% most dark pixels of the sample", like a variable edge threshold to start and stop a variance sequence metric.

So the trick would be, march lines through XY vectors randomly as a grid and/or from various angles around the object square to be measured, perhaps like 20-100 sample lines, and average the variance measurements from within approximate boolean dark boundaries.

You can also construct an orderly grid of variance based on vector sample paths through the image which can be compared to the vague boolean delimitation of the object defined by the greyscale, so that you have two maps that can be correlated and compared for perhaps a refined result of "object outlines based on their variance".

I don't know the really complex maths that exists to devide the image into zones of different "frequencies"... When you do a gaussian stack of the image or whatever The frequency of the image is analyzed into different frequency ranges... So you can also run vectors through the image at a variety of frequency deconstructions to search for boundaries of objects.

Perhaps you'll have to add lots of parameters to your algorythm for bias of different types of images and so that tricky exceptions like bokeh and clouds both require a specific check, i.e. for white and blue with clouds.