I am trying to segment very simple objects like boxes and bottles. I am using GrabCut and it works extremely well when there are no shadows or when the shadows are soft - but as soon as there is a harsh shadow, Grabcut includes that in the segment. The goal is very simple: determine the polygon that makes up the box. That's it.

Note that I am using Grabcut in the Rect mode. I cannot use a mask because I don't know where the shadows may fall.

Some further details that I forgot to add: this is a video based system. The user places the a box inside the red rectangle on-screen. Currently, I am using Grabcut on 12.5% scaled images to speed up the process so that the video feels it is in 'real-time'.

The purpose of the program is to compare the segmented object's simlarity using perpetual hash. As I mentioned, the system works extremely well as long as the shadows are minimised.

Here is an example:

enter image description here

Most objects aren't as simple and can have multiple colours inside them.

Some solutions that I have thought about:

Lighting Based This is part of a system that will be used in various locations where the lighting may vary. However, the system will have lighting built into it to compensate for the environment - but will that be enough? Can I use light to 'fill' in the shadows so that Grabcut works?

Use a pattern or grid beneath the object What I have noticed is that if the shadows are even partially outside the Rect, Grabcut is able to segment the object very nicely. I have been wondering if I include a pattern like this and then determine the dots the adjacent to the box, determine the contour/rectangle of the adjacent dots and then feed that Rectangle to Grabcut for segmentation. This assumes that the shadows are somewhat tall.

This method works fairly well but then I run into issue: some dots are partially covered by the object and they are unfortunately still included by grabcut. Perhaps I can consider some other pattern here?

Use some advanced shadow removal algorithm This seems like my best bet but it seems the majority of algorithms require a calibrated camera or professional camera. But I'm hoping there is some algorithm out there that can help me do this with a consumer grade webcam.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.