I am trying to segment very simple objects like boxes and bottles. I am using GrabCut and it works extremely well when there are no shadows or when the shadows are soft - but as soon as there is a harsh shadow, Grabcut includes that in the segment. The goal is very simple: determine the polygon that makes up the box. That's it.
Note that I am using Grabcut in the Rect mode. I cannot use a mask because I don't know where the shadows may fall.
Some further details that I forgot to add: this is a video based system. The user places the a box inside the red rectangle on-screen. Currently, I am using Grabcut on 12.5% scaled images to speed up the process so that the video feels it is in 'real-time'.
The purpose of the program is to compare the segmented object's simlarity using perpetual hash. As I mentioned, the system works extremely well as long as the shadows are minimised.
Here is an example:
Most objects aren't as simple and can have multiple colours inside them.
Some solutions that I have thought about:
Lighting Based This is part of a system that will be used in various locations where the lighting may vary. However, the system will have lighting built into it to compensate for the environment - but will that be enough? Can I use light to 'fill' in the shadows so that Grabcut works?
Use a pattern or grid beneath the object What I have noticed is that if the shadows are even partially outside the Rect, Grabcut is able to segment the object very nicely. I have been wondering if I include a pattern like this and then determine the dots the adjacent to the box, determine the contour/rectangle of the adjacent dots and then feed that Rectangle to Grabcut for segmentation. This assumes that the shadows are somewhat tall.
This method works fairly well but then I run into issue: some dots are partially covered by the object and they are unfortunately still included by grabcut. Perhaps I can consider some other pattern here?
Use some advanced shadow removal algorithm This seems like my best bet but it seems the majority of algorithms require a calibrated camera or professional camera. But I'm hoping there is some algorithm out there that can help me do this with a consumer grade webcam.
