3
$\begingroup$

I have two images – mannequin with and without garment.

Please refer sample images below. Ignore the jewels, footwear on the mannequin, imagine the second mannequin has only dress.

enter image description here enter image description here

I want to extract only the garment from the two images for further processing.

The complexity is that there is slight displacement in the position of camera when taking the two pictures. Due to this simple subtraction to generate the garment mask will not work.

Can anyone tell me how to handle it?

I think I need to do registration between the two images so that I can extract only the garment from the image?

Any references to blogs, articles and codes is highly appreciated.

-- Thanks

$\endgroup$
1
$\begingroup$

The approach you are alluding to involves a) first registering the images and b) using a subtraction operation to remove the background (which is the mannequin).

First on (a) (registration): Image registration deals with figuring out the displacement of corresponding pixels in two images. The problem with real-world images is that the world is 3D and because of the depth in the scene, nearer surfaces will move more than surfaces deeper in the scene. Also, a surface can self-occlude certain parts of the scene. I only state these so that you are aware of the complexities involved. Having said that, in the scenario you have posted above, you might be able to do very well with a simple translation model for the displacement. In other words, you can assume that all pixel in the second image move by $(\delta{x}, \delta{y})$ from its original position. So if a pixel was located at $(x,y)$ in $I_1$, then its location in $I_2$ will be $(x+\delta{x}, y+\delta{y})$. This can be assumed to be true for every pixel (global translation). You now need to estimate the horizontal shift $\delta_x$ and the vertical shift $\delta_y$. There are many approaches to this problem. Here is an academic reference to registration methods (may be TMI) Zitova and Flusser. You may wish to start off by trying the ITK toolkit and try using the "translation transform". If you find that after accounting for a translation transform, the image subtraction still leaves parts of the mannequins, you might want to try some more advanced transforms. For example, the camera may have rotated slightly in your hand. So in addition to a shift, you can also introduce a rotation. Also, you might have taken one shot closer to the other, so there may be a difference in scale. This necessitates the use of an affine transform which is also included in ITK. Barring these "global" methods which assumes that every pixel moved in the same way, you can also use local methods. For example, you can estimate a $(\delta_x, \delta_y)$ for every patch in the image. You can divide up the image in patches of $7 \times 7$ pixels and search for the best corresponding patch in the other image. Every pixel within this $7 \times 7$ window is assumed to have moved in the same manner as the motion that was estimated for the patch. A last resort, but the most general is to compute dense optical flow for every pixel in the image. This gives you a shift for every pixel in the image. You can find an optical flow implementation in the popular OpenCV software.

Finally, on (b) (occlusion): You will inevitably end up with some pixels without a valid shift because they simply aren't present on the other image. Or some pixels are uncovered in the second image which were covered on the first. See these paper for clues Occlusion aware optical flow and TV-L1 Optical Flow with Occlusion Detection

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.