How to extract the people and objects from an image separately?

Question

For example, I have an image

There are 7 people in the image and one apple and a bird. How can I extract every person and every object from this image and put the 7 people and 2 objects into 9 different new images? How can I also identify which is person and which is object?

Answer 1

I think what you're looking for is called "category level object recognition". As far as I know, it's still an active research subject. Meaning you can't expect to find high-quality libraries that you can use out of the box without understanding the magic behind. And even state-of-the art research code only achieves around 90% correct classification.

A common approach is the so called "bag of (visual) words" model. Meaning: You take a lot of labeled training images containing various objects. You use a keypoint detector&descriptor like SIFT or SURF to find the "visual words", i.e. similar-looking patches in the training examples, and you train a machine learning algorithm (like a support vector machine) to recognize the category labels based on the presence or absence of these words in each training image.

Then you again use the same keypoint descriptor, e.g. using a sliding window-technique on your test image, find the closest visual words, feed that information to your learning algorithm and you get an object classification for the sliding window (if you're lucky, that is).

Sadly, if you want to do that with Mathematica, you'll have to reinvent the wheel a lot. The SURF keypoint detector/descriptor is included in the ImageKeypoints function, but I don't think you can make it use sliding windows. And there is no machine learning functionality to speak of.