Why not use synthetic datasets for training machine vision (deep learning) algorithms?

Question

I am studying some examples of how to perform 3D instance segmentation in indoor scenes, and I have noticed many of the available datasets are from real environments. I was wondering with the availability of animation and computer graphics why people don't use these methods to create datasets ?

For one thing they are much, much simpler to annotate (for example for instance segmentation) since we already know the objects in the scenes. And we will have a lot more variety in the dataset since we have a much bigger freedom in in choosing and manipulating the objects in the scene (type of objects, color, lighting).

I do understand that for some more advanced tasks like analysing how humans interact with the environment the animations would be less realistic, but for tasks like instance segmentation it seems to me that the datasets would be accurate enough.

I do understand that creating these scenes (specially if we want to create an animation) could also be time consuming but wouldn't that be offset by the ease of annotation ? (specially for 3D scenes)

So what am I missing here ? why don't we have more of these types of datasets ? what is the drawback of using such a dataset ?

Answer 1

Well, when you're training something against a dataset, you're training it to work on that kind of data.

Since we have no idea of what the neural network "uses" from the input images to fulfills its purpose, these might be things where our computer graphics aren't quite like real-world images.

We're "trained" with real-world imaging, perceived through our eyes with our brains. The neural net has none of that. It might simply do the segmenting, and that's actually pretty likely, on the specific kind of anti-aliasing done by the 3D renderer. Great, now you have a neural net that can segment generated images very well, but can't deal with real-world photography. Not quite sure what you've gained there!

But: sometimes, you actually do that, if you can describe your real-world data well enough – but that's usually things like RF signals, abstracts datasets etc., and often you'd intentionally add noise, so that the noise equivocation is more prominent than the differences between synthesized data and the real world.

You'd always have to verify, either way.