I see in many reviews on Autonomous Car how they segment the images with person, cars, etc...

How is it achieved in Deep Learning?
Could anyone give an example of that?
How it is done?

  • $\begingroup$ Are you after the case of pre defined classes or object segmentation? $\endgroup$
    – Thomas
    Nov 29, 2021 at 8:36
  • $\begingroup$ Example of either will be great. $\endgroup$
    – David
    Nov 29, 2021 at 18:11

1 Answer 1


Well, I think the best way to tackle this question is a little background and a code as an example. I chose MATLAB for this example though PyTorch / Keras would probably be as easy.

This task requires the output to have the same dimensions as the input with per pixel classification (For other masks it could be regression as well).

A modern CNN architecture to tackle such task is the U-Net.
Utilizing skip connections allows is having a simple architecture while being very efficient in learning.

I created a function to build such net in MATLAB:

enter image description here

We basically have 2 nets with one begin encoder like and the other being decoder. The skip connections between the encoder and the decoder are the special sauce of this kind of nets.

In my net the merging of data from the skip connections is done by concatenation. One could chose other options as well (Add, Multiply, etc...).

To show how easy is to build and train such net I chose the Oxford Pets Data Set:

enter image description here

This simple dataset has enough samples with good masks.
We try to classify each pixel as either the pet, the background or the border between the two. I resized all images into 128x128 (One could also employ data augmentation which I skipped to keep things simple).

I ran the net for 15 Epochs:

enter image description here

As can be seen, It is easy to get almost 90% accuracy.
While it is nice, it is not a good metric for this kind of task for many reasons. One of them is that the data isn't well balanced so while it is easy to get the background pixels and most of the pet pixels, the border, which is the important part, might be badly missed.

Here is the result on one of the images of the set at the end of training:

enter image description here

We can see improvement over time but still far from very good. Data augmentation, label weight balancing, more epochs and more intelligent Learning Rate policy would get much better results.

The code is available at my StackExchange Codes Signal Processing Q79314 GitHub Repository (Look at the SignalProcessing\Q79314 folder).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.