My computer vision / image processing professor uses the term "Label", to represent part of an image, but cannot explain in a way that makes any sense to me what it is. She says this is a somewhat common term, but it is not clear what it is. Its somewhat similar to a pixel, but different. I can't find a definition anywhere. Any ideas / definitions?


Let me try to write a bit more encompassing answer. First of all, as @Aaron hinted in his answer, a label can be quite a general term. It can pertain to different things depending on your application.

So, generally, it's as @Aaron says: the labels correspond to class names in any occasion when you have to classify all or some of the image pixels. Also, as @sansuiso noted, the exact meaning of the labels depends on the context.

Labels can be both computer assigned, _computer determined / deduced _, or human assigned (usually by a person considered an "expert" in a certain field). Let me give some examples:

  • if the application is image segmentation

    There's several possible options here. The simplest one will be from the test sets, or ground truth sets for the segmentation: the "experts" will be asked to segment the image, and then name each part / segment they marked.

    The second one is running an segmentation algorithm without relying on test sets for learning. The image will be segmented somehow (maybe in to predetermined number of regions, or depending on signal to noise ration). Those regions will simply be assigned number labels by the algorithm: region 1, region 2, etc.

    The third option might come from combining the results of a segmentation algorithm with a ground truth database: the algorithm might then try to determine the actual descriptive labels for the image regions.

  • usages in object detection or object classification

    For both of this, you primarily need examples. Examples are ideally the images containing only the object you want to learn how to detect or classify, but sometimes can contain more objects. Each image in the database needs to be processed by a human, each object correctly marked ("segmented") and labeled (e.g. "bird", "cat", "tree", "book") to create the learning sets.

    One example of such a database is LabelMe, where MIT is asking people to label some images in their free time if they feel like it.

    Using those kind of databased, you can then do object detection or classification. For classification, you will usually only assign one label per image. You try to find the one object the image represents, and then label/classify the image by that. An image might then get the label "bird" or "dog".

    For object detection, only one or several regions of your image will be "marked" or "segmented" by your algorithm. Then each of those regions is assigned a label. Optionally, you might assign a label "background" to all remaining image pixels.

  • satellite image understanding (also kind of classification)

    I've seen examples when, for a satellite image, you have a label assigned to every pixel from a predetermined set of labels (usually "building, vegetation, road, water, ground" or something similar). You have a small example image, or maybe just 10% of your image is labeled. You then try to learn the labels for each pixel in the rest of the image.

    I'll try and find links to the works doing this, can't remember them right now.

As you can see, you can label very different things: the whole image, each pixel individually, isolated regions of the image, or all the regions in a segmentation.

Additionally, labels can be human determined, generic (numeric) labels determined by an algorithm, or learned (so that they are human-understandable) based on examples marked/labeled by humans.


A label is some unique ID that you attach to part of an image or to a pixel. You can have as many labels as objects (or classes) of interest for your application.

Some examples:

  • in connected component labeling, pixels are assigned a label that is usually just a number corresponding to the ID of the connected region the pixels belong to;
  • in object detection/recognition, the label is the name or the index of the class that a detected object belongs to;
  • in face recognition, the label will be the name of the identified person whose face is given as input;
  • etc.
  • $\begingroup$ So it doesn't necessarily mean anything without specific context? "Label == context specific ID == name"? $\endgroup$ – ConfusedStack Dec 15 '13 at 16:39
  • $\begingroup$ Yes, exactly. You need a context to correctly define the meaning. $\endgroup$ – sansuiso Dec 16 '13 at 8:17

In computer vision you are often trying to classify an image or parts of an image. The labels correspond to the class names. For example, if you were doing gender classification on an image of a face then the labels would be male and female. If you were doing segmentation then the labels could correspond to sky, car, street and so on.

  • $\begingroup$ Should a Label be an Object or an Attribute of the image? $\endgroup$ – ConfusedStack Dec 15 '13 at 16:41
  • $\begingroup$ It depends on what you are doing. If you are doing object recognition then it will be an object. If you are doing something else it might be an attribute $\endgroup$ – Aaron Dec 15 '13 at 19:35

I think the underlying distinction is called "level of measuerment" in theory of science.

Simply put, the question is what operations make sense for a value:

  • the total energy measured by a CCD sensor at some pixel is a ratio quantity: You can compare two brightnesses to find out which one is brighter than the other, you can add and subtract them, you can divide two brightness values or multiply them by a scalar constant, and you will get a meaningful result

  • the typical example for an ordinal property is Mohs scale of mineral hardness. If mineral A has a hardness value of 4 and mineral B has a hardness value of 2, then A will scratch B. You can compare hardness values. Of course, you can also add and subtract mineral hardness values, but the result is meaningless.

    If your camera's sensor isn't linear and you don't know the functional relationship between light energy and the resulting pixel value, you should treat the pixels as ordinal properties.

  • a nominal property is probably what your professor means by "label": A typical example is nationality (French, American, German, ...). Even if you represent them as integers in computer memory, you can't sensibly add or multiply these values. Greater than/less than between two nationalities isn't meaningful, either. The only meaningful relation between two labels is equality/inequality.

  • $\begingroup$ This makes sense. I will ask if that is it when I can. Thanks! $\endgroup$ – ConfusedStack Dec 15 '13 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.