How to choose the most appropriate color space for a computer vision application?

Question

A lot of image processing techniques used in computer vision consist (among other things) in switching from RGB to another color space (HSV, YUV, LMS...) : color transfer, visual tracking... It seems that different people use different color spaces for the same application.

In the case of statistical color transfer for example, you can use LMS (which is a device-independent color space) or use YUV (a device-dependant color space) and achieve pretty much the same result. Moreover the transformation from RGB to either of these 2 color spaces is achievable by multiplying the RGB pixel by a 3x3 matrix.

Besides the device-dependence/independence factor, there's the decorrelation between the channels. The RGB channels are inter-correlated but so are LMS.

My questions are:

• What are the characteristics to look for in a color space in order to be able to say whether it is appropriate or not to use it? (If you think that the scope of my question is too broad, take the example of statistical color transfer from an image to another)
  
• And why is the decorrelation between the channels of any color space that important?

Answer 1

Computer vision is a huge topic and your particular example (statistical color transfer) is too narrow to provide guidance on the remaning parts.

That being said, classically most algorithms have worked on black and white "color" space, which was either obtained directly from the sensor or was later converted to it. The reasons behind is: 1-It's much simpler, 2- It provides remarkably good results for most applications unless color is explicitly required.

Color is, however, inevitably required for certain machine vision applications. Then comes the problem of selecting the most suitable color space. RGB is the fundamental color space because those R,G,B signals are physically captured by the sensor. (This also came from human color perception and additive color formation based on it)

Correlation in RGB channels should be understood clearly. In general given an R value you cannot say anything about G or B. But given a brightness value and hue, then RGB will (may) be correlated.

YUV is the most useful color space used in TV broadcast technology, image and video coding applications (codecs, cameras, camcorders) and many technical applications where color is necessary. For some reasons YUV was derived based on special requirements of color tv broadcasting over existing black and white tvs with compatibility in mind. Y channel carries the brightness of the scene and typically used to replace the black and white image. U and V channels carry color value and saturation information.

HSV is a more complicated color space to derive from RGB. It can provide more accurate color discrimination than YUV. The HSV is definining image aspects like Hue, Saturation and Brightness which are fundamentally uncorrelated (or less correlated compared to RGB)

For technical reasons other color spaces also exist but they do not provide any advantages for signal processing and especially for machine vision.