Apply Principal Component Analysis (PCA) for RGB Images

Question

I've implemented a method to compute PCA on grayscale images.

I haven't seen PCA on RGB images yet, which left me wondering if it is possible to perform it.

With RGB images, is PCA done for each color channel separately? If so, is there any loss of information or inconsistent data when you combine the three channels?

The code I used is from the Medium blog post PCA (Principal Components Analysis) Applied to Images of Faces.

Does anyone know of any PCA implementations for RGB images?

Answer 1

General Idea

The general idea of Principal Component Analysis (PCA) is as following (Intuition over formalism):

Given a set of points in space (Inner Product Space) find a set of vectors (Directions) which are uncorrelated which span the data in the most energy preserving manner.

The tricky part is explaining "most energy preserving manner".
So we're talking on $ {L}_{2} $ norm and preserving means that if we ask for a given number of vectors, no other vectors will be able to approximate better the set in the $ {L}_{2} $ sense.

Working on Images

For a set of color images of size $ m \times n \times 3 $ we can work in either of the following approaches:

1. Vectorize image into single vector of size $ 3 m n $ and use regular PCA.
2. Use Tensor based approaches (Like the generalization of SVD to Tensors).

Task

The task is to build a dictionary for compression based on the PCA idea.
The data set for the task is based on the Utrecht ECVP face images data set.
The data set is composed of 131 images of size 900 x 1200 which were resized to 120 x 180.
In each approach the images were loaded image, converted to Float64 and scaled to range [0, 1]

Image Vectorization

Each images was vectorized to a single vector of length 64,800.
Taking all the images and removing the Mean Vector to create the data set.
The using SVD extracting the dictionary which spans the columns (Matrix $ U $ in the SVD).

Taking Image #1 in the data set with various numbers of components of the SVD:

The results are not as good as we usually get with Gray Scale images.
The reason that in the process the structure of image (3 Channels) is ignored and not leveraged.

So doing the same trick for each channels separately yield the following:

There is a small improvement (See the result for 75 components) but not significant.

Could we even do better?

Image as a Tensor

The trick here is we have many ways to project a tensor:

1. Project onto a tensor of a different space.
2. Project onto a vector.

Different approaches leads to different results as can be seen in A Survey of Multilinear Subspace Learning for Tensor Data:

MATLAB Code

The full code is available on my StackExchange Signal Processing Q58730 GitHub Repository (Look at the SignalProcessing\Q58730 folder).

Answer 2

One option is called Multilinear principal component analysis:

Multilinear principal component analysis (MPCA) is a multilinear extension of principal component analysis (PCA). MPCA is employed in the analysis of n-way arrays, i.e. a cube or hyper-cube of numbers, also informally referred to as a "data tensor".

And a RGB image is a sort of cube.

Now, is the RGB domain the best one for further analysis? Some prefer a prior decomposition (LAB, YUV). In multispectral imaging, different transformations across each dimensions are often preferred.

Answer 3

I've just recently reviewed a paper that used the t-svd as a multidimensional extension to the PCA. They've explicitly tested this on RGB images and claimed to achieve good results. Might be another candidate solution to look at.

Answer 4

A PCA is just an orthogonal transform of one vector from an (inner-product) vector space to another vector, whose dimensions are statistically uncorrelated.

So, it works on any inner product space, and since an RGB image simply has two dimensions more than a greyscale image, of course it works.