Building a Pipeline for Image Classification / Clustering Tasks with Features Extractor and Dimensionality Reduction (Example on MNIST Data)

Question

In MNIST, there are 28x28 images of hand written digits.
What features would one extract in order to classify then without any Deep Learning involved?

How does Dimensionality Reduction get in the processing pipeline?

Answer 1

Feature Extraction

There are many modern known features for images. Among them:

1. BRISK Feature.
2. FAST Feature.
3. Harris Feature.
4. KAZE Feature.
5. MSER Feature.
6. ORB Feature.
7. SIFT Feature.
8. SURF Feature.
9. LBP Feature.
10. HOG Features.

Those are classic and popular features. Since the blossom of Deep Learning people are less and less invest in researching newer features.

Basically the optimal feature extraction per task is the one which does both extraction of relevant feature and in low dimension. There is no single optimal feature extractor. Though for natural images we can see that modern nets (Deep Learning) can generalize very well.

The attached code in the following section is general and you may use different features to get better results.

Image Classification Pipeline (Without Deep Learning)

Remark: We'll analyze this work using the MNIST Data Set. Though the result is nice I haven't cross validated any step. So it is a representation of the concept. Yet for real world one should optimize each block.

In the classic machine learning a standard pipeline could be something like:

                               Input Image
                                    │
                                    │
      ┌─────────────────────────────▼─────────────────────────┐
      │                                                       │
      │                      Pre Processing                   │
      │                                                       │
      └─────────────────────────────┬─────────────────────────┘
                                    │
                                    │
      ┌─────────────────────────────▼─────────────────────────┐
      │                                                       │
      │ Feature Extraction and / or Dimensionality Reduction  │
      │                                                       │
      └─────────────────────────────┬─────────────────────────┘
                                    │
                                    │
      ┌─────────────────────────────▼─────────────────────────┐
      │                                                       │
      │                        Classifier                     │
      │                                                       │
      └───────────────────────────────────────────────────────┘

Remark
The flow above using ASCII Flow.

The heavy block here is the Feature Extraction / Dimensionality Reduction / Feature Selection (They can be done sequentially but there is som.

Each image we have in MNIST is an 28x28 image. Namely we have data in $ \mathbb{R}^{784} $.
We'll build our pipeline as following:

1. Pre Processing
   We'll do nothing. Though one could greatly increase the number of training samples with some small tricks (Rotation, Noise, Shift, etc...).
2. Feature Extraction
   We'll use the LPB Feature. This is a well known and pretty general feature for image processing tasks.
3. Dimensionality Reduction
   We'll use LDA for Supervised Linear Dimensionality reduction. One could use PCA as well. But then the properties of the output of this stage will be completely different.
4. Classifier
   We'll use SVM with Gaussian Kernel.

Remark: Pay attention that possibly using t-SNE directly on the data with a classifier would yield a better result. So in that context t-SNE is both a feature extractor and reduce dimensionality. Which any good feature extractor should do. But the issue would be using t-SNE for test set.

After applying the LDA we get (Looking only at the 2 first features):

The result is marginally better than what we got in Image Clustering Using Linear Discriminant Analysis (LDA) Compared to t-SNE / UMAP.

Using the classifier we get the following Confusion Matrix:

We got 91.48% success rate.
In our days it is not impressive at all. But still it shows a formulation of a pipeline.

The magic in Deep Learning is that the first layers are doing the job by learning instead of hard coded feature extractor / dimensionality reduction.

The code is available at my StackExchange Codes Signal Processing Q80949 GitHub Repository (Look at the SignalProcessing\Q80994 folder).