Building a Pipeline for Image Classification / Clustering Tasks with Features Extractor and Dimensionality Reduction (Example on MNIST Data)

In MNIST, there are 28x28 images of hand written digits.
What features would one extract in order to classify then without any Deep Learning involved?

How does Dimensionality Reduction get in the processing pipeline?

• Comments are not for extended discussion; this conversation has been moved to chat.
– Peter K.
Jan 13, 2022 at 18:43
• @PeterK., Could we open this question after my edit?
– Royi
Jan 14, 2022 at 7:35
• @Royi yes, that looks better.
– Peter K.
Jan 14, 2022 at 12:12
• @Royi Yes! that would work. Jan 15, 2022 at 2:38

Feature Extraction

There are many modern known features for images. Among them:

Those are classic and popular features. Since the blossom of Deep Learning people are less and less invest in researching newer features.

Basically the optimal feature extraction per task is the one which does both extraction of relevant feature and in low dimension. There is no single optimal feature extractor. Though for natural images we can see that modern nets (Deep Learning) can generalize very well.

The attached code in the following section is general and you may use different features to get better results.

Image Classification Pipeline (Without Deep Learning)

Remark: We'll analyze this work using the MNIST Data Set. Though the result is nice I haven't cross validated any step. So it is a representation of the concept. Yet for real world one should optimize each block.

In the classic machine learning a standard pipeline could be something like:

                               Input Image
│
│
┌─────────────────────────────▼─────────────────────────┐
│                                                       │
│                      Pre Processing                   │
│                                                       │
└─────────────────────────────┬─────────────────────────┘
│
│
┌─────────────────────────────▼─────────────────────────┐
│                                                       │
│ Feature Extraction and / or Dimensionality Reduction  │
│                                                       │
└─────────────────────────────┬─────────────────────────┘
│
│
┌─────────────────────────────▼─────────────────────────┐
│                                                       │
│                        Classifier                     │
│                                                       │
└───────────────────────────────────────────────────────┘



Remark
The flow above using ASCII Flow.

The heavy block here is the Feature Extraction / Dimensionality Reduction / Feature Selection (They can be done sequentially but there is som.

Each image we have in MNIST is an 28x28 image. Namely we have data in $$\mathbb{R}^{784}$$.
We'll build our pipeline as following:

1. Pre Processing
We'll do nothing. Though one could greatly increase the number of training samples with some small tricks (Rotation, Noise, Shift, etc...).
2. Feature Extraction
We'll use the LPB Feature. This is a well known and pretty general feature for image processing tasks.
3. Dimensionality Reduction
We'll use LDA for Supervised Linear Dimensionality reduction. One could use PCA as well. But then the properties of the output of this stage will be completely different.
4. Classifier
We'll use SVM with Gaussian Kernel.

Remark: Pay attention that possibly using t-SNE directly on the data with a classifier would yield a better result. So in that context t-SNE is both a feature extractor and reduce dimensionality. Which any good feature extractor should do. But the issue would be using t-SNE for test set.

After applying the LDA we get (Looking only at the 2 first features):

The result is marginally better than what we got in Image Clustering Using Linear Discriminant Analysis (LDA) Compared to t-SNE / UMAP.

Using the classifier we get the following Confusion Matrix:

We got 91.48% success rate.
In our days it is not impressive at all. But still it shows a formulation of a pipeline.

The magic in Deep Learning is that the first layers are doing the job by learning instead of hard coded feature extractor / dimensionality reduction.

The code is available at my StackExchange Codes Signal Processing Q80949 GitHub Repository (Look at the SignalProcessing\Q80994 folder).

• Very good answer! I like it. Why did LDA perform better now with MNIST, and not before? Jan 16, 2022 at 0:56
• Can you show different views from gscatter(mF(:, 1), mF(:, 2), vTrainSetLabel); ? E.g gscatter(mF(:, 1), mF(:, 3), vTrainSetLabel); and gscatter(mF(:, 2), mF(:, 3), vTrainSetLabel); Jan 16, 2022 at 1:21
• If you would use SVM to classify MNIST, wihout LDA. How would you prepere your MNIST data then? Jan 16, 2022 at 2:22
• @MrYui, You may change the code to you like and see the results. The LDA performs better (Though not significantly) because the extractor did it job. Pay attention the extractor isn't linear, hence it has the potential to give information to the LDA which it can't infer on its own. People could reach even better results with SVM. You may apply SVM on the data itself. Just instead sending it to the extractor send it to the classifier. As written, The idea was to show the pipeline, not the optimized solution.
– Royi
Jan 16, 2022 at 7:52
• Ok. So I can use SVM for classify MNIST? Should I open a new question about that? I wrote an algorithm called Point In Polygon. It's my "SVM", but it only works for 2D data. Jan 16, 2022 at 9:43