5
$\begingroup$

This is a a continuation of the discussion from Unsupervised Clustering of Images.

Image that we have MNIST database and we want to separate all the images like this. But we want to use Linear Discriminant Analysis (LDA) instead to see how the results differ.

Question

Will LDA perform better than t-SNE/UMAP if we know the classes of the data? Let's say that the X and Y axis of the plot is covaraiance.

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The Linear Discriminant Analysis (LDA) (Also the Fisher's Linear Discriminant, which the LDA is a generalization of) is a method to find a projection plane to separate data by linear projection Matrix multiplication).
Its main limitation is the use of linear projection.
On the other hand, it can be used in a supervised manner. Namely it can use the labels to find the optimal projection.

I implemented LDA in MATLAB and compared to the t-SNE from the previous question.

Supervised Dimensionality Reduction by LDA:

enter image description here

UnSupervised Dimensionality Reduction by t-SNE:

enter image description here

As one can see, though the LDA is supervised it can't compete with the t-SNE results. Though LDA could be very useful in other cases (Usually with fewer dimensions).
For instance, in order to validate my LDA implementation I used the UCI Machine Learning Repository Wine Data Set. I got the following result:

enter image description here

The code is available at my StackExchange Codes Signal Processing Q80949 GitHub Repository (Look at the SignalProcessing\Q80949 folder).

Resources

I found some resources about supervised dimensionality reduction:

Improve this answer
$\endgroup$
11
  • $\begingroup$ I see. LDA perform much worst compared to t-SNE. If I would have data that I need to separate from e.g if I have 10 classes of data, then I create a linear projection matrix $X$ and when I multiply that $X$ matrix with a specific new unknown data class, then that data class will "move avay" from other data. I don't now how to explain, but in LDA, you can find a $W$ matrix that project the data (e.g a vector) to another plane. Assume that you have $x$ vector, but $X*x = y$ where $y$ is a new projection. Can t-SNE create a projection matrix as LDA can? Or is t-SNE only a optimization algorithm? $\endgroup$
    – euraad
    Jan 10, 2022 at 17:52
  • $\begingroup$ What you're after is transformation of new data. Unfortunately, t-SNE doesn't support it natively. You can do some things to approximate it but I'd not do it. Yet UMAP does support this and it is as powerful as t-SNE. So you can use it for that usage. $\endgroup$
    – Royi
    Jan 10, 2022 at 18:41
  • $\begingroup$ Ok. Yes, I think UMAP is the right thing because LDA is quite dizzy for me when I see the plots. It's like all scatters have melted togeher. But I don't want to re-compute UMAP for every new unknown data. Do you know UMAP? If you could extend your answer with UMAP? $\endgroup$
    – euraad
    Jan 10, 2022 at 19:14
  • $\begingroup$ By the way! I like your MATLAB code! Very easy to read. $\endgroup$
    – euraad
    Jan 10, 2022 at 22:33
  • $\begingroup$ There is no official UMAP for MATLAB. If you use Python it is easy. I try keeping my code clear and concise as sometime I need it down the road. The more verbose it is the easier for me to reuse it. $\endgroup$
    – Royi
    Jan 11, 2022 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.