Unsupervised clustering of image data is tricky thing and requires adjusting the method to the content of the images set.
Assuming we're dealing with the MNIST data set we can do some nice things using known tools.
First, let's assume we're after 2 features, namely we're after a dimensionality reduction from 784 features / dimensions to 2.
The first approach you suggested, using Mean and Variance isn't fruitful for images.
This can be shown form the following image:
As we can see, the mean and variance features doesn't create separation between different classes of the data.
The other algorithm in your question, the t-SNE, originally developed for visualization is very capable in tasks like this.
It can reduce the dimensionality while keeping the classes information:
The idea is finding a method to embed data which is similar in high dimension closely in low dimension and keep data which is not similar far away.
Indeed in the figure above one could see that in most cases same digits are grouped together.
After the t-SNE we saw few other similar dimensionality reduction algorithm which are as capable (Even more) like UMAP.
The code is available at my StackExchange Codes Signal Processing Q80767 GitHub Repository (Look at the