From your question, I take it that you are attempting to analyze, in a completely unsupervised way, unlabelled speech data so as to unveil two groups that should (hopefully) line up well with the "male" and "female" labels.
Cross-correlation is useless for speech application. The notion of speech similarity (be it at the higher level of the actual words pronounced, or at the lower level of the perceived timbre) does not line up at all with the raw waveform correlation. If you really want to align pairs of recordings, you'll rather have to match sequences of feature vectors (such as MFCC) rather than waveforms. Even then, the MFCC of two different speakers saying the same words do not match - at the very least you have to consider a linear transform from one set of vectors to the other.
You have not explained how you planned to use principal component analysis here (PCA is not a clustering method in itself). I assume that your intention was to represent each speaker with a humongous $N$-dimension vector representing the entire waveform of the recording for this speaker; run a PCA on this $20 \times N$ matrix to get a lower dimensional representation of say $20 \times 2$, and use this for clustering. It won't work because without further feature extraction, the raw waveform does not carry any meaningful information for speech analysis.
Here's how I would naively do it (and this wouldn't require any alignment):
- For each recording, extract the pitch and MFCCs (and maybe their first and second derivatives) and train a GMM on them.
- Compute the pairwise distance between the models. This gives you a $20 \times 20$ matrix with the distance between each model.
- Once you have this matrix, it is ripe for use with spectral clustering techniques, or maybe something as simple as hierarchical agglomerative clustering.
Note that in this answer I have tried to follow your "clustering" approach - how to make difference in genders apparent in a collection of speech recordings, without any prior annotation.
If your problem is truly that of recognizing gender, and if you can afford a supervised approach (annotating data + training a model), then things would have to be done differently...