The Cocktail Party Problem is a Blind Source Separation (BSS) problem.
Given a linear mixture of signals:
$$ \boldsymbol{y} \left[ n \right] = A \boldsymbol{x} \left[ n \right] $$
We're trying to estimate the signal $ \boldsymbol{x} \left[ n \right] $.
The model can get even more complex with $ A $ being time varying:
$$ \boldsymbol{y} \left[ n \right] = A \left[ n \right] \boldsymbol{x} \left[ n \right] $$
We have 3 main approaches to this problem:
- Probabilistic Approach
Looking at the signals as an ensemble of points of a distribution and find the linear coordinate transform to guarantee some property. The PCA approach tries to remove correlation (2nd moment information) while the ICA tries to remove correlation in higher moments (Basically statistical independence). - Time Signal Processing Approach
In case of 2 signals one of them being a reference we can use the adaptive decorrelation filter. Basically we're after removing any time correlation from the signals. - Spatial Signal Processing Approach
We can utilize the known location of the microphones in the room to create adaptive beamforming. The idea is that delayed adaptive summation of the data can change the spatial curve of the array and making matching a certain direction.
Of course in late years we can find work on the subject utilizing Deep Learning approaches. Their main advantage is being able to incorporate additional information (Like using the properties of the signal, be it in a certain language, incorporating visual data on the scene like images and videos [Who is moving the lips when?]).
This is a vast subject and the main idea is to tailor the solution to the specific case of yours.
Modern Robust ICA and IVA (Independent Vector Analysis) can be very effective.
I'd try them first unless you have the case which matches the Adaptive Filter (Which can be proven to be matching the Beam Forming solution under some conditions).