Can Principal Component Analysis (PCA) Solve the Cocktail Party Problem?

Question

I'm looking into the cocktail party problem and trying to figure out whether something like Principal Component Analysis is enough to separate out all the various voices at the cocktail party into its constituent sound sources.

If its not enough, why? What other techniques should be used in conjunction with it, so that I wind up with distinct signals for each cocktail party patron's voice? Spatial filtering, such as various beamforming methods, has been suggested. But in researching PCA it seems that should (possibly) be enough to "split" the total signal of the cocktail party into individual signals for each human voice attending the party. Beamforming and similiar methods appear to be filters for subsequently focusing on just one of those voices and filtering the rest out.

Can anybody with PCA experience weigh in here with whether or not PCA can solve this, or if its requires additional processing?

Answer 1

The Cocktail Party Problem is a Blind Source Separation (BSS) problem.
Given a linear mixture of signals:

$$ \boldsymbol{y} \left[ n \right] = A \boldsymbol{x} \left[ n \right] $$

We're trying to estimate the signal $ \boldsymbol{x} \left[ n \right] $.
The model can get even more complex with $ A $ being time varying:

$$ \boldsymbol{y} \left[ n \right] = A \left[ n \right] \boldsymbol{x} \left[ n \right] $$

We have 3 main approaches to this problem:

1. Probabilistic Approach
   Looking at the signals as an ensemble of points of a distribution and find the linear coordinate transform to guarantee some property. The PCA approach tries to remove correlation (2nd moment information) while the ICA tries to remove correlation in higher moments (Basically statistical independence).
2. Time Signal Processing Approach
   In case of 2 signals one of them being a reference we can use the adaptive decorrelation filter. Basically we're after removing any time correlation from the signals.
3. Spatial Signal Processing Approach
   We can utilize the known location of the microphones in the room to create adaptive beamforming. The idea is that delayed adaptive summation of the data can change the spatial curve of the array and making matching a certain direction.

Of course in late years we can find work on the subject utilizing Deep Learning approaches. Their main advantage is being able to incorporate additional information (Like using the properties of the signal, be it in a certain language, incorporating visual data on the scene like images and videos [Who is moving the lips when?]).

This is a vast subject and the main idea is to tailor the solution to the specific case of yours.
Modern Robust ICA and IVA (Independent Vector Analysis) can be very effective.
I'd try them first unless you have the case which matches the Adaptive Filter (Which can be proven to be matching the Beam Forming solution under some conditions).

Answer 2

Speech Source Separation (SSS) or Audio Source Separation (ASS) can be seen as a specialized version of source separation. I mention these expressions under which one can find additional works. One acceptation of the "Cocktail Party Problem" is the task of hearing/recovering one specific sound of interest in a complex environment (one-source versus all), while your objective seems more ambitious: unmixing all sources (another common term). The 2015 paper The cocktail-party problem revisited: early processing and selection of multi-talker speech reviews speech-related and perception issues.

The possibility of their identification depends a lot on the quality and quantity of observations, and the formation model of the observed signals. If the model is linear and instantaneous, this is already complicated. Single channel source separation is a specific topic of interest. In non-linear environments, when the number of sources is greater than the observations, when convolutive effects or echoes happen, when noise is difficult to tackle, blind source separation techniques are unlikely to succeed without additional priors and ancillary information/models.

Humans use their binaural features a lot in this context. Therefore, using spatial information from similar sensors is useful, but this can be insufficient. Indeed, there is a whole domain of Audio-Visual Speech Source Separation because:

The separation of speech signals measured at multiple microphones in noisy and reverberant environments using only the audio modality has limitations because there is generally insufficient information to discriminate fully the different sound sources. Humans mitigate this problem by exploiting the visual modality which is insensitive to background noise and can provide contextual information about the audio scene.

Combining audio and visual sensors is an instance of increasing the diversity in the sources: the more the sources can avoid to overlap in the recorded domain, the highest the chance of separation. PCA is very limited to that respect because it is too-strongly-tied to correlation and orthogonality. It is linear, nonparametric and cannot (easily) incorporate prior knowledge. It can estimate decorrelated components, up to a rotation. In other words, suppose that we two are talking. A PCA could detect the following two sources: 1) your voice minus mine 2) your voice added to mine, not what you'd expect. However, as a whitening or compression method, PCA can be used as a preprocessing to other methods like Independent Component Analysis (ICA), see for instance A. Hyvarinen, J. arhunen, E. Oja, Independent Component Analysis, 2001.

Additional speech features, like sparsity in a time-frequency domain (see Sparse Representations for the Cocktail Party Problem, 2006), more informed (less-blind) source separation (see From blind to guided audio source separation: How models and side information can improve the separation of sound) can help.

Finally, the link Deep Learning Machine Solves the Cocktail Party Problem may provide some pointers to the use of machine learning and artificial intelligence.