Can Principal Component Analysis (PCA) Solve the Cocktail Party Problem?

I'm looking into the cocktail party problem and trying to figure out whether something like Principal Component Analysis is enough to separate out all the various voices at the cocktail party into its constituent sound sources.

If its not enough, why? What other techniques should be used in conjunction with it, so that I wind up with distinct signals for each cocktail party patron's voice? Spatial filtering, such as various beamforming methods, has been suggested. But in researching PCA it seems that should (possibly) be enough to "split" the total signal of the cocktail party into individual signals for each human voice attending the party. Beamforming and similiar methods appear to be filters for subsequently focusing on just one of those voices and filtering the rest out.

Can anybody with PCA experience weigh in here with whether or not PCA can solve this, or if its requires additional processing?

• The technical term is "blind source separation", and spatial diversity is highly recommended. Dec 29 '20 at 17:52
• They seem somewhat related, but I can’t speak to what exactly was meant by diversity. There’s a reason consumer smart speakers use an array of microphones to better separate the user from ambient sounds and other people. Common solutions are to use directional microphones semi-independently, or to use omnidirectional microphones and perform spatial filtering with hard/firm/software beam steering techniques. Dec 30 '20 at 3:18
• No, usually the microphones are identical. The objective is to separate sources purely based on their location in physical space. Beamforming/steering are the same thing in this context and are a type of spatial filtering. Delay-and-Sum beamforming would be a good place to start. For lower power computing, you may be better off with a directional array instead of trying to beamform on the device. There are challenges with audible sound beamforming as it is extremely wide bandwidth compared to the constituent frequencies which makes the lobe width non-uniform. Dec 30 '20 at 15:37
• If PCA or ICA were the solution to the cocktail party problem, you'd have to have at least as many ears as party guests. So, obviously not. Identifying and separating sound sources uses many cues, of which spatial cues are not even very important. Even a single channel without spatial information can be processed by the human brain and allow for identification of several simultaneous speakers and other sound sources . The greatest contribution is internal model-building in your brain. You know what sounds are there and know them individually. For speech, this is by far the paramount factor. Dec 31 '20 at 17:50
• Are you after some intuition why PCA / ICA work? I must say they don't work well all the time. There are cases they work. It depends on the assumption on the signals an their mixture.
– Royi
Jan 1 at 2:00

The Cocktail Party Problem is a Blind Source Separation (BSS) problem.
Given a linear mixture of signals:

$$\boldsymbol{y} \left[ n \right] = A \boldsymbol{x} \left[ n \right]$$

We're trying to estimate the signal $$\boldsymbol{x} \left[ n \right]$$.
The model can get even more complex with $$A$$ being time varying:

$$\boldsymbol{y} \left[ n \right] = A \left[ n \right] \boldsymbol{x} \left[ n \right]$$

We have 3 main approaches to this problem:

1. Probabilistic Approach
Looking at the signals as an ensemble of points of a distribution and find the linear coordinate transform to guarantee some property. The PCA approach tries to remove correlation (2nd moment information) while the ICA tries to remove correlation in higher moments (Basically statistical independence).
2. Time Signal Processing Approach
In case of 2 signals one of them being a reference we can use the adaptive decorrelation filter. Basically we're after removing any time correlation from the signals.
3. Spatial Signal Processing Approach
We can utilize the known location of the microphones in the room to create adaptive beamforming. The idea is that delayed adaptive summation of the data can change the spatial curve of the array and making matching a certain direction.

Of course in late years we can find work on the subject utilizing Deep Learning approaches. Their main advantage is being able to incorporate additional information (Like using the properties of the signal, be it in a certain language, incorporating visual data on the scene like images and videos [Who is moving the lips when?]).

This is a vast subject and the main idea is to tailor the solution to the specific case of yours.
Modern Robust ICA and IVA (Independent Vector Analysis) can be very effective.
I'd try them first unless you have the case which matches the Adaptive Filter (Which can be proven to be matching the Beam Forming solution under some conditions).

• Great answer @Royi (+1) and thank you! -- when it comes to beamforming, my understanding is you first need to apply an SSL (sound source location) algorithm to approximate the 3D coordinate of the sound source. Do you have any recommendations/inputs on what types of SSL algorithms would be most effective for this type of problem? Jan 1 at 15:38
• I think beamforming shines when you have control on the location of the microphones. Then you can arrange them such that the problem is 1D (Angle). Once you do that, you may need or not know the location of the speakers. It depends what's the goal. For instance if it is a pre process step for Speech Recognition algorithm you may try few angles and see how it affects the success rate of the recognition (Basically creating an SSL). If it is a stand alone block, indeed, you'd need some prior knowledge or algorithm to detect time of a single speaker and infer its angle by maximizing energy.
– Royi
Jan 1 at 15:45
• I suggest the brain does something like MMSE estimation of the ambient signals and successive interference cancellation (MMSE-SIC). Our ears+brains can ignore/subtract more than spatial processing alone could explain. The furnace fan humming, the microwave, the piano playing in the corner, etc. They have distinct time/freq signatures. Alternately, I suspect it would be impossible to reliably point blindfolded to one of two white noise generators in a room (at similar levels). Your ears are only two receivers. Jan 5 at 20:00
• I think the brain does something like scene recognition and prediction. Just like you see colors how the brain thinks they should be rendered. When we're in a loud scene the brain uses previous knowledge and its estimation to generate the signal to be muted.
– Royi
Jan 5 at 20:03

Speech Source Separation (SSS) or Audio Source Separation (ASS) can be seen as a specialized version of source separation. I mention these expressions under which one can find additional works. One acceptation of the "Cocktail Party Problem" is the task of hearing/recovering one specific sound of interest in a complex environment (one-source versus all), while your objective seems more ambitious: unmixing all sources (another common term). The 2015 paper The cocktail-party problem revisited: early processing and selection of multi-talker speech reviews speech-related and perception issues.

The possibility of their identification depends a lot on the quality and quantity of observations, and the formation model of the observed signals. If the model is linear and instantaneous, this is already complicated. Single channel source separation is a specific topic of interest. In non-linear environments, when the number of sources is greater than the observations, when convolutive effects or echoes happen, when noise is difficult to tackle, blind source separation techniques are unlikely to succeed without additional priors and ancillary information/models.

Humans use their binaural features a lot in this context. Therefore, using spatial information from similar sensors is useful, but this can be insufficient. Indeed, there is a whole domain of Audio-Visual Speech Source Separation because:

The separation of speech signals measured at multiple microphones in noisy and reverberant environments using only the audio modality has limitations because there is generally insufficient information to discriminate fully the different sound sources. Humans mitigate this problem by exploiting the visual modality which is insensitive to background noise and can provide contextual information about the audio scene.

Combining audio and visual sensors is an instance of increasing the diversity in the sources: the more the sources can avoid to overlap in the recorded domain, the highest the chance of separation. PCA is very limited to that respect because it is too-strongly-tied to correlation and orthogonality. It is linear, nonparametric and cannot (easily) incorporate prior knowledge. It can estimate decorrelated components, up to a rotation. In other words, suppose that we two are talking. A PCA could detect the following two sources: 1) your voice minus mine 2) your voice added to mine, not what you'd expect. However, as a whitening or compression method, PCA can be used as a preprocessing to other methods like Independent Component Analysis (ICA), see for instance A. Hyvarinen, J. arhunen, E. Oja, Independent Component Analysis, 2001.

Additional speech features, like sparsity in a time-frequency domain (see Sparse Representations for the Cocktail Party Problem, 2006), more informed (less-blind) source separation (see From blind to guided audio source separation: How models and side information can improve the separation of sound) can help.

Finally, the link Deep Learning Machine Solves the Cocktail Party Problem may provide some pointers to the use of machine learning and artificial intelligence.

• I suggest you to have a look to this video, the cocktail party problem is well explained and their results are astonishing ! youtube.com/watch?v=4SMWwpSrZE4&feature=emb_logo Jan 7 at 14:53
• I know this guy Jonathan Le Roux! Jan 7 at 19:27