# How to Extract Nonorthogonal PCA Principal Components

My problem is essentially a 'blind source separation' problem. I have 3 non-orthogonal sources (or basis functions) and $N$ random linear combinations (mixes) of said sources. My problem is to obtain the sources from the mixes.

Figure A shows the sources, B shows the mixes.

Approaches taken:

1. $\textbf{PCA}$ - I tried PCA on the mixes (fig.C), but the issue is that PCA will only give orthogonal bases, while my sources/bases are non-orthogonal. This issue with PCA is shown in figure D, where the data is clearly described by 2 non-orthogonal basis, but PCA (the solid lines) cant reconstruct them!
2. $\textbf{Factor Rotation}$ - I tried applying some solutions form Factor Analysis (not my forte). The promax rotation (matlab: nw = rotatefactors(cov,'method','promax') ) is shown in fig. D with the dashed lines. As far as I can tell, factor rotation works with the principal components, not the original matrix and thus I have no idea how it could reconstruct the right basis. I think this only works with factor matrices, not with generic ones...
3. $\textbf{ICA}$ - I tried overdetermined ICA with the fastICA algorithm (also not my forte, but I think I understand it better). I was hoping this would work since independent components (ICs) are non orthogonal. The solution is shown in fig.E. While the ICs are indeed nonorthogonal, they are NOT my original sources :-(

Any other potential tips or leads or solutions would be greatly appreciated. More mathematical Detail

The 3 source signals are themselves linear combos of the 5 laguerre basis functions. Thus, the problem can be formulated as such:

1. I take l=5 laguerre basis functions of length m=30 to get B, an [l,m] matrix
2. I randomly mix then is my x=3 sources, S: S=B*M1, where M1 is a random [l,x]=[5,3] mixing matrix. S is displayed in fig.A
3. I mix S n=100 times to get my mixed signals, K (fig.B), where K=S*M2, a [x,n]=[3,100] randomly generated mixing matrix.
4. I have the original basis, L, & the mixed matrix, K. K=S*M2=B*M1*M2. My goal is to recover S.

NOTE: I understand that this may seem impossible since there are many possible S's to choose from (ie the factorizatio S*M2=B*M1*M2 is not unique), however, the true S, will dramatically decrease model complexity by having only x=3 rather than l=5 basis functions which can exactly reproduce all the kernels.

Thanks!

• There is still no way to uniquely find S. You have there the equation K = SM2 = BM1*M2. This is not enough information to solve the problem. In other words, given K and B, there will be multiple ways of allocating M1 and M2 that both satisfy your equation. PCA is one of those ways but you said it is unsatisfactory to you. You need to put more constraints on M1 and M2 if you want to get a specific solution. Dec 26, 2013 at 6:01
• I think you should add an additional constraint, namely that the basis you look for makes your representation sparse. If I understand the problem correctly, you want your vectors in the basis you're looking for to either look like [a 0] or [0 b]. This is a problem very similar to compressed sensing, where I think you will find inspiration to solve it. Dec 26, 2013 at 10:18
• What linear combinations of the 5 PCA vectors give the 3 sources, in this particular case ? Are there near-0 coefficients that you can drop ? Jan 6, 2014 at 18:12

Let's say x,y and z are your basis vectors and you mix them to get what you are calling your mixed sources. If your basis vectors were (x-y), y, and (z+y) then you could mix them in a different way and end up with the same mixed sources.

Basically, there are many ways to come up with a set of basis vectors for your signals. You need to say what kind of basis you want. It is not enough to say that the basis vectors are non-orthogonal.

• This makes sense, however in the case of PCA (fig.D), intuitively I would think that there is a way to recover those clearly demarcated nonorthogonal components. If I can solve that problem I can solve my problem... Dec 26, 2013 at 1:08
• I'm not sure what you mean by non-orthogonal components. Dec 26, 2013 at 5:56

PCA requires the data to be orthogonal, so make it that way. I will assume a 2 signal source(i.e. 2D) for ease of explanation and visualization.

So add axes to make the data orthogonal in higher dimensions.

There are many ways to accomplish this task with iteration and conditions:

The first component of the PCA, black solid line, will be somewhere between the primary components. It tends to lean (not guaranteed, so you may need to check and randomly drop out samples to force the lean towards one line, it does not matter which one.)

Create two(or three for 3D spaces) new axes for each previous X or Y(X1) (or Z(X2)) coordinate with a Y value as the distance from the PCA calculated primary axis line.

This will collapse the primary PCA axis in each new coordinate system and allow the secondary PCAs to emerge for each new axis. Revealing an approximation original equations.

The cross product of the new PCA axes will allow refinement of the original sources, starting the process over until the required solution condition is met based on convergence of the error. If the error diverges, choose a new initial step:

The challenge occurs in the first step, promax may be used as an alternative to PCA the first step or subsequent steps or a mix.

Calculating a rotational center, through convergence of every axis line, using source translated polar coordinates as additional source dimensions would improve accuracy in the final calculations.

Due to the iterative nature based on assumptions, this answer should converge to one of many solutions, based on the initial parameters.

Any improvements are welcome.

A map from a weights vector $w$ to a ,,mixed signal'' $S$ would be just your 3 prototype signals as a matrix

$A=\left[\matrix{x & y & z}\right]$

so that a linearly combined signal could be written as

$S=Aw$

A ,,pseudoinverse'' to this matrix $A$ (in a few words: a matrix $A'$ so that $A'A=\mathbf{1}$) would map that signal back to $w$:

$A'S=w$

The problem is then finding such a pseudo-inverse. However, this problem is well-known and implementations are available.

• If I understand you correctly, I don't think this will work. The issue is that the 'weights' are randomly generated for each mixed signal. Thus if we have 100 mixed signals, k_1 ..k_100, each of them would have different weights so that Aw_i=k_i & Aw_j=k_j. Dec 26, 2013 at 1:04
• You'd of course have to multiply each signal seperately. In a certain numerics software, I'd write: > A=exp(linspace(1,-4,100)).*real(exp(I*linspace(0,15,100))); > B=exp(linspace(0.5,-5,100)).*real(exp(I*(3+linspace(0,25,100)))); > C=exp(linspace(1,-3,100)).*real(exp(I*(1+linspace(0,17,100)))); > # just making up some functions > K=[A',B',C']; # this is "A" > U=K*[0.5;0.7;0.3]; # this is "S" and "w" > pinv(K)*U ans = 0.50000 0.70000 0.30000 - of course I'd have to repeat the last step for different w, and I would store pinv(K) then. I'm sorry. I can't break lines in comments. Dec 26, 2013 at 1:09
• He's asking for a solution to the blind source separation problem. Both the matrix A and w are unknown. So it's impossible to find A with a pseudo-inverse of S. Dec 26, 2013 at 5:49
• How did he get the "original signals"? Dec 26, 2013 at 11:20