# How to Extract Nonorthogonal PCA Principal Components

My problem is essentially a 'blind source separation' problem. I have 3 non-orthogonal sources (or basis functions) and $N$ random linear combinations (mixes) of said sources. My problem is to obtain the sources from the mixes.

Figure A shows the sources, B shows the mixes.

Approaches taken:

1. $\textbf{PCA}$ - I tried PCA on the mixes (fig.C), but the issue is that PCA will only give orthogonal bases, while my sources/bases are non-orthogonal. This issue with PCA is shown in figure D, where the data is clearly described by 2 non-orthogonal basis, but PCA (the solid lines) cant reconstruct them!
2. $\textbf{Factor Rotation}$ - I tried applying some solutions form Factor Analysis (not my forte). The promax rotation (matlab: nw = rotatefactors(cov,'method','promax') ) is shown in fig. D with the dashed lines. As far as I can tell, factor rotation works with the principal components, not the original matrix and thus I have no idea how it could reconstruct the right basis. I think this only works with factor matrices, not with generic ones...
3. $\textbf{ICA}$ - I tried overdetermined ICA with the fastICA algorithm (also not my forte, but I think I understand it better). I was hoping this would work since independent components (ICs) are non orthogonal. The solution is shown in fig.E. While the ICs are indeed nonorthogonal, they are NOT my original sources :-(

Any other potential tips or leads or solutions would be greatly appreciated. More mathematical Detail

The 3 source signals are themselves linear combos of the 5 laguerre basis functions. Thus, the problem can be formulated as such:

1. I take l=5 laguerre basis functions of length m=30 to get B, an [l,m] matrix
2. I randomly mix then is my x=3 sources, S: S=B*M1, where M1 is a random [l,x]=[5,3] mixing matrix. S is displayed in fig.A
3. I mix S n=100 times to get my mixed signals, K (fig.B), where K=S*M2, a [x,n]=[3,100] randomly generated mixing matrix.
4. I have the original basis, L, & the mixed matrix, K. K=S*M2=B*M1*M2. My goal is to recover S.

NOTE: I understand that this may seem impossible since there are many possible S's to choose from (ie the factorizatio S*M2=B*M1*M2 is not unique), however, the true S, will dramatically decrease model complexity by having only x=3 rather than l=5 basis functions which can exactly reproduce all the kernels.

Thanks!

• There is still no way to uniquely find S. You have there the equation K = SM2 = BM1*M2. This is not enough information to solve the problem. In other words, given K and B, there will be multiple ways of allocating M1 and M2 that both satisfy your equation. PCA is one of those ways but you said it is unsatisfactory to you. You need to put more constraints on M1 and M2 if you want to get a specific solution. – Aaron Dec 26 '13 at 6:01
• I think you should add an additional constraint, namely that the basis you look for makes your representation sparse. If I understand the problem correctly, you want your vectors in the basis you're looking for to either look like [a 0] or [0 b]. This is a problem very similar to compressed sensing, where I think you will find inspiration to solve it. – Jazzmaniac Dec 26 '13 at 10:18
• What linear combinations of the 5 PCA vectors give the 3 sources, in this particular case ? Are there near-0 coefficients that you can drop ? – denis Jan 6 '14 at 18:12

Let's say x,y and z are your basis vectors and you mix them to get what you are calling your mixed sources. If your basis vectors were (x-y), y, and (z+y) then you could mix them in a different way and end up with the same mixed sources.

Basically, there are many ways to come up with a set of basis vectors for your signals. You need to say what kind of basis you want. It is not enough to say that the basis vectors are non-orthogonal.

• This makes sense, however in the case of PCA (fig.D), intuitively I would think that there is a way to recover those clearly demarcated nonorthogonal components. If I can solve that problem I can solve my problem... – DankMasterDan Dec 26 '13 at 1:08
• I'm not sure what you mean by non-orthogonal components. – Aaron Dec 26 '13 at 5:56

A map from a weights vector $w$ to a ,,mixed signal'' $S$ would be just your 3 prototype signals as a matrix

$A=\left[\matrix{x & y & z}\right]$

so that a linearly combined signal could be written as

$S=Aw$

A ,,pseudoinverse'' to this matrix $A$ (in a few words: a matrix $A'$ so that $A'A=\mathbf{1}$) would map that signal back to $w$:

$A'S=w$

The problem is then finding such a pseudo-inverse. However, this problem is well-known and implementations are available.

• If I understand you correctly, I don't think this will work. The issue is that the 'weights' are randomly generated for each mixed signal. Thus if we have 100 mixed signals, k_1 ..k_100, each of them would have different weights so that Aw_i=k_i & Aw_j=k_j. – DankMasterDan Dec 26 '13 at 1:04
• You'd of course have to multiply each signal seperately. In a certain numerics software, I'd write: > A=exp(linspace(1,-4,100)).*real(exp(I*linspace(0,15,100))); > B=exp(linspace(0.5,-5,100)).*real(exp(I*(3+linspace(0,25,100)))); > C=exp(linspace(1,-3,100)).*real(exp(I*(1+linspace(0,17,100)))); > # just making up some functions > K=[A',B',C']; # this is "A" > U=K*[0.5;0.7;0.3]; # this is "S" and "w" > pinv(K)*U ans = 0.50000 0.70000 0.30000 - of course I'd have to repeat the last step for different w, and I would store pinv(K) then. I'm sorry. I can't break lines in comments. – user7358 Dec 26 '13 at 1:09
• He's asking for a solution to the blind source separation problem. Both the matrix A and w are unknown. So it's impossible to find A with a pseudo-inverse of S. – Aaron Dec 26 '13 at 5:49
• How did he get the "original signals"? – user7358 Dec 26 '13 at 11:20