How to Extract Nonorthogonal PCA Principal Components

Question

My problem is essentially a 'blind source separation' problem. I have 3 non-orthogonal sources (or basis functions) and $N$ random linear combinations (mixes) of said sources. My problem is to obtain the sources from the mixes.

Figure A shows the sources, B shows the mixes.

Approaches taken:

1. $\textbf{PCA}$ - I tried PCA on the mixes (fig.C), but the issue is that PCA will only give orthogonal bases, while my sources/bases are non-orthogonal. This issue with PCA is shown in figure D, where the data is clearly described by 2 non-orthogonal basis, but PCA (the solid lines) cant reconstruct them!
2. $\textbf{Factor Rotation}$ - I tried applying some solutions form Factor Analysis (not my forte). The promax rotation (matlab: nw = rotatefactors(cov,'method','promax') ) is shown in fig. D with the dashed lines. As far as I can tell, factor rotation works with the principal components, not the original matrix and thus I have no idea how it could reconstruct the right basis. I think this only works with factor matrices, not with generic ones...
3. $\textbf{ICA}$ - I tried overdetermined ICA with the fastICA algorithm (also not my forte, but I think I understand it better). I was hoping this would work since independent components (ICs) are non orthogonal. The solution is shown in fig.E. While the ICs are indeed nonorthogonal, they are NOT my original sources :-(

Any other potential tips or leads or solutions would be greatly appreciated.

More mathematical Detail

The 3 source signals are themselves linear combos of the 5 laguerre basis functions. Thus, the problem can be formulated as such:

1. I take l=5 laguerre basis functions of length m=30 to get B, an [l,m] matrix
2. I randomly mix then is my x=3 sources, S: S=B*M1, where M1 is a random [l,x]=[5,3] mixing matrix. S is displayed in fig.A
3. I mix S n=100 times to get my mixed signals, K (fig.B), where K=S*M2, a [x,n]=[3,100] randomly generated mixing matrix.
4. I have the original basis, L, & the mixed matrix, K. K=S*M2=B*M1*M2. My goal is to recover S.

NOTE: I understand that this may seem impossible since there are many possible S's to choose from (ie the factorizatio S*M2=B*M1*M2 is not unique), however, the true S, will dramatically decrease model complexity by having only x=3 rather than l=5 basis functions which can exactly reproduce all the kernels.

Thanks!

Answer 1

Let's say x,y and z are your basis vectors and you mix them to get what you are calling your mixed sources. If your basis vectors were (x-y), y, and (z+y) then you could mix them in a different way and end up with the same mixed sources.

Basically, there are many ways to come up with a set of basis vectors for your signals. You need to say what kind of basis you want. It is not enough to say that the basis vectors are non-orthogonal.

Answer 2

PCA requires the data to be orthogonal, so make it that way. I will assume a 2 signal source(i.e. 2D) for ease of explanation and visualization.

So add axes to make the data orthogonal in higher dimensions.

There are many ways to accomplish this task with iteration and conditions:

The first component of the PCA, black solid line, will be somewhere between the primary components. It tends to lean (not guaranteed, so you may need to check and randomly drop out samples to force the lean towards one line, it does not matter which one.)

Create two(or three for 3D spaces) new axes for each previous X or Y(X1) (or Z(X2)) coordinate with a Y value as the distance from the PCA calculated primary axis line.

This will collapse the primary PCA axis in each new coordinate system and allow the secondary PCAs to emerge for each new axis. Revealing an approximation original equations.

The cross product of the new PCA axes will allow refinement of the original sources, starting the process over until the required solution condition is met based on convergence of the error. If the error diverges, choose a new initial step:

The challenge occurs in the first step, promax may be used as an alternative to PCA the first step or subsequent steps or a mix.

Calculating a rotational center, through convergence of every axis line, using source translated polar coordinates as additional source dimensions would improve accuracy in the final calculations.

Due to the iterative nature based on assumptions, this answer should converge to one of many solutions, based on the initial parameters.

Any improvements are welcome.

Answer 3

A map from a weights vector $w$ to a ,,mixed signal'' $S$ would be just your 3 prototype signals as a matrix

$A=\left[\matrix{x & y & z}\right]$

so that a linearly combined signal could be written as

$S=Aw$

A ,,pseudoinverse'' to this matrix $A$ (in a few words: a matrix $A'$ so that $A'A=\mathbf{1}$) would map that signal back to $w$:

$A'S=w$

The problem is then finding such a pseudo-inverse. However, this problem is well-known and implementations are available.