Suppose my dataset consists of $N$ example vectors $\mathbf{x}_{1}, \ldots, \mathbf{x}_{N}$ where $\mathbf{x}_{n} \in \mathbb{R}^{p}$ $\forall n$. I assume that each vector $\mathbf{x}_{n}$ is comprised of an underlying true datapoint $\mathbf{s}_{n}$ that is corrupted by additive Gaussian noise $\mathbf{x}_{n} = \mathbf{s}_{n} + \mathbf{w}_{n}$.

What I want to do is estimate the $\mathbf{s}_{n}$. Can I do this with ICA?

So the ICA model is of the form:

$$ \mathbf{x}_{n} = a_{1,n} \mathbf{s}_{1} + a_{2,n} \mathbf{s}_{2} + \ldots + a_{N,n} \mathbf{s}_{N} $$

Does this mean I need to fit my problem to the model by assuming that $ \mathbf{s}_{n} = a_{1,n} \mathbf{s}_{1} $ and $\mathbf{w}_{n} = a_{2,n} \mathbf{s}_{2}$ ? If so, how do I recover the vectors $\mathbf{s}_{n}$ $\forall n$?

I ask because I see the ICA problem formulated as:

$$ \mathbf{x} = \mathbf{A} \mathbf{s} $$

Would this then imply that: $\mathbf{s} = \mathbf{A}^{+} \mathbf{x}$, where $\mathbf{A}^{+}$ is the psuedo-inverse of $\mathbf{A}$? But how would I get the $\mathbf{s}_{n}$ out of this formulation?

Note: I cross-posted this on stats.stackexchange to get the statisticians' point of view, but I'd like to hear what the signal people have to say.


So I do not know $\mathbf{A}$ but I do know that each $\mathbf{s}_{n} \sim f_{s}(\cdot | \theta)$ for some parameter vector $\theta$. I have specified $f_{s}(\cdot | \cdot)$ but $\theta$ is unknown. I also know that the noise vectors are not sampled from that same distribution, but that they are independent of the signal vectors.


This appears similar to a classic least squared solution of an overdetermined equation that proceeds as follows:

Starting with:

$$ \mathbf{x} = \mathbf{A} \mathbf{s} $$

$\mathbf{A}$ is not a square matrix if overdetermined (more equations than unknownns) so therefore an inverse does not exist. What you do then is multiply both sides by the transpose of $\mathbf{A}$ since $\mathbf{A^TA}$ is a square matrix. From that, assuming and inverse exists for$\mathbf{A^TA}$, you can solve for s:

$$ \mathbf{A^Tx} = \mathbf{A^TA} \mathbf{s} $$

$$ \mathbf{s} = (\mathbf{A^TA})^{-1}\mathbf{A^Tx} $$


Your model $\mathbf{x}_n = \mathbf{s}_n + \mathbf{w}_n$ seems too simplistic. It basically says that your output is just some input corrupted by noise. (Unless $\mathbf{s}_n $ is not really your input but some transformed version of it.)

Usually, it's more complicated than that in physical systems. This is why a better model would be $\mathbf{x}= \mathbf{A s} + \mathbf{w}$, i.e., your input $\mathbf{s}$ first undergoes some (noiseless) transformation $\mathbf{A}$, then gets corrupted by noise $\mathbf{w}$. In my opinion, you should start by finding the adequate model to describe your data, then try finding a way to get the unknown quantity of interest to you (which should be represented as an unknown in your model).

Now, if you decide to use the model $\mathbf{x}= \mathbf{A s} + \mathbf{w}$, then the answer of Dan Boschen is what you need for finding $\mathbf{s}$, assuming you know $\mathbf{A}$; technically this is what we call an identification problem; you know the output of a system $\mathbf{x}$ along with the transformation $\mathbf{A}$ that generated it and you want to identify (find) the input $\mathbf{s}$.

If you don't know $\mathbf{A}$, then your problem becomes a blind identification problem where you only know the output $\mathbf{x}$ and what you seek to find this time is both $\mathbf{A}$ and $\mathbf{s}$. This case, is what you can use ICA for.

  • $\begingroup$ Indeed, I should have specified, I don't know $\mathbf{A}$. But I know that $\mathbf{s}_{n} \sim f_{s}(\cdot | \theta)$ for some parameter vector $\theta$ that is unknown. And I know the noise vectors are not sampled from that same distribution, but they are independent of the signal vectors. $\endgroup$ – The Dude Apr 10 '20 at 11:51
  • $\begingroup$ I believe you still need to do some work to figure out if your problem can be solved using ICA or not. You can take a look at the mathematical definition of the problem ICA solves, and if your problem follow this definition and checks the right (identifiability) conditions, then you can apply it. $\endgroup$ – Learn_and_Share Apr 10 '20 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.