# ICA for Noise Reduction over a Dataset

Suppose my dataset consists of $$N$$ example vectors $$\mathbf{x}_{1}, \ldots, \mathbf{x}_{N}$$ where $$\mathbf{x}_{n} \in \mathbb{R}^{p}$$ $$\forall n$$. I assume that each vector $$\mathbf{x}_{n}$$ is comprised of an underlying true datapoint $$\mathbf{s}_{n}$$ that is corrupted by additive Gaussian noise $$\mathbf{x}_{n} = \mathbf{s}_{n} + \mathbf{w}_{n}$$.

What I want to do is estimate the $$\mathbf{s}_{n}$$. Can I do this with ICA?

So the ICA model is of the form:

$$\mathbf{x}_{n} = a_{1,n} \mathbf{s}_{1} + a_{2,n} \mathbf{s}_{2} + \ldots + a_{N,n} \mathbf{s}_{N}$$

Does this mean I need to fit my problem to the model by assuming that $$\mathbf{s}_{n} = a_{1,n} \mathbf{s}_{1}$$ and $$\mathbf{w}_{n} = a_{2,n} \mathbf{s}_{2}$$ ? If so, how do I recover the vectors $$\mathbf{s}_{n}$$ $$\forall n$$?

I ask because I see the ICA problem formulated as:

$$\mathbf{x} = \mathbf{A} \mathbf{s}$$

Would this then imply that: $$\mathbf{s} = \mathbf{A}^{+} \mathbf{x}$$, where $$\mathbf{A}^{+}$$ is the psuedo-inverse of $$\mathbf{A}$$? But how would I get the $$\mathbf{s}_{n}$$ out of this formulation?

Note: I cross-posted this on stats.stackexchange to get the statisticians' point of view, but I'd like to hear what the signal people have to say.

Edits:

So I do not know $$\mathbf{A}$$ but I do know that each $$\mathbf{s}_{n} \sim f_{s}(\cdot | \theta)$$ for some parameter vector $$\theta$$. I have specified $$f_{s}(\cdot | \cdot)$$ but $$\theta$$ is unknown. I also know that the noise vectors are not sampled from that same distribution, but that they are independent of the signal vectors.

This appears similar to a classic least squared solution of an overdetermined equation that proceeds as follows:

Starting with:

$$\mathbf{x} = \mathbf{A} \mathbf{s}$$

$$\mathbf{A}$$ is not a square matrix if overdetermined (more equations than unknownns) so therefore an inverse does not exist. What you do then is multiply both sides by the transpose of $$\mathbf{A}$$ since $$\mathbf{A^TA}$$ is a square matrix. From that, assuming and inverse exists for$$\mathbf{A^TA}$$, you can solve for s:

$$\mathbf{A^Tx} = \mathbf{A^TA} \mathbf{s}$$

$$\mathbf{s} = (\mathbf{A^TA})^{-1}\mathbf{A^Tx}$$

Your model $$\mathbf{x}_n = \mathbf{s}_n + \mathbf{w}_n$$ seems too simplistic. It basically says that your output is just some input corrupted by noise. (Unless $$\mathbf{s}_n$$ is not really your input but some transformed version of it.)

Usually, it's more complicated than that in physical systems. This is why a better model would be $$\mathbf{x}= \mathbf{A s} + \mathbf{w}$$, i.e., your input $$\mathbf{s}$$ first undergoes some (noiseless) transformation $$\mathbf{A}$$, then gets corrupted by noise $$\mathbf{w}$$. In my opinion, you should start by finding the adequate model to describe your data, then try finding a way to get the unknown quantity of interest to you (which should be represented as an unknown in your model).

Now, if you decide to use the model $$\mathbf{x}= \mathbf{A s} + \mathbf{w}$$, then the answer of Dan Boschen is what you need for finding $$\mathbf{s}$$, assuming you know $$\mathbf{A}$$; technically this is what we call an identification problem; you know the output of a system $$\mathbf{x}$$ along with the transformation $$\mathbf{A}$$ that generated it and you want to identify (find) the input $$\mathbf{s}$$.

If you don't know $$\mathbf{A}$$, then your problem becomes a blind identification problem where you only know the output $$\mathbf{x}$$ and what you seek to find this time is both $$\mathbf{A}$$ and $$\mathbf{s}$$. This case, is what you can use ICA for.

• Indeed, I should have specified, I don't know $\mathbf{A}$. But I know that $\mathbf{s}_{n} \sim f_{s}(\cdot | \theta)$ for some parameter vector $\theta$ that is unknown. And I know the noise vectors are not sampled from that same distribution, but they are independent of the signal vectors. – The Dude Apr 10 '20 at 11:51
• I believe you still need to do some work to figure out if your problem can be solved using ICA or not. You can take a look at the mathematical definition of the problem ICA solves, and if your problem follow this definition and checks the right (identifiability) conditions, then you can apply it. – Learn_and_Share Apr 10 '20 at 13:34