# Extracting common signal without knowledge about noise [closed]

Given two noisy time series thought to contain a common signal, $$x_1(t) = s(t) + n_1(t), \quad x_2(t) = s(t) + n_2(t),$$ what is the best way to determine $s(t)$ without assuming any distribution for the noise terms $n_i(t)$?

In particular, the $n_i(t)$ are neither Gaussian nor stationary. The only assumption I am willing to make is that $n_1(t)$ and $n_2(t)$ have zero mean and are uncorrelated with each other, i.e. $E(n_1(t) n_2(t)) = 0$ (though even this assumption can be broken in my problem, but only slightly).

Is it true that the best estimate is simply $$\frac{x_1(t) + x_2(t)}{2}?$$

I am particularly interested in finding not only an estimate for $s(t)$, but also a measure of the uncertainty of this estimate.

• can you assume anything about $s$? – Marcus Müller Apr 26 '18 at 11:52
• Also, "best estimate" is an ambiguous term – can you mathematically define what measures "best"? Is it a maximum deviation between your estimate and $s$ over all $t$, is it the integral of the absolute difference over $t$, or over the square? Is it maybe something else? – Marcus Müller Apr 26 '18 at 11:59
• what is uncertainty? You'll need to give us some definition of "goodness" to start with, as otherwise we'd just have to write a book on estimation theory. Believe me, your question would be broad and hard enough to answer if you specified exactly what your measure for goodness (and uncertainty) was – you should probably first ask specifically, and then you can generalize. – Marcus Müller Apr 26 '18 at 14:52
• If you can say almost nothing about the properties of the noise signals and very little about the desired signal $s(t)$, then, by construction, you really can't tell them apart. Stating that $s(t)$ is smooth only lets you know that it might be bandlimited, or at least have only a little energy at high frequencies, but that's not enough. By your problem statement, the noise could have the exact same properties as the signal of interest. I.e. The signal of interest could look very much like the noise signals, so $x_1(t) = n_3(t) + n_1(t)$, $x_2(t) = n_3(t)+n_2(t)$ – Andy Walls Apr 26 '18 at 16:28
• Since I am allowed to assume almost anything about the noise, let's assume the power in $n_2(t)$ is at least 3 dB greater than the power in $s(t)$. Then your estimator $\dfrac{x_1(t) + x_2(t)}{2}$, is inferior to the estimator $x_1(t)$, so it is not true that your estimator is the best estimator. Of course there are many scenarios where $x_1(t)$ would be inferior. You really need some more bounds for a meaningful and useful answer. – Andy Walls Apr 26 '18 at 18:17

Taking the average is a reasonable heuristic. The term “best” needs a lot more clarification because it needs a criteria. Assuming that the signal is exactly the same on both channels is actually a strong assumption, so if you consider small perturbations to your model, say a time delay between the two channels, which is a lot more reasonable than perfect alignment, the character of the problem has changed.

There are techniques like ICA but they work well when they do and don’t when they don’t.

If your question is of a theoretical nature, there are answers to questions like is there a common signal in both channels using non parametric statistics.

If this is a practical question, then you should look at your data very closely and see whatever else you can reasonably assume.

• Hi: If you make enough assumptions about the error terms, then it's straightforward but my guess is that you don't want to make those assumptions which is why you're asking. second part of this link is derivation for the "nice" case. professorchappell.com/Econ727/handouts/signal.pdf. it's the time domain version of the weiner filter. – mark leeds Apr 26 '18 at 18:16

If by "best" estimator you mean the one with the lowest variance, you have the following (assuming that $s$ is deterministic):

$\newcommand{\Var}{\mathrm{Var}}$ $\newcommand{\E}{\mathrm{E}}$ For $i = 1,2$, $\E[x_i] = s$ (since $\E[n_i] = 0$) and $\Var(x_i) = \E[(x_i-s)^2] = \E[n_i^2]$.

Let's define $\hat{x} = \frac{x_1+x_2}{2}$: $$\E[\hat{x}] = \E[s + \frac{n_1+n_2}{2}] = s,$$ i.e. $\hat{x}$ is an unbiased estimator (this is also true for $x_1$ and $x_2$).

You can compute the variance as: $$\begin{split} \Var[\hat{x}] = &\E[(\hat{x}-s)^2]\\ = &\E[(\frac{x_1+x_2}{2}-s)^2]\\ = &\frac{1}{4} \E[((x_1-s)+(x_2-s))^2]\\ = & \frac{1}{4} \big[ \E[(n_1+n_2)^2]\\ = & \frac{1}{4} \big[ \E[n_1^2]+E[n_2^2] \big] \end{split}$$ since $\E[n_1n_2] = 0$.

In summary we have: $$\E[\hat{x}] = \E[\hat{x_1}] = \E[\hat{x_2}] = s$$ and: $$\Var(\hat{x}) = \frac{1}{4} (\Var(x_1)+\Var(x_2))$$ There is not much you can say if you don't have any further assumption on $n_1$ and $n_2$ ($n_1$ could have a very large variance, in which case $\Var(\hat{x})$ is also large and $\hat{x}$ is a "worst" estimator than $x_2$).

However if $\Var(x_1) = \Var(x_2)$ then: $$\Var(\hat{x}) = \frac{1}{2} \mathrm{\Var}(x_2),$$ i.e. the estimator $\hat{x}$ is "twice as good" (in terms of variance) as $x_1$ or $x_2$.

EDIT: In the more general case, for a linear estimator $\hat{x} = \alpha x_1 + (1-\alpha)x_2$ (with $0 \leq \alpha \leq 1$), you can show in a similar way that:
$$\Var(\hat{x}) = \frac{1}{4} (\alpha^2 \Var(x_1)+ (1-\alpha)^2\Var(x_2)).$$ Differentiating with respect to $\alpha$ shows that the lowest variance is attained for: $$\alpha = \frac{\Var(x_2)}{\Var(x_1)+\Var(x_2)}.$$ This is called the Best Linear Unbiased Estimator (BLUE), or the least-squares estimator.

However as said in the other answers, if you have further assumptions about the noise or signal (such as sparsity or smoothness), you can also use non-linear estimators.

The problem is underdetermined, as you have three sources (likely, one deterministic, two stochastic) and only two observations. Using an analogy: I have three numbers in mind. I give you their sum and product. Can you tell the three numbers? In general, the answer is no, unless you can additional information. This reminds me of the seven/eleven (9/11) problem: a customer buys four items in a 7/11 store. The cashier person says: great, it's 7.11 $. The customer is stunned "How lucky?" The cashier: "I just did the product of the individual items". The customer: "and the sum?" The cashier: "same". Then you get a unique solution in dollars and cents (numbers of shape x.xx). Smoothness, zero-mean, uncorrelatedness, positivity, sparsity, are priors that can definitely help find a local or a global optimum. You ought to recast your problem into an optimization problem, with a loss function and additional penalties. If the signal is smooth, you can penalize its derivatives, a total variation, etc. If the noise is weird, you can try to gaussianize it with a appropriate transformations: variance stabilizing, or whitening ones. If you only use$\frac{x_1(t)+x_2(t)}{2}$, you miss the smoothness. So a functional of the shape: $$\|T(x-\hat{x})\|^2+ f({x})+ g({n_1,n_2})$$ could be useful,$f$being related to derivatives,$g\$ to uncorrelatedness. From a more practical point-of-view, I would try first:

1. convert the data with a variance stabilizing transform (Anscombe, Box-Cox, etc.)
2. transform the data into several novel (invertible) bases or frames where signal and noise are better separated (it could be identity)
3. perform selection on coefficients, using at most uncorrelatedness: dependent shrinkage, etc.
4. invert the several (invertible) bases or frames and combine them into a more efficient estimator. For instance, you can aggregate estimators using: Aggregation of Affine Estimators

We consider the problem of aggregating a general collection of affine estimators for fixed design regression. Relevant examples include some commonly used statistical estimators such as least squares, ridge and robust least squares estimators. Dalalyan and Salmon (2012) have established that, for this problem, exponentially weighted (EW) model selection aggregation leads to sharp oracle inequalities in expectation, but similar bounds in deviation were not previously known. While results indicate that the same aggregation scheme may not satisfy sharp oracle inequalities with high probability, we prove that a weaker notion of oracle inequality for EW that holds with high probability.