MMSE Estimation - Fusion of 2 Measurements

Question

Let's say I have 2 measurements of the same phenomenon (for example current temperature) and I want to find the MMSE (minimum mean square error) estimator, i.e to minimize the MSE (mean square error). The measurements are independent and the noise in each one is Gaussian. The mathematical formulation is:

$$Y_1[n]=X[n]+V[n]$$

$$Y_2[n]=X[n]+W[n]$$ where:

• $X[n]$ is the real temperature at time $n$.
• $Y_1[n]$ and $Y_2[n]$ are the measurements at time $n$.
• $V[n]$ and $W[n]$ are zero-mean normally distributed random variables which represent the noise, and have variances $\sigma_v^2$ and $\sigma_w^2$, respectively.

I need a clue or reference for how to solve it.

Answer 1

*STOP! If you only want a hint and not the complete solution please see Stanley P.'s or Peter K.'s answers. *

Since you do not specify if there is model for the temperature evolving over time $n$, I will derive an estimator which is a combination of $Y_1$ and $Y_2$ for each fixed $n$. Let $\alpha \in (0,1)$ and suppose the estimate of $X$ can be written as $\hat X = \alpha Y_1 + (1-\alpha)Y_2$. The MSE is given by \begin{eqnarray} E[(\hat X-X)^2] &=& E[(\alpha Y_1 + (1-\alpha)Y_2 - X)^2]\\ &=& E[(\alpha X + (1-\alpha)X + \alpha V + (1-\alpha)W - X)^2]\\ &=& E[(\alpha V + (1-\alpha)W)^2]\\ &=& \alpha^2 \sigma_v^2 + (1-\alpha)^2\sigma_w^2. \end{eqnarray} To minimize this MSE, take the derivative with respect to $\alpha$ and set it to zero. After some algebra you get $\alpha = \frac{\sigma_w^2}{\sigma_v^2+\sigma_w^2}$ and so $$ \hat X = \frac{\sigma_w^2}{\sigma_v^2+\sigma_w^2} Y_1 + \frac{\sigma_v^2}{\sigma_v^2+\sigma_w^2 }Y_2. $$

Note that if the variances are unequal (say $\sigma_w^2 > \sigma_v^2$), then the noisier observation ($Y_2$) is given less weight than the less noisy observation ($Y_1)$. If the noise variances are equal, the two observations are weighted equally and we get just the arithmetic mean of the two observations.

If you do have a model for temperature evolution over time you should use a Kalman filter which is guaranteed to produce the MMSE estimate.

Answer 2

HINT:

Since in your model $X[n]$ is independent of $X[n-1]$ or $X[n+1]$ or any other shifts greater than zero, the linear estimate will have form:

$$ \hat{X}[n]= \alpha Y_1[n] + \beta Y_2[n] $$

One might consider how you would weight $\alpha$ and $\beta$ given $\sigma_v$ and $\sigma_w$.

One might also make a connection for Gaussian models, linear estimation, and MMSE.

Answer 3

I'm going to assume that we have no information about how $X[n]$ varies with time, so we can just do one-at-time estimation of $X[n]$ using $Y_1[n]$ and $Y_2[n]$.

One way to get an estimate from $Y_1$ and $Y_2$ is to average the measurements: $$ \hat{X}[n] = \frac{Y_1[n] + Y_2[n]}{2} $$

Using the fact that the sum of two independent Gaussian variables, $A$ and $B$ is distributed as: $$ N(\mu_A + \mu_B, \sigma^2_A + \sigma^2_B) $$ we can see that $\hat{X}[n]$ is distributed $$ N\left (X[n], \frac{\sigma^2_V + \sigma^2_W}{4} \right) $$ so it's unbiased and has a smaller variance if $V$ and $W$ are identically distributed.

From there, you want to think about an estimator like Atul's: if $V$ and $W$ are not identically distributed, what proportion of $Y_1$ and $Y_2$ makes sense rather than just half and half?

Answer 4

Remarks

• The MMSE is Bayesian Framework.
  Namely it should be employed between random variables which have joint distribution. In your case it seems $ x \left[ n \right] $ is a deterministic parameter hence Parameter Estimation framework should be employed.

• Parameter Estimation
  In the case above it seems the Maximum Likelihood Estimation fits. Since the data is Gaussian it will also be the Minimum Variance Unbiased Estimator (MVUE).

• Variance Minimization
  It seems that you are after minimizing the Variance of an unbiased estimator. Which the Maximum Likelihood will yield given the properties for your data (Gaussian).

Maximum Likelihood Estimator

You data can be modeled and generalized by a random vector $ y \in \mathbb{R}^{n} $ where $ n $ is the number of measurements. The distribution of $ y $ is given by:

$$ y \sim \mathcal{N} \left( \mu \boldsymbol{1}, C \right) $$

Where $ \mu $ is the expected value of the data, namely what you're after, $ x \left[ n \right] $ and $ C $ is the Covariance matrix:

$$ C = \begin{bmatrix} {\sigma}_{1}^{2} & & & \\ & {\sigma}_{2}^{2} & & \\ & & \ddots & \\ & & & {\sigma}_{n}^{2} \end{bmatrix} $$

Namely a diagonal matrix where the $ i $ -th element on the diagonal is the variance of the noise of the $ i $ -th sample.

Now we're basically after the Maximum Likelihood Estimator of the Mean Value of a Gaussian samples with the given covariance matrix.

$$\begin{align*} \hat{\mu} & = \arg \max_{\mu} P \left( y \mid \mu \right) & \text{} \\ & = \arg \max_{\mu} \det \left( 2 \pi C \right)^{-\frac{1}{2}} {e}^{-0.5 \left( y - \mu \boldsymbol{1} \right)^{T} {C}^{-1} \left( y - \mu \boldsymbol{1} \right) } & \text{Multivariate Gaussian Distribution} \\ & = \arg \min_{\mu} - \log \left( P \left( y \mid \mu \right) \right) & \text{The Log Likelihood Function (Monotonic)} \\ & = \arg \min_{\mu} \frac{1}{2} \left( y - \mu \boldsymbol{1} \right)^{T} {C}^{-1} \left( y - \mu \boldsymbol{1} \right) & \text{} \\ & = \frac{ \boldsymbol{1}^{T} {C}^{-1} }{ \boldsymbol{1}^{T} {C}^{-1} \boldsymbol{1} } y = \frac{ \sum_{i = 1}^{n} {\sigma}_{i}^{-2} {y}_{i} }{\sum_{i = 1}^{n} {\sigma}_{i}^{-2}} \end{align*}$$

If you plug in your parameters of the question you will get same result as others here.

The result also makes sense:

• The estimator is a weighted mean (Linear Combination) of the measurements.
• The weight is according to the SNR of the measurement.

Enjoy...