# Maximum Likelihood for Colored Noise

I have the following question about the maximum likelihood (ML) in presence of inter-symbol interference and colored noise.

Assume the communication system is as follows. Information source, modulator, transmit pulse filtering, channel, AWGN, matched receiver pulse filtering, sampler, and then ML estimator.

The received signal after the sampler is given as

$$y = h\star h\star c\:a + n\star h$$

where $y$ is the received samples (contains transmit symbols + ISI + noise), $a$ is the transmit symbols, $h$ transmit/receive pulse shaping filter, $c$ is the channel impulse response, $n$ is the AWGN, and $\star$ denotes convolution. This can be further written as

$$y = A\:a + \eta$$

where $A$ is a convolution matrix whose elements contains $h\star h\star c$ and $\eta$ is a colored noise.

• I wonder what will be the ML in this case?
• And how to derive/calculate it?

Thanks.

• In your question, are $h$ and $c$ known? – Royi Jul 25 '18 at 3:58
• I think your question is answered. Could you address the solution you were given? – Royi Jul 26 '18 at 15:48

Let's have a look on the following model:

$$y \left[ n \right] = \left( h \ast x \right) \left[ n \right] + \left( g \ast w \right) \left[ n \right]$$

Where $x \left[ n \right]$ is the signal of interest and $w \left[ n \right]$ is the AWGN with unit Variance.

In Matrix form it is written by:

$$\boldsymbol{y} = H \boldsymbol{x} + G \boldsymbol{w}$$

Where $H$ and $G$ are the convolution matrices of the model.

Let's define $v = G \boldsymbol{w}$ then $v \sim \mathcal{N} \left( \boldsymbol{0}, G {G}^{T} \right)$ which implies (If the input to Linear Operator is Gaussian Variable then the output is also Gaussian Variable):

$$\boldsymbol{y} \sim \mathcal{N} \left( H \boldsymbol{x}, G {G}^{T} \right)$$

Then the Maximum Likelihood is given by:

$$\arg \max_{ \boldsymbol{x} } \det \left( 2 \pi G {G}^{T} \right)^{-\frac{1}{2}} {e}^{ -\frac{1}{2} {\left( H \boldsymbol{x} - \boldsymbol{y} \right)}^{T} {\left( G {G}^{T} \right)}^{-1} \left( H \boldsymbol{x} - \boldsymbol{y} \right) }$$

It is easy to see you only need to deal with the term in the power of the Exponent, so it is equivalent of:

$$\arg \min_{ \boldsymbol{x} } {\left( H \boldsymbol{x} - \boldsymbol{y} \right)}^{T} {\left( G {G}^{T} \right)}^{-1} \left( H \boldsymbol{x} - \boldsymbol{y} \right)$$

Which is nothing more than Weighted Least Squares problem which matches the result of Linear Regression for Colored Noise (As expected for Gaussian Noise).

The optimal solution is given by:

$$\hat{x} = {\left( {A}^{T} {\left( G {G}^{T} \right)}^{-1} A \right)}^{-1} {A}^{T} H b$$

• Hi: I had only read upto the argmax determinant and I guess missed the argmin part below it. My mistake and my apologies. I would think that your expression must reduce to Aitken since it's also closed form. Note that, if the variance is not known ( in this case it's one but in real life, probably not ), your likelihood expression becomes 2 dimensional in which case FGLS is probably preferred. – mark leeds Jul 26 '18 at 16:58
• @markleeds, You seem to misunderstand what you call Aitken method. It is the exact as the above. Nothin more than the case of Weighted Least Squares. Read closely the model you linked to. – Royi Jul 26 '18 at 20:28
• I really don't understand your remarks. In case above the noise doesn't have Variance in the sense of a scalar (Not even necessarily Scalar Matrix). It has a Covariance Matrix which is assumed to be known as in the paper you attached. If the Covariance Matrix isn't known the ML become a complete different story. – Royi Jul 27 '18 at 5:43
• Hi Royi: I was referring to the case where the matrix is diagonal but the variances on the diagonal are all different. You are correct that the paper I linked to does not explain FGLS. There are so many ways that the assumption of known variance can be false. So, I've included three links. Obviously, you don't need to read them but my point was the assumption of know variance is unrealistic and, when one doesn't make that assumption, Aitken and your approach do not apply. Three links are in next comment. The last link is long winded and only section 5 of it is really relevant. – mark leeds Jul 27 '18 at 11:53
• – mark leeds Jul 27 '18 at 11:54

First, I think that the expression $A = h\star h\star c$ is not correct. Actually, $A$ is a convolution matrix whose elements contains $h\star h\star c$. The dimensions of $A$ will depend of the dimensions of $h$ and $c$. If $h$ has $N_{h}$ coefficients and $c$ has $N_{c}$ coefficients, and let $m = h\star h\star c$, then $m$ will have $N_{a} = 2N_{h} + N_{c} -2$ coefficients.

For example, if $h = [h_{1} \ h_{2} \ h_{3}]^{T}$ has 3 coefficients and $c = [c_{1} \ c_{2} \ c_{3} \ c_{4}]^{T}$ has four coefficients, then $m$ will have 8 coefficients.

The matrix $A$ will have $N_{m}$ rows and $N$ columns, where $N$ is the length of your $a$, so as to make the product $Aa$ equivalent to the convolution operation $m\star a$.

Pay attention the colored noise is the result of AWGN through a linear system, hence it is still an Additive Gaussian Noise So, the ML problem can written as:

$$\hat{a} = \arg \min_{a} {\left\| A a - y \right\|}_{2}^{2}$$

The solution to this ML problem can be found at any book about estimation theory and other materials like this. The analytical solution will depend on the statistics involved. The implementation of the ML can be made using the Viterbi algorithm, for example.

• Thank you for your insight. You are correct. $A$ is a convolution matrix whose elements contains $h*h*c$, I will update the question accordingly. Are you sure that $\hat{a} = arg \ min_{a}|Aa - y|^{2}.$ is the correct expression of the ML (given that $\eta$ is colored noise$? In the meantime, I will look into books of estimation theory. – Noor Apr 17 '16 at 1:41 • Hi Moor and JohnMarvin: Just a heads up that minimizing the squared deviation only results in an ML estimator in the case of the gaussian likelihood assumption. I'm not sure what the likelihood is in this case because the terminology is different from what I'm used tp since background is statistics-econometrics. interesting to read nonetheless. – mark leeds Jan 25 '18 at 14:23 • That is right mark leeds. He has that noise$ n $that is AWGN. However, it is filtered by the receive filter, generating the coloured noise$ \eta \$. It is important to notice that although the decorrelation between the samples is not guaranteed, the noise will still have a Gaussian distribution. So, as you said, the ML is still achieved when you minimize the squared deviation. :) – JohnMarvin Jan 28 '18 at 15:39
• @JohnMarvin. Thanks. It's really appreciated but even things like AWGN are foreign to me. If you don't mind, do you know of a reasonable level EE book that covers this type of material. I know that DSP books don't because I have a lot of those. My background is stat-econometrics but I' trying to become more knowledgable about EE topics. In some ways, the two fields are not so different. In other ways, EXTREMELY different. Thanks again. – mark leeds Feb 24 '18 at 17:53
• @JohnMarvin, I'm not so sure about your solution. When you write the ML function you'll have to deal with the Covariance Matrix of the colored noise. – Royi Jul 25 '18 at 4:45