I have the following question about the maximum likelihood (ML) in presence of inter-symbol interference and colored noise.
Assume the communication system is as follows. Information source, modulator, transmit pulse filtering, channel, AWGN, matched receiver pulse filtering, sampler, and then ML estimator.
The received signal after the sampler is given as
$$y = h\star h\star c\:a + n\star h$$
where $y$ is the received samples (contains transmit symbols + ISI + noise), $a$ is the transmit symbols, $h$ transmit/receive pulse shaping filter, $c$ is the channel impulse response, $n$ is the AWGN, and $\star$ denotes convolution. This can be further written as
$$y = A\:a + \eta$$
where $A$ is a convolution matrix whose elements contains $h\star h\star c$ and $\eta$ is a colored noise.
- I wonder what will be the ML in this case?
- And how to derive/calculate it?