# Determining pulse shape for Viterbi Equalization

I believe I understand well enough how the Viterbi algorithm can reverse the effects of ISI when the combined impulse response (transmit filter, multipath, receive filter) is known. I've read How I Learned to Love the Trellis by Bernard Sklar, and I also asked about it in a previous question: Applying Viterbi algorithm to compensate for ISI in PSK31.

But how can this technique be applied when the channel distortion isn't known? From my superficial understanding of equalizing algorithms like CMA, they attempt to infer the channel distortion and compensate for it dynamically. My particular interest is HF channels, which would certainly seem to benefit from some dynamic adjustment.

However, in my study of Viterbi equalization so far, I've not found an explanation of how the unknown impulse response of the channel is found. Without that information, the best we can accomplish is to compensate for known elements like the receive and transmit filters. And in a case with no inherent ISI (like RRC filters on each end), then does Viterbi equalization have any value at all?

• you don't need to determine the pulse shape because it is nearly impossible to determine it exactly. Instead you need some statistical model of the noise. See soft input Viterbi web.mit.edu/6.02/www/f2010/handouts/lectures/L9.pdf – AlexTP Jun 13 '17 at 7:59
• I could not be seeing the larger picture, but how is that article about equalization? My concern isn't noise, it's a non-ideal channel response such as multipath. – Phil Frost Jun 13 '17 at 11:40
• Okay, I will state the problematic in another way. "how can this technique be applied when the channel distortion isn't known". If channel distortion is not known deterministicly, you need a probabilistic model. Viterbi equalizer is to find the "most likely" path, and "most likely" means probabilistic. Sorry if my answer still confuses you but it is the best I think I can explain. – AlexTP Jun 13 '17 at 12:28
• If there's no ISI due to transmit or receive filtering we are trying to remove, and we give up on trying to incorporate multipath into the trellis, and there's no convolutional coding, then what state is there? If the symbols are equiprobable, and the noise is white, what information is there available to build a probabilistic model that's better than decoding symbols based on the sign of the observation? Or in other words, if we observe 0.00001 that may be a 1 with a low probability, what could possibly happen that would make that more likely to be a 0? – Phil Frost Jun 13 '17 at 13:13
• Question 1, I dont understand what you are trying to arrive. Question 2, the PDF of noise which is unknown (white is not distribution). When you are talking about the sign of observation as the "best" strategy, and if we think about "best" in term of maximum likelihood, you are implying that PDF is symmetric with zero mean. Question 3, again symmetric PDF as example, the mean of PDF will ditactate the result to be 0 or 1 for the same observation $x$, i.e. $\mathrm{Pr}(y=0|x=0.00001) > \mathrm{Pr}(y=1|x=0.00001)$. – AlexTP Jun 13 '17 at 13:53

## 1 Answer

I'll answer my own question with what I've learned in the past 28 days. Hopefully I've got it right!

Viterbi equalization is just one way to equalize the channel, as an alternative to some other method such as convolving the signal with an equalizing filter. Estimating the channel is an entirely separate problem which must be solved independently of the equalization method.

The trouble with an equalizing filter is it compensates for dips in the channel frequency response by amplifying those frequencies. Unavoidably, the noise is also amplified.

There's a tradeoff to be made: the flatter the frequency response is made, the lower the error due to ISI, but the higher the error due to noise. There's an optimal amount of equalization which minimizes the error due to noise and ISI. An equalizer that tries to hit this sweet spot is a minimum mean square error (MMSE) equalizer.

Viterbi equalization offers an alternative solution. The channel estimate will produce an estimated channel impulse response. The coefficients in this impulse response can be used to build the trellis, and the Viterbi algorithm can then find the most likely sequence of transmitted bits, taking into account the estimated distortion from the channel. The receive filter remains a simple matched filter, with no equalization. The Viterbi equalizer is not linear (unlike the equalizing filter alternative), so it's able to compensate for deep spectral nulls in the channel response in a way the equalizing filter can't. The cost is increased complexity.