What is a correct approach to baud-rate digital timing recovery for self-equalized line code

Question

In particular, I have a signal coming from a twisted pair of arbitrary length with a differential encoded biphase line code. What should be a correct approach to timing recovery in this case?

Currently I am trying to implement a baud-rate timing recovery algorithm using a Mueller&Muller TED (timing error detector), loop filter (average over 120 symbols) and counter-divider with adjustable initial value as a digital VCO. The source code for current implementation of the algorithm in python is here. For TED I use the following timing function:

$$ \frac{(a_k^2 - 5)}{16}(a_k x_{k-1} - a_{k-1} x_k ) $$

For simulation purposes I have sampled versions of the signal in half-duplex mode (i.e. without echo). Signal samples look like this for two different channel lengths:

For these signals the timing recovery algorithm converges to the phase near signal's zero crossings. This does not allow to correctly determine the original bit sequence (e.g. "+1 -1 +1 +1 -1 +1 ...") and I would like it to converge to the maximum eye opening, i.e. somewhere near the peaks.

I've managed to get this by adding a $(1 - aD)$ filter, where $a$ is $0,75$ for the first signal sample and $1,325$ for the second one. But because a channel has an arbitrary length, the shape of a signal is unknown during design phase and this filter with fixed coefficients seems like a bad solution. Moreover, adding this filter was purely an empirical solution and is not backed up by any math.

I am very new to timing recovery and would appreciate any advice.

Answer 1

A good option for timing function for such signal happens to be the canonical $(a_k x_{k-1} - a_{k-1} x_k )$ with additional condition to calculate its value only when $a_k$ equals $a_{k-1}$ (i.e. when "1" is preceded by "1" or "-1" is preceded by "-1"). As a result, no additional digital filter is needed.

The reason for this condition came from observations of S-curve for this timing function (see M. Rice «Chapter 8 Symbol Timing Synchronization» in "Digital communications: a discrete-time approach"). Having timing function (timing error function) $e(k)$ written as

$$e(k)=a(k) x(kT+\tau) - a(k-1) x((k-1)T+\tau),$$

where $T$ - is a symbol period, and $\tau$ - is a phase error,

the S-curve is an expected value of $e(k,\tau)$:

$$g(\tau)=E\{e(k,\tau)\}.$$

For the signal samples the proper S-curve with a single zero crossing was obtained only for "11", "00", "1100" and "0011" symbol patterns. The shorter symbol pattern (i.e. $a_k=a_{k-1}$) was selected to simplify further implementation.

Here is an example plot of a signal sample and values of $e(k)$ as a function of a phase (which vaguely represents $g(\tau)$), for different symbol patterns.