# Requirements on Signal Spectrum for Fractionally Spaced Equalization

I'm constructing an ultrasonic communication system and I am struggling with the correct parameters for my receiver.

In baseband, my channel input signal $$x(t)$$ has infinite bandwidth, since I use a rectangular pulse shape (which I have to because of technical constraints in the transmitter). However, since the spectrum of a rectangular pulse is a $$\text{sinc}$$, the main part of the power of the signal will be in the frequency range lower than a few times the inverse of the symbol rate $$T_s$$.

In the receiver, I have an equalizer. I know that using a fractionally spaced equalizer can make my communication system robust against random sampling delay. A requirement that I found in the literature is that the sampling rate for the input of the equalizer must be twice as high as the highest frequency in the received signal, so basically the Nyquist criterion. My transmit signal has unlimited bandwidth. That means that I either have to accept, that the signal's bandwidth is higher than what my equalizer can equalize, or I have to limit the frequency range of the received signal by low-pass filtering it before sampling.

My question is: Is this a tradeoff in the sense that I lose information by using the lowpass filter, or can I still recover the information from the transmitted signal perfectly (if no noise was present)? If it is, is there a way to quantify the error that I will introduce by removing the higher frequency components of the signal versus the error that is introduced by aliasing when the signal contains frequencies higher than half the sampling rate?

To make it more concrete: Right now I have a symbol rate of roughly 1250 symbols per second and a sampling rate of 10kHz. To limit the frequency response, I use an 8th-order Low-Pass filter with a cutoff frequency at 4kHz before sampling. That low-pass has increased equalizer performance with random sampling delay but has not eliminated it. I still see that on some messages the equalizer performs much worse than on others.

• You can't have an infinite bandwidth pulse. It must be bandlimited, albeit could be very high compared to your receiver sampling frequency. Some lowpass filtering at the input of the receiver is necessary. If your doing digital communications (as of your talking on bitrates etc.) then you can perfectly recover (with adequate signal processing) every transmitted bit, as long your transmission speed (its bitrate) does not cross the channel capacity limit. Note that channel capacity is not a static metric. It will change as the channel bandwidth or channel background noise changes. Nov 16, 2021 at 15:23

As far as equalizer performance and possible limitations I provide two key points below about the span of the equalizer and the number of samples per symbol to use.

The equalizer duration in time is set to match the time duration of the delay spread of the channel (since this is the distortion the equalizer will compensate for). If it is much less than there will be "echoes" (leading or trailing) that will not be properly regrouped into the combined signal, and if it is much longer than you are degrading the signal due to noise enhancement (due to the addition of independent random processes which each coefficient in the equalizer will contain). Fractionally spaced equalizers are desired to compensate for the full bandwidth of the waveform. For pulse-shaped waveforms where the occupied bandwidth is not much more than the symbol rate, then 2 samples per symbol would be more than sufficient. If the waveform has larger spectral occupancy, then we would follow Nyquist (which is twice the bandwidth NOT twice the highest frequency) to have a sufficient number of samples (sampling rate) to properly capture the full bandwidth of the waveform. Baud rate equalizers (1 sample/symbol) will still work but they will dominate their correction in the center bandwidth of the waveform, and not do very well at the edges due to the aliasing that such sampling would cause. I show this in the graphic below: As a final point on equalizer performance; the above describes "linear equalizers" which will not perform well in frequency selective channels. This occurs specifically when the delay spread of the channel exceeds the symbol duration (or more specifically for the OP's case, when the inverse of the delay spread is less than the occupied bandwidth of the waveform) as such to cause deep frequency selective nulls within the waveforms bandwidth (and if the bandwidth is wider such as rectangular pulses, this is also more likely to occur). A linear equalizer will cause noise enhancement on these null locations leading to degraded performance (and a non-linear equalizer such as a decision feedback equalizer may be a better choice). This would only occur if there are strong multipath components of similar magnitude and delayed longer than the inverse of the spectral occupancy (to cause deep variation between constructive and destructive interference).

With that in mind and specific to the OP's case of using a rectangular pulse, ultimately in any real-world system of finite energy, the pulse will be bandlimited. Further, when the channel is a shared resource, the metric of spectral occupancy drives the decision as to how much to bandlimit the channel. Also consider if there was no band-limiting (ultimately here in the receiver), the power spectral density of the waveform is decreasing (at rate $$1/f^2$$) while the noise is at least constant over a much wider frequency range (typically as the noise floor). The receiver will benefit by filtering the waveform regardless as to what was transmitted, and specifically with a matched filter such as to optimize the received SNR (which in this case of a rectangular pulse could be done as an integrate and dump over the symbol period). Such a filter has an equivalent noise bandwidth of $$1/T$$ where $$T$$ is the symbol duration. To address the OP's question as to impact of further band-limiting (regardless of equalizer implementation), that could be in comparison to an unlimited pulse assuming the matched filter receiver, and of course its performance in the presence of noise (which is always considered). What you will find that with the proper matched filter- adding more bandwidth (approximating a true rectangular pulse of infinite bandwidth) will always increase the SNR, but you will see as you integrate each additional sidelobe a clear point of diminishing returns-- a tiny fractional increase in SNR at the expense of bandwidth that in order to reap the receiver will also need to handle (with associated power and complexity). Filtering the waveform as the OP has attempted if done incorrectly could then add further inter-symbol interference (ISI) when the impulse response of the filter is non-zero at subsequent symbols, leading to more work for the equalizer. Typically such filtering is done with pulse shaping designed to have zero ISI but excellent filtering by having a long impulse response (long in time is short in frequency) but ensuring that it crosses zero at the ideal sampling location of the subsequent symbols based on their matched filter implementation in the receiver.

Specific to the question of quantifying the error by removing higher frequency components, I made an attempt to compute the actual diminishing returns of SNR benefit vs bandwidth for a perfect rectangular waveform with the ideal matched filter in the receiver. Assuming my thinking is correct, I believe the "processing gain" in the matched filter would be given by the following:

$$G_P(B) = \frac{\int_{-B}^B Sinc^2(\omega T/2)d\omega}{\sqrt{\int_{-B}^B Sinc^4(\omega T/2)d\omega}}$$

Here I am using the non-normalized form as $$sinc(x) = \sin(x)/x$$ and $$B$$ is the single-sided bandwidth in radians/sec. I am basically computing the SNR of the matched filter in frequency, and assuming white noise with the numerator representing the weights on the signal components and the denominator representing the weights on the noise.

In dB units of SNR gain this would be $$20Log_{10}(PG)$$. Graphing this relative to a symbol rate $$1/T$$ in Hz, we would get the following when normalized to $$G_P(B=\infty)$$ (To be clear, $$B$$ is single sided, so the first null of a rectangular waveform of rate $$T$$ would occur at $$1/T$$, so plot is normalized to the data rate): So this is suggesting, if we are not constrained by bandwidth for regulatory or other reasons, and are not concerned with the higher processing rate in the receiver, we can increase achievable the SNR by nearly 1 dB if we extend the bandwidth from the main lobe at $$1/T$$ out to $$10/T$$. However also note that wen going from $$4/T$$ to $$10/T$$ we only gain about 0.1 dB, hence the consideration of diminishing returns.

• Thanks for this detailed answer. So you say, Nyquist is not twice the highest frequency, but twice the highest bandwidth. Does that mean, that, if I have a symbol rate of, say, 1kHz, I will need 4kHz sampling rate, since my bandwidth includes the negative frequencies as well, ranging from -1kHz to +1kHz? Nov 17, 2021 at 9:59
• We need to be clear on what we define as bandwidth and a clarity if the signal is real and complex. I like to use the complex baseband signal which extends in the positive and negative frequencies as you are indicating. For generalized multipath distortion, the baseband signal must be complex since nothing about the multipath will ensure the positive and negative frequencies stay Hermitian symmetric (as for a real signal). For this case with a bandwidth that extends from $\pm B/2$ I would refer to the bandwidth as $B$ and the required complex sampling rate is $>B$ according to Nyquist. Nov 17, 2021 at 11:01
• For similar reasons as given above regarding the asymmetric characteristics of the distortion, the equalizer (if done at baseband) would be a complex equalizer. Nov 17, 2021 at 11:05
• Ok, thanks. I didn't mention it, but I was working with a complex equalizer and a complex input signal so far. That clarifies it. Nov 17, 2021 at 11:17
• One last thing: My biggest issue is the sampling delay $\tau$. So when the equalizer is trained on a pilot message sampled at times $t_k=k \cdot T$ and the next message is sampled at times $t'_k=k\cdot T + \tau$, the equalization result is much worse on messages with a large $\tau$ compared to messages with a small $\tau$. My understanding from the literature was, that the fractionally spaced equalizer should provide the same performance on any sampling delay, when the sampling frequency is larger than the Nyquist rate. Did I miss something here? Nov 17, 2021 at 11:17