The effect is in sidelobe level and is evaluated with consideration to number of samples per symbol used, waveform quality and the spectral mask. Increasing the length of the filter in total number of symbols or windowing the truncated impulse response used can both help to reduce sidelobe level. I recommend evaluating the spectrum for the equivalent signal that would be transmitted which would then include all subsequent interpolation and filtering in a specific implementation. Ultimately the transmitted spectrum, and the resulting waveform quality is to be evaluated against requirements and traded with implementation complexity. Below are examples showing the spectral effects when choosing different truncations.
Below is a plot showing the spectrum using a truncation of 2, 3, and 4 symbols for the case of $BT=0.5$ and 4 samples per symbol. Here we do see a significant improvement in sidelobe level when increasing the truncation from 2 to 3 symbols, and no further improvement past that.
A Gaussian pulse in the time domain has a Gaussian frequency response in the frequency domain, both of which extend to plus and minus infinity in each domain. For GMSK we further convolve the Gaussian pulse with a rectangular pulse, which thus results in the frequency domain being the product of the Gaussian response with a Sinc function, and we still have the condition of the ideal waveform extending to plus and minus infinity in each domain.
Note from this and the plot above that we have two effects distorting the signal: We have truncation in the time domain, which results in a convolution in the frequency domain of the ideal response with a sinc function, AND we have aliasing in the frequency domain due to using a finite sampling rate. Since the ideal waveforms extend to infinity, we can never extend the pulse long enough in the time domain, or sample high enough in the frequency domain to avoid these affects and achieve perfection. Thus we need to work from target requirements and extend both parameters to meet our target specifications.
To get further insight as to the underlying effects of truncation while minimizing the aliasing effects in the frequency domain, we can oversample the pulses and then observe the spectrums. For example, below shows the case when we oversample to 20 samples per symbol where we can now see the difference between pulses extended to 2, 3 and 4 symbols and the sidelobe level that results. This does not mean the samples per symbol used should be increased in the implementation, but shows us directly the trade space in samples per symbol versus frequency domain aliasing. Each null in the frequency response is spaced by $1/T$ where $T$ is the symbol period (as given by the sinc product due to convolving the Gaussian pulse with a rectangular pulse). So if we for example set the sampling rate $f_s$ such that the Nyquist frequency $f_s/2$ was at $2/T$ (at $f=0.1$ in the plot below), we can see in the plot that all aliasing components that are above $f_s/2$ would be lower than 80 dB. This is likely sufficient to meet any practical distortion target for GMSK and suggests that $f_s=4/T$ or 4 samples per symbol is more than sufficient. Similarly we see that extending the GMSK pulse for only 2 symbols will result in sidelobes less than 50 dB down which may be questionable to use without the requirement of additional filtering, while using 3 symbols will result in more than 100 dB, also likely sufficient for any practical distortion targets in GMSK applications.
Ultimately this is to suggest and demonstrate the importance of evaluating the spectrum for the specific implementation done (as well as waveform quality). By oversampling and overextending the pulse and observing the frequency domain results, we can choose how far to extend both the time domain (for the length of the GMSK pulse) and the frequency domain (the number of samples per symbol) to meet our target requirements.
Further since the ideal pulse extends to plus or minus infinity, frequency domain distortion will be minimized by multiplying the time domain pulse with a finite duration window that is optimized for such distortions (unlike the Gaussian). This adds some distortion in the time domain at the benefit of minimizing distortion in the frequency domain; and is just another tool that can be utilized in our signal processing utility belt. (It is likely however in this case that 4 samples per symbol and 3 symbol duration would be more than sufficient as is.).
The Python code for the above plots are as follows:
import numpy as np
import scipy.signal as sig
import matplotlib.pyplot as plt
import scipy.fftpack as fft
def gpulse(ts, b, fs, t):
# ts: symbol period
# b: 3 dB bandwidth
# (b * ts is the bandwidth-time product)
# fs: sampling rate
# t: time vector
sigma = np.sqrt(np.log(2)) / (2 * np.pi * b * ts)
rect = np.ones(int(fs/Ts))
h = 1 / (np.sqrt(2 * np.pi) * sigma * ts) * np.exp(-t**2 / (2 * sigma**2 * ts**2))
g= np.convolve(rect/int(fs/ts), h)[int(fs/ts//2-1):int(-fs/ts//2)]
Create truncated pulses:
nsyms = 4
ts = 1
b = 0.5
fs = 4
tsim = np.arange(nsyms * ts * fs)/fs - nsyms * ts/2
pulse4 = gpulse(ts, b, fs, tsim)
pulse2 = pulse4[np.where(np.abs(tsim)<1)]
pulse3 = pulse4[np.where(np.abs(tsim)<1.5)]
The above creates the following time domain pulse and then selects truncated versions to be 2, 3 and 4 symbols in duration.
The remaining plots are simply the dB magnitude of zero padded FFT's of these truncated pulses to provide the spectral envelope of the GMSK waveform:
faxis = fft.fftshift(fft.fftfreq(2**16))
fout2 = fft.fftshift(fft.fft(pulse2, 2**16))
fout3 = fft.fftshift(fft.fft(pulse3, 2**16))
fout4 = fft.fftshift(fft.fft(pulse4, 2**16))
plt.plot(faxis, 20*np.log10(np.abs(fout2)/fs), label= "2 symbols")
plt.plot(faxis, 20*np.log10(np.abs(fout3)/fs), label= "3 symbols")
plt.plot(faxis, 20*np.log10(np.abs(fout4)/fs), label= "4 symbols")