I have been exploring different ways of improving the EVM performance of quadrature demod block for GMSK demodulation. As we all know, the Gaussian filter introduces intentional ISI, which can be pretty destructive at lower BTs values. I suppose a Viterbi demodulator, which is the ML solution for GMSK, can deal with ISI without much trouble. Other sub-optimal solutions like the quadrature demod in GNU Radio can make use of a Wiener equalizer to reduce the EVM.
I saw in one CCSDS standard that a 3-tap [-0.0859984,1.0116342,-0.0859984] Wiener equalizer was used in their coherent demodulator. The demodulator in question uses Laurent taps as matched filters. I added an FIR filter with the recommended taps after the symbol sync. As it could be seen, in the constellation diagram and its distribution, there seems to be an improvement in EVM performance. I don't know if I use the Wiener equalizer correctly. I would also like to learn more about how to generate the Wiener equalizer taps so I could use more taps as needed.
Additionally, could anyone suggest more ways of reducing the EVM? All I want to see are two dots on the constellation diagram.