I'm validating a C++ oQPSK receiver. The 5Mbps oQPSK signal is generated by a satellite emulator. The receiver structure goes as follows:
SDR --> AGC --> Costa's Loop --> Delay --> Polyphase Symbol Sync (Maximimum Likelihood) --> Constellation demap
I have tested the Costas and the polyphase sync with BPSK and QPSK and they performed very well. However, the constellation I get for oQPSK seems to be unlocking at a very specific interval. Upon inspection, I found out that the problem lies with the polyphase symbol sync. As you can see below, the fractional interval drifts between 0.0 and 1.0 due to the difference in clock frequencies between the SDR and the satellite receiver. Upon further inspection, I found out that unlocking is caused by PFB filters corresponding to fractional indices of around $\mu=0.5\pm0.1$.
Could anyone provide me with any clue as to what could be behind this? Why PFB behaves like this for this range of fractional indices?