The OP is correct that the constellation is spinning due to a frequency offset, and removing the phase versus time is the right direction to take (which is frequency offset given frequency is the time derivative of phase). However the residual trend line that the OP determined is not the frequency offset due to the modulation changing the quadrant from sample to sample combined with the unwrapping led to the trend line shown.
A decision directed approach is commonly used for carrier recovery due to its simplicity and ability to remove the modulation and correctly determine over a longer term average the time rate of phase change. I detail the decision directed phase detector at this post which results in a phase error metric for each symbol given by:
$$\phi_\Delta[n] = \hat{I}[n]Q[n]-I[n]\hat{Q}[n]$$
Where $I[n],Q[n]$ is the complex waveform at sample $n$ (at the correct symbol timing locations given one sample per symbol) and $\hat{I}[n],\hat{Q}[n]$ is the closest QPSK decision for that sample. Below shows a graphic with the crosses at the correct "decision" locations for QPSK, along with an example sample as a complex vector given by $V_2 = I+jQ$, and its closest decision given by $V_1 = \hat{I}+j\hat{Q}$.

To select the correct one sample per symbol, timing recovery can be done prior to carrier recover using the Gardner Timing Error Detector which operates on multiple samples per symbol and converges under relatively large carrier offsets. From these samples the OP's code to remove the trend line would work in a post processing approach to demonstrate its operation. The typical approach is to use this within a carrier recovery loop for continuous tracking of the phase change versus time (frequency offset).
I detail a complete carrier recovery implementation at this post showing the decision directed phase detector combined with a numerically controlled oscillator (NCO) resulting in demodulated QPSK symbols with all carrier offsets removed.
Another simple approach applicable to QPSK and QAM, for frequency offsets that are less than a quarter of the Nyquist rate, is to first raise the waveform to the fourth power. This strips the modulation and creates a stronger spectral component at four times the carrier offset frequency which can then be easily tracked with a clean-up PLL. The carrier is then recovered by dividing this cleaned up reference by four.
These other posts also provide further details of carrier recovery for QPSK and QAM.
How do I know this? I teach courses on DSP and Python related to wireless comm through dsprelated.com and the ieee with new courses running soon!