The spread in the plot is the peak to peak phase, while the integrated phase noise is an rms quantity- The OP in his comment below this answer reported a measurement of 35° in the constellation and a 4.6° "integrated phase jitter". If this was rms phase then the measured peak as a ratio to the standard deviation would be $35\sigma/(2 x 4.6) = 3.8\sigma$ which is not an unreasonable result.
I had first assumed the OP was comparing an rms cycle-cycle jitter measurement in units of radians to the peak-peak phase deviation as shown on the constellation, hence the more detailed descriptions below that I will leave for future reference.
Determining RMS Phase
The rms phase error over the measurement duration can be determined from the SSB phase noise measurement $\mathscr{L}_\phi(f)$ using
$$\phi_{rms} = \sqrt{\int_{f=f_{low}}^{f_{BW}} S_\phi(f)df}$$
Where:
$S_\phi(f) = 2 \mathscr{L}\phi(f)$
$\mathscr{L}\phi(f)$: SSB Phase Noise (the values from the OP's plot above after converting from $\tt dBc/Hz$ to $\tt rad^2/Hz$)
$S_\phi(f)$: DSB Phase Noise (accounting for both upper and lower sidebands of phase modulation).
Th conversion from $\tt dBc/Hz$ to $\tt rad^2/Hz$ is $10^{k/10}$ where k is the value in $\tt dBc/Hz$.
With the range of integration further detailed below.
Determining RMS Jitter
A cycle-cycle jitter jitter measurement includes an effective high-pass filtering of the phase noise prior to integration, and converting to units of time.
The rms cycle to cylce jitter can be determined from the SSB phase noise measurement $\mathscr{L}_\phi(f)$ using:
$$t_{rms} = \frac{T}{\pi} \sqrt{\int_{f=f_{low}}^{f_{BW}} (1-\cos(2\pi f)) \mathscr{L}_\phi(f)df}$$
This is arrived at by doing the following:
- High pass filter $S_\phi(f)$ and integrate and square root to get rms phase:
$$\phi_{rms} = \sqrt{\int_{f=f_{low}}^{f_{BW}} |H(f)|^2S_\phi(f)df}$$
- Convert units of phase in radians to units of time in seconds using:
$$t_{rms} = \frac{\phi_{rms}}{2\pi } T$$
Where
$H(f)= 1-e^{-j2\pi f T}$: Frequency response of cycle to cycle subtraction
$|H(f)|^2= (1-e^{-j2\pi f T})(1-e^{+j2\pi f T}) = 2-2\cos(2\pi f)$
$T$: Cycle time in seconds
$f_{low} = 1/T_m$ in Hz
$T_m$: total time duration of the data used for the rms jitter measurement in seconds.
$f_{BW}$: Channel bandwidth (all filtering prior to jitter measurement) in Hz.
This is further explained as follows:
The measurement of cycle to cycle jitter is done by computing the rms of the difference in time from the expected zero crossing location between adjacent symbols.
This measurement of cycle to cycle jitter is a high pass process - the jitter in one cycle is subtracted from the jitter in the previous cycle so is the filtering process given by the subtraction of the phase error (given in units of time) in one cycle from the phase error in the previous cycle.
As a discrete process this is $H(z)= 1-z^{-1}$ followed by an rms computation (compute the standard deviation of the time difference). Since the phase noise measurement represents a continuous time process, I used instead a continuous time integration with the equivalent filter $H(s) = 1 - e^{-sT}$. This is a completely valid approach since selecting the time error at each symbol location is a decimation process of the underlying continuous time function (of error vs time), which folds in all the upper frequency noise components, which then get captured in the integration.
This is actually a comb filter response, with a zero at DC forming a highpass over the measurement bandwidth given by the symbol rate. Above cutoff and the noise is uncorrelated so adds in power for a +3 dB gain, and below cutoff the correlation increases by the inverse of the frequency providing a $1/(f_c-f)$ rejection of the phase noise. The easy way to see this is to consider the very very low frequency components of phase noise —- for those components the time shift in adjacent symbols is the same so is rejected when we compute the subtraction of the zero crossing offset between those two symbols. Below shows an example frequency response of the cycle to cycle measurement assuming a symbol rate of 10 MHz. Individual frequencies components above cut-off will have strong correlation every time the phase cycles through $2\pi$, hence all the nulls in the actual "comb" filter response.
Our observation time $T_m$ is given by the duration of the data set is another high pass filter (in order to see all the way down to DC you would need to observe forever!). An exact equation would add this additional high pass filter as an additional 20 dB/decade roll-off toward zero with a cutoff at $1/T_m$ and then extend the integration lower, but I have found it to not be necessary in all practical applications, getting the same result by simply starting the integral at $f_{low} = 1/T_m$ (similar to an equivalent noise bandwidth computation where the resultant filtering response is equivalent to a brickwall filter). So a brick-wall highpass with cutoff $1/T_m$ is used to model the observation duration, and similarly the channel bandwidth $f_{BW}$ is reasonably approximated as a brickwall lowpass filter since the phase noise is typically an insignificant contribution at the larger frequency offsets defined by the channel.
In summary, process the double-side-band (DSB) phase noise ($S_\phi(f)$) which is double the SSB phase noise ($\mathscr{L}\phi(f))$ with the high pass function that also has the additional noise power gain of +3 dB, and then integrate the result for total integrated power. Take the square root of this to get the standard deviation and this should match jitter once you convert between units of seconds as in jitter and units of radians as in phase noise using $t = \phi T/2\pi $ with $T$ as your symbol duration.
