With integer modulation indices, each symbol is actually an integer number of full oscillations. You DFT that, and get a sharp, discrete spectrum of tones, convolved with the pulse shape.
With non-integer indices, you get "cutoff" oscillations. That is, we need to dsitinguish between two cases:
- Continuous-Phase FSK: your next oscillation starts with a phase determined by the previous symbol(s) (as the phase continues where you left off)
- Non-continuous-Phase FSK: your symbol starts with a specific phase, no matter what the end phase of the last symbol was.
In the non-continuous case, you get a PSK-alike spectral component (because you suddenly change the phase). If you've been (virtually) letting the subcarrier oscillator run through while the other one was active, you'd only need convolve that PSK spectrum with dirac impulses to form your spectrum. If you "stopped" it, and always started a symbol with the same phase, you'd even get the PSK effect if constantly sending the same symbol. If you then also send both possible symbols alternatingly, you'd have to convolve that with the abrupt symbol phase change, so you'd get a slightly biased PSD. I think this is what we're seeing here.
EDIT Found out this is CP-FSK:
Well, the fact that you don't have full oscillations definitely explains why you don't get diracs in spectrum – especially since you're not really observing the PSD (which is an "invisible" property of a stochastic process), but a PSD estimate, very likely based simply on a mag² of a DFT – in other words, oscillations that don't fit into the DFT size an integer amount of times cannot ever be sharp spikes. In reality, this poses no problem (because for reliable detection, you need signal power, not peak height, and the power within the "broadened" peaks is, thanks to Parseval, the same).
Non-symmetry is an interesting effect here, and I wouldn't have expected it to be so clear (it should have been there, by the pure fact that the support of your subcarrier shapes in frequency domain would not be limited to "their" half of all frequencies, just not as clearly). The fact that the "right" half of the spectrum seems to be consitently higher worries me a bit – is it possible there's channel model/analog reasons for this?