The instantaneous frequency is the time derivative of phase. The time derivative operator is a high pass filter, so it on its own will enhance high frequency noise and attenuate lower frequencies. For example, a step change in time would become an impulse after taking a derivative.
It is possible that the spikes are from "steps" that are incorrectly introduced into the phase during processing; for example if the phase is wrapping at the $2\pi$ boundaries. I suggest reviewing a plot of the phase versus time used prior to taking the derivative and confirm if this phase should be "unwrapped" prior to the derivative operation. (unwrap is a function for doing this in Matlab, Octave and Python scipy.signal).
As a simple example consider the case below of a constant frequency, which would therefore be a ramp in phase. A small amount of noise was added to the phase which is barely visible in the plot of phase versus time, but clear in the demodulated frequency (which was done as a simple difference of phase from sample to sample).
If the phase was incorrectly "wrapped" to stay within the range of $\pm pi$, the same demodulator would produce the result as given by the plots below, with very significant negative spikes where the phase wrapping occurred. Even if we did the demodulation with a improved digital differentiator that also filtered the high frequency noise, a significant amount of these spikes would still feed through:
Not to conclude that phase wrapping is the source of the spikes in the OP's specific case as this is just one obvious / common source for this to occur; any other steps in phase (from actual noise sources) would also lead to spikes in the demodulated signal.
As a side-comment: this high frequency noise enhancement is a common 'feature' in FM demodulation, and is the reason for the use of pre-emphasis in some FM modulation implementations.
