This could happen as discriminator gain is increased with a filter discriminator approach since in many of those approaches the gain would be maximum and linear for small signals only and then the slope of the discriminator slowly goes down coinciding with the results in your plot (such that you no longer get a perfect sine wave out for a sine wave in—- so you can also look at it as converting some of your signal components to other harmonics at the discriminator output due to that non-linearity). Ultimately as you increase the gain further your signal will start to saturate and then even invert depending on the wider bandwidth shape of your discriminator.
Plot the derivative of your frequency discriminator to see this more clearly as to its usable range for a given frequency deviation- the ideal for no distortion is to have a constant slope.
I see from the link you provided that the slope is indeed constant over a limited usable range. So specifically in your case I assume as $k_f$ was increased, it would cause a proportionate amount of the signal to go beyond the discriminator range (where your response is 0).