What is the advantage of performing the FSK using IQ modulation?
- You only need one RF oscillator operating at a single frequency, instead of having 2 (or more in the case of M-ary FSK) oscillators operating at separate frequencies for each bit/symbol.
- Since you only have one oscillator, you don't have to worry about discontinuities in the phase of the transmitted RF signal, like this:
Sharp discontinuities contain high frequency which widens the bandwidth of the signal. With I/Q modulation you'll have a much easier time producing a continuous-phase FSK signal (CPFSK).
How to demodulate the FSK signal?
First lets assume we are trying to demodulate a simple FSK signal where the center frequency is $f_c$, a '1' is represented by the frequency $f_c+\Delta f$, and a '0' is represented by the frequency $f_c-\Delta f$.
Before IQ sampling, you tune your RF front end receiver to $f_c$. Then when you sample you have a baseband I/Q representation of the signal, centered around 0 Hz. The only thing to do is figure out if you have a positive frequency (+$\Delta f$) or negative frequency (-$\Delta f$). The easiest way to do that is to monitor the change in phase of the baseband IQ signal. A positive phase change corresponds to a positive frequency, and a negative phase change corresponds to a negative frequency.
So monitor the phase (the angle of $I + jQ$) and watch how it changes over one symbol period. Positive change = '1', negative change = '0'.
Does the FSK signal look triangular wave or sinewave?
If you're talking about the RF signal, it looks like a sinewave of varying frequency.
If you're talking about baseband IQ signal, it also looks sinusoidal, but with a bunch of phase shifts, like this (blue = I, orange = Q):
In this particular example, a '1' corresponds to a phase change of $+\pi/2$, and a '0' corresponds to a phase change of $-\pi/2$.