I'm wondering why GMSK use a gaussian filter instead of raised cosine filter as a pulse shaping filter? Is there special reason?
GMSK doesn't just use a Gaussian filter instead of a raised-cosine filter. It uses a Gaussian filter on the phase, before applying it to the modulator. This makes it a nonlinear operation.
When a raised-cosine filter is applied to some modulated signal, it is applied after modulation, as a linear operation.
So there's a whole lot of convenient rules about filters, ISI, "optimal" signal processing, etc., that get thrown out the window with GMSK, because all those rules are predicated on linear operations. But the advantages are worth it, sometimes.
Bottom line: GMSK can be transmitted at a constant amplitude, which means it can be amplified by cheaper, more-efficient class C (or possible E) amplifiers. That makes the increased ISI and decreased ease of analysis worth it.
As to "why Gaussian" -- probably because someone tried it, and it worked good. Good solid theoretical explanations are hard to come by when you stick nonlinearities into the mix -- if there is one, it's probably hand-wavy, and along the lines that a Gaussian filter is everywhere-continuous, unlike a raised-cosine, and so it make the phase transitions everywhere-continuous, too.
(Assuming the context is FSK signals, where the pulse shape is applied to the phase, instead of the amplitude as in linear modulation).
The main reason is that the sideband and out-of-band emissions of GMSK are lower than those produced with a raised cosine filter. See for example this plot from Wikipedia: https://en.wikipedia.org/wiki/File:GMSK_PSD.png
This property is especially important for mobile communications. Since spectrum is so limited, it has to be used very efficiently. Users may use channels that are adjacent to each other, with little or no guard band between them. Any power emitted outside the band assigned to a channel resuls in degraded SINR for other users in the cell.
GMSK requires a more complex receiver, because it reduces bandwdith at the cost of introducing ISI. However, in many scenarios this is an acceptable tradeoff, since bandwidth is more valuable than receiver simplicity.