MATLAB has a lot of ways to design filters. You can try
designfilt as a starting point: https://uk.mathworks.com/help/signal/ref/designfilt.html. You'd choose the options to set up a FIR LPF and give it your desired cutoff and other parameters (stopband attenuation, ripple allowance, method of filter design).
Choosing the cutoff frequency to be a function of the bit rate is not what you want though. Doing this will distort your signal which you are interested in. Instead, for GFSK, I'd recommend simply computing the 99, 95, or 90 percent bandwidth and setting the cutoff frequency to be half of that. MATLAB has an function called
obw which gives the 99% bandwidth but you could write a function
myObw(signal, percBandwidth) to calculate any percentage you'd like.
For something like this, it is a good idea to see if what you did actually makes sense. For example, plotting your filter's frequency response over the top of the signal's is a good place to start. The picture below illustrates what I mean and directly shows the problem with choosing the cutoff frequency to be half of your bit rate.