I have a time domain OFDM signal and to this I need to apply different linear filter however I am not sure on the cut off frequency. I am simulating for multiple symbols where each symbol has 1200 central subcarriers in a 2048 FFT/IFFT bin. The samp freq is 30.72MHz. While designing filters in MATLAB using commands like "fir1", I am not sure how to set cut off freq.
In a real hardware implementation, you always needs to implement additional filtering to an OFDM signal, in digital sometimes and then in analogue. The same is true in fact for all digital signals.
In a simulation, the filtering is realized in digital. The issue is so select the cut-off frequency effectively. It has to be slightly higher than the maximum signal frequency to avoid performance degradation, and short enough to be effective in suppressing the sidelobes.
Concerning performance degradation, the point is so have a cyclic prefix large enough to handle the time response of the filter, and to have a receiver able to equalize the transmission chain correctly. This transmission chain include the different filters, but also the multipath channel.
In my experience, the degradation brought by the filters are negligible compared to the degradation brought by the channel.
You don't need that filter. Your previous questions still indicate you shouldn't be using a filter. OFDM is a filterbank in itself, and applying a filter beforehand will increase delay spread, and thus you get ISI¹.
Sure, with a filter, you'll save on the noise power you'd get in carrier sidelobes, but that should typically be lower than what you lose in ISI.
So, no, you don't apply a different linear filter. You simply ignore the FFT bins that don't contain signal.
¹ you'll get ISI under the assumption that your OFDM system is designed with the minimum amount of subcarriers needed to compensate the maximum expectable delay spread on the channel. That makes a lot of sense because using significantly more subcarriers has no advantage, but makes your system depend on a longer coherency time. If you convolve the channel output with your receive filters, you get a "convolved channel-filter", with a length that is thus longer than your channel, and thus worse than your OFDM system can deal with.