Let's do the following, in this order:
- figure out how much power your signal has at the beginning of the stop band (i.e. 5 MHz away from the lowest or highest of the 1200 used carriers)
- compare that to how much attenuation you need relative to the peak power region
- convert the result to a filter specification
How much power does this OFDM system have 5 MHz from the used carriers?
I hope that number's right, but as far as I could find, the LTE subcarrier spacing is 15 kHz.
That means that within your 5 MHz transition width, 333 subcarriers fit.
Let's look at each sinc (the spectral shape of an OFDM subcarrier!) individually first:
The energy in its sidelobe monotonously falls with the "number" of the side lobe.
The exact value is rather hard to come up with, so I just wrote a python script that simply calculates the energy for the sidelobe in the middle between the sinc zeros; it's rather compact, so here it goes:
def f(k, N=1200):
positions = numpy.arange(k, k+N, 1)+0.5
f(13) will give you the sum energy of all the 1200 sidelobes that fall into the 13. sidelobe centre from the edge of the band of used subcarriers; that's a slight miscalculation for the maximum energy in the sidelobe (as the maxima don't exactly hit the center between zeros), but the error should be negligible. (If you think it's not: calculate the probability of the event "one of the two symbols that lead to the actual maximum energy is transmitted" and see that we really don't have to be afraid of regulators knocking on our doors.)
So, plotting that in a dB scale yielded this fine figure:
The orange vertical marks the 333. sidelobe – which, if the 15 kHz subcarrier spacing is right, is the sidelobe at 5 MHz from the highest used carrier.
Compare to how much attenuation you need
From the simulation, the 333. sidelobe has -36 dB power relative to main lobe.
You need -34 dB attenuation.
Hence, to little surprise, the number of occupied carriers (1200) relative to the full bandwidth of the system was chosen exactly to guarantee sufficient out-of-band suppression.
Convert the result to a filter specification
As predicted, an LTE signal fulfills the LTE spectral mask by itself.
You hence don't need any filters to fulfill the spectral mask.
Hence, you also shouldn't use any filters, to allow the flexibility of the OFDM system to correct the channel, not your superfluous filtering.
Hence, the filter specification is: do a FIR filter with taps
[1.0]. Or don't do one at all ;)