gives theoretical expressions for BER in Rayleigh fading channel. Can I use the expression for 64-QAM in this paper? The Authors have given a plot of the BER performance but there is no expression for it. So, I am unsure how to get the plot for Rayleigh case.
The model they have used is general and you can apply it to QAM as well.
In wireless communication systems, channel and noise are two different impairments affecting the overall transmission. Here as everywhere else, the noise is AWGN (additive white Gaussian; if you want to know what these terms mean, see this article). And the channel is a linear filter with taps whose magnitudes are modeled as Rayleigh random variables.
Yes, you can use the expression in that paper. The authors didn't give the expression because they might not have known it and simply generated it from Monte Carlo simulations.