I am having difficulty in understanding how the Authors in this paper, An EM based method for semi blind identification of linear systems driven by Chaotic signals

have used the expressions derived from applying an EM_EKF estimation method when the driving process is a non-linear input. They have considered the noise model as AWGN. They then apply their method for in equalization. They apply the estimators derived in Eq(12)--Eq(15) to estimate the MA process with Rayleigh fading coefficients. I have attached a picture of the method that they have used.

Question 1: I want to know what the EQ(1) would be for QAM communication and if the expression for the estimators would change if I want to apply to QAM communication. It is not clear to me if they have applied it to BPSK or to QAM model. If I want to equalize in Rayleigh fading channel, then would the noise be AWGN or Rayleigh noise? As they have applied the same estimators, I think the noise model is still AWGN. But in may other papers, when we say rayleigh model then don't we mean that the noise is rayleigh distributed?

Question 2: Link http://www.raymaps.com/index.php/theoretical-ber-of-m-qam-in-rayleigh-fading/

gives theoretical expressions for BER in Rayleigh fading channel. Can I use the expression for 64-QAM in this paper? The Authors have given a plot of the BER performance but there is no expression for it. So, I am unsure how to get the plot for Rayleigh case.