If I am passing a QAM signal through Root raised cosine Tx filter and measure the constellation at output, it seems very bad. If this bad output again pass with other another root raised cosine Rx filter, there is a significant improvement in EVM. Can I ask the reason behind this.
To satisfy the Nyquist criterion to eliminate the inter-symbolic interference (ISI) you need the whole link (TX, Channel and RX) to constitute a raised cosine filter, so you are having the root of the transfer function in the TX and the other square root in the RX. Since the overall response is just multiplication of the TX and RX transfer function, accordingly, you will get a full raised cosine when you do so.