I have an equalized constellation for 16 QAM. The constellation is equalized by LMS algorithm.
I want to calculate the EVM for the equalized constellation. How can I calculate this so that the EVM lies within the threshold limit?
The Error Vector is the Euclidean distance from the actual sample at the optimum timing location in each symbol to the actual symbol location in a reference constellation (as the distance to closest decision boundary, just prior to decision). The measurement metric Error Vector Magnitude (EVM) is computed as an rms quantity over multiple error vectors, where the waveform and the decision boundaries are scaled to either the rms amplitude or a maximum amplitude in the constellation, depending on which standard is used. We can therefore use one sample for every symbol to compute a sufficient number of errors to derive a statistical quantity.
Below shows a single Error Vector for 16 QAM, and the error vector magnitude would be the root-mean-square of all error vectors as given by the formula above (once the constellation and the received signal are properly normalized as described). It also must be mentioned that the normalization by the rms of the waveform assumes the symbols in the waveform are equally distributed, but if not normalization can occur through known training sequences. The idea is that the error vector magnitude should be zero if all the symbols land on the exact locations in the constellation.
To compute EVM normalize the constellation (by either the rms, the peak level of an outermost point, or the rms of the magnitudes depending on which definition is used). The waveform samples just prior to decision (one sample per symbol) are similarly normalized which would then minimize the computed EVM (this is consistent with what the receiver would do anyway just prior to decision to minimize error rate, so it is very easy to compute).
Once properly scaled the EVM computation in its general form is just the formula for the standard deviation of samples from a zero mean random process and given as a percentage quantity; so the average error magnitude given as a percentage of a normalized constellation:
$$EVM = \frac{\sqrt{\frac{\sum_n|t_n-\hat t_n|^2}{N}}}{S} \times 100\%$$
(Where $S$ is given by whatever normalization is used for the constellation)
Notice at Wikipedia en.wikipedia.org/wiki/Error_vector_magnitude it is defined as "The ideal signal amplitude reference can either be the maximum ideal signal amplitude of the constellation, or it can be the root mean square (RMS) average amplitude of all possible ideal signal amplitude values in the constellation. ". In this paper they define the normalization as the rms of the constellation: eprints.soton.ac.uk/263112/1/paper_101.pdf , while here the outermost point is used: ieee802.org/16/tg1/contrib/802161c-01_28.pdf .
Point being, if an EVM computation result is being provided or used, it is important to provide the definition and equation that is being used for that particular computation.
From my experience EVM is defined as $$ EVM = \sqrt{\frac{1}{NP_{avg}}\sum_{n=0}^{n=N-1}(|x_n-x^*_n|^2)}\\ EVM_{\%} = EVM \times 100 $$ where $x_n$ is the equalized symbol, and $x^*_n$ is the corresponding ideal value of the symbol. $N$ is the number of symbols, and $P_{avg}$ is the average power of equalized constellation. $P_{avg}=\frac{1}{N}\sum_0^{N-1} |x_n|^2$. Only thing you need to take care is scaling should be same for $x_n$ and $x^*_n$. If you are comparing $x_n$ against a normalized ideal constellation 16QAM set, that is $$ x^*_n \in \frac{1}{\sqrt{10}}(\pm 1+\pm j),\frac{1}{\sqrt{10}}(\pm 3+\pm j3),\frac{1}{\sqrt{10}}(\pm 3+\pm j),\frac{1}{\sqrt{10}}(\pm 1+\pm 3j) $$ then you need to scale $x^*_n$ by $\sqrt{P_{avg}}$.