How does antenna isolation affect single user spatial multiplexing MIMO performance in an antenna array? Text books talk about a critical antenna spacing of 1/2 of the wavelength provided that the environment is a rich scattering one . Would that be the sole criterion? or would the antenna-antenna isolation rather be the criterion to observe?
What you ultimately want is that the channel coefficients are uncorrelated. MIMO (and diversity techniques in general) depend on the fact that it's unlikely that many different channels will all be bad at the same time. When the channels are correlated, the performance of your receiver will go down dramatically, because correlation means that when one channel is bad, the others are likely to be bad too.
Regarding spacing: if the antennas are too close together, then they see essentially the same channel. So, some spacing is needed; the more distance between antennas, the better. The $\lambda/2$ antenna spacing is a rule of thumb that may or may not work in practice.
Regarding isolation: It turns out that when antennas are close together, they become "mutually coupled". The voltage produced by each antenna depends not only on the received signal, but on the voltages produced by the other antennas. Mutual coupling is reduced with increased spacing, or it can be attenuated using "decoupling" circuits.
Both of these effects result in an increase in channel correlation, and a decrease in performance. Most textbooks mention only spacing, and not coupling, even though this effect has been known in the RF community for a long time. See for example "Mutual coupling effects on the capacity of multielement antenna systems", by T. Svantesson et al.
To answer your question: in an actual system implementation, both spacing and coupling must be taken into account. However, keep in mind that the ultimate objective is to have decorrelated channels; antenna spacing and decoupling are just two techniques to achieve that.