For a set of $N$ orthogonal sequences $\textbf{A}^i = \{a_1^i,a_2^i,...,a_M^i\}, i =1 ..N$, the detection of these sequences can be done by calculating the inner product (dot product) of receive sequence $\textbf{Y}$ with all $N$ sequences $\textbf{A}^i$ to find the one maximizing the result:
$$\hat{i}=\text{argmax}_{1\leq i \leq N} \sum_{k=1}^M a^i_k y_k^* \tag{1}$$
This (1) works perfectly in AWGN channels: $$\textbf{Y} = \textbf{X} + \textbf{W} \tag{2}$$ where $\textbf{X} \in\{\textbf{A}^i \}_{1\leq i\leq N}$ and $\textbf{W}$ is complex Gaussian noise.
However, in fading, specifically Rayleigh fading, channels, (2) becomes $$\textbf{Y} = \textbf{H}\textbf{X} + \textbf{W} \tag{3}$$ where $\textbf{H}$ is fading coefficient matrix. To be more specific, the system is OFDM based and (3) is channel model in frequency domain. In practical conditions, channel can be severely selective, hence coherence bandwidth can be small (not like coherence time which is usually relatively large because of normal to walking-pace velocity). Then I assume the worst case where there is one $x_k$ per coherence bandwidth, hence the associated channel coefficients are decorrelated.
Therefore, (1) becomes $$\hat{i}=\text{argmax}_{1\leq i \leq N} \sum_{k=1}^M a^i_k y_k^* =\text{argmax}_{1\leq i \leq N} \sum_{k=1}^M a^i_k h_k^*x_k^* +w_k^*\tag{4}$$ which simply fails because $\{h_k\}_{1\leq k\leq M}$ are decorrelated (assumption: one $x_k$ per coherence bandwidth).
How could we detect orthogonal sequences in channel model (3)? I am interested in Rayleigh fading. But I am wonder if there is universal method for fading channels.
I think this is basics of CDMA (coherence bandwidth may be replaced by coherence time) or sequence correlation in mobile networks such as 4G and 5G. I do appreciate any answer or any paper reference.