Given is a direct-conversion I/Q up- and downconverter system. Receiver and transmitter share the same (10MHz) reference and hence the LO frequency is identical and there is an (unknown) phase difference between them. The transmitted RF signal is given as:
$$ x_{\rm rf} = x_i \cos(\omega_c t) + x_q \sin(\omega_c t) $$
Now I look at only one channel (e.g. the I channel) at the receiver (that means that any I/Q imbalance of the receiver is irrelevant). At the I branch, this signal is downconverted with an arbitrary phase shift phi and filtered with H:
$$ y_i = H( x_{\rm rf} \cos(\omega_c t + \phi) ) = \cdots \\ \approx H( \cos(\phi) x_i/2 - \sin(\phi) x_q/2 ) \\ = \frac{\cos(\phi)}{2}H(x_i + x_q \tan \phi) \\ = G(x_i + \alpha x_q) \\ = G(x_i) + \alpha G( x_q) \\ = G_1(x_i) + G_2(x_q) $$
The filter H is contains all filtering effects (mainly anti-aliasing filter).
I want to find a linear relationship between the time-domain samples $y_i[n]$, $x_i[n]$ and $x_q[n]$. A straight forward way is to bring this equation in a vector-/matrix form, model $G_1$ and $G_2$ as FIR filters with $Q$ taps and solve it via Least Squares:
$$ \mathbf{y}_i = \mathbf{X}_i \mathbf{g}_1 + \mathbf{X}_q \mathbf{g}_2 $$
This works nicely and I get the result as expected (in my case, the error between $\mathbf{y}_i$ and the received samples is -45dB). The caveat is I get a set of $2Q$ (real-valued) coefficients but according my derivation, $G_1$ should identical to $G_2$ up to a constant factor. If I would do the same for the $Q$ channel I get again $2Q$ coefficients and a total of $4Q$ real-valued coefficients if I were to model the I/Q relationship that way.
Clearly I can model everything directly in complex domain (with $Q$ complex-valued or $2Q$ real-valued coefficients) but then I cannot look at only the I channel without effect from the Q channel.
Hence a more proper way is to look at this relationship:
$$ \mathbf{y}_i = \mathbf{X}_i \mathbf{g}_1 + \mathbf{X}_q \alpha \mathbf{g}_1 $$
However, this is not a least squares problem any more. For that reason, I sweep $\alpha$ within a reasonable range, compute the LS solution and take the $\alpha$ with the minimum error (this seems to be convex). But if I do this, the error -35 dB (as opposed to -45dB above).
I really do not understand why this approach would not work. Why? And what is the proper way to model this?
Background info: I am testing my own direct conversion receiver. If I transmit data in only one channel (I or Q) and look at only one output channel (e.g. I) I get the performance that I expect based on linearity-, noise- and phase noise measurements of the receiver. If I look at I/Q at the same time (and model the the effect of the receiver as complex-valued FIR filter) then the error is more than 5 dB worse. For debugging I would like to avoid any interaction (such as I/Q imbalance) between I and Q and only look at the I channel output. But as soon as I transmit a full complex-valued input and look only at the I output my result is again at least 5 dB worse than predicted by noise, linearity, phase noise.
As another step, I use an Arbitrary Waveform Generator (Rohde & Schwarz SMW200A) used as transmitter and a VSA (Rohde & Schwarz FSW) as a receiver where I see the same issues. Both instruments are calibrated and the transmitter has superior image rejection. For these reasons I assume I made a conceptional or algebraic mistake in my derivations/assumptions.