Compute the dB of each and then subtract the two to get the difference in dB which should be very small. You can get the same result from converting the ratio of the two to dB directly.
For dB of voltage quantities of AC sources that are arbitrary waveforms you would use the rms value, so the difference between two different sources in dB using both approaches would be:
$$20Log_{10}(A_1) - 20Log_{10}(A_2)$$
$$20Log_{10}(A_1/A_2)$$
Where $A_1$ is the rms value for source 1 and $A_2$ is the rms value for source 2. The rms value can never be negative or complex. Since we are using a ratio, if the two waveforms are similar (both sinusoids for example), then any other consistent metric of amplitude could be used, such as peak amplitude instead of rms).
From the link the OP gave, the intention here is to derive an amplitude and phase imbalance specification and for that the relationship between SNR and imbalance is that the SNR will be the normalized error vector. For example, if the error vector normalized to ideal was 0.1 radians, the SNR would be $20Log_{10}(0.1) = 20$ dB. For small angle the amplitude error and phase error would have the same effect, so if the error vector was 0.1 due to amplitude imbalance, we would also get a 20 dB SNR. (Universally, it is the magnitude of the error vector). An error vector of 0.1 dB is an amplitude imbalance of $20Log_{10}(1\pm 0.1) = +0.8$ or $-0.9$ dB.