I want to simulate the IEEE 802.11ax waveform and include the IQ impairment and its mitigation. What is a reasonable/expected value of the Amplitude and Phase Imbalance coefficients? Please specify the units, linear or dB and degree or radians.
There's no such thing as a an expected imbalance in a communications standard. That's a property of the radio device, not of the standard!
In my humble experience, receiver imperfections vary a large deal – within the same class of devices across generations and manufacturers, and within the same device with different firmware configuration even.
That's not that surprising: IQ imbalance is typically removed using programmable units in OFDM baseband receivers (and there's also superheterodyne receivers for OFDM, though you don't usually find them in Wifi devices; but I wouldn't rule them out, especially in least-cost devices), and the way you program these makes a difference. IQ imbalance typically gets "calibrated out" when you tune to some faraway frequency after a prolonged amount of time, and depending on how intensely you do that, your tuning-and-recalibration routine takes longer or shorter.
How much you'll want to combat IQ imbalance depends on how much it distorts your signal – in OFDM systems like 802.11ax, that's mostly noticeable as ICI; but since your frequency stability, Doppler, SNR and synchronization error lead to ICI anyways, there's a natural limit for how low you need to suppress IQ imbalance before it stops paying off.
Also, receiver imperfection are related to manufacturing accuracy and factory trimming, and hence a cost- and binning aspect.
So, sorry. I'm not aware of a set of handy numbers I could give you.
Idea: However, if you use a training symbol that only uses the odd half of subcarriers, and then another training symbol that uses the even half of subcarriers, you should get something that is either purely imaginary or purely real in the time domain. You can then estimate IQ imbalance by comparing channel coefficient estimates of adjacent channels. I don't know how reliable that is, it will be worse the more your channel is frequency-selective.