Choosing the scaling parameters for a modem implementation

I have succesfully modeled a modem using floating-point data and with the following scheme, where the SRRC filter has gain G=6dB.

Modulator -> SRRC(UPx4) -> D/A -> Channel -> A/D -> AGC -> SRRC(DOWNx2) -> Remaining receiver blocks


The modulator outputs full-scale (-1 to +1) I/Q values and at the receiver I'm getting a signal whose power is 1/4=0.25W as expected, the digital AGC uses that value as target, and after the other filter I do get back a more-or-less unitary-energy signal. The reason to keep the ADC there is to provide the other blocks (e.g. Gardner synchronizer) with a signal whose amplitude is almost unitary.

Now, I'm not sure this is the best way to structure the TX/RX chain but I'm pretty sure this set-up won't fly that well when converted to fixed-point arithmetic: what I'm afraid of is that the AGC/SRRC combination as devised may cause the signal to clip/overflow and thus degrade the receiver performance. Is there a better and perhaps wiser way of designing the receiver chain? Use normalized filters? Use a different target value for the AGC?

The bit-growth caused by the two SRRC filters doesn't worry me, the plan is to compute the result with higher precision and then round/truncate it to 16-bit.

I'm also considering to lower a little the AGC target level, instead of targeting 1/4 of the quantized power scale the reference could be moved to a smaller value (e.g. 1.25) and reduce the chances of clipping.

I'm pretty new to all this DSP stuff so please bear with me, I'm trying to learn as much as possible while also trying to get a product out of the door :)

The chart is applicable to both real sampling with a single ADC as well as complex sampling with a dual ADC. For real sampling, full scale is where a Sine wave would start to clip and would be -3 dB on the horizontal axis. For complex sampling a similar "tone" would have constant magnitude as $$Ae^{j\omega t}$$ and would be 0 dB on the horizontal axis when at full scale where clipping starts to occur. The horizontal axis is the set point of the AGC relative to full scale and the vertical axis is the resulting total noise relative to the power in the signal (so negative SNR), depending on the Effective Number of Bits (ENOB) for the converter. For example if we had an ENOB of 10 bits, then for a real waveform the optimum AGC set point would be at approximately -13 dB on the horizontal axis, and the noise at this set point is approximately the best it can be at -51 dB. For a real signal, this is 10 dB below the that level where a sine wave would clip. So if in our given system we knew a sine wave would clip at +10 dBm (for example), we should AGC to 0 dBm and maximize our dynamic range and SNR.
I created this chart by combining the SNR due to quantization noise (using the well established formula $$SNR = 6.02 \text { dB/bit} + 1.76$$ dB with the predicted SNR due to clipping alone for a Gaussian distributed waveform. This SNR assumes the full Nyquist zone is being used, otherwise subsequent filtering will offer further improvement. Thankfully the detailed math for the case of clipping (from the area under the tail of a Gaussian) has already been worked out by Walt Kester and Rob Reeder in their article about Noise Power Ratio referenced in my graphic below.