This is a great question and comes down to the AGC design and optimizing the available dynamic range on the ADC, given a receiver minimum SNR, sensitivity and interference rejection requirements.
I first need to know or establish these requirements and then usually start a receiver design from the ADC options available within the cost, power and technology constraints. I work from the ADC in deciding what functionality would be analog and what would be digital (as most functionality can be either in terms of the "radio" operation). That said, the ADC must provide as a minimum the the dynamic range the waveform fundamentally requires (SNR required to meet bit error rate requirements for example) and then any excess can be utilized for digital AGC capability, thus simplifying the front-end by allowing for digital filtering of out of band interference or multi-carrier applications with wider dynamic range between channels. This approach would use an analog AGC to set the input signal to the optimum level below full-scale (see chart at bottom of post for that) at the input to the ADC and then provide digital AGC implementation after channel selection or out of band interference (that ended up controlling the pre-ADC signal level) has been removed.
The analog AGC can be stepped with fixed gain amplifiers or attenuators as you are doing, or can be implemented with variable gain attenuators or amplifiers that provide a smooth adjustment of signal level with a control voltage, maintaining a constant level at the ADC input.
Below shows a diagram for the key concepts and considerations in the mixed signal AGC approach with fixed gain steps prior to the ADC, showing the hysteresis to avoid chatter when crossing thresholds that would change the input power. The upper and lower range for the signal varies between the minimum signal level to meet the receiver design objectives under worst case interference conditions ("blockers") and the maximum signal where performance is also still achieved (before degradation due to clipping).
Importantly consider what occurs in a flawed digital AGC implementation with stepped gain front-end control as shown in the diagram below. The AGC as a loop will have a loop BW set to be as fast as possible to correct for amplitude changes without being so fast so as to track out amplitude components of the modulation (for waveforms that have amplitude modulation components such as OFDM prior to demod, QAM, etc). My rule of thumb is using a loop bandwidth anywhere from 1/20th to 1/50th of the symbol rate.
That said, when an AGC controller steps the gain of the analog front-end, following the hysteresis diagram above, the signal at the demodulator will immediately drop, the digital AGC will detect the change in power level and correct the level within the time allowed by the loop BW (again by design it cannot correct too fast or will remove AM modulation signal of importance).
A solution is to add a calibrated compensation in the digital AGC, that when upon controlling the front end gain, a compensatory gain is added post ADC prior to the digital AGC. This will minimize the step to a much shorter glitch (since the time between the two cannot be 0 but can be minimized), resulting in much less disruption to the receiver when changing front-end gain.
Important to the design considerations mentioned for AGC, both digital and analog, I have developed the following useful curves showing what level to set the AGC below full scale (in this chart, "full scale" is where a single complex IQ tone would clip, so the equivalent clipping for a real sine wave would be -3dB on the horizontal axis).
This is a very useful chart for maximizing the dynamic range with an AGC, and equivalently shows where the rms level of a modulated waveform needs to be digitally in order to minimize the number of bits in the datapath without distorting the signal due to clipping. Specifically this curve shows the optimum operating point for a Gaussian distributed waveform (as are well approximated by most modern modulations) between two distortion sources: quantization noise and clipping.
Two examples of using this chart: At ADC input and at baseband digital modem of the IQ waveform, both assuming Gaussian distributed signals.
ADC Input example: Suppose we wanted to maximize the available dynamic range for an 8 bit real converter of a modulated signal at IF. The purple line on the chart is for 8 bits and shows that the combined noise between clipping and quantization is minimized when the input signal is AGC'd to be 12 dB - 3 dB = 9 dB below the same level where a sine wave would clip. At this level we would have 40 dB SNR due to clipping and quantization: the total power due to clipping and quantization would be 40 dB below our signal.
Modem input example: We want to know the minimum number of bits required on I and Q for a waveform where we need 20 dB SNR for demodulation to meet design objectives. We could achieve that with a 4 bit data path IF we set the IQ signal level at the modem input to be 8 dB below the full scale IQ radius.
If the rms level of the signal is set too high, excessive clipping noise results; if it set too low, excessive quantization noise results--- or if we have excess dynamic range in the ADC (which we want to simplify the front-end), if we set it too low we lose dynamic range.
This point is further demonstrated with the following graphic. Many will see the few samples exceeding the range of the ADC and decide that the input signal is too high as clipping is occurring. However, as is clear from the chart above, they may lower the input signal level (set AGC level lower) to eliminate any clipping noise- but they will likely then be past the minimum- in which case more quantization noise is added for a net loss. The ideal operating point will have some amount of clipping, and we win overall by allowing that.
To Note importantly that these curves are based on the assumption that the waveform at the ADC input is Gaussian distributed (a very reasonable approximation for OFDM for example, but not particularly so for single carrier QAM). The upper part of the curves are derived from the tail of the Gaussian distribution as the predicted clipping noise while the lower part of the curves are from the variance of quantization noise as $q^2/12$ where $q$ is a quantization step. For waveforms of other distributions the lower part of the curve is not changed but the upper part will be specific to the actual waveform distribution. Ultimately for unknown waveforms I use this curve gives me to get an idea of a starting point and will resort to simulation and test with representative waveforms to find the optimum set point (rather than re-deriving the integral of the tail for non-Gaussian distributions which would be the complete analytical approach).
