My recommendation would be to do the processing in the frequency domain as methods are available that can directly be used to approach your problem. In many cases, speech "quality" is related to the signal-to-noise ratio (SNR) of the signal. At least, I assume this is the case for your application - i.e., telephone communication. In general, some measure for the "quality" can be derived from the SNR then with low SNR indicating "bad quality" and, conversely, high SNR indicating "good quality".
One approach to solve your problem would hence be to try and estimate SNR of the recording. In order to estimate the SNR, well-known noise power spectral density (PSD) estimation algorithms such as, e.g., the "minimum statistics" algorithm [1] or a minimum mean-square error based algorithm [2] can be used. An estimate for the SNR can then be computed from the noise PSD as
$$
\hat{\text{SNR}} = \frac{P_{xx}}{\hat{P}_{nn}}-1,
$$
with $P_{xx}$ the PSD of the noisy input signal and $\hat{P}_{nn}$ an estimate of the noise PSD. A single SNR estimate can be computed for the complete signal, but as the SNR changes strongly over time, it might be useful to compute an SNR estimate for short blocks of the signal - typical block lengths are in the order of $20\,$-$30\,\text{ms}$.
Instead of first estimating the noise PSD, the SNR can be estimated directly, e.g., with the method proposed in [3].
Alternatively, some non-intrusive (i.e., no clean reference signal is required) measure for either the speech quality or the speech intelligiblity can be used. You could try the speech-to-reverberation modulation energy ratio measure [4], of which implementations in Python and MATLAB are directly available.
References
[1] Rainer Martin, “Noise Power Spectral Density Estimation Based on Optimal Smoothing and Minimum Statistics,” IEEE Transactions on Speech and Audio Processing, vol. 9, no. 5, pp. 504–512, Jul. 2001. PDF MATLAB implementation
[2] T. Gerkmann and R. C. Hendriks, “Unbiased MMSE-Based Noise Power Estimation With Low Complexity and Low Tracking Delay,” Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 4, pp. 1383–1393, May 2012. PDF MATLAB implementation
[3] E. Nemer, R. Goubran, and S. Mahmoud, “SNR Estimation of Speech Signals using Subbands and Fourth-Order Statistics,” IEEE Signal Processing Letters, vol. 6, no. 7, pp. 171–174, Jul. 1999. IEEE Xplore
[4] T. H. Falk, C. Zheng, and W. Y. Chan, “A Non-Intrusive Quality and Intelligibility Measure of Reverberant and Dereverberated Speech,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1766–1774, Sep. 2010. IEEE Xplore Python implementation MATLAB implementation