Short answer: For a direct implementation of PWM in discrete-time, yes, for 48 kHz, 16-bit audio, a clock of 3.2 GHz is required.
Direct Stream Digital (DSD) format uses 1-bit $\Delta$-$\Sigma$ modulation (DSM), which is somewhat different than pulse-width modulation (PWM), although both generate an on-off switched sequence as the output.
Regular PWM uses a comparator (the sign function), where the input is a modulating (the desired signal) and a high-frequency carrier signal. The harmonic and intermodulation distortion between the desired signal and the carrier signal are what is going to determine the unwanted signal output limiting the signal-to-noise-and-distortion (SINAD) ratio. With a triangle-wave carrier with frequency $\omega_c$, the output for a single sinusoid with frequency $\omega_v$ and amplitude $M$ between 0 and 1 is:
$$
PWM(\omega) = \frac{1}{2} M \cos{\omega_v t} \\
+ 2 \sum_{m=1}^{\infty} \frac{J_{0}\left(\frac{1}{2} m \pi M \right)}{\pi m} \sin\left(\frac{1}{2} \pi m \right) \cos\left(m \omega_c t \right) \\
+ 2 \sum_{m=1}^{\infty} \sum_{n=\pm 1}^{\pm \infty} \frac{J_{n}\left(\frac{1}{2} \pi m M \right)}{\pi m} \sin\left(\frac{1}{2} \pi (m+n) \right) \cos\left((m \omega_c + n \omega_{v} ) t\right)
$$
$J_n$ denotes the n-th order Bessel function.
The first line has your desired sinusoidal signal, the second and third line are harmonic and intermodulation distortion. This is entirely deterministic and similar spectra can be computed for any input signals that can be described by Fourier-series. Sample rate does not play a role in the ideal, continuous-time case. The distortion products can typically be moved to higher frequencies and filtered out by a low-pass filter by choosing a higher carrier frequency $\omega_c$, but there is no guarantee that intermodulation products will not appear in the baseband. The third line contains what is called foldback distortion, and as you can see, there will always be distortion in the baseband, and there is no way to guarantee that some input signal will not cause some combination of superpositions that cause severe foldback distortion. The best you can hope for is that the foldback distortion is small for most signals (which it usually is, and it is dominated by other noise sources).
Discretization (quantization and sampling) effects cause additional distortion, and several PWM methods try to take this into account. As you have discovered, you need very high time-resolution in order to represent large word-widths where there is discretization in time. This does not mean that the (average) switching frequency has to be high, only that the time-resolution is high. That is, you need to represent your output sequence as a non-uniformly sampled signal. Hence you don't need a high sampling rate, only a mechanism on the output that can cause non-uniform switching with that accuracy. This does makes it difficult to accurately simulate PWM. There are several PWM-like methods that seek to solve these distortion problems.
DSM (noise-shaping using 1-bit) also uses the comparator, but there is no carrier signal. Instead there is feedback from the error between the input and output of the comparator. By virtue of Bode's sensitivity integral, it is then possible to spectrally shape the error, dependent on the feedback filter used. As the comparator is a discontinuous non-linearity (no Lipshitz constant), there does currently not exist any theory with regards to the stability of DSM, and the successful design is entirely dependent on simulations. It works most of the time because in most cases you have an actuating signal to the comparator that causes some sort of averaging, causing the discontinuity to become effectively continuous. The DSM does go unstable and produce weird results in some pathological cases. So in general, it does not work, it just works most of the time. In DSM the distortion is also entirely deterministic, although it can be chaotic. It is impossible to de-correlate the distortion in DSM from the input.
I do not know of any methodology other than (Monte-Carlo) simulation for reliably determining an approximate SINAD in either case.
I've added a couple of figures comparing PWM and DSM with 1-bit truncation.
Time-series after reconstruction:
Power Spectral Densities (up to 2 kHz):
For DSM a 5th order feedback-filter was used, oversampling rate of 128, and sample rate of 1 MHz. With this it is possible to achieve an ENOB in excess of 17 bit with a 2-kHz baseband. For audio this would have to be 20 kHz, hence a sampling rate of 10 MHz. The PWM carrier was at 4 kHz. Most of the distortion (the noise-floor) stem from time discretization, not having introduced proper pre-distortion needed for discrete-time PWM, and will not represent ideal (natural) PWM very well. I will see if I can update this later. Using proper pre-distortion you should see a SINAD in excess of 100 dBc (ENOB > 16 bit). A 1-bit truncated signal can have an ENOB less than one since the ENOB is based on the uniform quantization error distribution assumption with is only approximately valid for 7-bit quantization or higher.
Why not just use a 1-bit DSM re-quantization (e.g. DSD) which is much more linear than a PCM-PWM conversion? The answer to this is switching losses. With high OSR and high and irregular switching activity, the DSM bit-stream in its basic form is not very suitable to drive high power Class-D amplifiers. Likewise, PWM due to its two-level representation and high jitter susceptibility (the need for good switching time resolution) is not very suitable for high resolution small signal DACs.
It might also be noted, that it is quite common for Class-D amplifiers to re-quantize to e.g. 8-bit using DSM before producing the PWM signal, instead of using complicated pre-distortion and non-uniform switching techniques, as a low word length reduces the required timing-accuracy significantly.
Some literature to check out:
Direct Digital Pulse Width Modulation for Class D Amplifiers
Signal processing for high resolution pulse width modulation based digital-to-analogue conversion
Digital-to-Analog Conversion in High Resolution Audio
Understanding Delta-Sigma Data Converters
Oversampling Delta-Sigma Data Converters
Lee's rule extended (on DSM stability and design)
