A DCT is roughly equivalent to a DFT of a vector after it is doubled by mirroring by a symmetric reflection. This produces FFT input that does not have a discontinuity either in the middle or circularly. A DST is roughly equivalent to a DFT after an antisymmetric mirrored extension. This antisymmetric addition can easily result in discontinuities both in the middle and around the circle. Discontinuities are represented by energy in the high frequency bins in the FFT results. These high frequency artifacts are usually undesirable when using a transform for compression.
Since the DCT does not have this potential high frequency content due to circular discontinuities (as does a DST or FFT), the same total energy is thus spread lower in frequency, which potentially allows for greater compression of the high frequency DCT bins, while remaining below some visible threshold.