A common wavelet based standard is JPEG 2000 and a common DCT based standard is JPEG.
JPEG 2000 uses wavelets, but a good portion of the better compression it achieves than JPEG is due to the fact that JPEG uses a much much simpler entropy coder (JPEG does context-dependent Huffman codes and run length coding, JPEG 2000 does arithmetic coding with some extra tricks). You can gain a lot by tweaking the entropy coding stage. The particular wavelet transform you choose will make a difference as well. And how you decide to compress the coefficients makes a huge difference as well (JPEG reads its dct block coefficients in a zig-zag pattern and uses run length coding to compress long runs of zeros in the higher freq components due to the nature of the quantizers it uses).
The primary advantage of wavelets as used in JPEG 2000 versus rounding the DCT coefficients in the manner of JPEG is that it reduces blocking artifacts.
If you're not familiar with wavelets, you can look at an introductory text on wavelets, like Mallat's Wavelet tour of Signal Processing or Vetterli's new book (freely available online). Then, you can read the JPEG 2000 standard. These notes are kinda nice at a high level as well (and it has good pointers to the references).
For a quick review of JPEG, you can look at Gonzalez & Woods' Digital Image Processing (second or third edition should be fine).
