Are there images that are better suited to testing compression quality than others?
You are exactly right. A good compression algorithm is one that performs well on average considering all those types of images. In reality, a database of different types of image is used to evaluate a particular compression techniques.
Maybe an image of diagonal lines of specific colors at a specific angle would be more likely to show artifacts of algorithms / libraries than others?
Good test image makes sense when it comes to a specific feature of image we are interested to look at. For instance, assume we have an image compression based on discrete wavelet transform. Now we are interested to see how much gain can be achieved in terms of compression ratio if we replace the ordinary wavelet we use with a directional wavelet. In such case, an image full of straight lines with various directions can reveal the differences. An example test image of this type is Barbara. Or we may also use an image with a lot of smooth curves to see how the directional wavelet can decompose the curves. Lena image has more curves in this case.
High contrast colors? High contrast black and white?
Let's say we want to study the impact of smoothness of the wavelets we select on the compression ratio. So we need two groups of test image: highly edgy images with a lot of discontinuities where the non-smooth wavelet will perform better, and low-contrast images where the smooth wavelets are more appropriate.
Or assume the cross-channel compression capability (e.g. dependencies between chroma and luma) is the subject of test. So we need to test images that have (or don't have) significant dependency across these channels.
Checkerboards? Certain kinds of photos?
Sometimes block artifacts need to be tested considering different block sizes or at different scales. Or sometimes the compression algorithm is designed for a specific purpose such as remote sensing or satellite images. So we need to exclude e.g. portraits and focus on aerial test images. Or similarly for medical imaging...
Maybe there is a set of standard images to use to compare?
There are some known test images such as Lena, Barbara, Cameraman, etc. But they are mostly for academic papers, since their metrics (like PSNR at different BPPs) are quite widely known and people can evaluate the proposed techniques by looking at the reported metrics. Something like: "Lena (512) with 5-level cdf 9/7 with SPIHT at 0.5 BPP and arithmetic coding can achieve around 37.2 dB PSNR".
However, in practice (for design of a codec) a huge database is used. You can see for example something like here, here, or here.
Is it mostly subjective or are there objective criteria for comparing?
So based on the above discussion, we can see both subjective and objective tests apply, depending on what we are interested in. The compression performance of the image coded in general, or are we just want to test a specific idea or a particular part of the whole compression pipeline.