When using the discrete cosine transform are there commonly used alternatives to quantization to decide which dct components to keep/are important? If not, how do people come up with quantization matrices (such as in JPEG), is it purely arbitrary, based on empirical properties of common images, maths, etc.
DCT coefficients dwell in a domain akin to frequency. So if you have a model adapted to the frequency domain, it could be applied somehow to DCT coefficients. Only your tags talk about compression, so let us suppose we talk about any type of processing in the DCT domain. You can compute the median energy on 10% of the lowest coefficients, and keep only those which a two times higher. This would be a form of shrinkage, related the the above median threshold. You can keep a coefficient only if the (4 or 8) surrounding coefficients have at least a tenth of its value. It is a kind of block selection.
So, in a word, what is important should be cast in a model (almost quoting @MarcusMuller). For compression of scenes viewed by humans, one uses vision models and image behavior models. Details about the behavior of DCT coefficients are provided in this answer. As you have notice, the standard table is not symmetric: one quantizes more horizontal features than vertical features. For humans, and many terrestrial mammals, the sensitivity is a little higher for vertical components. The only explanation I have heard is that we are better adapted to danger emerging from the horizon.
But what if you decide to compress zero-mean mid-frequency textures? Then the basic matrix would not be well-suited. It is more effective to design a Q matrix that preserves the mid-freq features, albeit included in the header of the JPEG file. The size increase due to the new Q matrix, required at the decoder because unknown by the JPEG standard, is compensation by the quality improvement due to a better matrix.
I have been working on satellite image compression. There, horizontal and vertical are quite relative, and we used custom Q matrices.
From what I read there are certain quantisation matrices for different applications, Adobe Photoshop has like 15 or something. The idea is that although there are error based mathematical calculations for these matrices the best way to deduce a good quantisation matrix is by simply using a human eye (in the case of JPEGs)I haven't seen any quantisation matrices that don't follow the general trend of having heavy penalties(High number) on the lower right frequencies, although some images have a large high frequency coefficients the lower frequencies are generally larger.