I have a question: Do DCT each output coefficients have to be orthogonal? Here DCT is DCT-II type. It is used by the jpeg compression algorithm.
It doesn't have much meaning to ask if the coefficients are orthogonal. If you do a 1-d DCT transform then you get a 1-d vector as output. You need at least two vectors to talk about orthogonality.
The DCT transform however, is an orthogonal transform. This means that there exists an orthogonal matrix that transforms the input data into the dct coefficients.