I've read that upsampling performed in digital music playback can color the sound, produce artifacts, etc. For example, an audio file ripped from a CD might be 44.1Khz/16-bit, and then upconverted to 48Khz/16-bit and played via an optical digital audio output. Audiophiles say this is bad because the signal needs to be "bit perfect" to be reproduced correctly.
Is this correct? My vague knowledge of DSP from grad school leads me to think that all upsampling should do is increase the bandwidth of the signal. But, since the source signal is bandlimited, there shouldn't be anything new in the added bandwidth. And I don't see why the process would color the sound.
What am I missing?