Musical residual noise is created when a noisy signal passes through a noise reduction algorithm. For instance, please see the paper: Esch, T; Vary, P: "Efficient Musical Noise Suppression for Speech Enhancement Systems", IEEE Intl Conf Acoustics, Speech and Signal Proc, (ICASSP): 4409-4412, 2009.
Noise reduction techniques that are relying on spectral weighting rules often generate annoying musical noise artifacts in the processed signal.
What exactly does it sound like? Are there any sample clips available (for example, a wav file before passing through a noise reduction algorithm routine and a wav file after noise reduction with musical noise) for comparison?