In order to downsample a signal sampled at 48KHz, I implemented an anti-aliasing filter. An Elliptic LPF with a cutoff at 16KHz and order of 10. Everything looks OK until the input to this filter is a clipped signal, say a clipped sinewave. In that case, the output of the filter has a larger magnitude than the input (less power, but a higher max). Is there any theory to explain this behavior?
Once overshoot the threshold of clipping, the low-pass filter tries to smooth out the sharp changes. But in doing so, it introduces some oscillations (Gibbs phenomenon, as mentioned by Jdip in the comment) or ripples near the transition points. These oscillations cause the filtered signal to have a higher maximum magnitude (peak) than the original input signal, even though the overall power of the signal is reduced due to the low-pass filtering.