A real-valued system that doesn't distort the shape of the input signal must have the following input-output relation:
with arbitrary real-valued constants $A>0$ and $t_0$.
In the frequency domain, Eq. $(1)$ corresponds to
Consequently, the corresponding system is an LTI system ...
To downsample an FFT result for both magnitude and phase, it may work best to do an FFTShift before the FFT, then downsample the real component vector and then the imaginary component vector separately, before taking the atan2() of them to estimate downsampled phase.