UPDATE:
I reviewed the OP's data and conclude from that the overall delay is exceeding the time duration of the samples given causing aliasing in the time domain to occur.
Observe the following results:
Out of personal preference, I include fftshift in my plots of the impulse response to center the plot at t=0, plotting positive and negative time as an equivalence to the processing in the frequency domain with positive and negative frequencies. Similar to getting aliasing for frequencies above $f_s/2$ (negative frequencies) we can have similar effects in the time domain for delays above $T/2$ where $T$ is the total time duration of the signal being processed.
To illustrate, below is the process to plot the impulse response for a vector starting at t=0 with sample 1 followed by 1023 zeros (matching length of the OP's example, and coded on Octave (and instead of taking the real, I plot the complex magnitude):
xaxis = -512:511
plot(xaxis, fftshift(abs(ifft(exp(j*zeros(1024,1))))))):
Same processing for impulse response of OP's signal and it is clear that the overall delay of the signal has aliased beyond half the total time duration:
plot(xaxis, fftshift(abs(ifft(exp(j*phi)))))):
In general, for every delay of one sample, the phase will vary linearly from 0 to $-2\pi$ from DC to the sampling rate. The OP's phase is varying approximately -450 radians over 1024 frequency bins. $450/(2\pi)$ corresponds to a delay of 72 samples
plot(xaxis, fftshift(abs(ifft(exp(j*zeros(1024,1))))))):
Here is the OP's original waveform along with a 72 sample delay (which would then wrap cleanly at the positive/negative frequency boundary as well, but still resulted in the time domain aliasing the OP has computed):
Here is the impulse response computed from the phase of the 72 sample delay to confirm the prior processing:
plot(xaxis, fftshift(abs(ifft(exp(j*f)))))
Conclusion: I am rather surprised that the waveform would have that much additional delay due to the variation of the phase from the best linear fit (as an analogy to frequency translation occurring (in the frequency domain) from phase rotation in time, here we clearly see evidence of time translation (in the time domain) from the same phase rotation process apparently occurring in the frequency domain. Given this, the impulse response is being translated further than the original time sample can support, and the solution would therefore require a longer time sample which should then eliminate the aliasing. (Sometimes it is easier to reverse time and frequency to understand what may be occurring; as it may be more intuitive for many to understand how this could occur with frequency domain aliasing from undersampling--- here it is the same thing except instead of increasing the sampling rate we need to increase the time duration).