View your frequency response after your low pass filter on a dB scale to better show the limitations of your filter.
Use a multiband filter with the least squares algorithm for an optimized rejection filter for zero-fill interpolation. This will concentrate the rejection to be specifically where the images are that need to be removed.
Given your original signal extends to the band edge, you will have significant challenges interpolating this with zero insert and filtering since the filter complexity is driven by the width of your transition band. I give details below on such an interpolator filter design - notice specifically what defines the signal you want to pass and the signal you want to reject. To simplify that and make this approach feasible, consider first low pass filtering your waveform to reduce the high frequency components while still being acceptable for what you want to have in the time domain. Then you can interpolate using the zero-insert and multi-band filter design approach. The low pass step is not necessary (as the interpolate filter can be designed to do the same thing) but helpful as you will be able to observe your signal prior to further processing to access if the time domain is still acceptable.
To the extent you can pass your signal within the pass band without any distortion, and completely eliminate the images, you will achieve perfect interpolation (this is not possible without an infinitely long filter and infinite delay so you can only approximate this-- given enough taps you can do very well).
This first slide shows how images are created through zero-insert. The digital spectrum is already periodic, zero insert just extends the sampling rate without changing the periodicity or original shape of the spectrum-- so when we interpolate by 4 for example as done here, the images that were originally around $F_s$, $2F_s$ become part of our spectrum in the first Nyquist zone at $F_s/4$ and $F_s/2$. These must be filtered out to complete the interpolation.
An optimized interpolation filter need only reject the images while not distort the passband signal of interest. A least squares (firls in MATLAB/Octave) FIR filter is an excellent choice and support multiband filters where we can concentrate the rejection where it is needed most, providing greater rejection for the same number of taps.