This must be an artifact of the implementation for the
hilbert function (which to be clear is NOT the Hilbert Transform, but the analytic signal which consists of the signal plus it's Hilbert transform on the imaginary axis), in that any implementation of finite length will have a finite ripple. It doesn't appear the underlying filter using in the
hilbert function can be modified, so there are three suggestions to obtain a smoothed demodulated result:
Method 1: Determine the analytic signal using the FFT: take the FFT of the original signal, set all the negative frequencies to zero (the upper half of the FFT) and double the positive frequencies leaving bin 0 (DC) as is (and as Overlord pointed out in the comments, leave the Nyquist bin at $N/2$ as is when the FFT length $N$ is even). The IFFT will be the analytic signal and taking the absolute value of this signal will be the desired envelope. (As @aconcernedcitizen pointed out in the comments, this IS the method specifically implemented by Octave using the
hilbert function from the signal package.)
Method 2: Determine the analytic signal with quadrature phase tracking filters where the filter length versus ripple can be traded.
Method 3: Use a traditional AM demodulator by multiplying the modulated signal with the coherent carrier and then low pass filtering the result.
A demonstration of Method 1 is shown below, which is a suitable solution for the OP's case:
Zooming in on the start of the signal shows the distortion limited to the start-up condition and the smooth envelope after that.
The Matlab code for this is as follows:
N = length(sig);
sig_spectrum = fft(sig);
hilbert_spectrum = zeros(1, N);
hilbert_spectrum(1) = sig_spectrum(1);
hilbert_spectrum(2: ceil(N/2) - 1) = 2 * sig_spectrum(2:ceil(N/2) - 1);
if mod(N, 2) == 0
hilbert_spectrum(N/2) = sig_spectrum(N/2);
hilbert_time = ifft(hilbert_spectrum);
envelope = abs(hilbert_time);