This must be an artifact of the implementation for the
hilbert function, in that any implementation of finite length will have a finite ripple. It doesn't appear the underlying filter using in the
hilbert function can be modified, so there are three suggestions to obtain a smoothed demodulated result:
Method 1: Implement the Hilbert using the FFT: take the FFT of the original signal, set all the negative frequencies to zero (the upper half of the FFT) and double the positive frequencies leaving bin 0 (DC) as is (and as Overlord pointed out in the comments, leave the Nyquist bin at $N/2$ as is when the FFT length $N$ is even). The IFFT will be the Hilbert Transform and taking the absolute value of this signal will be the desired envelope. (As @aconcernedcitizen pointed out in the comments, this IS the method specifically implemented by Octave using the
hilbert function from the signal package.)
Method 2: Implement the Hilbert with quadrature phase tracking filters where the filter length versus ripple can be traded.
Method 3: Use a traditional AM demodulator by multiplying the modulated signal with the coherent carrier and then low pass filtering the result.
A demonstration of Method 1 is shown below, which is a suitable solution for the OP's case:
Zooming in on the start of the signal shows the distortion limited to the start-up condition and the smooth envelope after that.
The Matlab code for this is as follows:
N = length(sig);
sig_spectrum = fft(sig);
hilbert_spectrum = zeros(1, N);
hilbert_spectrum(1) = sig_spectrum(1);
hilbert_spectrum(2: ceil(N/2) - 1) = 2 * sig_spectrum(2:ceil(N/2) - 1);
if mod(N, 2) == 0
hilbert_spectrum(N/2) = sig_spectrum(N/2);
hilbert_time = ifft(hilbert_spectrum);
envelope = abs(hilbert_time);