So, your main concern is ADSR-shaped sawtooth waves. That's awesome, because with that having a discrete spectrum (Fig. 1), we can do the math for all the periodic signals, which all have discrete spectrum, with a good example.
Figure 1: Sawtooth wave in time and frequency domain.
Notice how the triangle wave is composed of discrete tones (lines in the spectrum) – you can decompose any periodic signal into a (potentially, like here) infinite sum of cosines that you adjust in amplitude and phase. In case of the sawtooth, there's a cosine every multiple of the fundamental frequency.
Now, what we'll do is
- get a feeling for what the envelope of something like this is
- apply that to the wave
- use that knowledge to let you do the hard part.
Now, 1. is easy for us: you seem to already have found the Hilbert transform, to give you an analytic signal. What that does, simply speaking: It removes the negative frequencies in the spectrum of a real signal. Look at Fig. 1 and remember that real-valued signals always have (Hermitian) symmetric Fourier transform (can't see the mirrorring of the imaginary part of the Fourier transform in a picture that just shows the magnitude of the complex values).
This means that for every single cosine tone in your signal (i.e., every line in the spectrum!), you use the original cosine as the real part of the analytic signal, and the sine of the same phase, frequency and amplitude as imagineary part. What happens is that you map the real-valued cosine oscillations to the complex sinusoid of the same argument:
\begin{align}
s(t) &= \sum_{k} A_k\cos(2\pi f_k t + \phi_k) \mapsto \sum_{k} A_ke^{j2\pi f_k t + \phi_k}.& A_k, f_k \ge 0, \phi_k \in [0;2\pi)\tag1\label{ana}
\end{align}
Now, this is really neat for envelope detection: Your cosine (of the real argument "phase") has an absolute value that changes over time, $\cos(x) \ne \text{const.}$, but the complex sinusoid has constant enveloper:
$$\left\lvert e^{jx}\right\rvert \equiv 1 \forall x\in \mathbb R.$$
In other words, when you transform a single cosine, phase shifted and at any frequency you want, to its analytic signal, it becomes something that has a constant envelope, since
$$\left\lvert A_ke^{j2\pi f_k t + \phi_k}\right\rvert = \lvert A_k \rvert \cdot \left\lvert e^{j2\pi f_k t + \phi_k}\right\rvert = \lvert A_k \rvert\cdot\lvert 1\rvert = A_k.$$
Small problem: You need to apply the absolute value to the whole sum in $\eqref{ana}$, and the absolute of a sum is not the sum of the absolutes¹.
So, this might not be the optimum approach!
Instead, what about we simply ignore all components of the signal, but exactly the one at 0 Hz: this will have a magnitude proportional to the amplitude of the sawtooth, so it's the same as our envelope (or proportional to it!). Sadly, that doesn't work out, because in a real-world analog circuit, your sawtooth will not go from $0$ to $A_0$, but might be AC-coupled (so from some value $-a$ to $+a$, making its average zero), or have just any DC offset. And the zero-Hertz frequency component is just that, the DC component.
So, we need to settle for first cosine, at the fundamental. OK. Let's assume we have a filter that filters out everything but that frequency, we just apply that to the input sawtooth, square the result, get a DC component proportional to the square of the of the amplitude of that cosine, and a cosine component at twice the fundamental frequency. We then filter with a low-pass filter to kill the double-fundamental component, and get the pure constant DC value we wanted. Fig. 2 illustrates the computation, Fig. 3 displays the result.
Figure 2: A GNU Radio Flow Graph doing the square-based envelope detection. Flow graph File.
Output comparing the original sawtooth to the the computed amplitude. Note that adjusting the sliders will adjust the amplitude or frequency of the sawtooth, and since the computation is correct, the envelope estimate follows.
Small problem: the filter mentioned here needs to be adjusted for the frequency of the sawtooth. So, you need to know that, or need to estimate it; the latter is possible, but I'd call it a topic for a different question.
Now, having that, imagine we apply a ADSR shaper to the sawtooth before we pass it to the amplitude estimator: what is a constant red line in Fig. 3 will now become a time-variant thing with exactly the shape of the ADSR - given that the ADSR is much slower than the sawtooth.
¹ The Hilbert transform / analytic signal is still highly useful for envelope detection, but you have to work at a carrier frequency higher than the content bandwidth of the signal of interest. With the sawtooth having infinite spectrum, I don't see a trivial method of doing that.
