I am trying to use Hilbert transform to extract the envelope of a residual signal. After implementing the Hilbert transform, I find that envelope jumps very high at its boundaries. May I ask the reason why this happens and how to alleviate such edging effect? Any help or suggestions are greatly appreciated!
The effect can be alleviated with appropriate padding, which imposes a 'statistical prior' (i.e. assumption).
No padding is equivalent to periodic padding, meaning signal's right joins its left, and vice versa; this is often unsound, as what happens a minute later may be completely irrelevant to $t=0$. There isn't a one-best-fits-all, but my recommendation for the general case is
reflect padding, which continues the signal from its local features. If the goal is instantaneous frequency/amplitude localization,
zero might work better. -- related post
On your signal:
Reflect works best when the derivative at boundary is continuous (i.e. padded part joins original "smoothly").
Why the edge effects?
Hilbert transform is a convolution; the full bag of problems that comes with it is hence applicable (and so are the remedies). In essence, "left draws from right", and vice versa, and to counter we extend "left" and "right" such that contributions from the 'other side' are negligible. How to extend is the topic of padding -- partly covered here.
"Because it's discrete" (rather than continuous) is a common explanation, but flawed: the main factor is absence of information near boundaries. The "continuous explanation" happens to be correct since continuous-time signals are usually defined perfectly over all of time - i.e. no missing info: if we zero the same continuous signal outside some interval, edge effects come pouring.
But there's a fundamental limitation: the $1/t$ Hilbert kernel is divergent in L1 sense (integral from any point to infinity diverges), so convolution contributions never drop below machine epsilon (though if we're concerned with an L2 measure, $1/t^2$ does converge). Thus we can't be "perfect" in discrete - but the effect can be made acceptably small.
Appropriate padding is an important topic in time-frequency analysis: STFT, CWT. For latter especially we use analytic wavelets, and a Hilbert-transformed signal is analytic - so the problems are analogous.
It comes down to picking the "best guess" for the signal's continuation, which is best done with knowledge of the source. If the goal is merely a smooth Hilbert envelope, this can always be achieved with an appropriate local interpolation - e.g. FFT-based or boundary wavelets.
These concepts apply to synthetic and "real" signals alike. I'll close by noting that if the goal is amplitude extraction, then CWT and synchrosqueezing are far preferred to a direct Hilbert envelope, as latter is limited to single-component signals (which vast majority of real-world signals are not) and former are robust.
Available at Github.