Let's understand why this definition is complicated to begin with.
Energy of signal is defined as $E_s := \langle x(t),x(t) \rangle$ which could equate to, in the continuous case as $ \Rightarrow \int |x(t)|^2 dx$ and in the discrete case as $ \Rightarrow \sum |x[n]|^2 $. Here, the energy is taken as an inner product $\langle , \rangle$ of the signal $x(t)$ on itself. An inner product, being a linear algebra concept, hints at the notion of "a signal as a vector". This little hint helps a ton with developing an understanding for spectral analysis or stochastic processes later in studies. Signals, as in spectral theory, could also be defined on complex values $(s=\alpha + i \omega)$. In such case, the formula above fails and must accommodate a conjugate as well $\int x(t) \overline{x(t)} dx$.
Further, it has been explicitly stated in signal-processing books (such as this ) that a signal is, generally, not (always) a physical concept. Thus, interpretation of the concept of "energy" in terms of signals could be misleading. It is considered best to comprehend "signal-energy" as "weight of a signal". Even in the most assumed case, where an EM wave acts as a signal; the definition of signal-energy doesn't match the exact semantic of energy as in physics.