Let $s_0(t)$ and $s_1(t)$ be the two different finite-duration signals that the transmitter is using --- exactly one of these is transmitted, and the receiver's job is to determine which one -- and the assumption of finite-duration is because the receiver wants to make the decision after having examined the entire signal, and doesn't want to wait an infinite time for the transmitter to finish transmitting. Assume further that "different signals" means that the difference signal $s_d(t) = s_0(t) - s_1(t)$ has positive energy: $$\int_{-\infty}^\infty |s_d(t)|^2 \,\mathrm dt = \int_{-\infty}^\infty |s_0(t) - s_1(t)|^2 \,\mathrm dt = \mathbb E >0.$$ The receiver consists of a filter with impulse response $h(t)$ and a sampler at time $T$ after the entire signal has been received, followed by a decision device that declares "$s_0(t)$ was transmitted" or "$s_1(t)$ was transmitted" and we would like this declaration to be correct with as high a probability as possible.
The receiver input is $r(t) = s(t)+N(t)$ where $s(t)$ denotes the transmitted signal (either $s_0(t)$ or $s_1(t)$ but the receiver doesn't know which) and $\{N(t)\}$ is a white Gaussian noise process, independent of which signal $s(t)$ is, and with power spectral density $\frac{N_0}{2}$. Then, the noise process $\{\hat N(t)\}$ at the filter output is a zero-mean stationary Gaussian process with variance $$\sigma^2 = \frac{N_0}{2}\int_{-\infty}^\infty |h(t)|^2 \,\mathrm dt. \tag{1}$$
Thus, no matter what sampling instant $T$ is chosen, the noise is a $\mathcal N(0,\sigma^2)$ random variable. The receiver filter output is $$\hat r(t) = \int_{-\infty}^\infty s(t-\tau)h(\tau) \,\mathrm d\tau + \hat N(t)$$ where $s(t)$ is equally likely to be $s_0(t)$ or $s_1(t)$. For $i=0,1$, let $$\hat s_i(t) = \int_{-\infty}^\infty s_i(t-\tau)h(\tau) \,\mathrm d\tau$$ denote the signal output of the received filter when $s_i(t)$ is the transmitted signal. Thus,
$$\hat r(t) = \begin{cases}\hat s_0(t) + \hat N(t) \sim \mathcal N(\hat s_0(t),\sigma^2), &\text{when }s_0(t)~\text{is the transmitted signal},\\
\hat s_1(t) + \hat N(t)\sim \mathcal N(\hat s_1(t),\sigma^2), & \text{when } s_1(t)~\text{is the transmitted signal}
\end{cases} \tag{2}$$
The first criterion for choosing the sampling time $T$ is that at time $T$, $a_0 = \hat s_0(T)$ and $a_1 = \hat s_1(T)$ must be different numbers because if $a_0$ were to equal $a_1$, the receiver could not determine which signal was transmitted even if the noise were completely absent. Assuming without loss of generality that $a_0 > a_1$, the receiver compares $\hat r(T)$ to a threshold $\Theta$ and declares that $s_0(t)$ or $s_1(t)$ was transmitted according as $\hat r(T)$ is larger than or smaller than $\Theta$. Now, if $s_0(t)$ was actually transmitted, and so $\hat r(T) \sim \mathcal N(a_0,\sigma^2)$, the receiver's declaration is incorrect if $\hat r(T) < \Theta$, and so $P_{e,0}$, the conditional probability of error when $s_0(t)$ was transmitted, is
$$P_{e,0} = P(\hat r(T) < \Theta\mid s_0) = \Phi\left(\frac{\Theta - a_0}{\sigma}\right) = Q\left(\frac{a_0-\Theta}{\sigma}\right). \tag{3}$$
Similarly, if $s_1(t)$actually transmitted, and so $\hat r(T) \sim \mathcal N(a_1,\sigma^2)$, the receiver's declaration is incorrect if $\hat r(T) > \Theta$, and so $P_{e,1}$, the conditional probability of error when $s_1(t)$ was transmitted, is
$$P_{e,1} = P(\hat r(T) > \Theta\mid s_1) = Q\left(\frac{\Theta - a_1}{\sigma}\right).\tag{4}$$
Assuming $s_0(t)$ and $s_1(t)$ are equally likely the be transmitted, the average error probability is
$$P_e = \left.\left.\frac 12 \right[Q\left(\frac{a_0 -\Theta}{\sigma}\right) + Q\left(\frac{\Theta-a_1}{\sigma}\right)\right]\tag{5}$$
and this average error probability is minimized if we choose $\Theta = \dfrac{a_0 + a_1}{2}$. The minimum value of $P_e$ is $$\min P_e = Q\left(\frac{a_0 - a_1}{2\sigma}\right)\tag{6}$$
exactly as promised by the OP's textbook. Note that Eq. $(6)$ also implicitly gives the second criterion for choosing the sampling time $T$; Choose $$T =
\max_t |\hat s_0(t) - \hat s_1(t)|$$ (and if it so happens that $\hat s_0(T) < \hat s_1(T)$, use an inverter to effectively change the impulse response of the receiver filter from $h(t)$ to $-h(t)$ ....)
Notice that all the above was for an arbitrary linear filter with impulse response $h(t)$. Can clever choice of linear filter give an even smaller probability of error? The answer is Yes, but note that simply increasing the gain of the filter doesn't help in the least because both signal and noise get amplified and their ratio, which appears in the argument of $Q(\cdot)$ in , is unchanged. The optimum choice of linear filter, called a matched filter, can be determined by writing the arguments of $Q\cdot)$ in $(6)$ in terms of integrals and then applying the Cauchy-Schwarz inequality using the method described in detail in this answer of mine. Specifically,
\begin{align}
a_0-a_1&= \hat s_0(T) - \hat s_1(T)\\
&= \int_{-\infty}^\infty s_0(T-\tau)h(\tau) \,\mathrm d\tau - \int_{-\infty}^\infty s_1(T-\tau)h(\tau) \,\mathrm d\tau\\
&= \int_{-\infty}^\infty [s_0(T-\tau) - s_1(T-\tau)]h(\tau) \,\mathrm d\tau\\
&= \int_{-\infty}^\infty s_d(T-\tau)h(\tau) \,\mathrm d\tau\\
&\leq \sqrt{\int_{-\infty}^\infty |s_d(T-\tau)|^2 \,\mathrm d\tau}\sqrt{\int_{-\infty}^\infty |h(\tau)|^2 \,\mathrm d\tau} \tag{7}\\
&= \sqrt{\mathbb E}\sqrt{\int_{-\infty}^\infty |h(\tau)|^2 \,\mathrm d\tau}\tag{8}
\end{align}
and so the argument $\dfrac{a_1-a_0}{2\sigma}$ of $Q(\cdot)$ in $(6)$ has maximum value
\begin{align}\frac{a_1-a_0}{2\sigma} &= \frac{\sqrt{\mathbb E}\sqrt{\int_{-\infty}^\infty |h(\tau)|^2 \,\mathrm d\tau}}{2\sqrt{\frac{N_0}{2}\int_{-\infty}^\infty |h(\tau)|^2 \,\mathrm d\tau}} = \sqrt{\frac{\mathbb E}{2N_0}}
\end{align}
exactly when $h(t) = \lambda s_d(T-t) = \lambda (s_0(T-t)-s_1(T-t)$ for some $\lambda > 0$, that is, the impulse response of the matched filter is just the difference signal reversed in time and delayed by $T$ and with an arbitrary (positive) gain constant, as has been mentioned previously. The minimum error probability (achieved with matched filtering) is
$$P_{e,\scriptstyle{\text{matched filtering}}} = Q\left(\sqrt{\frac{\mathbb E}{2N_0}}\right).$$
Note that $\mathbb E$ is the energy of the difference signal.
Sanity Check: For antipodal signals, $s_d(t) = 2s_0(t)$ and so $\mathbb E = 4E_b$ leading to everybody's most favorite formula $P_{e,\scriptstyle{\text{antipodal}}} = Q\left(\sqrt{\dfrac{2E_b}{N_0}}\right)$. For orthogonal equal-energy signals, $\mathbb E =2E_b$ leading to everybody's second most favorite formula $P_{e,\scriptstyle{\text{orthogonal}}} = Q\left(\sqrt{\dfrac{E_b}{N_0}}\right)$. More generally, $\mathbb E = E_0 + E_1 - 2\langle s_0, s_1\rangle$ where $E_0$ and $E_1$ are the respective signal energies of $s_0(t)$ and $s_1(t)$ and $\langle s_0, s_1\rangle$ is their inner product, and we get the less-familiar $P_e = Q\left(\sqrt{\frac{E_0 + E_1 - 2\langle s_0, s_1\rangle}{2N_0}}\right)$.
