I'm reading from Communication Systems, 5th Edition by Simon Haykin. At some point we derived that
Which is how the effect due to AWGN is minimized. Meanwhile, to minimize ISI we need $P(f)$ to be raised cosine spectrum which is attainable under the choices of $H(f)=\sqrt {P(f)}$ and $G(f)=\sqrt {P(f)}$.
My problem is that under these chis choice of $H(f)$ it no longer seems like the optimal version that minimized the probability of error due to noise. So either one of these is true:
1 - The new choice of $H(f)$ somehow as well minimizes the probability of error due to noise.
2 - It doesn't. We choose $H(f)$ based on whether we're interested in minimizing noise or ISI.
In case, it's 1 then I have no idea why that would be the case.
The book's notation is that $P(f)$ is the pulse in frequency after passing through the channel and the receive filter. $G(f)$ is the transmit filter (pulse shape) and $H(f)$ is the receive filter.