There are many varied impairments in a transceiver that we could call "noise", (see my graphic below), and ultimately I resort to simulations with good noise and impairment models for each effect, that is very dependent on the receiver type and operation.
This question is to broad to give a specific answer. However, let me describe my treatment of colored noise in the receiver in case this adds some insight. In most cases my approach is to first have a thorough understanding of the frequency domain characteristics of the noise process involved, and then to evaluate the equivalent frequency response of the receiver translated to baseband prior to final detection, and integrate the total noise across that bandwidth to get the total noise power in band. The signal bandwidth will have a low frequency limit (typically set by carrier and timing recovery loops, or the duration in time that a packet is observed-- you need to observe a signal for infinite duration to go down to true DC so it will always be a high pass!) and then an upper frequency limit set by the symbol rate if the receiver is optimized (typically a percentage over R/2 where R is the symbol rate and the percentage is based on the pulse shaping factor), or the actual pre-detection bandwidth if not optimized.
Here is an example with phase noise specifically. It is usually clear that the receiver will filter out the higher frequency components beyond the channel bandwidth, but often there is confusion with how to handle a low frequency limit, and knowing that the phase noise keeps increasing as we approach DC, we could get very different answers depending on what low frequency limit we choose!
If there is a carrier tracking loop involved, the low frequency limit is often established by that tracking loop, since it works by tracking out phase rotation (so slow rotations or movements of phase within the loop BW of this tracking loop would be tracked out). Depending on the loop this could be a first order high pass response as I show in the figure below.
It is the resulting phase noise density after being multiplied by this "filter" of the receiver that I would integrate (and double as half of the band is shown) to assess the phase noise impact on my receiver SNR.
The key point in this example is that with a thorough understanding of the frequency domain characteristics of the noise (the phase noise response, and beyond the scope of this example, how it maps to my receiver channel of interest), and a clear understanding of the receiver both in the low frequency limit and the upper frequency limit of the baseband channel with all operations prior to final detection; one can integrate the resulting colored noise density over the band of interest to establish the total noise in the receiver prior to detection. I also do time domain histograms to capture and access the effects of different noise distributions, but in most case the combination of multiple independent noise sources makes the Gaussian estimate reasonable.