First of all, you're understating your problem. The error probability depends not only on your noise power (density), but also on the actual distribution of noise. Also, you need to realize that bit error probability is not the same as symbol error probability, not even proportional, so at best, the formula you pasted (from an uncited source) is only an approximation for the special case of symbols being Gray-coded, and even then only for noise powers that are relatively small. I find it highly unlikely a general formula (for variable $M$, which you also didn't define, but I assume is the $M$ in M-PSK) is that simple.
I don't know the correct formula by heart, either, so me looking it up and check that my scenario makes exactly the same assumptions as the author of the thing I'm reading is as good as you doing it, with the difference that you actually know your assumptions, whereas I can only vaguely infer from your question statement what they might be.
One of the assumptions that I have infer is that you're considering additive, rotationally symmetric complex Gaussian noise to equiprobable symbols. And that you're assuming Gray coding. And that you're using a Maximum Likelihood detector.
It's likely you'll have to use the $Q$-function within your derivation at some point, but it's not as trivial as in the BPSK or QPSK case, since the decision boundaries aren't orthogonal to each other, so looking at the noise in I and Q separately simply doesn't work!
This is one of the classical problems with which stochastics tortures students (hint: that means there's a bazillion of resources out there). I'll outline the solution:
- As in your previous question, you have to define a noise model before you start calculating anything related to noise. No shortcut there. Write it down.
- Then, you calculate the probability density of your received value $R$ given that you've transmitted some specific symbol $s_T$ from the symbol alphabet $\mathcal S$: $f_{R|S_T}(r|s)$. In the smooth-pdf additive noise case, this boils down to the noise density $f_N$ shifted by the transmit symbol: $f_{R|S_T}(r|s)=f_N(r-s)$. Be careful, $R$ and $S_T$ are complex random variables w.l.o.g.
- Then, you need a mathematical model of your detector, which maps received values $r$ to symbols $S_s\in\mathcal S$, is some function $S_r = d(s): \mathbb C \mapsto \mathcal S$.
- Now you apply that model to your received symbols; this is often a coordinate transform $g(\cdot)$. For example, in PSK, you need to convert the received complex value from representation as real and imaginary part to polar coordinates. You will get a transformed random variable $\tilde R$, whose PDF you need to derive from $f_{R|S_T}$ and $g(R)$. If your stochastics for engineers course was anything like mine, you'll remember it here: Inverse of the mapping, Jacobi determinant, marginal density, done.
- Now, based on that, it should be possible to derive the probability that the transmitted symbol is detected as each of the possible symbols (including itself, i.e. the correct one). Summing over the probability of each symbol multiplied by the bits that differ between that symbol's bit mapping and the transmitted symbol's bits gives you the bit error probability given that specific transmitted symbol. Do that for all transmit symbols. Often, you'll notice that the symmetry of the decision boundaries and the noise properties allow you to only calculate one (like in PSK in AGN) or a few (like in QAM constellations in AGN) probabilities instead of all.
- Weigh each transmit symbol bit error probability with the transmit symbol's probability, sum, and get the overall bit error probability.
Notes
- I'm sorry to pester you with 1., but you need to really really be more stringent about writing down models. I can't look into your head and tell myself what noise model you're assuming!
- Point 2 is a dangerous game to play for many people: they infer the Maximum Likelihood decider is always based on the symbol with minimum distance to the received complex number. That's only the case for symmetric, concave noise PDFs.
- If you didn't have my "stochastics for engineers" course, "random variable transformation PDF" is what you want to look up
- Gray coding comes into play at 5., where a common simplification / approximation is to assume that the symbol errors that bring you into the neighboring wrong symbols are so much more likely than the symbol errors that take you multiple symbols away from what was transmitted, that you can neglect them altogether. Then, you can only consider the probability of the neighbors, which, due to Gray coding, have only a single bit error. However that's a strong assumption, especially if your noise isn't Gaussian or your SNR isn't good. Really note, whenever you're doing that simplification, that you're doing it, and why you think it's feasible.
- Here's where you'll notice that while in PSK, all symbols have the same bit error probability, that's not the case for other constellations. Thus, you'll find modern transmission systems where the channel encoder's output transmit symbols aren't equally probable by design. (this gets complicated, because not making the symbols in a QAM equally probable changes the average transmit power, so you need to rescale the transmission, but then your E_b/N_0 chances, and you need to re-optimize and then your number crunching cluster goes brrrt and gets warm.)
