# Error probability for orthogonal signaling with unequal symbol probabilities

I'm trying to derive the equation of error probability in orthogonal signaling. Here what I've tried:

Let $$\vec{s_1} = (\sqrt{E_b} , 0)$$ and $$\vec{s_2} = (0, \sqrt{E_b} )$$ be vector representation of $$s_1$$ and $$s_2$$. We know that $$f(\vec{y}|\vec{s_m}) = (\frac{1}{\sqrt{\pi N_0}})^2e^{-\frac{|\vec{y} - \vec{s_m}|^2}{N_0}} , \ m = 1,2$$ According to the Bayes' law, we can write $$P(\vec{s_m}|\vec{y}) = \frac{f(\vec{y}|\vec{s_m})P(\vec{s_m})}{f(\vec{y})}, m = 1,2$$Let $$P(\vec{s_1}) = p_1$$ and $$P(\vec{s_1}) = p_2$$. Using MAP rule, we want to find $$m$$ such that $$P(\vec{s_m}|\vec{y})$$ be maximized. So $$\vec{s_1}$$ is chosen if $$P(\vec{s_1}|\vec{y}) \gt P(\vec{s_2}|\vec{y})$$ and otherwise $$\vec{s_2}$$. After simplification we get $$\frac{N_0}{\sqrt{E_b}} \ln(\frac{p_1}{p_2}) \gt -2(y_1 - y_2)$$ Let $$a = \frac{-N_0 \ln(\frac{p_1}{p_2})}{2\sqrt{E_b}}$$ and note that $$y_1 - y_2$$ is a Gaussian random variable with mean $$\sqrt{E_b}$$ and variance $$\sigma^2 = N_0$$. So the probability of error is $$P_e = p_1\int_{-\infty}^a \frac{1}{\sqrt{2\pi N_0}}e^{-\frac{(z-\sqrt{E_b})^2}{2N_0}}dz + p_2\int_{a}^{+\infty} \frac{1}{\sqrt{2\pi N_0}}e^{-\frac{(z-\sqrt{E_b})^2}{2N_0}}dz$$ which simplifies to $$P_e = p_1\int_{-\infty}^{\frac{a - \sqrt{E_b}}{\sqrt{N_0}}} \frac{1}{\sqrt{2\pi}}e^{-\frac{u^2}{2}}du + p_2\int_{\frac{a - \sqrt{E_b}}{\sqrt{N_0}}}^{+\infty} \frac{1}{\sqrt{2\pi}}e^{-\frac{u^2}{2}}du = p_1Q(\frac{\sqrt{E_b} - a}{\sqrt{N_0}}) + p_2Q(\frac{a - \sqrt{E_b} }{\sqrt{N_0}})$$ Is this result correct? I couldn't find a source which includes error probability for the orthogonal signaling when prior probabilities are unequal.

Edit: Thanks to Dilip Sarwate's comment, the second exponent should be $$-\frac{(z+\sqrt{E_b})^2}{2N_0}$$ So the final answer is $$P_e = p_1Q(\frac{\sqrt{E_b} - a}{\sqrt{N_0}}) + p_2Q(\frac{\sqrt{E_b} + a }{\sqrt{N_0}})$$ Where $$a = \frac{-N_0 \ln(\frac{p_1}{p_2})}{2\sqrt{E_b}}$$. We can simplify this formula and get $$P_e = p_1Q(\sqrt{\frac{E_b}{N_0}} - \frac{\sqrt{N_0}}{2\sqrt{E_b}}\ln(\frac{p_2}{p_1})) + p_2Q(\sqrt{\frac{E_b}{N_0}} +\frac{\sqrt{N_0}}{2\sqrt{E_b}}\ln(\frac{p_2}{p_1}))$$ which agrees with Dilip Sarwate's answer.

• This is not quite right: the devil is in the details! Note that $y_1-y_2 \sim \mathcal N(E_b, N0)$ has when $x_1$ is the transmitted signal, but $y_1-y_2 \sim \mathcal N(-E_b, N0)$ has when $x_2$ is the transmitted signal, and you should use the appropriate densities when calculating $P_{e,1}$ and $P_{e,2}$. Also, I think that $a$ should be $\dfrac{-1}{2}\sqrt{\dfrac{N_0}{E_b}}\ln\left(\dfrac{p_1}{p_2}\right)$ May 12, 2021 at 20:10
• @DilipSarwate Thanks, you're right. That was the main mistake in my answer but I think the formula for $a$ is correct and now our answers are identical. May 12, 2021 at 20:40

I haven't checked the details fully but the OP's revised solution is correct. Once upon a time, I had posted a to a class webpage from 1998 where I had some lecture notes giving the answer $$P_e = \pi_0 Q\left(\mathsf{SNR} -(2\cdot\mathsf{SNR})^{-1}\ln\left(\frac{\pi_1}{\pi_0}\right)\right) + \pi_1 Q\left(\mathsf{SNR} +(2\cdot\mathsf{SNR})^{-1}\ln\left(\frac{\pi_1}{\pi_0}\right)\right) \tag{1}$$ where $$\mathsf{SNR} = \dfrac{\Vert s_0-s_1\Vert}{\sqrt{2N_0}} = \sqrt{\dfrac{E_b}{N_0}}$$ in this case. Note that when $$\pi_1 > \pi_0$$, $$\ln\left(\frac{\pi_1}{\pi_0}\right) > 0$$, and so $$Q\left(\mathsf{SNR} +(2\cdot\mathsf{SNR})^{-1}\ln\left(\frac{\pi_1}{\pi_0}\right)\right) < Q\left(\mathsf{SNR} -(2\cdot\mathsf{SNR})^{-1}\ln\left(\frac{\pi_1}{\pi_0}\right)\right),$$ that is, in the weighted sum of two error probabilities in $$(1)$$, the larger weight $$\pi_1$$ weights the smaller error probability, which is exactly how it should be.
• Thanks for the answer. You are right about the threshold and I've corrected it. I think in your answer SNR should be $\sqrt{\frac{E_b}{N_0}}$. Also I think it should be $(2(SNR))^{-1}$ instead of $2(SNR)^{-1}$. May 12, 2021 at 8:33
• I'm not sure whether both of probabilities should have exponent $-\frac{(z-\sqrt{E_b})^2}{2N_0}$ or one of them $-\frac{(z-\sqrt{E_b})^2}{2N_0}$ and the other one $-\frac{(z+\sqrt{E_b})^2}{2N_0}$ May 12, 2021 at 8:37
• @S.H.W With regard to your second comment, note that $y_1-y_2 \sim \mathcal N(E_b, N_0)$ when $x_1$ is the transmitted signal, but $y_1-y_2 \sim \mathcal N(-E_b, N_0)$ when $x_2$ is the transmitted signal, and you should use the appropriate densities when calculating $P_{e,1}$ and $P_{e,2}$. May 12, 2021 at 20:14