The way I learned this is there's two ways to find the probability of error. You can find the exact or as estimated probability. Exact is typically calculated for signal space where the points are in a rectangular pattern, because you're decision regions are nice and easy. You would see this rectangular pattern in a 4-QAM, 16-QAM, etc.

For estimated probability, we use the Union Bound. This states that the symbol probability of error = $P_{se|1} = \sum P_{se}($between point 1 and each other individual point$)$. Out total symbol probability of error

$P_{se} = $$\frac{\sum_{i=1}^M P_{es|i} }{M} $ where M is the number of points on our signal space ($M= 8$ in this case)

This means we're take a single point, and then with each other individual point we're going to create a new decision boundaries exactly half way between to the two and find the probability of error for just those two points. We do that for all the points, add them up, and now we have an estimated probability of error for the symbol. So let's say our noise is Gaussian, the variance of our noise $\sigma^2 = \frac{N_0}{2}$. Let's say D is the distance between two points. The probability of error between those two points $= Q(\frac{D}{2\sigma}) = Q(\frac{\sqrt2 \cdot D}{\sqrt N_0})$ So, let's look at point 000 in your examples. We can use cosine theory to find the distance between two points.

$D^2 = D_1^2 + D_2^2 -2\cdot D_1 \cdot D_2 \cdot cos(\theta)$ where $\theta$ is the angle between two points. Note $D_1$ and $D_2$ are the distances from the origin of our points, and in this case $D_1 = D_2$ because all points are the same distance from the origin.

$P_{se} = Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) + Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) +Q(\frac{\sqrt2\cdot d_1}{2 \sigma}) + Q(\frac{\sqrt2\cdot d_1}{2 \sigma}) +...$

$P_{se} = 2\cdot Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) + 2 \cdot Q(\frac{\sqrt2\cdot d_1}{2 \sigma}) +...$

We could go on, but since the $Q$ function falls off so quickly, we really only care about the point(s) closes to what we're comparing too. All the other terms are insignificant cared to that. So we end up with $P_se = 2\cdot Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma})$.

That's the symbol probability of error if we look at the point 000. We can calculate this for the other points, but it's just going to be the exact same due to symmetry. Since the Probability of error is the same for each symbol, we know the probability of error for all symbols is what we found above.

This is for symbol probability of error. To get to bit probability of error we have to look at our bit mapping. Since each point will only have two points that are closest to it, we can implement Gray coding. That means we can divide our symbol probability by $log_2(M)$ where $M =$ the number of different points we have on our signal space, which is 8 in this case. Therefore our $P_{be}$ is

$P_{be} = \frac{P_{es}}{log_2(8)} = \frac{2\cdot Q(\frac{d_1\cdot sqrt(2+sqrt(2))}{(2 \sigma)})}{3}$

The way I learned this is there's two ways to find the probability of error. You can find the exact or as estimated probability. Exact is typically calculated for signal space where the points are in a rectangular pattern, because you're decision regions are nice and easy. You would see this rectangular pattern in a 4-QAM, 16-QAM, etc.

For estimated probability, we use the Union Bound. This states that the symbol probability of error = $P_{se|1} = \sum P_{se}($between point 1 and each other individual point$)$. Out total symbol probability of error

$P_{se} = $$\frac{\sum_{i=1}^M P_{es|i} }{M} $ where M is the number of points on our signal space ($M= 8$ in this case)

This means we're take a single point, and then with each other individual point we're going to create a new decision boundaries exactly half way between to the two and find the probability of error for just those two points. We do that for all the points, add them up, and now we have an estimated probability of error for the symbol. So let's say our noise is Gaussian, the variance of our noise $\sigma^2 = \frac{N_0}{2}$. Let's say D is the distance between two points. The probability of error between those two points $= Q(\frac{D}{2\sigma}) = Q(\frac{\sqrt2 \cdot D}{\sqrt N_0})$ So, let's look at point 000 in your examples. We can use cosine theory to find the distance between two points.

$D^2 = D_1^2 + D_2^2 -2\cdot D_1 \cdot D_2 \cdot cos(\theta)$ where $\theta$ is the angle between two points. Note $D_1$ and $D_2$ are the distances from the origin of our points, and in this case $D_1 = D_2$ because all points are the same distance from the origin.

\begin{align} P_{se} &= Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) + Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) +Q(\frac{\sqrt2\cdot d_1}{2 \sigma}) + Q(\frac{\sqrt2\cdot d_1}{2 \sigma}) +\ldots\\ P_{se} &= 2\cdot Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma}) + 2 \cdot Q(\frac{\sqrt2\cdot d_1}{2 \sigma})+\ldots \end{align}

We could go on, but since the $Q$ function falls off so quickly, we really only care about the point(s) closes to what we're comparing too. All the other terms are insignificant cared to that. So we end up with $P_se = 2\cdot Q(\frac{d_1\cdot \sqrt {2+\sqrt2}}{2 \sigma})$.

That's the symbol probability of error if we look at the point 000. We can calculate this for the other points, but it's just going to be the exact same due to symmetry. Since the Probability of error is the same for each symbol, we know the probability of error for all symbols is what we found above.

This is for symbol probability of error. To get to bit probability of error we have to look at our bit mapping. Since each point will only have two points that are closest to it, we can implement Gray coding. That means we can divide our symbol probability by $log_2(M)$ where $M =$ the number of different points we have on our signal space, which is 8 in this case. Therefore our $P_{be}$ is

$P_{be} = \frac{P_{es}}{log_2(8)} = \frac{2\cdot Q(\frac{d_1\cdot sqrt(2+sqrt(2))}{(2 \sigma)})}{3}$