Unfortunately you did not mention what your channel is so I assume that merely AWGN is present. Furthermore I assume symbol-by-symbol detection is desired since a decoder for the Hammingcode is following later. So a single symbol $x$ is received as $y = x + n$ where $x \in [1,j,-j,-1]$ and $n$ is distributed according to a zero-mean complex normal distribution, i.e. $n \to CN(0, \sigma_n^2)$ so that:
\begin{equation}
p(n) = \frac{1}{\pi \sigma_n^2} \exp\left(-\frac{n n^*}{\sigma_n^2} \right)
\end{equation}
We know the conditional density $p(y|x)$ of the received symbol $y$ if the transmitted symbol $x$ is known. It is a non zero-mean complex normal distribution, i.e. $(y|x) \to CN(x, \sigma_n^2)$ and:
\begin{equation}
p(y|x) = \frac{1}{\pi \sigma_n^2} \exp\left(-\frac{(y-x) (y-x)^*}{\sigma_n^2} \right)
\end{equation}
However, we know $y$ and want to know the likelihood of $x$, so we utilize the bayes theorem:
\begin{equation}
p(x|y) = \frac{p(y|x) \cdot p(x)}{p(y)}
\end{equation}
$p(y|x)$ is already given above, $p(x)$ is the prior density of your transmit symbols. Often the systems are designed so that all symbols have equal probability so that $p(x) = \frac{1}{N_s}$ with $N_s$ being the amount of symbols in your alphabet. It is 0.25 in your case. Ignore $p(y)$ for the moment, it will cancel out later.
Now a softbit is usually defined in terms of it's log-likelihood, so that
\begin{equation}
l_0 = \log \frac{p(b_0=0|y)}{p(b_0=1|y)}
\end{equation}
The logarithm is mainly used since the calculations become more numerically stable since very often the probabilities will be either very close to zero or very close to one and quickly reach the precision limits of floating point processors, especially when multiplying many of those probabilities together. So now how can we express $p(b_0=0|y)$ in terms of $p(x_k|y)$? Assume the following bit mapping (use your own here):
b0 b1 -> x
0 0 -> 1
0 1 -> j
1 1 -> -1
1 0 -> -j
So we can see that for example $b_0=0$ if the transmitted symbol was a $1$ or a $j$ and hence:
\begin{equation}
p(b_0=0|y) = p(x=1|y) + p(x=j|y)
\end{equation}
and similar for $p(b_0=1|y)$. So the final softbit value for $l_0$ is given by:
\begin{equation}
l_0 = \log \frac{p(x=1|y) + p(x=j|y)}{p(x=-1|y) + p(x=-j|y)}
\end{equation}
Using the bayes theorem under the assumption that the prior $p(x)$ is equal for all symbols, we finally get:
\begin{equation}
l_0 = \log \frac{p(y|x=1) + p(y|x=j)}{p(y|x=-1) + p(y|x=-j)}
\end{equation}
which can be calculated using the definition of $p(y|x)$ given above. Similarly the softbit for $l_1$ can be calculated. The softbit will be positive if the bit is more likely to be zero and negative if it is more likely to be one. So you can get a hard decision just by looking at its sign. The magnitude of the log-likelihood corresponds to how much is known about the bit. If it is large, the bit is likely to be a zero or one. If it is close to zero, there is a higher chance the bit is wrong. If it is exactly zero, nothing is known about the bits value.
Lots of optimizations can be utilized to make this code run very fast but this would be a little much for an answer here.
b0 = -imag(IQ)andb1 = -real(IQ). Is this correct? $\endgroup$