Two successive symbols in the demodulator are $Z_1 = (X_1,Y_1)$ and $Z_2 =(X_2,Y_2)$
where $X$ is
the output of the I branch and $Y$ the output of the Q branch of the receiver. The
hard-decision DBPSK decision device considers the question:
Is the new symbol $Z_2$ closer to the old symbol $Z_1$ or to the
negative $-Z_1$ of the old symbol?
and thus compares
$$(X_2-X_1)^2 + (Y_2-Y_1)^2 \gtrless (X_2+X_1)^2 + (Y_2+Y_1)^2$$
which can be simplified to a sign comparison on $\langle Z_1,Z_2\rangle = X_1X_2+Y_1Y_2$.
Note that this is essentially asking
Are the two vectors $Z_1$ and $Z_2$
are pointing in roughly the same direction (in which case the inner product
or dot product is positive) or in roughly opposite direction (in which case
the dot product is negative)?
A third viewpoint thinks of $Z_1$ and $Z_2$ as
complex numbers and asks
Is $\text{Re}(Z_1Z_2^*) = X_1X_2+Y_1Y_2$ positive or negative?
The soft decision decision device simply
passes on the exact value of the dot product to the soft decision decoder
which may opt to quantize dot products that are very large in magnitude
into hard decisions and continue waffling on the rest. This is what
the decision rule stated in the OP's question is, where large is taken as
exceeding $1$ in magnitude.
In DQPSK, the encoding uses one of two conventions:
the signal phase is delayed by $0, \pi/2, \pi, 3\pi/2$ according
as the dibit to be transmitted is $00, 01, 11, 10$
the signal phase is advanced by $0, \pi/2, \pi, 3\pi/2$ according
as the dibit to be transmitted is $00, 01, 11, 10$
Note that a DQPSK signal is not the sum of two DBPSK signals
modulated on phase-orthogonal carriers, but the I and Q bits
jointly affect the net carrier phase.
For demodulating a DQPSK signal, the decision device needs to ask
Which of the four symbols $Z_1,\quad jZ_1 = (-Y_1,X_1),\quad -Z_1,\quad -jZ_1 = (Y_1,-X_1)$
is $Z_2$ closest to?
Thus, in addition to the comparison
$$(X_2-X_1)^2 + (Y_2-Y_1)^2 \gtrless (X_2+X_1)^2 + (Y_2+Y_1)^2$$
it is necessary to compare
$$(X_2+Y_1)^2 + (Y_2-X_1)^2 \gtrless (X_2-Y_1)^2 + (Y_2+X_1)^2$$
which works out to looking at $\text{Im}(Z_1Z_2^*)$ in
addition to $\text{Re}(Z_1Z_2^*)$ and making the decision
according as which quantity has the largest magnitude
and the sign of the largest magnitude. The details
of how the soft-decision decoder uses the decision
statistic $Z_1Z_2^* = (\text{Re}(Z_1Z_2^*), \text{Im}(Z_1Z_2^*))$
will determine how these numbers are further
massaged.