So, just to avoid naming confusion: I take it you get sample packets from your SDR – the fact that your $\frac\pi4$-QPSK communication might be packetized can't already be seen in these.
As a software architect, I do believe that a lot of good things come from getting an overview, if possible something like a simplifying block diagram. That allows us to reason about what needs to be done. Same for other complex problems like yours!
I can't give you a full intro on how to write that receiver – to be honest, our students wouldn't be ready to do that after the digital comms basic course, and that's 4 hours a week of education for a semester and a three hour test. And that after uni has primed them with three semesters of math basically leading up to that lecture. You're approaching a complex problem, and I deeply respect the effort you're putting in!
I hope, however, that me giving your problem a bit of structure allows you to divide it down to chunks that you can approach individually, and then ask awesome, specific, questions on the way to a solution. Cheers!
ASCII Block Diagram
So, the typical receiver architecture for a single-carrier transmission like yours does look something like the following
Waves +------------+ IQ samples +-----------------+
+----->+ SDR device +------------------->+ Synchronization | · Frequency
+------------+ (complex numbers +--------+--------+ · Phase
representing the | · Timing
RF signal, but |
shifted to 0 Hz) |
data +-------------+ +--------v--------+
bits | Channel | | Constellation |
<-------+ Decoding +<------------------+ de-mapping |
The transmitter added Converts points from
error-correction bits the complex plane back
to the data bits. to the bits that
Time for us to take determined which points
these, and correct were sent – i.e. the
the errors that "inverse" of the pi/4-
occurred. QPSK, if you will
SDR -IQ baseband samples-> Synchronization -> Constellation-Demapping -> Channel decoding -> data
So, your constellation is $\frac\pi4$-QPSK, so that's the kind of constellation that we need to revert to get the 2-bit groups back that were converted into the symbols.
To clarify: this isn't necessarily complete – I willfully neglected to add an equalizer to the system. I simply hope you don't need one. I can't tell however from your problem description. I also omitted things you'd typically find in textbook receiver diagrams like this one like the matched filter that matches the pulse shape, as that is often part of the timing synchronization algorithm, or the constellation de-mapping, depending on where you'd logically draw lines
But first, there's "Synchronization"; what does that entail?
- Frequency synchronization: Since the transmitter doesn't share the same oscillator as the receiver, it's very likely they have a difference in frequency. So, what your transmitter put at some frequency ends up on that frequency plus a fixed offset. The receiver needs to estimate that offset and correct it.
- Phase synchronization: Don't know how deep into the channel theory you've got, but the channel simply rotates the complex numbers you sent – and by a random phase offset! So, we need to estimate this and correct it, which means "rotating it back". For QPSK that's relatively easy (see what Dilip linked to above), but for \$\frac\pi4$-QPSK not so much.
- Timing synchronization: So you send Symbol1, Symbol2 … and so on, and because physics, there's going to be some transition of signal from one symbol "instant" to the next. (We call the filter that defines that transition Pulse Shaping.) Your receiver needs to figure out exactly when to "look" at the signal to get the actual symbol value, and not just the transition between symbols.
Note that the order of synchronizations isn't defined generally; the order I picked here is what I consider to be "most textbook & most likely to be applicable to Doug's problem", but it depends.
Or Decider, or whatever. It decides on the bits that were sent.
So, you know now that your signal is
- exactly centered around 0Hz (freq. sync.),
- probably rotated like it was sent (or just rotated by multiples of a
quarter circle, so that the bits could be wrong that you get out, but
you could trivially correct that later on by doing a Look Up in a
table for all four possible rotations), (phase sync), and
- you know exactly when to sample the signal to get the actual symbol value.
Time to take that symbol value and get the bits out of it!
You've probably learned that a $\frac\pi4$-QPSK is just a QPSK constellation, just that every other symbol gets rotated by $\frac18$ of a full circle. So, what you need is something to take these constellation points (which represent the bits the transmitter sent) and convert them back to the two-bit-groups that were used to select which of the four points of the QPSK were sent.
The problem here (as everywhere above and below) is that things are noisy – and that means that you don't actually receive what was sent, even if all the synchronization worked perfectly (it won't, ever, it's what drives us communication engineers mad). You get what was sent plus noise. Sometimes you'll make a wrong bit decision based on that. You can't avoid that. It's in communication technology's nature to be noisy.
Certainly something you'd do after you get bits out of your constellation de-mapper; use the forward error correction coding applied in the transmitter to get your original bits back. Sometimes, you'd not even know what your data and what your error correction bits were prior to decoding. Obviously, fully defined by the error correction code the transmitter used. If that is not known and things don't strikingly lot look like one of the "standard choices", then you've got to infer the coding method and the parameters – that can be extremely close to cryptanalysis!
If your transmitter transmits in packets, you'll need some way to detect the start and end of these – but how to do that depends very much on the packets' format, and how much effort you want to put in, and how important it is to detect every packet vs not detecting noise that is not actually packets. Since the place where you put that can't be determined without that, it's not in the block diagram.
You said you couldn't connect the blocks in GNU Radio – not quite sure whether that means you "mechanically" make the connections, or whether you don't know what to connect with which.
For the first problem: Try running a Virtual Machine (e.g. in virtual box) running a Linux system image that already contains GNU Radio. My Image might be what you'd like.
For the second problem: Maybe GNU Radio's Guided Tutorials is what you need.