My understanding is that a PLL(either in hardware or software) can recover the carrier frequency(or be used to correct the frequency offset) of a system. My question is how does this work when the transmitter is sending a pulse-shaped signal?
For a an approximate sinusoid the operation of the PLL makes sense but how does tracking work for carriers transmitting data?
As an example, suppose you're receiving a raised-cosine pulse, what do you do?
Here is a link to an example carrier tracking implementation for a QPSK and QAM system, applicable to full-response raised cosine pulse shaping. This post also has links to many other concerns with demodulating the waveform, including timing recovery and setting the parameters in the tracking loop implementation.
A x4 carrier recovery approach may be the simplest way to see how the pulse shaping effects the carrier recovery. For QPSK without pulse shaping you can recover the carrier by raising the waveform to the fourth power, and then dividing the resulting 4x carrier by four to complete the recovery. This is because phases add when you multiply, so if we have four possible phases in our modulated signal:
$$e^{(j\omega t + 0)}$$ $$e^{(j\omega t + \pi/2)}$$ $$e^{(j\omega t + \pi)}$$ $$e^{(j\omega t + 3\pi/2)}$$
Raising to the 4th power results in the following for all the cases above. $$e^{(j4\omega t + 0)}$$
If we observe our waveform over time and it instantly goes from one phase location to the other (no pulse shaping, infinite bandwidth), then the 4th power result will be a pure tone at 4x the carrier frequency with no other components.
However if we slowly transition from one phase to the other, we will still get a pure tone at 4x the carrier frequency, but there will be significant energy in sidebands close to this frequency as well. The 4x tone will still be the strongest in vicinity of this 4x signal and is easily tracked with a PLL to complete recovery. (This is evidenced by a positive mean in the baseband equivalent analytic signal). It is because this signal is the strongest that a PLL can track it (typically if the signal being tracked is +6 dB higher than the sidebands a PLL can lock to it), and it is because of the additional sideband energy that we need to the PLL to complete the recovery, effectively acting as narrow bandpass filter to provide a clean carrier.
I added some graphics below for the simpler case of BPSK with and without pulse-shaping that will hopefully provide an intuitive answer. Personally I find it much simpler to think and work with describing frequency using the complex exponential ($e^{j\omega t})$ rather than with cosines and sines, and for the same reason I avoid including an actual carrier but model at the baseband equivalent analytic signal. However I think in this case showing an actual sinusoidal carrier will more immediately get across the point I am trying to make. For the same reason, this is very illustrative in showing how the pulse shaping affects a tracking loop but it is not how I would typically do carrier recovery (see the link above for a typical software recovery loop I would implement).
This first image shows a sinusoidal carrier with an example (1 0 1) data pattern. Below the data pattern is shown the result of BPSK modulation with no pulse shaping applied (infinite bandwidth available). Observe in the third plot showing the modulated signal, that the phase is changing from 0 to $\pi$ radians as the data transitions. This abrupt change back and forth of the phase would prohibit a simple phase tracking PLL from tracking the phase: if it could even lock onto one phase state (because the data doesn't transition for some time for example), once the data transition occurs the very significant phase error that would occur would cause a simple tracking loop to break lock (unless the phase detector had 0°/180° ambiguity -- hint, hint at another recovery approach!). Another way of describing this, with a random data pattern that is 50% ones and 50% zeros, the carrier would be completely suppressed and therefore does not exist, so there is no signal at that frequency for any loop to lock onto. Notice too that in the example plot I give, the data transitions are not commensurate with the carrier. I did that purposely just to show that regardless of the data rate and where transitions occur, squaring this signal will result in a pure tone (when no pulse shaping or other bandwidth limiting is used) at exactly twice the rate of the carrier.
The fourth plot is simply the waveform in the third plot squared with no other filtering applied. This waveform is completely in phase with the original carrier, at exactly twice the frequency. In this case we could simply pass this signal into a frequency divide by two function (quite simple) and we would completely recover the carrier. No PLL is needed, but if there were any other noise sources present, a PLL would help increase the SNR of our recovered carrier in the presence of noise. However our waveform itself and the recovery process, in this case, contributes no additional noise or distortion.
What is occurring by squaring the signal is we are doubling the frequency (and the phase). This is clear from the trigonometric relationship:
$$cos(\alpha)*cos(\beta) = \frac{1}{2}cos(\alpha+\beta)+ \frac{1}{2}cos(\alpha-\beta)$$
$$cos(2\pi f_c t + \phi)cos(2\pi f_c t + \phi) = \frac{1}{2}cos(4\pi f_c t + 2 \phi)+ \frac{1}{2}cos(0)$$
$$ = \frac{1}{2}cos(4\pi f_c t + 2 \phi)+ \frac{1}{2}$$
So if the phase is only in two states: 0 (0°) and $\pi$ (180°), doubling this results in 0 and 2$\pi$ which is also 0.
Now observe what occurs in the same situation with pulse shaping as shown in the plot below. Here the only difference is a pulse shape has been added to each symbol. Notice that the result is to only add an amplitude variation (the pulse shape) to the squared signal but it has no effect on the phase. This is an important observation as to why a normal tracking loop could still track the phase in this situation, and also illustrates why the loop that is used should be able to "fly-wheel" in the absence of updates (as most tracking loops in both carrier and timing recovery). The phase in the doubled signal is in phase with the carrier at exactly twice the frequency but the amplitude is going up and down. This does of course impact the SNR of the recovered signal, as there are fewer phase error updates for tracking but unlike the case of the modulated signal itself, the phase does not abruptly change in a manner that would be disastrous for the loop to track. This also shows the importance of using a phase tracking loop in this case, which will ride through the amplitude variations and create a clean doubled carrier with no amplitude variation.
This is perhaps clearer by comparing the two final cases of the multiplied outputs directly as shown below. Here we clearly see that the pulse shaping only affects the amplitude and not the phase, hence a phase tracking loop should be able to track and recreate a cleaner replica of the doubled carrier signal, which we would then follow with a divide by two to completely recover the carrier.
The plots above should provide a satisfactory intuitive explanation in the effects of pulse shaping on carrier tracking loops, but not to suggest the favored approach to doing carrier recovery. In a digital or software implementation instead of squaring, PLL and divider, I would suggest decision directed and other similar digital Costas-Loop approaches as I have shown in the link above.
Another intuitive explanation is given by observing the above signals in the frequency domain, and noting how the carrier is suppressed prior to squaring and then recreated at twice the carrier frequency after squaring (or raising to the fourth power for QPSK, to the eight power for 8-PSK etc).
I assume that you are asking about phase synchronization in a linearly modulated (PSK, QAM, etc.) system and not particularly concerned about moderate to large frequency locking. A Frequency Locked Loop (FLL) is what you need if that is the case.
For implementing your phase synchronization system, you will find Dan's answer very helpful. He has written some of the very best posts I have read here.
Coming to your question of how the phase is recovered even when the modulation is present in a pulse shaped scenario, I call this the fundamental problem of synchronization. Usually, a PLL expects a continuous wave at its input, the phase of which can easily be locked onto. See a locking example of a PLL input and output below.
Imagine a QPSK signal input directly into a PLL designed to estimate and compensate for the phase and frequency of a sinusoid. Instead of the rectangular IQ expression you see for linear modulations, we can write a passband QAM waveform in polar form as $$ s(t) = \sum _m \sqrt{a_I^2[m] + a_Q^2[m]} \cdot p(t-mT_M) \cdot \sqrt{2} \cos \left(2\pi F_C t+ \tan^{-1} \frac{a_Q[m]}{a_I[m]}\right) $$
In this polar form, we have a single sinusoid whose amplitude is fixed for PSK modulation and phase is determined by symbols $a_I[m]$ and $a_Q[m]$. The modulated signal arriving at the Rx consists of two different phase components.
Depending on its design parameters, the PLL will try to lock onto the incoming phase within a specific time duration. However, instead of being a regular sinusoid, its phase is jumping around by $0^\circ$, $\pm 90^\circ$ or $180^\circ$ at every symbol boundary due to the modulating data, never allowing our PLL to converge. This is the fundamental problem every synchronization system needs to address and is illustrated in figure below.
Now we go to your question of finding the pulse shape in this phase recovery process. First, the carrier does not need to be involved in this process. With the expansion of unique digital processing solutions and shrinking of the analog side in the wireless communication systems, the signal is downconverted by a free running oscillator at a fixed frequency (leading to low phase jitter) and the phase offset can be compensated in the $IQ$ modulated data right there and then.
Most timing synchronization algorithms operate independent of the phase information. Therefore, almost all the DSP based carrier phase synchronization algorithms are timing-aided which means that the symbol boundaries in the Rx sampled waveform are established before the phase recovery block. Knowing the exact symbol boundaries is the same as identifying the optimal sampling instants where the eye opening is maximum and Inter-Symbol Interference (ISI) from the neighbouring symbols is zero. An example of a matched filter $IQ$ output is drawn in Figure below.
You can see the Raised Cosine pulse shape in the background as a red line. From here, a phase error detector removes the modulation such that the input to the loop filter is a track-able signal.
Many SDR enthusiasts outside of the core wireless area find it strange when they encounter a carrier phase synchronization block of code (e.g., a PLL or a Costas loop in GNU Radio) in which there is no `carrier' to be compensated for! Instead, they only find a phase de-rotation of the $IQ$ samples going into the symbol detector as shown above.
