EDIT: So the transfer function for my system is
$$ F_{nco}(z)=\beta E(z)/(z-1)$$ $$ z\Theta_{nco} (z)=\Theta_{nco} (z)+\alpha E(z)+\beta E(z)/(z-1)$$
So I would like to obtain $\Theta_{nco} (z) / \Theta(z)$ transfer function cause I think that would allow me to calculate whic range of $\alpha$ would be good for my PLL to lock. The thing is I don't really know how to obtain $ E(z)$ from $e[n]= \arg \big\{ x[n] \cdot (y[n])^* \big\} $
I'm trying to code my own NCO in order to use it for my own Costas loop (since I couldn't get this to work Implementing Costas loop for 4-PAM) and so far the NCO is working good except for it losing lock after a while. These are the update equations I'm using:
$$\theta_{nco}[n+1]= \theta_{nco}[n] + \alpha e[n] + f_{nco}[n]$$
$$f_{nco}[n+1] = f_{nco}[n] + \beta e[n]$$
So I generated this input with $f=50 Hz$ and $f_s = 1 kHz$ during $0.2 s$ (that is, 200 samples):
$$\begin{align} x[n] &= e^{j 2\pi n f/f_s+\theta} \\ &= \cos(2\pi n f/f_s+\theta)+j\sin(2\pi n f/f_s+\theta) \\ \end{align}$$
Then I'm running a for loop to update the NCO, which initially has 0 phase and frequency ($\theta_{nco}$ and $f_{nco}$). The parameters being $\alpha = 0.1$ and $\beta = 0.0025$.
The way I obtain the error is
$$e[n]= \arg \big\{ x[n] \cdot (y[n])^* \big\} $$
being $y[n]$ the output of the NCO:
$$y[n]=e^{j (2 \pi n f_{nco}[n] + \theta_{nco}[n])}$$
The thing is, it locks quite fast but it loses lock towards the few last samples (at around 0.2 s) (see attached picture). If I increase the number of samples, it loses lock even more.
I managed to fix it by changing alpha and beta to $\alpha = 0.05$ and $\beta = 0.000625$
So basically what I'd like to know is how I can know which are the best values for $\alpha$ and $\beta$ since I'm just randomly picking a low number for $\alpha$ and then doing $\beta=\alpha ^2 /4$. Thanks!