Given a complex input signal $x$ and real input signal $v$, a 4th order (for simplicity) transfer function $H(z)$ is first applied to $v$ to obtain $w$, which in the time domain is represented by the linear difference equation:
$$ w[n] = b_0v[n]+...+b_4v[n-4] - a_1w[n-1]-...-a_4w[n-4]$$
Output $y$ is then a function of $w$ and $x$: $$ y = F(x,w) = c_1x + c_2x|x| + c_3xw +c_4xw^2 $$
In the physical system being modeled, the inputs $x,v$ and output $y$ are known and observable, $w$ is not observable, however.
The coefficients of $H(z)$ are known to a fair degree of accuracy a priori, the question is how to adapt them in order to improve the model accuracy given only the input and output signals?
In reality there are many more higher order terms in $F$, but the representation here should be sufficient to understand the problem.
So far, I am able to solve for the coefficients of $F$ with an ordinary least squares approach. I can get some improvement by then using a non-linear least squares (Levenberg-Marquardt) approach to improve the estimate of the coefficients of $H$. Intuition tells me I should try to move the zeros and poles of $H$ to improve convergence, but so far I have not been able to connect the dots. Solutions and ideas are much appreciated!