I felt I needed to write an additional answer to try to clear my mind about the question. Here is the try, step by step. Caveat: for simplicity, I used the same notation $C$ of a function of reals $u$ and $v$, for its rewriting in $x=u+iv$ and $\bar x$, and on complex $x$ alone. I hope it is not confusing for the reader.
Let $C(x)$ be a function (we don't assume it real for now) of a complex vector $x$. If $x$ were real, it could come from the left (negative) or the right (positive) direction, and the notion of derivative is simpler. For instance, take $C(x)=x^2$; a derivative at $x=x_0$ is given by the limit (if it exists) of $$\frac{C(x_0+\Delta x) -C(x_0) }{\Delta x}=\frac{x_0^2+2x_0\Delta x+\Delta x^2 - x_0^2}{\Delta x}=2x_0+\Delta x\,.$$
So, the limit is well-defined here.
When dealing with optimization in real variables, this can be satisfactory if one performs least-square optimization, as detailed by Matt L.. With reals, $x$ can approach $x_0$ only from the left (negative) or the right (positive), which makes differentiation relatively easy. With complex numbers, derivation seems more involved, as $x$ can approach $x_0$ from all angles/directions. And $x\to x^2$ is no longer the graph of a 2D nice positive parabola. So what happens when one want to optimize a simple nice convex function such as:
$$C(x)=|x|^2 = x\cdot \overline{x}\,?$$
$$\frac{C(x_0+\Delta x) -C(x_0) }{\Delta x}=\frac{x_0\overline{x_0}+x_0\overline{\Delta x}+\overline{x_0}\Delta x+|\Delta x|^2 - x_0\overline{x_0}}{\Delta x}=x_0\frac{\overline{\Delta x}}{{\Delta x}}+\overline{x_0}+\overline{\Delta x}\,.$$
In the real case, $\frac{\overline{\Delta x}}{{\Delta x}}=1$ and $\overline{x_0}={x_0}$, and we go back to normal with a $2x_0$ derivative. Sadly, in the complex case, the term with $\frac{\overline{\Delta x}}{{\Delta x}}$ is unresolved, as it has not clear limit (it has unit modulus), unless with $x_0=0$.
Yet, we would like to have a broader notion of derivation (for extremum extraction), allowing this function to be differentiable. So we can reformulate it with $x=u+iv$ (or "get back to real") and look for stationnary points where:
$$ \frac{\partial C(u,v)}{du} = \frac{\partial C(u,v)}{dv} =0\,.$$
This may work, but require to convert an all $x$ (complex) function in real and imaginary parts, which can be cumbersome. Since $u=\frac{1}{2}(x+\bar x)$, $v=\frac{1}{2}(x-\bar x)$, one can rewrite $C(u,v)$ as ${C}(x,\overline{x})$.
With the chain rule:
$$\frac{\partial}{\partial u}=\frac{\partial x}{\partial u}\frac{\partial}{\partial x}+\frac{\partial\bar x}{\partial u}\frac{\partial}{\partial\bar x}=\frac{\partial}{\partial x}+\frac{\partial}{\partial\bar x}\,.$$
Similarly, one gets:
$$\frac{\partial}{\partial v}=i\left(\frac{\partial}{\partial x}-\frac{\partial}{\partial\bar x}\right)\,.$$
Then, resolving the above (invertible) system in both $x$ and $\bar x$, that can be treated as another pair of "independent" variables, and one get the Wirtinger derivatives:
$$\frac{\partial}{\partial x}=\frac{1}{2}\left(\frac{\partial}{\partial u}-i\frac{\partial}{\partial v}\right)$$
and
$$\frac{\partial}{\partial \bar x}=\frac{1}{2}\left(\frac{\partial}{\partial u}+i\frac{\partial}{\partial v}\right)\,.$$
Wirtinger calculus has its merits: since $\frac{\partial}{\partial x} \bar x = \frac{\partial}{\partial \bar x} x =0 $, $\bar x $ can be regarded as a constant when differentiating with respect to $ x $ and vice-versa (hence the notion of "independence" used above. For the squared error ${C}(x,\overline{x}) = x.\bar x$ undergoes the Wirtinger derivation as:
$$\frac{\partial}{\partial \bar x}{C}(x,\overline{x}) = x$$
and
$$\frac{\partial}{\partial x}{C}(x,\overline{x}) = \bar x\,.$$
In the general case, Cauchy–Riemann equations do appear: if $C$ is Fréchet-differentiable (T. Tao, Notes 1: complex differentiation, exercise 23), $C$ is holomorphic if and only if $\frac{\partial}{\partial \bar x}{C}(x,\overline{x})=0$ everywhere on some open domain of $\mathbb{C}$. In other words, ${C}$ need to be $\overline{x}$-free to be holomorphic. Roughly, a real-function over $\mathbb{C}$ is rarely holomorphic, unless it is constant. This is somewhat intuitive: to be real, a $x$ in the expression of ${C}$ should be counterbalanced by some $\overline{x}$ that un-complexes it. So, Wirtinger derivatives elegantly re-express CS conditions in a compact way.
To find stationnary points, you now can look where $\partial_u$ and $\partial_v$ vanish, or equivalently where $\partial_{x}$ and $\partial_\overline{x}$ do. You have not gain too much, apart from delaing with the complex variables directly, instead of converting everything to real and imaginary parts.
Now, what happen if $C$ is real? Then, there is some redundancy in the equations, that can be exploited. For instance, both partial derivatives $\frac{\partial}{\partial u}$ and $\frac{\partial}{\partial v}$ ought to be real. In other words, they are equal to their conjugate. And it turns out that this amounts to the derivative of ${C}(x,\overline{x})$ with respect to the first variable being zero.
So now, only one equation suffices to find the stationnary points of a real function of a complex variable:
$$\frac{\partial C(x)}{\partial x}=0\,,$$
and the steepest ascent points (under some conditions) to the direction of:
$$\frac{\partial C(x)}{\partial \bar x}\,.$$
Resultantly, a natural extention of gradient descent is therefore written as:
$$x_{k+1}=x_{k}-\mu \frac{\partial C(x)}{\partial \bar x}x_{k}\,,$$
which can find good approximations of the extremas, instead of trying to solve the vanishing equation in $\partial x$.
Hoping I did not made mistakes in the notations, more information can be found in:
- Wirtinger’s Calculus in general Hilbert Spaces, Pantelis Bouboulis, 2010
- An Introduction to Complex Differentials and Complex Differentiability, Raphael Hunger, 2007, Technical Report, TUM-LNS-TR-07-06
- Complex differentiation is quite complex, Stephen Tu, 2016
- Notes 1: complex differentiation, Terence Tao, 2016
- dz and dz bar: How to derive the Wirtinger derivatives
- What is the intuition behind the Wirtinger derivatives?