Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this community
I have a real function $C$ of a complex vector $x$. While taking the gradient of the function $C$ for minimising the same, why do we take the derivatives with respect to the complex conjugate of $x$, i.e. $\bar{x}$ and not the actual vector $x$?
I have tried looking up for help but I am directed again and again to Cauchy–Riemann equations, which I know is not pertinent here.
That's a trick which you will also find in a DSP context, that's why I choose to provide an answer here. It is related to the Wirtinger derivative, and you can find more details about it in this answer over at math.SE.
In practice this trick is often used to compute the extremum (minimum or maximum) of a real-valued function depending on a complex variable (or vector/matrix). In order to find the extremum, you formally take the derivative with respect to the complex conjugate of the variable of interest, set this derivative equal to zero, and from this equation derive the optimum value of the (possibly vector-/matrix-valued) variable.
As a simple example, take the minimization (with respect to the vector $x$) of the following error quantity:
$$\epsilon=||Ax-b||_2^2\tag{1}$$
where the vectors $x$ and $b$, and the matrix $A$ are assumed to be complex-valued. The error can be written as
$$\epsilon=x^HA^HAx-x^HA^Hb-b^HAx+b^Hb\tag{2}$$
where $^H$ denotes the Hermitian (conjugate) transpose. Now you take the derivative of $(2)$ with respect to $x^H$, formally treating this vector as being independent of $x$. The result is
$$\frac{\partial\epsilon}{\partial x^H}=A^HAx-A^Hb\tag{3}$$
Setting this derivative equal to zero and assuming that $A^HA$ is invertible, the optimal value of $x$ is given by
$$x_o=(A^HA)^{-1}A^Hb\tag{4}$$
where the matrix $(A^HA)^{-1}A^H$ on the right hand side of $(4)$ is the Moore-Penrose inverse.
I felt I needed to write an additional answer to try to clear my mind about the question. Here is the try, step by step. Caveat: for simplicity, I used the same notation $C$ of a function of reals $u$ and $v$, for its rewriting in $x=u+iv$ and $\bar x$, and on complex $x$ alone. I hope it is not confusing for the reader.
Let $C(x)$ be a function (we don't assume it real for now) of a complex vector $x$. If $x$ were real, it could come from the left (negative) or the right (positive) direction, and the notion of derivative is simpler. For instance, take $C(x)=x^2$; a derivative at $x=x_0$ is given by the limit (if it exists) of $$\frac{C(x_0+\Delta x) -C(x_0) }{\Delta x}=\frac{x_0^2+2x_0\Delta x+\Delta x^2 - x_0^2}{\Delta x}=2x_0+\Delta x\,.$$
So, the limit is well-defined here.
When dealing with optimization in real variables, this can be satisfactory if one performs least-square optimization, as detailed by Matt L.. With reals, $x$ can approach $x_0$ only from the left (negative) or the right (positive), which makes differentiation relatively easy. With complex numbers, derivation seems more involved, as $x$ can approach $x_0$ from all angles/directions. And $x\to x^2$ is no longer the graph of a 2D nice positive parabola. So what happens when one want to optimize a simple nice convex function such as:
$$C(x)=|x|^2 = x\cdot \overline{x}\,?$$
$$\frac{C(x_0+\Delta x) -C(x_0) }{\Delta x}=\frac{x_0\overline{x_0}+x_0\overline{\Delta x}+\overline{x_0}\Delta x+|\Delta x|^2 - x_0\overline{x_0}}{\Delta x}=x_0\frac{\overline{\Delta x}}{{\Delta x}}+\overline{x_0}+\overline{\Delta x}\,.$$
In the real case, $\frac{\overline{\Delta x}}{{\Delta x}}=1$ and $\overline{x_0}={x_0}$, and we go back to normal with a $2x_0$ derivative. Sadly, in the complex case, the term with $\frac{\overline{\Delta x}}{{\Delta x}}$ is unresolved, as it has not clear limit (it has unit modulus), unless with $x_0=0$.
Yet, we would like to have a broader notion of derivation (for extremum extraction), allowing this function to be differentiable. So we can reformulate it with $x=u+iv$ (or "get back to real") and look for stationnary points where:
$$ \frac{\partial C(u,v)}{du} = \frac{\partial C(u,v)}{dv} =0\,.$$
This may work, but require to convert an all $x$ (complex) function in real and imaginary parts, which can be cumbersome. Since $u=\frac{1}{2}(x+\bar x)$, $v=\frac{1}{2}(x-\bar x)$, one can rewrite $C(u,v)$ as ${C}(x,\overline{x})$. With the chain rule: $$\frac{\partial}{\partial u}=\frac{\partial x}{\partial u}\frac{\partial}{\partial x}+\frac{\partial\bar x}{\partial u}\frac{\partial}{\partial\bar x}=\frac{\partial}{\partial x}+\frac{\partial}{\partial\bar x}\,.$$
Similarly, one gets:
$$\frac{\partial}{\partial v}=i\left(\frac{\partial}{\partial x}-\frac{\partial}{\partial\bar x}\right)\,.$$
Then, resolving the above (invertible) system in both $x$ and $\bar x$, that can be treated as another pair of "independent" variables, and one get the Wirtinger derivatives:
$$\frac{\partial}{\partial x}=\frac{1}{2}\left(\frac{\partial}{\partial u}-i\frac{\partial}{\partial v}\right)$$ and $$\frac{\partial}{\partial \bar x}=\frac{1}{2}\left(\frac{\partial}{\partial u}+i\frac{\partial}{\partial v}\right)\,.$$
Wirtinger calculus has its merits: since $\frac{\partial}{\partial x} \bar x = \frac{\partial}{\partial \bar x} x =0 $, $\bar x $ can be regarded as a constant when differentiating with respect to $ x $ and vice-versa (hence the notion of "independence" used above. For the squared error ${C}(x,\overline{x}) = x.\bar x$ undergoes the Wirtinger derivation as:
$$\frac{\partial}{\partial \bar x}{C}(x,\overline{x}) = x$$ and $$\frac{\partial}{\partial x}{C}(x,\overline{x}) = \bar x\,.$$
In the general case, Cauchy–Riemann equations do appear: if $C$ is Fréchet-differentiable (T. Tao, Notes 1: complex differentiation, exercise 23), $C$ is holomorphic if and only if $\frac{\partial}{\partial \bar x}{C}(x,\overline{x})=0$ everywhere on some open domain of $\mathbb{C}$. In other words, ${C}$ need to be $\overline{x}$-free to be holomorphic. Roughly, a real-function over $\mathbb{C}$ is rarely holomorphic, unless it is constant. This is somewhat intuitive: to be real, a $x$ in the expression of ${C}$ should be counterbalanced by some $\overline{x}$ that un-complexes it. So, Wirtinger derivatives elegantly re-express CS conditions in a compact way. To find stationnary points, you now can look where $\partial_u$ and $\partial_v$ vanish, or equivalently where $\partial_{x}$ and $\partial_\overline{x}$ do. You have not gain too much, apart from delaing with the complex variables directly, instead of converting everything to real and imaginary parts.
Now, what happen if $C$ is real? Then, there is some redundancy in the equations, that can be exploited. For instance, both partial derivatives $\frac{\partial}{\partial u}$ and $\frac{\partial}{\partial v}$ ought to be real. In other words, they are equal to their conjugate. And it turns out that this amounts to the derivative of ${C}(x,\overline{x})$ with respect to the first variable being zero.
So now, only one equation suffices to find the stationnary points of a real function of a complex variable:
$$\frac{\partial C(x)}{\partial x}=0\,,$$ and the steepest ascent points (under some conditions) to the direction of:
$$\frac{\partial C(x)}{\partial \bar x}\,.$$
Resultantly, a natural extention of gradient descent is therefore written as:
$$x_{k+1}=x_{k}-\mu \frac{\partial C(x)}{\partial \bar x}x_{k}\,,$$ which can find good approximations of the extremas, instead of trying to solve the vanishing equation in $\partial x$. Hoping I did not made mistakes in the notations, more information can be found in:
I have found an alternate answer which is very simple and comprehensive, so thinking to share with all.
In order to differentiate an expression $f(z)$ with respect to a complex $z$, the Cauchy-Riemann equations have to be satisfied:
\begin{equation} \label{eq:1} \frac{\partial f(z) }{\partial z} = \frac{\partial \mathbb{R}(f(z)) }{\partial \Re z} + i \ \frac{\partial \Im (f(z)) }{\partial \Re z} \end{equation}
and
\begin{equation} \label{eq:2} \frac{\partial f(z) }{\partial z} = - i \ \frac{\partial \mathbb{R}(f(z)) }{\partial \Im z} + \frac{\partial \Im (f(z)) }{\partial \Im z} \end{equation}
A complex function that satisfies the Cauchy-Riemann equations for a set of points in region $R$ is said to be an analytic in the region $R$. In general, expressions involving complex conjugate or conjugate transpose do not satisfy the Cauchy-Riemann equations. In order to avoid this, a more generalised definition of complex derivative is used.
The Generalised Complex Derivative which is
\begin{equation} \label{eq:3} \frac{d f(z) }{d z} = \frac{1}{2}\left ( \ \frac{\partial f(z) }{\partial \Re z} - i\ \frac{\partial f(z) }{\partial \Im z} \right ) \end{equation}
and the Conjugate Complex Derivative which is
\begin{equation} \label{eq:4} \frac{d f(z) }{d z^*} = \frac{1}{2}\left ( \ \frac{\partial f(z) }{\partial \Re z} + i\ \frac{\partial f(z) }{\partial \Im z} \right ) \end{equation}
The genralised complex derivative equals the normal derivative when $f$ is an analytic function. For non-analytic function such as $f(z)= z^*$, the derivative equals zero.
The conjugate complex derivative equals zero when $f$ is an analytic function, and the conjugate complex derivative is used when deriving a complex gradient.
Also note that
\begin{equation} \label{eq:5} \frac{d f(z) }{d z} \neq \ \frac{\partial f(z) }{\partial \Re z} + i\ \frac{\partial f(z) }{\partial \Im z} \end{equation}
as would apparently seem to be.
Now, if $f$ is a real function of a complex vector/ matrix $z$, then the complex gradient vector is given by
\begin{equation} \label{eq:6} \Delta f(z)= 2 \ \frac{d f(z) }{d z^*} = \ \frac{\partial f(z) }{\partial \Re z} + i\ \frac{\partial f(z) }{\partial \Im z} \end{equation}
which is the expression that we actually desire.
