Neural Networks and Complex Valued Inputs

Question

[not sure if this or stats.stackexchange was the correct location for this post, so put it on both for now.]

I've seen some recent papers describing complex valued neural networks like this one: Deep Complex Networks, 2017, Trabelsi et al.. What I'm wondering is, rather than invent a novel complex network pipeline that takes a complex value input as a single channel, why not just separate the real and imaginary components into two channels fed into a regular neural network, and then let the network figure out the relations, without necessarily knowing that one channel represents the real component while the other represents the imaginary component?

I assume there must be some disadvantage to doing it this way, or some relation that the neural network can't pick up on, so if that's the case would someone please provide me with a high-level explanation of why this two-channel standard network approach is inferior to the novel single-channel complex network?

(By the way, the application I have in mind for researching deep complex networks is RF signal classification.)

Answer 1

The power of complex representations remains an open topic to me. I still do strive the understand Fourier transformations.

An underlying question is, to me: why would complex transformations be useful for real data? More generally, when data dwell in a set $S$, is $S$ the most appropriate set of analysis, or is it more appropriate to resort to a bigger set $S^*$? For instance, for real-valued polynomials, we know that the field of complex numbers provides a more elegant extension. This might not be so distant, as $z$-transforms (extended polynomials) are tools of choice for real time-series analysis. Real linear-time invariant systems are root signals (here eigenvectors) that are complex (cisoids). To better separate frequencies, time-frequency analyses often employ analytic signals and the Hilbert transform, or analytic, dual-tree multiscale tools like wavelets. The recent paper Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety deals with more stochastic observations:

Complex-valued signals occur in many areas of science and engineering and are thus of fundamental interest. In the past, it has often been assumed, usually implicitly, that complex random signals are proper or circular. A proper complex random variable is uncorrelated with its complex conjugate, and a circular complex random variable has a probability distribution that is invariant under rotation in the complex plane. While these assumptions are convenient because they simplify computations, there are many cases where proper and circular random signals are very poor models of the underlying physics. When taking impropriety and noncircularity into account, the right type of processing can provide significant performance gains. There are two key ingredients in the statistical signal processing of complex-valued data: 1) utilizing the complete statistical characterization of complex-valued random signals; and 2) the optimization of real-valued cost functions with respect to complex parameters. In this overview article, we review the necessary tools, among which are widely linear transformations, augmented statistical descriptions, and Wirtinger calculus. We also present some selected recent developments in the field of complex-valued signal processing, addressing the topics of model selection, filtering, and source separation.

But going back in time (Oppenheim's works for instance), one knows that complex phase can capture non-stationarity, discontinuity (edges and jumps) and oscillatory behaviors (textures). My present belief is that, at a given scale, a complex feature computed as a whole is more efficient at capturing local behavior that a pair of real and imaginary parts computed somewhat separately, incorporating some invariances with respect to translation or rotation.

As for neural networks, and of course deep learning, the recent theory of scattering networks, and subsequent works, have provided a solid ground for understanding how deep learning works, from a solid mathematical point of view, based on complex wavelet frames and non-linear operators. Another interesting paper for your question is:

M. Tygert et al., 2016, A Mathematical Motivation for Complex-Valued Convolutional Networks, Neural Computation

Courtesy of the exact correspondence, the remarkably rich and rigorous body of mathematical analysis for wavelets applies directly to (complex-valued) convnets.

Convolutional networks (convnets) have become increasingly important to artificial intelligence in recent years, as reviewed by LeCun, Bengio, and Hinton (2015).This note presents a theoretical argument for complex-valued convnets and their remarkable performance. Complex-valued convnets turn out to calculate data-driven multiscale windowed spectra characterizing certain stochastic processescommonin the modeling of time series (such as audio) and natural images (including patterns and textures).We motivate the construction of such multiscale spectra using local averages of multiwavelet absolute values or, more generally, nonlinear multiwavelet packets.

And maybe, complex numbers are not enough, and we should turn to quaternions...

Answer 2

Training is essentially an optimization and assuming you will have complex weights $z$, and a real valued objective, $$ \min_{z} f(z) \quad f(z) \quad \text{Real} $$ because of the Cauchy Riemann Condition, $f(z)$ is not analytic (essentially doesn't have a Taylor Series in $z$) so this is handled as: $$ \min_{z,z^*} f(z,z^*) $$ where $z$ and $z^*$ are considered independent, which seems a bit strange at first but you can linearly (and invertibly) transform from $z,z^*$ to $\text{Real}(z), \text{Imag}(z)$ and have an equivalent optimization $$\min_z g( \text{Real}(z), \text{Imag}(z)) $$

A good article is

@article{RN18,
   author = {Sorber, Laurent and Barel, Marc Van and Lathauwer, Lieven De},
   title = {Unconstrained optimization of real functions in complex variables},
   journal = {SIAM Journal on Optimization},
   volume = {22},
   number = {3},
   pages = {879-898},
   ISSN = {1052-6234},
   year = {2012},
   type = {Journal Article}
}

which has a reference to Bramwood, which is the way it is typically introduced in array processing.

Going Complex or in terms of real and imaginary are equivlent , given that you understand the optimization in terms of complex variables is necessarily in term of $z,z^*$

Answer 3

Your question makes sense. The difference between a complex network and a regular network with twice the amount of channels, on a mechanical level, is the multiplication operation which ties pairs of channels together. This can be viewed as a restriction of the hypothesis class the network can express compared to a real-valued network with the same parameter count.

Ideally, this should help if the restrictions encode something useful about the problem you're trying to solve (e.g. how a CNN can be viewed as a neural network enforcing translational symmetry). I found a Master's Thesis that shows a complex-valued image-recognition network for a very-specific task generalized than the equivalent real-valued network, at the cost of training being unstable. The analysis there is that problems suited to complex-valued neural networks are ones where interesting input has phase-coherent structure, however they did only show one specific problem where the type of network worked well.