The power of complex representations remains an open topic to me. I still do strive the understand Fourier transformations.
An underlying question is, to me: why would complex transformations be useful for real data? More generally, when data dwell in a set $S$, is $S$ the most appropriate set of analysis, or is it more appropriate to resort to a bigger set $S^*$? For instance, for real-valued polynomials, we know that the field of complex numbers provides a more elegant extension. This might not be so distant, as $z$-transforms (extended polynomials) are tools of choice for real time-series analysis. Real linear-time invariant systems are root signals (here eigenvectors) that are complex (cisoids). To better separate frequencies, time-frequency analyses often employ analytic signals and the Hilbert transform, or analytic, dual-tree multiscale tools like wavelets. The recent paper Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety deals with more stochastic observations:
Complex-valued signals occur in many areas of science and engineering
and are thus of fundamental interest. In the past, it has often been
assumed, usually implicitly, that complex random signals are proper or
circular. A proper complex random variable is uncorrelated with its
complex conjugate, and a circular complex random variable has a
probability distribution that is invariant under rotation in the
complex plane. While these assumptions are convenient because they
simplify computations, there are many cases where proper and circular
random signals are very poor models of the underlying physics. When
taking impropriety and noncircularity into account, the right type of
processing can provide significant performance gains. There are two
key ingredients in the statistical signal processing of complex-valued
data: 1) utilizing the complete statistical characterization of
complex-valued random signals; and 2) the optimization of real-valued
cost functions with respect to complex parameters. In this overview
article, we review the necessary tools, among which are widely linear
transformations, augmented statistical descriptions, and Wirtinger
calculus. We also present some selected recent developments in the
field of complex-valued signal processing, addressing the topics of
model selection, filtering, and source separation.
But going back in time (Oppenheim's works for instance), one knows that complex phase can capture non-stationarity, discontinuity (edges and jumps) and oscillatory behaviors (textures). My present belief is that, at a given scale, a complex feature computed as a whole is more efficient at capturing local behavior that a pair of real and imaginary parts computed somewhat separately, incorporating some invariances with respect to translation or rotation.
As for neural networks, and of course deep learning, the recent theory of scattering networks, and subsequent works, have provided a solid ground for understanding how deep learning works, from a solid mathematical point of view, based on complex wavelet frames and non-linear operators. Another interesting paper for your question is:
M. Tygert et al., 2016, A Mathematical Motivation for Complex-Valued Convolutional Networks, Neural Computation
Courtesy of the exact correspondence, the remarkably rich and rigorous
body of mathematical analysis for wavelets applies directly to
(complex-valued) convnets.
Convolutional networks (convnets) have become increasingly important
to artificial intelligence in recent years, as reviewed by LeCun,
Bengio, and Hinton (2015).This note presents a theoretical argument
for complex-valued convnets and their remarkable performance.
Complex-valued convnets turn out to calculate data-driven multiscale
windowed spectra characterizing certain stochastic processescommonin
the modeling of time series (such as audio) and natural images
(including patterns and textures).We motivate the construction of such
multiscale spectra using local averages of multiwavelet absolute
values or, more generally, nonlinear multiwavelet packets.
And maybe, complex numbers are not enough, and we should turn to quaternions...