7
$\begingroup$

[not sure if this or stats.stackexchange was the correct location for this post, so put it on both for now.]

I've seen some recent papers describing complex valued neural networks like this one: Deep Complex Networks, 2017, Trabelsi et al.. What I'm wondering is, rather than invent a novel complex network pipeline that takes a complex value input as a single channel, why not just separate the real and imaginary components into two channels fed into a regular neural network, and then let the network figure out the relations, without necessarily knowing that one channel represents the real component while the other represents the imaginary component?

I assume there must be some disadvantage to doing it this way, or some relation that the neural network can't pick up on, so if that's the case would someone please provide me with a high-level explanation of why this two-channel standard network approach is inferior to the novel single-channel complex network?

(By the way, the application I have in mind for researching deep complex networks is RF signal classification.)

$\endgroup$
3
  • 2
    $\begingroup$ Hi: It's probably easy enough to check if your idea results in the same result from the NN. If not, then it's probably due to the backpropagation algorithm being somehow dependent on the correlation of the real part and imaginary part. This isn't an answer but the backprop algoritm takes partials so it may matter if you send them in separately. I'm very very fuzzy on backprop and this is not a direct answer to your question, but, my point is that, if the results are different, then the backprop algorithm nuances are almost surely the reason why. $\endgroup$
    – mark leeds
    Apr 25, 2018 at 19:03
  • $\begingroup$ I plan to try this out as soon as I get the code working that goes along with the paper I linked. I'm looking to get an intuitive enough understanding that I can explain it to others. Thanks though, that gives me at least some idea of what the issue is. $\endgroup$
    – Austin
    Apr 25, 2018 at 19:07
  • 2
    $\begingroup$ Hi: Me again. I just read only the abstract but convolutional networks-deep learning is somewhat different from straight basic NN's so my answer may not be applicable to your question. I have zero knowledge of deep learning aside from regular NN's so can't provide any insight other than above. Apologies for any confusion. $\endgroup$
    – mark leeds
    Apr 25, 2018 at 19:08

3 Answers 3

6
$\begingroup$

The power of complex representations remains an open topic to me. I still do strive the understand Fourier transformations.

An underlying question is, to me: why would complex transformations be useful for real data? More generally, when data dwell in a set $S$, is $S$ the most appropriate set of analysis, or is it more appropriate to resort to a bigger set $S^*$? For instance, for real-valued polynomials, we know that the field of complex numbers provides a more elegant extension. This might not be so distant, as $z$-transforms (extended polynomials) are tools of choice for real time-series analysis. Real linear-time invariant systems are root signals (here eigenvectors) that are complex (cisoids). To better separate frequencies, time-frequency analyses often employ analytic signals and the Hilbert transform, or analytic, dual-tree multiscale tools like wavelets. The recent paper Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety deals with more stochastic observations:

Complex-valued signals occur in many areas of science and engineering and are thus of fundamental interest. In the past, it has often been assumed, usually implicitly, that complex random signals are proper or circular. A proper complex random variable is uncorrelated with its complex conjugate, and a circular complex random variable has a probability distribution that is invariant under rotation in the complex plane. While these assumptions are convenient because they simplify computations, there are many cases where proper and circular random signals are very poor models of the underlying physics. When taking impropriety and noncircularity into account, the right type of processing can provide significant performance gains. There are two key ingredients in the statistical signal processing of complex-valued data: 1) utilizing the complete statistical characterization of complex-valued random signals; and 2) the optimization of real-valued cost functions with respect to complex parameters. In this overview article, we review the necessary tools, among which are widely linear transformations, augmented statistical descriptions, and Wirtinger calculus. We also present some selected recent developments in the field of complex-valued signal processing, addressing the topics of model selection, filtering, and source separation.

But going back in time (Oppenheim's works for instance), one knows that complex phase can capture non-stationarity, discontinuity (edges and jumps) and oscillatory behaviors (textures). My present belief is that, at a given scale, a complex feature computed as a whole is more efficient at capturing local behavior that a pair of real and imaginary parts computed somewhat separately, incorporating some invariances with respect to translation or rotation.

As for neural networks, and of course deep learning, the recent theory of scattering networks, and subsequent works, have provided a solid ground for understanding how deep learning works, from a solid mathematical point of view, based on complex wavelet frames and non-linear operators. Another interesting paper for your question is:

M. Tygert et al., 2016, A Mathematical Motivation for Complex-Valued Convolutional Networks, Neural Computation

Courtesy of the exact correspondence, the remarkably rich and rigorous body of mathematical analysis for wavelets applies directly to (complex-valued) convnets.

Convolutional networks (convnets) have become increasingly important to artificial intelligence in recent years, as reviewed by LeCun, Bengio, and Hinton (2015).This note presents a theoretical argument for complex-valued convnets and their remarkable performance. Complex-valued convnets turn out to calculate data-driven multiscale windowed spectra characterizing certain stochastic processescommonin the modeling of time series (such as audio) and natural images (including patterns and textures).We motivate the construction of such multiscale spectra using local averages of multiwavelet absolute values or, more generally, nonlinear multiwavelet packets.

And maybe, complex numbers are not enough, and we should turn to quaternions...

$\endgroup$
2
  • 1
    $\begingroup$ This is very interesting thanks for the paper links! $\endgroup$
    – Austin
    Apr 26, 2018 at 0:51
  • $\begingroup$ And I believe this is an ongoing trend: Complex is good for reals :) $\endgroup$ May 1, 2018 at 14:17
3
$\begingroup$

Training is essentially an optimization and assuming you will have complex weights $z$, and a real valued objective, $$ \min_{z} f(z) \quad f(z) \quad \text{Real} $$ because of the Cauchy Riemann Condition, $f(z)$ is not analytic (essentially doesn't have a Taylor Series in $z$) so this is handled as: $$ \min_{z,z^*} f(z,z^*) $$ where $z$ and $z^*$ are considered independent, which seems a bit strange at first but you can linearly (and invertibly) transform from $z,z^*$ to $\text{Real}(z), \text{Imag}(z)$ and have an equivalent optimization $$\min_z g( \text{Real}(z), \text{Imag}(z)) $$

A good article is

@article{RN18,
   author = {Sorber, Laurent and Barel, Marc Van and Lathauwer, Lieven De},
   title = {Unconstrained optimization of real functions in complex variables},
   journal = {SIAM Journal on Optimization},
   volume = {22},
   number = {3},
   pages = {879-898},
   ISSN = {1052-6234},
   year = {2012},
   type = {Journal Article}
}

which has a reference to Bramwood, which is the way it is typically introduced in array processing.

Going Complex or in terms of real and imaginary are equivlent , given that you understand the optimization in terms of complex variables is necessarily in term of $z,z^*$

$\endgroup$
5
  • $\begingroup$ I'll have to look up the math references as they are a bit above my head for now, but to understand your conclusion are you saying there is not necessarily a performance degradation from treating the real and imaginary components as separate channel real values in a standard neural network? Or did I miss your point completely? $\endgroup$
    – Austin
    Apr 25, 2018 at 20:19
  • $\begingroup$ Just to double check that my original question was clear, I meant in the one scenario to have everything be real-valued, that is take the real component as one real channel, the imaginary component as another real channel, real valued weights, and a real valued activation function. So like instead of inputting [a+bi], just input [a,b] separately without explicitly telling the network that the second channel is imaginary. $\endgroup$
    – Austin
    Apr 25, 2018 at 20:23
  • 2
    $\begingroup$ I'm just saying you can get to the same place, 2 different ways. Didn't say that getting there takes the same time, but aside from the time to make sure and test your training algorithm for the complex case, there should be about the same number of math operations, so about the same, but given the amount of time it took C to have intrinsic Complex types, I wouldn't just assume the same level of optimization. The SIAM paper favours the $z,z^*$ approach. If you are more interested in the network, the real,imaginary approaches is going to be quicker and safer. Programing wise $\endgroup$
    – user28715
    Apr 25, 2018 at 20:33
  • $\begingroup$ Thanks. I'm more asking from the standpoint of accuracy rather than compute time. I was wondering if there was any reason to believe that the complex case directly would achieve higher classification accuracy for complex data than splitting it into two channels. I heard some talk about neural nets not being able to understand the relation between real and imaginary components when fed in seperately because of the cyclical patterns of the data. $\endgroup$
    – Austin
    Apr 25, 2018 at 20:35
  • 1
    $\begingroup$ When it comes to non Convex optimization, your guess is as good as mine. The paper isn't that hard with Wikipedia as a backup for some clarifications. You should let it make the case, either way. $\endgroup$
    – user28715
    Apr 25, 2018 at 20:41
2
$\begingroup$

Your question makes sense. The difference between a complex network and a regular network with twice the amount of channels, on a mechanical level, is the multiplication operation which ties pairs of channels together. This can be viewed as a restriction of the hypothesis class the network can express compared to a real-valued network with the same parameter count.

Ideally, this should help if the restrictions encode something useful about the problem you're trying to solve (e.g. how a CNN can be viewed as a neural network enforcing translational symmetry). I found a Master's Thesis that shows a complex-valued image-recognition network for a very-specific task generalized than the equivalent real-valued network, at the cost of training being unstable. The analysis there is that problems suited to complex-valued neural networks are ones where interesting input has phase-coherent structure, however they did only show one specific problem where the type of network worked well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.