0
$\begingroup$

I am working through the book Geometric Deep Learning (https://arxiv.org/abs/2104.13478) and have hit the following descrition of Rectified Linear Unit (Chapter 5.1, page 70).

Perhaps the most popular example at the time of writing is the Rectified Linear Unit (ReLU): σ(x) = max(x, 0). This non-linearity effectively rectifies the signals, pushing their energy towards lower frequencies, and enabling the computation of high-order interactions across scales by iterating the construction.

In the text, there is no explanation as to what "energy" and "frequencies" means in the context of ReLU. It seems that energy refer to this "energy" and frequencies to the frenquency domain in Fourier transform. If that's the case, how is the above statement justified/explained? Is there any references, papers, textbooks that I can check?

$\endgroup$

1 Answer 1

1
$\begingroup$

Half-wave rectification is a highly non-linear operation which makes it difficult to make general statements like this.

First and foremost it will create a large bias (or DC offset) and the DC component (at 0Hz) will always be the strongest component in the spectrum.

After that, it gets more complicated. Ignoring DC: white noise just stays white noise, pink noise stays pink. Rectifying a sine wave will create harmonics so in the case you are shoving energy to higher frequencies. For bandlimited noise, you will see a little bit of both.

Which way the energy goes (other than to DC, that is) depends a lot on the actual signal and personally I don't think the statement as written is particularly helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.