I have a question about invariances in neural networks.
In general, neural nets with enough layers can learn arbitrarily complicated nonlinear functions. Therefore, it's not hard to understand how a neural net can learn a Fourier transform or convolution: (3rd to last paragraph of this page: http://www.dspguide.com/ch26/3.htm).
Fourier transforms have a useful translation (shift) invariance property, i.e. the magnitude of the FT of a function/image remains the same after you translate it.
Other transforms also have useful invariance properties, e.g. the Mellin transform is scale invariant, i.e. the magnitude of the Mellin transform of a function/image remains the same after you scale it.
So my question is about why neural networks don't already have built in invariances. Is it purely a matter of number of neurons? If I have an enormously large number of neurons and/or layers, do I start to get built in invariances?
(E.g. as described in above link, you need 2 layers of N nodes to learn convolution of an N sample signal. Also, not every single neuron is going to be weighted in such a way to participate in learning convolution, so you'll need many more neurons than just 2N.)