This is a good idea, I was pondering upon awhile ago. I drop some thoughts.
A Linear(no activations) MLP with a single hidden layer performs the
same or sometimes better than a Multi-layer model with ReLU
activations. What might be the cause of this?
The dimension reduction transform, i.e. taking CS measurements is a linear operation, a measurement is ...