For the additive white noise channel, it can be shown that matched filtering maximizes SNR; and for constant illumination, the "transmitting pulse filter" is a rectangle, and the match to that is a rectangular weighting, i.e. an average, too.
The idea is simply that you "my signal is a vector; multiplication / convolution then becomes a dot product. Cauchy-Schwarz Inequality gives me an upper bound for the norm of signal·receive window / filter, and I reach that when what I do at the receiving end is a linear multiple of what I do at the transmitting end".
For unknown spectral distributions, you'd try to find a filter on the receiving end of things that is as orthogonal as possible to the spectrum of the noise. You'd need to estimate that first, though! Devise some calibration method; but, really, when you're observing the same spot for long enough, maybe just look at the spectrum and ignore the constant value?
Assuming you have some non-white noise, you can "abuse" techniques from the world of spread-spectrum communications design. Imagine the following:
You can modulate the intensity of your light source to two values, a and b, over time. Imagine sending a pseudo-random sequence consisting of times when you send the higher intensity, and of the lower intensity:
a ---- -------- ----
b ---- ----
At the receiver, the first thing you do is simply subtract the average of the received signal intensity; that way, what you receive becomes:
c ---- -------- ----
-c ---- ----
Now, that's a very simple spread spectrum signal (in fact, DSSS), and it's almost certain that its spectral properties are pretty different from these of noise.
Now, you just correlate with the high-low sequence that you've sent, considering "high" a "1", and "low" a "-1". When you shift that "+1 -1 +1 +1 -1 +1" sequence correctly, you'll see that by point-wise multiplication between "c -c +c +c -c +c" (which you received), you get a very high amplitude: 1·c + (-1)·(-c) + 1·c + 1·c + (-1)·(c) + 1·c = 6c . In reality, you use lengths much higher than 6!
You still get the same gain as when averaging white noise over a length of $N=6$, but for low-frequency-heavy noise distributions, you'd get noise suppression, too. (you buy that by getting less noise suppression at higher frequencies! No such thing as a free lunch; but you might want to do that, because you might have other light sources in your room – for example, 100 Hz modulating room light, which you can eliminate efficiently).
A bit of general commentary:
That averaging gives you the gain you state depends on the assumption that noise is uncorrelated with signal! In general, that's not the case for you: The energy of noise is very much dependent on current forward current in a semiconductor junction; that's kind of a bummer. What the spread-spectrum approach does is in fact a bit of noise shaping, and it might well suppress a bit of the harmonics that nonlinearity introduces, but I'm not expert enough in photonics to tell you how severe these nonlinearities are.
Also, I don't know the reflectivity of what you're measuring. Just be warned that white LEDs aren't really white: they consist of multiple light sources that generate different wavelengths; same goes for your photo sensors: They aren't uniformly sensitive to all wavelengths. If you haven't already, calibrate your sensor with a color proof sheet!