Oversample or undersample correlated noise?

I wish to measure nontrivial (non-Gaussian) broadband current noise for its distribution and power spectral density. The noise is amplified (with transimpedance amplifier) with some characteristic risetime (tau ~30 usec), that I assume makes the signal correlated with the same timescale. The noise itself may be correlated with different timescale (~1 usec to 1 msec).

Is it best to oversample (sample frequency > 1/tau) or undersample (< 1/tau) the signal? Does it really matter and what is the difference/effect?

I thought that slight undersampling draws directly from the distribution and the distribution is not biased to "double counting" (measure distortion) of the signal, however a colleague advised to overasample, thus no information gets lost.

To make it clear, the signal is random noise, and the purpose is to properly characterize it (so I'm not sure how Nyquist theorem fits in or relevant to it. Maybe other sampling theorems may be useful).

I would strongly suggest oversampling at a frequency much higher than the bandwidth of concern, and specifically setting up the test with proper filtering to ensure that there is insignificant energy beyond half the sampling rate or $$f_s/2$$ (assuming real sampling). This would imply using a low pass filter prior to the sampler and also reviewing the resulting power spectral density that the energy is sufficiently lower as you approach the Nyquist frequency of $$f_s/2$$. The analog input frequency that extends to $$f_s/2$$ is referred to as the first Nyquist band.
You may be "bandpass" sampling in which case "undersampling" would apply with the same guidance above to "oversample" at more than 2x the bandwidth of interest with sufficient margin for filting. In this case the filter is a bandpass filter centered on the higher Nyquist zone (the first Nyquist zone for real signals is $$f=0$$ to $$f_s/2$$ as noted above, the second Nyquist zone is $$f_s/2$$ to $$f_s$$, the third Nyquist zone is $$f_s$$ to $$3f_s/2$$ etc all of which may be undersampled.
However in all cases it is imperative to be filtering the signal of interest and to oversample it with a sampling rate > 2x the resulting bandwidth of the filter. The reason is when you undersample below that bandwidth, while you will still be able to measure the overall total integrated power density, you will have lost all information as to what frequency your noise issues are at which would typically be of great interest in dealing with the noise. When sampling, everything that is in a band over a multiple of the sampling rate will fold into the same frequency band in your digital spectrum that extends from $$-f_s/2$$ to $$+f_s/2$$ (which is equivalently $$f=0$$ to $$f_s/2$$ for real signals). For example, every frequency $$Nf_s + \Delta$$ will alias to $$\Delta$$ in your digital spectrum.