Noise floor estimation is usually done after applying an FFT to windowed data segments. By tracking the noise floor in each frequency band, the frequency dependence of the noise is taken into account. If the noise is non-stationary, its time dependence can be tracked by regularly updating the noise floor estimate in each frequency band. This usually requires the noise to vary more slowly than the desired signal. Have a look at this paper to get an idea of how it can work, and also look at the references therein. Search the web for "noise floor estimation" and "noise floor tracking" to find many more papers on the subject.