Let's not get too in the weeds of using bits or noise densities to determine your minimum detectable signal (MDS) just yet. What you're asking is a more fundamental question about determining what value (in terms of SNR) you need to declare a detection. The answer to the question "What SNR do I need in order to detect a signal in noise?" is actually up to you. I'll use a simple radar example here since detection schemes tend to be simpler when compared to more complicated communication systems like PSK, FSK, etc. where you're detecting a sequence of bits or whatever.
In detection theory there are the concepts of probabilty of detection and probability of false alarm. In a simple detector you are trying to choose some absolute threshold value such that
- You want to detect the signal(s) of interest (detection)
- You want to avoid false detections from noise (false alarm)
Immediately you can see that there are two competing requirements when setting the threshold:
- Setting a high threshold will help in avoiding detections from noise, but will also make it harder to detect the signal of interest.
- A low threshold will make it easier to detect the signal of interest, but now noise can also be detected more easily causing a false alarm.
An example here is a radar return from a target at 700 m. It has been processed using a matched filter.
The dotted lines are two thresholds we are considering. If we use the blue one, we'll easily avoid false alarms but we're also increasing the chance that the target peak will dip below the threshold, in which we'll miss a detection. The red one will make it very likely that our target will peak above it, but also will the noise which leads to more false alarms.
So, decreasing the threshold increases the probability of detection and the probability of false alarms. Increasing it does the opposite. Having said that, it is up to you to decide which probability to prioritize, since some systems care more about false alarms (or vice versa) and then determine the other. Decide what probabilities you're willing to live with and then set your threshold accordingly. There is no magic number.
There is more to this as this was a simple single pulse example. You can start adding more pulses and play other games, but the basic considerations are the same. You can extend these concepts to communication systems where you can start talking about bit and symbol error rates, etc.