What is the intuition explaining the Shannon-Hartley theorem?

Question

In the digital domain, the signal to noise ratio (SNR) may be interpreted loosely but usefully as the number of discrete or quantized levels one can transmit reliably. For instance, loosely speaking, if the maximum input voltage to a circuit channel with with a noise level of $\pm 0.5 V$ is $10 V$ so that the $S/N$ is related to $\frac{10}{1}$ and the maximal number of discrete integer levels one can pass is roughly $11$, which are the integer values of voltage from $0 V$ through $10 V$. The logarithm to the base two simply reflects the number of bits required to transmit this maximum number of levels. This intuition is particularly useful when interpreting the Shannon-Hartley theorem represented by $C=B\log_2(1 + S/N)$, wherein the channel capacity $C$ or reliable bit rate is directly proportional to the bandwidth, $B$, and the logarithm of the SNR to the base two. Clearly, $\log_2 (1 + S/N)$ reflects the maximum number of bits one can pass through the channel reliably while $B$ is the rate at which one can pass any number of bits. The interpretation is also corroborated on observing the units of the quantities on the left and right hand side of the equation representing the theorem.

In the analog domain, does the interpretation of the SNR as the number of discrete or quantized levels one can transmit reliably, apply readily as in the digital domain, and if not, is there a similar interpretation which provide us the intuition to apply the basic Shannon-Hartley theorem to analog signals?

Edit: Altered text to clarify heuristic interpretation.

Answer 1

Fundamentally, the signal-to-noise-ratio (SNR or $S/N$) is the ratio of signal power to noise power and that power ratio and is usually expressed in $dB$. Shannon and Hartley (likely collaborating at Bell labs, 1928) made the connection of the SNR (expressed as the unit-less ratio and not in $dB$) with the bandwidth and capacity a channel has, to (reliably) move data through it.

We shall try to construct a thought experiment that might help illustrate where the Shannon channel capacity theorem might come from. This is nowhere near a rigorous proof and I am making shit things up to help us intuit the likely genesis of the theorem.

---

Intuition 1: Relationship between channel bandwidth and bit-rate capacity

Suppose your data has high entropy, in that every bit, $d_w$ appears to be independent from every other bit. At some discrete time, $n$, you can group these bits together into a $W$-bit word of data as an integer, $i[n]$

$$ i[n] = \sum\limits_{w=0}^{W-1} d_w[n] 2^w $$

here $0 \le i[n] \le 2^W - 1$. Now suppose we use an ideal $W$-bit Digital-to-Analog converter (DAC) to output a bipolar voltage,

$$x[n] = \Delta \left( i[n] - \frac{2^W - 1}{2} \right)$$

representing data $i[n]$. We can see that $-\Delta\frac{2^W - 1}{2} \le x[n] \le +\Delta\frac{2^W - 1}{2} $.

Now suppose that DAC output is connected to a $W$-bit Analog-to-Digital converter (ADC) that is scaled exactly (having the same stepsize $\Delta$) as the DAC is so that the analog voltage of $x[n]$ is read exactly correctly by the ADC. At some discrete time $n$, you have $W$ bits of data $d_w[n]$ that is assembled into a unipolar value $i[n]$, converted into a bipolar DAC voltage of value $x[n]$ that is passed to an ADC that converts the voltage back to the same $W$-bit value $i[n]$, and you have your $W$ bits, $d_w[n]$ back at the other end. You were able to send these $W$ bits of information across this (so far noiseless) channel to the other end and accurately retrieve the bits.

And in the following discrete sample instance, $n+1$, you can send another $W$ bits, (these bits are $d_w[n+1]$) across the channel and retrieve these bits with your perfectly-scaled ideal ADC. Now how often can you do that? If your channel has bandwidth $B$ Hz, then you can send $2B$ samples of $x[n]$ through this channel every second. The sample rate is $2B$. The ideal DAC output would be:

$$ x(t) = \sum\limits_{n=-\infty}^{\infty} x[n] \operatorname{sinc}\left(2Bt-n \right) $$

where

$$ \operatorname{sinc}(u) \triangleq \begin{cases} \frac{\sin(\pi u)}{\pi u} \qquad & u \ne 0 \\ 1 \qquad & u = 0 \\ \end{cases}$$

That means you're sending $2B\cdot W$ bits of information through this noiseless channel of bandwidth $B$ every second.

---

Intuition 2: Signal power of digital (binary) signal

Now let's calculate the signal power assuming these bits $d_w[n]$ are totally random and independent of each other.

So assuming the step-size, $\Delta$, of the DAC is uniform the different voltages that the DAC outputs, $x[n]$, are

$$x[n] = \Delta \cdot i[n] - \Delta\frac{2^W - 1}{2},$$

where $i[n]=\sum_{w=0}^{W-1} d_w[n] 2^w$ is some integer $0 \le i[n] \le 2^W - 1$. Since we assume that the bits are uniformly random and independent, then every combination of $W$ bits for the integer word, $i[n]$, from 0000...0000 to 1111...1111 is equally likely. Therefore, the signal power which is defined as the expectation of the square of the signal, is obtained for the signal considered as

$$\begin{align} \overline{|x[n]|^2} &= \frac{1}{2^W}\sum\limits_{i=0}^{2^W - 1} \left(\Delta \cdot i - \Delta\frac{2^W - 1}{2}\right)^2 \\ \\ &= \frac{\Delta^2}{2^W} \left( \sum\limits_{i=0}^{2^{W-1} - 1} \left( i - \frac{2^W - 1}{2}\right)^2 + \sum\limits_{i=0}^{2^{W-1} - 1} \left(i + 2^{W-1} - \frac{2^W - 1}{2}\right)^2 \right) \\ \\ &= \ \ \frac{2\Delta^2}{2^W}\sum\limits_{i=0}^{2^{W-1} - 1} (i+\tfrac12)^2 \\ \\ &= \frac{\Delta^2}{2^{W-1}}\sum\limits_{i=0}^{2^{W-1} - 1} (i^2 + i + \tfrac14) \\ \\ &= \frac{\Delta^2}{2^{W-1}} \cdot \left( \frac{(2^{W-1}-1)2^{W-1}(2^W-1)}{6} + \frac{(2^{W-1}-1)2^{W-1}}{2} + 2^{W-1}\tfrac14 \right) \\ \\ &= \Delta^2\cdot\left( \frac{(2^{W-1}-1)(2^W-1)}{6} \ + \ \frac{2^{W-1}-1}{2} \ + \ \frac14 \right) \\ \\ &= \Delta^2\cdot\left( \frac{2^{W-1}2^W - (2^{W-1}+2^W) + 1}{6} \ + \ \frac{2^{W-1}-1}{2} \ + \ \frac14 \right) \\ \\ &= \Delta^2\cdot\left( \frac{2^{2W-1} - 3\cdot2^{W-1} + 1}{6} \ + \ \frac{3\cdot 2^{W-1}-3}{3\cdot2} \ + \ \frac14 \right) \\ \\ &= \Delta^2\cdot\left( \frac{2^{2W-1}-2}{6} \ + \ \frac14 \right) \\ \\ &= \Delta^2\cdot\left( \frac{2^{2W}-4}{12} \ + \ \frac{3}{12} \right) \\ \\ &= \frac{\Delta^2}{12}\cdot(2^{2W}-1). \end{align}$$

The expression above is the signal power or the $S$ in $S/N$.

---

Intuition 3: Relationship between SNR and effective word or pulse width, $W$, which can be passed reliably through the channel

So far, with the DAC, we're able to transmit $2B$ words of information per second through the channel and each word of information has width $W$ and that many bits of information in it.

Now, how much noise can we add to that signal and still expect the ADC to correctly estimate the value of $x[n]$? In the interest of answering this question simply, in order to illustrate the intuition behind the channel capacity, we make a big bad assumption as follows. Shannon assumes noise is (sampled from) a standard Normal or Gaussian random variable with distribution or probability density function (PDF) having variance (power is defined to be the variance) $0<\sigma^2$. However, we assume, purely for mathematical convenience to improve our understanding, that this added noise is uniform PDF of width no bigger than the DAC and ADC step-size $\Delta$. If we call this channel noise an error signal $\epsilon[n]$ that has uniform probability such that

$$-\frac{\Delta}{2} < \epsilon[n] < \frac{\Delta}{2},$$

then, the signal, $x[n]$ with the noise $\epsilon[n]$ added is

$$y[n] = x[n] + \epsilon[n].$$

We assume that adding that noise $|\epsilon[n]|$ is just barely small enough that this will not cause the ADC to spuriously round to a different word other than $i[n]$ which is transmitted as $x[n]$. If $\epsilon[n]$ were larger in magnitude, it could potentially cause the ADC to report a different integer value than $i[n]$ as the output for the input $i[n]$ to the DAC.

The PDF of $\epsilon[n]$ is

$$ p_\epsilon(u) = \begin{cases} \frac{1}{\Delta} \qquad & |u| < \frac{\Delta}{2} \\ 0 \qquad & |u| \ge \frac{\Delta}{2} \\ \end{cases}$$

The noise power which is defined as the variance or expectation of the squared deviation or mean squared deviation or the squared standard deviation of a noise signal, is obtained for the noise considered as

$$\overline{|\epsilon[n]|^2} = \frac{\Delta^2}{12},$$

The expression above can be verified from this post on Stack Exchange which calculates the variance of a random variable with zero mean uniform PDF.

So the SNR is

$$\begin{align} \frac{\overline{|x[n]|^2}}{\overline{|\epsilon[n]|^2}} &= \frac{\frac{\Delta^2}{12}\cdot(2^{2W}-1)}{\frac{\Delta^2}{12}} \\ \\ &= 2^{2W}-1 \\ \end{align} $$

Thus, we have a relationship between the number of levels which can be passed reliably through the channel, $W$, or word width and the SNR given as

$$W = \tfrac12 \log_2 \left(1 + \frac{\overline{|x[n]|^2}}{\overline{|\epsilon[n]|^2}} \right).$$

Further, notice that the effective (that which can be passed reliably through the channel) number of analog (say, voltage) levels is given as

$$2^W = \sqrt{1+SNR},$$

since $W = \frac{1}{2}\log_2 (1+SNR) = \log_2 \sqrt{1+SNR}$.

---

Shannon-Hartley theorem: Maximum channel capacity or theoretical upper bound on net bit or pulse rate is equal to the product of the bandwidth and logarithm to the base 2 of the SNR of the channel

Based on the intuitive analysis above, we obtain the the (reliable) rate of passage of information or channel capacity as

$$C = 2 B \cdot W = B \log_2 \left(1 + \frac{\overline{|x[n]|^2}}{\overline{|\epsilon[n]|^2}} \right).$$

Answer 2

Echoing what already answered: you are approaching this backwards. SNR is a concept that's very fundamental and applicable to way more things that just channel capacity.

If you have a signal $y(t)$ that's the sum of a desired signal $x(t)$ and a noise signal $r(t)$ , i.e. $y(t) = x(t) + r(t)$ the SNR in dB is simply

$$ SNR_\mathrm{dB} = 10 \cdot \log_{10} \frac{\langle x^2 \rangle}{\langle r^2 \rangle}$$ where $\langle \cdot \rangle$ is the mean value operator.

Note that this a power ratio. There no assumption whatsoever about spectral, temporal or amplitude distribution of the noise (or the desired signal). I SOME cases the SNR can indeed be used to estimate the channel capacity. The Shannon channel capacity explicitly assumes that the it's additive white Gaussian noise (AWGN) channel. I.e. the noise is spectrally white, has a Gaussian amplitude distribution and is stationary in time. For different type of noises, things are are different and the simple formula does not hold any more.

if the maximum input voltage to a circuit channel with with noise of $\pm 0.5 V$ is $10 V$ or $SNR=\frac{10}{1}$

That's not a useful way to think about it. First of all you need to specify whether your numbers are peak or RMS (i.e. root mean square) values. The definition of SNR uses RMS, however your argument uses peak value. For Gaussian noise, individual samples or instances of the noise can be much larger than the RMS and there is no easy way to even define a Peak value. Hence it requires a statistical argument.