I'm currently playing around with some compression algorithms and I'm asking myself if there is a type of data distribution / noise distribution that is easier to target with quantization (meaning less distortions at same rate). To my understanding i.d.d. Gaussian is the upper bound on "compression difficulty".
Fat32's answer and your distortion aspect are on-spot here:
The continuous Gaussian distribution is the distribution that maximizes differential entropy.
But things get a bit hairy in your general problem:
a type of data distribution / noise distribution that is easier to target with quantization (meaning less distortions at same rate).
discrete input, lossless compression
So, I'd argue that a data distribution is already given in discrete quantities – and for discrete sources it's trivial to show that the minimum entropy is 0 (discrete information is non-negative, and hence so is its expectation, the entropy), and can be realized by a source where all but one element have probability 0 (and thus, the remaining element probability 1), since then the expectation value of probability-weighed informations collapses to the information in the only occurring value, and $\log_2(P=1)\equiv 0$.
Hence, the source that always gives the same value has 0 bit of information, and can be compressed the best. By simply ignoring the source. As it gives no information.
Now assume you've got a discrete source of information that you want to compress losslessly. Can you map that to a different distribution that has lower entropy without losing info? No, you can't.
- Either you find a mapping that combines multiple input values into one output value, which means you lose information, and hence are lossy,
- or you only increase the entropy (or, best case, you just keep the same entropy), by splitting events that are identical into multiple bins, thus approximating the discrete uniform source more, which has maximum entropy.
So, losslessly, you can't do anything to improve your source if its i.i.d.; compression that works better than a plain Huffman codec on average (for example, lossless audio codecs) uses the fact that observations aren't independent; no matter what you do, your losslessly compressed data still has at least as much bits as there was entropy in the source (that's a fundamental result of basic information theory).
Now, the more interesting case is that of noise, or signal, in the continuous case.
continuous source, quantization
The problem you're describing, *how much information do I lose when I subject a source of information to a specific quantization" is called Rate-Distortion Theory. It's a fun field full of fun integrals!
So, let's start with a short consideration of what quantization is:
It takes a continuous source and maps it to discrete case. Quite intuitively, this is a lossy process (or your source wasn't really continuous from the start): The probability of any one of the continuously possible values is (surely) 0 – thus, the information in the event "value $v$ has been observed" is $-\log_2\left(P_X(v)\right)=-\log_2(0)$, and that is unbound (the event "has infinite information", but I hesitate saying that, because "infinite" is not something you should quantize things with). Thus, the continous source has unbounded entropy in the discrete information theory sense – and since any quantization in this world can only give you a finite amount of bits, you're bound to lose some of the original information. (I could've explained that more plastically – how do you quantize $e$, $3$ and $\pi$ with the same quantizer and still "hit" all three values exactly?)
So, the question is, what continuous distribution $F_X$ of the source $X$ suffers the least distortion when being sampled (by a quantization function $Q: \{x\in X\}\mapsto \{y\in Y\}$ with $Q^{-1}(y)=x\;\forall y\in Y$, i.e. a quantization that leaves the values it can "exactly" produce untouched) with a fixed bit depth $r$, giving us the discrete source $Y$?
Now, following the usual Rate-Distortion theory approach, we should first define a distortion function $D$ that tells us how "far" the quantized value $x$ is from the input value $y$, but we don't have to: any reasonable metric would have the property of being $D=0$ for $y=x$, and always $D\ge0$.
Hence, the infimum of the distortion would occur under optimal distribution $\tilde F_X$ of $X$ (let's assume said distribution is differentiable to its density $\tilde f$, else things just get uglier):
\begin{align}
d &=
\inf_{F_X} E\left[D(X,Y=Q(X))\right]\\
&= \inf_{F_X} \int\limits_X f_X(x)D(x,y=Q(x))\,\mathrm{d}x \\
&= \int\limits_X \tilde f_X(x)D(x,Q(x))\,\mathrm{d}x \\
&= \int\limits_{\left\{x\in X| Q(x)=x\right\}} \tilde f_X(x)D(x,Q(x))\,\mathrm{d}x +
\int\limits_{X \backslash \left\{x\in X| Q(x)=x\right\}} \tilde f_X(x)D(x,Q(x))\,\mathrm{d}x \\
&= \int\limits_{X \backslash \left\{x\in X| Q(x)=x\right\}} \tilde f_X(x)D(x,Q(x))\,\mathrm{d}x \\
&=\int\limits_{X \backslash \left\{x\in X| Q(x)=x\right\}} \tilde f_X(x)D(x,Q(x))\,\mathrm{d}x \\
&= 0 \\
&\iff \tilde f(x) = 0\; \forall \notin Y\\
&\implies \tilde F_X = F_Y
\end{align}
So, this might not be a surprising result, but it's quite fundamental to understand: No matter how you (sensibly) define distortion/loss, the closer you get to the output distribution of your quantizer, the less distortion you see.
Optimizing for something better than just transmitting the least bits possible
With that result, we can move on: we already know the optimally compressible output distribution $F_Y$, namely the constant output mentioned above, so the optimal continous source distribution in terms of compression is just the same.
Assuming you really want your quantizer to give you an $r$-bit observation of reality, you'd build it to have $2^r$ steps of equal probability, and since "wasting" steps is a bad idea, we just pick $r$ to be an integer.
Hence, you'd look at the source distribution and find a continuous mapping so that the output of $Q$ is discretely uniform.
That means you'd just find a mapping so that the density mass in each of the quantization bins is equal.
In the uniform step width quantization ("linear ADC"), that means finding the inverse of $F_X(x)$.
