# Mean square error for a quantizer - problem in integral

There are similar questions about related problems that I honestly didn't understand. I believe this might be a lack of statistics knowledge of my part. Quantization error standard deviation Error analysis for uniform quantization of uniform input Signal to Quantization Noise ratio concept

I was reading David R. Smith's Digital Transmission Systems and in it I found the following derivation for the average error due to quantization.

I don't understand going from 3.8 to 3.9 It might be a math problem on my end.

The last two integrals will disappear, because they correspond to the overload regions of the quantizer, and we are assuming that the input signal, x, does not have values in that region, meaning, in the last 2 integrals $$p(x)=0$$ Thus, they are irrelevant.

Looking at just the first integral (ignoring the summation, because it doesn't bother me), for those that can't see the picture we have

$$\int_{x_{j-1}}^{x_j} (y_j - x)^2 p(x) dx$$

$$x_j$$ and $$x_{j-1}$$ correspond to input values of the quantizer that are the limits of a region in which the output of the quantizer is $${y_j}$$. For example, if the input of the quantizer is between 2V to 3V let's say the output of the quantizer is 2.5V. $$x_j$$ for this case is 3V,$$x_{j-1}$$ is 2V and {y_j} is 2.5V $$p(x)$$ is just the probability that the input signal takes on the value $$x$$, so $$p(2.3)$$ is the probability that the input signal takes on the value 2.3V

Now for the following step we assume that $$p(x)$$ is a constant equal to $$p(y_j)$$ That I understood, but I can't understand how (I removed the constant probability from the integral to simplify) $$\int_{x_{j-1}}^{x_j} (y_j - x)^2 dx$$ turned into $$\frac{(x_j-x_{j-1})^2}{12} * (x_j-x_{j-1})$$

Where did the $$y_j$$ go?

• It's a good question. This treatment is not very good. There needs to be an explicit relation that says something like $$y_j = (x_j - x_{j-1}) \left\lfloor \frac{x}{x_j - x_{j-1}} \right\rfloor$$ where $\lfloor \cdot \rfloor$ is the floor( ) function (rounds down to the nearest integer). May 10 at 19:01
• @robertbristow-johnson thank you so much! I was feeling very stupid :( May 10 at 21:49

$$y_j=\frac{x_j+x_{j-1}}{2}$$
\begin{align}\int_{x_{j-1}}^{x_j}(y_j-x)^2dx&=\frac{(y_j-x_{j-1})^3-(y_j-x_{j})^3}{3}\\&=\frac{\left(\frac{x_j+x_{j-1}}{2}-x_{j-1}\right)^3-\left(\frac{x_j+x_{j-1}}{2}-x_{j}\right)^3}{3}\\&=\frac{\left(\frac{x_j-x_{j-1}}{2}\right)^3-\left(\frac{x_{j-1}-x_j}{2}\right)^3}{3}\\&=\frac{(x_j-x_{j-1})^3}{12}\end{align}