0
$\begingroup$

I understand Central limit theorem, but cannot understand the result when it applies to diverse situation.

As far as I understand, Central limit theorem states follows:

  • No matter what random variable has its distribution, if we pick n samples and mean (or just sum) them and do this many time, those values will follow normal distribution (if n is big enough).

And there is a good example for this: a Brownian motion. If we measure the distance that particles moved suspending in a fluid, distance will follow normal distribution because sum of random hit(movement) of fluid particle will follow normal distribution according to Central limit theorem.

However, for central limit theorem applies, random variables which are merged should follow all the same means and variances. In upper example, all random particle movement will follow same mean and variances, because they have same temperature.

But as far as I understand, there are many cases where noise follows normal distribution but there components are not necessarily follow same mean.

(S_n = (X_1 + X_2 + ... X_n) / n and S_n follows normal distribution but X_i s have all DIFFERENT means)

For example, electrical noise will be composed of many other things which are not necessarily follow equal to each mean. But why electrical noise can be approximated as gaussian noise? Not only electrical noise, why many other things are treated as normal distribution(S_n follows normal distribution) although their components don't have equal mean?(X_i s have all DIFFERENT mean and variances)

Thank you for reading!

$\endgroup$
2
  • $\begingroup$ To be completely honest, this is a candidate for math.SE, not really for signals.SE, but: your definition is not sufficient, you need to bound the variance of the RV. The CLT has more general forms (see the English CLT wikipedia article, it actually lists Lyapunov CLT), and with these you can make the statements you want to make. In general, no, not every sum of independent RVs approaches normal distribution. $\endgroup$ – Marcus Müller Feb 27 at 15:53
  • 1
    $\begingroup$ I'm going to reiterate something @MarcusMüller said: the CLT does not apply at all to random variables with infinite variance. Such things do exist, and you'll never sum (or average) them to a Gaussian. Also, if you have random variables with a long-tail distribution then taking an average over just a few samples will not work in practice -- you need to average enough samples so that the "long tail" gets smoothed out along with everything else. $\endgroup$ – TimWescott Feb 27 at 17:12
2
$\begingroup$

You can add random variables with different means and variances. For example, look at two continuous uncorrelated normal-distributed random variables $x_1$ and $x_2$ with means and variances $\mu_1, \sigma_1^2$ and $\mu_2, \sigma_2^3$

Then $x_3 = x_1 + x_2 $ is also a normal-distributed random variable with $\mu_3 = \mu_1 + \mu_2$ and $ \sigma_3^2 = \sigma_1^2 + \sigma_2^2 $. For uncorrelated random variables the means and variances just sum.

The distribution of a sum of two continuous random variables is the convolution of the individual distributions. That's actually the core of the central limit theorem. If you convolve a Gaussian with a Gaussian you get another Gaussian (see for example https://jeremy9959.net/Math-5800-Spring-2020/notebooks/convolution_of_gaussians.html) . If you keep convolving reasonably looking functions with each other eventually it will approach a Gaussian.

A simple example is summing a few uniformly distributed random variables. The first one has a rectangular distribution. The second one is triangular. The third one already shows a nice round belly in the middle and once you are up to summing 10 or 12 it looks very much Gaussian. That's in fact one method how gaussian random number generators are implemented.

For the Central Limit Theorem, there is no requirement for random variables to have the same mean or variance.

$\endgroup$
18
  • $\begingroup$ I'd actually be careful. Had something like this before (this is from a especially dark fold of my brain, so treat with suspicion): $X_i \in\{-a, a, 0\},\, i=1,\ldots,N,\, N < a\in\mathbb R, \quad P(X_i=a)=P(X_i=-a)=e^{-i^2}$; then for $S_N=\sum_{i=1}^N X_i$, the cdf $\lim_{N\to\infty} F_{S_N}(s)$ doesn't converge to a normal cdf, although the variances are all bounded and the means are all 0. (this arises when adding up things that get increasingly less likely, like errors that slip through multiple regression tests) $\endgroup$ – Marcus Müller Feb 27 at 16:44
  • 1
    $\begingroup$ Agreed. I was trying to keep it reasonably "simple" and cover the majority of cases. Hence I restricted it to continuous random variables in the first place. $\endgroup$ – Hilmar Feb 27 at 18:36
  • $\begingroup$ Fair point! You can run into trouble with these nevertheless (just give these $X_i$ a normal, but low-variance "skirt", and it still fails), but it's less commonly a problem in continuous variables. $\endgroup$ – Marcus Müller Feb 27 at 19:30
  • $\begingroup$ @Hilmar: I know that you know your stuff big time and always appreciate your great insights ( even when I don't understand them ) but the first 2 paragraphs in your reply only hold for normal random variables. You cannot obtain the mean and variance of the addition of any two rv's by adding their means and variances. I felt obligated to say this in case future readers use read thread for something. Oh, your last 3 paragraphs are el perfecto. And to summarize, the OP's question just needs that property for normal RV's that you described in the first two paragraphs. $\endgroup$ – mark leeds Feb 28 at 2:45
  • 1
    $\begingroup$ @markleeds I would have thought (and could certainly be wrong!), for uncorrelated random variables, it would hold-- the mean for sure- do you have an example where the variances would not actually sum (importantly for the uncorrelated cases as he wrote)? $\endgroup$ – Dan Boschen Feb 28 at 2:57
1
$\begingroup$

I think we need to be a fair bit more specific:

No matter what random variable has its distribution,

… If that distribution has a finite variance and mean (counterexample: Cauchy-distributed variables, e.g. $\operatorname*{Im}(z)/\operatorname*{Re}(z)$ of complex normal $z$),

if we pick n samples and mean (or just sum) them and do this many time, those values will follow normal distribution (if n is big enough).

It's called central limit theorem for a reason: the distribution of the of the sum of normalized, centralized iid RVs, divided by the square root of the number of summands, converges against the standard normal distribution:

$$ \lim_{n\to\infty}P\left(\frac{1}{\sqrt{n}}\sum_{i=1}^n (X_i-E(X_i))/\sigma_i \le x\right)=\Phi(x)$$

(and if you don't want to normalize, just multiply both sides of that inequality with a factor of the $\sqrt{\sum\sigma_i^2}$; you can work with noncentrals simply by adding to the value: Things are nice and linear)

That doesn't mean it will be exactly a normal distribution for any finite $n$. I mean, you wouldn't expect the sum of $n=3$ random variables to be normally distributed, right. There's some distributions where the convergence speed is high, there's others where it's low.

$$S_n = \frac{X_1 + X_2 + \ldots + X_n}n$$ and $S_n$ follows normal distribution but $X_i$ have all DIFFERENT means

Absolutely no problem here! When summing up two independent random variables (RVs) with probability density functions (pdf) $f_a$ and $f_b$, respectively, then the sum has a density that's the convolution of each.

Now, a normal distribution with variance $\sigma_i^2$ and mean $\mu_i$ has pdf

$$f_i(x) = \frac{1}{\sigma_i \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu_i}{\sigma_i}\right)^2},$$

and if you convolve two of these, you'll see that the convolution is again a normal variable, with $\sigma_{\Sigma}^2=\sigma_1^2+\sigma_2^2$ and $\mu_\Sigma=\mu_1+\mu_2$; you need no CLT for that at all, that's just stubborn evaluation of the convolution (or going to Fourier domain, doing a multiplcation there, and then back, which is generally considered less work).

$\endgroup$
6
  • $\begingroup$ Lots of mis-statements here. The Weak Law of Large Numbers assures us that the average of $N$ i.i.d. finite variance random variables converges to a constant (the common mean of the random vairiables) and not to a nornal distribution. What the CLT actually says is far different from what the OP thinks it does; please don't follow him down that rabbit hole. $\endgroup$ – Dilip Sarwate Feb 28 at 21:44
  • $\begingroup$ @DilipSarwate Is what Marcus said actually incorrect? As $N$ approaches infinity, wouldn't the distribution approach a Gaussian within the constraints he gave? $\endgroup$ – Dan Boschen Mar 1 at 3:40
  • $\begingroup$ [Redoing this is a slightly more relaxed mindset]: You're right, I shouldn't have used the word "average" there. That just confuses, as the convergence there is to a normal distribution of arbitrarily low variance; you have to be a bit careful with what the WLoLN says, though, as it can lead to increased blood pressure in your peers :) The WLoLN just states that the probability of getting an average that differs from the mean is lower than any positive bound, and that is fully compatible with the classical CLT. You were giving me real moments of terror there! $\endgroup$ – Marcus Müller Mar 1 at 7:18
  • $\begingroup$ @DanBoschen My comment applied to the original version of this answer. The revised version is much better in many respects but still has incorrect statements. The variance of a sum of independent random variables is the sum of the variances, and so the $S_n$ that Marcus defines has variance $\dfrac{\sigma^2}{n}\to 0$ as $n$ increases. Hence, $S_n$ is converging to the constant $\mu$ as the WLLN tells us, and Marcus acknowledges in his comment. $\endgroup$ – Dilip Sarwate Mar 1 at 22:42
  • $\begingroup$ @DilipSarwate Thanks Dilip-- I didn't see the original response so that all makes sense to me now. $\endgroup$ – Dan Boschen Mar 1 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.