Central Limit Theorem

Let $x_1, x_2, \ldots, x_N$ be a set of $N$ Independent random variates and each $x_i$ have an arbitrary probability distribution $P(x_1,\ldots,x_N)$ with Mean $\mu_i$ and a finite Variance ${\sigma_i}^2$. Then the normal form variate

$$X_{\rm norm} \equiv {\sum_{i=1}^N x_i - \sum_{i=1}^N \mu_i\over\sqrt{\sum_{i=1}^N {\sigma_i}^2}}$$ (1)

has a limiting distribution which is Normal (Gaussian) with Mean $\mu=0$ and Variance $\sigma^2 = 1$. If conversion to normal form is not performed, then the variate

$$X\equiv{1\over N}\sum_{i=1}^N x_i$$ (2)

is Normally Distributed with $\mu_X=\mu_x$ and $\sigma_X={\sigma_x/\sqrt{N}}$. To prove this, consider the Inverse Fourier Transform of $P_X(f)$.

$${\mathcal F}^{-1}[P_X(f)] \equiv \int_{-\infty}^\infty e^{2\pi i f X}p(X)\,dX$$

$$= \int_{-\infty}^\infty \sum_{n=0}^\infty {(2\pi i f X)^n\over n!} p(X)\,dX$$

$$= \sum_{n=0}^\infty {(2\pi i f)^n\over n!} \int_{-\infty}^\infty X^np(X)\,dX$$

$$= \sum_{n=0}^\infty {(2\pi i f)^n\over n!} \left\langle{X^n}\right\rangle{}.$$ (3)

Now write

$\left\langle{X^n}\right\rangle{} = \left\langle{N^{-n}(x_1+x_2+\ldots+x_N)^n}\right\rangle{}$
$= \int_{-\infty}^\infty N^{-n}(x_1+\ldots+x_N)^n p(x_1)\cdots p(x_N)\, dx_1\cdots dx_N,$
	(4)

so we have

${\mathcal F}^{-1}[P_X(f)] = \sum_{n=0}^\infty {(2\pi i f)^n\over n!}\left\langle{X^n}\right\rangle{}$
$= \sum_{n=0}^\infty {(2\pi i f)^n\over n!} \int_{-\infty}^\infty N^{-n}(x_1+\ldots+x_N)^n p(x_1)\cdots p(x_N)\,dx_1\cdots dx_N$
$= \int_{-\infty}^\infty \sum_{n=0}^\infty \left[{2\pi i f(x_1+\ldots+x_N)\over N}\right]^n {1\over n!} p(x_1)\cdots p(x_N)\,dx_1\cdots dx_N$
$= \int_{-\infty}^\infty e^{2\pi i f(x_1+\ldots+x_N)/N} p(x_1)\cdots p(x_N)\,dx_... ... dx_N \cdots \left[{\int_{-\infty}^\infty e^{2\pi ifx_N/N} p(x_N)\,dx_N}\right]$
$= \left[{\int_{-\infty}^\infty e^{2\pi i f x/N}p(x)\,dx}\right]^N$
$= \left\{{\int_{-\infty}^\infty \left[{1+\left({2\pi i f \over N}\right)x +{1\over 2}\left({2\pi i f \over N}\right)^2 x^2 +\ldots}\right]p(x)\,dx}\right\}^N$
$= \left[{\int_{-\infty}^\infty p(x)\,dx+{2\pi i f\over N}\int_{-\infty}^\infty ... ...)^2\over 2N^2} \int_{-\infty}^\infty x^2p(x)\,dx+{\mathcal O}(N^{-3})}\right]^N$
$= \left[{1+{2\pi i f\over N}\left\langle{x}\right\rangle{}-{(2\pi f)^2\over 2N^2}\left\langle{x^2}\right\rangle{}+{\mathcal O}(N^{-3})}\right]^N$
$= \mathop{\rm exp}\nolimits \left\{{N\ln\left[{1+{2\pi i f\over N}\left\langle{... ...er 2N^2}\left\langle{x^2}\right\rangle{}+{\mathcal O}(N^{-3})}\right]}\right\}.$	(5)

Now expand

$$\ln(1+x)=x-{\textstyle{1\over 2}}x^2+{\textstyle{1\over 3}}x^3+\ldots,$$ (6)

so

$${\mathcal F}^{-1}[P_X(f)] \approx \mathop{\rm exp}\nolimits \left\{{N\left[{{2\pi if\over N}\left\l... ...ver N^2}\left\langle{x}\right\rangle{}^2 +{\mathcal O}(N^{-3})}\right]}\right\}$$

$$= \mathop{\rm exp}\nolimits \left[{2\pi if\left\langle{x}\right\ran... ...angle{}-\left\langle{x}\right\rangle{}^2)\over 2N}+{\mathcal O}(N^{-2})}\right]$$

$$\approx \mathop{\rm exp}\nolimits \left[{2\pi if\mu_x-{(2\pi f)^2{\sigma_x}^2\over 2N}}\right],$$ (7)

since

$$\mu_x \equiv \left\langle{x}\right\rangle{}$$ (8)

$${\sigma_x}^2 \equiv \left\langle{x^2}\right\rangle{}-\left\langle{x}\right\rangle{}^2.$$ (9)

Taking the Fourier Transform,

$$P_X \equiv \int_{-\infty}^\infty e^{-2\pi ifx}{\mathcal F}^{-1}[P_X(f)]\,df$$

$$= \int_{-\infty}^\infty e^{2\pi if(\mu_x-x)-(2\pi f)^2{\sigma_x}^2/2N}\,df.$$ (10)

This is of the form

$$\int_{-\infty}^\infty e^{iaf-bf^2}\,df,$$ (11)

where $a\equiv 2\pi(\mu_x-x)$ and $b\equiv (2\pi\sigma_x)^2/2N$. But, from Abramowitz and Stegun (1972, p. 302, equation 7.4.6),

$$\int_{-\infty}^\infty e^{iaf-bf^2}\,df = e^{-a^2/4b} \sqrt{\pi\over b}.$$ (12)

Therefore,

$$P_X = \sqrt{\pi\over{(2\pi\sigma_x)^2\over 2N}} \mathop{\rm exp}\nolimits \left\{{-[{2\pi(\mu_x-x)]^2}\over {4{(2\pi\sigma_x)^2\over 2N}}}\right\}$$

$$= \sqrt{2\pi N\over 4\pi^2{\sigma_x}^2} \mathop{\rm exp}\nolimits \left[{-{4\pi^2(\mu_x-x)^2 2N\over 4\cdot 4\pi^2{\sigma_x}^2}}\right]$$

$$= {\sqrt{N}\over\sigma_x\sqrt{2\pi}} e^{-(\mu_x-x)^2N/2{\sigma_x}^2}.$$ (13)

But $\sigma_X={\sigma_x/\sqrt{N}}$ and $\mu_X=\mu_x$, so

$$P_X = {1\over\sigma_X\sqrt{2\pi}} e^{-(\mu_X-x)^2/2{\sigma_X}^2}.$$ (14)

The ``fuzzy'' central limit theorem says that data which are influenced by many small and unrelated random effects are approximately Normally Distributed.

See also Lindeberg Condition, Lindeberg-Feller Central Limit Theorem, Lyapunov Condition

References

Abramowitz, M. and Stegun, C. A. (Eds.). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th printing. New York: Dover, 1972.

Spiegel, M. R. Theory and Problems of Probability and Statistics. New York: McGraw-Hill, pp. 112-113, 1992.

Zabell, S. L. ``Alan Turing and the Central Limit Theorem.'' Amer. Math. Monthly 102, 483-494, 1995.