The Central Limit Theorem

Theorem (The Central Limit Theorem (CLT))

Let $X_1,\dots ,X_n$ be IID with mean $\mu$ and variance ${\sigma }^2$. Let ${\overline{X}}_n=n^{-1}{\sum }_{i=1}^n{X}_i$. Then

$$Z_n\equiv \frac{{\overline{X}}_n}{\sqrt{\mathbb{V}({\overline{X}}_n)}}=\frac{\sqrt{n}({\overline{X}}_n-\mu )}{\sigma }⇝Z$$

where $Z\sim N(0,1)$. In other words,

$$\underset{n\to \infty }{\lim }\mathbb{P}(Z_n\le z)\Phi (z)={\int }_{-\infty }^z\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2}dx.$$

The Law of Large Numbers and the CLT both describe what happens to the sample mean as the sample size grows, but they emphasize different aspects of that behavior.

• The LLN is a statement about stability: averaging many observations washes out randomness and pulls the sample mean toward the population mean $\mu$.
• The CLT is a statement about the shape of the remaining randomness: after centering at $\mu$ and scaling by $\sqrt{n}$, the error in the sample mean is approximately normal.

In that sense, the LLN explains why the average becomes reliable, while the CLT explains what the leftover uncertainty looks like for a large but finite sample. This is why the LLN is most useful when the main point is that an empirical average should be close to the truth, whereas the CLT is most useful when approximating probabilities or building confidence intervals. Informally, the LLN says averaging makes noise die out, while the CLT says the leftover noise is approximately Gaussian and has size about $1/\sqrt{n}$.

The interpretation is that probability statements about ${\overline{X}}_n$ can be approximated using a Normal distribution. Its the probability statements that we are approximating, not the random variable itself.

In addition to $Z_n⇝N(0,1)$, there are several forms of notation to denote that the distribution of $Z_n$ is converging to a Normal. They all mean the same thing:

$$\begin{array}{rl}{Z}_n& \approx N(0,1)\\ {\overline{X}}_n& \approx N\left(\mu ,\frac{{\sigma }^2}{n}\right)\\ {\overline{X}}_n-\mu & \approx N\left(0,\frac{{\sigma }^2}{n}\right)\\ \sqrt{n}({\overline{X}}_n-\mu )& \approx N(0,{\sigma }^2)\\ \frac{\sqrt{n}({\overline{X}}_n)}{\sigma }& \approx N(0,1).\end{array}$$

Example

Suppose that the number of errors per computer program has a Poisson distribution with mean 5. We get 125 programs. Let $X_1,\dots ,X_{125}$ be the number of errors in the programs. We want to approximate $\mathbb{P}({\overline{X}}_n<5.5)$. Let $\mu =\mathbb{E}(X_1)=\lambda =5$ and ${\sigma }^2=\mathbb{V}(X_1)=\lambda =5$. Then,

$$\begin{array}{rl}\mathbb{P}({\overline{X}}_n<5.5)& =\mathbb{P}\left(\frac{\sqrt{n}({\overline{X}}_n-\mu )}{\sigma }<\frac{\sqrt{n}(5.5-\mu )}{\sigma }\right)\\ & \approx \mathbb{P}(Z<2.5)=\mathrm{0.9938.}\end{array}$$

One important thing to note is that we rarely know $\sigma$. However, we can estimate ${\sigma }^2$ from $X_1,\dots ,X_n$ by

$$S_n^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-{\overline{X}}_n{)}^2.$$

Interestingly, if $\sigma$ is replaced with $S_n$, the central limit theorem still holds:

Theorem

Assume the same conditions as the CLT. Then,

$$\frac{\sqrt{n}({\overline{X}}_n-\mu )}{S_n}⇝N(0,1).$$