The Law of Large Numbers

The following theorem states that, the distribution of ${\overline{X}}_n$ becomes more concentrated around $\mu$ as $n$ gets large.

Theorem (The Weak Law of Large Numbers)

If $X_1,\dots ,X_n$ are IID, then ${\overline{X}}_n\stackrel{\mathrm{P}}{\to }\mu$.

Proof

Assume that $\sigma <\infty$. This is not necessary but it simplifies the proof. Using Chebyshev’s inequality,

$$\mathbb{P}(|{\overline{X}}_n-\mu |>ϵ)\le \frac{\mathbb{V}({\overline{X}}_n)}{{ϵ}^2}=\frac{{\sigma }^2}{n{ϵ}^2}$$

which tends to 0 as $n\to \infty$.

Example

Consider flipping a coin for which the probability of heads is $p$. Let $X_i$ denote the outcome of a single toss (0 or 1). Suppose that $p=1/2$. How large should $n$ be so that $\mathbb{P}(0.4\le {\overline{X}}_n\le 0.6)\ge 0.7$? Since $\mathbb{E}({\overline{X}}_n)=p=1/2$ and $\mathbb{V}({\overline{X}}_n)={\sigma }^2/n=p(1-p)/n=1/(4n)$, from Chebyshev’s inequality,

$$\begin{array}{rl}\mathbb{P}(0.4\le {\overline{X}}_n\le 0.6)& =\mathbb{P}(|{\overline{X}}_n-\mu |\le 0.1)\\ & =1-\mathbb{P}(|{\overline{X}}_n-\mu |>0.1)\\ & \ge 1-\frac{1}{4n(0.1{)}^2}=1-\frac{25}{n}.\end{array}$$

The last expression will be larger than 0.7 if $n=84$.