The Empirical Distribution Function

Definition (Empirical Distribution Function)

The empirical distribution function ${\hat{F}}_n$ is the CDF that puts mass $1/n$ at each data point $X_i$. Formally,

$${\hat{F}}_n(x)=\frac{{\sum }_{i=1}^n\mathbb{I}\{X_i\le x\}}{n}$$

where $\mathbb{I}$ is the indicator function.

Theorem

For any fixed value of $x$,

$$\begin{array}{rl}\mathbb{E}\left[{\hat{F}}_n(x)\right]& =F(x),\\ \mathbb{V}\left({\hat{F}}_n(x)\right)& =\frac{F(x)(1-F(x))}{n},\\ \mathrm{MSE}& =\frac{F(x)(1-F(x))}{n}\to 0,\\ {\hat{F}}_n(x)& \stackrel{\mathrm{P}}{\to }F(x).\end{array}$$

The DKW Inequality

The following inequality bounds the probability that the random function ${\hat{F}}_n$ differs from $F$ by more than given constant $ϵ>0$ anywhere on the real line.

Theorem (The Dvoretzky-Kiefer-Wolfowitz (DKW) Inequality)

Let $X_1,\dots ,X_n\sim F$. Then, for any $ϵ>0$,

$$\mathbb{P}\left(\underset{x}{\sup }|F(x)-{\hat{F}}_n(x)|>ϵ\right)\le 2e^{-2n{ϵ}^2}.$$

Building CDF Bands

From the DKW inequality, we can construct a confidence bound as follows:

Definition (A Nonparametric $1-\alpha$ Confidence Band for $F$)

Define,

$$\begin{array}{rl}L(x)& =\max \{{\hat{F}}_n(x)-{ϵ}_n,0\}\\ U(x)& =\min \{{\hat{F}}_n(x)+{ϵ}_n,1\}\\ \mathrm{where}{ϵ}_n& =\sqrt{\frac{1}{2n}\log \left(\frac{2}{\alpha }\right)}.\end{array}$$

It follows that for any $F$,

$$\mathbb{P}\left(L(x)\le F(x)\le U(x)\text{for all}x\right)\ge 1-\alpha .$$