Bootstrapping

Definition

The bootstrap is a method for estimating standard errors and computing confidence intervals.

Let $T_n=g(X_1,\dots ,X_n)$ be a statistic, that is, $T_n$ is any function of the data. Suppose we want to know ${\mathbb{V}}_F(T_n)$, the variance of $T_n$. Note that the subscript $F$ emphasizes that the variance usually depends on the unknown distribution $F$. For example, if $T_n={\bar{X}}_n$, then ${\mathbb{V}}_F(T_n)={\sigma }^2/n$ where ${\sigma }^2=\int (x-\mu {)}^2/dF(x)$ and $\mu =\int xdF(x)$. Thus, the variance of $T_n$ is a function of $F$.

Intuition

If we only use the observed sample directly, we can certainly compute the statistic itself, such as the sample mean, median, or correlation. But that gives us only one realized value of the statistic. It does not tell us how much that statistic would change if we drew a different sample from the population.

That is the real goal of the bootstrap: not to recompute the statistic for its own sake, but to approximate the sampling distribution of the statistic. Once we have an approximation to that sampling distribution, we can estimate standard errors, variances, and confidence intervals.

Since the true population distribution $F$ is unknown, we replace it by the empirical distribution ${\hat{F}}_n$, which puts mass $1/n$ on each observed data point. Resampling with replacement from the observed sample is exactly the same as drawing from ${\hat{F}}_n$. Repeating this many times lets us see how $T_n$ would vary if ${\hat{F}}_n$ were the population.

The bootstrap has two steps:

1. Estimate ${\mathbb{V}}_F(T_n)$ with ${\mathbb{V}}_{{\hat{F}}_n}(T_n)$.
2. Approximate ${\mathbb{V}}_{{\hat{F}}_n}(T_n)$ using simulation

For $T_n={\bar{X}}_n$, we have for Step 1 that ${\mathbb{V}}_{{\hat{F}}_n}(T_n)={\hat{\sigma }}^2/n$, where ${\hat{\sigma }}^2=n^{-1}{\sum }_{i=1}^n(X_i-{\bar{X}}_n)$. In this case, Step 1 is enough. However, in more complicated cases, we cannot write down a simple formula for ${\mathbb{V}}_{{\hat{F}}_n}(T_n)$ which is why we need Step 2.

Simulation

Suppose we draw an IID sample $Y_1,\dots ,Y_B$ from a distribution $G$. By the Law of Large Numbers,

$${\hat{Y}}_n=\frac{1}{B}\sum _{j=1}^B{Y}_j\stackrel{\mathrm{P}}{\to }\int ydG(y)=\mathbb{E}(Y)$$

as $B\to \infty$. So if we draw a large sample from $G$, we can use the sample mean ${\bar{Y}}_n$ to approximate $\mathbb{E}(Y)$. In a simulation, we can make $B$ as large as we like, in which case the difference between ${\hat{Y}}_n$ and $\mathbb{E}(Y)$ is negligible.

More generally, if $h$ is any function with finite mean, then

$$\frac{1}{B}\sum _{j=1}^{B}h(Y_j)\stackrel{\mathrm{P}}{\to }\int h(y)dG(y)=\mathbb{E}[h(Y)]$$

as $B\to \infty$. In particular,

$$\begin{array}{rl}\frac{1}{B}\sum _{j=1}^B(Y_j-\bar{Y}{)}^2& =\frac{1}{B}\sum _{j=1}^B{Y}_j^B-{\left(\frac{1}{B}\sum _{j=1}^B{Y}_j\right)}^2\\ & \stackrel{\mathrm{P}}{\to }\int y^2/dF(y)-{\left(\int ydF(y)\right)}^2=\mathbb{V}(Y).\end{array}$$

Hence, we can use the sample variance of the simulated values to approximate $\mathbb{V}(Y)$.

Limitations

The bootstrap is powerful, but it is not automatic or universally reliable.

• It only uses the observed sample. If the sample is biased or unrepresentative, the bootstrap inherits that problem.
• With small samples, the empirical distribution ${\hat{F}}_n$ may be a poor approximation to the true distribution $F$, so the bootstrap approximation can be inaccurate.
• It cannot recover behavior that is missing from the data, such as rare events or poorly observed tail behavior.
• The ordinary bootstrap assumes the data are IID. For dependent data, such as time series or clustered observations, the usual bootstrap should not be used without modification.
• Some statistics are not well approximated by the bootstrap, especially irregular or non-smooth ones such as maxima, minima, or post-model-selection estimators.
• Bootstrap confidence intervals can have poor coverage in small samples or highly skewed settings unless more careful variants are used.
• It can be computationally expensive when the statistic is costly to recompute many times.

So the bootstrap is most useful when the sample is reasonably representative, the IID assumption is plausible, and the statistic has a complicated sampling distribution but behaves regularly.