Bootstrap Confidence Intervals

There are several ways to construct bootstrap confidence intervals

Normal Interval

The Normal Interval

The simplest method is the Normal interval

$$T_n\pm z_{\alpha /2}{\widehat{\mathrm{SE}}}_{\mathrm{boot}}$$

where ${\widehat{\mathrm{SE}}}_{\mathrm{boot}}=\sqrt{v_{\mathrm{boot}}}$ is the bootstrap estimate of the standard error.

Warning

This interval is not accurate unless the distribution of $T_n$ is close to Normal.

Pivotal Intervals

Let $\theta =T(F)$ and ${\hat{\theta }}_n=T({\hat{F}}_n)$ and define the pivot $R_n={\hat{\theta }}_n-\theta$. Let ${\hat{\theta }}_{n,1}^{\ast },\dots ,{\hat{\theta }}_{n,B}^{\ast }$ denote the bootstrap replications of ${\hat{\theta }}_n$. Let $H(r)$ denote the CDF of the pivot:

$$H(r)={\mathbb{P}}_F(R_n\le r).$$

Define $C_n^{\ast }(a,b)$ where

$$a={\hat{\theta }}_n-H^{-1}\left(1-\frac{\alpha }{2}\right)\mathrm{and}b={\hat{\theta }}_n-H^{-1}\left(\frac{\alpha }{2}\right).$$

It follows that

$$\begin{array}{rl}\mathbb{P}(a\le \theta \le b)& =\mathbb{P}(a-{\hat{\theta }}_n\le \theta -{\hat{\theta }}_n\le b-{\hat{\theta }}_n)\\ & =\mathbb{P}({\hat{\theta }}_n-b\le {\hat{\theta }}_n-\theta \le {\hat{\theta }}_n-a)\\ & =\mathbb{P}({\hat{\theta }}_n-b\le R_n\le {\hat{\theta }}_n-a)\\ & =H({\hat{\theta }}_n-a)-H({\hat{\theta }}_n-b)\\ & =H\left(H^{-1}\left(1-\frac{\alpha }{2}\right)\right)-H\left(H^{-1}\left(\frac{\alpha }{2}\right)\right)\\ & =1-\frac{\alpha }{2}-\frac{\alpha }{2}=1-\alpha .\end{array}$$

Hence, $C_n^{\ast }$ is an exact $1-\alpha$ confidence interval for $\theta$. Unfortunately, $a$ and $b$ depend on the unknown distribution of $H$, but we can form a bootstrap estimate of $H$:

$$\hat{H}(r)=\frac{1}{B}\sum _{b=1}^B\mathbb{I}(R_{n,b}^{\ast }\le r),$$

where $R_{n,b}^{\ast }={\hat{\theta }}_{n,b}^{\ast }-{\hat{\theta }}_n$. Let $r_{\beta }^{\ast }$ denote the $\beta$ sample quantile of $(R_{n,1}^{\ast },\dots ,R_{n,B})$ and let ${\theta }_{\beta }^{\ast }$ denote the $\beta$ sample quantile of $({\hat{\theta }}_{n,1}^{\ast },\dots ,{\hat{\theta }}_{n,B}^{\ast })$. Note that $r_{\beta }^{\ast }={\theta }_{\beta }^{\ast }-{\hat{\theta }}_n$. It follows that an approximate $1-\alpha$ confidence interval is $C_n=(\hat{a},\hat{b})$ where

$$\begin{array}{rl}\hat{a}& ={\hat{\theta }}_n-{\hat{H}}^{-1}\left(1-\frac{\alpha }{2}\right)={\hat{\theta }}_n-r_{1-\alpha /2}^{\ast }\\ & =2{\hat{\theta }}_n-{\theta }_{1-\alpha /2}^{\ast }\\ \hat{b}& ={\hat{\theta }}_n-{\hat{H}}^{-1}\left(\frac{\alpha }{2}\right)={\hat{\theta }}_n-r_{\alpha /2}^{\ast }\\ & =2{\hat{\theta }}_n-{\theta }_{\alpha /2}^{\ast }.\end{array}$$

Bootstrap Pivotal Confidence Interval

The $1-\alpha$ bootstrap pivotal confience interval is

$$C_n=\left(2{\hat{\theta }}_n-{\hat{\theta }}_{1-\alpha /2}^{\ast },2{\hat{\theta }}_n-{\theta }_{\alpha /2}^{\ast }\right).$$

Intuition

The pivot $R_n={\hat{\theta }}_n-\theta$ is the estimation error. If we knew the distribution of this error, then we could turn typical errors into a confidence interval for $\theta$ by solving for the values of $\theta$ that make the observed estimate ${\hat{\theta }}_n$ look typical.

The bootstrap mimics this unknown error distribution by replacing the unknown truth $\theta$ with the observed estimate ${\hat{\theta }}_n$. In the bootstrap world, the analogue of the error is

$$R_n^{\ast }={\hat{\theta }}_n^{\ast }-{\hat{\theta }}_n.$$

So the bootstrap pivotal interval says: look at the typical bootstrap errors, then reflect them around ${\hat{\theta }}_n$ to get plausible values of the true parameter. This is why the endpoints have the form

$${\hat{\theta }}_n-r_{1-\alpha /2}^{\ast },{\hat{\theta }}_n-r_{\alpha /2}^{\ast }.$$

Equivalently, if the bootstrap estimates are unusually far above ${\hat{\theta }}_n$, then plausible values of $\theta$ should be correspondingly far below ${\hat{\theta }}_n$, since those values would make the observed estimate have the same size error.

Theorem

Under weak conditions on $T(F)$,

$${\mathbb{P}}_F\left(T(F)\in C_n\right)\to 1-\alpha$$

as $n\to \infty$, where $C_n$ is given above.