Confidence Intervals

Definition (Confidence Interval)

A $1-\alpha$ confidence interval for a parameter $\theta$ is an interval $C_n=(a,b)$ where $a=a(X_1,\dots ,X_n)$ and $b=b(X_1,\dots ,X_n)$ are functions of the data such that

$$\begin{array}{rl}{\mathbb{P}}_{\theta }(\theta \in C_n)& ={\mathbb{P}}_{\theta }(a<\theta <b)\\ & \ge 1-\alpha \text{for all}\theta \in \Theta \end{array}$$

Important

$C_n$ is random and $\theta$ is fixed!

Commonly, people use 95% confidence intervals, which corresponds to choosing $\alpha =0.05$.

Warning

A confidence interval is not a probability statement about $\theta$ since $\theta$ is a fixed quantity, not a random variable. A way to interpret confidence intervals is that if we continue constructing confidence intervals for a sequence of unrelated parameters ${\theta }_1,{\theta }_2,\dots$, then 95 percent of the intervals will trap the true parameter value.

Methods of Derivation

Normal-based Confidence Intervals

Point estimators often have a limiting normal distribution, that is, ${\hat{\theta }}_n\approx N(\theta ,\widehat{{\mathrm{SE}}^2})$. In this case, we can construct (approximate) confidence intervals as follows.

Theorem (Normal-based Confidence Interval)

Suppose that ${\hat{\theta }}_n\approx N(\theta ,\widehat{{\mathrm{SE}}^2})$. Let $\Phi$ be the CDF of the standard Normal and let $z_{\alpha /2}={\Phi }^{-1}(1-(\alpha /2))$, that is, $\mathbb{P}(Z>z_{\alpha /2})=\alpha /2$ and $\mathbb{P}(-z_{\alpha /2}<Z<z_{\alpha /2})=1-\alpha$ where $Z\sim N(0,1)$. Let

$$C_n=({\hat{\theta }}_n-z_{\alpha /2}\widehat{\mathrm{SE}},{\hat{\theta }}_n+z_{\alpha /2}\widehat{\mathrm{SE}}).$$

Then,

$$\mathbb{P}(\theta \in C_n)\to 1-\alpha .$$

Note

The normal-based confidence interval is appropriate when the estimator is approximately Normal, usually because it has a limiting normal distribution and the sample size is large enough for the approximation to be accurate. The main source of this approximation is the Central Limit Theorem, which says that averages of IID random variables with finite variance are approximately Normally distributed for large $n$. This is why normal-based confidence intervals are so common for sample means, sample proportions, and estimators built from averages. More generally, one also uses this method for estimators that are asymptotically normal.

In practice, this is most reasonable when the observations are independent (or only weakly dependent), the variance is finite, and the sampling distribution of ${\hat{\theta }}_n$ is not too skewed at the given sample size. One should be more cautious for small samples, highly skewed or heavy-tailed data, or parameters near the boundary of the parameter space.

Proof

Let $Z_n=({\hat{\theta }}_n-\theta )/\widehat{\mathrm{SE}}$. By assumption $Z_n⇝Z$ where $Z \sim N(0,1)$$. Hence,

$$\begin{array}{rl}{\mathbb{P}}_{\theta }(\theta \in C_n)& ={\mathbb{P}}_{\theta }\left({\hat{\theta }}_n-z_{\alpha /2}\widehat{\mathrm{SE}}<\theta <{\hat{\theta }}_n+z_{\alpha /2}\widehat{\mathrm{SE}}\right)\\ & ={\mathbb{P}}_{\theta }\left(-z_{\alpha /2}<\frac{{\hat{\theta }}_n-\theta }{\widehat{\mathrm{SE}}}<z_{\alpha /2}\right)\\ & ⇝\mathbb{P}\left(-z_{\alpha /2}<Z<z_{\alpha /2}\right)\\ & =1-\alpha .\end{array}$$

Example

Let $X_1,\dots ,X_n\sim \mathrm{Bernoulli}(p)$ and let ${\hat{p}}_n=n^{-1}{\sum }_{i=1}^n{X}_i$. Then,

$$\begin{array}{rl}\mathbb{V}(\hat{p})& =n^{-2}\sum _{i=1}^n\mathbb{V}(X_i)\\ & =n^{-2}\sum _{i=1}^{n}p(1-p)\\ & =n^{-2}np(1-p)\\ & =\frac{p(1-p)}{n}.\end{array}$$

Hence, $\mathrm{SE}=\sqrt{p(1-p)/n}$ and $\widehat{\mathrm{SE}}=\sqrt{{\hat{p}}_n(1-{\hat{p}}_n)/n}$. By the Central Limit Theorem, ${\hat{p}}_n\approx N(p,\widehat{{\mathrm{SE}}^2})$. Therefore, an approximate $1-\alpha$ confidence interval is

$${\hat{p}}_n\pm z_{\alpha /2}\widehat{\mathrm{SE}}={\hat{p}}_n\pm z_{\alpha /2}\sqrt{\frac{{\hat{p}}_n(1-{\hat{p}}_n)}{n}}.$$