Hypothesis Testing

Suppose that we partition the parameter space $\Theta$ into two disjoint sets ${\Theta }_0$ and ${\Theta }_1$ and that we wish to test

$$H_0:\theta \in {\Theta }_0\mathrm{versus}{H}_1:\theta \in {\Theta }_1.$$

We call $H_0$ the null hypothesis and $H_1$ the alternative hypothesis.

Definition (Null Hypothesis)

The null hypothesis, often denoted $H_0$, is the claim that the effect being studied does not exist. The null hypothesis can also be described as the hypothesis in which no relationship exists between two sets of data or variables being analyzed.

If the null hypothesis exists, any experimentally observed effect is due to change alone.

Definition (Alternative Hypothesis)

In contrast with the null hypothesis, an alternative hypothesis (often denoted $H_A$ or $H_1$) is developed, which claims that a relationship does exist between two variables.

Let $X$ be a random variable, and let $\mathcal{X}$ be the range of $X$. We test a hypothesis by finding a appropriate subset of outcomes $R\subset \mathcal{X}$ called the reject region. If $X\in R$ we reject the null hypothesis; otherwise, we do not reject the null hypothesis.

$$\begin{array}{rl}X\in R& \Rightarrow \text{reject}{H}_0\\ X\notin R& \Rightarrow \text{retain (do not reject)}{H}_0\end{array}$$

Usually, the rejection region $R$ is of the form

$$R=\left\{x:T(x)>c\right\}$$

where $T$ is a test statistic and $c$ is a critical value. The problem in hypothesis testing is to fund an appropriate test statistic $T$ and an appropriate critical value $c$.

For a very common example of this template, see The Wald Test, where the test statistic is a standardized estimator of the form

$$T=\left|\frac{\hat{\theta }-{\theta }_0}{\widehat{\mathrm{SE}}}\right|.$$

Warning

There is a tendency to use hypothesis testing methods even when they are not appropriate. Often, estimation and confidence intervals are better tools. Use hypothesis testing only when you want to test a well-defined hypothesis.

For example, suppose a company compares an old and new website design and wants to know whether the new design improves average time-on-site. A hypothesis test might show a statistically significant increase, but with a large enough sample even a tiny increase can be significant. In that setting, a confidence interval for the size of the increase is often more useful, because the real question is whether the improvement is large enough to matter in practice.

Definition

The power function of a test with rejection region $R$ is defined by

$$\beta (\theta )={\mathbb{P}}_{\theta }(X\in R)$$

The size of a test is defined to be

$$\alpha =\underset{\theta \in {\Theta }_0}{\sup }\beta (\theta ).$$

A test is said to have level $\alpha$ if its size is less than or equal to $\alpha$.

Intuition (Size)

The power function $\beta (\theta )$ is the probability that the test rejects when the true parameter is $\theta$. If $\theta \in {\Theta }_0$, then rejecting means we rejected the null hypothesis even though it was true.

So, within the null hypothesis, $\beta (\theta )$ is the false positive rate at that particular parameter value. The size takes the largest such value over all $\theta \in {\Theta }_0$. In other words, the size is the worst-case probability of incorrectly rejecting the null when the null is true.

This is why a level $\alpha$ test is valuable: it guarantees that no matter which parameter value in $H_0$ is the truth, the probability of a false rejection is at most $\alpha$.

Types of Hypothesis Tests

Definition (Simple Hypothesis)

A hypothesis of the form $\theta ={\theta }_0$ is called a simple hypothesis.

Definition (Composite Hypothesis)

A hypothesis of the form $\theta >{\theta }_0$ or $\theta <{\theta }_0$ is called a composite hypothesis.

Definition (Two-Sided Test)

A test of the form

$$H_0:\theta ={\theta }_0\mathrm{versus}{H}_1:\theta \ne {\theta }_0$$

is called a two-sided test.

Definition (One-Sided Test)

A test of the form

$$H_0:\theta \le {\theta }_0\mathrm{versus}{H}_1:\theta >{\theta }_0$$

or

$$H_0:\theta <{\theta }_0\mathrm{versus}{H}_1:\theta \ge {\theta }_0$$

is called a one-sided test.

The most common tests are two-sided tests.

Common Testing Methods

Different hypothesis tests fit different data-generating settings. The right test depends on the type of data, the null hypothesis being tested, and how trustworthy the underlying assumptions are.

Method	Good to use when	Not a good choice when	Main assumptions / limitations
The Wald Test	You have an estimator that is approximately Normal and a reliable estimated standard error, especially in large samples. Common for regression coefficients and scalar parameters.	Sample sizes are small, the estimator is skewed, or the parameter is near the boundary of its parameter space.	Relies on asymptotic Normality and a good standard error estimate. Can be unstable when the Normal approximation is poor.
Likelihood Ratio Test	You can write down a likelihood and want to compare a restricted model under $H_0$ against a less restricted model. Especially useful for multi-parameter and nested-model problems.	The model is hard to fit, the MLE does not behave regularly, or the null puts parameters on the boundary.	Uses likelihood-based modeling and usually large-sample ${\chi }^2$ approximations via Wilks’ theorem. Regularity conditions can fail in nonstandard problems.
Pearsons Chi-Squared Test	You have categorical count data, such as multinomial or contingency-table data, and want to compare observed counts to expected counts under $H_0$.	Expected counts are too small, categories are sparse, or observations are not independent.	Typically justified asymptotically. Works best when expected cell counts are not too small and the data are counts, not arbitrary continuous measurements.
The Permutation Test	You want a nonparametric test, especially for small samples, and the null hypothesis implies the labels are exchangeable. Useful when comparing two groups without trusting parametric assumptions.	The observations are dependent, the labels are not exchangeable under the null, or exhaustive resampling is computationally infeasible.	Exact in ideal settings, but depends on exchangeability under $H_0$. In practice it is often approximated by random resampling rather than all permutations.

These methods are all instances of the same general idea: build a test statistic that should look typical under $H_0$, then reject when the observed value looks too extreme.

Example

Let $X_1,\dots ,X_n\sim N(\mu ,{\sigma }^2)$ where $\sigma$ is known. We want to test $H_0:\mu \le 0$ versus $H_1:\mu >0$. Hence, ${\Theta }_0=(-\infty ,0]$ and ${\Theta }_1=(0,\infty )$. Consider the test

$$\mathrm{reject}{H}_0\mathrm{if}T>C$$

where $T=\bar{X}$. The rejection region is

$$R=\left\{(x_1,\dots ,x_n):T(x_1,\dots ,x_n)>c\right\}.$$

Let $Z$ denote a standard normal random variable. The power function is

$$\begin{array}{rl}\beta (\mu )& ={\mathbb{P}}_{\mu }(\bar{X}>c)\\ & ={\mathbb{P}}_{\mu }\left(\frac{\sqrt{n}(\bar{X}-\mu )}{\sigma }>\frac{\sqrt{n}(c-\mu )}{\sigma }\right)\\ & =\mathbb{P}\left(Z>\frac{\sqrt{n}(c-\mu )}{\sigma }\right)\\ & =1-\Phi \left(\frac{\sqrt{n}(c-\mu )}{\sigma }\right).\end{array}$$

This function is increasing in $\mu$. Hence,

$$\mathrm{size}=\underset{\mu \le 0}{\sup }\beta (\mu )=\beta (0)=1-\Phi \left(\frac{\sqrt{n}c}{\sigma }\right).$$

For a size $\alpha$ test, we set this equal to $\alpha$ and solve for $c$ to get

$$c=\frac{\sigma {\Phi }^{-1}(1-\alpha )}{\sqrt{n}}.$$

We reject when $\bar{X}>\sigma {\Phi }^{-1}(1-\alpha )/\sqrt{n}$. Equivalently, we reject when

$$\frac{\sqrt{n}(\bar{X}-0)}{\sigma }>z_{\alpha },$$

where $z_{\alpha }={\Phi }^{-1}(1-\alpha )$.