The Wald Test

Let $\theta$ be a scalar parameter, let $\hat{\theta }$ be an estimate of $\theta$, and let $\widehat{\mathrm{SE}}$ be the estimated standard error of $\hat{\theta }$.

Hypothesis testing says that a test is built by choosing a test statistic $T$ and a critical value $c$, then rejecting when $T$ looks too extreme under $H_0$. The Wald test is one of the most common ways to do this when an estimator is approximately Normal.

Definition (The Wald Test)

Consider testing

$$H_0:\theta ={\theta }_0\mathrm{versus}{H}_1:\theta \ne {\theta }_0.$$

Assume that $\hat{\theta }$ is asymptotically Normal:

$$\frac{\hat{\theta }-\theta }{\widehat{\mathrm{SE}}}⇝N(0,1).$$

The size $\alpha$ Wald Test is: reject $H_0$ when $|W|>z_{\alpha /2}$ where

$$W=\frac{\hat{\theta }-{\theta }_0}{\widehat{\mathrm{SE}}}.$$

Theorem

Asymptotically, the Wald test has a size $\alpha$, that is,

$${\mathbb{P}}_{{\theta }_0}\left(|W|>z_{\alpha /2}\right)\to \alpha$$

as $n\to \infty$.

Intuition

The Wald statistic

$$W=\frac{\hat{\theta }-{\theta }_0}{\widehat{\mathrm{SE}}}$$

measures how many estimated standard errors the observed estimate $\hat{\theta }$ is away from the null value ${\theta }_0$.

If $H_0$ is true, then $\theta ={\theta }_0$, so we expect

$$W\approx N(0,1).$$

That means values of $W$ near $0$ are typical under the null, while very large positive or negative values are unlikely under the null. So the Wald test says:

Intuition

Pretend the null value ${\theta }_0$ is the truth. Then ask: would an estimate this far from ${\theta }_0$, relative to its usual noise level, be surprising?

If yes, reject $H_0$. If no, keep $H_0$.

This is exactly the general hypothesis testing template with

$$T(x)=\left|\frac{\hat{\theta }(x)-{\theta }_0}{\widehat{\mathrm{SE}}(x)}\right|\text{and}c=z_{\alpha /2}.$$

So the rejection region is

$$R=\left\{x:\left|\frac{\hat{\theta }(x)-{\theta }_0}{\widehat{\mathrm{SE}}(x)}\right|>z_{\alpha /2}\right\}.$$

Why Standardize?

The raw difference $\hat{\theta }-{\theta }_0$ by itself is not enough, because the same absolute difference can be very meaningful or not meaningful depending on the estimator’s variability.

• If $\widehat{\mathrm{SE}}$ is small, even a modest difference from ${\theta }_0$ is strong evidence against $H_0$.
• If $\widehat{\mathrm{SE}}$ is large, the same difference may just be sampling noise.

Dividing by $\widehat{\mathrm{SE}}$ puts the discrepancy on a common scale: “how many noise levels away from the null are we?”

Connection to Confidence Intervals

The Wald test is the testing version of a normal-based confidence interval. The usual approximate $1-\alpha$ interval is

$$\hat{\theta }\pm z_{\alpha /2}\widehat{\mathrm{SE}}.$$

The null hypothesis $H_0:\theta ={\theta }_0$ is rejected exactly when ${\theta }_0$ is outside this interval, since

$$\left|\frac{\hat{\theta }-{\theta }_0}{\widehat{\mathrm{SE}}}\right|>z_{\alpha /2}$$

is equivalent to saying that ${\theta }_0$ lies more than $z_{\alpha /2}$ standard errors away from $\hat{\theta }$.

This is often the cleanest way to remember the Wald test:

Note

A Wald test rejects a null value precisely when that null value is not plausible according to the corresponding normal-based confidence interval.

Example

Suppose $X_1,\dots ,X_n\sim \mathrm{Bernoulli}(p)$ and we want to test

$$H_0:p=p_0\mathrm{versus}{H}_1:p\ne p_0.$$

Let $\hat{p}=n^{-1}{\sum }_{i=1}^n{X}_i$. Since

$$\hat{p}\approx N\left(p,\frac{p(1-p)}{n}\right),$$

we estimate the standard error by

$$\widehat{\mathrm{SE}}=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}.$$

The Wald statistic is then

$$W=\frac{\hat{p}-p_0}{\sqrt{\hat{p}(1-\hat{p})/n}}.$$

If $|W|>z_{\alpha /2}$, we reject $H_0$.

Informally: if the observed sample proportion is several estimated standard errors away from the hypothesized proportion $p_0$, then $p_0$ is not a convincing explanation for the data.

Caveat

The Wald test is simple and widely used, but it can be inaccurate in small samples, for skewed estimators, or when the parameter is near the boundary of its parameter space. In those settings, score tests, likelihood-ratio tests, or bootstrap-based methods can behave better.