Mean Squared Error

Estimator

Definition

The quality of a point estimate is sometimes assessed by the mean squared error, or MSE, defined by

$$\mathrm{MSE}={\mathbb{E}}_{\theta }({\hat{\theta }}_n-\theta {)}^2.$$

Keep in mind that ${\mathbb{E}}_{\theta }(\cdot )$ refers to the expectation with respect to the distribution

$$f(x_1,\dots ,x_n;\theta )=\prod _{i=1}^{n}f(x_i;\theta )$$

that generated the data. It does not mean we are averaging over a distribution for $\theta$. In other words, $\theta$ is treated as fixed but unknown, while ${\hat{\theta }}_n$ is random because it depends on the random sample $(X_1,\dots ,X_n)$. Thus, the MSE is the average squared error we would obtain over repeated samples drawn from the population when the true parameter value is $\theta$.

More generally, even when the true parameter value is unknown, we can still analyze an estimator theoretically by deriving its bias, variance, or MSE as functions of the unknown parameter and the sample size. Then, once we observe a sample and compute the estimator, those theoretical properties help us interpret how reliable the estimate is under repeated sampling.

Theorem

The MSE can be written as

$$\mathrm{MSE}=\mathbb{V}({\hat{\theta }}_n)+{\mathrm{bias}}^2({\hat{\theta }}_n).$$

Proof

Let ${\bar{\theta }}_n=\mathbb{E}({\hat{\theta }}_n)$. Then,

$$\begin{array}{rl}{\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-\theta {)}^2\right]& ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n-{\bar{\theta }}_n-\theta {)}^2\right]\\ & ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n{)}^2+2({\hat{\theta }}_n-{\bar{\theta }}_n)({\bar{\theta }}_n-\theta )+({\bar{\theta }}_n-\theta {)}^2\right]\\ & ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n{)}^2\right]+{\mathbb{E}}_{\theta }\left[2({\hat{\theta }}_n-{\bar{\theta }}_n)({\bar{\theta }}_n-\theta )\right]+{\mathbb{E}}_{\theta }\left[({\bar{\theta }}_n-\theta {)}^2\right]\\ & ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n{)}^2\right]+2({\bar{\theta }}_n-\theta ){\mathbb{E}}_{\theta }\left[{\hat{\theta }}_n-{\bar{\theta }}_n\right]+({\bar{\theta }}_n-\theta {)}^2\\ & ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n{)}^2\right]+2({\bar{\theta }}_n-\theta )({\bar{\theta }}_n-{\bar{\theta }}_n)+({\bar{\theta }}_n-\theta {)}^2\\ & ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-{\bar{\theta }}_n{)}^2\right]+({\bar{\theta }}_n-\theta {)}^2\\ & =\mathbb{V}({\hat{\theta }}_n)+{\mathrm{bias}}^2({\hat{\theta }}_n)\end{array}\text{\hspace{1em}(1)(2)}$$

Where (1) is due to ${\bar{\theta }}_n-\theta$ being a constant, and (2) ${\bar{\theta }}_n$ being a constant.

A shorter proof can be achieved using that for a random variable $X$, $\mathbb{E}(X^2)=\mathbb{V}(X)+(\mathbb{E}[X]{)}^2$:

Proof

$$\begin{array}{rl}\mathrm{MSE}({\hat{\theta }}_n)& ={\mathbb{E}}_{\theta }\left[({\hat{\theta }}_n-\theta {)}^2\right]\\ & =\mathbb{V}({\hat{\theta }}_n-\theta )+(\mathbb{E}[{\hat{\theta }}_n-\theta ]{)}^2\\ & =\mathbb{V}({\hat{\theta }}_n)+{\mathrm{bias}}^2({\hat{\theta }}_n)\end{array}$$

Theorem

If $\mathrm{bias}\to 0$ and the standard error $\mathrm{SE}\to 0$ as $n\to \infty$, then ${\hat{\theta }}_n$ is consistent, that is, ${\hat{\theta }}_n\stackrel{\mathrm{p}}{\to }\theta$.

Proof

If $\mathrm{bias}\to 0$ and $\mathrm{SE}\to 0$, then by the above theorem, $\mathrm{MSE}\to 0$. It follows from convergence in quadratic mean that ${\hat{\theta }}_n\stackrel{\mathrm{qm}}{\to }\theta$. The result then follows from the convergence properties that ${\hat{\theta }}_n\stackrel{\mathrm{qm}}{\to }\theta$ implies that ${\hat{\theta }}_n\stackrel{\mathrm{P}}{\to }\theta$.

Predictor

If a vector of $n$ predictions is generated from a sample of $n$ data points on all random variables, and $Y$ is the vector of observed values of the variable being predicted, with $\hat{Y}$ being the predicted values, then the within-sample MSE of the predictor is computed as

$$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n{\left(Y_i-{\hat{Y}}_i\right)}^2.$$

In matrix notation,

$$\mathrm{MSE}=\frac{1}{n}\sum _{i=1}^n(e_i{)}^2=\frac{1}{n}{\mathbf{e}}^{\mathrm{T}}\mathbf{e},$$

where $e_i$ is $Y_i-{\hat{Y}}_i$, and $\mathbf{e}$ is a $n\times 1$ column vector.

This version of MSE is used when the goal is prediction rather than estimation of a population parameter. In that setting, we evaluate how close the predicted responses are to the observed outcomes, whereas the estimator version of MSE evaluates how close an estimator tends to be to a fixed but unknown parameter value.

Examples

Mean

Example

Suppose we have a random sample of size $n$ from a population, $X_1,\dots X_n$. The usual estimator for the population mean $\mu$ is the sample mean

$$\bar{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$$

which has an expected value equal to the true mean $\mu$ (so it is unbiased), and a mean squared error of

$$\mathrm{MSE}(\bar{X})=\mathbb{E}\left[(\bar{X}-\mu {)}^2\right]={\left(\frac{\sigma }{\sqrt{n}}\right)}^2=\frac{{\sigma }^2}{n}$$

where $\sigma$ is the population variance.

Variance

Example

The usual estimator for the variance is the corrected sample variance

$$S_{n-1}^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-{\bar{X}}_n{)}^2.$$

This is unbiased (its expected value is ${\sigma }^2$), and its MSE is

$$\mathrm{MSE}(S_{n-1}^2)=\frac{1}{n}\left({\mu }_4-\frac{n-3}{n-1}{\sigma }^4\right),$$

where ${\mu }_4$ is the fourth central moment of the distribution or population.