Standard Error

Definition (Standard Error)

The standard deviation of a statistic (an estimator of a parameter) ${\hat{\theta }}_n$ is called the standard error, denoted by $\mathrm{SE}$:

$$\mathrm{SE}=\mathrm{SE}({\hat{\theta }}_n)=\sqrt{\mathbb{V}({\hat{\theta }}_n)}.$$

Often, the standard error depends on the unknown $F$. In those cases, $\mathrm{se}$ is an unknown quantity, but we usually can estimate it. The estimated standard error is denoted by $\widehat{\mathrm{SE}}$.

Note

If ${\hat{\theta }}_n$ is an unbiased estimator of $\theta$, then $\mathbb{E}({\hat{\theta }}_n)=\theta$, and so

$$\mathrm{MSE}({\hat{\theta }}_n)=\mathbb{E}\left[({\hat{\theta }}_n-\theta {)}^2\right]=\mathbb{V}({\hat{\theta }}_n),$$

where $\mathrm{MSE}$ is the mean squared error. Therefore, in the unbiased case,

$$\sqrt{\mathrm{MSE}({\hat{\theta }}_n)}=\mathrm{SE}({\hat{\theta }}_n).$$

Example

Let $X_1,\dots ,X_n\sim \mathrm{Bernoulli}(p)$ and let ${\hat{p}}_n=n^{-1}{\sum }_i{X}_i$. Then $\mathbb{E}({\hat{p}}_n)=n^{-1}{\sum }_i\mathbb{E}(X_i)=p$, so ${\hat{p}}_n$ is unbiased. The standard error is $\mathrm{SE}=\sqrt{\mathbb{V}({\hat{p}}_n)}=\sqrt{p(1-p)/n}$. The estimated standard error is $\widehat{\mathrm{SE}}=\sqrt{\hat{p}(1-\hat{p})/n}$.

Standard Error of the Sample Mean

Suppose we have a random sample of size $n$ from a population, $X_1,\dots X_n$. The usual estimator for the population mean $\mu$ is the sample mean

$$\bar{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$$

which has an expected value equal to the true mean $\mu$ (so it is unbiased), and a mean squared error of

$$\mathrm{MSE}(\bar{X})=\mathbb{E}\left[(\bar{X}-\mu {)}^2\right]={\left(\frac{\sigma }{\sqrt{n}}\right)}^2=\frac{{\sigma }^2}{n}$$

where $\sigma$ is the population variance. Therefore, the standard error of the sample mean is

$$\mathrm{SE}(\bar{X})=\sqrt{\mathrm{MSE}(\bar{X})}=\frac{\sigma }{\sqrt{n}}.$$

In practice, however, the population variance ${\sigma }^2$ is usually unknown, so the standard error is also unknown. In that case, we replace $\sigma$ by the sample standard deviation. The usual choice is to use the sample variance

$$S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2,$$

so that

$$S=\sqrt{\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2}.$$

Using this, the estimated standard error of the sample mean is

$$\widehat{\mathrm{SE}}(\bar{X})=\frac{S}{\sqrt{n}}.$$

The version with $n-1$ in the denominator is the one usually used in statistics, because $S^2$ is an unbiased estimator of ${\sigma }^2$. If one instead divides by $n$, then the resulting sample variance tends to underestimate the population variance slightly. For large $n$, the difference between dividing by $n$ and by $n-1$ is usually small, but by convention and theory the corrected version is standard.