The standard deviation of a statistic (an estimator of a parameter) is called the standard error, denoted by :
Often, the standard error depends on the unknown . In those cases, is an unknown quantity, but we usually can estimate it. The estimated standard error is denoted by .
Let and let . Then , so is unbiased. The standard error is . The estimated standard error is .
Standard Error of the Sample Mean
Suppose we have a random sample of size from a population, . The usual estimator for the population mean is the sample mean
which has an expected value equal to the true mean (so it is unbiased), and a mean squared error of
where is the population variance. Therefore, the standard error of the sample mean is
In practice, however, the population variance is usually unknown, so the standard error is also unknown. In that case, we replace by the sample standard deviation. The usual choice is to use the sample variance
so that
Using this, the estimated standard error of the sample mean is
The version with in the denominator is the one usually used in statistics, because is an unbiased estimator of . If one instead divides by , then the resulting sample variance tends to underestimate the population variance slightly. For large , the difference between dividing by and by is usually small, but by convention and theory the corrected version is standard.
Sources
Wasserman, L. (2010). All of Statistics: A concise Course in Statistical Inference. Chapter 6.