Point Estimation

Point estimation refers to providing a single best guess of some quantity of interest. This could be a parameter in a parametric model, a CDF $F$, a probability density function $f$, a regression function $r$, or a prediction for a future value $Y$ of some random variable.

By convention, we denote a point estimate of $\theta$ by $\hat{\theta }$ or ${\hat{\theta }}_n$. Remember that $\theta$ is a fixed unknown quantity. The estimate $\hat{\theta }$ depends on the data, and so $\hat{\theta }$ is a random variable.

Definition

Let $X_1,\dots ,X_n$ be $n$ IID data points from some distribution $F$. A point estimator ${\hat{\theta }}_n$ of a parameter $\theta$ is some function of $X_1,\dots ,X_n$:

$${\hat{\theta }}_n=g(X_1,\dots ,X_n).$$

The bias of the estimator is defined by

We say that ${\hat{\theta }}_n$ is unbiased if $\mathbb{E}({\hat{\theta }}_n)=\theta$. Many of the estimators studied are biased. A reasonable requirement for an estimator is that it should converge to the true parameter value as we collect more and more data.

Definition (Consistent)

A point estimator ${\hat{\theta }}_n$ of a parameter $\theta$ is consistent if ${\hat{\theta }}_n\stackrel{\mathrm{P}}{\to }\theta$.

Definition (Sampling Distribution)

The distribution of ${\hat{\theta }}_n$ is called the sampling distribution.

Limiting Normal Distribution

Definition

An estimator is asymptotically Normal if

$$\frac{{\hat{\theta }}_n-\theta }{\mathrm{SE}}⇝N(0,1).$$