Maximum Likelihood Estimation

Maximum Likelihood

The most common method for estimating parameters in a parametric model is the maximum likelihood method.

Definition (Likelihood function)

Let $X_1,\dots ,X_n$ be IID with PDF $f(x;\theta )$. The likelihood function is defined by

$${\mathcal{L}}_n(\theta )=\prod _{i=1}^{n}f(X_i;\theta ).$$

The log-likelihood function is defined by ${\ell }_n(\theta )=\log {\mathcal{L}}_n(\theta ).$

Intuition

The likelihood function is just the join density of the data, except that we treat it as a function of the parameter $\theta$. Thus, ${\mathcal{L}}_n:\Theta \to [0,\infty )$. The likelihood function is not a density function in general. It is not true that ${\mathcal{L}}_n(\theta )$ integrates to 1 (with respect to $\theta$).

Definition (MLE)

The maximum likelihood estimator (MLE), denoted by ${\hat{\theta }}_n$, is the value of $\theta$ that maximizes ${\mathcal{L}}_n(\theta )$.

Since $\log$ is a monotonic function, the maximum of ${\ell }_n(\theta )$ occurs at the same place as the maximum of ${\mathcal{L}}_n(\theta )$, so maximizing the log-likelihood leads to the same answer as maximizing he likelihood. Often, it is easier to work with the log-likelihood.

Properties of Maximum Likelihood Estimators

Under certain conditions of the model, the maximum likelihood estimator ${\hat{\theta }}_n$ possesses many properties that make it an appealing choice of estimator. The main properties of MLE are:

1. The MLE is consistent: ${\hat{\theta }}_n\to {\theta }_{\star }$ where ${\theta }_{\star }$ denotes the true value of the parameter $\theta$.
2. The MLE is equivariant: if ${\hat{\theta }}_n$ is the MLE of $\theta$, then $g({\hat{\theta }}_n)$ is the MLE of $g(\theta )$.
3. The MLE is asymptotically Normal: $(\hat{\theta }-{\theta }_{\star })/\widehat{\mathrm{SE}}⇝N(0,1)$; also, the estimated standard error $\widehat{\mathrm{SE}}$ can often be computed analytically.
4. The MLE is asymptotically optimal: roughly, this means that among all well-behaved estimators, the MLE ahs the smallest variance , at least for large samples.
5. The MLE is approximately the Bayes estimator.

Consistency of Maximum Likelihood Estimators

Consistency means that the MLE converges in probability to the true value. If $f$ and $g$ are PDFs, define the Kullback-Leibler distance between $f$ and $g$ to be

$$D(f,g)=\int f(x)\log \frac{f(x)}{g(x)}dx.$$

For any $\theta ,\psi \in \Theta$ write $D(\theta ,\psi )$ to mean $D(f(x;\theta ),f(x;\psi ))$.

Definition

A model $\mathfrak{F}$ is identifiable if $\theta \ne \psi$ implies that $D(\theta ,\psi )>0$. This means that different values of the parameter correspond to different distributions.

Assume that the model is identifiable. Let ${\theta }_{\star }$ denote the true value of $\theta$. Maximizing ${\ell }_n(\theta )$ is equivalent to maximizing

$$M_n(\theta )=\frac{1}{n}\sum _i\log \frac{f(X_i;\theta )}{f(X_i;{\theta }_{\star })}.$$

This follows since $M_n(\theta )=n^{-1}({\ell }_n(\theta )-{\ell }_n({\theta }_{\star }))$ and ${\ell }_n({\theta }_{\star })$ is a constant (with respect to $\theta$). For each fixed $\theta$, the randomness is in the sample $X_1,\dots ,X_n$, and since the data are assumed to be drawn from the true model $f(\cdot ;{\theta }_{\star })$, the expectation below is taken with respect to that distribution. By the Law of Large Numbers, $M_n(\theta )$ converges to

$$\begin{array}{rl}{\mathbb{E}}_{{\theta }_{\star }}\left(\log \frac{f(X_i;\theta )}{f(X_i;{\theta }_{\star })}\right)& =\int \log \left(\frac{f(x;\theta )}{f(x;{\theta }_{\star })}\right)f(x;{\theta }_{\star })dx\\ & =-\int \log \left(\frac{f(x;{\theta }_{\star })}{f(x;\theta )}\right)f(x;{\theta }_{\star })dx\\ & =-D({\theta }_{\star },\theta ).\end{array}$$

Hence, $M_n(\theta )\approx -D({\theta }_{\star },\theta )$ which is maximized at ${\theta }_{\star }$ since $-D({\theta }_{\star },{\theta }_{\star })=0$ and $-D({\theta }_{\star },\theta )<0$ for $\theta \ne {\theta }_{\star }$. This suggests that the maximizer of $M_n(\theta )$, and hence the MLE, should converge to ${\theta }_{\star }$, with a fully rigorous proof requiring additional regularity conditions.

Asymptotic Normality

In turns out that the distribution of ${\hat{\theta }}_n$ is approximately Normal and we can compute its approximate variance analytically.

Definition

The score function is defined to be

$$s(X;\theta )=\frac{\partial \log f(X;\theta )}{\partial \theta }.$$

The Fisher Information is defined to be

$$\begin{array}{rl}{I}_n(\theta )& ={\mathbb{V}}_{\theta }\left(\sum _{i=1}^{n}s(X_i;\theta )\right)\\ & =\sum _{i=1}^n{\mathbb{V}}_{\theta }\left(s(X_i;\theta )\right)\end{array}$$

where the second equation is due to the variance of the sum of independent random variables being the sum of their individual variances.

Intuition

The Fisher information measures how much information the data carry about the parameter $\theta$. The score $s(X;\theta )$ is the derivative of the log-density, so it measures how sensitive the model is to changes in $\theta$ for a single observation. If changing $\theta$ tends to change the log-density a lot, then the score will typically have larger magnitude, so its variance will be larger. Thus, the definition $I(\theta )={\mathbb{V}}_{\theta }(s(X;\theta ))$ says that Fisher information is large when the model responds strongly to changes in $\theta$, and small when nearby values of $\theta$ are hard to distinguish. In this sense, larger Fisher information means $\theta$ can be estimated more precisely.

It can be shown that ${\mathbb{E}}_{\theta }(s(X;\theta ))=0$. It then follows that ${\mathbb{V}}_{\theta }(s(X;\theta ))={\mathbb{E}}_{\theta }(s^2(X;\theta ))$. A further simplification of $I_n(\theta )$ is given in the next result.

Theorem

$I_n(\theta )=nI(\theta )$, where $I(\theta ):=I_1(\theta )$. Also,

$$\begin{array}{rl}I(\theta )& =-{\mathbb{E}}_{\theta }\left(\frac{{\partial }^2\log f(X;\theta )}{\partial {\theta }^2}\right)\\ & =-\int \left(\frac{{\partial }^2\log f(x;\theta )}{\partial {\theta }^2}\right)f(x;\theta )dx.\end{array}$$

Intuition

This formula gives the same quantity a second interpretation in terms of curvature. The second derivative measures how curved the log-likelihood is as a function of $\theta$. Near a maximum this quantity is usually negative, so the minus sign makes the information positive. If the log-likelihood is sharply curved near the true value, then moving $\theta$ slightly makes the fit much worse, so the data determine $\theta$ more precisely and the Fisher information is large. If it is relatively flat, then nearby values of $\theta$ fit almost equally well, so the Fisher information is small. Therefore, the variance-of-score definition and the negative-expected-curvature definition are two equivalent ways of measuring the same idea: how sensitive the model is to changes in $\theta$.

Theorem (Asymptotic Normality of the MLE)

Let $\mathrm{SE}=\sqrt{\mathbb{V}({\hat{\theta }}_n)}$. Under appropriate regularity conditions, the following hold:

1. $\mathrm{SE}\approx \sqrt{1/I_n(\theta )}$ and

$$\frac{{\hat{\theta }}_n-\theta }{\mathrm{SE}}⇝N(0,1).$$

2. Let $\widehat{\mathrm{SE}}=\sqrt{1/I_n({\hat{\theta }}_n)}$. Then,

$$\frac{{\hat{\theta }}_n-\theta }{\widehat{\mathrm{SE}}}⇝N(0,1).$$

• The first statement says that ${\hat{\theta }}_n\approx N(\theta ,\mathrm{SE})$ where the approximate standard error of ${\hat{\theta }}_n$ is $\mathrm{SE}=\sqrt{1/I_n(\theta )}$.
• The second statement says that this is still true even if we replace the standard error by its estimated standard error $\widehat{\mathrm{SE}}=\sqrt{1/I_n({\hat{\theta }}_n)}$.

Informally, the theorem says that the distribution of the MLE can be approximated with $N(\theta ,\widehat{{\mathrm{SE}}^2})$.

Theorem

Let

$$C_n=\left({\hat{\theta }}_n-z_{\alpha /2}\widehat{\mathrm{SE}},{\hat{\theta }}_n+z_{\alpha /2}\widehat{\mathrm{SE}}\right).$$

Then ${\mathbb{P}}_{\theta }(\theta \in C_n)\to 1-\alpha$ as $n\to \infty$.

Optimality

Suppose that $X_1,\dots ,X_n\sim N(\theta ,{\sigma }^2)$. The MLE is ${\hat{\theta }}_n={\bar{X}}_n$. Another reasonable estimator of $\theta$ is the sample median ${\tilde{\theta }}_n$. The MLE satisfies

$$\sqrt{n}({\hat{\theta }}_n-\theta )⇝N(0,{\sigma }^2).$$

It can be proved that the median satisfies

$$\sqrt{n}({\tilde{\theta }}_n-\theta )⇝N\left(0,{\sigma }^2\frac{\pi }{2}\right).$$

This means that the median converges to the right value but has a larger variance than the MLE.

More generally, consider two estimators $T_n$ and $U_n$ and suppose that

$$\sqrt{n}(T_n-\theta )⇝N(0,t^2),$$

and that

$$\sqrt{n}(U_n-\theta )⇝N(0,u^2).$$

We define the asymptotic relative efficiency of $U$ to $T$ by $\mathrm{ARE}(U,T)=t^2/u^2$. In the Normal example, $\mathrm{ARE}({\tilde{\theta }}_n,{\hat{\theta }}_n)=2/\pi =0.63$. The interpretation is that if you use the median, you are effectively using only a fraction of the data.

Theorem

If ${\hat{\theta }}_n$ is the MLE and ${\tilde{\theta }}_n$ is any other estimator, then

$$\mathrm{ARE}({\tilde{\theta }}_n,{\hat{\theta }}_n)\le 1.$$

Interpretation

Thus, the MLE has the smallest (asymptotic) variance and we say that the MLE is efficient or asymptotically optimal.