Types of Convergence

Convergence in Probability and Convergence in Distribution

There are two main types of convergence.

Definition

Let $X_1,X_2,\dots$ be a sequence of random variables, and let $X$ be another random variable. Let $F_n$ denote the CDF of $X_n$ and let $F$ denote the CDF of $X$.

1. $X_n$ converges to $X$ in probability, written $X_n\stackrel{\mathrm{P}}{\to }X$, if, for every $ϵ>0$,

$$\mathbb{P}(|X_n-X|>ϵ)\to 0$$

as $n\to \infty$. 2. $X_n$ converges to $X$ in distribution, written $X_n⇝X$, if

$$\underset{n\to \infty }{\lim }{F}_n(t)=F(t)$$

for all $t$ for which $F$ is continuous.

In words:

• $X_n\stackrel{\mathrm{P}}{\to }X$ means the random variables themselves get close with high probability.
• $X_n⇝X$ means only the distributions get close; the random variables do not have to be close on the same outcome.
• In probability, we compare the values of $X_n(\omega )$ and $X(\omega )$ at the same outcome $\omega$.

When the limiting factor is a point mass, we change the notation slightly. If $\mathbb{P}(X=c)=1$ and $X_n\stackrel{\mathrm{P}}{\to }X$, then we write $X_n\stackrel{\mathrm{P}}{\to }c$. Similarly, if $X_n⇝X$ we write $X_n⇝c$.

Same distribution, different convergence

Let $Z\sim N(0,1)$ and define $X_n=(-1{)}^{n}Z$.

Since the standard normal distribution is symmetric about $0$, we have $Z\stackrel{d}{=}-Z$. So each $X_n$ has the same distribution as $Z$, and therefore $X_n⇝Z$.

But $X_n$ does not converge to $Z$ in probability, because for odd $n$ we have $X_n=-Z$ on the same outcome $\omega$. If $Z(\omega )=3$, then $X_n(\omega )=-3$, not $3$. So

$$|X_n-Z|=|-Z-Z|=2|Z|.$$

This does not get small with probability going to $1$, so $X_n/\stackrel{\mathrm{P}}{\to }Z$.

Convergence in both probability and distribution

Let $Z\sim N(0,1)$ and write $X_n=\frac{Z}{\sqrt{n}}.$ Then $X_n\sim N(0,1/n)$.

To show convergence in probability to $0$, fix $ϵ>0$. Then

$$\mathbb{P}(|X_n-0|>ϵ)=\mathbb{P}\left(\left|\frac{Z}{\sqrt{n}}\right|>ϵ\right)=\mathbb{P}(|Z|>ϵ\sqrt{n}).$$

Since $ϵ\sqrt{n}\to \infty$ and $Z$ has a fixed distribution, this probability goes to $0$. So $X_n\stackrel{\mathrm{P}}{\to }0$. Interestingly, this can also be shown using Markov’s Inequality. For any $ϵ>0$,

$$\begin{array}{rl}\mathbb{P}(|X_n|>ϵ)& =\mathbb{P}(|X_n{|}^2>{ϵ}^2)\\ & \le \frac{\mathbb{E}(X_n^2)}{{ϵ}^2}=\frac{\frac{1}{n}}{{ϵ}^2}\to 0\end{array}$$

as $n\to \infty$. Hence, $X_n\stackrel{\mathrm{P}}{\to }0$.

To show convergence in distribution, let $F_n$ be the CDF of $X_n$. For any $t$,

$$F_n(t)=\mathbb{P}\left(\frac{Z}{\sqrt{n}}\le t\right)=\mathbb{P}(Z\le t\sqrt{n})=\Phi (t\sqrt{n}),$$

where $\Phi$ is the standard normal CDF.

If $t<0$, then $t\sqrt{n}\to -\infty$, so $F_n(t)\to 0$. If $t>0$, then $t\sqrt{n}\to \infty$, so $F_n(t)\to 1$. This is exactly the CDF of a point mass at $0$, so $X_n⇝0$.

Convergence in Quadratic Mean

There is another type of convergence which is useful for proving convergence in probability:

Definition

$X_n$ converges to $X$ in quadratic mean (also called convergence in $L_2$), written $X_n\stackrel{\mathrm{qm}}{\to }X$, if

$$\mathbb{E}(X_n-X{)}^2\to 0$$

As $n\to \infty$.

Again, if $X$ is a point mass at $c$, we write $X_n\stackrel{\mathrm{qm}}{\to }c$ instead of $X_n\stackrel{\mathrm{qm}}{\to }X$.

Convergence Properties

The following theorem gives the relationship between the types of convergence.

Theorem

The following relationships hold:

• a) $X_n\stackrel{\mathrm{qm}}{\to }X$ implies that $X_n\stackrel{\mathrm{P}}{\to }X$.
• b) $X_n\stackrel{\mathrm{P}}{\to }X$ implies that $X_n⇝X$.
• c) If $X_n⇝X$ and if $\mathbb{P}(X=c)=1$ for some real number $c$, then $X_n\stackrel{\mathrm{P}}{\to }X$.

There are also some additional useful properties.

Theorem

Let $X_n,X,Y_n,Y$ be random variables. Let $g$ be a continuous function.

• (a) If $X_n\stackrel{\mathrm{P}}{\to }X$ and $Y_n\stackrel{\mathrm{P}}{\to }Y$, then $X_n+Y_n\stackrel{\mathrm{P}}{\to }X+Y$.
• (b) If $X_n\stackrel{\mathrm{qm}}{\to }X$ and $Y_n\stackrel{\mathrm{qm}}{\to }Y$, then $X_n+Y_n\stackrel{\mathrm{qm}}{\to }X+Y$.
• (c) If $X_n⇝X$ and $Y_n⇝c$, then $X_n+Y_n⇝X+c$.
• (d) If $X_n\stackrel{\mathrm{P}}{\to }X$ and $Y_n\stackrel{\mathrm{P}}{\to }Y$, then $X_n{Y}_n\stackrel{\mathrm{P}}{\to }XY$.
• (e) If $X_n⇝X$ and $Y_n⇝c$, then $X_n{Y}_n⇝cX$.
• (f) If $X_n\stackrel{\mathrm{P}}{\to }X$, then $g(X_n)\stackrel{\mathrm{P}}{\to }g(X)$.
• (g) If $X_n⇝X$, then $g(X_n)⇝g(X)$.