Variance and Covariance

Variance

The variance measures the spread of a distribution.

Definition

Let $X$ be a random variable with mean $\mu$. The variance of $X$, denoted by ${\sigma }^2$ or ${\sigma }_X^2$ or $\mathbb{V}(X)$, is defined by

$${\sigma }^2=\mathbb{E}(X-\mu {)}^2=\int (x-\mu {)}^{2}dF(x),$$

assuming this expectation exists. The standard deviation is $\text{sd}(X)=\sqrt{\mathbb{V}(x)}$, and is denoted by $\sigma$ and ${\sigma }_X$.

We can’t use $\mathbb{E}(X-\mu )$ as a measure of spread, since $\mathbb{E}(X-\mu )=\mathbb{E}(X)-\mu =\mu -\mu =0$. We can sometimes use $\mathbb{E}|X-\mu |$ as a measure of spread, but often we use the variance.

Theorem

Assuming the variance is well defined, it has the following properties:

1. $\mathbb{V}(X)=\mathbb{E}(X^2)-{\mu }^2$.
2. If $a$ and $b$ are constants, then $\mathbb{V}(aX+b)=a^2\mathbb{V}(X)$.
3. If $X_1,\dots ,X_n$ are independent and $a_1,\dots ,a_n$ are constants, then

$$\mathbb{V}\left(\sum _{i=1}^n{a}_i{X}_i\right)=\sum _{i=1}^n{a}_i^2\mathbb{V}(X_i).$$

Sample Mean and Sample Variance

Definition

If $X_1,\dots ,X_n$ are random variables, then we define the sample mean to be

$${\bar{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i,$$

and the sample variance to be

$$S_{n-1}^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-{\bar{X}}_n{)}^2.$$

This is the unbiased (or corrected) sample variance. If we divided by $n$ instead, then on average we would underestimate the true variance. A shorter way to see this is to use the identity

$$\sum _{i=1}^n(X_i-\bar{X}{)}^2=\sum _{i=1}^n(X_i-\mu {)}^2-n(\bar{X}-\mu {)}^2.$$

To derive it, write

$$\begin{array}{rl}{X}_i-\bar{X}& =(X_i-\mu )-(\bar{X}-\mu ),\\ \sum _{i=1}^n(X_i-\bar{X}{)}^2& =\sum _{i=1}^n{\left((X_i-\mu )-(\bar{X}-\mu )\right)}^2\\ & =\sum _{i=1}^n(X_i-\mu {)}^2-2(\bar{X}-\mu )\sum _{i=1}^n(X_i-\mu )+\sum _{i=1}^n(\bar{X}-\mu {)}^2\\ & =\sum _{i=1}^n(X_i-\mu {)}^2-2(\bar{X}-\mu )n(\bar{X}-\mu )+n(\bar{X}-\mu {)}^2\\ & =\sum _{i=1}^n(X_i-\mu {)}^2-n(\bar{X}-\mu {)}^2.\end{array}$$

Then

$$\begin{array}{rl}\mathbb{E}\left[\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2\right]& =\mathbb{E}\left[\frac{1}{n}\sum _{i=1}^n(X_i-\mu {)}^2-(\bar{X}-\mu {)}^2\right]\\ & =\frac{1}{n}\sum _{i=1}^n\mathbb{E}[(X_i-\mu {)}^2]-\mathbb{E}[(\bar{X}-\mu {)}^2]\\ & =\frac{1}{n}\sum _{i=1}^n\mathbb{V}(X_i)-\mathbb{V}(\bar{X})\\ & =\frac{1}{n}\cdot n\mathbb{V}(X)-\frac{1}{n}\mathbb{V}(X)\\ & =\frac{n-1}{n}\mathbb{V}(X)\end{array}$$

The reason is that the data are being measured relative to the sample mean ${\bar{X}}_n$, not the true mean $\mu$. Since ${\bar{X}}_n$ is computed from the same sample, it is the value that makes the sample look as centered as possible, so the deviations $(X_i-{\bar{X}}_n)$ are typically a bit smaller than the true deviations $(X_i-\mu )$.

A concrete way to see the loss of one degree of freedom is that the centered values must satisfy

$$\sum _{i=1}^n(X_i-{\bar{X}}_n)=0.$$

Once you know any $n-1$ of these deviations, the last one is forced to be whatever makes the total equal to $0$. So there are really only $n-1$ independent pieces of variation left after estimating the mean. Dividing by $n-1$ compensates for this built-in shrinkage, which is why $\mathbb{E}(S_{n-1}^2)={\sigma }^2$.

Theorem

Let $X_1,\dots ,X_n$ be IID and let $\mu =\mathbb{E}(X_i)$, ${\sigma }^2=\mathbb{V}(X_i)$. Then,

$$\mathbb{E}({\bar{X}}_n)=\mu ,\mathbb{V}({\bar{X}}_n)=\frac{{\sigma }^2}{n},\text{and}\mathbb{E}(S_{n-1}^2)={\sigma }^2,$$

i.e., the sample mean and sample variance are unbiased estimators of $\mu$ and ${\sigma }^2$ respectively.

Covariance and Correlation

If $X$ and $Y$ are random variables, then the covariance and correlation between $X$ and $Y$ measure how strong the relationship is between $X$ and $Y$.

Definition

let $X$ and $Y$ be random variables with means ${\mu }_X$ and ${\mu }_Y$, and standard deviations ${\sigma }_X$ and ${\sigma }_Y$. Define the covariance between $X$ and $Y$ by

$$\text{Cov}(X,Y)=\mathbb{E}\left((X-{\mu }_X)(Y-{\mu }_Y)\right),$$

and the correlation by

$$\rho ={\rho }_{X,Y}=\rho (X,Y)=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}.$$

Theorem

The covariance satisfies

$$\text{Cov}(X,Y)=\mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y).$$

The correlation satisfies

$$-1\le \rho (X,Y)\le 1.$$

If $Y=aX+b$ for some constants $a$ and $b$, then $\rho (X,Y)=1$ if $a>0$, and $\rho (X,Y)=-1$ if $a<0$. If $X$ and $Y$ are independent, then $\text{Cov}(X,Y)=\rho =0$. The converse is not true in general.

Theorem

$\mathbb{V}(X+Y)=\mathbb{V}(X)+\mathbb{V}(Y)+2\text{Cov}(X,Y)$, and $\mathbb{V}(X-Y)=\mathbb{V}(X)+\mathbb{V}(Y)-2\text{Cov}(X,Y)$. More generally, for random variables $X_1,\dots ,X_n$,

$$\mathbb{V}\left(\sum _i{a}_i{X}_i\right)=\sum _i{a}_i^2\mathbb{V}(X_i)+2\sum \sum _{i<j}{a}_i{a}_j\text{Cov}(X_i,X_j).$$

Definition

For a random vector $\mathbf{X}$, the covariance matrix or variance-covariance matrix $\sum$ is a square matrix giving the covariance between each pair of random variables of the vector $\mathbf{X}$. It is defined by

$$\mathbb{V}(\mathbf{X})=\left[\begin{array}{cccc}\mathbb{V}(X_1)& \text{Cov}(X_1,X_2)& \cdots & \text{Cov}(X_1,X_n)\\ \text{Cov}(X_2,X_1)& \mathbb{V}(X_2)& \cdots & \text{Cov}(X_2,X_n)\\ ⋮& ⋮& \ddots & ⋮\\ \text{Cov}(X_n,X_1)& \text{Cov}(X_n,X_2)& \cdots & \text{V}(X_n)\end{array}\right].$$

Intuitively, the covariance matrix generalizes the notion of variance to multiple dimensions.