Variance and Covariance

Variance

The variance measures the spread of a distribution.

Definition

Let $X$ be a random variable with mean $\mu$. The variance of $X$, denoted by ${\sigma }^2$ or ${\sigma }_X^2$ or $\mathbb{V}(X)$, is defined by

$${\sigma }^2=\mathbb{E}(X-\mu {)}^2=\int (x-\mu {)}^{2}dF(x),$$

assuming this expectation exists. The standard deviation is $\text{sd}(X)=\sqrt{\mathbb{V}(x)}$, and is denoted by $\sigma$ and ${\sigma }_X$.

We can’t use $\mathbb{E}(X-\mu )$ as a measure of spread, since $\mathbb{E}(X-\mu )=\mathbb{E}(X)-\mu =\mu -\mu =0$. We can sometimes use $\mathbb{E}|X-\mu |$ as a measure of spread, but often we use the variance.

Theorem

Assuming the variance is well defined, it has the following properties:

1. $\mathbb{V}(X)=\mathbb{E}(X^2)-{\mu }^2$.
2. If $a$ and $b$ are constants, then $\mathbb{V}(aX+b)=a^2\mathbb{V}(X)$.
3. If $X_1,\dots ,X_n$ are independent and $a_1,\dots ,a_n$ are constants, then

$$\mathbb{V}\left(\sum _{i=1}^n{a}_i{X}_i\right)=\sum _{i=1}^n{a}_i^2\mathbb{V}(X_i).$$

Sample Mean and Sample Variance

Definition

If $X_1,\dots ,X_n$ are random variables, then we define the sample mean to be

$${\bar{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i,$$

and the sample variance to be

$$S_n^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-{\bar{X}}_n{)}^2.$$

Theorem

Let $X_1,\dots ,X_n$ be IID and let $\mu =\mathbb{E}(X_i)$, ${\sigma }^2=\mathbb{V}(X_i)$. Then,

$$\mathbb{E}({\bar{X}}_n)=\mu ,\mathbb{V}({\bar{X}}_n)=\frac{{\sigma }^2}{n},\text{and}\mathbb{E}(S_n^2)={\sigma }^2.$$

Covariance and Correlation

If $X$ and $Y$ are random variables, then the covariance and correlation between $X$ and $Y$ measure how strong the relationship is between $X$ and $Y$.

Definition

let $X$ and $Y$ be random variables with means ${\mu }_X$ and ${\mu }_Y$, and standard deviations ${\sigma }_X$ and ${\sigma }_Y$. Define the covariance between $X$ and $Y$ by

$$\text{Cov}(X,Y)=\mathbb{E}\left((X-{\mu }_X)(Y-{\mu }_Y)\right),$$

and the correlation by

$$\rho ={\rho }_{X,Y}=\rho (X,Y)=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}.$$

Theorem

The covariance satisfies

$$\text{Cov}(X,Y)=\mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y).$$

The correlation satisfies

$$-1\le \rho (X,Y)\le 1.$$

If $Y=aX+b$ for some constants $a$ and $b$, then $\rho (X,Y)=1$ if $a>0$, and $\rho (X,Y)=-1$ if $a<0$. If $X$ and $Y$ are independent, then $\text{Cov}(X,Y)=\rho =0$. The converse is not true in general.

Theorem

$\mathbb{V}(X+Y)=\mathbb{V}(X)+\mathbb{V}(Y)+2\text{Cov}(X,Y)$, and $\mathbb{V}(X-Y)=\mathbb{V}(X)+\mathbb{V}(Y)-2\text{Cov}(X,Y)$. More generally, for random variables $X_1,\dots ,X_n$,

$$\mathbb{V}\left(\sum _i{a}_i{X}_i\right)=\sum _i{a}_i^2\mathbb{V}(X_i)+2\sum \sum _{i<j}{a}_i{a}_j\text{Cov}(X_i,X_j).$$

Definition

For a random vector $\mathbf{X}$, the covariance matrix or variance-covariance matrix $\sum$ is a square matrix giving the covariance between each pair of random variables of the vector $\mathbf{X}$. It is defined by

$$\mathbb{V}(\mathbf{X})=\left[\begin{array}{cccc}\mathbb{V}(X_1)& \text{Cov}(X_1,X_2)& \cdots & \text{Cov}(X_1,X_n)\\ \text{Cov}(X_2,X_1)& \mathbb{V}(X_2)& \cdots & \text{Cov}(X_2,X_n)\\ ⋮& ⋮& \ddots & ⋮\\ \text{Cov}(X_n,X_1)& \text{Cov}(X_n,X_2)& \cdots & \text{V}(X_n)\end{array}\right].$$

Intuitively, the covariance matrix generalizes the notion of variance to multiple dimensions.