Inequalities for Expectations

Cauchy-Schwartz Inequality

https://en.wikipedia.org/wiki/Cauchy%E2%80%93Schwarz_inequality

Theorem (Cauchy-Schwartz Inequality)

If $X$ and $Y$ have finite variances, then

$$|\mathbb{E}[XY]|\le \sqrt{\mathbb{E}[X^2]\mathbb{E}[Y^2]}.$$

$\mathbb{E}[XY]$ measures how much $X$ and $Y$ align on average (positive when they tend to have the same sign and large together, negative when opposite). Cauchy-Schwartz essentially says that no matter how cleverly they align, the average product is bounded by the inner product of their root-mean-square.

If $\mathbb{E}[XY]$ shows up and we don’t know the joint distribution well, Cauchy-Schwartz lets you reduce it to $\mathbb{E}[X^2]$ and $\mathbb{E}[Y^2]$ which are often easier to control. It also shows up in covariance bounds

$$|\text{Cov}(X,Y)|=|\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]|\le \sqrt{\text{Var}\left(X\right)\text{Var}\left(Y\right)}.$$

Jensen’s Inequality

https://en.wikipedia.org/wiki/Jensen%27s_inequality

A function $g$ is convex if for each $x,y$ and each $\alpha \in [0,1]$,

$$g\left(\alpha x+(1-\alpha )y\right)\le \alpha g(x)+(1-\alpha )g(y).$$

If $g$ is twice differentiable and $g^{\prime \prime }(x)\ge 0$ for all $x$, then $g$ is convex. It can be shown that if $g$ is convex, then $g$ lies above any line that touches $g$ at some point, called the tangent line. A function $g$ is concave if $-g$ is convex.

Theorem (Jensen's Inequality)

If $g$ is convex, then

$$\mathbb{E}[g(X)]\ge g\left(\mathbb{E}[X]\right).$$

If $g$ is concave, then

$$\mathbb{E}[g(X)]\le g\left(\mathbb{E}[X]\right).$$

This can be visualized pictorially below.

Proof

Let $L(x)=a+bx$ be a line, tangent to $g(x)$ at the point $\mathbb{E}[X]$. Since $g$ is convex, it lies above the line $L(x)$. So,

$$\mathbb{E}[g(X)]\ge \mathbb{E}[L(X)]=\mathbb{E}[a+bX]=a+b\mathbb{E}[X]=L(\mathbb{E}[X])=g\left(\mathbb{E}[X]\right).$$