Kullback-Leibler Divergence

The Kullback-Leibler Divergence (also called relative entropy), denoted $D_{\mathrm{KL}}(P\mid \mid Q)$, is a type of statistical distance, measuring how much an approximating probability distribution $Q$ is different from a true probability distribution $P$.

Definition

For discrete probability distributions $P$ and $Q$ defined on a sample space $\mathcal{X}$, the relative entropy from $Q$ to $P$ is defined to be

$$D_{\mathrm{KL}}(P\mid \mid Q)=\sum _{x\in \mathcal{X}}P(x)\log \frac{P(x)}{Q(x)}.$$

For distributions $P$ and $Q$ of a continuous random variable with probability density functions $p$ and $q$ respectively, relative entropy is defined to be

$$D_{\mathrm{KL}}(P\mid \mid Q)={\int }_{-\infty }^{\infty }p(x)\log \frac{p(x)}{q(x)}dx.$$

In other words, the KL divergence is the expected excess suprise from using the approximation $Q$ instead of $P$ when the actual is $P$. Note that it is not an actual metric, since it is not symmetric.

Often, $P$ represents the data, the observations, or a measured probability distribution, and $Q$ represents a model.