Entropy

The core idea of information theory is that the informational value of a communicated message depends on the degree to which the content of the message is surprising. If a highly likely event occurs, the message carries little information. On the other hand, if a highly unlikely event occurs, the message is much more informative.

The information content, also called the surprise or self-information, of an event $E$ is a function that increases as the probability $p(E)$ of an event decreases.

Definition

The information, or surprise, of an event $E$ is defined by

$$I(E)=\log \left(\frac{1}{p(E)}\right),$$

or equivalently,

$$I(E)=-\log \left(p(E)\right).$$

The logarithm gives 0 surprise when the probability of the event is 1. In fact, $\log$ is the only function that satisfies a specific set of conditions for information theory.

Definition (Entropy)

The entropy of a random variable $X$ with distribution $p$, denoted $\mathbb{H}(X)$, is a measure of its uncertainty. The entropy of a discrete random variable $X$, which takes values in the set $\mathcal{X}$ and is distributed accordingly to $p:\mathcal{X}\to [0,1]$ such that $p(x):=\mathbb{P}[X=x]$, is

$$\mathbb{H}(X)=\mathbb{E}[I(X)]=\mathbb{E}[-\log p(X)].$$

Note that $I(X)$ is itself a random variable. The entropy can be explicitly written as

$$\mathbb{H}(X)=-\sum _{x\in \mathcal{X}}p(x)\log p(x).$$

For continuous random variables, with probability density function $f(x)$, the differential entropy (or continuous entropy) is given by

$$\mathbb{H}(X)=\mathbb{E}[-\log f(X)]=-{\int }_{\mathcal{X}}f(x)\log f(x)dx.$$