# Maximum Entropy

Information entropy is a function of probability distributions measure the quantity of information contained in a random variable (typically in bits).  Consider a binary random variable that takes on two values, $H$ and $L$.  The probability it takes the value $H$ is $p$ and the probability it takes value $L$ is $1-p$.  Then the average information content of this random variable is measured as,

\begin{align}
    H(p) &= -\left( p \log(p) + (1-p)\log(1-p) \right)
\end{align}

Choose the value of $p$ that maximizes the entropy function (i.e., maximizes the uncertainty, or information content of the binary random variable.)


```python
from sympy import *
from sympy.plotting import plot
init_printing()
```


```python
# Declare our symbols and setup the
# entropy function.
p = Symbol('p', real=True, positive=True)
H = -1*(p * log(p) + (1 - p) * log(1-p))

Eq(Symbol('H(p)'), H)
```


```python
# Take the derivative with respect to
# the probability of H: p.
DH = H.diff(p)
Eq(Symbol('\\frac{\partial H(p)}{\partial p}'), DH)
```


```python
# Set the derivative equal to zero
# and solve for p to find the value that
# satisfies the first order condition for
# a maximum.
p_star = solve(DH, p)[0]
Eq(Symbol('p^*'), p_star)
```


```python
# Find the second derivative so we can
# test the second order condition for
# a maximum.
D2H = H.diff(p).diff(p)
Eq(Symbol('\\frac{\partial^2 H(p)}{\partial p^2}'), D2H)
```


```python
# Evaluate the sign of the second derivative
# at the critical point we found, p*.
Eq(Symbol('\\frac{\partial^2 H(p)}{\partial p^2}|_{p^*}'), D2H.subs(p, p_star))
```

As we can see, the second derivative is negative and the critical point we found $p^* = 1/2$ is a maximum.  Thus, maximum entropy occurs at the uniform distribution $(p, 1-p) = (1/2, 1/2)$.


```python
plot(H, (p, 0.01, 0.99))
```

8d319c47498d8eb6034aff1ca488a1f89ff726f6
ipynb
Jupyter Notebook
assets/pdfs/math_bootcamp/notebooks/maximum_entropy.ipynb
joepatten/joepatten.github.io
4b9acc8720f3a33337368fee719902b54a6f2f68
MIT
assets/pdfs/math_bootcamp/notebooks/maximum_entropy.ipynb
joepatten/joepatten.github.io
4b9acc8720f3a33337368fee719902b54a6f2f68
MIT
2020-08-09T16:28:31.000Z
2020-08-10T14:48:57.000Z
assets/pdfs/math_bootcamp/notebooks/maximum_entropy.ipynb
joepatten/joepatten.github.io
4b9acc8720f3a33337368fee719902b54a6f2f68
MIT
Qwen/Qwen-72B
1. YES
2. YES