### Mutual Information Theory Derivation

Let us begin with the definition of mutual information. 



Observations $x$ come from an unknown data generating distribution $x \sim P(X)$. The random variable $Y$ represents the weights of a neuron and its pdf is a Gaussian. Because it will be referenced frequently, we denote it with a $q(y)$. The weights of the neuron are updated after being stimulated by samples from the data generating distribution $P$. Therefore there is a dependence between the two and $p(x,y) \neq p(x)q(y)$. One way to quantify this interdependence with the mutual information

$$I(X;Y) = \int_{X\times Y} p(x,y) \log \frac{p(x,y)}{p(x)q(y)}d(x,y)$$

Fubini-Tonelli's Theorem tell us that as long as $X$ and $Y$ are $\sigma$-finite measure spaces and $f$ is a measurable function then

$${\displaystyle \int _{X\times Y}\big|f(x,y)\big|\,{\text{d}}(x,y)} = \int _{X}\left(\int _{Y}\big|f(x,y)\big|\,{\text{d}}y\right)\,{\text{d}}x=\int _{Y}\left(\int _{X}\big|f(x,y)\big|\,{\text{d}}x\right)\,{\text{d}}y$$

If we break up the integrand using logarithmic properties we can simplify the formula

$$I(X;Y) = \int_X \int_Y p(x,y) \log \frac{p(x,y)}{p(x)}{\text{d}}y \,{\text{d}}x - \int_Y \int_X p(x,y) \log p(y) {\text{d}}x \,{\text{d}}y$$

To simplify things futher wesubstitute $p(x,y) = p(y|x)p(x)$ in the left term

\begin{align}
I(X;Y) &= \int_X \int_Y  p(y|x)p(x) \log p(y|x){\text{d}}y \,{\text{d}}x - \int_Y \bigg( \int_X p(x,y) {\text{d}}x \bigg) \log p(y) {\text{d}}y \\
&= \int_X p(x) \left( \int_Y  p(y|x) \log p(y|x){\text{d}}y\right) \,{\text{d}}x - \int_Y p(y) \log p(y) {\text{d}}y \\ 
&= \mathbb{E}_X \left[ h(x) \right] + H(Y)
\end{align}

where $h(x)$ is the negative entropy of the conditional distribution $P(Y|X=x )$

### **To be done in the future**: Show that $h(x) \approx q(x)$

Assuming that $h(x) \approx q(x)$...

\begin{align}
q(\boldsymbol{x} | \theta) &= \frac{1}{\sqrt{(2\pi)^{k} 
|\Sigma|}} \exp \bigg\{ -\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^T\Sigma^{-1}
(\boldsymbol{x}-\boldsymbol{\mu}) \bigg \} \\
&= \frac{1}{C}\exp \{ -\frac{1}{2}d^2(\boldsymbol{x}, \boldsymbol{\mu};\Sigma)\}
\end{align}

where $C=\sqrt{(2\pi)^{k} |\Sigma|}$ is the volume under an unnormalized Gaussian and $d(\boldsymbol{x}, \boldsymbol{\mu};\Sigma)$ is the mahalanobis distance with respect to covariance matrix $\Sigma$.

Our objective is to learn a set of parameters $\theta$ that maximize the mutual information $I(X;Y)$, so we represent this as an objective function in terms of the parameters

\begin{align}
f(\mu,\Sigma) &= \mathbb{E}_X [q(\boldsymbol{x} | \mu, \Sigma) ] + H[q(\boldsymbol{x} | \mu, \Sigma)] \\ 
\end{align}

In addition we can calculate the differential entropy of the multivariate gaussian pdf for $Y$ analytically and substitute it into our objective.

\begin{align}
H[q(\boldsymbol{x} | \mu, \Sigma)] &= \frac{1}{2}\log | 2 \pi e \Sigma | \\
f(\mu,\Sigma) &= \mathbb{E}_X [q(\boldsymbol{x} | \mu, \Sigma) ] + \frac{1}{2}\log | 2 \pi e \Sigma | \\ 
\end{align}

To maximize the mutual information we maximize the objective using traditional techniques

\begin{align}
\frac{\partial f}{\boldsymbol{\mu}} &= \mathbb{E}_X \big[ \frac{\partial q(x|\boldsymbol{\mu}, \Sigma)}{\partial \boldsymbol{\mu}} \big] \\
\frac{\partial f}{\partial \Sigma} &= \mathbb{E}_X \big[ \frac{\partial q(x|\boldsymbol{\mu}, \Sigma)}{\partial \Sigma} \big] + \frac{1}{2}\Sigma^{-1} \\
\end{align}

We begin by recalling the partials of the multivariate gaussian pdf for both the mean $\boldsymbol{\mu}$ and the covariance $\Sigma$.

\begin{align}
\frac{\partial q}{\boldsymbol{\mu}} &= q(x|\boldsymbol{\mu}, \Sigma) \Sigma^{-1} (\boldsymbol{x}-\boldsymbol{\mu}) \\
\frac{\partial q}{\partial \Sigma} &= -q(x|\boldsymbol{\mu}, \Sigma) \frac{1}{2}(\Sigma^{-1}-\Sigma^{-1}(\boldsymbol{x}-\boldsymbol{\mu})(\boldsymbol{x}-\boldsymbol{\mu})^T\Sigma^{-1}) \\
\end{align}

(see appendix for derivations)

Substituting these back into our gradient terms we arrive at the following for the mean $\mu$

\begin{align}
\frac{\partial f}{\boldsymbol{\mu}} &=  \mathbb{E}_X \big[ q(x|\boldsymbol{\mu}, \Sigma)\Sigma^{-1} (\boldsymbol{x}-\boldsymbol{\mu}) \big] \\
&=  \Sigma^{-1} \mathbb{E}_X \big[ q(x|\boldsymbol{\mu}, \Sigma)(\boldsymbol{x}-\boldsymbol{\mu}) \big] \\
\end{align}

and below for $\Sigma$

\begin{align}
\frac{\partial f}{\partial \Sigma} &= \mathbb{E}_X \big[ -q(x|\boldsymbol{\mu}, \Sigma) \frac{1}{2}(\Sigma^{-1}-\Sigma^{-1}(\boldsymbol{x}-\boldsymbol{\mu})(\boldsymbol{x}-\boldsymbol{\mu})^T\Sigma^{-1}) \big] + \frac{1}{2}\Sigma^{-1} \\
&= \frac{1}{2}\Sigma^{-1} \bigg[ \mathbb{E}_X \big[q(x|\boldsymbol{\mu}, \Sigma) ((\boldsymbol{x}-\boldsymbol{\mu})(\boldsymbol{x}-\boldsymbol{\mu})^T-\Sigma) \big] + \Sigma \bigg] \Sigma^{-1} \\
\end{align}

If the expectation in the above expression equals zero, then then entire partial derivative is zero. Furthermore, if $\Sigma$ is invertible then it is the only way the partial can be zero.

Lastly we substitute the definition of the Gaussian $q$ from above.

\begin{align}
\frac{\partial f}{\partial \mu} &=  \Sigma^{-1} \mathbb{E}_X \big[ q(x|\boldsymbol{\mu}, \Sigma)(\boldsymbol{x}-\boldsymbol{\mu}) \big] \\
&=  \frac{1}{C}\Sigma^{-1} \mathbb{E}_X \big[\exp \{ -\frac{1}{2}d^2(\boldsymbol{x}, \boldsymbol{\mu};\Sigma) \}(\boldsymbol{x}-\boldsymbol{\mu}) \big] \\
\end{align}

Doing the same for the partial with respect to $\Sigma$ we arrive at

\begin{align}
\frac{\partial f}{\partial \Sigma} &= \frac{1}{2}\Sigma^{-1} \bigg[ \mathbb{E}_X \big[q(x|\boldsymbol{\mu}, \Sigma) ((\boldsymbol{x}-\boldsymbol{\mu})(\boldsymbol{x}-\boldsymbol{\mu})^T-\Sigma) \big] - \Sigma \bigg] \Sigma^{-1} \\
&= \frac{1}{2C}\Sigma^{-1} \bigg[ \mathbb{E}_X \big[\exp \{ -\frac{1}{2}d^2(\boldsymbol{x}, \boldsymbol{\mu};\Sigma) \} [(\boldsymbol{x}-\boldsymbol{\mu})(\boldsymbol{x}-\boldsymbol{\mu})^T-\Sigma] \big] + \Sigma\sqrt{(2\pi)^{k} |\Sigma|} \bigg] \Sigma^{-1} \\
\end{align}

Finally we can produce our stochastic approximation update equations...

\begin{align}
\mu_{i+1} &= \mu_i + \exp \{ -\frac{1}{2}d^2(\boldsymbol{x}_i, \boldsymbol{\mu}_i;\Sigma_i) \}(\boldsymbol{x}_i-\boldsymbol{\mu}_i) \\
\Sigma_{i+1} & = \Sigma_i + \exp \{ -\frac{1}{2}d^2(\boldsymbol{x}_i, \boldsymbol{\mu}_i;\Sigma_i) \} [(\boldsymbol{x}_i-\boldsymbol{\mu}_i)(\boldsymbol{x}_i-\boldsymbol{\mu}_i)^T-\Sigma_i] + \Sigma_i\sqrt{(2\pi)^{k} |\Sigma_i|} 
\end{align}


```python

```


```python

```

71a8bed2ef9fc21e41e7b43de115273f47071bc0
ipynb
Jupyter Notebook
dissertation/Mutual Information Derivation.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
2019-12-04T11:04:45.000Z
2019-12-04T11:04:45.000Z
dissertation/Mutual Information Derivation.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
dissertation/Mutual Information Derivation.ipynb
mathnathan/notebooks
63ae2f17fd8e1cd8d80fef8ee3b0d3d11d45cd28
MIT
Qwen/Qwen-72B
1. YES
2. YES