of how informative an observation will be. We do this by marginalizing over the Bayesian evidence, $p(x)$ to obtain $H(\Theta|X)$: $$\begin{aligned}
H(\Theta|X) &= \int_{x\in X} p(x)H(\Theta|x)dx \nonumber \\
            &= -\int_{x\in X} \int_{\theta\in \Theta} p(x)p(\theta|x)\log_2 p(\theta|x)d\theta dx,\end{aligned}$$

The average amount of information obtained from a given observation is equal to the difference between the entropy of the prior and the posterior distribution, $H(\Theta) - H(\Theta|X)$, which is also known as the mutual information, $I(\Theta,X)$, a quantity that is familiar in the context of Bayesian experimental design [@lindley1956; @Chaloner95bayesianexperimental; @journals/ploscb/LiepeFKS13]. The formal definition of the mutual information is: $$I(\Theta,X) = \int_\Theta \int_X p(x,\theta) \log_2\left(\frac{p(x,\theta)}{p(x)p(\theta)}\right) dxd\theta,$$ where $p(x,\theta)$ is the joint probability distribution, defined as $p(x,\theta) = p(\theta|x)p(x) = p(x|\theta)p(\theta)$, from Bayes’ Theorem: $$p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}.$$ Here, $p(\theta)$ is the prior distribution of the model parameters, $\theta$, and $p(x|\theta)$ is the likelihood, determined by the noise levels for the observation. Again, $p(x)$ is the Bayesian evidence (which in practice is handled internally by an MCMC algorithm).

Note that the mutual information is also symmetric: $I(\Theta,X) = I(X,\Theta)$. If $X$ and $\Theta$ are completely uncorrelated, then $I(\Theta,X) = 0$. The mutual information can also be defined in terms of the Kullback-Leibler divergence [@Kullback51klDivergence]: $$I(\Theta,X) 