This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size. MINE is  back-propable and we prove that it is strongly consistent. We illustrate a handful of applications in which MINE is succesfully applied  to enhance the property of generative models in both unsupervised and supervised settings. We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems. Our results  demonstrate substantial added flexibility and improvement in these settings.
 Mutual information is an important quantity for expressing and understanding the relationship between random variables. As a fundamental tool of data science, it has found application in a range of domains and tasks, including applications to biomedical sciences, blind source separation (BSS, e.g., independent component analysis, BID23 , information bottleneck (IB, BID45 , feature selection BID28 BID36 , and causality BID8 .In contrast to correlation, mutual information captures the absolute statistical dependency between two variables, and thus can act as a measure of true dependence. Put simply, mutual information is the shared information of two random variables, X and Z, defined on the same probability space, (X ⇥ Z, F), where X ⇥ Z is the domain over both variables (such as R m ⇥ R n ), and F is the set of all possible outcomes over both variables. The mutual information has the form 1 : DISPLAYFORM0 where P XZ : F ! [0, 1] is a probabilistic measure (commonly known as a joint probability distribution in this context), and P X = R Z dP XZ and P Z = R X dP XZ are the marginals. The mutual information is notoriously difficult to compute. Exact computation is only tractable with discrete variables (as the sum can be computed exactly) or with a limited family of problems where the probability distributions are known and for low dimensions. For more general problems, common approaches include binning BID18 BID14 , kernel density estimation BID32 BID28 , Edgeworth expansion based estimators BID47 and likelihood-ratio estimators based on support vector machines (SVMs, e.g., BID43 . While the mutual information can be estimated from empirical samples with these estimators, they still make critical assumptions about the underlying distribution of samples, and estimate errors can reflect this. In addition , these estimators typically do not scale well with sample size or dimension.More recently, there has been great progress in the estimation of f -divergences BID34 and integral probability metrics (IPMs, Sriperumbudur et al., 2009 ) using deep neural networks (e.g., in the context of f -divergences and the Wasserstein distance or Fisher IPMs, BID35 BID4 BID33 . These methods are at the center of generative adversarial networks (GANs Goodfellow et al., 2014) , which train a generative model without any explicit assumptions about the underlying distribution of the data. One perspective on these works is that, given the correct constraints on a neural network, the network can be used to compute a variational lower-bound on the distance or divergence of implicit probability measures.In this paper we look to extend this estimation strategy to mutual information as given in equation 1, which we note corresponds to the Kullback-Leibler (KL-) divergence BID27 between the joint, P XZ and the product of the marginal distributions, P X ⌦ P Z , i.e., D KL (P XZ || P X ⌦ P Z ). This observation can be used to help formulate variational Bayes in terms of implicit distributions BID30 or INFOMAX BID7 .We introduce an estimator for the mutual information based on the Donsker-Varadhan representation of the KL-divergence BID38 . As with those introduced by BID35 , our estimator is scalable, flexible, and is completely trainable via back-propagation. The contributions of this paper are as follows.• We introduce the mutual information neural estimator (MINE), providing its theoretical bases and generalizability to other information metrics.• We illustrate that our estimator can be used to train a model with improved support coverage and richer learned representation for training adversarial models (such as adversariallylearned inferences, ALI, Dumoulin et al., 2016 ).• We demonstrate how to use MINE to improve reconstructions and inference in Adversarially Learned Inference Dumoulin et al. FORMULA0 on large scale Datasets.• We show that our estimator provides a method of performing the Information Bottleneck method BID45 in a continuous setting, and that this approach outperforms variational bottleneck methods BID1 . We proposed a mutual information estimator, which we called the mutual information neural estimator (MINE), that is scalable in dimension and sample-size. We demonstrated the efficiency of this estimator by applying it in a number of settings. First, a term of mutual information can be introduced alleviate mode-dropping issue in generative adversarial networks (GANs, Goodfellow et al., 2014) . Mutual information can also be used to improve inference and reconstructions in adversarially-learned inference (ALI, Dumoulin et al., 2016) . Finally, we showed that our estimator allows for tractable application of Information bottleneck methods BID45 in a continuous setting.through co-occurrence. We illustrate this perspective by considering distributions on natural image manifolds.Consider a random image in [0, 1] d by randomly sampling the intensity of each pixel independently. This image will show very little structure when compared to an image sampled form the manifold of natual images, M nature ⇢ [0, 1] d , as the latter is is bound to respect a number of physically possible priors (such as smoothness). We expect the mutual information of the pixels of images arising from M nature to be high. Differently put, the larger the number of simultaneously co-occurring subset of pixels in [0, 1] d , the higher the mutual information. In the language of cumulants tensors, the larger ponderation of higher order cumulants tensor in the cumulant generating function of the joint distribution over the pixels, the higher the mutual information, and the more structure there is to be found amongst the pixels. Note that the case of mutually independent pixels corresponds to joint distribution where the only cumulants contributing the joint distribution are of order one. This is the corner case where the joint distribution equals the product of marginals. Thus in order to assess the amount of structure it is enough to score how the joint distribution is different from the product of marginals. As we show in the paper, this principle can be extended to different divergences as well.
A scalable in sample size and dimensions mutual information estimator.