R. Moon[^1]'
- 'Kumar Sricharan[^2]'
- 'Alfred O. Hero III[^3]'
bibliography:
- 'References.bib'
title: Ensemble Estimation of Mutual Information
---

\#1[\_[\#1,h\_[\#1]{}]{}]{} \#1[\_[\#1]{}]{}

\#1[\_[\#1,h\_[\#1]{}]{}]{} \#1[\_[\#1,h]{}]{} \#1[\_[\#1,h\_[\#1]{}(l)]{}]{} \#1[\_[\#1,h\_[\#1]{}]{}]{} \#1[\_[\#1,h]{}]{}

§ \#1[\_]{}

\#1[\_[\#1,k\_[\#1]{}]{}]{} \#1[\_[\#1,k\_[\#1]{}+1]{}]{}

\#1[\_[\#1,k\_[\#1]{}]{}]{} \#1\#2[\_[\#1,k(\#2)]{}]{} \#1[\_[\#1,k\_[\#1]{}+1]{}]{} \#1[|\_[\#1,k\_[\#1]{}]{}]{} \#1[|\_[\#1,k\_[\#1]{}+1]{}]{}

\#1\#2[\_[\#1\#2,h\_[\#1]{},h\_[\#2]{}]{}]{}

\#1\#2\#3[\_[\#2]{}\^[(\#3)]{}]{}

Introduction {#sec:intro}
============

Mutual information (MI) estimation has many applications in machine learning including MI has been used in fMRI data processing [@chai2009fMRI], structure learning [@structure2016], independent subspace analysis [@pal2010estimation], forest density estimation [@liu2012exponential], clustering [@lewi2006real], neuron classification [@schneidman2003information], and intrinsically motivated reinforcement learning [@mohamed2015variational; @salge2014changing]. Another particularly common application is feature selection or extraction where features are chosen to maximize the MI between the chosen features $\mathbf{X}$ and the outcome variables $\mathbf{Y}$ [@torkkola2003feature; @vergara2014review; @peng2005feature; @kwak2002input]. In many of these applications, the predictor labels have discrete components (e.g. classification labels) while the input variables have continuous components. To the best of our knowledge, there are currently no nonparametric MI estimators that are known to achieve the parametric mean squared error (MSE) convergence rate $1/N$ when $\X$ and/or $\Y$ contain discrete components. Also, while many nonparametric estimators of MI exist, most can only be applied to specific information measures (e.g. Shannon or Rényi information). In this paper, we provide a framework for 