In  this  preliminary  work,  we  study  the  generalization  properties  of  infinite  ensembles  of infinitely-wide neural networks.   Amazingly, this model family admits tractable calculations for many information-theoretic quantities.   We report analytical and empirical investigations in the search for signals that correlate with generalization. A major area of research is to understand deep neural networks' remarkable ability to generalize to unseen examples. One promising research direction is to view deep neural networks through the lens of information theory (Tishby and Zaslavsky, 2015) . Abstractly, deep connections exist between the information a learning algorithm extracts and its generalization capabilities (Bassily et al., 2017; Banerjee, 2006) . Inspired by these general results, recent papers have attempted to measure information-theoretic quantities in ordinary deterministic neural networks (Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2017; Achille and Soatto, 2019) . Both practical and theoretical problems arise in the deterministic case (Amjad and Geiger, 2018; Saxe et al., 2018; Kolchinsky et al., 2018) . These difficulties stem from the fact that mutual information (MI) is reparameterization independent (Cover and Thomas, 2012) . 1 One workaround is to make a network explicitly stochastic, either in its activations (Alemi et al., 2016) or its weights (Achille and Soatto, 2017). Here we take an alternative approach, harnessing the stochasticity in our choice of initial parameters. That is, we consider an ensemble of neural networks, all trained with the same training procedure and data. This will generate an ensemble of predictions. Characterizing the generalization properties of the ensemble should characterize the generalization of individual draws from this ensemble. Infinitely-wide neural networks behave as if they are linear in their parameters (Lee et al., 2019) . Their evolution is fully described by the neural tangent kernel (NTK). The NTK is constant in time and can be tractably computed (Anonymous, 2020) . For our purposes, it can be considered to be a function of the network's architecture, e.g. the number and the structure of layers, nonlinearity, initial parameters' distributions, etc. All told, the output of an infinite ensemble of infinitely-wide neural networks initialized with Gaussian weights and biases and trained with gradient flow to minimize a square loss is simply a conditional Gaussian distribution: where z is the output of the network and x is its input. The mean µ(x, τ ) and covariance Σ(x, τ ) functions can be computed (Anonymous, 2020) . For more background on the NTK and NNGP as well as full forms of µ and Σ, see appendix A. This simple form allows us to bound several interesting information-theoretic quantities including: the MI between the representation and the targets (I(Z; Y ), appendix C.2), the MI between the representation and the inputs after training (I(Z; X|D), appendix C.3), and the MI between the representations and the training set, conditioned on the input (I(Z; D|X), appendix C.4), We are also able to compute in closed form: the Fisher information metric (appendix C.5), the distance the parameters move (appendix C.6), and the MI between the parameters and the data (I(Θ; D), appendix C.7). Because infinitely-wide neural networks are linear in their parameters, their information geometry in parameter space is very simple. The Fisher information metric is constant and flat, so the trace of the Fisher does not evolve as in Achille and Soatto (2019) . While the Euclidean distance the parameters move is small (Lee et al., 2019) , the distance they move according to the Fisher metric is finite. Finally, the MI between the data and the parameters tends to infinity, rendering PAC Bayes style bounds on generalization vacuous (Achille and Soatto, 2017; Banerjee, 2006; Bassily et al., 2017) . Infinite ensembles of infinitely-wide neural networks provide an interesting model family. Being linear in their parameters they permit a high number of tractable calculations of information-theoretic quantities and their bounds. Despite their simplicity, they still can achieve good generalization performance (Arora et al., 2019) . This challenges existing claims for the purported connections between information theory and generalization in deep neural networks. In this preliminary work, we laid the ground work for a larger-scale empirical and theoretical study of generalization in this simple model family. Given that real networks approach this family in their infinite width limit, we believe a better understanding of generalization in the NTK limit will shed light on generalization in deep neural networks. This makes them particularly analytically tractable. An infinitely-wide neural network, trained by gradient flow to minimize squared loss admits a closed form expression for evolution of its predictions as a function of time: Here z denotes the output of our neural network acting on the input x. τ is a dimensionless representation of the time of our training process. X denotes the whole training set of examples, with their targets Y. z 0 (x) ≡ z(x, τ = 0) denotes the neural networks output at initialization. The evolution is governed by the neural tangent kernel (NTK) Θ (Jacot et al., 2018) . For a finite width network, the NTK corresponds to JJ T , the gram matrix of neural network gradients. As the width of a network increases to infinity, this kernel converges in probability to a fixed value. There exist tractable ways to calculate the exact infinite-width kernel for wide classes of neural networks (Anonymous, 2020). The shorthand Θ denotes the kernel function evaluated on the train data (Θ ≡ Θ(X , X )). Notice that the behavior of infinitely-wide neural networks trained with gradient flow and squared loss is just a time-dependent affine transformation of their initial predictions. As such, if we now imagine forming an infinite ensemble of such networks as we vary their initial weight configurations, if those weights are sampled from a Gaussian distribution, the law of large numbers enforces that the distribution of outputs of the ensemble of networks at initialization is Gaussian, conditioned on its input. Since the evolution is an affine transformation of the initial predictions, the predictions remain Gaussian at all times. For more details see Lee et al. (2019) . Here, K denotes yet another kernel, the neural network gaussian process kernel (NNGP). For a finite width network, the NNGP corresponds to the expected gram matrix of the outputs: E zz T . In the infinite width limit, this concentrates on a fixed value. Just as for the NTK, the NNGP can be tractably computed (Anonymous, 2020), and should be considered just a function of the neural network architecture.
Infinite ensembles of infinitely wide neural networks are an interesting model family from an information theoretic perspective.