We study the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information I(X;T) between the input X and internal representations T decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true I(X;T) over these networks is provably either constant (discrete X) or infinite (continuous X). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which I(X;T) is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for I(X;T) in noisy DNNs and observe compression in various models. By relating I(X;T) in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the T space. Finally, we return to the estimator of I(X;T) employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest. Recent work by BID10 uses the Information Bottleneck framework BID13 BID12 to study the dynamics of DNN learning. The framework considers the mutual information pair I(X; T ), I(Y ; T ) between the input X or the label Y and the network's hidden layers T . Plotting the evolution of these quantities during training, BID10 made two interesting observations: (1) while I(Y ; T ) remains mostly constant as the layer index increases, I(X; T ) decreases, suggesting that layers gradually shed irrelevant information about X; and (2) after an initial fitting phase, there is a long compression phase during which I(X; T ) slowly decreases. It was suggested that this compression is responsible for the generalization performance of DNNs. A follow-up paper contends that compression is not inherent to DNN training, claiming double-sided saturating nonlinearities yield compression while single-sided/non-saturating ones do not necessarily compress. BID10 and present many plots of I(X; T ), I(Y ; T ) evolution across training epochs. These plots, however, are inadvertently misleading: they show a dynamically changing I(X; T ) when the true mutual information is provably either infinite or a constant independent of the DNN's parameters (see BID1 for a discussion of further degeneracies related to to the Information Bottleneck framework). Recall that the mutual information I(X; T ) is a functional of the joint distribution of (X, T ) ∼ P X,T = P X P T |X , and that, in standard DNNs, T is a deterministic function of X. Hence, if P X is continuous, then so is T , and thus I(X; T ) = ∞ (cf. (Polyanskiy & Wu, 2012 , Theorem 2.4)). If P X is discrete (e.g., when the features are discrete or if X adheres to an empirical distribution over the dataset), then the mutual information is a finite constant that does not depend on the parameters of the DNN. Specifically, for deterministic DNNs, the mapping from a discrete X to T is injective for strictly monotone nonlinearities such as tanh or sigmoid, except for a measure-zero set of weights. In other words, deterministic DNNs can encode all information about a discrete X in arbitrarily fine variations of T , causing no loss of information and implying I(X; T ) = H(X), even if deeper layers have fewer neurons.The compression observed in BID10 and therefore cannot be due to changes in mutual information. This discrepancy between theory and experiments originates from a theoretically unjustified discretization of neuron values in their approximation of I(X; T ). To clarify, the quantity computed and plotted in these works is I(X; Bin(T )), where Bin is a per-neuron discretization of each hidden activity of T into a user-selected number of bins. This I X; Bin(T ) is highly sensitive to the selection of bin size (as illustrated in FIG0 ) and does not track I(X; T ) for any choice of bin size.1 Nonetheless, compression results based on I X; Bin(T ) are observed by BID10 and in many interesting cases.To understand this curious phenomenon we first develop a rigorous framework for tracking the flow of information in DNNs. In particular, to ensure I(X; T ) is meaningful for studying the learned representations, we need to make the map X → T a stochastic parameterized channel whose parameters are the DNN's weights and biases. We identify several desirable criteria that such a stochastic DNN framework should fulfill for it to provide meaningful insights into commonly used practical systems.(1 ) The stochasticity should be intrinsic to the operation of the DNN, so that the characteristics of mutual information measures are related to the learned internal representations, and not to an arbitrary user-defined parameter. ( 2) The stochasticity should relate the mutual information to the deterministic binned version I X; Bin(T ) , since this is the object whose compression was observed; this requires the injected noise to be isotropic over the domain of T analogously to the per-neuron binning operation. And most importantly, (3) the network trained under this stochastic model should be closely related to those trained in practice.We propose a stochastic DNN framework in which independent and identically distributed (i.i.d.) Gaussian noise is added to the output of each of the DNN's neurons. This makes the map from X to T stochastic, ensures the data processing inequality (DPI) is satisfied, and makes I(X; T ) reflect the true operating conditions of the DNN, following Point (1). Since the noise is centered and isotropic, Point (2) holds. As for Point (3), Section 2 experimentally shows the DNN's learned representations and performance are not meaningfully affected by the addition of noise, for variances β 2 not too large. Furthermore, randomness during training has long been used to improve neural network performance, e.g., to escape poor local optima BID4 , improve generalization performance BID11 , encourage learning of disentangled representations BID0 , and ensure gradient flow with hard-saturating nonlinearities BID3 . Under the stochastic model, I(X; T ) has no exact analytic expression and is impossible to approximate numerically. In Section 3 we therefore propose a sampling technique that decomposes the estimation of I(X; T ) into several instances of a simpler differential entropy estimation problem: estimating h(S + Z) given n samples of the d-dimensional random vector S and knowing the distribution of Z ∼ N (0, DISPLAYFORM0 We analyze this problem theoretically and show that any differential entropy estimator over the noisy DNN requires at least exponentially many samples in the dimension d. Leveraging the explicit modeling of S + Z, we then propose a new estimator that converges 1 Another approach taken in considers I(X; T + Z) (instead of I X; Bin(T ) ), where Z is an independent Gaussian with a user-defined variance. This approach has two issues: (i) the values as a function of may violate the data processing inequality, and (ii) they do not reflect the operation of the actual DNN, which was trained without noise. We focus on I X; Bin(T ) because it was commonly used in BID10 and , and since both methods have a similar effect of blurring T .as O (log n) d/4 / √ n , which significantly outperforms the convergence rate of general-purpose differential entropy estimators when applied to the noisy DNN framework.We find that I(X; T ) exhibits compression in many cases during training of small DNN classifiers. To explain compression in an insightful yet rigorous manner, Section 4 relates I(X; T ) to the well-understood notion of data transmission over additive white Gaussian noise (AWGN) channels. Namely , I(X; T ) is the aggregate information transmitted over the channel P T |X with input X drawn from a constellation defined by the data samples and the noisy DNN parameters. As training progresses, the representations of inputs from the same class tend to cluster together and become increasingly indistinguishable at the channel's output, thereby decreasing I(X; T ). Furthermore , these clusters tighten as one moves into deeper layers, providing evidence that the DNN's layered structure progressively improves the representation of X to increase its relevance for Y .Finally, we examine clustering in deterministic DNNs. We identify methods for measuring clustering that are valid for both noisy and deterministic DNNs, and show that clusters of inputs in learned representations typically form in both cases. We complete the circle back to I X; Bin(T ) by clarifying why this binned mutual information measures clustering. This explains what previous works were actually observing: not compression of mutual information, but increased clustering by hidden representations. The geometric clustering of hidden representations is thus the fundamental phenomenon of interest, and we aim to test its connection to generalization performance, theoretically and experimentally, in future work. In this work we reexamined the compression aspect of the Information Bottleneck theory (ShwartzZiv & Tishby, 2017) , noting that fluctuations of I(X; T ) in deterministic networks with strictly monotone nonlinearities are theoretically impossible. Setting out to discover the source of compression observed in past works, we: (i) created a rigorous framework for studying and accurately estimating information-theoretic quantities in DNNs whose weights are fixed; (ii) identified clustering of the learned representations as the phenomenon underlying compression; and (iii) demonstrated that the compression-related experiments from past works were in fact measuring this clustering through the lens of the binned mutual information. In the end, although binning-based measures do not accurately estimate mutual information, they are simple to compute and prove useful for tracking changes in clustering, which is the true effect of interest in deterministic DNNs. We believe that further study of geometric phenomena driven by DNN training is warranted to better understand the learned representations and to potentially establish connections with generalization.
Deterministic deep neural networks do not discard information, but they do cluster their inputs. This paper provides a principled way to examine the compression phrase in deep neural networks by providing an theoretical sounding entropy estimator to estimate mutual information. 