To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions. In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression. In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions. Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions. With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions. We also find that there is a large amount of variation in compression between different network initializations. Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting. Finally, we show that only compression of the last layer is positively correlated with generalization. Although deep learning (reviewed by BID15 ) has produced astonishing advances in machine learning BID17 , a rigorous statistical explanation for the outstanding performance of deep neural networks (DNNs) is still to be found.According to the information bottleneck (IB) theory of deep learning BID18 BID16 ) the ability of DNNs to generalize can be seen as a type of representation compression. The theory proposes that DNNs use compression to eliminate noisy and task-irrelevant information from the input, while retaining information about the relevant segments BID1 . The information bottleneck method BID19 quantifies the relevance of information by considering an intermediate representation T between the original signal X and the salient data Y . T is the most relevant representation of X, and is said to be an information bottleneck, when it maximally compresses the input, retaining only the most relevant information, while maximizing the information it shares with the target variable Y . Formally, the information bottleneck minimizes the Lagrangian: DISPLAYFORM0 where I(·) is mutual information. In this Lagrangian β is the Lagrange multiplier, determining the trade-off between compression and retention of information about the target. In the context of deep learning, T is a layer's hidden activity represented as a single variable, X is a data set and Y is the set of labels. Compression for a given layer is signified by a decrease in I(T, X) value, while I(T, Y ) is increasing during training. Fitting behaviour refers to both values increasing. BID16 visualized the dynamic of training a neural network by plotting the values of I(T, X) and I(T, Y ) against each other. This mapping was named the information plane. According to IB theory the learning trajectory should move the layer values to the top left of this plane. In fact what was observed was that a network with tanh activation function had two distinct phases: fitting and compression. The paper and the associated talks 1 show that the compression phase leads to layers stabilizing on the IB bound. When this study was replicated by BID14 with networks using ReLU BID12 activation function instead of tanh, the compression phase did not happen, and the information planes only showed fitting throughout the whole training process. This behaviour required more detailed study, as a constant increase in mutual information between the network and its input implies increasing memorization, an undesired trait that is linked to overfitting and poor generalization BID11 .Measuring differential mutual information in DNNs is an ill-defined task, as the training process is deterministic BID14 . Mutual information of hidden activity T with input X is: DISPLAYFORM1 If we consider the hidden activity variable T to be deterministic then entropy is: DISPLAYFORM2 However, if T is continuous then the entropy formula is: DISPLAYFORM3 In the case of deterministic DNNs, hidden activity T is a continuous variable and p(T |X) is distributed as the delta function. For the delta function : DISPLAYFORM4 Thus, the true mutual information value I(T, X) is in fact infinite. However, to observe the dynamics of training in terms of mutual information, finite values are needed. The simplest way to avoid trivial infinite mutual information values, is to add noise to hidden activity.Two ways of adding noise have been explored previously by BID16 and BID14 . One way is to add noise Z directly to T and get a noisy variableT = T + Z. Then H(T |X) = H(Z) and mutual information is I(T , X) = H(T ) + H(Z). When the additive noise is Gaussian, the mutual information can be approximated using kernel density estimation (KDE), with an assumption that the noisy variable is distributed as a Gaussian mixture BID9 . The second way to add noise is to discretize the continuous variables into bins. To estimate mutual information , BID16 and BID14 primarily relied on binning hidden activity. The noise comes embedded with the discretization that approximates the probability density function of a random variable. In context of neural networks , adding noise can be done by binning hidden activity and approximating H(T ) as a discrete variable. In this case H(T |X) = 0 since the mapping is deterministic and I(T, X) = H(T ).Generally, when considering mutual information in DNNs, the analyzed values are technically the result of the estimation process and, therefore, are highly sensitive to it. For this reason it is vital to maintain consistency when estimating mutual information. The problem is not as acute when working with DNNs implemented with saturating activation functions, since all hidden activity is bounded. However, with non-saturating functions, and the resulting unbounded hidden activity, the level of noise brought by the estimation procedure has to be proportional and consistent, adapting to the state of every layer of the network at a particular epoch.In the next section adaptive estimation schemes are presented, both for the binning and KDE estimators. It is shown that for networks with unbounded activation functions in their hidden layers, the estimates of information change drastically. Moreover, the adaptive estimators are better able to evaluate different activation functions in a way that allows them to be compared. This approach shows considerable variation in compression for different activation functions. It also shows that L2 regularization leads to more compression and clusters all layers to the same value of mutual information. When compression in hidden layers is quantified with a compression metric and compared with generalization, no significant correlation is observed. However, compression of the last softmax layer is correlated with generalization. In this paper we proposed adaptive approaches to estimating mutual information in the hidden layers of DNNs. These adaptive approaches allowed us to compare behaviour of different activation functions and to observe compression in DNNs with non-saturating activation functions. However, unlike saturating activation functions, compression is not always present and is sensitive to initialization. This may be due to the minimal size of the network architecture that was tested. Experiments with larger convolutional neural networks could be used to explore this possibility.Different non-saturating activation functions compress information at different rates. While saturation plays a role in compression rates, we show that its absence does not imply absence of compression. Even seemingly similar activation functions, such as softplus and centered softplus, gave different compression scores. Compression does not always happen in later stages of training, but can happen from initialization. Further work is needed to understand the other factors contributing to compression.We also found that DNNs implemented with L2 regularization strongly compress information, forcing layers to forget information about the input. The clustering of mutual information to a single point on the information plane has never been reported previously. This result could lay the ground for further research to optimize the regularization to stabilize the layers on the information bottleneck bound to achieve better generalization BID0 , as well as linking information compression to memorization in neural networks BID20 .There are a few limitations to the analysis presented here. Principally , for tractability, the networks we explored were much smaller and more straightforward than many state of the art networks used for practical applications. Furthermore , our methods for computing information, although adaptive for any distribution of network activity, were not rigorously derived. Finally, our compression metric is ad-hoc. However, overall we have three main observations: first, compression is not restricted to saturating activation functions, second, L2 regularization induces compression, and third, generalization accuracy is positively correlated with the degree of compression only in the last layer and is not significantly affected by compression of hidden layers.
We developed robust mutual information estimates for DNNs and used them to observe compression in networks with non-saturating activation functions This paper studied the popular belief that deep neural networks do information compression for supervised tasks This paper proposes a method for the estimation of mutual information for networks with unbounded activation functions and the use of L2 regularization to induce more compression.