Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy. Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization. In order to understand the intriguing generalization properties of deep neural networks highlighted by BID22 BID33 BID15 , the identification of numerical indicators of generalization performance that remain applicable across a diverse set of training settings is critical. A well-known and extensively studied example of such indicator is the width of the minima the network has converged to BID11 BID15 .In this paper, we present empirical evidence supporting the discovery of a novel indicator of generalization: the evolution across training of the cosine distance between each layer's weight vector and its initialization (denoted by layer rotation). Indeed , we show across a diverse set of experiments (with varying datasets, networks and training procedures), that larger layer rotations (i.e. larger cosine distance between final and initial weights of each layer) consistently translate into better generalization performance. In addition to providing an original perspective on generalization, our experiments suggest that layer rotation also BID0 ICTEAM, Université catholique de Louvain, Louvain-LaNeuve, Belgium. <simon.carbonnelle@uclouvain.be>.benefits from the following properties compared to alternative indicators of generalization:• It is easily monitored and, since it only depends on the evolution of the network's weights, can be controlled along the optimization through appropriate weight update adjustments • It has a network-independent optimum (all layers reaching a cosine distance of 1) • It provides a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.In comparison, other indicators usually provide a metric to optimize (e.g. the wider the minimum, the better) but no clear optimum to be reached (what is the optimal width?), nor a precise methodology to tune it (how to converge to a minimum with a specific width?). By disclosing simple guidelines to tune layer rotations and an easy-to-use controlling tool, our work can also help practitioners get the best out of their network with minimal hyper-parameter tuning.The presentation of our experimental study is structured according to three successive steps:1. Development of tools to monitor and control layer rotation (Section 2); 2. Systematic study of layer rotation configurations in a controlled setting (Section 3); 3. Study of layer rotation configurations in standard training settings, with a special focus on SGD, weight decay and adaptive gradient methods (Section 4).Related work is discussed in Supplementary Material.
This paper presents empirical evidence supporting the discovery of an indicator of generalization: the evolution across training of the cosine distance between each layer's weight vector and its initialization.