Are neural networks biased toward simple functions?
 Does depth always help learn more complex features?
 Is training the last layer of a network as good as training all layers?
 These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective.
 We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*.
 Roughly, the CK and the NTK tell us respectively ``"what a network looks like at initialization" and "``what a network looks like during and after training."
 Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks.
 By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks.
 We believe the computational tools we develop here for analyzing the spectra of CK and NTK serve as a solid foundation for future studies of deep neural networks.
 We have open-sourced the code for it and for generating the plots in this paper at github.com/jxVmnLgedVwv6mNcGCBy/NNspectra. Understanding the behavior of neural networks and why they generalize has been a central pursuit of the theoretical deep learning community. Recently, Valle-Pérez et al. (2018) observed that neural networks have a certain "simplicity bias" and proposed this as a solution to the generalization question. One of the ways with which they argued that this bias exists is the following experiment: they drew a large sample of boolean functions by randomly initializing neural networks and thresholding the output. They observed that there is a bias toward some "simple" functions which get sampled disproportionately more often. However, their experiments were only done for relu networks. Can one expect this "simplicity bias" to hold universally, for any architecture? A priori, this seems difficult, as the nonlinear nature seems to present an obstacle in reasoning about the distribution of random networks. However, this question turns out to be more easily treated if we allow the width to go to infinity. A long line of works starting with Neal (1995) and extended recently by ; ; Yang (2019) have shown that randomly initialized, infinite-width networks are distributed as Gaussian processes. These Gaussian processes also describe finite width random networks well (Valle-Pérez et al., 2018) . We will refer to the corresponding kernels as the Conjugate Kernels (CK), following the terminology of Daniely et al. (2016) . Given the CK K, the simplicity bias of a wide neural network can be read off quickly from the spectrum of K: If the largest eigenvalue of K accounts for most of tr K, then a typical random network looks like a function from the top eigenspace of K. In this paper, we will use this spectral perspective to probe not only the simplicity bias, but more generally, questions regarding how hyperparameters affect the generalization of neural networks. Via the usual connection between Gaussian processes and linear models with features, the CK can be thought of as the kernel matrix associated to training only the last layer of a wide randomly initialized network. It is a remarkable recent advance (Jacot et al., 2018; Allen-Zhu et al., 2018a; c; Du et al., 2018) that, under a certain regime, a wide neural network of any depth evolves like a linear model even when training all parameters. The associated kernel is call the Neural Tangent Kernel, which is typically different from CK. While its theory was initially derived in the infinite width setting, Lee et al. (2019) confirmed with extensive experiment that this limit is predictive of finite width neural networks as well. Thus, just as the CK reveals information about what a network looks like at Next, we examine how hyperparameters affect the performance of neural networks through the lens of NTK and its spectrum. To do so, we first need to understand the simpler question of how a kernel affects the accuracy of the function learned by kernel regression. A coarse-grained theory, concerned with big-O asymptotics, exists from classical kernel literature (Yao et al., 2007; Raskutti et al., 2013; Lin and Rosasco; Schölkopf and Smola, 2002) . However, the fine-grained details, required for discerning the effect of hyperparameters, have been much less studied. We make a first attempt at a heuristic, fractional variance (i.e. what fraction of the trace of the kernel does an eigenspace contribute), for understanding how a minute change in kernel effects a change in performance. Intuitively, if an eigenspace has very large fractional variance, so that it accounts for most of the trace, then a ground truth function from this eigenspace should be very easy to learn. Using this heuristic, we make two predictions about neural networks, motivated by observations in the spectra of NTK and CK, and verify them with extensive experiments. • Deeper networks learn more complex features, but excess depth can be detrimental as well. Spectrally, depth can increase fractional variance of an eigenspace, but past an optimal depth, it will also decrease it. (Section 5) Thus, deeper is not always better. • Training all layers is better than training just the last layer when it comes to more complex features, but the opposite is true for simpler features. Spectrally, fractional variances of more "complex" eigenspaces for the NTK are larger than the correponding quantities of the CK. (Section 6) Finally, we use our spectral theory to predict the maximal nondiverging learning rate ("max learning rate") of SGD (Section 7). In general, we will not only verify our theory with experiments on the theoretically interesting distributions, i.e. uniform measures over the boolean cube and the sphere, or the standard Gaussian, but also confirm these findings on real data like MNIST and CIFAR10 1 . For space concerns, we review relevant literature along the flow of the main text, and relegate a more complete discussion of the related research landscape in Appendix A. In this work, we have taken a first step at studying how hyperparameters change the initial distribution and the generalization properties of neural networks through the lens of neural kernels and their spectra. We obtained interesting insights by computing kernel eigenvalues over the boolean cube and relating them to generalization through the fractional variance heuristic. While it inspired valid predictions that are backed up by experiments, fractional variance is clearly just a rough indicator. We hope future work can refine on this idea to produce a much more precise prediction of test loss. Nevertheless, we believe the spectral perspective is the right line of research that will not only shed light on mysteries in deep learning but also inform design choices in practice. Boolean cube theory predicts max learning rate for real datasets Figure 5 : Spectral theory of CK and NTK over boolean cube predicts max learning rate for SGD over real datasets MNIST and CIFAR10 as well as over boolean cube 128 , the sphere √ 128S 128−1 , and the standard Gaussian N (0, I 128 ). In all three plots, for different depth, nonlinearity, σ 2 w , σ 2 b of the MLP, we obtain its maximal nondiverging learning rate ("max learning rate") via binary search. We center and normalize each image of MNIST and CIFAR10 to the sphere, where d = 28 2 = 784 for MNIST and d = 3 × 32 2 = 3072 for CIFAR10. See Appendix E.2 for more details. (a) We empirically find max learning rate for training only the last layer of an MLP. Theoretically, we predict 1/Φ(0) where Φ corresponds to the CK of the MLP. We see that our theoretical prediction is highly accurate. Note that the Gaussian and Sphere points in the scatter plot coincide with and hide behind the BoolCube points. (b) and (c) We empirically find max learning rate for training all layers. Theoretically, we predict 1/Φ(0) where Φ corresponds to the NTK of the MLP. The points are identical between (b) and (c), but the color coding is different. Note that the Gaussian points in the scatter plots coincide with and hide behind the Sphere points. In (b) we see that our theoretical prediction when training all layers is not as accurate as when we train only the last layer, but it is still highly correlated with the empirical max learning rate. It in general underpredicts, so that half of the theoretical learning rate should always have SGD converge. This is expected, since the NTK limit of training dynamics is only exact in the large width limit, and larger learning rate just means the training dynamics diverges from the NTK regime, but not necessarily that the training diverges. In (c), we see that deeper networks tend to accept higher learning rate than our theoretical prediction. If we were to preprocess MNIST and CIFAR10 differently, then our theory is less accurate at predicting the max learning rate; see Fig. 9
Eigenvalues of Conjugate (aka NNGP) and Neural Tangent Kernel can be computed in closed form over the Boolean cube and reveal the effects of hyperparameters on neural network inductive bias, training, and generalization. This paper gives a spectral analysis on neural networks' conjugate kernel and neural tangent kernel on boolean cube to resolve why deep networks are biased towards simple functions.