We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks.   The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks.   Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped).   Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc. Neural networks (NNs), both in general BID0 as well as in their most recent incarnation as deep neural networks (DNNs) as used in deep learning BID1 , are of interest not only for their remarkable empirical performance on a variety of machine learning (ML) tasks, but also since they exhibit rather complex properties that have led researchers to quite disparate conclusions about their behavior. For example, some papers lead with the claim that DNNs are robust to a massive amount of noise in the data and/or that noise can even help training (3; 4; 5) , while others discuss how they are quite sensitive to even a modest amount of noise (6; 7); some papers express surprise that the popular Probably Approximately Correct (PAC) theory and Vapnik-Chervonenkis (VC) theory do not describe well their properties BID6 , while others take it as obvious that those theories are not particularly appropriate for understanding NN learning (8; 9; 10; 11; 12; 13); many papers point out how the associated optimization problems are extremely non-convex and lead to problems like local minima, while others point out how non-convexity and local minima are never really an issue (14; 15; 16; 17; 18; 19) ; some advocate for convergence to flat minimizers BID19 , while others seem to advocate that convergence to sharp minima can generalize just fine BID20 ; and so on.These tensions have been known for a long time in the NN area, e.g., see (22; 23; 24; 25; 26; 10; 27; 14) , but they have received popular attention recently due to the study of Zhang et al. BID6 . This recent study considered the tendency of state-of-the-art DNNs to overtrain when presented with noisy data, and its main conclusions are the following.Observation 1 (Neural networks can easily overtrain.) State-of-the-art NNs can easily minimize training error, even when the labels and/or feature vectors are noisy, i.e., they easily fit to noise and noisy data (although, we should note, we found that reproducing this result was not so easy). This implies that state-of-the-art deep learning systems, when presented with realistic noisy data, may always overtrain.Observation 2 (Popular ways to regularize may or may not help.) Regularization (more precisely, many recently-popular ways to implement regularization) fails to prevent this. In particular, methods that implement regularization by, e.g., adding a capacity control function to the objective and approximating the modified objective, performing dropout, adding noise to the input, and so on, do not substantially improve the situation. Indeed, the only control parameter 1 that has a substantial regularization effect is early stopping.To understand why this seems peculiar to many people trained in statistical data analysis, consider an SVM, where this does not happen. Let's say one has a relatively-good data set, and one trains an SVM with, say, 90% training accuracy. Then, clearly, the SVM generalization accuracy, on some other test data set, is bounded above by 90%. If one then randomizes, say, 10% of the labels, and one retrains the SVM, then one may overtrain and spuriously get a 90% training accuracy. Textbook discussions, however, state that one can always avoid overtraining by tuning regularization parameters to get better generalization error on the test data set. In this case, one expects the tuned training and generalization accuracies to be bounded above by roughly 90 − 10 = 80%. Observation 1 and Observation 2 amount to saying that DNNs behave in a qualitatively different way.Given the well-known connection between the capacity of models and bounds on generalization ability provided by PAC/VC theory and related methods based on Rademacher complexity, etc. (28; 29) , a grand conclusion of Zhang et al. FORMULA20 is that understanding the properties of DNN-based learning "requires rethinking generalization." We agree. Moreover, we think this rethinking requires going beyond recently-popular ML methods to revisiting old ideas on generalization and capacity control from the statistical mechanics of NNs (9; 30; 11; 31) .Here , we consider the statistical mechanics (SM) theory of generalization, as applied to NNs and DNNs. We show how a very simple application of it can provide a qualitative explanation of recently-observed empirical properties that are not easily-understandable from within PAC/VC theory of generalization, as it is commonly-used in ML. The SM approach (described in more detail in Sections 2 and A.2) can be formulated in either a "rigorous" or a "non-rigorous" manner. The latter approach, which does not provide worst-case a priori bounds, is more common, but the SM approach can provide precise quantitative agreement with empirically-observed results (as opposed to very coarse bounds) along the entire learning curve, and it is particularly appropriate for models such as DNNs where the complexity of the model grows with the number of data points. In addition, it provides a theory of generalization in which, in appropriate limits, certain phenomenon such as phases, phase transitions, discontinuous learning, and other complex learning behavior arise very naturally, as a function of control parameters of the ML process. Most relevant for our discussion are load-like parameters and temperature-like parameters. While the phenomenon described by the SM approach are not inconsistent with the more well-known PAC/VC approach, the latter is coarse and typically formulated in such a way that these phenomenon are not observed in the theory. Schematic of error plots, phase diagrams, and the process of adding noise to input data and then adjusting algorithm knobs for our new VSDL model of classification in DNN learning models. We describe this in Claims 1, 2 and 3 in Section 3.We propose that the two parameters used by Zhang et al. FORMULA20 (and many others), which are control parameters used to control the learning process, are directly analogous to load-like and temperature-like parameters in the traditional SM approach to generalization. (Some readers may be familiar with these two parameters from the different but related Hopfield model of associative memory (32; 33) , but the existence of two or more such parameters holds more generally (9; 30; 11; 34; 31) .) Given these two identifications, which are novel to this work, general considerations from the SM theory of generalization, applied even to very simple models like the VSDL model, suggest that complex and non-trivial generalization properties-including the inability not to overfit to noisy data-emerge very naturally, as a function of these two control parameters. In particular , we note the following (which amount to explaining Observations 1 and 2).• One-dimensional phase diagram. FIG1 illustrates the behavior of the generalization error as a function of increasing (from left to right, or decreasing, from right to left) the load parameter α. There is a critical value α c where the the generalization properties change dramatically, and for other values of α the generalization properties change smoothly.• Two-dimensional phase diagram. FIG1 illustrates the phase diagram in the two-dimensional space defined by the α and τ parameters. In this figure, the boundaries between different phases mark sharp transitions in the generalization properties of the system, and within a given phase the generalization properties of the system vary smoothly.• Adding noise and parameter fiddling . FIG1 illustrates the process of adding noise to data and adjusting algorithm knobs to compensate. Starting from the (α, τ ) point A, which exhibits good generalization behavior, adding noise casues α to decrease, leading to point B, which exhibits poor generalization. This can be offset by adjusting (for A → B → C, this means decreasing) the number of iterations to modify the τ parameter, again leading to good generalization. FIG1 (c) also illustrates that, starting from the (α, τ ) point A , adding noise casues α to decrease, leading to point B , which also has poor generalization, and this can be offset by adjusting (except for A → B → C , this means increasing) the number of iterations to modify the τ parameter to obtain point C .The VSDL model and these consequences are described in more detail in Sections 3.1 and 3.2.We should note that the SM approach to generalization can lead to quantitative results, but to achieve this can be technically quite complex (9; 30; 11; 34; 31) . Thus, in this paper, we do not focus on these technical complexities, lest the simplicity of our main contribution be lost, but we instead leave that for future work. On the other hand, the basic ideas and qualitative results are quite simple, even if somewhat different than the ideas underlying the more popular PAC/VC approach (9; 30; 11; 34; 31) .While it should go without saying, one should of course be careful about naïvely interpreting our results to make extremely broad claims about realistic DNN systems. Realistic DNNs have many more control parameters-the amount of dropout, SGD block sizes, learning rate schedules, the number of iterations, layer normalization, weight norm constraints, etc.-and these parameters can interact in very complicated ways. Thus, an important more general insight from our approach is that-depending strongly on the details of the model, the specific details of the learning algorithm, the detailed properties of the data and their noise, etc. (which are not usually described sufficiently well in publications to reproduce their main results)-going beyond worst-case bounds can lead to a rich and complex array of manners in which generalization can depend on the control parameters of the ML process.In the next section, Section 2, we will review some relevant background; and then, in Section 3, we will present our main contributions on connecting practical DNN control parameters with load-like parameters, temperature-like parameters, and non-trivial generalization behavior in a VSDL model. In Section A, we will provide a more detailed discussion and explanation of our main result; and in Section 4, we will provide a brief discussion and conclusion. The approach we have adopted to rethinking generalization is to ask what is the simplest possible model that reproduces non-trivial properties of realistic DNNs. In the VSDL model, we have idealized very complex DNNs as being controlled by two control parameters, one describing an effective amount of data or load on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we have explained how a very simple application of ideas from the SM theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of DNNs not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.As we were writing up this paper, we became aware of recent work with a similar flavor (44; 45; 46) . In BID44 , the authors consider a more refined scale-sensitive analysis involving a Lipshitz constant of the network, and they make connections with margin-based boosting methods to scale the Lipshitz constant. In BID45 , the authors use Information Bottleneck ideas to analyze how information is compressed early versus late in the running of stochastic optimization algorithms, when training error improves versus when it does not. These lines of work provide a nice complement to our approach, and the connections with our results merit further examination.To conclude, it is worth remembering that these types of questions have a long history, albeit in smaller and less data-intensive situations, and that revisiting old ideas can be fruitful. Indeed, recent empirical evidence suggests the obvious conjecture that "every" DNN has, as a function of its control parameters, some kind of generalization phase diagram, as in FIG1 ; and that fiddling with algorithm knobs has the effect of moving around some kind of parameter space, as in FIG1 . In these diagrams, there will be a phase where generalization changes gradually, roughly as PAC/VC-based intuition would suggest, and there will also be a "low temperature" spin glass like phase, where learning and generalization break down, potentially dramatically. At this point, it is hard to evaluate this conjecture, not only since existing methods tend to conflate (algorithmic) optimization and (statistical) regularization issues (suggesting we should better delineate the two in our theory), but also since empirical results are very sensitive to the many knobs and are typically non-reproducible. BID10 By the way, in addition to providing an "explanation" of the main observations of Zhang et al. FORMULA20 , the VSDL model and the SM approach provides an "explanation" for many other phenomena that are observed empirically: e.g., strong discontinuities in the generalization performance as a function of control parameters; that the generalization performance can depend sensitively on details of the model, details of the algorithms that perform approximate computation, the implicit regularization properties associated with these approximate computations, the detailed properties of the data and their noise, that the generalization can decay in the asymptotic regime as a power law with an exponent other than 1 or 1/2, or with some other functional form, etc. [
Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior The authors suggest that statistical mechanics ideas will help to understand generalization properties of deep neural networks, and give an approach that provides strong qualitative descriptions of empirical results regarding deep neural networks and learning algorithms. A set of ideas related to theoretical understanding generalization properties of multilayer neural networks, and a qualitative analogy between behaviours in deep learning and results from quantitative statistical physics analysis of single and two-layer neural networks.