This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.) evaluated for a model arising at (any point of) a training run. This pattern is universal in the sense that this one to one relationship is identical across architectures (such as VGG, Resnet, Densenet etc.), algorithms (SGD and SGD with momentum) and training loss functions (cross entropy and mean squared error). Neural networks are state of the art models for various tasks in machine learning, especially those in computer vision and natural language processing. While there has been significant progress in designing and applying neural networks for various tasks, our understanding of most aspects of their behavior has not yet caught up with these advances. One significant challenge in furthering our understanding is the huge variation in deep learning models. In the context of image classification (which will be the context of the current paper), there are several well known models that have been developed by the machine learning community: VGG BID5 , Resnet BID2 , Densenet BID3 to name a few; all of them having their own unique structure. Is it possible to understand the behavior of all of these models through a common lens?The main contribution of the current paper is to propose and demonstrate that, despite the diversity in the structure of these different models, there are some striking resemblances in the behavior of all of these models. More concretely, the training curves across any two loss functions (such as cross entropy vs 0/1 error or cross entropy vs mean squared error) essentially overlap across all of the above models. See FIG1 for a pictorial description and Section 3 for a rigorous description.This observation suggests that training of most (if not all) deep neural networks follows the same pattern. The existence of such universal patterns is quite exciting as it points to the possibility of understanding the behavior (in this instance training behavior) of different neural networks through a single approach. We also note that, while this similarity in behavior extends, to some extent, also to the test data, there are limitations (see Section 4.2).Paper organization: In Section 2, we will present the setup and required definitions. Section 3 formally describes the phenomenon identified in this paper. Section 4 presents the main experimental results. We conclude in Section 5. More experimental results are presented in the appendix.Notation: Vectors are written in bold small case letters (such as p and x). DISPLAYFORM0 The canonical basis for R K is represented as {e i , . . . , e K } where e ij = 1 if i = j else e ij = 0. We propose and demonstrate that there is a training dynamics which is universal for various neural networks. While similar behavior does not seem to hold for test data, preliminary results call for further investigation. One possible explanation is that distribution of predictions through the training process follows a universal law. However, measuring standard distances (e.g., Wasserstein) requires a large number of samples. One potential solution is to measure weaker distances such as neural net distances BID0 . We believe that our observations could lead to a new way of understanding neural networks in a unified manner. A. Appendix
We identify some universal patterns (i.e., holding across architectures) in the behavior of different surrogate losses (CE, MSE, 0-1 loss) while training neural networks and present supporting empirical evidence.