Understanding the optimization trajectory is critical to understand training of deep neural networks. We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along this trajectory. Based on a theoretical model, we predict that using a high learning rate or a small batch size in the early phase of training leads SGD to regions of the parameter space with (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We show that the point on the trajectory after which these effects hold, which we refer to as the break-even point, is reached early during training. We demonstrate these effects empirically for a range of deep neural networks applied to multiple different tasks. Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a high learning rate to achieve loss smoothing effects attributed previously to BN alone. The choice of the optimization method implicitly regularizes deep neural networks (DNNs) by influencing the optimization trajectory in the loss surface (Neyshabur, 2017; Arora, 2019) . In this work, we theoretically and empirically investigate how the learning rate and the batch size used at the beginning of training determine properties of the entire optimization trajectory. Figure 1: Visualization of the early part of the training trajectory on CIFAR-10 (before reaching 65% training accuracy) of a simple CNN model (see Sec. 4 for details) optimized using SGD with learning rate η = 0.1 (red) and η = 0.01 (blue). Each model, shown as a point on the trajectory, is represented by its test predictions embedded into a two-dimensional space using UMAP. The background color indicates the spectral norm of K (left) and the training accuracy (right). Depending on η, after reaching what we call the break-even point, trajectories are steered towards regions characterized by different K (left) for the same training accuracy (right). See Sec. 4.1 for details. We focus our analysis on two objects that quantify different properties of the optimization trajectory: the covariance of gradients (K) 1 , and the Hessian of the training loss (H). The matrix K quantifies noise induced by noisy estimate of the full-batch gradient, and has been linked to the generalization error (Roux et al., 2008; . The matrix H describes the curvature of the loss surface. Better conditioning of H has been attributed as the main reason behind the efficacy of batch normalization (Bjorck et al., 2018; Ghorbani et al., 2019) . Our first and main contribution is predicting, and empirically demonstrating two effects in the early phase of training influenced by the choice of the hyperparameters in stochastic gradient descent (SGD): (1) reduced spectral norms of K and H and (2) improved conditioning of K and H. These effects manifest themselves after a certain point on the optimization trajectory, to which we refer to as the break-even point. See Fig. 1 for an illustration of this phenomenon. We make our predictions based on a theoretical model of the initial phase of training, which incorporates recent observations on the instability and oscillations in the parameter space that characterize the learning dynamics of neural networks (Masters & Luschi, 2018; Xing et al., 2018; Lan et al., 2019) . As our second contribution, we apply our analysis to a network with batch normalization (BN) layers and find that our predictions are valid in this case too. Delving deeper in this direction of investigation, we show that using a large learning rate is necessary to reach better-conditioned, relatively to a network without BN layers, regions of the loss surface, which was previously attributed to BN alone (Bjorck et al., 2018; Ghorbani et al., 2019; Page, 2019) . Based on a theoretical model, we conjectured and empirically argued for the existence of the breakeven point on the optimization trajectory induced by SGD. Next, we demonstrated that using a high learning rate or a small batch size in SGD has two effects on K and H along the trajectory that we referred to as (1) variance reduction and (2) pre-conditioning. There are many potential implications of the existence of the break-even point. We investigated one in particular, and demonstrated that using a high learning rate is necessary to achieve the loss smoothing effects previously attributed to batch normalization alone. Additionally, the break-even occurs typically early during training, which might be related to the recently discovered phenomenon of the critical learning period in training of deep networks (Achille et al., 2017; Golatkar et al., 2019) . We plan to investigate this connection in the future.
In the early phase of training of deep neural networks there exists a "break-even point" which determines properties of the entire optimization trajectory. This work analyzes the optimization of deep neural networks by considering how the batch size and step-size hyper-parameters modify learning trajectories.