Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results. Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better? We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam. We then analyze its robustness to learning rate misspecification and objective curvature variation. Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD. YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly. We empirically show YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to $3.28$x in synchronous and up to $2.69$x in asynchronous settings. Accelerated forms of stochastic gradient descent (SGD), pioneered by BID0 and BID1 , are the de-facto training algorithms for deep learning. Their use requires a sane choice for their hyperparameters: typically a learning rate and momentum parameter BID2 . However, tuning hyperparameters is arguably the most time-consuming part of deep learning, with many papers outlining best tuning practices written BID4 BID6 . Deep learning researchers have proposed a number of methods to deal with hyperparameter optimization, ranging from grid-search and smart black-box methods BID7 BID8 to adaptive optimizers. Adaptive optimizers aim to eliminate hyperparameter search by tuning on the fly for a single training run: algorithms like AdaGrad BID9 , RMSProp BID10 and Adam BID11 use the magnitude of gradient elements to tune learning rates individually for each variable and have been largely successful in relieving practitioners of tuning the learning rate. Recently some researchers have started favoring simple momentum SGD over the previously mentioned adaptive methods BID12 BID13 , often reporting better test scores BID14 . Motivated by this trend, we ask the question: can simpler adaptive methods, based on momentum SGD perform as well or better? We empirically show that, with hand-tuned learning rate, Polyak's momentum SGD achieves faster convergence than Adam for a large class of models. We then formulate the optimization update as a dynamical system and study certain robustness properties of the momentum operator. Building on our analysis, we design YELLOWFIN, an automatic hyperparameter tuner for momentum SGD. YELLOWFIN simultaneously tunes the learning rate and momentum on the fly, and can handle the complex dynamics of asynchronous execution. Specifically:• In Section 2, we show that momentum presents convergence robust to learning rate misspecification and curvature variation in a class of non-convex objectives; this robustness is desirable for deep learning. They stem from a known but obscure fact: the momentum operator's spectral radius is constant in a large subset of the hyperparameter space.• In Section 3, we use these robustness insights and a simple quadratic model analysis to design YELLOWFIN, an automatic tuner for momentum SGD. YELLOWFIN uses on-the-fly measurements from the gradients to tune both a single learning rate and momentum.• In Section 3.3, we discuss common stability concerns related to the phenomenon of exploding gradients . We present a natural extension to our basic tuner, using adaptive gradient clipping, to stabilize training for objectives with exploding gradients.• In Section 4 we present closed-loop YELLOWFIN, suited for asynchronous training. It uses a novel component for measuring the total momentum in a running system, including any asynchrony-induced momentum, a phenomenon described in BID16 . This measurement is used in a negative feedback loop to control the value of algorithmic momentum.We provide a thorough evaluation of the performance and stability of our tuner. In Section 5, we demonstrate empirically that on ResNets and LSTMs YELLOWFIN can converge in fewer iterations compared to: (i) hand-tuned momentum SGD (up to 1.75x speedup); and (ii) default Adam (0.8x to 3.3x speedup). Under asynchrony, the closed-loop control architecture speeds up YELLOWFIN, making it up to 2.69x faster than Adam. Our experiments include runs on 7 different models, randomized over at least 5 different random seeds. YELLOWFIN is stable and achieves consistent performance: the normalized sample standard deviation of test metrics varies from 0.05% to 0.6%. We released PyTorch and TensorFlow implementations, that can be used as drop-in replacements for any optimizer. YELLOWFIN has also been implemented in other various packages. Its large-scale deployment in industry has taught us important lessons about stability; we discuss those challenges and our solution in Section 3.3. We conclude with related work and discussion in Section 6 and 7. We presented YELLOWFIN, the first optimization method that automatically tunes momentum as well as the learning rate of momentum SGD. YELLOWFIN outperforms the state-of-the-art adaptive optimizers on a large class of models both in synchronous and asynchronous settings. It estimates statistics purely from the gradients of a running system, and then tunes the hyperparameters of momentum SGD based on noisy, local quadratic approximations. As future work, we believe that more accurate curvature estimation methods, like the bbprop method (Martens et al., 2012) can further improve YELLOWFIN. We also believe that our closed-loop momentum control mechanism in Section 4 could accelerate convergence for other adaptive methods in asynchronous-parallel settings. A PROOF OF LEMMA 2To prove Lemma 2, we first prove a more generalized version in Lemma 6. By restricting f to be a one dimensional quadratics function, the generalized curvature h t itself is the only eigenvalue. We can prove Lemma 2 as a straight-forward corollary. Lemma 6 also implies, in the multiple dimensional correspondence of (4), the spectral radius ⇢(A t ) = p µ if the curvature on all eigenvector directions (eigenvalue) satisfies (5). Lemma 6. Let the gradients of a function f be described by DISPLAYFORM0 with H (x t ) 2 R n 7 ! R n⇥n . Then the momentum update can be expressed as a linear operator: DISPLAYFORM1 where DISPLAYFORM2 . Now, assume that the following condition holds for all eigenvalues (H ( DISPLAYFORM3 then the spectral radius of A t is controlled by momentum with ⇢(A t ) = p µ.
YellowFin is an SGD based optimizer with both momentum and learning rate adaptivity. Proposes a method to automatically tuning the momentum parameter in momentum SGD methods, which achieves better results and fast convergence speed that state-of-the-art Adam algorithm.