We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. 
 We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) 
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).
 We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. 
 Our source code will become available after the review process. Adaptive gradient methods, such as AdaGrad BID3 , RMSProp BID19 , and Adam BID12 have become a default method of choice for training feedforward and recurrent neural networks BID21 BID5 BID16 . Nevertheless, state-of-the-art results for popular image classification datasets, such as CIFAR-10 and CIFAR-100 BID13 , are still obtained by applying SGD with momentum BID7 BID4 BID15 BID4 . Furthermore, Wilson et al. (2017) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks such as image classification, character-level language modeling and constituency parsing. Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima BID11 BID2 and inherent problems of adaptive gradient methods BID20 . In this paper, we show that a major factor in the poor generalization of the most popular adaptive gradient method, Adam, lies in its dysfunctional implementation of weight decay; the issue we identify in Adam also pertains to other adaptive gradient methods.Specifically, our analysis of Adam given in this paper leads to the following observations:The standard way to implement L 2 regularization/weight decay in Adam is dysfunctional.One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that L 2 regularization/weight decay are implemented suboptimally in common deep learning libraries. Therefore, on tasks/datasets where the use of L 2 regularization is beneficial (e.g., on many popular image classification datasets), Adam leads to worse results than SGD with momentum (for which L 2 regularization behaves as expected). L 2 regularization and weight decay are not the same thing. Contrary to common belief, the two techniques are not equivalent. For SGD, they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate; this is not the case for Adam. In particular, when combined with adaptive gradients, L 2 regularization leads to weights with large gradients being regularized less than they would be when using weight decay.Optimal weight decay is a function (among other things) of the total number of batch passes/weight updates. Our empirical analysis of Adam suggests that the longer the runtime/number of batch passes to be performed, the smaller the optimal weight decay. This effect tends to be neglected because hyperparameters are often tuned for a fixed or a comparable number of training epochs. As a result, the values of the weight decay found to perform best for short runs do not generalize to much longer runs.Our contributions are aimed at fixing the issues described above:Decoupling weight decay from the gradient-based update (Section 2). We suggest to decouple the gradient-based update from weight decay for both SGD and Adam. The resulting SGD version SGDW decouples optimal settings of the learning rate and the weight decay factor, and the resulting Adam version AdamW generalizes substantially better than Adam. Normalizing the values of weight decay (Section 3). We propose to parameterize the weight decay factor as a function of the total number of batch passes. This leads to a greater invariance of the hyperparameter settings in the sense that the values found to perform best for short runs also perform well for many times longer runs. Adam with warm restarts and normalized weight decay (Section 4). After we fix the weight decay in Adam and design AdamW, we introduce AdamWR to obtain strong anytime performance by performing warm restarts.The main motivation of this paper is to fix the weight decay in Adam to make it competitive w.r.t. SGD with momentum even for those problems where it did not use to be competitive. We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should help to reduce the common issue of selecting dataset/task-specific training algorithms and their hyperparameters. Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum BID20 , we identified at least one possible explanation to this phenomenon: the dysfunctional use of L 2 regularization and weight decay. We proposed a simple fix to deal with this issue, yielding substantially better generalization performance in our AdamW variant. We also proposed normalized weight decay and warm restarts for Adam, showing that a more robust hyperparameteer selection and a better anytime performance can be achieved in our new AdamWR variant.Our preliminary results obtained with AdamW and AdamWR on image classification datasets must be verified on a wider range of tasks, especially the ones where the use of regularization is expected to be important. It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam BID22 . While we focussed our experimental analysis on Adam, we believe that similar results also hold for other adaptive gradient methods, such as AdaGrad BID3 and RMSProp BID19 .The results shown in FIG2 suggest that Adam and AdamW follow very similar curves most of the time until the third phase of the run where AdamW starts to branch out to outperform Adam. As pointed out by an anonymous reviewer, it would be interesting to investigate what causes this branching and whether the desired effects are observed at the bottom of the landscape. One could investigate this using the approach of BID9 to switch from Adam to AdamW at a given epoch index. Since it is quite possible that the effect of regularization is not that pronounced in the early stages of training, one could think of designing a version of Adam which exploits this by being fast in the early stages and well-regularized in the late stages of training. The latter might be achieved with a custom schedule of the weight decay factor.In this paper, we argue that the popular interpretation that weight decay = L 2 regularization is not precise. Instead, the difference between the two leads to the following important consequences. Two algorithms as different as SGD and Adam will exhibit different effective rates of weight decay even if the same regularization coefficient is used to include L 2 regularization in the objective function. Moreover, when decoupled weight decay is applied, two algorithms as different as SGDW and AdamW will optimize two effectively different objective functions even if the same weight decay factor is used. Our findings suggest that the original Adam algorithm with L 2 regularization affects effective rates of weight decay in a way that precludes effective regularization, and that effective regularization is achievable by decoupling the weight decay. BID0 analytically showed that in the limited data regime of deep networks the presence of eigenvalues that are zero forms a frozen subspace in which no learning occurs and thus smaller (e.g., zero) initial weight norms should be used to achieve best generalization results. Our future work shall consider adapting initial weight norms or weight norm constraints BID17 at each warm restart. BID10 proposed a family of regularization techniques which are specific to the current batch and its size. Similarly to L 2 regularization and weight decay, the latter techniques might be attempted to be transformed to act directly on weights.1 SUPPLEMENTARY MATERIAL
Fixing weight decay regularization in adaptive gradient methods such as Adam Proposes idea to decouple the weight decay from the number of steps taken by the optimization process. The paper presents an alternative way to implement weight decay in Adam with empirical results shown Investigates weight decay issues lied in the SGD variants and proposes the decoupling method between weight decay and the gradient-based update.