Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing for capability on large-scale sparse datasets. On top of that, they are computationally efficient and insensitive to the hyper-parameter settings. In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT. Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients. The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly. We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings. Employing first order optimization methods, such as stochastic gradient descent (SGD), is a key of solving large-scale problems. The classic gradient descent algorithm is widely used to update the model parameters, denoted by x, x t+1 = x t − η∇f (x t ), where the gradient is denoted by ∇f (x t ) and the step size by η. While the method has shown its efficiency for many contemporary tasks, the adaptive variants of SGD outperform the vanilla SGD methods on their rapid training time. Specifically, the step size η is substituted by an adaptive step size η/ √ v t , and v t is generated from the squared gradient [∇f (x t )] 2 . Several variants of the popular adaptive optimizers can be summarized into such common format. These optimizers share gradients calculation and parameters updating functions, but specify different moving average schemes for calculating the parameter-wise adaptive learning rate v t . For example, AdaGrad (Duchi et al., 2011) takes the arithmetic average of historical squared gradients [∇f (x t )] 2 . Compared with the conventional momentum method, it adapts the learning rate to each parameter to suit the sparse data structure, and thus gains a rapid convergence speed (Ruder, 2016) . Later, Tieleman & Hinton (2012) proposed RMSProp to reduce the aggressiveness of the decay rate in AdaGrad. The method modifies v t to the exponentially decayed squared gradients. Similar implementations could also be found in ADADELTA (Zeiler, 2012) . Instead of the squared gradients, the method applies squared parameter updates to define the adaptive learning rate. As a result, each update guarantees the same hypothetical units as the parameter. Later, Adam (Kingma & Ba, 2015) modifies RMSProp with the idea from momentum methods (Qian, 1999) . Except for the second moment moving average, the new rule also replaces the gradient ∇f (x t ) at the end of the Equation (1) to the first-moment estimation. The method has practically shown its superiority regarding the converge speed and memory requirement. While the aforementioned methods are the most famous frameworks, there are also many variants for each of them. The examples include NAdam (Dozat, 2016) , AMSGrad (Reddi et al., 2018) and Adafom (Chen et al., 2019) . So far, the adaptive methods with exponential moving average gradients have gained great attention with huge success in many deep learning tasks. However, it remains unsolved whether the simple exponential smoothing results or the level information is sufficient in capturing the landscape of the cost surface. When clear upward or downward pattern could be recognized within the moving routine, it is suggested to add a trend term on top of the single level information. In this paper, we modify the Adam rule with trend-corrected exponential smoothing schemes, namely AdamT, to obtain the local minima with a faster speed. To the best of our knowledge, our research is the first to apply the trend-corrected features on gradients scaling and parameters updating. It shall be emphasized that our framework is universally implementable for all adaptive update methods that apply the exponential average term, including but not restricted to ADADELTA, RMSProp, AdaMAX and other well-recognized methods. For the sake of conciseness, in this specific paper, we focus on Adam regarding rule modification and performance comparison. Our contributions in this paper could be summarized in three-fold: 1. We propose the notion of trend corrected exponential smoothing to modify the conventional application of exponential moving average in optimizers with adaptive gradients. Our AdamT method collaborates the trend information into the update rule of Adam. 2. We show the conditions for the method to converge in convex settings. The regret bound is in consistent to Adam at O( √ T ). 3. We demonstrate AdamT's convergence in both convex and non-convex settings. The performance is compared with Adam, where AdamT shows clear superiority on both the training set and the test set, especially for non-convex problems. For the remainder of the paper, we present the fundamental idea of Adam and Holt's linear methods in Section 2. In Section 3 and 4, we detail the update rules and experimental analysis, respectively. In addition, Section 5 reviews recent developments of Adam-typed optimizers. While many of them focus more on non-convex optimizations, there is a potential to incorporate our methods with such frameworks and this extension is expected for future settings. In this work, we have modified the scheme to calculate the adaptive step size from exponential moving average to trend-corrected exponential smoothing. Empirical results demonstrate that our method, AdamT, works well in practice and constantly beats the baseline method Adam. We leave some potentials for future developments. First, although we focused primarily on ADAM for theoretical and experimental analysis, we believe that similar ideas could also extend to other adaptive gradient methods, such as RMSProp (Tieleman & Hinton, 2012) and AMSGrad (Reddi et al., 2018) . Also, this work, the same as the original ADAM method, relies on the theoretical assumption of convex problems settings. We have demonstrated its computational ability on the non-convex settings, and it is possible to extend the theoretical framework to non-convex scenarios. Some potential candidates in the latest research are listed in Section 5. To find how the expectation of the trend estimates b m t relates to the expectation of the difference between the level estimates at successive timesteps ( m t − m t−1 ), we take the expectation for both sides of the above equation: where ζ can be considered as a small constant, since the factor (γ 1 φ 1 ) t−i will be tiny if the associated expectation E[( )] is stationary, the constant ζ will be zero. To further simplify the above equation, we apply the formula for the sum of geometric sequence: This suggests that we can use the term ] to correct the bias and close the discrepancy between the above two expectations at the presence of the damping factor φ 1 .
We present a new framework for adapting Adam-typed methods, namely AdamT, to include the trend information when updating the parameters with the adaptive step size and gradients. A new type of Adam variant that uses Holt's linear method to compute the smoothed first order and second order momentum instead of using exponential weighted average.