Title: FAME: Fast Adaptive Moment Estimation based on Triple Exponential Moving Average. Abstract: Network optimization is a key step in deep learning, which broadly impacts different domains (e.g. natural language, computer vision). Over the years, several optimizers have been developed - some are adaptive and converge quickly, while others are not adaptive but may be more accurate. However, due to the fact that most current optimizers' use simple Exponential Moving Average, gradient trends and their rapid changes may not be accurately identified, resulting in sub-optimal network performance. In this paper, we propose the first deep optimizer based on the Triple Exponential Moving Average (TEMA), a technical indicator originally developed to predict stock market trends. TEMA adds richer multi-level information about data changes and trends compared to the simple Exponential Moving Average. As a result, the gradients moments are better estimated. Furthermore, instead of using TEMA in the same way as the stock domain, here we use it as part of a continuous average during an optimization procedure. We extensively validated our method. Five benchmarks (CIFAR-10, CIFAR-100, PASCAL-VOC, MS-COCO, and Cityscapes) were used to test our method, as well as 14 different learning architectures, five different optimizers, and various vision tasks (detection, segmentation, and classification). The results clearly indicate that the robustness and accuracy of our FAME optimizer are superior to those of others.