Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training. Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages. We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter. We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods. We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point. Training deep neural networks (DNNs) is well-known to be time and energy consuming, motivating the development of new methods and hardware to make training more efficient. One way to improve training efficiency is to use numerical representations that are more hardware-friendly. This is the reason that the IEEE 754 32-bit single-precision floating point format (FP32) is more widely used for training DNNs than the more precise double precision format (FP64), which is commonly used in other areas of high-performance computing. In an effort to further improve hardware efficiency, there has been increasing interest in using data types with even lower precision than FP32 for training Wang et al., 2018; Kalamkar et al., 2019; Sakr et al., 2019) . Of these, the IEEE half-precision floating-point (FP16) format is already well supported by modern GPU vendors (Choquette et al., 2018) . Using FP16 for training can reduce the memory footprint by half compared to FP32 and significantly improve the runtime performance and power efficiency. Nevertheless, numerical issues like overflow, underflow, and rounding errors frequently occur when training in low precision only. Percentage of underflow (%) Underflow rate among activation gradients accross all layers iter=10000 iter=50000 iter=80000 iter=110000 (a) Underflow rate is calculated by counting the absolute gradients below 2 −24 , the smallest positive FP16 number. Loss scale expected by each layer iter=10000 iter=50000 iter=80000 iter=110000 (b) Expected loss scale of each layer is calculated by 1 over the (0.01N )-th smallest absolute gradient, where N is the size of each gradient and 0.01 is the largest underflow rate permitted. Figure 1: Statistics of activation gradients collected from training SSD by FP32. Data are collected from different training iterations (120k in total). Layer ID are assigned in the topological order of backpropagation computation. Layers with higher ID are closer to the input. start of the backward pass so that the computed (scaled) gradients can then be properly represented in FP16 without significant underflow. For an appropriate choice of α, loss scaling can achieve state of the art results that are competitive with regular FP32 training. Unfortunately, there is no single value of α that will work in arbitrary models, and so it often needs to be tuned per model. Its value must be chosen large enough to prevent underflow issues from affecting training accuracy. However, if α is chosen too large, it could amplify rounding errors caused by swamping (Higham, 1993) or even result in overflow. This observed sensitivity to the particular choice of loss scale is also reported by , who find that different values can lead to very different ResNet-50 MPT convergence behavior. Furthermore, the data distribution of gradients can vary both between layers and between iterations (Figure 1 ), which implies that a single scale is insufficient. For instance, gradients closer to the input require a higher loss scale that may cause overflow or severe rounding errors if the same value were used in layers closer to the output. Including the time spent tuning α, the total training time of MPT can even exceed regular FP32 training. We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use. We hope that this will help to utilize better existing hardware with support for fast FP16 operations. Our method improves the usability of MPT compared to existing methods by removing the need to tune a model-specific loss scale hyperparameter, while retaining (and in some cases surpassing) the accuracy of regular FP32 training. We achieve this by introducing layer-wise loss scale values which are automatically computed and dynamically updated during training to deal with underflow more effectively than existing methods. Experimental results on several examples show that MPT with adaptive loss scaling can achieve the best model accuracy and the shortest overall training time, especially when training deep models on large datasets. This paper presents adaptive loss scaling, a method that calculates layer-wise loss scale during runtime, to improve the performance and usability of MPT. Empirically we find it works better than plain MPT, existing loss scaling methods, and even FP32 in some cases, regarding model accuracy and the time taken to converge. Future work includes evaluating adaptive loss scaling on other tasks and models, especially those for Natural Language Processing; and trying to find a tighter upper bound of loss scale for each layer, e.g., based on the variance analysis in (Sakr et al., 2019) , such that each layer can be scaled more effectively; extending it to FP8 is also intriguing to try. A DETAILED ANALYSIS ON CIFAR RESULTS Table 1 shows that adaptive loss scaling is beneficial for training ResNet-110, while less advantageous for ResNet-20 and ResNet-56. We hypothesize the reason behind is that underflow causes more numerical problems when the model is deeper. For shallower models, the difference between the oracle gradient values and the underflowing ones is moderate and can even be viewed as a form of regularization. This argument is supported by the fact that the training accuracy of ResNet models on CIFAR can always reach 100%. In this way, even though adaptive loss scaling can improve the accuracy of the computed gradients, this does not necessarily always translate to improved test accuracy. 19% 93.19% 93.19% We dive deeper into this argument by reviewing Table 4 , which shows the test accuracy of the two shallower ResNet models on CIFAR-10. For both models, the test accuracy first increases to a maxima at 16, then there is a sudden drop at 128, and finally it climbs up to a plateau. Our hypothetical interpretation is as follows: 1. Initially the test accuracy is low. Here the underflow rate is expected to be at its highest, and it is the major cause for the low test accuracy. 2. The test accuracy then increases with loss scale, mainly due to the mitigation of underflow by loss scaling. However, as the gradients become more accurate, the regularizing effect from underflow is also reduced and the test accuracy will drop, until the loss scale reaches around 128. 3. If the loss scale continues to increase, the high rounding error and swamping problem caused by large scales will arise. It adds another kind of regularization, which is relatively more harmful than what underflow may cause, and the test accuracy cannot improve much. Even though this interpretation is hypothetical, this empirical evaluation in Table 4 shows that the relationship between the goodness of a loss scaling scheme and test accuracy is complicated when the model tends to overfit.
We devise adaptive loss scaling to improve mixed precision training that surpass the state-of-the-art results. Proposal for an adaptive loss scaling method during backpropagation for mix precision training where scale rate is decided automatically to reduce the underflow. The authors propose a method to train models in FP16 precision that adopts a more elaborate way to minimize underflow in every layer simultaneously and automatically.