\;=\;\frac{lr_0}{1+deacay*selfiteration}$$ Compared with LR itself, the deep learning model is less sensitive to *lr$_0$* and *k*.\

![Linear decay of learning rate with time-based (left) and drop-based (right) schedules[]{data-label="Figure 1"}](1.png)

Exponential decay is another widely used schedule [@li15].Compared with linear decay, an exponential schedule provides a more drastic decay at the beginning and a gentle decay when approaching convergence (Figure \[Figure 2\]). The basic mathematical form of exponential decay is as follows: $$lr \;=\;{lr_0}\cdot exp(-kt)$$ where *lr$_0$* and *k* are hyper-parameters for exponential learning decay. Similar to linear decay, *t* could be the number of self-iterations or epochs. If the number of epochs is used, the LR can be expressed with a similar form: $$lr \;=\;{lr_0}\cdot floor^{\frac{epoch}{epochsdrop}}$$ In addition to the initial LR and epoch drop rate, *floor* is another hyper-parameter. Generally, the LR is set to drop every 10 epochs (*epochsdrop*=10) by half (*floor*=0.5),but these can be decided by the model. EfficientNet [@tan19b] uses an exponential schedule when trained with ImageNet [@krizhevsky12]. Its choice for *lr$_0$*, *floor* and *epochsdrop* are too tricky to be designed purely by hand. For example, the LR is updated every 2.4 epochs (*epochsdrop*=2.4) by 0.97 (*floor*=0.97). Automatic hyper-parameter tuning is supposed to be applied in 