tricky task to determine a good value for LR. With a proper constant LR, the network is able to be trained to a passable but unsatisfactory accuracy because the initial value could always be overlarge, especially in the final few steps. A small improvement based on the constant value is to set an initial rate as 0.1, for example, adjusting to 0.01 when the accuracy is saturated, and to 0.001 if necessary [@tanksale18].\
Linear LR decay is a common choice for researchers who wish to set a schedule. It changes gradually during the training process based on time or step. The basic mathematical function of linear decay is $$lr \;=\;\frac{lr_0}{1+kt}$$ where *lr$_0$* and *k* are hyper-parameters for linear learning decay, indicating the initial LR and decay rate. A typical choice of *t* could be training time or number for self-iteration. If *t* is training time, the schedule of LR is a continuous change, as shown in Figure \[Figure 1\]. If t stands for the number of iterations, the LR drops every few iterations. A typical choice is to drop the LR by half every 10 epochs [@lau2017learning] by 0.1 every 20 epochs [@li15]. This has the following mathematical form: $$lr 