this case.\

![Exponential decay of learning rate[]{data-label="Figure 2"}](2.png)

A typical LR schedule may encounter some challenges when applied to a certain model. One problem is that users must determine all hyper-parameters in the schedule in advance of training, which is a task that depends on experience. Another problem is that in the abovementioned schedules, the LR only varies with time or steps, and the same value is applied for all layers in a model. The first problem is typically solved with automated HPO or the cyclical LR method [@smith17]. The LR is updated in a triangle rule within a certain bound value, and the bound value is decayed in a certain cyclic schedule (Figure \[Figure 3\]).\

![Learning rate decay in a cyclic schedule (source: https://github.com/bckenstler/CLR)[]{data-label="Figure 3"}](3.png)

Regarding the second problem, this study [@you17] suggested an algorithm based on layer-wise adaptive rate scaling (LARS). In their study, every layer had a local LR related to its weight and gradient. Users decide how to adjust the LR according to the layers and related hyper-parameters.\
In reality, selecting the optimal LR or its optimum schedule is a challenge. A small LR leads to slow convergence, whereas a large LR may prevent the model 