from converging (Figure \[Figure 4\]). The LR schedule must be adjusted according to the optimizer algorithm [@lau2017learning]. If the LR schedule must be determined in advance, it is recommended to simultaneously tune the hyper-parameters of the LR schedule and corresponding optimizers.\

![Effect of learning rate (adapted figure from: https://www.jeremyjordan.me/nn-learning-rate/)[]{data-label="Figure 4"}](4.png)

The choice of LR varies with specific tasks, but generally some tips exist that are based on our experience. They can be viewed as a general rule for hyper-parameter tuning.\

-   In practice, it is difficult to decide the importance of a hyper-parameter if one has no experience of it. A sensitivity test is suggested [@hamby94; @breierova1996introduction] to ensure the influence of a certain hyper-parameter.

-   Initial values of hyper-parameters are influential and must be carefully determined. The initial LR could be a comparatively large value because it will decay during training. In the early stage of training, a large LR will lead to fast convergence with fewer risks (Figure \[Figure 4\])

-   Use log scale to update the LR. Thus, exponential decay could be a better choice. An exponential schedule could be applicable for many other tuning hyper-parameters, such as momentum and weight decay.

-   Try more schedules. Exponential 