0.9; *$\beta_2$* is the parameter of RMSprop, with a suggested value of 0.999; $\epsilon$ here is used to avoid dividing by 0; and $10^{-8}$ is the default value. The default configurations are suitable for most problems, and therefore Adam is relatively easy to apply [@brownlee17]. The suggested values are adapted by most deep learning frameworks, including TensorFlow [@abadi16], Keras [@chollet18], Caffe [@jia14], MxNet [@chen15], and PyTorch [@paszke19].\
Generally, optimization algorithms must be adjusted along with the LR schedule. Most hyper-parameters for neural network training are highly related to optimizers and LR. RMSprop and Adam can be applied in similar situations. In most cases, Adam can act as the default optimizer [@ruder16] because it is a more reliable improvement based on RMSprop. If Adam is used as the optimizer, LR must be adjusted accordingly, whereas default values are sufficient for most other hyper-parameters. Compared with RMSprop and Adam, SGD with momentum may require more time to find the optimum.\
The abovementioned subsections discussed the most critical hyper-parameters related to model training. According to lecture notes of Ng (2017), the importance of hyper-parameters are listed as follows:\

1.  LR, as in our discussion above.

2.  Momentum beta, for RMSprop etc.

3.  Mini-batch 