decay is not always the best choice; it depends on the model and dataset.

Optimizer
---------

Optimizers, or optimization algorithms, play a critical role in improving accuracy and training speed. Hyper-parameters related to optimizers include the choice of optimizer, mini-batch size, momentum, and beta. Selecting an appropriate optimizer is a tricky task. This section discusses the most widely adopted optimizers (mini-batch gradient descent, RMSprop, and Adam), related hyper-parameters, and suggested values.\
The aim of mini-batch gradient descent [@li14]is to solve the following two problems. Compared with vanilla gradient descent [@li15],mini-batch gradient descent (mini-batch GD) accelerates the training process, especially on a large dataset. Compared with SGD with a mini-batch size of 1, mini-batch GD reduces the noise and increases the probability of convergence [@ng2017improving].Mini-batch size is a hyper-parameter, and the value is highly related to the memory of the computation unit. Its value is suggested to be a power of 2 because of the access of CPU/GPU memory. The model runs faster if a power of 2 is used as the mini-batch size, and 32 could be a good default value [@bengio12; @masters18]. The maximum mini-batch size fitting in CPU/GPU memory. With a constant LR, researchers may find that their 