determine the speed of convergence, the tuning of which should always be ensured. Hyper-parameters for model design are more related to the structure of neural networks, the most typical example being the number of hidden layers and width of layers. These hyper-parameters usually measure the model’s learning capacity, which is determined by the complexity of function [@agrawal18].\
This section provides an in-depth discussion on hyper-parameters of great importance for model structures and training, as well as introduce their effect in models and provide suggested values or schedules.

Learning Rate
-------------

LR is a positive scalar that determines the length of step during SGD [@goodfellow16]. In most cases, the LR must be manually adjusted during model training, and this adjustment is often necessary for enhanced accuracy [@bengio12].An alternative choice of fixed LR is a varied LR over the training process. This method is referred to as the LR schedule or LR decay [@goodfellow16]. Adaptive LR can be adjusted in response to the performance or structure of the model, and it is supported by a learning algorithm [@smith17; @brownlee19].\
Constant LR, as the simplest schedule, is often set as the default by deep learning frameworks (e.g., Keras). It is an important but 