clarifies the targeted users for each of them.

-   The potential research direction regarding to existing problems are suggested on algorithms, applications and techniques.

Major Hyper-Parameters and Search Space
=======================================

Considering the computational resources required, hyper-parameters with greater importance receive preferential treatment in the process of HPO. Hyper-parameters with a stronger effect on weights during training are more influential for neural network training [@ng2017improving]. It is difficult to quantitatively determine which of the hyper-parameters are the most significant for final accuracy. In general, there are more studies on those with higher importance, and their importance has been decided by previous experience.\
Hyper-parameters can be categorized into two groups: those used for training and those used for model design. A proper choice of hyper-parameters related to model training allows neural networks to learn faster and achieve enhanced performance. Currently, the most adopted optimizer for training a deep neural network is stochastic gradient descent [@robbins51] with momentum [@qian99] as well as its variants such as AdaGrad [@duchi11], RMSprop [@hinton12b], and Adam [@kingma14]. In addition to the choice of optimizer, corresponding hyper-parameters (e.g., momentum) are critical for certain networks. During the training process, batch size and LR draw the most attention because they 