building and training neural networks, including their influence on models, potential search spaces, and empirical values or schedules based on previous experience. Section 3 focuses on widely used algorithms in hyper-parameter searching, and these approaches are categorized into searching algorithms and trial schedulers. This section also evaluates the efficiency and applicability of these algorithms for different machine learning models. Section 4 provides an overview of mainstream HPO toolkits and services, compares their pros and cons, and presents some practical and implementation details. Section 5 more comprehensively compares existing HPO methods and highlights the efficient methods of model evaluation, and finally, Section 6 provides the study’s conclusions.\
The contribution of this study is summarized as follows:

-   Hyper-parameters are systematically categorized into structure-related and training-related. The discussion of their importance and empirical strategies are helpful to determine which hyper-parameters are involved in HPO.

-   HPO algorithms are analyzed and compared in detail, according to their accuracy, efficiency and scope of application. The analysis on previous studies is not only committed to include state-of-the-art algorithms, but also to clarify limitations on certain scenarios.

-   By comparing HPO toolkits, this study gives insights of the design of close-sourced libraries and open-sourced services, and 