the learning rate (LR) of stochastic gradient descent (SGD), batch size, and optimizer [@hyper].The history of HPO dates back to the early 1990s [@ripley93; @king95],and the method is widely applied for neural networks with the increasing use of machine learning. HPO can be viewed as the final step of model design and the first step of training a neural network. Considering the influence of hyper-parameters on training accuracy and speed, they must carefully be configured with experience before the training process begins [@rodriguez18]. The process of HPO automatically optimizes the hyper-parameters of a machine learning model to remove humans from the loop of a machine learning system. As a trade of human efforts, HPO demands a large amount of computational resources, especially when several hyper-parameters are optimized together. The questions of how to utilize computational resources and design an efficient search space have resulted in various studies on HPO on algorithms and toolkits. Conceptually, HPO’s purposes are threefold [@feurer19]: to reduce the costly menial work of artificial intelligence (AI) experts and lower the threshold of research and development; to improve the accuracy and efficiency of neural network training [@melis17]; and to make the choice of hyper-parameter set more convincing and 