While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multi-fidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations. In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations. cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost. Furthermore, cfKG can be generalized, following Wu et al. (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously. Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning. In hyperparameter tuning of machine learning models, we seek to find a set of hyperparameters x in some set A to minimize the validation error f (x), i.e., to solve min x∈A f (x) (1.1)Evaluating f (x) can take substantial time and computational power BID0 , and may not provide gradient evaluations. Thus, machine learning practitioners have turned to Bayesian optimization for solving (1.1) BID19 because it tends to find good solutions with few function evaluations BID6 .As the computational expense of training and testing a modern deep neural network for a single set of hyperparameters has grown as long as days or weeks, it has become natural to seek ways to solve (1.1) more quickly by supplanting some evaluations of f (x) with computationally inexpensive lowfidelity approximations. Indeed , when training a neural network or most other machine learning models, we can approximate f (x) by training on less than the full training data, or using fewer training iterations. Both of these controls on fidelity can be set to achieve either better accuracy or lower computational cost across a range of values reasonably modeled as continuous.In this paper, we consider optimization with evaluations of multiple fidelities and costs where the fidelity is controlled by one or more continuous parameters. We model these evaluations by a realvalued function g(x, s) where f (x) := g(x, 1 m ) and s ∈ [0, 1] m denotes the m fidelity-control parameters. g(x, s) can be evaluated, optionally with noise, at a cost that depends on x and s. In the context of hyperparameter tuning, we may take m = 2 and let g(x, s 1 , s 2 ) denote the loss on the validation set when training using hyperparameters x with a fraction s 1 of the training data and a fraction s 2 of some maximum allowed number of training iterations. We may also set m = 1 and let s index either training data or training iterations. We assume A is a compact connected uncountable set into which it is easy to project, such as a hyperrectangle.This problem setting also appears outside of hyperparameter tuning, in any application where the objective is expensive to evaluate and we may observe cheap low-fidelity approximations parameterized by a continuous vector. For example , when optimizing a system evaluated via a Monte Carlo simulator, we can evaluate a system configuration approximately by running with fewer replications. Also, when optimizing an engineering system modeled by a partial differential equation (PDE), we can evaluate a system configuration approximately by solving the PDE using a coarse grid.Given this problem setting, we use the knowledge gradient approach BID3 to design an algorithm to adaptively select the hyperparameter configuration and fidelity to evaluate, to best support solving (1.1). By generalizing a computational technique based on the envelope theorem first developed in Wu et al. (2017) , our algorithm supports parallel function evaluations, and also can take advantage of derivative observations when they are available. This algorithm chooses the point or set of points to evaluate next that maximizes the ratio of the value of information from evaluation against its cost.Unlike most existing work on discrete-and continuous-fidelity Bayesian optimization, our approach considers the impact of our measurement on the future posterior distribution over the full feasible domain, while existing expected-improvement-based approaches consider its impact at only the point evaluated. One exception is the entropy-search-based method [10] , which also considers the impact over the full posterior. Our approach differs from entropy search in that it chooses points to sample to directly minimize expected simple regret, while entropy search seeks to minimize the entropy of the location or value of the global optimizer, indirectly reducing simple regret.We summarize our contributions as follows. We propose a novel continuous-fidelity BO algorithm, cfKG, which generalizes naturally to batch and derivative settings. This algorithm can find good solutions to global optimization problems with less cost than state-of-art algorithms in applications including deep learning and kernel learning.
We propose a Bayes-optimal Bayesian optimization algorithm for hyperparameter tuning by exploiting cheap approximations. Studies hyperparameter-optimization by Bayesian optimization, using the Knowledge Gradient framework and allowing the Bayesian optimizer to tune fidelity against cost.