A Study of Bayesian Neural Network Surrogates for Bayesian Optimization
Bayesian optimization is a highly efficient approach to optimizing objective
functions which are expensive to query. These objectives are typically
represented by Gaussian process (GP) surrogate models which are easy to
optimize and support exact inference. While standard GP surrogates have been
well-established in Bayesian optimization, Bayesian neural networks (BNNs) have
recently become practical function approximators, with many benefits over
standard GPs such as the ability to naturally handle non-stationarity and learn
representations for high-dimensional data. In this paper, we study BNNs as
alternatives to standard GP surrogates for optimization. We consider a variety
of approximate inference procedures for finite-width BNNs, including
high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics
such as deep ensembles. We also consider infinite-width BNNs, linearized
Laplace approximations, and partially stochastic models such as deep kernel
learning. We evaluate this collection of surrogate models on diverse problems
with varying dimensionality, number of objectives, non-stationarity, and
discrete and continuous inputs. We find: (i) the ranking of methods is highly
problem dependent, suggesting the need for tailored inductive biases; (ii) HMC
is the most successful approximate inference procedure for fully stochastic
BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is
relatively competitive; (iv) deep ensembles perform relatively poorly; (v)
infinite-width BNNs are particularly promising, especially in high dimensions.
Yucen Lily Li, Tim G. J. Rudner, Andrew Gordon Wilson
# A Study of Bayesian Neural Network Surrogates

###### Abstract

Bayesian optimization is a highly efficient approach to optimizing objective functions which are expensive to query. These objectives are typically represented by Gaussian process (GP) surrogate models which are easy to optimize and support exact inference. While standard GP surrogates have been well-established in Bayesian optimization, Bayesian neural networks (BNNs) have recently become practical function approximators, with many benefits over standard GPs such as the ability to naturally handle non-stationarity and learn representations for high-dimensional data. In this paper, we study BNNs as alternatives to standard GP surrogates for optimization. We consider a variety of approximate inference procedures for finite-width BNNs, including high-quality Hamiltonian Monte Carlo, low-cost stochastic MCMC, and heuristics such as deep ensembles. We also consider infinite-width BNNs and partially stochastic models such as deep kernel learning. We evaluate this collection of surrogate models on diverse problems with varying dimensionality, number of objectives, non-stationarity, and discrete and continuous inputs. We find: (i) the ranking of methods is highly problem dependent, suggesting the need for tailored inductive biases; (ii) HMC is the most successful approximate inference procedure for fully stochastic BNNs; (iii) full stochasticity may be unnecessary as deep kernel learning is relatively competitive; (iv) infinite-width BNNs are particularly promising, especially in high dimensions.

## 1 Introduction

_Bayesian optimization_[31] is a distinctly compelling success story of Bayesian inference. In Bayesian optimization, we place a prior over the objective we wish to optimize, and use a _surrogate model_ to infer a posterior predictive distribution over the values of the objective at all feasible points in space. We then combine this predictive distribution with an _acquisition function_ that trades-off exploration (moving to regions of high uncertainty) and exploitation (moving to regions with a high expected value, for maximization). The resulting approach converges quickly to a global optimum, with strong performance in many expensive black-box settings ranging from experimental design, to learning parameters for simulators, to hyperparameter tuning [7, 10].

While many acquisition functions have been proposed for Bayesian optimization [e.g. 8, 44], Gaussian processes (gps) with standard Matern or RBF kernels are almost exclusively used by default as a surrogate model for the objective, without checking whether other alternatives would be more appropriate, despite the fundamental role that the surrogate model plays in Bayesian optimization.

Thus, despite promising advances in Bayesian optimization research, there is an elephant in the room: _should we be considering other surrogate models?_ It has become particularly timely to evaluate Bayesian neural network (bnn) surrogates as alternatives to Gaussian processes with standard kernels: In recent years, there has been extraordinary progress in making bnn practical [e.g. 4, 17, 34, 42, 47]. Moreover, bnn can flexibly represent the
non-stationary behavior typical of optimization objectives, discover similarity measures as part of representation learning which is useful for higher dimensional inputs, and naturally handle multi-output objectives. In parallel, new _Monte-Carlo_ acquisition functions [1] have been developed which only require posterior samples, significantly lowering the barrier to using non-gp surrogates that do not provide closed-form predictive distributions.

In this paper, we exhaustively evaluate Bayesian neural networks as surrogate models for Bayesian optimization. We consider conventional fully stochastic multilayer bnn with a variety of approximate inference procedures, ranging from high-quality full-batch Hamiltonian Monte Carlo [15, 27, 28], to stochastic gradient Markov Chain Monte Carlo [3], to heuristics such as deep ensembles [21]. We also consider infinite-width bnn[22, 29], corresponding to gps with fixed non-stationary kernels derived from a neural network architecture, as well as partially Bayesian last-layer deep kernel learning methods [48]. This particularly wide range of neural network-based surrogates allows us to evaluate the role of representation learning, non-stationarity, and stochasticity in modeling Bayesian optimization objectives. Moreover, given that so much is unknown about the role of the surrogate model, we believe it is particularly valuable not to have a "horse in the race", such as a special bnn model particularly designed for Bayesian optimization, in order to conduct an unbiased scientific study where any outcome is highly informative.

We also extensively study a variety of synthetic and real-world objectives--with a wide range of input space dimensionalities, single- and multi-dimensional output spaces, and both discrete and continuous inputs, and non-stationarities.

Our study provides several key findings: (1) while stochasticity is often prized in Bayesian optimization [10, 35], fully stochastic bnn do not consistently dominate deep kernel learning, which is not stochastic about network parameters, due to the small data sizes in Bayesian optimization; (2) of the fully stochastic bnn, hmc generally works the best for Bayesian optimization, and deep ensembles work surprisingly poorly, given their success in other settings; (3) on standard benchmarks, standard gps are relatively competitive, due to their strong priors and simple exact inference procedures; (4) there is no single method that dominates across most problems, demonstrating that there is significant variability across Bayesian optimization objectives, where tailoring the surrogate to the objective has particular value; (5) infinite-width bnn are surprisingly effective at high-dimensional optimization, even relative to dkl and stochastic multilayer bnn. These results suggest that the non-Euclidean similarity metrics constructed from neural networks are valuable for high-dimensional Bayesian optimization, but representation learning (provided by dkl and finite-width bnn) is not as valuable as a strong prior derived from a neural network architecture (provided by the infinite-width bnn).

This study also serves as an evaluation framework for considering alternative surrogate models for Bayesian optimization. Our code is available at [https://github.com/yucenli/bnn-bo](https://github.com/yucenli/bnn-bo).

## 2 Related Work

There is a large body of literature on improving the performance of Bayesian optimization. However, an overwhelming majority of this research only considers Gaussian process surrogate models, focusing on developing new acquisition functions [e.g. 8, 44], additive covariance functions [9, 16], the inclusion of gradient information [49], multi-objectives [41], trust region methods that use input partitioning for higher dimensional and non-stationary data [6], and covariance functions for discrete inputs and strings [26]. For a comprehensive review, see Garnett [10].

There has been some prior work focusing on other types of surrogate models for Bayesian optimization, such as random forests [13] and tree-structured Parzen estimators [2]. Moreover, Snoek et al. [36] apply a Bayesian linear regression model to the last layer of a deterministic neural network for computational scaling, which can be helpful for the added number of
objective queries associated with higher dimensional inputs. Deep kernel learning [48], which transforms the inputs of a Gaussian process kernel with a deterministic neural network, is sometimes also used with Bayesian optimization, especially in specialized applications like protein engineering [40]. Additionally, the linearized-Laplace approximation to produce a linear model from a neural network has recently been applied to Bayesian optimization in concurrent work [19].

Despite the extraordinary recent practical advances in developing Bayesian neural networks for many tasks [e.g. 47], and recent Monte-Carlo acquisition functions which make it easier to use surrogates like bnn that do not provide closed-form predictive distributions [1], there is a vanishingly small body of work that considers bnn as surrogates for Bayesian optimization. This is surprising, since we would indeed expect bnn to have properties naturally aligned with Bayesian optimization, such as the ability to learn non-stationary functions without explicit modeling interventions and gracefully handle high-dimensional input and output spaces.

The possible first attempt to use a Bayesian neural network surrogate for Bayesian optimization [38] came before most of these advances in bnn research, and used a form of stochastic gradient Hamiltonian Monte Carlo (sghmc) [3] for inference. Like Snoek et al. [36], the focus was largely on scalability advantages over Gaussian processes; however, the reported gains were marginal, and puzzling in that they were largest for a _small_ number of objective function queries (where the neural net would not be able to learn a rich representation, and scalability would not be required). Kim et al. [18] used the same method for bnn with Bayesian optimization, also with sghmc, targeted at scientific problems with known structures and high dimensionality. In these applications, bnn leverage auxiliary information, domain knowledge, and intermediate data, which would not typically be available in many Bayesian optimization problems.

Our paper provides several key contributions in the context of this prior work, where standard gp surrogates are nearly always used with Bayesian optimization. While finite-width bnn surrogates have been attempted, they have been largely limited to sghmc inference, and are often applied in specialized settings without an effort to understand their properties. Little is known about whether bnn could generally be used as an alternative to gps for Bayesian optimization, especially in light of more recent general advances in bnn research. This is the first paper to provide a comprehensive study of bnn surrogates, considering a range of model types and experimental settings. We test the utility of bnn in a variety of contexts, exploring their behavior as we change the dimensionality of the problem and the number of objectives, investigating their performance on non-stationary functions, and also incorporating problems with a mix of discrete and continuous input parameters. Moreover, we are the first to study infinite bnn models in Bayesian optimization, and to consider the role of stochasticity and representation learning in neural network based Bayesian optimization surrogates. Finally, rather than champion a specific approach, we provide an objective assessment, also highlighting the benefits of gp surrogates for general Bayesian optimization problems.

## 3 Surrogate Models

We consider a wide variety of surrogate models, helping to separately understand the role of stochasticity, representation learning, and strong priors in Bayesian optimization surrogates. We provide additional information about these surrogates, as well as background about Bayesian optimization in Appendix A.

Gaussian Processes.Throughout our experiments, when we refer to Gaussian processes, we always mean _standard_ Gaussian processes, with the Matern-5/2 kernel that is typically used in Bayesian optimization [35]. These Gaussian processes have the advantage of simple
exact inference procedures, strong priors, and only a few hyperparameters, such as length-scale, which controls rate of variability. On the other hand, these models are _stationary_, meaning the covariance function is translation invariant and models the objective as having similar properties (such as rate of variation) at different points in input space. They also provide a similarity metric for data points based on simple Euclidean distance of inputs, which is often not suitable for higher dimensional input spaces.

Fully Stochastic Finite-Width Bayesian Neural Networks.These models treat all of the parameters of the neural network as random variables, which necessitates approximate inference. We consider four different mechanisms of approximate inference: (1) Hamiltonian Monte Carlo (hmc), an MCMC procedure which is the computationally expensive gold standard [15, 27, 28]; (2) Stochastic Gradient Hamiltonian Monte Carlo (sgmc), a scalable MCMC approach that works with mini-batches [3]; (3) Deep Ensembles, considered a practically effective heuristic that ensembles a neural network retrained multiple times, and has been shown to approximate fully Bayesian inference [15, 21]. These approaches are fully stochastic, and can do _representation learning_, meaning that they can learn appropriate distance metrics for the data as well as particular types of non-stationarity.

Deep Kernel Learning.Deep kernel learning (dkl) [48] is a hybrid Bayesian deep learning model, which layers a GP on top of a neural network feature extractor. This approach can do non-Euclidean representation learning, handle non-stationarity, and also uses exact inference. However, it is only stochastic about the last layer.

Figure 1: **The design of the bnn has a significant impact on the uncertainty estimates. We visualize the uncertainty estimates and function draws produced by full-batch hmc on a simple toy objective function with four function queries (denoted in black). For the visualizations above, we fix all other design choices with the following base parameters: likelihood variance \(=1\), prior variance \(=1\), number of hidden layers \(=3\), and width \(=128\). We see that varying the different aspects of the model leads to significantly different posterior predictive distributions.**
Linearized Laplace Approximation.The linearized-Laplace approximation (lda) is a deterministic approximate inference method that uses the Laplace approximation [24, 14] to produce a linear model from a neural network, and has recently been considered for Bayesian optimization in concurrent work [19].

Infinite-Width Bayesian Neural Networks.Infinite-width neural networks (i-bnn) refer to the behavior of neural networks as the number of nodes per hidden layer increases to infinity. Neal [27] famously showed with a central limit theorem argument that a bnn with a single infinite-width hidden layer converges to a gp with a neural network covariance function, and this result has been extended to deep neural networks by Lee et al. [22]. i-bnn are fully stochastic and very different from standard gps, as it can handle non-stationarity and provides a non-Euclidean notion of distance inspired by a neural network. However, it cannot do representation learning and instead has a fixed covariance function that provides a relatively strong prior.

### Role of Architecture

We conduct a sensitivity study into the role of architecture and other key design choices for bnn surrogate models. We highlight results for hmc inference, as it is the gold standard for approximate inference in bnns [15].

Gaussian processes involve relatively few design choices -- essentially only the covariance function, which is often simply set to the RBF or Matern kernel. Additionally, we are able to have an intuitive understanding of what the induced distributions over functions look like for the different choices of covariance functions.

In contrast, with bnns, we must consider the architecture, the prior over parameters, and the approximate inference procedure. It is also less clear how different modeling choices in bnns affect the inferred posterior predictive distributions. To illustrate these differences for varying network, prior, and variance parameters, we plot the inferred posterior predictive distributions over functions for different network widths and depths, the activation functions, and likelihood and variance parameters in Figure 1, and we evaluate the performance under different model choices for three synthetic data problems in Figure 2. We focus on fully-connected multi-layer perceptrons for this study: while certain architectures have powerful inductive biases for computer vision and language tasks, generic regression tasks such as Bayesian optimization tend to be well-suited for fully-connected multi-layer perceptrons, which have relatively mild inductive biases and make loose assumptions about the structure of the function we are modeling.

Model Hyperparameters.We consider isotropic priors over the neural network parameters with zero mean and variance parameters 0.1, 1, and 10. Similarly, we consider Gaussian likelihood functions with variance parameters 0.1, 1, and 10. The corresponding posterior predictive distributions for full-batch hmc are shown in Figure 0(a). As would be expected, an increase in the likelihood variance results in a poor fit of the data and virtually no posterior collapse. In contrast, increasing the prior variance results in a higher predictive variance between data points with a good fit to the data points, whereas a prior variance that is too small leads to over-regularization and uncertainty collapse. As shown in Figure 1(a) and Figure 1(c), lower likelihood variance parameters and larger prior variance parameters tended to perform best across three synthetic data experiments.

Network Width and Depth.To better understand the effects of the network size on inference, we explore the differences in performance when varying the number of hidden layers and the number of parameters per layer, each corresponding to an increase in model complexity. In Figure 0(c), we can see that there is a significant increase in uncertainty as we increase the number of hidden layers from two to three, and a bnn with four hidden layers
has even larger uncertainty with higher variations in the function draws. Figure (d)d also shows an increase in uncertainty as we increase the width of the network, where a smaller width leads to function draws that are much flatter than function draws from a larger width. However, the best size to choose seems to be problem-dependent, as shown in Figure (b)b and Figure (d)d.

Activation Function.The choice of activation function in a neural network determines important characteristics of the function class, such as smoothness or periodicity. The impact of the activation function can be seen in Appendix D, with function draws from the ReLU bnn appearing more jagged and function draws from the tanh bnn more closely resembling the draws from a gp with a Squared Exponential or Matern 5/2 covariance function.

## 4 Empirical Evaluation

We provide an extensive empirical evaluation of bnn surrogates for Bayesian optimization. We first assess how bnn compare to gps in relatively simple and well-understood settings through commonly used synthetic objective functions, and we perform an empirical comparison between gps and different types of bnns (hmc, sghmc, lla, ensemble, i-bnn, and dkl). To further ascertain whether bnn may be a suitable alternative to gps in real-world Bayesian optimization problems, we study six real-world datasets used in prior work on Bayesian optimization with gp surrogates [5, 6, 25, 30, 43]. We also provide evidence that the performance of bnns could be further improved with a careful selection of network hyperparameters. We conclude our evaluation with a case study of Bayesian optimization tasks where simple Gaussian process models may fail but bnn models would be expected to prevail. To this end, we design a set of experiments to assess the performance of gps and bnns as a function of the input dimensionality and in settings where the objective function is non-stationary.

Figure 2: **There is no single architecture for hmc that performs the best across all problems. We compare the impact of the design on the Bayesian optimization performance for different benchmark problems. For each set of experiments, we fix all other aspects of the design and plot the values of the maximum reward found using hmc after 100 function evaluations over 10 trials.**
### Synthetic Benchmarks

We evaluate bnn and gp surrogates on a variety of synthetic benchmarks, and we choose problems with a wide span of input dimensions to understand how the performance differs as we increase the dimensionality of the data. We also select problems that vary in the number of objectives to compare the performance of the different surrogate models. Detailed problem descriptions can be found in Appendix B, and we include the experiment setup in Appendix C. We use Monte-Carlo based Expected Improvement [1] as our acquisition function for all problems.

As shown in Figure 3, we find bnn surrogate models to show promising results; however, the specific performance of different bnn varies considerably per problem. dkl matches gps in Branin and BraninCurrin, but seems to perform poorly on highly non-stationary problems such as Ackley. i-bnn also seem to slightly underperform compared to gps on these synthetic problems, many of which have a small number of input dimensions. In contrast, we find finite-width bnn using full hmc to be comparable to gps, performing very similarly in many of the experiments, slightly underperforming in Hartmann and DTLZ5, and outperforming standard gps in the 10-dimensional Ackley experiment. However, this behavior is not generalizable to all approximate inference methods: the performance of sghmc and lla vary significantly per problem, matching the performance of hmc and gps in some experiments while failing to approach the maximum value in others. Deep ensembles also consistently underperform the other surrogate models, plateauing at noticeably lower objective values on multi-objective problems like BraninCurrin and DTLZ1. This result is surprising, since ensembles are often seen to as an effective way to measure uncertainty (Appendix D).

### Real-World Benchmarks

To provide an evaluation of bnn surrogates in more realistic optimization problems, we consider a diverse selection of real-world applications which span a variety of domains, such as

Figure 3: **bnns are often comparable to gps on standard synthetic benchmarks.** However, the type of bnn used has a big impact: hmc is similar to gps and even outperforms them on Ackley, while sghmc and deep ensembles seem to have less reliable performance and are often unable to effectively find the maximum. i-bnns also seem to struggle on these low-dimensional problems. For each benchmark function, we include \(d\) for the number of input dimensions, and \(o\) for the number of objectives. We plot the mean and one standard error of the mean over 5 trials.
solving differential equations and monitoring cellular network coverage [5, 6, 25, 30, 43]. Many of these problems, such as the development of materials to clean oil spills, have consequential applications; however, these objectives are often multi-modal and are difficult to optimize globally. Additionally, unlike the synthetic benchmarks, many real-world applications consist of input data with ordinal or categorical values, which may be difficult for gps to handle. Several of the problems also require multiple objectives to be optimized. Detailed problem descriptions are provided in Appendix B.

We share the results of our experiments in Figure 4, and details about the experiment setup can be found in Appendix C. The results are mixed: bnns are able to significantly outperform gps in the Pest Control dataset, while gps find the maximum reward in the Cell Coverage and Lunar Lander experiments. The Pest Control, Cell Coverage, and Oil Spill Sorbent experiments all include discrete input parameters, and there seems to be a slight trend of gp and i-bnns performing well, and sghmc and ensembles performing more poorly. Similar to the findings from the synthetic benchmarks, we see that the different approximate inference methods for finite-width bnns lead to significantly different Bayesian optimization performance, with hmc generally finding higher rewards compared to sghmc, lla, and deep ensembles. Additionally, it appears that gps perform well in the two multi-objective problems, although that may not be generalizable to additional multi-objective problems and may be more related to the curvature of the specific problem space.

### Neural Architecture Search

To better understand the impact of architecture on the performance of bnns, we conduct an extensive neural architecture search over a selection of the benchmark problems, varying the width, depth, prior variance, likelihood variance, and activation function. For our experiments, we use SMAC3 [23], a framework which uses Bayesian optimization to select the best hyperparameters, and we detail the experiment setup in Appendix C. While a thorough search over architectures is often impractical for realistic settings of Bayesian

Figure 4: **Real world benchmarks show mixed results. bnns outperform gps on some problems and underperform on others, and there does not seem to be a noticeable preference for any particular surrogate as we increase the number of input dimensions. Additionally, there does not appear to be a clear separation between the top row of experiments, which optimize over continuous parameters, and the bottom row of experiments, which also include some discrete inputs. For each benchmark, we include \(d\) for the number of input dimensions, and \(o\) for the number of objectives. We plot the mean and one standard error of the mean over 5 trials.**
optimization since it requires a very large number of function evaluations, we use this experiment to demonstrate the flexibility of bnn and to showcase its potential when the design is well-suited for the problem.

We show the effect of neural architecture search on hmc surrogate models in Figure 5. On the Cell Coverage problem, the architecture search did not drastically change the performance of hmc. In contrast, extensively optimizing the hyperparameters made a significant difference on the Pest Control problem, leading to hmc finding higher rewards than gps while using fewer function evaluations; however, on this problem i-bnn, which does not require specifying an architecture, still performs best. Neural architecture search was also able to improve the results on dtlz5, leading hmc to be competitive with other surrogate models such as i-bnn and dkl. The difference in the benefits of the search may be attributed to some problems having less inherent structure than others, where extensive hyperparameter optimization may not be as necessary. Additionally, our original hmc surrogate model choice may already have been a suitable choice for some problems, so an extensive search over architectures may not significantly improve the performance.

### Limitations of GP Surrogate Models

Although popular, gps suffer from well-known limitations that directly impact their usefulness as surrogate models. To contrast bnn and gp surrogates, we explore two failure modes of gps and demonstrate that the increased flexibility provided by bnn surrogate models can overcome these issues and improve performance in Bayesian optimization.

Non-Stationary Objective Functions.To use gp surrogates, we must specify a kernel function class that governs the covariance structure over data points and learn the kernel hyperparameters. We typically constrain model selection to models with kernel of the form \(k(\mathbf{x},\mathbf{x}^{\prime})=k(\|\mathbf{x}-\mathbf{x}^{\prime}\|)\). This constraint makes it easier to describe the functional form and learn the hyperparameters of the kernel. However, because the covariance between two values only depends on their distance and not on the values themselves, this setup assumes the function is stationary and has similar mean and smoothness throughout the input space. Unfortunately, this assumption of stationarity does not hold true in many real-world settings. For example, in the common Bayesian optimization application of choosing hyperparameters of a neural network, the true loss function landscape may have vastly different behavior in one part of the input space compared to another. bnn surrogates, in contrast to gp surrogates, are able to model non-stationary functions without similar constraints.

Figure 5: **The impact of neural architecture search on hmc is problem-dependent. The dashed green line indicates the performance of hmc after an extensive neural architecture search, compared to the solid green line representing the hmc model selected from a much smaller pool of hyperparameters. We see that it has minimal impact on Cell Coverage (left), moderate impact on dtlz5 (center), and extensive impact on Pest Control (right), even outperforming gps. For each benchmark, we include \(d\) for the number of input dimensions, and \(o\) for the number of objectives. We plot the mean and one standard error of the mean over 5 trials.**
In Appendix D, we show the performance of Bayesian optimization with bnn and gp surrogate models for a non-stationary objective function. Because the gp assumes that the behavior of the function is the same throughout the input domain, it cannot accurately model the input-dependent variation of the function and leads to underfitting around the true optimum. In contrast, bnn surrogates can learn the non-stationarity of the function.

High-Dimensional Input Spaces.Due to the curse of dimensionality, gps do not scale well to high-dimensional input spaces without careful human intervention. Common covariance functions may fail to faithfully represent high-dimensional input data, making the design of custom-tailored kernel functions necessary. In contrast, neural networks are well-suited for modeling high-dimensional input data [20].

To measure the effect of dimensionality on the performance of gps and bnns, we first benchmark the ability of the surrogate models to maximize synthetic test functions provided by high-dimensional polynomial functions and function draws from neural networks. To better understand the behaviors of different surrogate models in real-world settings, we also construct a high-dimensional problem by using Bayesian optimization to set the parameters of a neural network in the context of knowledge distillation. Knowledge distillation refers to the act of "distilling" information from a larger teacher model to a smaller student model by matching model outputs [11], and it is known to be a difficult optimization problem [39]. For full descriptions of the high-dimensional problems, see Appendix B.

We share the results of our findings in Figure 6 and Appendix D. There is a clear trend that i-bnns perform very well in these high-dimensional settings. The i-bnn has several conceptual advantages in this setting: (1) it provides a non-Euclidean and non-stationary similarity metric, which can be particularly valuable in high-dimensions; (2) it does not have any hyperparameters for learning, and thus is not "data hungry" -- especially important in high dimensional problems with small data sizes, which provide relatively little information for representation learning.

Additionally, we find that other bnn surrogate models also outperform gps across the high-dimensional problems, providing a compelling motivation for bnns as surrogate models for Bayesian optimization.

Figure 6: **i-bnns outperform other surrogates in many high-dimensional settings**. We show the results of maximizing a polynomial function (left), maximizing a fixed function draw from a neural network (center), and optimizing the parameters of a neural network in the context of knowledge distillation (right). All of these objectives are high-dimensional and non-stationary, and we find that bnns consistently find higher rewards than gps across all problems. We plot the mean and one standard error of the mean over 5 trials, and \(d\) corresponds to the number of input dimensions.
Discussion

While Bayesian optimization research has made significant progress over the last few decades [10], the surrogate model is a crucial and highly underexplored design choice. Although standard gp models are the default surrogate, it is not because they have been shown to be superior to alternatives -- we simply have had almost no evidence about how alternatives would perform. It is particularly timely to consider neural network surrogates, given significant recent advances in Bayesian neural networks and related approaches. The setting of Bayesian optimization is also quite different from where bnn are typically applied, involving particularly small datasets and online learning.

The fact that dkl is competitive with bnn calls into question the importance of fully stochastic surrogates -- a surprising finding given the small datasets common to Bayesian optimization, where overfitting would be a more significant concern. Moreover, infinite-width bnn show promising performance in general, but especially for higher dimensional settings. Given that they also do not involve learning many hyperparameters, and do not require approximate inference, it is possible they could become a de facto standard surrogate for Bayesian optimization.

We also show that the Bayesian optimization objectives are sufficiently different such that one method does not generally dominate. This finding supports the use of simple models with strong but generic assumptions, such as standard gp models, which indeed provide relatively competitive performance. On the other hand, perhaps it is self-fulfilling that standard gps would be competitive on standard benchmarks, since most problems were designed with very few alternative models in mind. It may be time for the community to consider new benchmarks that evolve with the advances in our choice of surrogate models.

### Acknowledgements

We thank Greg Benton for helpful guidance in the beginning stages of this research, and Sanyam Kapoor for discussions. This work is supported by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A1, NSF IIS-1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and an Amazon Research Award.