Several technical details unclear: see "Questions" section. These are not absolutely critical to the paper but I would appreciate clarification.
Unclear effect of "different objective" vs "exact solution": the proposed method has 2 deviations from using the empirical mean/variance of a tree ensemble for Bayesopt: 1) making predictions with a tree kernel GP instead of empirical mean/variance 2) optimizing UCB exactly using a cone solver. In section 5 it was generally unclear to me how much of the difference in final performance came from using the trees to make predictions in a different way (GP instead of empirical mean/variance), and how much came from performing [nearly] exact maximization of the acquisition function. A good experiment to do would be use the GP formulation to find a promising set of candidate points (perhaps using multiple seeds), then use the original empirical mean/variance UCB objective to select a point to query in the end. This would be combining "exact optimization" with the standard way of making predictions.
Code requires a non-public library gurobi. This is not a deal-breaker, and I realize there are academic licenses available, but it would be really nice to have a fully open-source version for the camera-ready paper. I ask some additional questions about the choice of solver below. Suggestions
Given that the paper focuses specifically on non-continuous spaces, why is x
defined to be a real number when introducing Bayesopt on line 44? I think x
should just be an element of a general space X
Many readers at NeurIPS may not be very familiar with cone problems / conic solvers. It would be nice to put some information on these in the background, even if it is just a short section saying that "it is a very general way of formulating certain optimization problems, with a variety of established methods for solving them"
"LCB" is a fairly uncommon formulation of Bayesopt; I think many readers are accustomed to seeing problems posed as UCB maximization. Perhaps it would be slightly helpful to the reader to frame the problem as maximization and multiple everything in the paper by − 1 ?
I saw in the appendix that you use Adam to optimize GP kernel hyperparameters. One problem with Adam is that it may converge very slowly if the learning rate is too low, or not fully converge if it is too large. Especially since you are only optimizing 2 hyperparameters, I recommend using L-BFGS instead. In my personal experience this method is much more robust.
The objective in equation 12 would also be another interesting baseline to see: it would show how much the GP variance actually contributes, as opposed to just finding a point with a sufficient amount of disagreement among ensemble members. Overall
Originality: seems high, but I am not very familiar with related work in this area
Quality: high, good paper, good method, good evaluation
Clarity: very high!
Significance: high. The proposed method solves a real problem and is likely to be used in practice, especially if a good open-source library is published.
I definitely recommend accepting this paper and my score reflects that. I would consider increasing my score slightly if my suggestions are addressed. I could also imagine decreasing my score if:
The novelty is not as high as I thought (I am not very familiar with related work in this area, so perhaps there is a paper that does something very similar which was not cited)
There were technical flaws that I did not catch (I know only the basics of convex optimization/cone problems, so this is possible)
As far as I can tell, the limitations of this work are discussed adequately.
1) making predictions with a tree kernel GP instead of empirical mean/variance