While the proposed approach is interesting, I believe that there is a substantial amount of complexity that is currently unjustified. Moreover, little is offered in terms of intuition as to why the involved components are jointly able to produce a more accurate surrogate model. I believe the paper has potential, but these outstanding issues would have to be addressed in a satisfactory manner for the paper to be accepted. Specifically,
(To avoid confusion, I will denote the BO surrogate as the BO GP and the student as the SGP)
### __1. The complexity of the method:__
In addition to the Vanilla GP, there are three learnable components.
- The teacher, trained on two real and synthetic data based on its own classification ability and the student's performance
- The student, trained on synthetic data and validated on real data (synthetic and real)
- The synthetic data generation process, which is optimized to minimize the feedback loss of the first two
This nested scheme makes it difficult to discern whether synthetic data are sensible, are classified correctly and what drives performance - essentially why this process makes sense. As such, plots demonstrating the fit of the student model, the synthetically generated data or anything else which boosts intuition is paramount. As of now, the method is not sufficiently digestible.
### __2. The concept of adding synthetic data:__
In BO, The (GP) surrogate models the underlying objective in a principled manner which yields calibrated uncertainty estimates, assuming that it is properly trained. Since the GP in itself is principles, I am not convinced that adding synthetic data is an intrinsically good idea. The proposed approach suggests that the GP is either un-calibrated or does not sufficiently extrapolate the observed data. While this seems possible, there is no insight into which possible fault (of the GP) is being addressed, nor how the added synthetic data accomplishes it. Is it
1. The teacher MLP that aggressively (and accurately) extrapolates in synthetic data generation?
2. That the vanilla GP is simply under-confident, so that adding synthetic data acts to reduce the uncertainty by adding more data?
3. Some other reason/combination of the two
Note that I am specifically asking for intuition as to how the _modeling_ (and not the BO results) can improve by adding synthetic data.
### __3. The accuracy and role of the teacher:__
If the teacher is able to generate accurate data (synthetic or real), why not use it as a surrogate instead of the conventional GP? Comments like
#### _"Evaluating the performance of the student that is trained with random unlabeled data far away from the global optimum, may not provide relevant feedback for tuning the teacher toward finding the global optimum."_
suggest that it is the teacher's (and not the BO loop's) task to find the global optimum. If the teacher adds data that it believes is good to the BO surrogate, the teacher is indirectly conducting the optimization.
### __4. The role of the student:__
If the student is trained only on synthetic data, validated on a (promising) subset of the true data, and its loss is coupled with the teacher's. As such, the student's only role appears to be as a regularizer (i.e. to incorporate GP-like smoothness into the MLP) while producing approximately the same predictions on labeled data. Can the authors shed light on the role on whether this is true, and if so, what difference the student offers as opposed to conventional regularization.
### __5. Few tasks in results:__
Three benchmarks constitutes a fairly sparse evaluation. Does the proposed method make sense on conventional test functions, and can a couple of those be added?
### __6. Missing references:__
The key idea of 5.1 is _very_ similar to MES (Wang and Jegelka 2017), and so this is a must-cite. They also use a GEV (specifically a Gumbel) to fit $p(y^*)$. __Minor:__
- App. B1: proposes -->purposes, remove ", however"
- Sec 5.0 of systematically sample --> of systematically sampling
- Legend fontsize is tiny, Fig.1 fontsize is too small as well
2. That the vanilla GP is simply under-confident, so that adding synthetic data acts to reduce the uncertainty by adding more data?