, and while I recommend acceptance, I think that it deserves to be substantially improved before publication. One of the main concerns is that part of the relevant literature has been ignored, and also importantly that the proposed approach has not really been extensively compared to potential competitors (that might need to be adapted to the multi-source framework; not e also that single-fidelity experiments could be run in order to better understand how the proposed acquisition function compares to others from the literature). Another main concern in connection with the previous one is that the presented examples remain relatively simple, one testcase being an analytical function and the other one a one-dimensional mototonic function. While I am not necessarily requesting a gigantic benchmark or a list of complicated high-dimensional real-world test cases, the paper would significantly benefit from a more informative application section. Ideally, the two aspects of improving the representativity of numerical test cases and of better benchmarking against competitor strategies could be combined. As of missing approaches from the literature, some entry points follow: * Adaptive Designs of Experiments for Accurate Approximation of a Target Region (Picheny et al. 2010) http://mechanicaldesign.asmedigitalcollection.asme.org/article.aspx?articleid=1450081 *Fast kriging-based stepwise uncertainty reduction with application to the identification of an excursion set (Chevalier et al. 2014a) http://amstat.tandfonline.com/doi/full/10.1080/00401706.2013.860918 * NB: approaches from the two articles above and more (some cited in the submitted paper but not benchmarked against) are coded for instance in the R package "KrigInv". The following article gives an overview of some of the package's functionalities: KrigInv: An efficient and user-friendly implementation of batch-sequential inversion strategies based on kriging (Chevalier et al. 2014b) * In the following paper, an entropy-based approach (but not in the same fashion as the one proposed in the submitted paper) is used, in a closely related reliability framework: Gaussian process surrogates for failure detection: A Bayesian experimental design approach (Wang et al. 2016) https://www.sciencedirect.com/science/article/pii/S002199911600125X * For an overall discussion on estimating and quantifying uncertainty on sets under GP priors (with an example in contour line estimation), see Quantifying Uncertainties on Excursion Sets Under a Gaussian Random Field Prior (Azzimonti et al. 2016) https://epubs.siam.org/doi/abs/10.1137/141000749 NB: the presented approaches to quantify uncertainties on sets under GP priors could also be useful here to return a more complete output (than just the contour line of the predictive GP mean) in the CLoVER algorithm. * Coming to the multi-fidelity framework and sequential design for learning quantities (e.g. probabilities of threshold exceedance) of interest, see notably Assessing Fire Safety using Complex Numerical Models with a Bayesian Multi-fidelity Approach (Stroh et al. 2017) https://www.sciencedirect.com/science/article/pii/S0379711217301297?via%3Dihub Some further points * Somehow confusing to read "relatively inexpensive" in the abstract and then "expensive to evaluate" in the first line of the introduction! * L 55 "A contour" should be "A contour line"? * L88: what does "f(l,x) being the normal distribution..." mean? * It would be nice to have a point-by-point derivation of equation (10) in the supplementary material (that would among others help readers including referees proofchecking the calculation). * About the integral appearing in the criterion, some more detail on how its computation is dealt with could be worth. ##### added after the rebuttal I updated my overall grade from 7 to 8 as I found the response to the point and it made me confident that the final paper would be improved (by suitably accounting for my remarks and those of the other referees) upon acceptance. Let me add a comment about related work by RÃ©mi Stroh. The authors are right, the Fire Safety paper is of relevance but does actually not address the design of GP-based multi-fidelity acquisition functions (in the BO fashion). However, this point has been further developed by Stroh et al; see "Sequential design of experiments to estimate a probability of exceeding a threshold in a multi-fidelity stochastic simulator" in conference contributions listed in http://www.l2s.centralesupelec.fr/en/perso/remi.stroh/publications
* Coming to the multi-fidelity framework and sequential design for learning quantities (e.g. probabilities of threshold exceedance) of interest, see notably Assessing Fire Safety using Complex Numerical Models with a Bayesian Multi-fidelity Approach (Stroh et al. 2017) https://www.sciencedirect.com/science/article/pii/S0379711217301297?via%3Dihub Some further points * Somehow confusing to read "relatively inexpensive" in the abstract and then "expensive to evaluate" in the first line of the introduction!