- My main concern is that the performance improvement, though generally better, is not particularly too significant, not to mention that those proxy-based method achieves also pretty good IM results while using only a negligible amount of time compared to BOIM (or other simulation-based method in general)
- Other choices of graph kernel are not considered and experimented with such as random walk or Graphlet kernel? There are probably easy tricks to turn them into valid GP kernels.
- Despite the time reduction introduced by BOIM, proxy-based methods are still substantially cheaper. Would it be possible to use proxy-based methods as heuristics to seed BOIM or other zero-order optimization method (e.g., CMA-ES).
- While the author has shown that GSS has theoretically lower variance, it’d be nice to compare against with random sampling and check empirically how well it performs.
- Results presentation can be improved. For example, in Figure 2 and 3, the y-axis is labeled as “performance” which is ambiguous, and the runtime is not represented in those figure. A scatter plot with x/y axes being runtime/performance could help the reader better understand and interpret the results. Best results in tables can also be highlighted. Minor:
- Typo in Section 2: “Mockus (1998) and has since become…” → “Mockus (1998) has since become…”
- Results presentation can be improved. For example, in Figure 2 and 3, the y-axis is labeled as “performance” which is ambiguous, and the runtime is not represented in those figure. A scatter plot with x/y axes being runtime/performance could help the reader better understand and interpret the results. Best results in tables can also be highlighted. Minor: