- The class of problems is not very well motivated. The CIFAR example is contrived and built for demonstration purposes. It is not clear what application would warrant sequentially (or in batches) and jointly selecting tasks and parameters to simultaneously optimize multiple objective functions. Although one could achieve lower regret in terms of total task-function evaluations by selecting the specific task(s) to evaluate rather than evaluating all tasks simultaneously, the regret may not be better with respect to timesteps. For example, in the assemble-to-order, even if no parameters are evaluated for task function (warehouse s) at timestep t, that warehouse is going to use some (default) set of parameters at timestep t (assuming it is in operation---if this is all on a simulator then the importance of choosing s seems even less well motivated). There are contextual BO methods (e.g. Feng et al 2020) that address the case of simultaneously tuning parameters for multiple different contexts (tasks), where all tasks are evaluated at every timestep. Compelling motivating examples would help drive home the significance of this paper. - The authors take time to discuss how KG handles the continuous task setting, but there are no experiments with continuous tasks - It’s great that entropy methods for conditional optimization are derived in Section 7 in the appendix, but why are these not included in the experiments? How does the empirical performance of these methods compare to ConBO? - The empirical performance is not that strong. EI is extremely competitive and better in low-budget regimes on ambulance and ATO - The performance evaluation procedure is bizarre: “We measure convergence of each benchmark by sampling a set of test tasks S_test ∼ P[s] ∝ W(s) which are never used during optimization”. Why are the methods evaluated on test tasks not used during the optimization since all benchmark problems have discrete (and relatively small) sets of tasks? Why not evaluate performance on the expected objective (i.e. true, weighted) across tasks? - The asymptotic convergence result for Hybrid KG is not terribly compelling - It is really buried in the appendix that approximate gradients are used to optimize KG using Adam. I would feature this more prominently. - For the global optimization study on hybrid KG, it would be interesting to see performance compared to other recent kg work (e.g. one-shot KG, since that estimator formulation can be optimized with exact gradients)
Writing: - L120: this is a run-on sentence - Figure 2: left title “poster mean” -> “posterior mean” - Figure 4: mislabeled plots. The title says validation error, but many subplots appear to show validation accuracy. Also, “hyperaparameters” -> hyperparameters - L286: “best validation error (max y)” is contradictory - L293: “We apply this trick to all algorithms in this experiment”: what is “this experiment”? - The appendix is not using NeurIPS 2021 style files - I recommend giving the appendix a proofread:
Some things that jump out
P6: “poster mean”, “peicewise-linear”
P9: “sugggest”
Limitations and societal impacts are discussed, but the potential negative societal impacts could be expounded upon.
- The authors take time to discuss how KG handles the continuous task setting, but there are no experiments with continuous tasks - It’s great that entropy methods for conditional optimization are derived in Section 7 in the appendix, but why are these not included in the experiments? How does the empirical performance of these methods compare to ConBO?