)
The idea of incorporating the training dynamics to the Bayesian optimization tuning process to construct online BO is novel.
The experiments are conducted on two complex datasets CIFAR-10, CIFAR-100. Weaknesses:
No deep analysis is conducted to understand why the proposed method can lead to better generalization.
I feel unclear with several technical details: 1) What is the x-axis in Figures 1 & 2? Is it the number of epochs? 2) How many experiments are repeated in Figures 1&2 and Table 1? 3) How to set the search space S for GSdyn? In your experiments in Section 5, how do the authors set the search space S? 4) What is the objective function for GSdyn and FABOLAS? In Section 5.1, it is mentioned that the DNN’s accuracy is the objective function, but which accuracy? The accuracy on the validation dataset or on the test dataset? 5) To evaluate BO, the standard practice is to find the hyperparameter set with the best accuracy on the validation dataset. Why in this work, the accuracy on the test dataset (but not validation dataset) is compared between baseline methods (Figures 1 &2)? And which accuracies are there in Table 1? I understand that GSdyn leads to good generalization but the accuracy on the validation dataset is also needed to be shown as it is the objective of the vanilla BO? 6) The experiments might include different hyper-parameters, and more hyper-parameters.
Minor comments:
In the figures, the labels of each axis need to be added.
Third bullet in the summarized contributions in Section 1: Beyes --> Bayes
Line 5 of Algorithm 1: Should be either \sigma_0 or \sigma, not a mix of them?
Line 5 of Algorithm 2: What is Sample function? I understand it is the acquisition function but a rigorous formula of the acquisition function needs to be provided.
2) How many experiments are repeated in Figures 1&2 and Table 1?