- The contribution goes in several directions which makes the paper hard to evaluate; is the main contribution the selection of existing datasets, introducing new datasets or new versions of datasets, the empirical evaluation or the software tooling? - The dataset does not describe existing datasets and benchmarks, and so it is hard to judge the exact differences between the proposed datasets and currently used datasets. A more direct comparison might be useful, and it's not clear why existing, smaller datasets are not included in the collection. - For some of the datasets, it's unclear if or how they have been used or published before. In particular, the datasets from Moleculenet seem to be mostly reproduced, using the splitting strategy that was suggested in their paper, with the modification potentially being addition of new features. - If the selection of the datasets is a main contribution, the selection process should be made more clear. What was the pool of datasets that was drawn from, and how were datasets selected? An example of such a work is the OpenML100 and OpenML CC-18 for classification, see Bischl et. al. "OpenML Benchmarking Suites". or Gijsbers et al "An Open Source AutoML Benchmark" In addition to selection of the datasets, the selection of the splitting procedure and split ratios also seems ad-hoc and is not detailed. - Neither the software package, nor the datasets, nor the code for the experiments has been submitted as supplementary material, and the details in the paper are unlikely to be enough to reproduce the creation of the datasets or the experiments given the datasets. - Given that many methods aim at one of the three tasks, having 5, 6 and 4 datasets for the tasks respectively, might not be enough for a very rigorous evaluation, in particular if some of the datasets are so large that not all algorithms can be used on them. Addendum: Thank you to the authors for their detailed reply. A repository and online platform for reproducing the experiments was provided, and it was clarified that the datasets are substantially novel. Motivations for the number and choice of datasets were given and I updated my assessment to reflect that.
- Given that many methods aim at one of the three tasks, having 5, 6 and 4 datasets for the tasks respectively, might not be enough for a very rigorous evaluation, in particular if some of the datasets are so large that not all algorithms can be used on them. Addendum: Thank you to the authors for their detailed reply. A repository and online platform for reproducing the experiments was provided, and it was clarified that the datasets are substantially novel. Motivations for the number and choice of datasets were given and I updated my assessment to reflect that.