- The number of datasets used is relatively small, to really access the importance of different design decisions, it would probably be good to use further datasets, e.g., the classical GeoQuery dataset.
- I would have appreciated a discussion of the statistical properties of the results - with the given number of tests, what is the probability that differences are generated by random noise and does a regression on the different design decisions give us a better idea of the importance of the factors?
- The paper mentions that a heuristic is used to identify variable names in the Django corpus, however, I could not find information on how this heuristic works. Another detail that was not clear to me is whether the BERT model was fine tuned and how the variable strings were incorporated into the BERT model (the paper mentions that they were added to the vocabulary, but not how). For a paper focused on determining what actually matters in building a text to code system, I think it is important to be precise on these details.
It would take some time to implement your task for other corpora, which potentially use different programming languages, but it might be possible to still strengthen your results using bootstrapping. You could resample some corpora from the existing two and see how stable your results are.
If you have some additional space, it would also be interesting to know if you have discuss results based on types of examples - e.g., do certain decisions make more of a difference if there are more variables?
Typos: - Page 1: "set of value" -> "set of values" "For instance, Orlanski and Gittens (2021) fine-tunes BART" -> "fine-tune" - Page 2: "Non determinism" -> "Non-Determinism"
- I would have appreciated a discussion of the statistical properties of the results - with the given number of tests, what is the probability that differences are generated by random noise and does a regression on the different design decisions give us a better idea of the importance of the factors?