1. From an experimental design perspective, the experimental design suggested by the authors has been used widely for open-domain dialogue systems with the caveat of it not being done in live interactive settings.
2. The authors have not referenced those works that use continuous scales in the evaluation and there is a large bunch of literature missing from the paper. Some of the references are provided in the comments section.
3. Lack of screenshots of the experimental interface
Comments: 1. Please add screenshots of the interface that was designed.
2. Repetition of the word Tables in Line 549 3. In Appendix A.3, the GLEU metric is reference as GLUE.
Questions: 1. In table 1, is there any particular reason for the reduction in pass rate % from free run 1 and free run2?
2. What is the purpose of the average duration reported in Table 1? There is no supporting explanation about it. Does it include time spent by the user waiting for the model to generate a response?
3. With regards to the model section, is there any particular reason that there was an emphasis on choosing retriever-based transformer models over generative models? Even if the models are based on ConvAI2, there are other language modeling GPT2 based techniques that could have been picked.
4. In figure 6, what are the models in the last two columns lan_model_p and lan_model?
Missing References: 1. Howcroft, David M., Anja Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. " Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions." In Proceedings of the 13th International Conference on Natural Language Generation, pp. 169-182. 2020.
2. Santhanam, S. and Shaikh, S., 2019. Towards best experiment design for evaluating dialogue system output. arXiv preprint arXiv:1909.10122.
3. Santhanam, S., Karduni, A. and Shaikh, S., 2020, April. Studying the effects of cognitive biases in evaluation of conversational agents. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
4. Novikova, J., Dušek, O. and Rieser, V., 2018. RankME: Reliable human ratings for natural language generation. arXiv preprint arXiv:1803.05928.
5. Li, M., Weston, J. and Roller, S., 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087.
2. What is the purpose of the average duration reported in Table 1? There is no supporting explanation about it. Does it include time spent by the user waiting for the model to generate a response?