1. The case made for adopting the proposed strategy for a new automated evaluation paradigm - auto-rewrite (where the questions that are not valid due to a coreference resolution failure in terms of the previous answer get their entity replaced to be made consistent with the gold conversational history) - seems weak. While the proposed strategy does seem to do better in terms of being closer to how humans evaluated the 4 models (all in the context of one specific English dataset), it is not clear how the proposed strategy - a) does better than the previously proposed strategy of using model-predicted history (auto-pred). Looking at the comparison results for different evaluations - in terms of table 1, there definitely does not seem to be much difference between the two strategies (auto-rewrite and auto-pred). In fig 5, for some (2/6) pairs, the pred-history strategy has higher agreement than the proposed auto-rewrite strategy while they are all at the same agreement for 1/6 pairs. b) gets to the fundamental problem with automated evaluation raised in the paper, which is that "when placed in realistic settings, the models never have access to the ground truth (gold answers) and are only exposed to the conversational history and the passage." The proposed strategy seems to need gold answers as well, which is incompatible with the real-world use case. The previously proposed auto-pred strategy, however, uses only the questions and the model's own predictions to form the conversational history - which seems to be more compatible with the real-world use case. In summary, it is not clear why the proposed new way of automatically evaluating CQA systems is better or should be adopted as opposed to the previously proposed automated evaluation method of using a model's predictions as the conversational history (auto-pred), and the comparison between the results for these two automated strategies seems to be a missing exploration and discussion.
Questions to the authors (which also act as suggestions): Q1. - Line 151: "four representative CQA models" - what does representative mean here? representative in what sense? In terms of types or architectures of models? This needs clarification and takes on importance because the discrepancy, in terms of how models get evaluated on human vs automated evaluation, depends on these four models in a sense. Q2. Line 196: "We noticed that the annotators are biased when evaluating the correctness of answers" - are any statistics on this available? Q3. Section 3.1: For Mechanical Turk crowdsourcing work, what was the compensation rate for the annotators? This should be mentioned, if not in the main text, then add to appendix and point to it in the main text. Also, following the work in Card et al (2020) ("With Little Power Comes Great Responsibility.") - were there any steps taken to ensure the human annotation collection study was appropriately powered? ( If not, consider noting or discussing this somewhere in the paper as it helps with understanding the validity of human experiments) Q4. Lines 264-265: "The gap between HAM and ExCorD is significant in Auto-Gold" - how is significance measured here?
Q5. Lines 360-364: "We determine whether e∗_j = e_j by checking if F1(s∗_{j,1}, s_{j,1}) > 0 .... .... as long as their first mentions have word overlap." Two questions here - 5a. It is not clear why word overlap was used and not just an exact match here? What about cases where there is some word overlap but the two entities are indeed different, and therefore, the question is invalid (in terms of coreference resolution) but deemed valid?
5b. How accurate is this invalid question detection strategy? In case this has not already been measured, perhaps a sample of instances where predicted history invalidates questions via unresolved coreference (marked by humans) can be used to then detect if the automated method catches these instances accurately. Having some idea of how well invalid question detection happens is needed to get a sense of if or how many of the invalid questions will get rewritten. Comments, suggestions, typos: - Line 031: "has the promise to revolutionize" - this should be substantiated further, seems quite vague. - Line 048: "extremely competitive performance of" - what is 'performance' for these systems? ideally be specific since, at this point in the paper, we do not know what is being measured, and 'extremely competitive' is also quite vague. - The abstract is written well and invokes intrigue early - could potentially be made even better if, for "evaluating with gold answers is inconsistent with human evaluation" - an example of the inconsistency, such as models get ranked differently is also given there. - Line 033: "With recent development of large-scale datasets" -> the* recent development, but more importantly - which languages are these datasets in? And for this overall work on CQA, the language which is focused on should be mentioned early on in the introduction and ideally in the abstract itself. - Line 147: "more modeling work has been done than in free-form question answering" - potential typo, maybe it should be "maybe more modeling work has been done 'in that'" - where that refers to extractive QA?
- Line 222: "In total, we collected 1,446 human-machine con- versations and 15,059 question-answer pairs" - suggestion: It could be reasserted here that this dataset will be released as this collection of conversations is an important resource and contribution and does not appear to have been highlighted as much as it could. - Figure 2: It is a bit unintuitive and confusing to see the two y-axes with different ranges and interpret what it means for the different model evaluations. Can the same ranges on the y-axes be used at least even if the two metrics are different? Perhaps the F1 can use the same range as Accuracy - it would mean much smaller gold bars but hopefully, still get the point across without trying to keep two different ranges in our head? Still, the two measures are different - consider making two side-by-side plots instead if that is feasible instead of both evaluations represented in the same chart. - Lines 250-252: "the absolute numbers of human evaluation are much higher than those of automatic evaluations" - saying this seems a bit suspect - what does absolute accuracy numbers being higher than F1 scores mean? They are two different metrics altogether and should probably not be compared in this manner. Dropping this, the other implications still hold well and get the point across - the different ranking of certain models, and Auto-Gold conveying a gap between two models where Human Eval does not. - Line 348: "background 4, S∗ - latex styling suggestion, add footnote marker only right after the punctuation for that renders better with latex, so - "background,\footnote{} ..." in latex.
- Footnote 4: 'empirically helpful' - should have a cite or something to back that there. - Related Work section: a suggestion that could make this section but perhaps also the broader work stronger and more interesting to a broader audience is making the connection to how this work fits with other work looking at different NLP tasks that looks at failures of the popular automated evaluation strategy or metrics failing to capture or differing significantly from how humans would evaluate systems in a real-world setting.
- The abstract is written well and invokes intrigue early - could potentially be made even better if, for "evaluating with gold answers is inconsistent with human evaluation" - an example of the inconsistency, such as models get ranked differently is also given there.