1. Despite the well-motivated problem formulation, the simulation is not realistic. The author does not really collect feedback from human users but derives them from labeled data. One can imagine users can find out that returned answers are contrastive to commonsense. For instance, one can know that “Tokyo” is definitely a wrong answer to the question “What is the capital of South Africa?”. However, it is not very reasonable to assume that the users are knowledgeable enough to provide both positive and negative feedback. If so, why do they need to ask QA models? And what is the difference between collecting feedback and labeling data?
In conclusion, it would be more realistic to assume that only a small portion of the negative feedback is trustworthy, and there may be little or no reliable positive feedback. According to the experiment results, however, 20% of feedback perturbation makes the proposed method fail.
Therefore, the experiment results cannot support the claim made by authors.
2. There is a serious issue of missing related work. As mentioned above, Campos et al. (2020) has already investigated using user feedback to fine-tune deployed QA models. They also derive feedback from gold labels and conduct experiments with both in-domain and out-of-domain evaluation. The proposed methods are also similar: upweighting or downweighting the likelihood of predicted answer according to the feedback.
Moreover, Campos et al. (2020) has a more reasonable formulation, where there could be multiple feedback for a certain pair of questions and predicted answers. 3. The adopted baseline models are weak. First of all, the author does not compare to Campos et al. (2020), which also uses feedback in QA tasks. Second, they also do no comparison with other domain adaptation methods, such as those work cited in Section 8.
Line 277: “The may be attributed…” -> “This may be attributed…
3. The adopted baseline models are weak. First of all, the author does not compare to Campos et al. (2020), which also uses feedback in QA tasks. Second, they also do no comparison with other domain adaptation methods, such as those work cited in Section 8. Line 277: “The may be attributed…” -> “This may be attributed…