UPDATE: The authors addressed most of my concerns however, I believe that the first and second points are still valid and should be discussed as potential limitations (i.e., there are too many confounding variables to claim that one is investigating an impact of different training methods; and the datasets might have been - and probably were - used for RLHF).
UPDATE 2: the concerns were addressed by the authors to the extend it was possible with the current design.
- The authors claim to investigate the effect of different training methods on the processing of discourse-level information (as one of three main experiments), however, it is questionable whether what we see is the effect of different training methods, different training data, or perhaps the data used for RLHF (rather than RLHF alone, that is it is possible that the MT datasets used in this study were used to create RLHF examples). Since the authors research black box models behind an API, I do not think we can make any claims about the effect of the training method (of which we know little) on the model's performance.
- Data contamination might have influenced the evaluation - The authors employ various existing datasets. While two of these datasets do have publication date past the openAI's models' training cutoff point (made public in August 2022), this seems not to be the case for the other datasets employed in this study (including the dataset with contextual errors). It is likely that these were included in the training data of the LLMs being evaluated. Furthermore, with the RLHF models, it is also possible (and quite likely) that MT datasets published post-training were employed to create the RLHF data. For instance the WMT22 dataset was made public in August 2022, which gives companies like OpenAI plenty of time to retrieve it, reformulate it into training examples, and use for RLHF.
- The authors discuss how certain methods are significantly different from others, yet no significance testing is done to support these claims. For example, in line 486 the authors write "The conversational ability of ChatGPT and GPT-4 significantly boosts translation quality and discourse awareness" -- the difference between zh->en ChatGPT (17.4 d-BLEU; 2.8/2.9 humeval) and GPT-4 (18.8 d-BLEU; 2.6/2.8 humeval) and the scores for FeedME-2 (16.1 d-BLEU; 2.2/2.3 humeval) and PPO (17.2 d-BLEU; 2.6/2.7 humeval) is minimal and it's hard to say whether it is significant without proper testing (including checking the distribution and accounting for multiple comparisons).
- The main automatic method used throughout this paper is d-BLEU, which works best with multiple references, yet I believe only one is given. I understand that there are limited automatic options for document level evaluation where the sentences cannot be aligned. Some researchers used sliding windows for COMET, but that's not ideal (yet worth trying?). That is why the human evaluation is so important, and the one in this paper is lacking.
- Human evaluation - many important details are missing so it is hard to judge the research design (more questions below); however what bothers me most is that the authors construct an ordinal scale with a clear cutoff point between 2 and 3 (for general quality especially), yet they present only average scores. I do believe that "5: Translation passes quality control; the overall translation is excellent (...)" minus "4: Translation passes quality control; the overall translation is very good" is not the same "one" as "3: Translation passes quality control; the overall translation is ok. (...)" minus "2: Translation does not pass quality control; the overall translation is poor. (...)". It is clear that the difference between 5 and 4 is minimal, while between 3 and 2 is much bigger. Simple average is not a proper way to analyze this data (a proper analysis would include histograms of scores, possibly a stacked bar with proportion of choices, statistical testing).
- Another issue with the human evaluation is that it appears that the evaluators were asked to evaluate an entire document by assigning it one score. Note that this is a cognitively demanding and difficult task for the evaluators. The results are likely to be unreliable (please see Sheila Castilho's work on this topic). There is also no indication that the annotators were at least given some practice items.
- "Discourse-aware prompts" - I am not sure what this experiment was about. It seems that the idea was to evaluated how the availability of discourse information can improve the translation, but if that is so, then all three setups did have discourse level information (hence this evaluation is impossible). The only thing this seems to be doing is checking in which way the information should be presented (one sentence at a time in a chat, all sentences at once but clearly marked, or the entire document at once without sentence boundaries).
- The authors discuss how certain methods are significantly different from others, yet no significance testing is done to support these claims. For example, in line 486 the authors write "The conversational ability of ChatGPT and GPT-4 significantly boosts translation quality and discourse awareness" -- the difference between zh->en ChatGPT (17.4 d-BLEU; 2.8/2.9 humeval) and GPT-4 (18.8 d-BLEU; 2.6/2.8 humeval) and the scores for FeedME-2 (16.1 d-BLEU; 2.2/2.3 humeval) and PPO (17.2 d-BLEU; 2.6/2.7 humeval) is minimal and it's hard to say whether it is significant without proper testing (including checking the distribution and accounting for multiple comparisons).