Based on the analyses presented in the paper, the scaling behavior of document-level machine translation models (in terms of number of parameters and number of data points) is not crucially different from the scaling behavior of sentence-level machine translation systems, which has already been studied by [Ghorbani et al. (2021)](https://arxiv.org/pdf/2109.07740.pdf).
In my opinion, the experiments (or their reporting) could be more thorough in many places to back up the claims made by the authors, e.g.
- Figure 1 indicates a causal relationship between the number of parameters and translation quality by affecting sample efficiency and optimal sequence length. While the graphic is confusing to me (e.g. it looks to me as if the number of parameters is affecting the corpus size), I don’t think that the intended claim (as reiterated at the end of the “Introduction” section) is supported by the results. None of the experiments actually show causation but rather association between the variables.
- There is no information on how the function for the optimal sequence length was estimated (Equation 1) and how reliable we expect this model to be.
- In Section 5.2, the authors should define what they mean by “error accumulation” explicitly. Is the problem that the models base the translation of later sentences made on erroneous translation of earlier sentences? If that is what the authors meant by “error accumulation”, it would need more analysis to verify such a phenomenon. Personally, I’d be interested in details like the maximum sequence length used for these experiments and the number of training examples per bin (which are not reported yet), as the observations made by the authors could be related to tendencies of the Transformer to overfit to length statistics of the training data, see [Varis and Bojar (2021)](https://aclanthology.org/2021.emnlp-main.650.pdf).
- The results presented in Section 5.3 look rather noisy to me as the accuracy on ContraPro decreases from context length 60 to 120 and from 120 to 250 while it improves considerably for larger context lengths. Intuitively, a substantial context length increase (e.g. from 60 to 250 tokens) should not hurt the accuracy. The authors do not make an attempt to explain this trend.
- It would be helpful to also provide the confidence intervals to strengthen the conclusions from the experiments with one or multiple factors.
- The authors mention in Section 5.1 that the cross entropy loss “fails to fully depict the translation quality”. I don’t think that this conclusion is valid based on the authors' experiments. The authors are measuring translation quality in terms of (d-)BLEU, i.e. in terms of a metric that has many limitations, especially for document-level MT, see [Kocmi et al. (2021)](https://aclanthology.org/2021.wmt-1.57.pdf); [Post and Junczys-Dowmunt (2023)](https://arxiv.org/pdf/2304.12959.pdf). If general translation quality is not properly reflected by the metric, then it can’t really be determined whether the loss is a good indicator of general translation quality.
Some minor comments are:
- For most of the experiments in Section 4, the factor that is held constant is not reported, e.g. for the experiment of the joint effect of maximum sequence length and data scale (in Section 4.2), I couldn’t find the model size.
- The authors mention that previous work, in particular, Beltagy et al. (2020) and Press et al. (2021) demonstrate that model performance improves with a larger context. Let me note here that their methods are different from the ones presented in this paper and thus conclusions can be different for a number of reasons.
- In my opinion, a lot of the details on training and inference configurations can be moved to the appendix.
- The abbreviation “MAC” is used in line 194 already but only explained later in the paper.
- There is no information on how the function for the optimal sequence length was estimated (Equation 1) and how reliable we expect this model to be.