1. The choice of the word-alignment baseline seems odd. The abstract claims that “Word alignment has proven to benefit many-to-many neural machine translation (NMT).” which is supported by (Lin et al., 2020). However, the method proposed by Lin et al was used as baseline. Instead, the paper compared to an older baseline proposed by (Garg et al., 2019). Besides, this baseline by Garg et al (+align) seems to contradict the claim in the abstract since it always performs worse than the baseline without word-alignment (Table 2). If for some practical reason, the baseline of (Lin et al., 2020) can’t be used, it needs to be explained clearly.
2. In Table 2, the proposed approaches only outperform the baselines in 1 setup (out of 3). In addition, there is no consistent trend in the result (i.e. it’s unclear which proposed method (+w2w) or (+FA) is better). Thus, the results presented are insufficient to prove the benefits of the proposed methods. To better justify the claims in this paper, additional experiments or more in-depth analysis seem necessary.
3. If the claim that better word-alignment improves many-to-many translation is true, why does the proposed method have no impact on the MLSC setup (Table 3)? Section 4 touches on this point but provides no explanation.
1. Please provide more details for the sentence retrieval setup (how sentences are retrieved, from what corpus, is it the same/different to the setup in (Artetxe and Schwenk, 2019) ? ).
From the paper, “We found that for en-kk, numbers of extracted word pairs per sentence by word2word and FastAlign are 1.0 and 2.2, respectively. In contrast, the numbers are 4.2 and 20.7 for improved language pairs”. Is this because word2word and FastAlign fail for some language pairs or is this because there are few alignments between these language pairs? Would a better aligner improve result further?
2. For Table 3, are the non-highlighted cells not significant or not significantly better? If it’s the latter, please also highlight cells where the proposed approaches are significantly worse. For example, from Kk to En, +FA is significantly better than mBART (14.4 vs 14.1, difference of 0.3) and thus the cell is highlighted. However, from En to Kk, the difference between +FA and mBART is -0.5 (1.3 vs 1.8) but this cell is not highlighted.
2. In Table 2, the proposed approaches only outperform the baselines in 1 setup (out of 3). In addition, there is no consistent trend in the result (i.e. it’s unclear which proposed method (+w2w) or (+FA) is better). Thus, the results presented are insufficient to prove the benefits of the proposed methods. To better justify the claims in this paper, additional experiments or more in-depth analysis seem necessary.