# An Automatic Evaluation
of the WMT22 General Machine Translation Task

Benjamin Marie
<EMAIL_ADDRESS>
###### Abstract

This report presents an automatic evaluation of the general machine
translation task of the Seventh Conference on Machine Translation (WMT22). It
evaluates a total of 185 systems for 21 translation directions including high-
resource to low-resource language pairs and from closely related to distant
languages. This large-scale automatic evaluation highlights some of the
current limits of state-of-the-art machine translation systems. It also shows
how automatic metrics, namely chrF, BLEU, and COMET, can complement themselves
to mitigate their own limits in terms of interpretability and accuracy.

##  1 Introduction

In its 2022 edition, the Conference on Machine Translation
(WMT22)111https://www.statmt.org/wmt22/ organized a new “general” machine
translation shared task. The participants in this task were asked to prepare
machine translation (MT) systems for translating texts from potentially any
domains and most likely “news, social, conversational, and ecommerce.”

For this task, WMT22 proposed a large variety of language pairs:
Czech$\leftrightarrow$English (cs-en), Czech$\leftrightarrow$Ukrainian (cs-
uk), German$\leftrightarrow$English (de-en), French$\leftrightarrow$German
(fr-de), English$\rightarrow$Croatian (en-hr),
English$\leftrightarrow$Japanese (en-ja), English$\leftrightarrow$Livonian
(en-liv), English$\leftrightarrow$Russian (en-ru),
Russian$\leftrightarrow$Yakut (ru-sah), English$\leftrightarrow$Ukrainian (en-
uk), and English$\leftrightarrow$Chinese (en-zh). The organizers classified
all these language pairs as in Table 1 according to the relatedness of the
languages and the quantity of resources available for training an MT system.
In total, there are 11 language pairs and 21 translation directions.

In this paper, we present an automatic evaluation222This is not an official
paper of WMT22. WMT22 organizers will perform a human evaluation. of the 185
systems submitted333It includes the “Online-*” systems added by the
organizers. to the general machine translation task. We used the translations
officially released by WMT22.444https://github.com/wmt-conference/wmt22-news-
systems We analyze the results with different metrics and methods. Our main
observations are as follows:

  * •

Low-resource distant language pairs remain extremely challenging. The best
systems for en$\leftrightarrow$liv and ru$\leftrightarrow$sah are very far
from reaching the reference. ja$\leftrightarrow$en and en$\rightarrow$uk also
remain challenging despite being medium-resource. Among the high-resource
language pairs, the systems built for en$\rightarrow$ru struggled the most to
approximate the reference translation.

  * •

A BLEU score difference higher than 0.9 is always statistically significant in
this task.

  * •

A naive combination of all the evaluated systems using minimum Bayes risk
(MBR) decoding with COMET can improve the BLEU, chrF and COMET scores.

  * •

Only a few systems seem to have exploited document-level features or a context
larger than one segment.

  * •

Absolute COMET scores are meaningless and must be completed with another
metric score to get an estimate of the system’s translation quality.

| High-resource | Medium-resource | Low-resource
---|---|---|---
Closely related |  | cs-uk |
Same family | cs-en, de-en, en-ru | de-fr, en-uk | en-hr
Distant | en-zh | en-ja | en-liv, ru-sah
Table 1: Language pairs in terms of resource available for training and
language similarity as defined by WMT22.

The paper is organized as follows. In each section, we present an overview of
the main findings with the numerous associated tables given in the appendix.
In Section 2, we score and rank the systems with three different metrics.
These rankings are completed by a statistical significance testing on all
possible pairs of systems in Section 3. Section 4 presents the results of a
combination of all the submissions using MBR and an oracle guided by COMET.
Section 5 presents various statistics of the submitted systems to better
highlight how they stand out. For a more in-depth evaluation with COMET, we
evaluate all the systems with 5 different COMET models in Section 6. Finally,
Section 7 concludes this evaluation.

##  2 Ranking with Metrics

For this evaluation, we will rely on three different automatic metrics:

  * •

chrF (Popović, 2015): A tokenization independent metric operating at
character-level with a higher correlation with human judgments than BLEU.

  * •

BLEU (Papineni et al., 2002): The standard BLEU.

  * •

COMET (Rei et al., 2020): A state-of-the-art metric based on a pre-trained
language model. We used the default model “wmt20-comet-da.”

Note that chrF and BLEU are merely used for diagnostic purposes and to answer
the question: How far are we from reaching particular reference translations?
On the other hand, we used COMET mainly to produce more accurate rankings that
would better correlate with the human evaluation.

chrF and BLEU scores are computed with SacreBLEU (Post,
2018).555https://github.com/mjpost/sacrebleu If there are two reference
translations available, both are used to compute the chrF and BLEU
scores,666We did not use the reference “stud” created by students. while we
only used the first available reference to compute COMET scores. The systems
are ranked given their scores but a rank is assigned only to the systems that
have been declared “constrained” by their authors, i.e., systems that only
used the data provided by the organizers. The rankings and scores are given in
Section A, from Table 2 to 22. In these tables, systems with a rank “n/a” are
systems that are not constrained.

Having two references, we obtain absolute BLEU scores rarely seen in the
machine translation research literature with, for instance, 60.9 BLEU points
for JDExploreAcademy for cs$\rightarrow$en. Even higher BLEU scores are
observed for en$\rightarrow$zh due to the use of smaller tokens that makes the
4-gram matching a much easier task. Such high scores may be misleading about
the translation quality. Absolute BLEU scores do not inform us on the
translation quality itself, but they do inform us that these systems produce
many 4-grams that are in the reference translations.

While chrF and BLEU directly indicate how well the translation matches the
references with a score between 0 and 100 points, COMET scores are not bounded
between 0 and 100 making the interpretation of absolute scores extremely
challenging. For instance, at the extremes, AMU obtains 104.9 COMET points for
uk$\rightarrow$cs and AIST obtains -152.7 COMET points for liv$\rightarrow$en.
BLEU and chrF absolute scores can be used for diagnostic purposes and answer
basic questions: How close are we from the reference with a given
tokenization? Is the system likely generating text in the target languages?
etc. COMET cannot, but is much more reliable for ranking systems as
demonstrated in previous work (Rei et al., 2020).

BLEU and chrF yields the same rankings for only 5 translation directions:
uk$\rightarrow$cs, de$\rightarrow$fr, sah$\rightarrow$ru, and
ru$\rightarrow$sah. Nonetheless, BLEU and chrF yields ranking that are very
similar and find the same best systems for 20 among the 21 translation
directions. On the other hand, for 11 among the 21 language pairs, COMET finds
a best system that is not among the best systems found by BLEU and chrF. For
some system pairs, the difference between BLEU and COMET can be very
significant (more on this in Section 3). For instance for cs$\rightarrow$uk,
in terms of score differences, the difference between Online-B and AMU is of
3.6 BLEU points and -5.1 COMET points.

Surprisingly, for some translation directions, constrained systems outperform
systems that are not constrained. According to COMET, this is the case for
cs$\rightarrow$uk, uk$\rightarrow$cs, de$\rightarrow$en, ja$\rightarrow$en,
and en$\rightarrow$ja.

For some other directions, online systems seem to be better by a large margin.
For instance, for de$\rightarrow$fr, Online-W is better than the best
constrained system by 18.3 BLEU points.

##  3 Statistical Significance Testing

We would like to measure how reliable is the conclusion that a system is
better than another one according to some metric. In other words, we would
like to test whether the difference between systems’ metric scores is
statistically significant. There are a several tools and techniques to perform
statistical significance testing. For this evaluation, we chose the most
commonly used: paired bootstrap resampling (Koehn, 2004). We used the
implementation in SacreBLEU with its default parameters for chrF and BLEU. For
COMET, we used the default parameters of “comet-compare.” Results are reported
in Section B, from Table 23 to 85. In these tables, the difference between the
metric scores is reported for each system pair. The background color is darker
for more significant differences (lower p-value) and the score difference is
underline if the p-value is below 0.05.

A first interesting observation we have is that a difference in BLEU higher
than 0.9 points (cs$\rightarrow$uk) is always significant (p-value $<0.05$).
In chrF, the largest difference that is not significant is 0.6 points
(en$\rightarrow$zh), while it reaches 2.6 points (liv$\rightarrow$en) for
COMET.777Note that this would highly vary depending on the model used with
COMET.

The significance of the difference between systems can be very different
depending on the metric. Overall, there is no clear pattern. For some
translation directions, one metric will find a cluster of similar systems
while another metric will find that they are all significantly different. This
is well-illustrated with en$\rightarrow$ru where BLEU finds a cluster of
similar systems while COMET finds that they are almost all significantly
different.

The three metrics only agree on a system that is significantly better than all
the others for a few translation directions: cs$\rightarrow$en (Online-W),
fr$\rightarrow$de (Online-W), en$\rightarrow$liv (TAL-SJTU),
sah$\rightarrow$ru (Online-G), and en$\rightarrow$uk (Online-B).

It will be interesting to observe how well the clusters of systems found with
statistical significance testing match the clusters found with WMT22 human
evaluation.

##  4 Naive MBR and Oracle Combinations

As shown by the tables presented in Section 3, most of the evaluated
translations are significantly different according to COMET. One way to
exploit this diversity is to combine the translations to generate a better
translation. There are many methods to combine machine translation outputs. In
this section, we only present two different methods for combination:

  * •

Minimum Bayes Risk (MBR): Naively use all the evaluated translations and
performs MBR decoding using COMET.

  * •

Oracle: Select the best segments among all the evaluated translations
according to their COMET score. This is an oracle experiment since we use the
COMET scores computed with a reference translation. This experiment is mainly
to get an idea of what would be the best translation obtained from the
combination of all the systems.

The results of these combinations are reported in Section C, from Table 86 to
96. These MBR and oracle combinations rely on COMET but we also report on the
chrF and BLEU scores of the resulting translations. “Baseline” denotes the
score of the best single system for each metric.

As we observe in the tables, the MBR combination improves on all the three
metrics for cs$\rightarrow$uk, uk$\rightarrow$cs (with 1.9 BLEU points of
improvement), de$\rightarrow$en, en$\rightarrow$de, and liv$\rightarrow$en.

However, for some translation directions, the MBR combination largely fails in
improving the metric scores and may even decrease the COMET scores compared to
the best single system. For instance, the MBR combination leads to lower COMET
scores for cs$\rightarrow$en, de$\rightarrow$fr, ru$\rightarrow$sah,
sah$\rightarrow$ru, and uk$\rightarrow$en. This is presumably due to the
presence of several translations of a very different and lower quality
compared to the best single translation.

On the other hand, the oracle combination improves the scores on all metrics
for all translation directions. This shows that BLEU and chrF tend to agree,
mostly, that a segment with a higher COMET score is better.

##  5 Sentence Matching and Normalization

In this section, we look at various statistics to try to identify some salient
characteristics of the evaluated systems. Statistics are presented in Section
E from Table 108 to 118.

### Exact Match

We counted how many segments in the system translation are identical to their
corresponding segments in the reference translation.

### Self Mismatch

We counted the number of non-unique source segments translated differently by
the system and the reference translation. It leads to interesting
observations. Most systems always identically translate the same source
segments, but there are some notable exceptions. For instance, Online-W
generated different translations for the same segment for almost all its
translation directions. For 47 of the 122 non-unique source segments of the
en$\rightarrow$ja, Online-W generates a different translation. Among the other
submissions to this translation direction, only LanguageX (1) and NAIST-NICT-
TIT (22) generated different translations for the same source segments. In
contrast, the human translator(s?) produced 63 different translations for the
122 non-unique segments. This is the highest number among all the translation
directions. The case of Japanese is particularly interesting here because in
this language the context can frequently and radically lead to different
translations for the same text. Getting different translations for the same
segment is a clue that the system either takes into account document-level
information and/or use sampling strategies with some randomness at inference
time. CUNI-DocTransformer is another interesting example for cs$\rightarrow$en
since it translates differently as many non-unique source segments (7) as the
reference translation, probably by exploiting document-level features as
suggested by the name of the system. For 10 translation directions, the
reference translation translates all the non-unique segments identically.

### Sentence Normalization

We also normalized the translation outputs to observe how BLEU and COMET are sensitive to changes in the punctuation marks and encoding issues. It can also highlight whether a system relied on some special post-processing to increase the metric scores. For normalization, we used the following sequence of Moses (Koehn et al., 2007) scripts:888https://github.com/moses-smt/mosesdecoder/tree/master/scripts tokenizer/replace-unicode-punctuation.perl | tokenizer/normalize-punctuation.perl -l <target_language> | /tokenizer/remove-non-printing-char.perl. As expected, COMET is almost insensitive to this normalization.999There are exception for the Japanese and Chinese languages for which Moses scripts were not designed to deal with. The scores of the normalized translations should be ignored for these languages. On the other hand, this normalization has a stronger impact on the BLEU scores, but it can greatly vary from one system to another. For instance for en$\rightarrow$cs, it has no effect on JDExploreAcademy while the score of Online-Y drops by 1.4 BLEU points. For de$\rightarrow$fr, the normalization increases the BLEU score of Online-A by 4.9 point and becomes better than Online-W for which the normalization has no effect on BLEU. Nonetheless, Online-W remains around 10 COMET points better than Online-A.

##  6 Evaluation With Different COMET Models

In Section 2, we observed that COMET scores can be very high (above 100) or
very low (below -100) making their interpretation very difficult. In this
section, in an attempt to obtain scores easier to interpret we reevaluate all
the translations with 5 different COMET models:

  * •

wmt20-comet-da: The same model used in Section 2 trained with direct
assessment (DA).

  * •

wmt21-comet-da: A more recent model trained with DA on more data.

  * •

wmt21-comet-mqm: A model trained with MQM human judgments.

  * •

wmt21-comet-qe-mqm: A reference-less model trained with MQM human judgments.

  * •

wmt21-comet-qe-da: A reference-less model trained with DA.

The results using these 5 models are presented in Section D from Table 97 to
107.

We can observe that wmt20-comet-da scores are actually quite different from
all the other models. While the maximum score obtained by a system with
wmt20-comet-da is 104.9 (uk$\rightarrow$cs), the scores obtained with the
other 4 models never exceed 15.9 for all translation directions. The absolute
scores remain difficult to interpret with the 4 new models. More particularly,
with wmt21-comet-da for ja$\rightarrow$en, the best system is scored at 1.1
while for zh$\rightarrow$en, wmt21-comet-da scores are negative for all the
systems. We can conclude that absolute COMET scores are not informative
whatever model we use. Negative COMET scores can be assigned to an excellent
machine translation.

Nonetheless, the rankings obtained by all 5 models are very similar albeit the
reference-less models that tend to produce more different rankings.

##  7 Conclusion

In this paper, we presented a large-scale evaluation of 185 systems for the 21
translation directions of 11 language pairs. The variety of scenarios, going
from high-resource to low-resource language pairs, for closely related or
distant language pairs, draws an interesting picture of the state of machine
translation. According to three different automatic metrics, significant
progress remains to be done to get close to a reference translation,
especially for the most difficult scenario involving distant low-resource
language pairs.

For only 5 translation directions, WMT22 systems submitted are significantly
better than online systems according to COMET. This suggests that the online
systems for these translation directions used less training data.

We highlight that while COMET is a state-of-the-art metric for ranking
systems, its score are uninformative, or misleading for the readers that are
unaware of COMET, as even systems that looks reasonably good may only achieve
a negative COMET score. We believe that this peculiarity101010The large
majority of NLP evaluation metrics only yield positive scores. is a major
obstacle for its wide adoption in MT research.

We also highlighted that most submitted systems still do not exploit document-
level features or context larger than one segment.

It will be interesting to observe how the human evaluation conducted by WMT22
will align with these findings.

## Acknowledgments

We would like to thank the WMT organizers for releasing the translations and
Tom Kocmi for providing preliminary results as well as insightful comments and
suggestions on the first draft of this report.

## References

  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.

##  Appendix A Ranking Tables

From Table 2 to 22.

System | Rank | chrF
---|---|---
Online-W | n/a | 79.7
JDExploreAcademy | 1 | 75.0
Lan-Bridge | n/a | 74.6
Online-B | n/a | 74.5
Online-A | n/a | 74.1
CUNI-DocTransformer | 2 | 73.0
CUNI-Transformer | 3 | 72.4
Online-G | n/a | 71.1
SHOPLINE-PL | 4 | 70.1
Online-Y | n/a | 68.8
ALMAnaCH-Inria | 5 | 58.3

System | Rank | BLEU
---|---|---
Online-W | n/a | 68.7
JDExploreAcademy | 1 | 60.9
Lan-Bridge | n/a | 60.8
Online-B | n/a | 60.6
Online-A | n/a | 59.4
CUNI-DocTransformer | 2 | 57.9
CUNI-Transformer | 3 | 57.6
Online-G | n/a | 54.3
SHOPLINE-PL | 4 | 53.5
Online-Y | n/a | 51.0
ALMAnaCH-Inria | 5 | 36.8

System | Rank | COMET
---|---|---
Online-W | n/a | 77.5
JDExploreAcademy | 1 | 74.8
Online-B | n/a | 71.9
Lan-Bridge | n/a | 71.8
CUNI-DocTransformer | 2 | 70.6
Online-A | n/a | 69.8
CUNI-Transformer | 3 | 69.2
Online-G | n/a | 63.0
SHOPLINE-PL | 4 | 61.2
Online-Y | n/a | 58.7
ALMAnaCH-Inria | 5 | 19.3

Table 2:  Scores for the cs$\rightarrow$en translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 71.3
Lan-Bridge | 1 | 70.4
Online-W | n/a | 70.4
Online-A | n/a | 67.5
JDExploreAcademy | 2 | 67.2
CUNI-DocTransformer | 3 | 66.0
CUNI-Bergamot | 4 | 65.1
Online-G | n/a | 64.6
Online-Y | n/a | 64.5
CUNI-Transformer | 5 | 64.2

System | Rank | BLEU
---|---|---
Online-B | n/a | 56.5
Lan-Bridge | 1 | 53.4
Online-W | n/a | 53.1
JDExploreAcademy | 2 | 49.4
Online-A | n/a | 49.1
CUNI-DocTransformer | 3 | 47.8
CUNI-Bergamot | 4 | 46.2
CUNI-Transformer | 5 | 45.6
Online-G | n/a | 44.8
Online-Y | n/a | 44.7

System | Rank | COMET
---|---|---
Online-W | n/a | 97.9
Online-B | n/a | 97.5
CUNI-Bergamot | 1 | 96.1
JDExploreAcademy | 2 | 95.3
Lan-Bridge | 3 | 94.7
Online-A | n/a | 92.2
CUNI-DocTransformer | 4 | 91.7
CUNI-Transformer | 5 | 86.7
Online-Y | n/a | 83.7
Online-G | n/a | 82.3

Table 3: Scores for the en$\rightarrow$cs translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 64.0
Lan-Bridge | n/a | 64.0
HuaweiTSC | 1 | 62.6
Online-A | n/a | 62.2
CUNI-JL-JH | 2 | 61.6
CUNI-Transformer | 3 | 61.6
AMU | 4 | 61.5
CharlesTranslator | n/a | 61.5
Online-G | n/a | 60.8
Online-Y | n/a | 59.6
ALMAnaCH-Inria | 5 | 54.5

System | Rank | BLEU
---|---|---
Online-B | n/a | 38.3
Lan-Bridge | n/a | 38.3
HuaweiTSC | 1 | 36.0
Online-A | n/a | 35.9
CUNI-Transformer | 2 | 35.0
CUNI-JL-JH | 3 | 34.8
AMU | 4 | 34.7
CharlesTranslator | n/a | 34.3
Online-G | n/a | 32.5
Online-Y | n/a | 32.1
ALMAnaCH-Inria | 5 | 26.8

System | Rank | COMET
---|---|---
AMU | 1 | 99.4
Online-B | n/a | 94.3
Lan-Bridge | n/a | 91.8
CharlesTranslator | n/a | 90.8
HuaweiTSC | 2 | 90.7
CUNI-JL-JH | 3 | 90.1
Online-G | n/a | 88.4
Online-A | n/a | 87.9
CUNI-Transformer | 4 | 87.4
Online-Y | n/a | 78.4
ALMAnaCH-Inria | 5 | 61.4

Table 4: Scores for the cs$\rightarrow$uk translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
AMU | 1 | 60.7
Lan-Bridge | n/a | 60.4
Online-B | n/a | 60.3
HuaweiTSC | 2 | 59.6
CharlesTranslator | n/a | 59.0
CUNI-Transformer | 3 | 59.0
CUNI-JL-JH | 4 | 58.7
Online-A | n/a | 57.5
Online-G | n/a | 56.3
Online-Y | n/a | 55.3
ALMAnaCH-Inria | 5 | 50.7

System | Rank | BLEU
---|---|---
AMU | 1 | 37.0
Lan-Bridge | n/a | 36.5
Online-B | n/a | 36.4
HuaweiTSC | 2 | 36.0
CharlesTranslator | n/a | 35.9
CUNI-Transformer | 3 | 35.8
CUNI-JL-JH | 4 | 35.1
Online-A | n/a | 33.3
Online-G | n/a | 31.5
Online-Y | n/a | 29.6
ALMAnaCH-Inria | 5 | 25.3

System | Rank | COMET
---|---|---
AMU | 1 | 104.9
Online-B | n/a | 96.5
Lan-Bridge | n/a | 94.6
HuaweiTSC | 2 | 91.5
CharlesTranslator | n/a | 90.2
CUNI-JL-JH | 3 | 89.1
CUNI-Transformer | 4 | 88.5
Online-A | n/a | 85.4
Online-G | n/a | 84.2
Online-Y | n/a | 78.7
ALMAnaCH-Inria | 5 | 62.5

Table 5: Scores for the uk$\rightarrow$cs translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-A | n/a | 66.4
Online-G | n/a | 66.3
Lan-Bridge | n/a | 66.2
JDExploreAcademy | 1 | 65.8
Online-B | n/a | 65.7
Online-Y | n/a | 65.7
PROMT | n/a | 65.6
Online-W | n/a | 65.4
LT22 | 2 | 58.4

System | Rank | BLEU
---|---|---
Online-A | n/a | 50.2
Lan-Bridge | n/a | 50.1
Online-B | n/a | 49.7
Online-G | n/a | 49.7
JDExploreAcademy | 1 | 49.3
Online-Y | n/a | 49.3
PROMT | n/a | 49.2
Online-W | n/a | 48.8
LT22 | 2 | 40.3

System | Rank | COMET
---|---|---
JDExploreAcademy | 1 | 58.0
Online-B | n/a | 57.0
Lan-Bridge | n/a | 56.6
Online-G | n/a | 55.3
Online-Y | n/a | 54.7
Online-A | n/a | 54.5
Online-W | n/a | 54.4
PROMT | n/a | 51.8
LT22 | 2 | 25.6

Table 6: Scores for the de$\rightarrow$en translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 69.7
JDExploreAcademy | 1 | 69.2
Lan-Bridge | n/a | 69.1
Online-A | n/a | 68.8
Online-W | n/a | 68.7
Online-Y | n/a | 68.7
Online-G | n/a | 68.4
PROMT | n/a | 67.5
OpenNMT | n/a | 66.7

System | Rank | BLEU
---|---|---
Online-B | n/a | 52.3
JDExploreAcademy | 1 | 51.6
Online-Y | n/a | 50.5
Online-A | n/a | 50.1
Online-G | n/a | 49.6
Lan-Bridge | n/a | 49.4
PROMT | n/a | 49.0
Online-W | n/a | 48.9
OpenNMT | n/a | 48.4

System | Rank | COMET
---|---|---
Online-W | n/a | 65.6
JDExploreAcademy | 1 | 63.3
Online-B | n/a | 62.3
Online-Y | n/a | 61.1
Online-A | n/a | 60.6
Online-G | n/a | 60.2
Lan-Bridge | n/a | 58.8
OpenNMT | n/a | 57.3
PROMT | n/a | 55.9

Table 7: Scores for the en$\rightarrow$de translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 74.6
Online-Y | n/a | 66.8
Online-W | n/a | 65.5
Online-A | n/a | 64.5
Online-G | n/a | 62.7
LT22 | 1 | 54.4

System | Rank | BLEU
---|---|---
Online-B | n/a | 58.4
Online-Y | n/a | 46.2
Online-W | n/a | 43.6
Online-A | n/a | 41.3
Online-G | n/a | 39.0
LT22 | 1 | 28.3

System | Rank | COMET
---|---|---
Online-B | n/a | 70.5
Online-W | n/a | 63.7
Online-Y | n/a | 57.8
Online-A | n/a | 52.3
Online-G | n/a | 44.9
LT22 | 1 | 10.5

Table 8: Scores for the de$\rightarrow$fr translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-W | n/a | 81.2
Online-B | n/a | 68.7
eTranslation | 1 | 68.4
Online-Y | n/a | 67.5
Online-A | n/a | 67.2
Online-G | n/a | 66.0
Lan-Bridge | n/a | 65.0

System | Rank | BLEU
---|---|---
Online-W | n/a | 64.8
Online-B | n/a | 46.6
eTranslation | 1 | 46.5
Online-Y | n/a | 45.0
Online-A | n/a | 44.4
Lan-Bridge | n/a | 41.8
Online-G | n/a | 41.1

System | Rank | COMET
---|---|---
Online-W | n/a | 77.9
Online-B | n/a | 63.8
Online-Y | n/a | 61.6
Online-A | n/a | 59.3
eTranslation | 1 | 55.5
Lan-Bridge | n/a | 51.2
Online-G | n/a | 48.3

Table 9: Scores for the fr$\rightarrow$de translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 58.5
Lan-Bridge | n/a | 58.5
HuaweiTSC | 1 | 56.8
Online-A | n/a | 56.5
SRPOL | 2 | 56.3
NiuTrans | 3 | 56.3
Online-Y | n/a | 54.3
Online-G | n/a | 53.2

System | Rank | BLEU
---|---|---
Online-B | n/a | 31.5
Lan-Bridge | n/a | 31.5
HuaweiTSC | 1 | 29.9
NiuTrans | 2 | 29.3
Online-A | n/a | 29.1
SRPOL | 3 | 29.1
Online-Y | n/a | 26.6
Online-G | n/a | 25.7

System | Rank | COMET
---|---|---
Online-B | n/a | 80.4
Lan-Bridge | n/a | 79.6
Online-A | n/a | 69.5
SRPOL | 1 | 69.5
HuaweiTSC | 2 | 67.7
NiuTrans | 3 | 65.6
Online-G | n/a | 64.3
Online-Y | n/a | 56.8

Table 10: Scores for the en$\rightarrow$hr translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-W | n/a | 51.7
NT5 | 1 | 51.3
JDExploreAcademy | 2 | 50.1
Online-B | n/a | 49.9
DLUT | 3 | 49.8
LanguageX | n/a | 49.1
Lan-Bridge | n/a | 48.7
Online-A | n/a | 48.4
NAIST-NICT-TIT | 4 | 48.3
Online-Y | n/a | 48.2
AISP-SJTU | 5 | 48.0
Online-G | n/a | 45.7
KYB | 6 | 43.4
AIST | n/a | 11.4

System | Rank | BLEU
---|---|---
Online-W | n/a | 27.8
NT5 | 1 | 26.6
JDExploreAcademy | 2 | 25.6
DLUT | 3 | 24.8
Online-B | n/a | 24.7
Online-A | n/a | 22.8
Lan-Bridge | n/a | 22.8
NAIST-NICT-TIT | 4 | 22.7
LanguageX | n/a | 22.4
AISP-SJTU | 5 | 22.0
Online-Y | n/a | 21.5
Online-G | n/a | 19.7
KYB | 6 | 18.1
AIST | n/a | 0.1

System | Rank | COMET
---|---|---
NT5 | 1 | 42.1
Online-W | n/a | 41.2
JDExploreAcademy | 2 | 40.6
Online-B | n/a | 39.7
DLUT | 3 | 37.3
NAIST-NICT-TIT | 4 | 33.4
LanguageX | n/a | 33.0
Online-A | n/a | 32.9
Online-Y | n/a | 32.4
Lan-Bridge | n/a | 32.0
AISP-SJTU | 5 | 30.2
Online-G | n/a | 22.3
KYB | 6 | 17.4
AIST | n/a | -152.7

Table 11: Scores for the ja$\rightarrow$en translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
NT5 | 1 | 36.8
JDExploreAcademy | 2 | 36.1
LanguageX | n/a | 36.1
DLUT | 3 | 36.1
Online-B | n/a | 35.5
Online-W | n/a | 35.2
Online-Y | n/a | 34.4
Lan-Bridge | n/a | 34.1
Online-A | n/a | 34.1
AISP-SJTU | 4 | 33.9
NAIST-NICT-TIT | 5 | 33.8
KYB | 6 | 28.6
Online-G | n/a | 28.0

System | Rank | BLEU
---|---|---
NT5 | 1 | 27.6
LanguageX | n/a | 27.2
DLUT | 2 | 27.2
Online-W | n/a | 27.1
JDExploreAcademy | 3 | 27.0
Online-B | n/a | 26.5
Online-Y | n/a | 25.2
AISP-SJTU | 4 | 25.0
Lan-Bridge | n/a | 24.8
NAIST-NICT-TIT | 5 | 24.6
Online-A | n/a | 24.0
KYB | 6 | 18.9
Online-G | n/a | 18.6

System | Rank | COMET
---|---|---
JDExploreAcademy | 1 | 65.2
NT5 | 2 | 64.1
LanguageX | n/a | 62.1
Online-B | n/a | 60.8
DLUT | 3 | 60.5
Online-W | n/a | 59.8
Online-Y | n/a | 56.9
Lan-Bridge | n/a | 56.6
Online-A | n/a | 53.7
NAIST-NICT-TIT | 4 | 53.4
AISP-SJTU | 5 | 52.4
KYB | 6 | 31.8
Online-G | n/a | 25.0

Table 12: Scores for the en$\rightarrow$ja translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:ja-mecab-0.996-IPA|smooth:exp|version:2.1.0),
COMET (wmt20-comet-da).

System | Rank | chrF
---|---|---
TartuNLP | n/a | 53.5
TAL-SJTU | n/a | 53.2
HuaweiTSC | n/a | 48.4
Liv4ever | n/a | 46.7
NiuTrans | 1 | 35.6

System | Rank | BLEU
---|---|---
TAL-SJTU | n/a | 30.4
TartuNLP | n/a | 29.9
HuaweiTSC | n/a | 23.4
Liv4ever | n/a | 23.3
NiuTrans | 1 | 13.0

System | Rank | COMET
---|---|---
TartuNLP | n/a | -5.7
TAL-SJTU | n/a | -8.3
HuaweiTSC | n/a | -27.2
Liv4ever | n/a | -44.0
NiuTrans | 1 | -88.3

Table 13: Scores for the liv$\rightarrow$en translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
TAL-SJTU | n/a | 43.8
Liv4ever | n/a | 39.6
TartuNLP | n/a | 39.2
HuaweiTSC | n/a | 37.7
NiuTrans | 1 | 30.5

System | Rank | BLEU
---|---|---
TAL-SJTU | n/a | 17.0
TartuNLP | n/a | 15.0
Liv4ever | n/a | 14.7
HuaweiTSC | n/a | 12.8
NiuTrans | 1 | 12.3

System | Rank | COMET
---|---|---
TAL-SJTU | n/a | -29.5
TartuNLP | n/a | -36.8
HuaweiTSC | n/a | -38.8
Liv4ever | n/a | -39.4
NiuTrans | 1 | -81.9

Table 14: Scores for the en$\rightarrow$liv translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-G | n/a | 70.0
JDExploreAcademy | 1 | 68.9
Lan-Bridge | n/a | 68.5
HuaweiTSC | 2 | 68.5
Online-B | n/a | 68.3
Online-A | n/a | 68.3
Online-Y | n/a | 68.2
SRPOL | 3 | 67.2
Online-W | n/a | 66.3
ALMAnaCH-Inria | 4 | 57.9

System | Rank | BLEU
---|---|---
Online-G | n/a | 46.7
Lan-Bridge | n/a | 45.2
JDExploreAcademy | 1 | 45.1
HuaweiTSC | 2 | 45.1
Online-B | n/a | 45.0
Online-A | n/a | 43.9
Online-Y | n/a | 43.8
SRPOL | 3 | 43.6
Online-W | n/a | 42.6
ALMAnaCH-Inria | 4 | 30.3

System | Rank | COMET
---|---|---
Online-G | n/a | 65.2
JDExploreAcademy | 1 | 64.9
Online-Y | n/a | 64.1
Lan-Bridge | n/a | 63.1
Online-B | n/a | 63.1
Online-A | n/a | 62.2
Online-W | n/a | 61.6
HuaweiTSC | 2 | 60.9
SRPOL | 3 | 59.5
ALMAnaCH-Inria | 4 | 26.9

Table 15: Scores for the ru$\rightarrow$en translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 59.7
Online-G | n/a | 59.5
Lan-Bridge | n/a | 59.0
JDExploreAcademy | 1 | 58.4
Online-W | n/a | 58.3
Online-Y | n/a | 58.3
Online-A | n/a | 58.1
SRPOL | 2 | 56.4
PROMT | n/a | 56.1
HuaweiTSC | 3 | 56.1
eTranslation | 4 | 55.8

System | Rank | BLEU
---|---|---
Online-B | n/a | 34.9
Online-Y | n/a | 33.2
Online-A | n/a | 33.1
Online-G | n/a | 32.8
JDExploreAcademy | 1 | 32.7
Lan-Bridge | n/a | 32.6
Online-W | n/a | 32.4
HuaweiTSC | 2 | 30.8
PROMT | n/a | 30.6
SRPOL | 3 | 30.4
eTranslation | 4 | 29.8

System | Rank | COMET
---|---|---
Online-W | n/a | 75.1
Online-G | n/a | 73.2
Online-B | n/a | 72.9
Online-Y | n/a | 69.8
JDExploreAcademy | 1 | 69.7
Lan-Bridge | n/a | 67.4
Online-A | n/a | 67.4
PROMT | n/a | 60.4
SRPOL | 2 | 59.8
HuaweiTSC | 3 | 59.2
eTranslation | 4 | 58.0

Table 16: Scores for the en$\rightarrow$ru translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-G | n/a | 55.5
Lan-Bridge | 1 | 28.3

System | Rank | BLEU
---|---|---
Online-G | n/a | 29.6
Lan-Bridge | 1 | 7.1

System | Rank | COMET
---|---|---
Online-G | n/a | 31.1
Lan-Bridge | 1 | -75.9

Table 17: Scores for the sah$\rightarrow$ru translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Lan-Bridge | 1 | 56.6
Online-G | n/a | 47.0

System | Rank | BLEU
---|---|---
Lan-Bridge | 1 | 15.3
Online-G | n/a | 14.7

System | Rank | COMET
---|---|---
Online-G | n/a | -17.1
Lan-Bridge | 1 | -48.0

Table 18: Scores for the ru$\rightarrow$sah translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Lan-Bridge | n/a | 67.3
Online-B | n/a | 67.2
Online-G | n/a | 66.0
Online-A | n/a | 65.2
PROMT | n/a | 64.7
Online-Y | n/a | 64.6
ARC-NKUA | n/a | 64.6
HuaweiTSC | 1 | 63.9
ALMAnaCH-Inria | 2 | 55.6

System | Rank | BLEU
---|---|---
Lan-Bridge | n/a | 44.6
Online-B | n/a | 44.4
Online-G | n/a | 43.2
Online-A | n/a | 42.3
PROMT | n/a | 42.1
ARC-NKUA | n/a | 41.9
Online-Y | n/a | 41.8
HuaweiTSC | 1 | 41.6
ALMAnaCH-Inria | 2 | 30.0

System | Rank | COMET
---|---|---
Online-B | n/a | 62.6
Lan-Bridge | n/a | 62.4
Online-G | n/a | 57.5
Online-A | n/a | 52.2
HuaweiTSC | 1 | 50.1
Online-Y | n/a | 49.8
PROMT | n/a | 49.6
ARC-NKUA | n/a | 49.6
ALMAnaCH-Inria | 2 | 21.8

Table 19: Scores for the uk$\rightarrow$en translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 59.3
Lan-Bridge | n/a | 58.8
Online-G | n/a | 57.2
Online-A | n/a | 56.0
Online-Y | n/a | 54.9
eTranslation | 1 | 54.8
HuaweiTSC | 2 | 54.8
ARC-NKUA | n/a | 54.0

System | Rank | BLEU
---|---|---
Online-B | n/a | 32.5
Lan-Bridge | n/a | 29.5
Online-A | n/a | 28.0
Online-G | n/a | 27.2
Online-Y | n/a | 26.9
HuaweiTSC | 1 | 26.5
eTranslation | 2 | 26.2
ARC-NKUA | n/a | 25.2

System | Rank | COMET
---|---|---
Online-B | n/a | 73.3
Online-G | n/a | 69.9
Lan-Bridge | n/a | 65.8
Online-A | n/a | 60.9
eTranslation | 1 | 54.6
HuaweiTSC | 2 | 54.4
Online-Y | n/a | 52.0
ARC-NKUA | n/a | 49.2

Table 20: Scores for the en$\rightarrow$uk translation task: chrF
(nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
JDExploreAcademy | 1 | 62.4
LanguageX | n/a | 61.7
Online-G | n/a | 60.9
AISP-SJTU | 2 | 60.5
HuaweiTSC | n/a | 59.6
Online-B | n/a | 59.4
Lan-Bridge | n/a | 59.1
Online-Y | n/a | 58.6
Online-A | n/a | 58.4
NiuTrans | 3 | 57.2
DLUT | 4 | 56.3
Online-W | n/a | 55.6

System | Rank | BLEU
---|---|---
JDExploreAcademy | 1 | 37.9
LanguageX | n/a | 36.4
AISP-SJTU | 2 | 34.4
HuaweiTSC | n/a | 34.3
Online-G | n/a | 34.1
Online-B | n/a | 33.3
Lan-Bridge | n/a | 32.9
Online-A | n/a | 31.6
Online-Y | n/a | 31.1
NiuTrans | 3 | 30.5
DLUT | 4 | 29.0
Online-W | n/a | 28.1

System | Rank | COMET
---|---|---
Online-G | n/a | 45.7
JDExploreAcademy | 1 | 45.2
LanguageX | n/a | 45.0
Lan-Bridge | n/a | 43.1
HuaweiTSC | n/a | 42.9
Online-B | n/a | 42.2
AISP-SJTU | 2 | 41.7
Online-Y | n/a | 40.9
Online-A | n/a | 35.3
Online-W | n/a | 31.7
NiuTrans | 3 | 31.4
DLUT | 4 | 30.7

Table 21: Scores for the zh$\rightarrow$en translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

System | Rank | chrF
---|---|---
Online-B | n/a | 70.3
HuaweiTSC | n/a | 61.0
LanguageX | n/a | 59.5
Online-A | n/a | 58.1
Manifold | 1 | 57.7
Online-Y | n/a | 57.0
JDExploreAcademy | 2 | 55.8
AISP-SJTU | 3 | 55.4
Lan-Bridge | n/a | 53.7
DLUT | 4 | 53.1
Online-G | n/a | 52.4
Online-W | n/a | 51.4

System | Rank | BLEU
---|---|---
Online-B | n/a | 80.0
HuaweiTSC | n/a | 73.3
LanguageX | n/a | 72.9
Manifold | 1 | 69.4
Online-A | n/a | 68.6
JDExploreAcademy | 2 | 68.4
Online-Y | n/a | 68.3
AISP-SJTU | 3 | 67.6
Lan-Bridge | n/a | 66.5
DLUT | 4 | 63.9
Online-G | n/a | 63.2
Online-W | n/a | 62.0

System | Rank | COMET
---|---|---
LanguageX | n/a | 63.9
Online-B | n/a | 61.8
JDExploreAcademy | 1 | 61.8
Lan-Bridge | n/a | 61.4
Online-W | n/a | 61.0
Manifold | 2 | 60.2
Online-Y | n/a | 59.8
HuaweiTSC | n/a | 59.6
Online-A | n/a | 57.4
AISP-SJTU | 3 | 56.6
DLUT | 4 | 52.2
Online-G | n/a | 51.3

Table 22:  Scores for the en$\rightarrow$zh translation task: chrF
(nrefs:2|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0), BLEU
(nrefs:2|case:mixed|eff:no|tok:zh|smooth:exp|version:2.1.0), COMET
(wmt20-comet-da).

##  Appendix B Statistical Significance Testing Tables

From Table 23 to 85.

| Onl. | JDE. | Lan. | Onl. | Onl. | CUN. | CUN. | Onl. | SHO. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 4.7 | 5.1 | 5.2 | 5.6 | 6.7 | 7.3 | 8.6 | 9.6 | 10.9 | 21.4
JDExploreAcademy |  | 0.0 | 0.4 | 0.5 | 1.0 | 2.1 | 2.7 | 4.0 | 4.9 | 6.3 | 16.7
Lan-Bridge |  |  | 0.0 | 0.1 | 0.6 | 1.7 | 2.3 | 3.6 | 4.5 | 5.9 | 16.3
Online-B |  |  |  | 0.0 | 0.5 | 1.6 | 2.2 | 3.5 | 4.4 | 5.8 | 16.2
Online-A |  |  |  |  | 0.0 | 1.1 | 1.7 | 3.0 | 4.0 | 5.3 | 15.7
CUNI-DocTransformer |  |  |  |  |  | 0.0 | 0.6 | 1.9 | 2.8 | 4.2 | 14.6
CUNI-Transformer |  |  |  |  |  |  | 0.0 | 1.3 | 2.3 | 3.6 | 14.0
Online-G |  |  |  |  |  |  |  | 0.0 | 0.9 | 2.3 | 12.7
SHOPLINE-PL |  |  |  |  |  |  |  |  | 0.0 | 1.4 | 11.8
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 10.4
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 23:  Statistical significance testing of the chrF score difference for each system pair for the cs$\rightarrow$en. | Onl. | JDE. | Lan. | Onl. | Onl. | CUN. | CUN. | Onl. | SHO. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 7.8 | 7.9 | 8.1 | 9.2 | 10.8 | 11.1 | 14.4 | 15.2 | 17.7 | 31.8
JDExploreAcademy |  | 0.0 | 0.1 | 0.3 | 1.5 | 3.0 | 3.3 | 6.6 | 7.4 | 9.9 | 24.1
Lan-Bridge |  |  | 0.0 | 0.2 | 1.3 | 2.9 | 3.2 | 6.5 | 7.3 | 9.8 | 23.9
Online-B |  |  |  | 0.0 | 1.2 | 2.7 | 3.0 | 6.3 | 7.1 | 9.6 | 23.8
Online-A |  |  |  |  | 0.0 | 1.5 | 1.8 | 5.1 | 5.9 | 8.4 | 22.6
CUNI-DocTransformer |  |  |  |  |  | 0.0 | 0.3 | 3.6 | 4.4 | 6.9 | 21.0
CUNI-Transformer |  |  |  |  |  |  | 0.0 | 3.3 | 4.1 | 6.6 | 20.8
Online-G |  |  |  |  |  |  |  | 0.0 | 0.8 | 3.3 | 17.4
SHOPLINE-PL |  |  |  |  |  |  |  |  | 0.0 | 2.5 | 16.6
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 14.1
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 24: Statistical significance testing of the BLEU score difference for each system pair for the cs$\rightarrow$en. | Onl. | JDE. | Onl. | Lan. | CUN. | Onl. | CUN. | Onl. | SHO. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 2.7 | 5.6 | 5.7 | 6.9 | 7.7 | 8.3 | 14.5 | 16.3 | 18.8 | 58.2
JDExploreAcademy |  | 0.0 | 2.9 | 3.0 | 4.2 | 5.0 | 5.6 | 11.8 | 13.6 | 16.1 | 55.5
Online-B |  |  | 0.0 | 0.1 | 1.3 | 2.1 | 2.7 | 8.9 | 10.7 | 13.2 | 52.6
Lan-Bridge |  |  |  | 0.0 | 1.2 | 2.0 | 2.6 | 8.8 | 10.6 | 13.1 | 52.5
CUNI-DocTransformer |  |  |  |  | 0.0 | 0.8 | 1.4 | 7.6 | 9.4 | 11.9 | 51.3
Online-A |  |  |  |  |  | 0.0 | 0.6 | 6.8 | 8.6 | 11.1 | 50.5
CUNI-Transformer |  |  |  |  |  |  | 0.0 | 6.2 | 8.0 | 10.5 | 49.9
Online-G |  |  |  |  |  |  |  | 0.0 | 1.8 | 4.3 | 43.7
SHOPLINE-PL |  |  |  |  |  |  |  |  | 0.0 | 2.5 | 41.9
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 39.4
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 25: Statistical significance testing of the COMET score difference for each system pair for the cs$\rightarrow$en. | Onl. | Lan. | Onl. | Onl. | JDE. | CUN. | CUN. | Onl. | Onl. | CUN.
---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.9 | 0.9 | 3.8 | 4.1 | 5.3 | 6.2 | 6.7 | 6.8 | 7.1
Lan-Bridge |  | 0.0 | -0.0 | 2.9 | 3.2 | 4.4 | 5.3 | 5.8 | 6.0 | 6.2
Online-W |  |  | 0.0 | 2.9 | 3.2 | 4.4 | 5.4 | 5.8 | 6.0 | 6.2
Online-A |  |  |  | 0.0 | 0.3 | 1.6 | 2.5 | 2.9 | 3.1 | 3.3
JDExploreAcademy |  |  |  |  | 0.0 | 1.2 | 2.1 | 2.6 | 2.8 | 3.0
CUNI-DocTransformer |  |  |  |  |  | 0.0 | 0.9 | 1.4 | 1.5 | 1.8
CUNI-Bergamot |  |  |  |  |  |  | 0.0 | 0.5 | 0.6 | 0.9
Online-G |  |  |  |  |  |  |  | 0.0 | 0.2 | 0.4
Online-Y |  |  |  |  |  |  |  |  | 0.0 | 0.2
CUNI-Transformer |  |  |  |  |  |  |  |  |  | 0.0
Table 26: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$cs. | Onl. | Lan. | Onl. | JDE. | Onl. | CUN. | CUN. | CUN. | Onl. | Onl.
---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 3.1 | 3.4 | 7.0 | 7.4 | 8.7 | 10.2 | 10.9 | 11.6 | 11.8
Lan-Bridge |  | 0.0 | 0.3 | 3.9 | 4.3 | 5.6 | 7.1 | 7.8 | 8.5 | 8.7
Online-W |  |  | 0.0 | 3.7 | 4.0 | 5.3 | 6.9 | 7.5 | 8.3 | 8.4
JDExploreAcademy |  |  |  | 0.0 | 0.3 | 1.6 | 3.2 | 3.9 | 4.6 | 4.7
Online-A |  |  |  |  | 0.0 | 1.3 | 2.9 | 3.5 | 4.3 | 4.4
CUNI-DocTransformer |  |  |  |  |  | 0.0 | 1.6 | 2.2 | 3.0 | 3.1
CUNI-Bergamot |  |  |  |  |  |  | 0.0 | 0.7 | 1.4 | 1.5
CUNI-Transformer |  |  |  |  |  |  |  | 0.0 | 0.7 | 0.9
Online-G |  |  |  |  |  |  |  |  | 0.0 | 0.1
Online-Y |  |  |  |  |  |  |  |  |  | 0.0
Table 27: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$cs. | Onl. | Onl. | CUN. | JDE. | Lan. | Onl. | CUN. | CUN. | Onl. | Onl.
---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 0.4 | 1.8 | 2.6 | 3.2 | 5.7 | 6.2 | 11.2 | 14.2 | 15.6
Online-B |  | 0.0 | 1.4 | 2.2 | 2.8 | 5.3 | 5.8 | 10.8 | 13.8 | 15.2
CUNI-Bergamot |  |  | 0.0 | 0.8 | 1.4 | 3.9 | 4.4 | 9.4 | 12.4 | 13.8
JDExploreAcademy |  |  |  | 0.0 | 0.6 | 3.1 | 3.6 | 8.6 | 11.6 | 13.0
Lan-Bridge |  |  |  |  | 0.0 | 2.5 | 3.0 | 8.0 | 11.0 | 12.4
Online-A |  |  |  |  |  | 0.0 | 0.5 | 5.5 | 8.5 | 9.9
CUNI-DocTransformer |  |  |  |  |  |  | 0.0 | 5.0 | 8.0 | 9.4
CUNI-Transformer |  |  |  |  |  |  |  | 0.0 | 3.0 | 4.4
Online-Y |  |  |  |  |  |  |  |  | 0.0 | 1.4
Online-G |  |  |  |  |  |  |  |  |  | 0.0
Table 28: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$cs. | Onl. | Lan. | Hua. | Onl. | CUN. | CUN. | AMU. | Cha. | Onl. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.1 | 1.5 | 1.8 | 2.5 | 2.5 | 2.5 | 2.5 | 3.3 | 4.4 | 9.6
Lan-Bridge |  | 0.0 | 1.4 | 1.7 | 2.4 | 2.4 | 2.4 | 2.4 | 3.2 | 4.4 | 9.5
HuaweiTSC |  |  | 0.0 | 0.4 | 1.0 | 1.0 | 1.1 | 1.0 | 1.8 | 3.0 | 8.1
Online-A |  |  |  | 0.0 | 0.7 | 0.7 | 0.7 | 0.7 | 1.4 | 2.6 | 7.7
CUNI-JL-JH |  |  |  |  | 0.0 | 0.0 | 0.0 | 0.0 | 0.8 | 2.0 | 7.1
CUNI-Transformer |  |  |  |  |  | 0.0 | 0.0 | 0.0 | 0.8 | 2.0 | 7.1
AMU |  |  |  |  |  |  | 0.0 | -0.0 | 0.7 | 1.9 | 7.0
CharlesTranslator |  |  |  |  |  |  |  | 0.0 | 0.8 | 1.9 | 7.1
Online-G |  |  |  |  |  |  |  |  | 0.0 | 1.2 | 6.3
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 5.1
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 29: Statistical significance testing of the chrF score difference for each system pair for the cs$\rightarrow$uk. | Onl. | Lan. | Hua. | Onl. | CUN. | CUN. | AMU. | Cha. | Onl. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.1 | 2.4 | 2.5 | 3.3 | 3.6 | 3.6 | 4.0 | 5.8 | 6.2 | 11.5
Lan-Bridge |  | 0.0 | 2.3 | 2.4 | 3.2 | 3.5 | 3.5 | 3.9 | 5.8 | 6.2 | 11.4
HuaweiTSC |  |  | 0.0 | 0.1 | 1.0 | 1.2 | 1.2 | 1.7 | 3.5 | 3.9 | 9.1
Online-A |  |  |  | 0.0 | 0.9 | 1.1 | 1.1 | 1.6 | 3.4 | 3.8 | 9.0
CUNI-Transformer |  |  |  |  | 0.0 | 0.2 | 0.3 | 0.7 | 2.5 | 2.9 | 8.2
CUNI-JL-JH |  |  |  |  |  | 0.0 | 0.0 | 0.5 | 2.3 | 2.7 | 7.9
AMU |  |  |  |  |  |  | 0.0 | 0.4 | 2.2 | 2.6 | 7.9
CharlesTranslator |  |  |  |  |  |  |  | 0.0 | 1.8 | 2.2 | 7.5
Online-G |  |  |  |  |  |  |  |  | 0.0 | 0.4 | 5.7
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 5.3
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 30: Statistical significance testing of the BLEU score difference for each system pair for the cs$\rightarrow$uk. | AMU. | Onl. | Lan. | Cha. | Hua. | CUN. | Onl. | Onl. | CUN. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
AMU | 0.0 | 5.1 | 7.6 | 8.6 | 8.7 | 9.3 | 11.0 | 11.5 | 12.0 | 21.0 | 38.0
Online-B |  | 0.0 | 2.5 | 3.5 | 3.6 | 4.2 | 5.9 | 6.4 | 6.9 | 15.9 | 32.9
Lan-Bridge |  |  | 0.0 | 1.0 | 1.1 | 1.7 | 3.4 | 3.9 | 4.4 | 13.4 | 30.4
CharlesTranslator |  |  |  | 0.0 | 0.1 | 0.7 | 2.4 | 2.9 | 3.4 | 12.4 | 29.4
HuaweiTSC |  |  |  |  | 0.0 | 0.6 | 2.3 | 2.8 | 3.3 | 12.3 | 29.3
CUNI-JL-JH |  |  |  |  |  | 0.0 | 1.7 | 2.2 | 2.7 | 11.7 | 28.7
Online-G |  |  |  |  |  |  | 0.0 | 0.5 | 1.0 | 10.0 | 27.0
Online-A |  |  |  |  |  |  |  | 0.0 | 0.5 | 9.5 | 26.5
CUNI-Transformer |  |  |  |  |  |  |  |  | 0.0 | 9.0 | 26.0
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 17.0
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 31: Statistical significance testing of the COMET score difference for each system pair for the cs$\rightarrow$uk. | AMU. | Lan. | Onl. | Hua. | Cha. | CUN. | CUN. | Onl. | Onl. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
AMU | 0.0 | 0.3 | 0.4 | 1.1 | 1.6 | 1.7 | 2.0 | 3.2 | 4.3 | 5.3 | 10.0
Lan-Bridge |  | 0.0 | 0.1 | 0.8 | 1.4 | 1.4 | 1.7 | 2.9 | 4.1 | 5.1 | 9.7
Online-B |  |  | 0.0 | 0.7 | 1.2 | 1.2 | 1.6 | 2.8 | 3.9 | 4.9 | 9.6
HuaweiTSC |  |  |  | 0.0 | 0.6 | 0.6 | 0.9 | 2.1 | 3.2 | 4.3 | 8.9
CharlesTranslator |  |  |  |  | 0.0 | 0.0 | 0.3 | 1.5 | 2.7 | 3.7 | 8.3
CUNI-Transformer |  |  |  |  |  | 0.0 | 0.3 | 1.5 | 2.7 | 3.7 | 8.3
CUNI-JL-JH |  |  |  |  |  |  | 0.0 | 1.2 | 2.3 | 3.3 | 8.0
Online-A |  |  |  |  |  |  |  | 0.0 | 1.2 | 2.2 | 6.8
Online-G |  |  |  |  |  |  |  |  | 0.0 | 1.0 | 5.6
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 4.6
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 32: Statistical significance testing of the chrF score difference for each system pair for the uk$\rightarrow$cs. | AMU. | Lan. | Onl. | Hua. | Cha. | CUN. | CUN. | Onl. | Onl. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
AMU | 0.0 | 0.5 | 0.6 | 1.0 | 1.1 | 1.2 | 1.9 | 3.7 | 5.5 | 7.5 | 11.8
Lan-Bridge |  | 0.0 | 0.1 | 0.5 | 0.6 | 0.7 | 1.4 | 3.2 | 5.0 | 7.0 | 11.2
Online-B |  |  | 0.0 | 0.4 | 0.5 | 0.6 | 1.3 | 3.1 | 4.9 | 6.9 | 11.1
HuaweiTSC |  |  |  | 0.0 | 0.1 | 0.2 | 0.9 | 2.7 | 4.5 | 6.4 | 10.7
CharlesTranslator |  |  |  |  | 0.0 | 0.1 | 0.8 | 2.6 | 4.3 | 6.3 | 10.6
CUNI-Transformer |  |  |  |  |  | 0.0 | 0.7 | 2.5 | 4.3 | 6.2 | 10.5
CUNI-JL-JH |  |  |  |  |  |  | 0.0 | 1.8 | 3.6 | 5.5 | 9.8
Online-A |  |  |  |  |  |  |  | 0.0 | 1.8 | 3.8 | 8.1
Online-G |  |  |  |  |  |  |  |  | 0.0 | 2.0 | 6.3
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 4.3
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 33: Statistical significance testing of the BLEU score difference for each system pair for the uk$\rightarrow$cs. | AMU. | Onl. | Lan. | Hua. | Cha. | CUN. | CUN. | Onl. | Onl. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---|---
AMU | 0.0 | 8.4 | 10.3 | 13.4 | 14.7 | 15.8 | 16.4 | 19.5 | 20.7 | 26.2 | 42.4
Online-B |  | 0.0 | 1.9 | 5.0 | 6.3 | 7.4 | 8.0 | 11.1 | 12.3 | 17.8 | 34.0
Lan-Bridge |  |  | 0.0 | 3.1 | 4.4 | 5.5 | 6.1 | 9.2 | 10.4 | 15.9 | 32.1
HuaweiTSC |  |  |  | 0.0 | 1.3 | 2.4 | 3.0 | 6.1 | 7.3 | 12.8 | 29.0
CharlesTranslator |  |  |  |  | 0.0 | 1.1 | 1.7 | 4.8 | 6.0 | 11.5 | 27.7
CUNI-JL-JH |  |  |  |  |  | 0.0 | 0.6 | 3.7 | 4.9 | 10.4 | 26.6
CUNI-Transformer |  |  |  |  |  |  | 0.0 | 3.1 | 4.3 | 9.8 | 26.0
Online-A |  |  |  |  |  |  |  | 0.0 | 1.2 | 6.7 | 22.9
Online-G |  |  |  |  |  |  |  |  | 0.0 | 5.5 | 21.7
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 16.2
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  |  | 0.0
Table 34: Statistical significance testing of the COMET score difference for
each system pair for the uk$\rightarrow$cs.

| Onl. | Onl. | Lan. | JDE. | Onl. | Onl. | PRO. | Onl. | LT2.
---|---|---|---|---|---|---|---|---|---
Online-A | 0.0 | 0.1 | 0.2 | 0.6 | 0.7 | 0.7 | 0.7 | 0.9 | 8.0
Online-G |  | 0.0 | 0.1 | 0.5 | 0.6 | 0.6 | 0.7 | 0.9 | 7.9
Lan-Bridge |  |  | 0.0 | 0.4 | 0.5 | 0.5 | 0.5 | 0.8 | 7.8
JDExploreAcademy |  |  |  | 0.0 | 0.1 | 0.1 | 0.2 | 0.4 | 7.4
Online-B |  |  |  |  | 0.0 | 0.0 | 0.1 | 0.3 | 7.3
Online-Y |  |  |  |  |  | 0.0 | 0.0 | 0.3 | 7.3
PROMT |  |  |  |  |  |  | 0.0 | 0.2 | 7.2
Online-W |  |  |  |  |  |  |  | 0.0 | 7.0
LT22 |  |  |  |  |  |  |  |  | 0.0
Table 35: Statistical significance testing of the chrF score difference for each system pair for the de$\rightarrow$en. | Onl. | Lan. | Onl. | Onl. | JDE. | Onl. | PRO. | Onl. | LT2.
---|---|---|---|---|---|---|---|---|---
Online-A | 0.0 | 0.0 | 0.4 | 0.5 | 0.8 | 0.8 | 1.0 | 1.4 | 9.8
Lan-Bridge |  | 0.0 | 0.4 | 0.5 | 0.8 | 0.8 | 1.0 | 1.3 | 9.8
Online-B |  |  | 0.0 | 0.1 | 0.4 | 0.4 | 0.6 | 0.9 | 9.4
Online-G |  |  |  | 0.0 | 0.3 | 0.3 | 0.5 | 0.9 | 9.3
JDExploreAcademy |  |  |  |  | 0.0 | -0.0 | 0.2 | 0.5 | 9.0
Online-Y |  |  |  |  |  | 0.0 | 0.2 | 0.5 | 9.0
PROMT |  |  |  |  |  |  | 0.0 | 0.4 | 8.8
Online-W |  |  |  |  |  |  |  | 0.0 | 8.5
LT22 |  |  |  |  |  |  |  |  | 0.0
Table 36: Statistical significance testing of the BLEU score difference for each system pair for the de$\rightarrow$en. | JDE. | Onl. | Lan. | Onl. | Onl. | Onl. | Onl. | PRO. | LT2.
---|---|---|---|---|---|---|---|---|---
JDExploreAcademy | 0.0 | 1.0 | 1.4 | 2.7 | 3.3 | 3.5 | 3.6 | 6.2 | 32.4
Online-B |  | 0.0 | 0.4 | 1.7 | 2.3 | 2.5 | 2.6 | 5.2 | 31.4
Lan-Bridge |  |  | 0.0 | 1.3 | 1.9 | 2.1 | 2.2 | 4.8 | 31.0
Online-G |  |  |  | 0.0 | 0.6 | 0.8 | 0.9 | 3.5 | 29.7
Online-Y |  |  |  |  | 0.0 | 0.2 | 0.3 | 2.9 | 29.1
Online-A |  |  |  |  |  | 0.0 | 0.1 | 2.7 | 28.9
Online-W |  |  |  |  |  |  | 0.0 | 2.6 | 28.8
PROMT |  |  |  |  |  |  |  | 0.0 | 26.2
LT22 |  |  |  |  |  |  |  |  | 0.0
Table 37: Statistical significance testing of the COMET score difference for each system pair for the de$\rightarrow$en. | Onl. | JDE. | Lan. | Onl. | Onl. | Onl. | Onl. | PRO. | Ope.
---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.6 | 0.7 | 1.0 | 1.1 | 1.1 | 1.4 | 2.3 | 3.1
JDExploreAcademy |  | 0.0 | 0.1 | 0.4 | 0.5 | 0.5 | 0.8 | 1.7 | 2.5
Lan-Bridge |  |  | 0.0 | 0.3 | 0.4 | 0.4 | 0.7 | 1.6 | 2.4
Online-A |  |  |  | 0.0 | 0.1 | 0.1 | 0.4 | 1.3 | 2.1
Online-W |  |  |  |  | 0.0 | 0.0 | 0.3 | 1.2 | 2.0
Online-Y |  |  |  |  |  | 0.0 | 0.3 | 1.2 | 2.0
Online-G |  |  |  |  |  |  | 0.0 | 0.9 | 1.7
PROMT |  |  |  |  |  |  |  | 0.0 | 0.8
OpenNMT |  |  |  |  |  |  |  |  | 0.0
Table 38: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$de. | Onl. | JDE. | Onl. | Onl. | Onl. | Lan. | PRO. | Onl. | Ope.
---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.7 | 1.8 | 2.2 | 2.7 | 2.9 | 3.3 | 3.5 | 3.9
JDExploreAcademy |  | 0.0 | 1.1 | 1.5 | 2.0 | 2.2 | 2.6 | 2.8 | 3.2
Online-Y |  |  | 0.0 | 0.4 | 0.9 | 1.1 | 1.5 | 1.6 | 2.1
Online-A |  |  |  | 0.0 | 0.5 | 0.8 | 1.2 | 1.3 | 1.7
Online-G |  |  |  |  | 0.0 | 0.3 | 0.7 | 0.8 | 1.2
Lan-Bridge |  |  |  |  |  | 0.0 | 0.4 | 0.5 | 1.0
PROMT |  |  |  |  |  |  | 0.0 | 0.1 | 0.6
Online-W |  |  |  |  |  |  |  | 0.0 | 0.5
OpenNMT |  |  |  |  |  |  |  |  | 0.0
Table 39: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$de. | Onl. | JDE. | Onl. | Onl. | Onl. | Onl. | Lan. | Ope. | PRO.
---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 2.3 | 3.3 | 4.5 | 5.0 | 5.4 | 6.8 | 8.3 | 9.7
JDExploreAcademy |  | 0.0 | 1.0 | 2.2 | 2.7 | 3.1 | 4.5 | 6.0 | 7.4
Online-B |  |  | 0.0 | 1.2 | 1.7 | 2.1 | 3.5 | 5.0 | 6.4
Online-Y |  |  |  | 0.0 | 0.5 | 0.9 | 2.3 | 3.8 | 5.2
Online-A |  |  |  |  | 0.0 | 0.4 | 1.8 | 3.3 | 4.7
Online-G |  |  |  |  |  | 0.0 | 1.4 | 2.9 | 4.3
Lan-Bridge |  |  |  |  |  |  | 0.0 | 1.5 | 2.9
OpenNMT |  |  |  |  |  |  |  | 0.0 | 1.4
PROMT |  |  |  |  |  |  |  |  | 0.0
Table 40: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$de. | Onl. | Onl. | Onl. | Onl. | Onl. | LT2.
---|---|---|---|---|---|---
Online-B | 0.0 | 7.8 | 9.1 | 10.1 | 12.0 | 20.2
Online-Y |  | 0.0 | 1.3 | 2.3 | 4.1 | 12.4
Online-W |  |  | 0.0 | 1.0 | 2.8 | 11.1
Online-A |  |  |  | 0.0 | 1.8 | 10.1
Online-G |  |  |  |  | 0.0 | 8.2
LT22 |  |  |  |  |  | 0.0
Table 41: Statistical significance testing of the chrF score difference for each system pair for the de$\rightarrow$fr. | Onl. | Onl. | Onl. | Onl. | Onl. | LT2.
---|---|---|---|---|---|---
Online-B | 0.0 | 12.2 | 14.8 | 17.0 | 19.4 | 30.0
Online-Y |  | 0.0 | 2.6 | 4.8 | 7.2 | 17.8
Online-W |  |  | 0.0 | 2.2 | 4.6 | 15.2
Online-A |  |  |  | 0.0 | 2.4 | 13.0
Online-G |  |  |  |  | 0.0 | 10.6
LT22 |  |  |  |  |  | 0.0
Table 42: Statistical significance testing of the BLEU score difference for each system pair for the de$\rightarrow$fr. | Onl. | Onl. | Onl. | Onl. | Onl. | LT2.
---|---|---|---|---|---|---
Online-B | 0.0 | 6.8 | 12.7 | 18.2 | 25.6 | 60.0
Online-W |  | 0.0 | 5.9 | 11.4 | 18.8 | 53.2
Online-Y |  |  | 0.0 | 5.5 | 12.9 | 47.3
Online-A |  |  |  | 0.0 | 7.4 | 41.8
Online-G |  |  |  |  | 0.0 | 34.4
LT22 |  |  |  |  |  | 0.0
Table 43: Statistical significance testing of the COMET score difference for each system pair for the de$\rightarrow$fr. | Onl. | Onl. | eTr. | Onl. | Onl. | Onl. | Lan.
---|---|---|---|---|---|---|---
Online-W | 0.0 | 12.5 | 12.8 | 13.7 | 14.0 | 15.2 | 16.2
Online-B |  | 0.0 | 0.3 | 1.2 | 1.5 | 2.7 | 3.7
eTranslation |  |  | 0.0 | 0.9 | 1.2 | 2.4 | 3.4
Online-Y |  |  |  | 0.0 | 0.3 | 1.5 | 2.5
Online-A |  |  |  |  | 0.0 | 1.2 | 2.2
Online-G |  |  |  |  |  | 0.0 | 1.0
Lan-Bridge |  |  |  |  |  |  | 0.0
Table 44: Statistical significance testing of the chrF score difference for each system pair for the fr$\rightarrow$de. | Onl. | Onl. | eTr. | Onl. | Onl. | Lan. | Onl.
---|---|---|---|---|---|---|---
Online-W | 0.0 | 18.2 | 18.3 | 19.8 | 20.4 | 23.0 | 23.8
Online-B |  | 0.0 | 0.2 | 1.6 | 2.3 | 4.9 | 5.6
eTranslation |  |  | 0.0 | 1.4 | 2.1 | 4.7 | 5.4
Online-Y |  |  |  | 0.0 | 0.6 | 3.3 | 4.0
Online-A |  |  |  |  | 0.0 | 2.6 | 3.3
Lan-Bridge |  |  |  |  |  | 0.0 | 0.7
Online-G |  |  |  |  |  |  | 0.0
Table 45: Statistical significance testing of the BLEU score difference for each system pair for the fr$\rightarrow$de. | Onl. | Onl. | Onl. | Onl. | eTr. | Lan. | Onl.
---|---|---|---|---|---|---|---
Online-W | 0.0 | 14.1 | 16.3 | 18.6 | 22.4 | 26.7 | 29.6
Online-B |  | 0.0 | 2.2 | 4.5 | 8.3 | 12.6 | 15.5
Online-Y |  |  | 0.0 | 2.3 | 6.1 | 10.4 | 13.3
Online-A |  |  |  | 0.0 | 3.8 | 8.1 | 11.0
eTranslation |  |  |  |  | 0.0 | 4.3 | 7.2
Lan-Bridge |  |  |  |  |  | 0.0 | 2.9
Online-G |  |  |  |  |  |  | 0.0
Table 46: Statistical significance testing of the COMET score difference for each system pair for the fr$\rightarrow$de. | Onl. | Lan. | Hua. | Onl. | SRP. | Niu. | Onl. | Onl.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.1 | 1.7 | 2.0 | 2.2 | 2.2 | 4.2 | 5.3
Lan-Bridge |  | 0.0 | 1.7 | 1.9 | 2.2 | 2.2 | 4.1 | 5.2
HuaweiTSC |  |  | 0.0 | 0.3 | 0.5 | 0.5 | 2.4 | 3.6
Online-A |  |  |  | 0.0 | 0.2 | 0.2 | 2.2 | 3.3
SRPOL |  |  |  |  | 0.0 | 0.0 | 2.0 | 3.1
NiuTrans |  |  |  |  |  | 0.0 | 2.0 | 3.1
Online-Y |  |  |  |  |  |  | 0.0 | 1.1
Online-G |  |  |  |  |  |  |  | 0.0
Table 47: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$hr. | Onl. | Lan. | Hua. | Niu. | Onl. | SRP. | Onl. | Onl.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.0 | 1.6 | 2.2 | 2.4 | 2.4 | 4.9 | 5.8
Lan-Bridge |  | 0.0 | 1.6 | 2.1 | 2.4 | 2.4 | 4.9 | 5.8
HuaweiTSC |  |  | 0.0 | 0.5 | 0.8 | 0.8 | 3.3 | 4.2
NiuTrans |  |  |  | 0.0 | 0.2 | 0.3 | 2.8 | 3.6
Online-A |  |  |  |  | 0.0 | 0.0 | 2.5 | 3.4
SRPOL |  |  |  |  |  | 0.0 | 2.5 | 3.4
Online-Y |  |  |  |  |  |  | 0.0 | 0.9
Online-G |  |  |  |  |  |  |  | 0.0
Table 48: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$hr. | Onl. | Lan. | Onl. | SRP. | Hua. | Niu. | Onl. | Onl.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.8 | 10.9 | 10.9 | 12.7 | 14.8 | 16.1 | 23.6
Lan-Bridge |  | 0.0 | 10.1 | 10.1 | 11.9 | 14.0 | 15.3 | 22.8
Online-A |  |  | 0.0 | 0.0 | 1.8 | 3.9 | 5.2 | 12.7
SRPOL |  |  |  | 0.0 | 1.8 | 3.9 | 5.2 | 12.7
HuaweiTSC |  |  |  |  | 0.0 | 2.1 | 3.4 | 10.9
NiuTrans |  |  |  |  |  | 0.0 | 1.3 | 8.8
Online-G |  |  |  |  |  |  | 0.0 | 7.5
Online-Y |  |  |  |  |  |  |  | 0.0
Table 49: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$hr. | Onl. | NT5. | JDE. | Onl. | DLU. | Lan. | Lan. | Onl. | NAI. | Onl. | AIS. | Onl. | KYB. | AIS.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 0.4 | 1.5 | 1.8 | 1.9 | 2.6 | 2.9 | 3.3 | 3.4 | 3.5 | 3.6 | 6.0 | 8.2 | 40.3
NT5 |  | 0.0 | 1.1 | 1.4 | 1.5 | 2.2 | 2.6 | 2.9 | 3.0 | 3.1 | 3.3 | 5.6 | 7.8 | 39.9
JDExploreAcademy |  |  | 0.0 | 0.2 | 0.4 | 1.1 | 1.4 | 1.7 | 1.9 | 1.9 | 2.1 | 4.4 | 6.7 | 38.8
Online-B |  |  |  | 0.0 | 0.1 | 0.8 | 1.2 | 1.5 | 1.6 | 1.7 | 1.9 | 4.2 | 6.4 | 38.5
DLUT |  |  |  |  | 0.0 | 0.7 | 1.0 | 1.4 | 1.5 | 1.6 | 1.7 | 4.1 | 6.3 | 38.4
LanguageX |  |  |  |  |  | 0.0 | 0.4 | 0.7 | 0.8 | 0.9 | 1.1 | 3.4 | 5.6 | 37.7
Lan-Bridge |  |  |  |  |  |  | 0.0 | 0.3 | 0.4 | 0.5 | 0.7 | 3.0 | 5.3 | 37.3
Online-A |  |  |  |  |  |  |  | 0.0 | 0.1 | 0.2 | 0.4 | 2.7 | 4.9 | 37.0
NAIST-NICT-TIT |  |  |  |  |  |  |  |  | 0.0 | 0.1 | 0.3 | 2.6 | 4.8 | 36.9
Online-Y |  |  |  |  |  |  |  |  |  | 0.0 | 0.2 | 2.5 | 4.7 | 36.8
AISP-SJTU |  |  |  |  |  |  |  |  |  |  | 0.0 | 2.3 | 4.6 | 36.7
Online-G |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 2.2 | 34.3
KYB |  |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 32.1
AIST |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 50: Statistical significance testing of the chrF score difference for each system pair for the ja$\rightarrow$en. | Onl. | NT5. | JDE. | DLU. | Onl. | Onl. | Lan. | NAI. | Lan. | AIS. | Onl. | Onl. | KYB. | AIS.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 1.1 | 2.1 | 2.9 | 3.0 | 5.0 | 5.0 | 5.1 | 5.4 | 5.8 | 6.3 | 8.0 | 9.7 | 27.7
NT5 |  | 0.0 | 1.0 | 1.8 | 1.9 | 3.9 | 3.8 | 4.0 | 4.3 | 4.6 | 5.2 | 6.9 | 8.5 | 26.6
JDExploreAcademy |  |  | 0.0 | 0.8 | 0.9 | 2.8 | 2.8 | 2.9 | 3.3 | 3.6 | 4.1 | 5.9 | 7.5 | 25.6
DLUT |  |  |  | 0.0 | 0.1 | 2.1 | 2.0 | 2.1 | 2.5 | 2.8 | 3.3 | 5.1 | 6.7 | 24.8
Online-B |  |  |  |  | 0.0 | 2.0 | 1.9 | 2.0 | 2.4 | 2.7 | 3.3 | 5.0 | 6.6 | 24.7
Online-A |  |  |  |  |  | 0.0 | -0.0 | 0.1 | 0.4 | 0.8 | 1.3 | 3.0 | 4.7 | 22.7
Lan-Bridge |  |  |  |  |  |  | 0.0 | 0.1 | 0.4 | 0.8 | 1.3 | 3.1 | 4.7 | 22.7
NAIST-NICT-TIT |  |  |  |  |  |  |  | 0.0 | 0.3 | 0.7 | 1.2 | 2.9 | 4.6 | 22.6
LanguageX |  |  |  |  |  |  |  |  | 0.0 | 0.4 | 0.9 | 2.6 | 4.3 | 22.3
AISP-SJTU |  |  |  |  |  |  |  |  |  | 0.0 | 0.5 | 2.2 | 3.9 | 21.9
Online-Y |  |  |  |  |  |  |  |  |  |  | 0.0 | 1.7 | 3.4 | 21.4
Online-G |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 1.6 | 19.7
KYB |  |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 18.0
AIST |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 51: Statistical significance testing of the BLEU score difference for each system pair for the ja$\rightarrow$en. | NT5. | Onl. | JDE. | Onl. | DLU. | NAI. | Lan. | Onl. | Onl. | Lan. | AIS. | Onl. | KYB. | AIS.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
NT5 | 0.0 | 0.9 | 1.5 | 2.4 | 4.8 | 8.7 | 9.1 | 9.2 | 9.7 | 10.1 | 11.9 | 19.8 | 24.7 | 194.8
Online-W |  | 0.0 | 0.6 | 1.5 | 3.9 | 7.8 | 8.2 | 8.3 | 8.8 | 9.2 | 11.0 | 18.9 | 23.8 | 193.9
JDExploreAcademy |  |  | 0.0 | 0.9 | 3.3 | 7.2 | 7.6 | 7.7 | 8.2 | 8.6 | 10.4 | 18.3 | 23.2 | 193.3
Online-B |  |  |  | 0.0 | 2.4 | 6.3 | 6.7 | 6.8 | 7.3 | 7.7 | 9.5 | 17.4 | 22.3 | 192.4
DLUT |  |  |  |  | 0.0 | 3.9 | 4.3 | 4.4 | 4.9 | 5.3 | 7.1 | 15.0 | 19.9 | 190.0
NAIST-NICT-TIT |  |  |  |  |  | 0.0 | 0.4 | 0.5 | 1.0 | 1.4 | 3.2 | 11.1 | 16.0 | 186.1
LanguageX |  |  |  |  |  |  | 0.0 | 0.1 | 0.6 | 1.0 | 2.8 | 10.7 | 15.6 | 185.7
Online-A |  |  |  |  |  |  |  | 0.0 | 0.5 | 0.9 | 2.7 | 10.6 | 15.5 | 185.6
Online-Y |  |  |  |  |  |  |  |  | 0.0 | 0.4 | 2.2 | 10.1 | 15.0 | 185.1
Lan-Bridge |  |  |  |  |  |  |  |  |  | 0.0 | 1.8 | 9.7 | 14.6 | 184.7
AISP-SJTU |  |  |  |  |  |  |  |  |  |  | 0.0 | 7.9 | 12.8 | 182.9
Online-G |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 4.9 | 175.0
KYB |  |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 170.1
AIST |  |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 52: Statistical significance testing of the COMET score difference for each system pair for the ja$\rightarrow$en. | NT5. | JDE. | Lan. | DLU. | Onl. | Onl. | Onl. | Lan. | Onl. | AIS. | NAI. | KYB. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
NT5 | 0.0 | 0.7 | 0.8 | 0.8 | 1.3 | 1.6 | 2.4 | 2.7 | 2.8 | 3.0 | 3.1 | 8.3 | 8.9
JDExploreAcademy |  | 0.0 | 0.1 | 0.1 | 0.6 | 0.9 | 1.7 | 2.0 | 2.1 | 2.3 | 2.4 | 7.6 | 8.2
LanguageX |  |  | 0.0 | 0.0 | 0.6 | 0.8 | 1.6 | 2.0 | 2.0 | 2.2 | 2.3 | 7.5 | 8.1
DLUT |  |  |  | 0.0 | 0.5 | 0.8 | 1.6 | 2.0 | 2.0 | 2.2 | 2.3 | 7.5 | 8.1
Online-B |  |  |  |  | 0.0 | 0.3 | 1.1 | 1.4 | 1.5 | 1.6 | 1.7 | 6.9 | 7.5
Online-W |  |  |  |  |  | 0.0 | 0.8 | 1.1 | 1.2 | 1.4 | 1.5 | 6.7 | 7.3
Online-Y |  |  |  |  |  |  | 0.0 | 0.3 | 0.4 | 0.5 | 0.6 | 5.9 | 6.4
Lan-Bridge |  |  |  |  |  |  |  | 0.0 | 0.1 | 0.2 | 0.3 | 5.5 | 6.1
Online-A |  |  |  |  |  |  |  |  | 0.0 | 0.2 | 0.3 | 5.5 | 6.1
AISP-SJTU |  |  |  |  |  |  |  |  |  | 0.0 | 0.1 | 5.3 | 5.9
NAIST-NICT-TIT |  |  |  |  |  |  |  |  |  |  | 0.0 | 5.2 | 5.8
KYB |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.6
Online-G |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 53: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$ja. | NT5. | Lan. | DLU. | Onl. | JDE. | Onl. | Onl. | AIS. | Lan. | NAI. | Onl. | KYB. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
NT5 | 0.0 | 0.4 | 0.4 | 0.5 | 0.6 | 1.1 | 2.4 | 2.6 | 2.8 | 3.0 | 3.6 | 8.7 | 9.0
LanguageX |  | 0.0 | 0.0 | 0.2 | 0.3 | 0.7 | 2.0 | 2.2 | 2.4 | 2.7 | 3.2 | 8.3 | 8.6
DLUT |  |  | 0.0 | 0.2 | 0.3 | 0.7 | 2.0 | 2.2 | 2.4 | 2.7 | 3.2 | 8.3 | 8.6
Online-W |  |  |  | 0.0 | 0.1 | 0.5 | 1.8 | 2.1 | 2.2 | 2.5 | 3.0 | 8.2 | 8.5
JDExploreAcademy |  |  |  |  | 0.0 | 0.4 | 1.7 | 2.0 | 2.1 | 2.4 | 2.9 | 8.0 | 8.4
Online-B |  |  |  |  |  | 0.0 | 1.3 | 1.5 | 1.7 | 2.0 | 2.5 | 7.6 | 7.9
Online-Y |  |  |  |  |  |  | 0.0 | 0.2 | 0.4 | 0.7 | 1.2 | 6.3 | 6.6
AISP-SJTU |  |  |  |  |  |  |  | 0.0 | 0.2 | 0.4 | 1.0 | 6.1 | 6.4
Lan-Bridge |  |  |  |  |  |  |  |  | 0.0 | 0.3 | 0.8 | 5.9 | 6.2
NAIST-NICT-TIT |  |  |  |  |  |  |  |  |  | 0.0 | 0.5 | 5.6 | 6.0
Online-A |  |  |  |  |  |  |  |  |  |  | 0.0 | 5.1 | 5.4
KYB |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.3
Online-G |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 54: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$ja. | JDE. | NT5. | Lan. | Onl. | DLU. | Onl. | Onl. | Lan. | Onl. | NAI. | AIS. | KYB. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
JDExploreAcademy | 0.0 | 1.1 | 3.1 | 4.4 | 4.7 | 5.4 | 8.3 | 8.6 | 11.5 | 11.8 | 12.8 | 33.4 | 40.2
NT5 |  | 0.0 | 2.0 | 3.3 | 3.6 | 4.3 | 7.2 | 7.5 | 10.4 | 10.7 | 11.7 | 32.3 | 39.1
LanguageX |  |  | 0.0 | 1.3 | 1.6 | 2.3 | 5.2 | 5.5 | 8.4 | 8.7 | 9.7 | 30.3 | 37.1
Online-B |  |  |  | 0.0 | 0.3 | 1.0 | 3.9 | 4.2 | 7.1 | 7.4 | 8.4 | 29.0 | 35.8
DLUT |  |  |  |  | 0.0 | 0.7 | 3.6 | 3.9 | 6.8 | 7.1 | 8.1 | 28.7 | 35.5
Online-W |  |  |  |  |  | 0.0 | 2.9 | 3.2 | 6.1 | 6.4 | 7.4 | 28.0 | 34.8
Online-Y |  |  |  |  |  |  | 0.0 | 0.3 | 3.2 | 3.5 | 4.5 | 25.1 | 31.9
Lan-Bridge |  |  |  |  |  |  |  | 0.0 | 2.9 | 3.2 | 4.2 | 24.8 | 31.6
Online-A |  |  |  |  |  |  |  |  | 0.0 | 0.3 | 1.3 | 21.9 | 28.7
NAIST-NICT-TIT |  |  |  |  |  |  |  |  |  | 0.0 | 1.0 | 21.6 | 28.4
AISP-SJTU |  |  |  |  |  |  |  |  |  |  | 0.0 | 20.6 | 27.4
KYB |  |  |  |  |  |  |  |  |  |  |  | 0.0 | 6.8
Online-G |  |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 55: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$ja. | Tar. | TAL. | Hua. | Liv. | Niu.
---|---|---|---|---|---
TartuNLP | 0.0 | 0.4 | 5.1 | 6.8 | 17.9
TAL-SJTU |  | 0.0 | 4.7 | 6.4 | 17.5
HuaweiTSC |  |  | 0.0 | 1.7 | 12.8
Liv4ever |  |  |  | 0.0 | 11.1
NiuTrans |  |  |  |  | 0.0
Table 56: Statistical significance testing of the chrF score difference for each system pair for the liv$\rightarrow$en. | TAL. | Tar. | Hua. | Liv. | Niu.
---|---|---|---|---|---
TAL-SJTU | 0.0 | 0.5 | 7.0 | 7.1 | 17.4
TartuNLP |  | 0.0 | 6.5 | 6.6 | 16.9
HuaweiTSC |  |  | 0.0 | 0.1 | 10.4
Liv4ever |  |  |  | 0.0 | 10.3
NiuTrans |  |  |  |  | 0.0
Table 57: Statistical significance testing of the BLEU score difference for each system pair for the liv$\rightarrow$en. | Tar. | TAL. | Hua. | Liv. | Niu.
---|---|---|---|---|---
TartuNLP | 0.0 | 2.6 | 21.5 | 38.3 | 82.6
TAL-SJTU |  | 0.0 | 18.9 | 35.7 | 80.0
HuaweiTSC |  |  | 0.0 | 16.8 | 61.1
Liv4ever |  |  |  | 0.0 | 44.3
NiuTrans |  |  |  |  | 0.0
Table 58: Statistical significance testing of the COMET score difference for each system pair for the liv$\rightarrow$en. | TAL. | Liv. | Tar. | Hua. | Niu.
---|---|---|---|---|---
TAL-SJTU | 0.0 | 4.2 | 4.6 | 6.1 | 13.3
Liv4ever |  | 0.0 | 0.4 | 1.9 | 9.1
TartuNLP |  |  | 0.0 | 1.6 | 8.7
HuaweiTSC |  |  |  | 0.0 | 7.2
NiuTrans |  |  |  |  | 0.0
Table 59: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$liv. | TAL. | Tar. | Liv. | Hua. | Niu.
---|---|---|---|---|---
TAL-SJTU | 0.0 | 2.0 | 2.3 | 4.2 | 4.7
TartuNLP |  | 0.0 | 0.3 | 2.2 | 2.7
Liv4ever |  |  | 0.0 | 1.9 | 2.4
HuaweiTSC |  |  |  | 0.0 | 0.5
NiuTrans |  |  |  |  | 0.0
Table 60: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$liv. | TAL. | Tar. | Hua. | Liv. | Niu.
---|---|---|---|---|---
TAL-SJTU | 0.0 | 7.3 | 9.3 | 9.9 | 52.4
TartuNLP |  | 0.0 | 2.0 | 2.6 | 45.1
HuaweiTSC |  |  | 0.0 | 0.6 | 43.1
Liv4ever |  |  |  | 0.0 | 42.5
NiuTrans |  |  |  |  | 0.0
Table 61: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$liv. | Onl. | JDE. | Lan. | Hua. | Onl. | Onl. | Onl. | SRP. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---
Online-G | 0.0 | 1.1 | 1.5 | 1.5 | 1.7 | 1.7 | 1.8 | 2.8 | 3.7 | 12.1
JDExploreAcademy |  | 0.0 | 0.4 | 0.4 | 0.6 | 0.6 | 0.7 | 1.7 | 2.6 | 11.0
Lan-Bridge |  |  | 0.0 | 0.0 | 0.2 | 0.2 | 0.3 | 1.3 | 2.2 | 10.6
HuaweiTSC |  |  |  | 0.0 | 0.2 | 0.2 | 0.3 | 1.3 | 2.2 | 10.6
Online-B |  |  |  |  | 0.0 | -0.0 | 0.1 | 1.1 | 2.0 | 10.4
Online-A |  |  |  |  |  | 0.0 | 0.1 | 1.1 | 2.0 | 10.4
Online-Y |  |  |  |  |  |  | 0.0 | 1.0 | 1.9 | 10.3
SRPOL |  |  |  |  |  |  |  | 0.0 | 0.9 | 9.3
Online-W |  |  |  |  |  |  |  |  | 0.0 | 8.4
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  | 0.0
Table 62: Statistical significance testing of the chrF score difference for each system pair for the ru$\rightarrow$en. | Onl. | Lan. | JDE. | Hua. | Onl. | Onl. | Onl. | SRP. | Onl. | ALM.
---|---|---|---|---|---|---|---|---|---|---
Online-G | 0.0 | 1.5 | 1.6 | 1.6 | 1.6 | 2.8 | 2.8 | 3.1 | 4.0 | 16.3
Lan-Bridge |  | 0.0 | 0.2 | 0.1 | 0.2 | 1.3 | 1.4 | 1.6 | 2.6 | 14.9
JDExploreAcademy |  |  | 0.0 | -0.0 | 0.0 | 1.2 | 1.2 | 1.5 | 2.4 | 14.7
HuaweiTSC |  |  |  | 0.0 | 0.1 | 1.2 | 1.2 | 1.5 | 2.4 | 14.8
Online-B |  |  |  |  | 0.0 | 1.1 | 1.2 | 1.4 | 2.4 | 14.7
Online-A |  |  |  |  |  | 0.0 | 0.1 | 0.3 | 1.3 | 13.6
Online-Y |  |  |  |  |  |  | 0.0 | 0.2 | 1.2 | 13.5
SRPOL |  |  |  |  |  |  |  | 0.0 | 1.0 | 13.3
Online-W |  |  |  |  |  |  |  |  | 0.0 | 12.3
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  | 0.0
Table 63: Statistical significance testing of the BLEU score difference for each system pair for the ru$\rightarrow$en. | Onl. | JDE. | Onl. | Lan. | Onl. | Onl. | Onl. | Hua. | SRP. | ALM.
---|---|---|---|---|---|---|---|---|---|---
Online-G | 0.0 | 0.3 | 1.1 | 2.1 | 2.1 | 3.0 | 3.6 | 4.3 | 5.7 | 38.3
JDExploreAcademy |  | 0.0 | 0.8 | 1.8 | 1.8 | 2.7 | 3.3 | 4.0 | 5.4 | 38.0
Online-Y |  |  | 0.0 | 1.0 | 1.0 | 1.9 | 2.5 | 3.2 | 4.6 | 37.2
Lan-Bridge |  |  |  | 0.0 | 0.0 | 0.9 | 1.5 | 2.2 | 3.6 | 36.2
Online-B |  |  |  |  | 0.0 | 0.9 | 1.5 | 2.2 | 3.6 | 36.2
Online-A |  |  |  |  |  | 0.0 | 0.6 | 1.3 | 2.7 | 35.3
Online-W |  |  |  |  |  |  | 0.0 | 0.7 | 2.1 | 34.7
HuaweiTSC |  |  |  |  |  |  |  | 0.0 | 1.4 | 34.0
SRPOL |  |  |  |  |  |  |  |  | 0.0 | 32.6
ALMAnaCH-Inria |  |  |  |  |  |  |  |  |  | 0.0
Table 64: Statistical significance testing of the COMET score difference for each system pair for the ru$\rightarrow$en. | Onl. | Onl. | Lan. | JDE. | Onl. | Onl. | Onl. | SRP. | PRO. | Hua. | eTr.
---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.3 | 0.7 | 1.3 | 1.4 | 1.4 | 1.6 | 3.3 | 3.6 | 3.6 | 3.9
Online-G |  | 0.0 | 0.5 | 1.0 | 1.1 | 1.1 | 1.3 | 3.0 | 3.4 | 3.4 | 3.6
Lan-Bridge |  |  | 0.0 | 0.6 | 0.7 | 0.7 | 0.9 | 2.6 | 2.9 | 2.9 | 3.1
JDExploreAcademy |  |  |  | 0.0 | 0.1 | 0.1 | 0.3 | 2.0 | 2.3 | 2.3 | 2.6
Online-W |  |  |  |  | 0.0 | 0.0 | 0.2 | 1.9 | 2.2 | 2.2 | 2.5
Online-Y |  |  |  |  |  | 0.0 | 0.2 | 1.9 | 2.2 | 2.2 | 2.5
Online-A |  |  |  |  |  |  | 0.0 | 1.7 | 2.0 | 2.0 | 2.3
SRPOL |  |  |  |  |  |  |  | 0.0 | 0.3 | 0.3 | 0.6
PROMT |  |  |  |  |  |  |  |  | 0.0 | -0.0 | 0.2
HuaweiTSC |  |  |  |  |  |  |  |  |  | 0.0 | 0.3
eTranslation |  |  |  |  |  |  |  |  |  |  | 0.0
Table 65: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$ru. | Onl. | Onl. | Onl. | Onl. | JDE. | Lan. | Onl. | Hua. | PRO. | SRP. | eTr.
---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 1.7 | 1.9 | 2.1 | 2.2 | 2.3 | 2.5 | 4.2 | 4.4 | 4.5 | 5.1
Online-Y |  | 0.0 | 0.2 | 0.4 | 0.5 | 0.6 | 0.8 | 2.5 | 2.7 | 2.9 | 3.4
Online-A |  |  | 0.0 | 0.3 | 0.4 | 0.5 | 0.7 | 2.3 | 2.5 | 2.7 | 3.2
Online-G |  |  |  | 0.0 | 0.1 | 0.2 | 0.4 | 2.0 | 2.2 | 2.4 | 3.0
JDExploreAcademy |  |  |  |  | 0.0 | 0.1 | 0.3 | 1.9 | 2.1 | 2.3 | 2.9
Lan-Bridge |  |  |  |  |  | 0.0 | 0.2 | 1.8 | 2.0 | 2.2 | 2.8
Online-W |  |  |  |  |  |  | 0.0 | 1.6 | 1.8 | 2.0 | 2.6
HuaweiTSC |  |  |  |  |  |  |  | 0.0 | 0.2 | 0.4 | 0.9
PROMT |  |  |  |  |  |  |  |  | 0.0 | 0.2 | 0.7
SRPOL |  |  |  |  |  |  |  |  |  | 0.0 | 0.6
eTranslation |  |  |  |  |  |  |  |  |  |  | 0.0
Table 66: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$ru. | Onl. | Onl. | Onl. | Onl. | JDE. | Lan. | Onl. | PRO. | SRP. | Hua. | eTr.
---|---|---|---|---|---|---|---|---|---|---|---
Online-W | 0.0 | 1.9 | 2.2 | 5.3 | 5.4 | 7.7 | 7.7 | 14.7 | 15.3 | 15.9 | 17.1
Online-G |  | 0.0 | 0.3 | 3.4 | 3.5 | 5.8 | 5.8 | 12.8 | 13.4 | 14.0 | 15.2
Online-B |  |  | 0.0 | 3.1 | 3.2 | 5.5 | 5.5 | 12.5 | 13.1 | 13.7 | 14.9
Online-Y |  |  |  | 0.0 | 0.1 | 2.4 | 2.4 | 9.4 | 10.0 | 10.6 | 11.8
JDExploreAcademy |  |  |  |  | 0.0 | 2.3 | 2.3 | 9.3 | 9.9 | 10.5 | 11.7
Lan-Bridge |  |  |  |  |  | 0.0 | 0.0 | 7.0 | 7.6 | 8.2 | 9.4
Online-A |  |  |  |  |  |  | 0.0 | 7.0 | 7.6 | 8.2 | 9.4
PROMT |  |  |  |  |  |  |  | 0.0 | 0.6 | 1.2 | 2.4
SRPOL |  |  |  |  |  |  |  |  | 0.0 | 0.6 | 1.8
HuaweiTSC |  |  |  |  |  |  |  |  |  | 0.0 | 1.2
eTranslation |  |  |  |  |  |  |  |  |  |  | 0.0
Table 67: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$ru. | Lan. | Onl.
---|---|---
Lan-Bridge | 0.0 | 9.7
Online-G |  | 0.0
Table 68: Statistical significance testing of the chrF score difference for each system pair for the ru$\rightarrow$sah. | Lan. | Onl.
---|---|---
Lan-Bridge | 0.0 | 0.6
Online-G |  | 0.0
Table 69: Statistical significance testing of the BLEU score difference for each system pair for the ru$\rightarrow$sah. | Onl. | Lan.
---|---|---
Online-G | 0.0 | 30.9
Lan-Bridge |  | 0.0
Table 70: Statistical significance testing of the COMET score difference for each system pair for the ru$\rightarrow$sah. | Onl. | Lan.
---|---|---
Online-G | 0.0 | 27.2
Lan-Bridge |  | 0.0
Table 71: Statistical significance testing of the chrF score difference for each system pair for the sah$\rightarrow$ru. | Onl. | Lan.
---|---|---
Online-G | 0.0 | 22.6
Lan-Bridge |  | 0.0
Table 72: Statistical significance testing of the BLEU score difference for each system pair for the sah$\rightarrow$ru. | Onl. | Lan.
---|---|---
Online-G | 0.0 | 107.0
Lan-Bridge |  | 0.0
Table 73: Statistical significance testing of the COMET score difference for each system pair for the sah$\rightarrow$ru. | Lan. | Onl. | Onl. | Onl. | PRO. | Onl. | ARC. | Hua. | ALM.
---|---|---|---|---|---|---|---|---|---
Lan-Bridge | 0.0 | 0.1 | 1.3 | 2.1 | 2.6 | 2.8 | 2.7 | 3.4 | 11.7
Online-B |  | 0.0 | 1.2 | 2.0 | 2.5 | 2.7 | 2.6 | 3.3 | 11.6
Online-G |  |  | 0.0 | 0.8 | 1.3 | 1.5 | 1.4 | 2.1 | 10.5
Online-A |  |  |  | 0.0 | 0.5 | 0.7 | 0.6 | 1.3 | 9.6
PROMT |  |  |  |  | 0.0 | 0.1 | 0.1 | 0.8 | 9.1
Online-Y |  |  |  |  |  | 0.0 | -0.1 | 0.6 | 9.0
ARC-NKUA |  |  |  |  |  |  | 0.0 | 0.7 | 9.0
HuaweiTSC |  |  |  |  |  |  |  | 0.0 | 8.3
ALMAnaCH-Inria |  |  |  |  |  |  |  |  | 0.0
Table 74: Statistical significance testing of the chrF score difference for each system pair for the uk$\rightarrow$en. | Lan. | Onl. | Onl. | Onl. | PRO. | ARC. | Onl. | Hua. | ALM.
---|---|---|---|---|---|---|---|---|---
Lan-Bridge | 0.0 | 0.1 | 1.4 | 2.3 | 2.5 | 2.7 | 2.8 | 2.9 | 14.6
Online-B |  | 0.0 | 1.2 | 2.2 | 2.4 | 2.6 | 2.6 | 2.8 | 14.5
Online-G |  |  | 0.0 | 0.9 | 1.2 | 1.4 | 1.4 | 1.6 | 13.3
Online-A |  |  |  | 0.0 | 0.2 | 0.4 | 0.5 | 0.6 | 12.3
PROMT |  |  |  |  | 0.0 | 0.2 | 0.3 | 0.4 | 12.1
ARC-NKUA |  |  |  |  |  | 0.0 | 0.0 | 0.2 | 11.9
Online-Y |  |  |  |  |  |  | 0.0 | 0.2 | 11.9
HuaweiTSC |  |  |  |  |  |  |  | 0.0 | 11.7
ALMAnaCH-Inria |  |  |  |  |  |  |  |  | 0.0
Table 75: Statistical significance testing of the BLEU score difference for each system pair for the uk$\rightarrow$en. | Onl. | Lan. | Onl. | Onl. | Hua. | Onl. | PRO. | ARC. | ALM.
---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.2 | 5.1 | 10.4 | 12.5 | 12.8 | 13.0 | 13.0 | 40.8
Lan-Bridge |  | 0.0 | 4.9 | 10.2 | 12.3 | 12.6 | 12.8 | 12.8 | 40.6
Online-G |  |  | 0.0 | 5.3 | 7.4 | 7.7 | 7.9 | 7.9 | 35.7
Online-A |  |  |  | 0.0 | 2.1 | 2.4 | 2.6 | 2.6 | 30.4
HuaweiTSC |  |  |  |  | 0.0 | 0.3 | 0.5 | 0.5 | 28.3
Online-Y |  |  |  |  |  | 0.0 | 0.2 | 0.2 | 28.0
PROMT |  |  |  |  |  |  | 0.0 | 0.0 | 27.8
ARC-NKUA |  |  |  |  |  |  |  | 0.0 | 27.8
ALMAnaCH-Inria |  |  |  |  |  |  |  |  | 0.0
Table 76: Statistical significance testing of the COMET score difference for each system pair for the uk$\rightarrow$en. | Onl. | Lan. | Onl. | Onl. | Onl. | eTr. | Hua. | ARC.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 0.5 | 2.0 | 3.3 | 4.4 | 4.4 | 4.5 | 5.3
Lan-Bridge |  | 0.0 | 1.5 | 2.8 | 3.9 | 3.9 | 4.0 | 4.8
Online-G |  |  | 0.0 | 1.3 | 2.4 | 2.4 | 2.4 | 3.2
Online-A |  |  |  | 0.0 | 1.1 | 1.1 | 1.2 | 2.0
Online-Y |  |  |  |  | 0.0 | 0.0 | 0.1 | 0.9
eTranslation |  |  |  |  |  | 0.0 | 0.0 | 0.8
HuaweiTSC |  |  |  |  |  |  | 0.0 | 0.8
ARC-NKUA |  |  |  |  |  |  |  | 0.0
Table 77: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$uk. | Onl. | Lan. | Onl. | Onl. | Onl. | Hua. | eTr. | ARC.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 3.1 | 4.5 | 5.3 | 5.6 | 6.0 | 6.3 | 7.4
Lan-Bridge |  | 0.0 | 1.5 | 2.2 | 2.6 | 3.0 | 3.3 | 4.3
Online-A |  |  | 0.0 | 0.8 | 1.1 | 1.5 | 1.8 | 2.8
Online-G |  |  |  | 0.0 | 0.3 | 0.7 | 1.0 | 2.0
Online-Y |  |  |  |  | 0.0 | 0.4 | 0.7 | 1.7
HuaweiTSC |  |  |  |  |  | 0.0 | 0.3 | 1.3
eTranslation |  |  |  |  |  |  | 0.0 | 1.0
ARC-NKUA |  |  |  |  |  |  |  | 0.0
Table 78: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$uk. | Onl. | Onl. | Lan. | Onl. | eTr. | Hua. | Onl. | ARC.
---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 3.4 | 7.5 | 12.4 | 18.7 | 18.9 | 21.3 | 24.1
Online-G |  | 0.0 | 4.1 | 9.0 | 15.3 | 15.5 | 17.9 | 20.7
Lan-Bridge |  |  | 0.0 | 4.9 | 11.2 | 11.4 | 13.8 | 16.6
Online-A |  |  |  | 0.0 | 6.3 | 6.5 | 8.9 | 11.7
eTranslation |  |  |  |  | 0.0 | 0.2 | 2.6 | 5.4
HuaweiTSC |  |  |  |  |  | 0.0 | 2.4 | 5.2
Online-Y |  |  |  |  |  |  | 0.0 | 2.8
ARC-NKUA |  |  |  |  |  |  |  | 0.0
Table 79: Statistical significance testing of the COMET score difference for each system pair for the en$\rightarrow$uk. | JDE. | Lan. | Onl. | AIS. | Hua. | Onl. | Lan. | Onl. | Onl. | Niu. | DLU. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---
JDExploreAcademy | 0.0 | 0.7 | 1.5 | 1.9 | 2.8 | 3.0 | 3.3 | 3.8 | 4.0 | 5.2 | 6.1 | 6.8
LanguageX |  | 0.0 | 0.8 | 1.1 | 2.1 | 2.3 | 2.6 | 3.0 | 3.2 | 4.5 | 5.3 | 6.0
Online-G |  |  | 0.0 | 0.3 | 1.3 | 1.5 | 1.8 | 2.2 | 2.4 | 3.7 | 4.5 | 5.2
AISP-SJTU |  |  |  | 0.0 | 1.0 | 1.2 | 1.5 | 1.9 | 2.1 | 3.3 | 4.2 | 4.9
HuaweiTSC |  |  |  |  | 0.0 | 0.2 | 0.5 | 0.9 | 1.2 | 2.4 | 3.3 | 4.0
Online-B |  |  |  |  |  | 0.0 | 0.3 | 0.8 | 1.0 | 2.2 | 3.1 | 3.8
Lan-Bridge |  |  |  |  |  |  | 0.0 | 0.5 | 0.7 | 1.9 | 2.8 | 3.5
Online-Y |  |  |  |  |  |  |  | 0.0 | 0.2 | 1.4 | 2.3 | 3.0
Online-A |  |  |  |  |  |  |  |  | 0.0 | 1.2 | 2.1 | 2.8
NiuTrans |  |  |  |  |  |  |  |  |  | 0.0 | 0.9 | 1.6
DLUT |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.7
Online-W |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 80: Statistical significance testing of the chrF score difference for each system pair for the zh$\rightarrow$en. | JDE. | Lan. | AIS. | Hua. | Onl. | Onl. | Lan. | Onl. | Onl. | Niu. | DLU. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---
JDExploreAcademy | 0.0 | 1.5 | 3.5 | 3.5 | 3.8 | 4.6 | 5.0 | 6.3 | 6.8 | 7.4 | 8.9 | 9.8
LanguageX |  | 0.0 | 2.0 | 2.1 | 2.3 | 3.1 | 3.5 | 4.8 | 5.3 | 5.9 | 7.4 | 8.3
AISP-SJTU |  |  | 0.0 | 0.0 | 0.3 | 1.0 | 1.5 | 2.7 | 3.2 | 3.8 | 5.3 | 6.3
HuaweiTSC |  |  |  | 0.0 | 0.3 | 1.0 | 1.4 | 2.7 | 3.2 | 3.8 | 5.3 | 6.3
Online-G |  |  |  |  | 0.0 | 0.8 | 1.2 | 2.5 | 3.0 | 3.6 | 5.1 | 6.0
Online-B |  |  |  |  |  | 0.0 | 0.4 | 1.7 | 2.2 | 2.8 | 4.3 | 5.2
Lan-Bridge |  |  |  |  |  |  | 0.0 | 1.3 | 1.8 | 2.4 | 3.9 | 4.8
Online-A |  |  |  |  |  |  |  | 0.0 | 0.5 | 1.1 | 2.6 | 3.5
Online-Y |  |  |  |  |  |  |  |  | 0.0 | 0.6 | 2.1 | 3.0
NiuTrans |  |  |  |  |  |  |  |  |  | 0.0 | 1.5 | 2.4
DLUT |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.9
Online-W |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 81: Statistical significance testing of the BLEU score difference for each system pair for the zh$\rightarrow$en. | Onl. | JDE. | Lan. | Lan. | Hua. | Onl. | AIS. | Onl. | Onl. | Onl. | Niu. | DLU.
---|---|---|---|---|---|---|---|---|---|---|---|---
Online-G | 0.0 | 0.5 | 0.7 | 2.6 | 2.8 | 3.5 | 4.0 | 4.8 | 10.4 | 14.0 | 14.3 | 15.0
JDExploreAcademy |  | 0.0 | 0.2 | 2.1 | 2.3 | 3.0 | 3.5 | 4.3 | 9.9 | 13.5 | 13.8 | 14.5
LanguageX |  |  | 0.0 | 1.9 | 2.1 | 2.8 | 3.3 | 4.1 | 9.7 | 13.3 | 13.6 | 14.3
Lan-Bridge |  |  |  | 0.0 | 0.2 | 0.9 | 1.4 | 2.2 | 7.8 | 11.4 | 11.7 | 12.4
HuaweiTSC |  |  |  |  | 0.0 | 0.7 | 1.2 | 2.0 | 7.6 | 11.2 | 11.5 | 12.2
Online-B |  |  |  |  |  | 0.0 | 0.5 | 1.3 | 6.9 | 10.5 | 10.8 | 11.5
AISP-SJTU |  |  |  |  |  |  | 0.0 | 0.8 | 6.4 | 10.0 | 10.3 | 11.0
Online-Y |  |  |  |  |  |  |  | 0.0 | 5.6 | 9.2 | 9.5 | 10.2
Online-A |  |  |  |  |  |  |  |  | 0.0 | 3.6 | 3.9 | 4.6
Online-W |  |  |  |  |  |  |  |  |  | 0.0 | 0.3 | 1.0
NiuTrans |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.7
DLUT |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 82: Statistical significance testing of the COMET score difference for each system pair for the zh$\rightarrow$en. | Onl. | Hua. | Lan. | Onl. | Man. | Onl. | JDE. | AIS. | Lan. | DLU. | Onl. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 9.3 | 10.8 | 12.3 | 12.7 | 13.4 | 14.5 | 15.0 | 16.7 | 17.2 | 17.9 | 18.9
HuaweiTSC |  | 0.0 | 1.5 | 2.9 | 3.3 | 4.0 | 5.2 | 5.6 | 7.3 | 7.9 | 8.6 | 9.6
LanguageX |  |  | 0.0 | 1.5 | 1.9 | 2.6 | 3.7 | 4.1 | 5.9 | 6.4 | 7.1 | 8.1
Online-A |  |  |  | 0.0 | 0.4 | 1.1 | 2.2 | 2.7 | 4.4 | 5.0 | 5.7 | 6.6
Manifold |  |  |  |  | 0.0 | 0.7 | 1.8 | 2.3 | 4.0 | 4.6 | 5.2 | 6.2
Online-Y |  |  |  |  |  | 0.0 | 1.1 | 1.6 | 3.3 | 3.8 | 4.5 | 5.5
JDExploreAcademy |  |  |  |  |  |  | 0.0 | 0.5 | 2.2 | 2.7 | 3.4 | 4.4
AISP-SJTU |  |  |  |  |  |  |  | 0.0 | 1.7 | 2.3 | 3.0 | 4.0
Lan-Bridge |  |  |  |  |  |  |  |  | 0.0 | 0.6 | 1.2 | 2.2
DLUT |  |  |  |  |  |  |  |  |  | 0.0 | 0.7 | 1.7
Online-G |  |  |  |  |  |  |  |  |  |  | 0.0 | 1.0
Online-W |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 83: Statistical significance testing of the chrF score difference for each system pair for the en$\rightarrow$zh. | Onl. | Hua. | Lan. | Man. | Onl. | JDE. | Onl. | AIS. | Lan. | DLU. | Onl. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---
Online-B | 0.0 | 6.7 | 7.1 | 10.5 | 11.4 | 11.6 | 11.7 | 12.4 | 13.4 | 16.1 | 16.7 | 18.0
HuaweiTSC |  | 0.0 | 0.5 | 3.9 | 4.8 | 4.9 | 5.1 | 5.7 | 6.8 | 9.5 | 10.1 | 11.4
LanguageX |  |  | 0.0 | 3.4 | 4.3 | 4.5 | 4.6 | 5.3 | 6.3 | 9.0 | 9.6 | 10.9
Manifold |  |  |  | 0.0 | 0.9 | 1.0 | 1.2 | 1.8 | 2.9 | 5.6 | 6.2 | 7.5
Online-A |  |  |  |  | 0.0 | 0.1 | 0.3 | 0.9 | 2.0 | 4.7 | 5.3 | 6.6
JDExploreAcademy |  |  |  |  |  | 0.0 | 0.1 | 0.8 | 1.9 | 4.5 | 5.2 | 6.5
Online-Y |  |  |  |  |  |  | 0.0 | 0.7 | 1.7 | 4.4 | 5.0 | 6.3
AISP-SJTU |  |  |  |  |  |  |  | 0.0 | 1.1 | 3.7 | 4.4 | 5.6
Lan-Bridge |  |  |  |  |  |  |  |  | 0.0 | 2.7 | 3.3 | 4.6
DLUT |  |  |  |  |  |  |  |  |  | 0.0 | 0.6 | 1.9
Online-G |  |  |  |  |  |  |  |  |  |  | 0.0 | 1.3
Online-W |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 84: Statistical significance testing of the BLEU score difference for each system pair for the en$\rightarrow$zh. | Lan. | Onl. | JDE. | Lan. | Onl. | Man. | Onl. | Hua. | Onl. | AIS. | DLU. | Onl.
---|---|---|---|---|---|---|---|---|---|---|---|---
LanguageX | 0.0 | 2.1 | 2.1 | 2.5 | 2.9 | 3.7 | 4.1 | 4.3 | 6.5 | 7.3 | 11.7 | 12.6
Online-B |  | 0.0 | 0.0 | 0.4 | 0.8 | 1.6 | 2.0 | 2.2 | 4.4 | 5.2 | 9.6 | 10.5
JDExploreAcademy |  |  | 0.0 | 0.4 | 0.8 | 1.6 | 2.0 | 2.2 | 4.4 | 5.2 | 9.6 | 10.5
Lan-Bridge |  |  |  | 0.0 | 0.4 | 1.2 | 1.6 | 1.8 | 4.0 | 4.8 | 9.2 | 10.1
Online-W |  |  |  |  | 0.0 | 0.8 | 1.2 | 1.4 | 3.6 | 4.4 | 8.8 | 9.7
Manifold |  |  |  |  |  | 0.0 | 0.4 | 0.6 | 2.8 | 3.6 | 8.0 | 8.9
Online-Y |  |  |  |  |  |  | 0.0 | 0.2 | 2.4 | 3.2 | 7.6 | 8.5
HuaweiTSC |  |  |  |  |  |  |  | 0.0 | 2.2 | 3.0 | 7.4 | 8.3
Online-A |  |  |  |  |  |  |  |  | 0.0 | 0.8 | 5.2 | 6.1
AISP-SJTU |  |  |  |  |  |  |  |  |  | 0.0 | 4.4 | 5.3
DLUT |  |  |  |  |  |  |  |  |  |  | 0.0 | 0.9
Online-G |  |  |  |  |  |  |  |  |  |  |  | 0.0
Table 85:  Statistical significance testing of the COMET score difference for
each system pair for the en$\rightarrow$zh.

##  Appendix C MBR and Oracle Combination Tables

From Table 86 to 96.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 77.5 | 79.3 | 64.2
MBR | 77.3 | 75.4 | 56.5
Oracle | 86.9 | 82.9 | 69.2
(a) cs$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 97.9 | 69.0 | 48.2
MBR | 102.5 | 67.2 | 44.6
Oracle | 111.3 | 75.9 | 57.8
(b) en$\rightarrow$cs

Table 86: MBR and oracle combination using COMET for cs-en.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 99.4 | 64.0 | 38.3
MBR | 104.3 | 64.9 | 38.8
Oracle | 110.8 | 70.8 | 46.5
(a) cs$\rightarrow$uk

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 104.9 | 60.7 | 37.0
MBR | 107.0 | 62.1 | 38.9
Oracle | 114.2 | 67.8 | 46.6
(b) uk$\rightarrow$cs

Table 87: MBR and oracle combination using COMET for cs-uk.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 58.0 | 58.7 | 33.7
MBR | 60.3 | 59.3 | 34.2
Oracle | 69.0 | 62.5 | 38.8
(a) de$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 65.6 | 64.6 | 38.4
MBR | 66.2 | 64.9 | 38.5
Oracle | 74.6 | 69.1 | 44.5
(b) en$\rightarrow$de

Table 88: MBR and oracle combination using COMET for de-en.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 70.5 | 74.6 | 58.4
MBR | 70.8 | 69.0 | 49.1
Oracle | 84.3 | 75.2 | 58.8
(a) de$\rightarrow$fr

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 77.9 | 81.2 | 64.8
MBR | 74.3 | 72.5 | 52.6
Oracle | 86.5 | 82.7 | 67.8
(b) fr$\rightarrow$de

Table 89: MBR and oracle combination using COMET for de-fr. System | COMET | chrF | BLEU
---|---|---|---
Baseline | 80.4 | 58.5 | 31.5
MBR | 84.6 | 58.2 | 31.0
Oracle | 92.8 | 60.8 | 34.6
(a) en$\rightarrow$hr Table 90: MBR and oracle combination using COMET for en-
hr.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 42.1 | 51.7 | 27.8
MBR | 48.6 | 51.4 | 26.2
Oracle | 66.2 | 57.0 | 33.7
(a) ja$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 65.2 | 36.8 | 27.6
MBR | 69.7 | 36.3 | 27.0
Oracle | 82.7 | 42.0 | 33.2
(b) en$\rightarrow$ja

Table 91: MBR and oracle combination using COMET for en-ja.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | -5.7 | 53.5 | 30.4
MBR | 4.3 | 54.4 | 31.2
Oracle | 11.1 | 56.0 | 33.1
(a) liv$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | -29.5 | 43.8 | 17.0
MBR | -24.9 | 42.4 | 16.1
Oracle | -14.6 | 44.4 | 17.9
(b) en$\rightarrow$liv

Table 92: MBR and oracle combination using COMET for en-liv.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 75.1 | 59.7 | 34.9
MBR | 77.9 | 59.8 | 34.5
Oracle | 87.6 | 64.4 | 40.1
(a) en$\rightarrow$ru

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 65.2 | 70.0 | 46.7
MBR | 68.0 | 70.0 | 46.8
Oracle | 76.4 | 73.6 | 53.1
(b) ru$\rightarrow$en

Table 93: MBR and oracle combination using COMET for en-ru.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | -17.1 | 56.6 | 15.3
MBR | -41.7 | 50.2 | 13.3
Oracle | -5.6 | 60.6 | 22.9
(a) ru$\rightarrow$sah

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 31.1 | 55.5 | 29.6
MBR | 20.1 | 51.8 | 26.1
Oracle | 32.9 | 55.5 | 29.7
(b) sah$\rightarrow$ru

Table 94: MBR and oracle combination using COMET for ru-sah.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 62.6 | 67.3 | 44.6
MBR | 62.0 | 67.4 | 45.1
Oracle | 71.4 | 71.6 | 51.0
(a) uk$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 73.3 | 59.3 | 32.5
MBR | 73.5 | 58.0 | 30.0
Oracle | 84.2 | 62.1 | 34.7
(b) en$\rightarrow$uk

Table 95: MBR and oracle combination using COMET for en-uk.

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 45.7 | 61.1 | 33.5
MBR | 52.9 | 60.5 | 31.8
Oracle | 64.8 | 63.9 | 36.2
(a) zh$\rightarrow$en

System | COMET | chrF | BLEU
---|---|---|---
Baseline | 63.9 | 49.1 | 54.3
MBR | 67.3 | 46.7 | 51.3
Oracle | 77.7 | 53.6 | 57.8
(b) en$\rightarrow$zh

Table 96: MBR and oracle combination using COMET for en-zh.

##  Appendix D Rankings With Different COMET Models

From Table 97 to 107.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-W | 77.5 (1) | 11.4 (1) | 4.5 (1) | 11.3 (1) | 7.9 (3)
JDExploreAcademy | 74.8 (2) | 10.1 (2) | 4.5 (2) | 11.3 (2) | 8.3 (1)
Online-B | 71.9 (3) | 9.0 (4) | 4.4 (4) | 11.2 (3) | 8.0 (2)
Lan-Bridge | 71.8 (4) | 9.0 (3) | 4.4 (3) | 11.1 (4) | 7.9 (4)
CUNI-DocTransformer | 70.6 (5) | 8.4 (5) | 4.3 (5) | 11.0 (5) | 7.2 (5)
Online-A | 69.8 (6) | 8.2 (6) | 4.3 (6) | 11.0 (6) | 7.0 (6)
CUNI-Transformer | 69.2 (7) | 7.9 (7) | 4.3 (7) | 11.0 (7) | 7.0 (7)
Online-G | 63.0 (8) | 6.0 (8) | 4.2 (8) | 10.7 (8) | 5.7 (9)
SHOPLINE-PL | 61.2 (9) | 5.3 (9) | 4.1 (9) | 10.7 (9) | 5.8 (8)
Online-Y | 58.7 (10) | 4.3 (10) | 4.1 (10) | 10.6 (10) | 5.3 (10)
ALMAnaCH-Inria | 19.3 (11) | -7.1 (11) | 3.4 (11) | 9.5 (11) | -1.4 (11)
(a) cs$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-W | 97.9 (1) | 15.9 (2) | 4.9 (2) | 12.2 (1) | 13.9 (1)
Online-B | 97.5 (2) | 16.2 (1) | 4.9 (1) | 11.9 (4) | 12.7 (4)
CUNI-Bergamot | 96.1 (3) | 13.9 (5) | 4.8 (3) | 12.0 (3) | 13.4 (2)
JDExploreAcademy | 95.3 (4) | 15.1 (4) | 4.8 (4) | 12.1 (2) | 13.4 (3)
Lan-Bridge | 94.7 (5) | 15.4 (3) | 4.8 (5) | 11.8 (6) | 12.5 (5)
Online-A | 92.2 (6) | 13.5 (6) | 4.7 (7) | 11.7 (8) | 11.9 (7)
CUNI-DocTransformer | 91.7 (7) | 13.3 (7) | 4.7 (6) | 11.8 (5) | 12.2 (6)
CUNI-Transformer | 86.7 (8) | 11.9 (8) | 4.6 (8) | 11.7 (7) | 11.6 (8)
Online-Y | 83.7 (9) | 10.7 (9) | 4.5 (10) | 11.3 (10) | 9.7 (10)
Online-G | 82.3 (10) | 10.5 (10) | 4.5 (9) | 11.3 (9) | 9.7 (9)
(b) en$\rightarrow$cs

Table 97:  COMET scores (ranking) with different models for cs-en.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
AMU | 99.4 (1) | 14.1 (1) | 4.6 (1) | 12.5 (1) | 14.1 (1)
Online-B | 94.3 (2) | 13.6 (2) | 4.5 (2) | 12.0 (2) | 11.6 (2)
Lan-Bridge | 91.8 (3) | 12.7 (3) | 4.4 (3) | 11.9 (3) | 11.0 (3)
CharlesTranslator | 90.8 (4) | 11.8 (5) | 4.4 (4) | 11.7 (7) | 10.7 (5)
HuaweiTSC | 90.7 (5) | 12.2 (4) | 4.4 (5) | 11.9 (4) | 11.0 (4)
CUNI-JL-JH | 90.1 (6) | 11.3 (7) | 4.3 (6) | 11.7 (8) | 10.4 (7)
Online-G | 88.4 (7) | 11.3 (8) | 4.3 (7) | 11.9 (5) | 10.5 (6)
Online-A | 87.9 (8) | 11.4 (6) | 4.3 (8) | 11.8 (6) | 10.1 (9)
CUNI-Transformer | 87.4 (9) | 11.0 (9) | 4.3 (9) | 11.7 (9) | 10.3 (8)
Online-Y | 78.4 (10) | 8.4 (10) | 4.1 (10) | 11.3 (10) | 7.9 (10)
ALMAnaCH-Inria | 61.4 (11) | 3.4 (11) | 3.8 (11) | 10.8 (11) | 4.8 (11)
(a) cs$\rightarrow$uk

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
AMU | 104.9 (1) | 15.7 (1) | 5.0 (1) | 12.9 (1) | 13.8 (1)
Online-B | 96.5 (2) | 13.8 (2) | 4.8 (2) | 12.5 (2) | 11.4 (2)
Lan-Bridge | 94.6 (3) | 13.3 (3) | 4.7 (3) | 12.4 (3) | 11.0 (3)
HuaweiTSC | 91.5 (4) | 12.0 (4) | 4.6 (4) | 12.2 (4) | 10.6 (4)
CharlesTranslator | 90.2 (5) | 11.4 (5) | 4.6 (5) | 12.1 (5) | 10.2 (5)
CUNI-JL-JH | 89.1 (6) | 11.0 (7) | 4.6 (6) | 12.1 (6) | 9.9 (7)
CUNI-Transformer | 88.5 (7) | 11.2 (6) | 4.6 (7) | 12.1 (7) | 10.1 (6)
Online-A | 85.4 (8) | 10.2 (8) | 4.5 (8) | 12.1 (8) | 9.3 (8)
Online-G | 84.2 (9) | 9.7 (9) | 4.4 (9) | 11.9 (9) | 8.9 (9)
Online-Y | 78.7 (10) | 8.0 (10) | 4.3 (10) | 11.8 (10) | 7.9 (10)
ALMAnaCH-Inria | 62.5 (11) | 3.1 (11) | 4.0 (11) | 11.2 (11) | 4.8 (11)
(b) uk$\rightarrow$cs

Table 98: COMET scores (ranking) with different models for cs-uk.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
JDExploreAcademy | 58.0 (1) | 4.6 (1) | 4.1 (1) | 11.6 (1) | 9.8 (1)
Online-B | 57.0 (2) | 4.2 (2) | 4.1 (2) | 11.5 (2) | 9.3 (2)
Lan-Bridge | 56.6 (3) | 4.1 (3) | 4.1 (3) | 11.5 (3) | 9.2 (3)
Online-G | 55.3 (4) | 3.7 (4) | 4.1 (4) | 11.4 (4) | 8.9 (4)
Online-Y | 54.7 (5) | 3.5 (7) | 4.0 (5) | 11.4 (5) | 8.9 (5)
Online-A | 54.5 (6) | 3.6 (5) | 4.0 (6) | 11.4 (6) | 8.8 (7)
Online-W | 54.4 (7) | 3.6 (6) | 4.0 (7) | 11.4 (7) | 8.9 (6)
PROMT | 51.8 (8) | 2.6 (8) | 4.0 (8) | 11.2 (8) | 8.2 (8)
LT22 | 25.6 (9) | -4.7 (9) | 3.5 (9) | 10.6 (9) | 3.3 (9)
(a) de$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-W | 65.6 (1) | 9.7 (1) | 4.6 (1) | 11.9 (1) | 12.2 (1)
JDExploreAcademy | 63.3 (2) | 8.8 (2) | 4.6 (2) | 11.8 (2) | 12.0 (2)
Online-B | 62.3 (3) | 8.6 (3) | 4.5 (3) | 11.7 (3) | 11.8 (3)
Online-Y | 61.1 (4) | 7.9 (4) | 4.5 (4) | 11.7 (4) | 11.5 (4)
Online-A | 60.6 (5) | 7.8 (5) | 4.5 (5) | 11.6 (5) | 11.4 (5)
Online-G | 60.2 (6) | 7.7 (7) | 4.5 (6) | 11.6 (6) | 11.3 (6)
Lan-Bridge | 58.8 (7) | 7.8 (6) | 4.5 (7) | 11.6 (7) | 11.0 (7)
OpenNMT | 57.3 (8) | 6.5 (8) | 4.4 (8) | 11.6 (8) | 10.9 (8)
PROMT | 55.9 (9) | 6.2 (9) | 4.4 (9) | 11.4 (9) | 10.4 (9)
(b) en$\rightarrow$de

Table 99: COMET scores (ranking) with different models for de-en.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-B | 70.5 (1) | 8.4 (1) | 4.4 (1) | 11.6 (2) | 6.7 (2)
Online-W | 63.7 (2) | 6.1 (2) | 4.3 (2) | 11.9 (1) | 8.0 (1)
Online-Y | 57.8 (3) | 4.3 (3) | 4.1 (3) | 11.6 (3) | 6.3 (3)
Online-A | 52.3 (4) | 3.4 (4) | 4.1 (4) | 11.5 (4) | 5.9 (4)
Online-G | 44.9 (5) | 1.1 (5) | 3.9 (5) | 11.2 (5) | 4.9 (5)
LT22 | 10.5 (6) | -6.8 (6) | 3.5 (6) | 10.7 (6) | 0.9 (6)
(a) de$\rightarrow$fr

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-W | 77.9 (1) | 13.6 (1) | 4.7 (1) | 11.5 (1) | 7.2 (1)
Online-B | 63.8 (2) | 8.0 (2) | 4.3 (2) | 11.3 (2) | 6.0 (2)
Online-Y | 61.6 (3) | 7.1 (3) | 4.3 (3) | 11.3 (3) | 5.8 (3)
Online-A | 59.3 (4) | 6.7 (4) | 4.2 (4) | 11.2 (4) | 5.5 (4)
eTranslation | 55.5 (5) | 5.8 (5) | 4.2 (5) | 11.0 (5) | 4.9 (5)
Lan-Bridge | 51.2 (6) | 4.3 (6) | 4.1 (6) | 11.0 (6) | 4.6 (6)
Online-G | 48.3 (7) | 3.5 (7) | 4.0 (7) | 10.6 (7) | 3.3 (7)
(b) fr$\rightarrow$de

Table 100: COMET scores (ranking) with different models for de-fr. System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-B | 80.4 (1) | 8.8 (1) | 4.3 (1) | 11.1 (1) | 11.1 (1)
Lan-Bridge | 79.6 (2) | 8.6 (2) | 4.3 (2) | 11.1 (2) | 10.9 (2)
Online-A | 69.5 (3) | 5.5 (4) | 4.1 (3) | 10.7 (5) | 8.8 (5)
SRPOL | 69.5 (4) | 5.6 (3) | 4.1 (4) | 10.8 (4) | 9.0 (3)
HuaweiTSC | 67.7 (5) | 5.3 (5) | 4.1 (5) | 10.7 (6) | 8.9 (4)
NiuTrans | 65.6 (6) | 4.9 (6) | 4.0 (6) | 10.7 (7) | 8.3 (7)
Online-G | 64.3 (7) | 4.2 (7) | 4.0 (7) | 11.0 (3) | 8.8 (6)
Online-Y | 56.8 (8) | 2.0 (8) | 3.8 (8) | 10.2 (8) | 6.3 (8)
(a) en$\rightarrow$hr Table 101: COMET scores (ranking) with different models
for en-hr.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
NT5 | 42.1 (1) | 1.1 (1) | 3.7 (1) | 10.8 (1) | 5.9 (2)
Online-W | 41.2 (2) | 1.1 (2) | 3.7 (2) | 10.8 (2) | 5.6 (3)
JDExploreAcademy | 40.6 (3) | 0.8 (3) | 3.7 (3) | 10.8 (3) | 6.1 (1)
Online-B | 39.7 (4) | 0.6 (4) | 3.6 (4) | 10.6 (4) | 5.1 (4)
DLUT | 37.3 (5) | -0.3 (5) | 3.6 (5) | 10.5 (5) | 4.8 (5)
NAIST-NICT-TIT | 33.4 (6) | -1.5 (6) | 3.5 (6) | 10.4 (6) | 4.4 (7)
LanguageX | 33.0 (7) | -1.5 (7) | 3.5 (8) | 10.2 (10) | 3.6 (9)
Online-A | 32.9 (8) | -1.6 (8) | 3.5 (7) | 10.3 (7) | 4.2 (8)
Online-Y | 32.4 (9) | -1.7 (9) | 3.5 (9) | 10.3 (8) | 4.5 (6)
Lan-Bridge | 32.0 (10) | -1.7 (10) | 3.4 (10) | 10.2 (11) | 3.6 (10)
AISP-SJTU | 30.2 (11) | -2.3 (11) | 3.4 (11) | 10.3 (9) | 3.6 (11)
Online-G | 22.3 (12) | -4.3 (12) | 3.3 (12) | 9.9 (13) | 2.1 (12)
KYB | 17.4 (13) | -5.5 (13) | 3.2 (13) | 10.0 (12) | 1.9 (13)
AIST | -152.7 (14) | -65.3 (14) | -0.6 (14) | 4.0 (14) | -38.0 (14)
(a) ja$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
JDExploreAcademy | 65.2 (1) | 10.4 (1) | 4.4 (1) | 12.1 (1) | 13.3 (1)
NT5 | 64.1 (2) | 10.2 (2) | 4.4 (2) | 12.0 (2) | 13.0 (2)
LanguageX | 62.1 (3) | 9.4 (3) | 4.4 (3) | 11.9 (4) | 12.2 (5)
Online-B | 60.8 (4) | 9.4 (4) | 4.4 (4) | 11.9 (5) | 12.5 (3)
DLUT | 60.5 (5) | 9.1 (5) | 4.3 (6) | 11.9 (6) | 12.4 (4)
Online-W | 59.8 (6) | 9.1 (6) | 4.4 (5) | 12.0 (3) | 12.2 (6)
Online-Y | 56.9 (7) | 8.1 (7) | 4.2 (8) | 11.8 (7) | 11.7 (8)
Lan-Bridge | 56.6 (8) | 8.0 (8) | 4.3 (7) | 11.8 (8) | 11.7 (9)
Online-A | 53.7 (9) | 7.5 (9) | 4.2 (9) | 11.8 (9) | 11.9 (7)
NAIST-NICT-TIT | 53.4 (10) | 7.4 (10) | 4.2 (10) | 11.7 (10) | 11.6 (10)
AISP-SJTU | 52.4 (11) | 7.0 (11) | 4.2 (11) | 11.7 (11) | 11.0 (11)
KYB | 31.8 (12) | 1.5 (12) | 3.8 (12) | 11.2 (12) | 7.8 (12)
Online-G | 25.0 (13) | 0.5 (13) | 3.7 (13) | 11.0 (13) | 6.7 (13)
(b) en$\rightarrow$ja

Table 102: COMET scores (ranking) with different models for en-ja.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
TAL-SJTU | -29.5 (1) | -15.5 (1) | 0.7 (1) | 2.9 (2) | -40.0 (3)
TartuNLP | -36.8 (2) | -17.1 (2) | 0.5 (3) | 2.9 (3) | -40.2 (4)
HuaweiTSC | -38.8 (3) | -17.7 (3) | 0.4 (4) | 2.8 (4) | -41.7 (5)
Liv4ever | -39.4 (4) | -18.4 (4) | 0.6 (2) | 3.2 (1) | -37.6 (1)
NiuTrans | -81.9 (5) | -31.3 (5) | 0.1 (5) | 2.8 (5) | -38.4 (2)
(a) en$\rightarrow$liv

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
TartuNLP | -5.7 (1) | -12.3 (1) | 2.6 (1) | 6.9 (1) | -18.1 (1)
TAL-SJTU | -8.3 (2) | -12.6 (2) | 2.6 (2) | 6.9 (2) | -18.2 (2)
HuaweiTSC | -27.2 (3) | -18.5 (3) | 2.2 (3) | 6.2 (3) | -20.4 (3)
Liv4ever | -44.0 (4) | -23.0 (4) | 1.9 (4) | 5.6 (4) | -21.4 (4)
NiuTrans | -88.3 (5) | -36.6 (5) | 1.0 (5) | 4.7 (5) | -25.1 (5)
(b) liv$\rightarrow$en

Table 103: COMET scores (ranking) with different models for en-liv.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-G | 65.2 (1) | 8.2 (1) | 4.4 (1) | 11.3 (2) | 8.1 (2)
JDExploreAcademy | 64.9 (2) | 8.1 (2) | 4.4 (2) | 11.5 (1) | 8.6 (1)
Online-Y | 64.1 (3) | 7.7 (3) | 4.4 (3) | 11.3 (3) | 7.9 (3)
Lan-Bridge | 63.1 (4) | 7.4 (4) | 4.4 (4) | 11.3 (4) | 7.9 (4)
Online-B | 63.1 (5) | 7.4 (5) | 4.4 (5) | 11.3 (5) | 7.9 (5)
Online-A | 62.2 (6) | 7.1 (6) | 4.3 (6) | 11.2 (7) | 7.6 (7)
Online-W | 61.6 (7) | 7.0 (7) | 4.3 (7) | 11.3 (6) | 7.8 (6)
HuaweiTSC | 60.9 (8) | 6.8 (8) | 4.3 (8) | 11.2 (8) | 7.5 (8)
SRPOL | 59.5 (9) | 6.3 (9) | 4.3 (9) | 11.2 (9) | 7.4 (9)
ALMAnaCH-Inria | 26.9 (10) | -4.0 (10) | 3.6 (10) | 10.1 (10) | 1.4 (10)
(a) ru$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-W | 75.1 (1) | 11.3 (1) | 4.6 (1) | 11.8 (1) | 12.7 (1)
Online-G | 73.2 (2) | 10.8 (2) | 4.6 (2) | 11.7 (2) | 12.3 (2)
Online-B | 72.9 (3) | 10.7 (3) | 4.6 (3) | 11.5 (3) | 11.6 (3)
Online-Y | 69.8 (4) | 9.6 (4) | 4.5 (4) | 11.5 (4) | 11.1 (5)
JDExploreAcademy | 69.7 (5) | 9.6 (5) | 4.5 (5) | 11.5 (5) | 11.4 (4)
Lan-Bridge | 67.4 (6) | 9.1 (6) | 4.4 (6) | 11.3 (6) | 10.5 (7)
Online-A | 67.4 (7) | 8.6 (7) | 4.4 (7) | 11.3 (7) | 10.6 (6)
PROMT | 60.4 (8) | 6.9 (8) | 4.3 (8) | 11.2 (8) | 9.8 (8)
SRPOL | 59.8 (9) | 6.4 (9) | 4.2 (9) | 11.0 (10) | 9.2 (10)
HuaweiTSC | 59.2 (10) | 6.3 (10) | 4.2 (10) | 11.1 (9) | 9.8 (9)
eTranslation | 58.0 (11) | 6.0 (11) | 4.2 (11) | 11.0 (11) | 9.0 (11)
(b) en$\rightarrow$ru

Table 104: COMET scores (ranking) with different models for en-ru.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-G | -17.1 (1) | -18.3 (1) | 1.0 (1) | 4.8 (1) | -30.4 (1)
Lan-Bridge | -48.0 (2) | -19.0 (2) | 0.1 (2) | 3.3 (2) | -38.8 (2)
(a) ru$\rightarrow$sah

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-G | 31.1 (1) | 1.2 (1) | 3.8 (1) | 8.7 (1) | -10.4 (1)
Lan-Bridge | -75.9 (2) | -30.0 (2) | 1.8 (2) | 6.5 (2) | -19.3 (2)
(b) sah$\rightarrow$ru

Table 105: COMET scores (ranking) with different models for ru-sah.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-B | 62.6 (1) | 7.4 (1) | 4.4 (1) | 11.5 (1) | 7.0 (1)
Lan-Bridge | 62.4 (2) | 7.4 (2) | 4.4 (2) | 11.5 (2) | 6.9 (2)
Online-G | 57.5 (3) | 5.8 (3) | 4.3 (3) | 11.3 (3) | 6.0 (3)
Online-A | 52.2 (4) | 4.1 (4) | 4.2 (4) | 11.1 (4) | 5.1 (4)
HuaweiTSC | 50.1 (5) | 3.4 (5) | 4.2 (5) | 11.1 (5) | 5.0 (5)
Online-Y | 49.8 (6) | 3.3 (7) | 4.1 (6) | 10.9 (8) | 4.5 (8)
PROMT | 49.6 (7) | 3.4 (6) | 4.1 (7) | 11.0 (6) | 4.9 (6)
ARC-NKUA | 49.6 (8) | 3.3 (8) | 4.1 (8) | 11.0 (7) | 4.6 (7)
ALMAnaCH-Inria | 21.8 (9) | -5.7 (9) | 3.5 (9) | 10.0 (9) | -0.7 (9)
(a) uk$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-B | 73.3 (1) | 10.4 (1) | 4.5 (1) | 11.6 (1) | 11.9 (1)
Online-G | 69.9 (2) | 9.2 (2) | 4.4 (2) | 11.6 (2) | 11.7 (2)
Lan-Bridge | 65.8 (3) | 8.5 (3) | 4.3 (3) | 11.3 (3) | 10.4 (3)
Online-A | 60.9 (4) | 6.8 (4) | 4.2 (4) | 11.2 (4) | 10.0 (4)
eTranslation | 54.6 (5) | 4.9 (5) | 4.1 (5) | 11.0 (5) | 9.0 (5)
HuaweiTSC | 54.4 (6) | 4.7 (6) | 4.1 (6) | 11.0 (6) | 8.7 (6)
Online-Y | 52.0 (7) | 4.0 (7) | 4.0 (7) | 10.8 (7) | 7.9 (7)
ARC-NKUA | 49.2 (8) | 3.4 (8) | 4.0 (8) | 10.7 (8) | 7.7 (8)
(b) en$\rightarrow$uk

Table 106: COMET scores (ranking) with different models for en-uk.

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
Online-G | 45.7 (1) | -0.0 (1) | 3.3 (1) | 9.3 (2) | 4.6 (1)
JDExploreAcademy | 45.2 (2) | -0.3 (2) | 3.3 (2) | 9.2 (3) | 4.1 (5)
LanguageX | 45.0 (3) | -0.3 (3) | 3.3 (3) | 9.2 (4) | 4.0 (6)
Lan-Bridge | 43.1 (4) | -0.5 (4) | 3.3 (4) | 9.4 (1) | 4.3 (2)
HuaweiTSC | 42.9 (5) | -0.7 (5) | 3.2 (5) | 9.2 (5) | 4.0 (7)
Online-B | 42.2 (6) | -0.7 (6) | 3.2 (6) | 9.2 (6) | 4.2 (4)
AISP-SJTU | 41.7 (7) | -1.2 (7) | 3.2 (7) | 9.1 (8) | 3.6 (8)
Online-Y | 40.9 (8) | -1.5 (8) | 3.2 (8) | 9.2 (7) | 4.3 (3)
Online-A | 35.3 (9) | -2.8 (9) | 3.1 (9) | 8.9 (9) | 3.1 (9)
Online-W | 31.7 (10) | -3.3 (10) | 3.0 (11) | 8.8 (12) | 2.6 (10)
NiuTrans | 31.4 (11) | -3.5 (11) | 3.1 (10) | 8.9 (10) | 2.5 (11)
DLUT | 30.7 (12) | -4.0 (12) | 3.0 (12) | 8.9 (11) | 2.4 (12)
(a) zh$\rightarrow$en

System | wmt20-comet-da | wmt21-comet-da | wmt21-comet-mqm | wmt21-comet-qe-mqm | wmt21-comet-qe-da
---|---|---|---|---|---
LanguageX | 63.9 (1) | 7.6 (1) | 4.2 (1) | 11.3 (3) | 8.0 (5)
Online-B | 61.8 (2) | 6.8 (2) | 4.2 (2) | 11.3 (4) | 8.1 (4)
JDExploreAcademy | 61.8 (3) | 6.8 (3) | 4.2 (3) | 11.3 (5) | 8.0 (6)
Lan-Bridge | 61.4 (4) | 6.5 (4) | 4.2 (4) | 11.4 (1) | 8.2 (2)
Online-W | 61.0 (5) | 6.2 (5) | 4.1 (5) | 11.3 (6) | 8.2 (3)
Manifold | 60.2 (6) | 6.2 (6) | 4.1 (6) | 11.4 (2) | 8.3 (1)
Online-Y | 59.8 (7) | 5.6 (8) | 4.1 (7) | 11.3 (7) | 8.0 (7)
HuaweiTSC | 59.6 (8) | 6.0 (7) | 4.1 (8) | 11.3 (8) | 7.8 (8)
Online-A | 57.4 (9) | 4.9 (10) | 4.0 (10) | 11.2 (9) | 7.5 (9)
AISP-SJTU | 56.6 (10) | 5.1 (9) | 4.1 (9) | 11.2 (10) | 7.2 (10)
DLUT | 52.2 (11) | 3.7 (11) | 4.0 (11) | 11.2 (11) | 7.0 (11)
Online-G | 51.3 (12) | 3.0 (12) | 3.9 (12) | 10.9 (12) | 5.7 (12)
(b) en$\rightarrow$zh

Table 107:  COMET scores (ranking) with different models for en-zh.

##  Appendix E Sentence Matching Statistic Tables

From Table 108 to 118.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-W | 145 | 3/7/10/1448 | 68.7/68.6 | 77.5/77.5
JDExploreAcademy | 98 | 0/7/10/1448 | 60.9/60.9 | 74.8/74.8
Lan-Bridge | 84 | 0/7/10/1448 | 60.8/60.7 | 71.8/71.8
Online-B | 84 | 0/7/10/1448 | 60.6/60.5 | 71.9/71.9
CUNI-DocTransformer | 80 | 7/7/10/1448 | 57.9/57.9 | 70.6/70.6
Online-A | 83 | 0/7/10/1448 | 59.4/59.4 | 69.8/69.8
CUNI-Transformer | 82 | 0/7/10/1448 | 57.6/57.8 | 69.2/69.2
Online-G | 75 | 0/7/10/1448 | 54.3/54.2 | 63.0/63.0
SHOPLINE-PL | 55 | 0/7/10/1448 | 53.5/53.4 | 61.2/61.2
Online-Y | 57 | 0/7/10/1448 | 51.0/51.5 | 58.7/58.7
ALMAnaCH-Inria | 12 | 0/7/10/1448 | 36.8/37.1 | 19.3/19.3

(a) cs$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
CUNI-Bergamot | 49 | 0/2/122/2037 | 46.2/45.3 | 96.1/95.9
CUNI-DocTransformer | 100 | 48/2/122/2037 | 47.8/46.4 | 91.7/91.5
CUNI-Transformer | 84 | 0/2/122/2037 | 45.6/44.4 | 86.7/86.5
JDExploreAcademy | 107 | 0/2/122/2037 | 49.4/49.4 | 95.3/95.3
Lan-Bridge | 226 | 0/2/122/2037 | 53.4/53.4 | 94.7/94.7
Online-A | 105 | 0/2/122/2037 | 49.1/49.1 | 92.2/92.2
Online-B | 296 | 0/2/122/2037 | 56.5/55.3 | 97.5/97.4
Online-G | 90 | 0/2/122/2037 | 44.8/44.8 | 82.3/82.4
Online-W | 137 | 28/2/122/2037 | 53.1/53.1 | 97.9/97.9
Online-Y | 99 | 0/2/122/2037 | 44.7/43.3 | 83.7/83.5

(b) en$\rightarrow$cs

Table 108:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
AMU | 24 | 0/0/0/1930 | 34.7/34.3 | 99.4/99.4
Online-B | 80 | 0/0/0/1930 | 38.3/37.6 | 94.3/94.2
Lan-Bridge | 66 | 0/0/0/1930 | 38.3/38.2 | 91.8/91.8
CharlesTranslator | 46 | 0/0/0/1930 | 34.3/34.0 | 90.8/90.7
HuaweiTSC | 51 | 0/0/0/1930 | 36.0/35.7 | 90.7/90.7
CUNI-JL-JH | 39 | 0/0/0/1930 | 34.8/34.4 | 90.1/90.0
Online-G | 29 | 0/0/0/1930 | 32.5/32.5 | 88.4/88.3
Online-A | 43 | 0/0/0/1930 | 35.9/35.5 | 87.9/87.8
CUNI-Transformer | 42 | 0/0/0/1930 | 35.0/34.8 | 87.4/87.3
Online-Y | 40 | 0/0/0/1930 | 32.1/31.9 | 78.4/78.4
ALMAnaCH-Inria | 21 | 0/0/0/1930 | 26.8/26.7 | 61.4/61.3

(a) cs$\rightarrow$uk

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
AMU | 117 | 4/0/24/2812 | 37.0/36.4 | 104.9/104.7
Online-B | 127 | 0/0/24/2812 | 36.4/35.7 | 96.5/96.3
Lan-Bridge | 147 | 0/0/24/2812 | 36.5/36.5 | 94.6/94.6
HuaweiTSC | 132 | 0/0/24/2812 | 36.0/35.7 | 91.5/91.4
CharlesTranslator | 125 | 0/0/24/2812 | 35.9/35.0 | 90.2/90.0
CUNI-JL-JH | 122 | 0/0/24/2812 | 35.1/34.6 | 89.1/89.0
CUNI-Transformer | 126 | 0/0/24/2812 | 35.8/34.8 | 88.5/88.4
Online-A | 116 | 0/0/24/2812 | 33.3/33.0 | 85.4/85.3
Online-G | 106 | 0/0/24/2812 | 31.5/31.4 | 84.2/84.2
Online-Y | 88 | 0/0/24/2812 | 29.6/29.1 | 78.7/78.4
ALMAnaCH-Inria | 75 | 0/0/24/2812 | 25.3/25.0 | 62.5/62.3

(b) uk$\rightarrow$cs

Table 109:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
JDExploreAcademy | 62 | 0/1/5/1984 | 49.3/49.3 | 58.0/58.0
Online-B | 58 | 0/1/5/1984 | 49.7/49.7 | 57.0/57.0
Lan-Bridge | 67 | 0/1/5/1984 | 50.1/50.1 | 56.6/56.6
Online-G | 59 | 0/1/5/1984 | 49.7/49.7 | 55.3/55.3
Online-Y | 45 | 0/1/5/1984 | 49.3/48.6 | 54.7/54.6
Online-A | 61 | 0/1/5/1984 | 50.2/50.2 | 54.5/54.5
Online-W | 66 | 1/1/5/1984 | 48.8/48.8 | 54.4/54.4
PROMT | 59 | 0/1/5/1984 | 49.2/49.2 | 51.8/51.8
LT22 | 27 | 0/1/5/1984 | 40.3/40.3 | 25.6/25.6

(a) de$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-W | 75 | 29/0/122/2037 | 48.9/48.8 | 65.6/65.6
JDExploreAcademy | 70 | 0/0/122/2037 | 51.6/50.2 | 63.3/63.1
Online-B | 74 | 0/0/122/2037 | 52.3/51.0 | 62.3/62.2
Online-Y | 55 | 0/0/122/2037 | 50.5/49.1 | 61.1/60.9
Online-A | 75 | 0/0/122/2037 | 50.1/50.1 | 60.6/60.6
Online-G | 47 | 0/0/122/2037 | 49.6/49.6 | 60.2/60.2
Lan-Bridge | 70 | 0/0/122/2037 | 49.4/49.4 | 58.8/58.8
OpenNMT | 86 | 0/0/122/2037 | 48.4/46.7 | 57.3/56.9
PROMT | 64 | 0/0/122/2037 | 49.0/47.7 | 55.9/55.7

(b) en$\rightarrow$de

Table 110:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-B | 264 | 0/0/5/1984 | 58.4/58.3 | 70.5/70.4
Online-W | 90 | 3/0/5/1984 | 43.6/43.6 | 63.7/63.6
Online-Y | 80 | 0/0/5/1984 | 46.2/45.2 | 57.8/57.3
Online-A | 62 | 0/0/5/1984 | 41.3/45.2 | 52.3/52.5
Online-G | 55 | 0/0/5/1984 | 39.0/39.0 | 44.9/44.9
LT22 | 23 | 0/0/5/1984 | 28.3/28.5 | 10.5/10.5

(a) de$\rightarrow$fr

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-W | 315 | 8/0/31/2006 | 64.8/64.8 | 77.9/77.9
Online-B | 153 | 0/0/31/2006 | 46.6/46.2 | 63.8/63.6
Online-Y | 121 | 0/0/31/2006 | 45.0/44.3 | 61.6/61.4
Online-A | 133 | 0/0/31/2006 | 44.4/44.4 | 59.3/59.3
eTranslation | 162 | 0/0/31/2006 | 46.5/46.0 | 55.5/55.2
Lan-Bridge | 107 | 0/0/31/2006 | 41.8/41.8 | 51.2/51.2
Online-G | 85 | 0/0/31/2006 | 41.1/41.0 | 48.3/48.4

(b) fr$\rightarrow$de

Table 111:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-B | 80 | 0/5/11/1671 | 31.5/31.5 | 80.4/80.4
Lan-Bridge | 77 | 0/5/11/1671 | 31.5/31.4 | 79.6/79.6
Online-A | 64 | 0/5/11/1671 | 29.1/29.1 | 69.5/69.5
SRPOL | 61 | 0/5/11/1671 | 29.1/29.1 | 69.5/69.5
HuaweiTSC | 62 | 0/5/11/1671 | 29.9/29.9 | 67.7/67.7
NiuTrans | 65 | 0/5/11/1671 | 29.3/29.3 | 65.6/65.6
Online-G | 48 | 0/5/11/1671 | 25.7/25.7 | 64.3/64.3
Online-Y | 54 | 0/5/11/1671 | 26.6/26.5 | 56.8/56.7

(a) en$\rightarrow$hr Table 112:  Sentence Matching Statistics. “Exact Match”
counts how many segments in the translation are identical to the corresponding
segments of the reference translation. “Self Mismatch” provides various
statistics: the number of non-unique source segments translated differently by
the system/the number of non-unique source segments translated differently by
the reference/the number of duplicated source segments/the total number of
source segments. “BLEU” and “COMET” give the scores computed on the
original/normalized translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
NAIST-NICT-TIT | 35 | 13/11/53/2008 | 22.7/22.9 | 33.4/33.4
AIST | 0 | 24/11/53/2008 | 0.1/0.1 | -152.7/-152.6
NT5 | 48 | 0/11/53/2008 | 26.6/26.6 | 42.1/42.1
Online-W | 45 | 14/11/53/2008 | 27.8/27.7 | 41.2/41.2
JDExploreAcademy | 75 | 0/11/53/2008 | 25.6/25.6 | 40.6/40.6
Online-B | 63 | 0/11/53/2008 | 24.7/24.8 | 39.7/39.6
DLUT | 66 | 0/11/53/2008 | 24.8/24.8 | 37.3/37.3
Online-A | 39 | 0/11/53/2008 | 22.8/22.9 | 32.9/32.9
LanguageX | 36 | 0/11/53/2008 | 22.4/22.3 | 33.0/32.9
Online-Y | 32 | 0/11/53/2008 | 21.5/22.4 | 32.4/32.3
Lan-Bridge | 44 | 0/11/53/2008 | 22.8/22.7 | 32.0/31.9
AISP-SJTU | 36 | 0/11/53/2008 | 22.0/21.9 | 30.2/30.2
Online-G | 34 | 0/11/53/2008 | 19.7/20.1 | 22.3/22.5
KYB | 30 | 0/11/53/2008 | 18.1/18.2 | 17.4/17.5

(a) ja$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
JDExploreAcademy | 10 | 0/63/122/2037 | 25.4/2.9 | 65.2/59.9
NT5 | 15 | 0/63/122/2037 | 18.2/2.9 | 64.1/59.1
LanguageX | 15 | 1/63/122/2037 | 24.5/2.7 | 62.1/57.1
Online-B | 12 | 0/63/122/2037 | 8.0/2.4 | 60.8/57.1
DLUT | 15 | 0/63/122/2037 | 12.8/2.7 | 60.5/55.3
Online-W | 14 | 47/63/122/2037 | 17.6/2.3 | 59.8/55.1
Online-Y | 10 | 0/63/122/2037 | 17.5/2.9 | 56.9/52.2
Lan-Bridge | 11 | 0/63/122/2037 | 15.1/2.9 | 56.6/51.9
Online-A | 5 | 0/63/122/2037 | 13.3/2.8 | 53.7/48.8
NAIST-NICT-TIT | 13 | 22/63/122/2037 | 13.6/2.9 | 53.4/48.5
AISP-SJTU | 13 | 0/63/122/2037 | 16.6/2.8 | 52.4/47.9
KYB | 5 | 0/63/122/2037 | 13.3/2.7 | 31.8/26.6
Online-G | 6 | 0/63/122/2037 | 12.4/2.8 | 25.0/20.2

(b) en$\rightarrow$ja

Table 113:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
TartuNLP | 3 | 0/0/0/420 | 29.9/29.9 | -5.7/-5.7
TAL-SJTU | 4 | 0/0/0/420 | 30.4/30.4 | -8.3/-8.3
HuaweiTSC | 1 | 0/0/0/420 | 23.4/23.3 | -27.2/-27.2
Liv4ever | 3 | 0/0/0/420 | 23.3/23.3 | -44.0/-44.0
NiuTrans | 0 | 0/0/0/420 | 13.0/12.9 | -88.3/-88.3

(a) liv$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
TAL-SJTU | 0 | 0/0/0/420 | 17.0/16.9 | -29.5/-29.5
TartuNLP | 1 | 0/0/0/420 | 15.0/14.3 | -36.8/-36.9
HuaweiTSC | 1 | 0/0/0/420 | 12.8/12.9 | -38.8/-39.0
Liv4ever | 0 | 0/0/0/420 | 14.7/14.5 | -39.4/-39.6
NiuTrans | 0 | 0/0/0/420 | 12.3/11.4 | -81.9/-81.9

(b) en$\rightarrow$liv

Table 114:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-G | 132 | 0/0/5/2016 | 46.7/46.6 | 65.2/65.2
JDExploreAcademy | 107 | 0/0/5/2016 | 45.1/45.1 | 64.9/64.9
Online-Y | 98 | 0/0/5/2016 | 43.8/44.7 | 64.1/64.1
Lan-Bridge | 98 | 0/0/5/2016 | 45.2/45.2 | 63.1/63.1
Online-B | 97 | 0/0/5/2016 | 45.0/45.1 | 63.1/63.1
Online-A | 96 | 0/0/5/2016 | 43.9/43.8 | 62.2/62.2
Online-W | 71 | 2/0/5/2016 | 42.6/42.6 | 61.6/61.6
HuaweiTSC | 97 | 0/0/5/2016 | 45.1/45.1 | 60.9/60.9
SRPOL | 85 | 3/0/5/2016 | 43.6/43.6 | 59.5/59.5
ALMAnaCH-Inria | 35 | 0/0/5/2016 | 30.3/30.3 | 26.9/26.9

(a) ru$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-W | 81 | 27/25/122/2037 | 32.4/32.4 | 75.1/75.1
Online-G | 74 | 2/25/122/2037 | 32.8/32.8 | 73.2/73.2
Online-B | 100 | 0/25/122/2037 | 34.9/33.3 | 72.9/72.7
Online-Y | 103 | 0/25/122/2037 | 33.2/32.1 | 69.8/69.6
JDExploreAcademy | 81 | 1/25/122/2037 | 32.7/32.7 | 69.7/69.7
Lan-Bridge | 83 | 0/25/122/2037 | 32.6/32.3 | 67.4/67.3
Online-A | 70 | 0/25/122/2037 | 33.1/31.9 | 67.4/67.2
PROMT | 73 | 0/25/122/2037 | 30.6/29.7 | 60.4/60.3
SRPOL | 66 | 16/25/122/2037 | 30.4/30.4 | 59.8/59.8
HuaweiTSC | 60 | 0/25/122/2037 | 30.8/30.8 | 59.2/59.2
eTranslation | 58 | 0/25/122/2037 | 29.8/28.9 | 58.0/57.9

(b) en$\rightarrow$ru

Table 115:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-G | 7 | 0/0/14/1123 | 14.7/14.7 | -17.1/-17.2
Lan-Bridge | 0 | 0/0/14/1123 | 15.3/14.0 | -48.0/-48.9

(a) ru$\rightarrow$sah

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-G | 32 | 0/1/15/1123 | 29.6/29.6 | 31.1/31.0
Lan-Bridge | 0 | 0/1/15/1123 | 7.1/7.1 | -75.9/-75.7

(b) sah$\rightarrow$ru

Table 116:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-B | 101 | 0/0/0/2018 | 44.4/44.4 | 62.6/62.5
Lan-Bridge | 106 | 0/0/0/2018 | 44.6/44.6 | 62.4/62.4
Online-G | 73 | 0/0/0/2018 | 43.2/43.0 | 57.5/57.4
Online-A | 76 | 0/0/0/2018 | 42.3/42.1 | 52.2/52.1
HuaweiTSC | 71 | 0/0/0/2018 | 41.6/41.5 | 50.1/50.1
Online-Y | 75 | 0/0/0/2018 | 41.8/41.0 | 49.8/49.7
PROMT | 73 | 0/0/0/2018 | 42.1/42.1 | 49.6/49.6
ARC-NKUA | 76 | 0/0/0/2018 | 41.9/41.9 | 49.6/49.6
ALMAnaCH-Inria | 42 | 0/0/0/2018 | 30.0/29.7 | 21.8/21.7

(a) uk$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-B | 75 | 0/6/122/2037 | 32.5/31.6 | 73.3/73.1
Online-G | 36 | 2/6/122/2037 | 27.2/27.2 | 69.9/69.9
Lan-Bridge | 71 | 0/6/122/2037 | 29.5/29.5 | 65.8/65.7
Online-A | 53 | 0/6/122/2037 | 28.0/27.5 | 60.9/60.8
eTranslation | 52 | 0/6/122/2037 | 26.2/25.2 | 54.6/54.4
HuaweiTSC | 45 | 0/6/122/2037 | 26.5/26.3 | 54.4/54.4
Online-Y | 49 | 0/6/122/2037 | 26.9/25.9 | 52.0/51.8
ARC-NKUA | 50 | 0/6/122/2037 | 25.2/24.4 | 49.2/49.1

(b) en$\rightarrow$uk

Table 117:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
Online-G | 15 | 0/1/4/1875 | 34.1/33.7 | 45.7/45.5
JDExploreAcademy | 22 | 0/1/4/1875 | 37.9/37.9 | 45.2/45.2
LanguageX | 22 | 0/1/4/1875 | 36.4/36.4 | 45.0/45.0
Lan-Bridge | 21 | 0/1/4/1875 | 32.9/32.9 | 43.1/43.1
HuaweiTSC | 24 | 0/1/4/1875 | 34.3/34.3 | 42.9/42.9
Online-B | 18 | 0/1/4/1875 | 33.3/33.3 | 42.2/42.1
AISP-SJTU | 15 | 0/1/4/1875 | 34.4/34.3 | 41.7/41.7
Online-Y | 8 | 0/1/4/1875 | 31.1/30.8 | 40.9/40.7
Online-A | 10 | 0/1/4/1875 | 31.6/31.7 | 35.3/35.2
Online-W | 16 | 1/1/4/1875 | 28.1/28.1 | 31.7/31.7
NiuTrans | 12 | 0/1/4/1875 | 30.5/30.5 | 31.4/31.3
DLUT | 9 | 0/1/4/1875 | 29.0/29.0 | 30.7/30.7

(a) zh$\rightarrow$en

System | Exact Match | Self Mismatch | BLEU | COMET
---|---|---|---|---
LanguageX | 95 | 0/3/122/2037 | 24.9/4.9 | 63.9/60.6
Online-B | 65 | 0/3/122/2037 | 48.1/9.2 | 61.8/58.5
JDExploreAcademy | 59 | 1/3/122/2037 | 24.0/5.2 | 61.8/58.7
Lan-Bridge | 49 | 20/3/122/2037 | 23.6/5.1 | 61.4/58.0
Online-W | 32 | 36/3/122/2037 | 16.4/4.4 | 61.0/58.0
Manifold | 47 | 0/3/122/2037 | 23.4/5.9 | 60.2/57.3
Online-Y | 44 | 0/3/122/2037 | 28.0/6.0 | 59.8/56.5
HuaweiTSC | 43 | 0/3/122/2037 | 25.3/4.9 | 59.6/56.5
Online-A | 50 | 0/3/122/2037 | 26.4/6.3 | 57.4/53.8
AISP-SJTU | 52 | 0/3/122/2037 | 25.7/4.8 | 56.6/53.2
DLUT | 35 | 0/3/122/2037 | 21.1/5.2 | 52.2/48.5
Online-G | 38 | 3/3/122/2037 | 16.1/5.0 | 51.3/48.3

(b) en$\rightarrow$zh

Table 118:  Sentence Matching Statistics. “Exact Match” counts how many
segments in the translation are identical to the corresponding segments of the
reference translation. “Self Mismatch” provides various statistics: the number
of non-unique source segments translated differently by the system/the number
of non-unique source segments translated differently by the reference/the
number of duplicated source segments/the total number of source segments.
“BLEU” and “COMET” give the scores computed on the original/normalized
translations.