We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.

 Evaluation metrics play a central role in the machine learning community. They direct research efforts and define the state of the art models. In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) . Both approaches rely on counting the matching n-grams in the candidate text to n-grams in the reference text. BLEU is precision focused while ROUGE is recall focused. These metrics have posed serious limitations and have already been criticized by the academic community (Reiter, 2018) (Callison-Burch et al., 2006) (Sulem et al., 2018) (Novikova et al., 2017) . In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass. Furthermore we propose concrete ways to use recent advances in NLP to design data-driven metrics addressing the weaknesses found in BLEU and ROUGE while scoring high on the criteria for a sound evaluation metric. In this work, we have established a framework to assess metrics comparing the quality of reference and hypothesis summary/translations. Based on these criteria, we compare evaluators using recent Transformers (in this case RoBERTa-STS) to BLEU and ROUGE. We also show how this good performance on our scorecard translates on a previously unseen machine translation datasets. Such results highlight the potential to replace BLEU and ROUGE with data-driven models such as RoBERTa-STS. A APPENDIX In the appendix, we will discuss the failure cases of BLEU, ROUGE and RoBERTa-STS in detail to provide a better understanding of how these models can fall short in language evaluation. This is important because a good metric scorecard has to represent the quality of an evaluator. These experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator. We will start by taking examples from the BLEU and ROUGE dataset. As in the paper the BLEU scores used are always a uniform average up to 4-grams and the ROUGE score is the average of ROUGE-1 and ROUGE-2. Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5. In table 6 we see examples of many different error cases and ,in most sentences, we also have more than one cause for the drastic difference between BLEU/ROUGE and the label. For instance, in rows 1 and 6 we see that the cause for the error is the reordering of sub-sentences, spelling/punctuation and newly introduced words that don't change the meaning but merely extend it. While BLEU and ROUGE are failing in these examples, we see that the RoBERTa-STS model scores similarly to the label. In line 7, we can see that the RoBERTa-STS model score is above 5. In rows 2 and 7, we see that the main difference is the form or tense of the verb in a sentence. This makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below. This characteristic of BLEU reinforces the point that BLEU and ROUGE are not useful in tracking the state of the art and comparing the best methods but are tools to weed out bad models fairly simply. In rows 3 and 9 we see sentences that differ due to using descriptive phrases instead of a word or extending the sentence with more information. These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them to reach a related word as in the famous example of king -men + woman = queen Mikolov et al. (2013) . In rows 4,8 and 5 we see general paraphrases with the same meaning represented in a generally different sentence. In all cases we see a drastic difference between BLEU/ROUGE and the label but these cases also unearth a specific characteristic of the neural evaluator. In 4 and 8 we see that the error of the RoBERTa-STS model comparatively lower than row 5. While it is hard to determine the exact cause through only looking at these examples table 7 for the RoBERTa-STS failure cases will make this case more compelling. While language models have a general sense of the context in a given sentence, they still lack a general knowledge of the world. Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic. To test this we added "while riding" or "on a bike" at the end of a sentence and the score immediately went up to 3.6/5 while barely changing the BLEU* and the ROUGE score. In row 4 and 8 however, the context of the sentence is defined explicitly with the key phrases. We see this bias affecting RoBERTa-STS scoring in the examples below. In the above examples, we will find two points that will helps us better understand the RoBERTa-STS as a neural evaluator. Firstly, we see that the neural network sometimes lacks a sense of context that is not given in the sentence explicitly. While these language models are trained on a large corpus and capture a sense of the words and language, we still see that their performance is not perfect. We see these examples in row 4, where the model cannot relate a navy seal as a military personnel. Or as in row 1, where the model cannot model an idiom. The second and more critical place where we need further development is especially detecting whether the core argument/message in a sentence is the same beyond whether if they are talking about the same things. As in rows 2 and 3. We see the same landmark words and can clearly say that the sentences are talking about the same things, but what a human can distinguish is that they are saying unrelated things. This is one of the key motivations in including the language inference task in the scorecard. Since detecting whether a pair of sentences are related on what level is a key part of detecting sentence similarity. One last thing we will mention is that while RoBERTa-STS and BLEU/ROUGE have different error cases, their performance on these error cases is also remarkably different in favor of the former. Table 8 shows the mean error of BLEU* and the RoBERTa-STS model on each others top 500, which is one third of the development set, error cases. We see in table 8 that BLEU* has a remarkable error in both its failure cases and also the failure cases of RoBERTa-STS while RoBERTa-STS outperforms BLEU* in each category. While neural evaluators have also room for improvement, we can with confidence say that they are outperforming classical methods and with a methodical way of improving them can bolster progress of NLP research.
Introduces JAUNE: a methodology to replace BLEU and ROUGE score with multidimensional, model-based evaluators for assessing summaries This paper proposes a new JAUNE metric for the evaluation of machine translation and text summarization systems, showing that their model corresponds better to ground truth similarity labels than BLEU.