# Exploring Prompting Large Language Models as Explainable Metrics

Ghazaleh Mahmoudi
School of Computer Engineering
Iran University of Science and Technology, Iran
<EMAIL_ADDRESS>
###### Abstract

This paper describes the IUST NLP Lab submission to the Prompting Large
Language Models as Explainable Metrics Shared Task at the Eval4NLP 2023
Workshop on Evaluation & Comparison of NLP Systems. We have proposed a zero-
shot prompt-based strategy for explainable evaluation of the summarization
task using Large Language Models (LLMs). The conducted experiments demonstrate
the promising potential of LLMs as evaluation metrics in Natural Language
Processing (NLP), particularly in the field of summarization. Both few-shot
and zero-shot approaches are employed in these experiments. The performance of
our best provided prompts achieved a Kendall correlation of 0.477 with human
evaluations in the text summarization task on the test data. Code and results
are publicly available on GitHub 111https://github.com/ghazaleh-
mahmoodi/Prompting_LLMs_AS_Explainable_Metrics.

Exploring Prompting Large Language Models as Explainable Metrics

Ghazaleh Mahmoudi School of Computer Engineering Iran University of Science
and Technology, Iran<EMAIL_ADDRESS>

##  1 Introduction

Summarization is crucial for quickly understanding large textual documents.
The goal of text summarization is to condense lengthy documents into a
concise, coherent, and easily understandable format while retaining the
essential information from the original source. However, assessing the quality
and performance of summarization systems has proven to be a challenging task.
Commonly used evaluation methods for summarization, such as ROUGE scores Lin
(2004), have certain limitations. They fail to capture the overall quality,
coherence, and interpretability of summaries. Additionally, they rely on
human-generated reference summaries, which are time-consuming and subjective.
Given the limitations of traditional evaluation approaches, it is important to
explore alternative evaluation methods that leverage the capabilities of LLMs
and offer explainable metrics.

LLMs, such as GPT-3 Brown et al. (2020) and LLaMA Touvron et al. (2023), have
shown remarkable summarization capabilities. They can generate coherent and
contextually grounded summaries. This makes them ideal for evaluation
purposes. LLMs provide both interpretability and inherent summarization
abilities. They can generate explanations and reasoning for their predictions,
giving evaluators a deeper understanding of system strengths and weaknesses.

Moreover, LLMs reduce the dependency on gold-standard reference summaries. By
using LLMs as evaluators, we can generate comparative summaries and
objectively assess system-generated summaries. This eliminates potential
biases from human references.

In summary, using LLMs as explainable metrics in summarization evaluation
offers several benefits. It overcomes the limitations of traditional methods,
provides interpretability, and reduces reliance on human-generated references.
This emerging field of research holds promise for a more comprehensive and
objective assessment of summarization systems.

The main contribution of this paper is the investigation of various prompt-
based methods for explainable evaluation of summarization tasks. We explore
both few-shot and zero-shot approaches in our experiments. The best
performance prompt follows the zero-shot strategy and is introduced in the
paper under the name P1. In this prompt, the criteria for assessing the
quality of summaries are described (e.g., how well the main idea of the main
document is captured in the summary). This prompt achieves a Kendall
correlation score of 0.477, outperforming other methods in comparison. Our
conducted experiments highlight the promising potential of LLMs as evaluation
metrics in the field of NLP, with a specific focus on summarization.

##  2 Related Work

Several recent studies have focused on utilizing LLMs for the evaluation of
several different tasks in NLP (e.g., text generation, machine translation,
and summarization).

GPTScore Fu et al. (2023) is a novel framework for evaluating generated texts
using large pre-trained language models, particularly GPT-3. The framework
leverages the emergent abilities of generative pre-trained models, such as
zero-shot instruction, to score generated texts. GPTScore operates under the
assumption that a large pre-trained language model is more likely to assign
higher probabilities to high-quality generated text when provided with
adequate instruction and context. By leveraging the power of GPT-3, GPTScore
aims to assess the quality of generated text based on the model’s generative
capabilities.

In a similar vein, Wang et al. (2023) conducted a preliminary survey on using
ChatGPT, a variant of the GPT model, as a natural language generation (NLG)
evaluator. The study explores the potential of ChatGPT in evaluating the
quality of generated text in various NLG tasks.

In the context of translation quality assessment, GEMBA Kocmi and Federmann
(2023) is introduced as a GPT-based metric that can effectively evaluate
translations with or without a reference translation. The evaluation focuses
on zero-shot prompting and involves comparing four prompt variants in two
modes, depending on the availability of a reference. Results from the
evaluation demonstrate that GEMBA achieves state-of-the-art accuracy when
compared to MQM-based human labels, as evidenced by the WMT22 Metrics shared
task.

Instructscore Xu et al. (2023) is an open-source and explainable evaluation
metric for text generation. This model fine-tunes the LLaMA model to predict a
fine-grained error diagnosis of machine translated content. This work presents
a novel framework for explainable text generation evaluation, addressing the
limitations associated with black-box metrics and showcasing the potential of
LLMs to provide meaningful and interpretable evaluations.

G-Eval Liu et al. (2023), is a framework that utilizes LLMs with chain-of-
thoughts (CoT) and a form-filling paradigm to assess the quality of NLG
outputs, specifically in text summarization and dialogue generation tasks. The
experiments demonstrate that G-Eval, utilizing GPT-4 as the backbone model,
achieves a high Spearman correlation of 0.514 with human evaluations in the
text summarization task, outperforming previous methods significantly.

##  3 Task Description

The topic of Eval4NLP shared task Leiter et al. (2023) is to provide
explainable metrics for summarization and machine translation evaluation by
prompting LLMs. The main goal is to investigate different prompting methods
(e.g., zero-shot, few-shot, Chain of Thought, Fine-Grained, Majority Vote,
Self-Refinement), therefore, fine-tuning the LLMs is not allowed. Also, a
number of LLMs are allowed to be used. The shared task has two tracks based on
the model sizes (One for models bigger than 25B parameters, and one for
smaller models).

This work has been done on the summarization task and using small models. In
the following, the dataset and the evaluation metric are explained.

###  3.1 Data

The Shared Task organizers opted for SummEval during the training and
development phase for summarization. Fabbri et al. (2021) introduced SummEval
as an evaluation benchmark, aiming to compare various methods for assessing
summarization. This benchmark entails the assignment of human ratings on four
key dimensions of every summary, including fluency, coherence, consistency,
and relevance. SummEval draws upon the renowned CNN/DailyMail dataset proposed
by Hermann et al. (2015) for its construction.

Furthermore, in the testing phase, a new reference-free dataset with summary-
level quality scores is collected for summarization. As source data, sentences
and paragraphs were collected from English Wikipedia pages created after
15.07.2023. Test-phase scores are constructed from fine-grained scores.

###  3.2 Evaluation Metrics

To determine how well LLMs explainable metrics correlate with humans, We
follow the evaluation protocol of the WMT22 metrics shared task. we use
Kendall’s Tau correlation Freitag et al. (2022). In addition to Kendall
correlation, Pearson Mukaka (2012) and Spearman Zar (2005) are also used in
the test phase.

##  4 Methodology

In this research, We have used pre-trained orca_mini_v3_7b (tuned Llama2-7b
model) Mathur (2023) on the HuggingFace
Transformers222https://huggingface.co/pankajmathur/orca_mini_v3_7b. We
employed two strategies, zero-shot and few-shot, for constructing prompts.

The zero-shot strategy included the combination of evaluation criteria for the
quality of summarization in the form of questions or detailed explanations
provided to the model.

There are examples of summarization evaluation written in the few-shot
strategy. In this way, the main document, the summarized document, and the
scores received are mentioned precisely.

To assess summarization quality via prompting an LLM, the following parameters
are needed:

  * •

Prompt variant (from a pre-defined set)

  * •

Main document Source Text

  * •

Summary Summary

###  4.1 Prompt variants

For P1 (Table 2), we formulated the main criteria for assessing summary
quality, which were originally expressed by humans. In this prompt, the
following items are mentioned to be checked:

  * •

Comparing the key points.

  * •

Capturing the main ideas.

  * •

Score on a continuous scale from 0 to 100.

  * •

Meaning of zero score: irrelevant, factually incorrect, and not readable
summary.

  * •

Meaning of a hundred score: relevant, factually correct, good readability
summary.

  * •

Explain the result.

To create prompt P2(Table 3), we consulted the ChatGPT4 Bot and asked what
questions would be relevant for evaluating summarization. We then modified and
adapted the generated questions accordingly. In this prompt, in addition to
the items mentioned in P1, the following items have been added in the form of
questions.

  * •

The overall length of the summary. Concise representation of the original
document.

  * •

Grammatical accuracy and fluency of the summary.

  * •

Evaluate The ranking of information in the summary.

  * •

Analyze the level of abstraction in the summary.

  * •

Contextual understanding is exhibited by the summary.

Prompts P1 and P2 also include an Explanation for the model’s output score,
thus containing questions that aid in better understanding the received
score’s reasoning.

Prompt P3(Table 4) and P4(Table 5) are similar to the P1 prompts, and only the
wording and the way of expression have changed.

In P5 (Table 6), we guide the model to calculate the desired score by
calculating the similarity of the main and summarized documents. P5 includes
examples of how one can calculate the similarity of two documents.

Prompt P6 (Table 7) follows the few-shot strategy, where two examples
consisting of the main document, and the written summary, along with their
respective score, are included in the input prompt.

##  5 Results and Analysis

We experiment with six different distinct prompt types. One of them is few-
shot (P6) and the rest are zero-shot. Table 1 shows results for all prompt
variants we have experimented with.

|  Kendall | Pearson | Spearman
---|---|---|---
P1 | 0.477 | 0.495 | 0.619
P2 | 0.470 | 0.468 | 0.607
P3 | 0.472 | 0.498 | 0.612
P4 | 0.467 | 0.504 | 0.610
P5 | 0.454 | 0.543 | 0.589
P6 | 0.283 | 0.513 | 0.376
Table 1:  Test phase Segment-level Kendall ($\tau$) and Pearson ($\rho$) and
Spearman correlation. Prompt P1:
---
Score the effectiveness of the summarization by comparing the key points and
overall coherence of the summarized with the main document.
Checked whether the summary captures the main ideas, maintains the intended
tone and style, and provides a concise yet comprehensive overview of the main
document.
Score the summarization with respect to the summarized document on a
continuous scale from 0 to 100, where a score of zero means irrelevant,
factually incorrect and good readable and score of 100 means relevant,
factually correct, no readability summarized.
Also explain your process to get this score to summary.
Also please perform error Analysis of given summary.
What should we change to have a better result?",
main document: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Explanation: gen ’explanation’
Table 2: The best-performing prompt based on zero-shot prompting strategy
expecting a score between 0–100.

The execution of each prompt takes approximately 1 hour. If we also include
the explanation of the results in the output, each execution of the test data
takes close to 4 hours.

Based on the Kendall measure (which serves as the primary evaluation metric),
the best result is associated with P1. This prompt follows the zero-shot
strategy. In this prompt, some of the SummEval criteria are also mentioned.
Additionally, P1 achieves the highest value in the Spearman measure and serves
as the final strategy for the test phase.

The results of P2, P3, and P4 are very close to each other. The reason for the
difference observed is the variation in the way the evaluation method is
expressed. In this regard, it can be said that LLMs are sensitive to manner of
expression. However, considering the proximity of the Kendall output value, it
can be concluded that they have a low sensitivity to the mentioned changes.

Furthermore, considering the results of P5, it can be stated that introducing
scientific methods for examining the similarity between summaries and the main
document did not effectively guide the model. Instead, criteria such as
"captures the main ideas" yielded better results.

Contrary to our expectations, P6 (few-shot approach) obtains the lowest score
in the Kendall measure. We expected that the few-shot strategy would
outperform zero-shot since it allows the model to observe multiple instances
of scoring, thereby enhancing its capabilities. However, our experiments
yielded results contrary to this assumption. There may be several reasons for
this result. It is possible that a larger number of input samples would have
been beneficial. Furthermore, the quality of the input samples might not have
been sufficient for the model to comprehend the problem-solving process fully.

In conclusion, based on the obtained results, it can be inferred that by
explicitly defining evaluation metrics, language models can be utilized as an
interpretable method for evaluating the summarization task.

##  6 Conclusion

In this paper, we have investigated different prompts to define explainable
evaluation metrics for summarization Using LLMs.

The experiments conducted indicate that LLMs have great potential as
evaluation metrics in NLP tasks, especially summarization. In these
experiments, both the few-shot and zero-shot approaches were used. Our best
prompt achieved a Kendall correlation of 0.477 compared to the human score.

In future work, other prompt strategies, such as the Chain of Thought, can
also be explored. Experiments can also be repeated with existing prompts and
other Language Models and compare the results obtained to determine the effect
of the Language Model on changing the quality of the output.

##  7 Limitations

Due to hardware limitations, we were unable to investigate other eligible
models in this series of experiments. In future research, it would be
beneficial to examine the impact of other models on the introduced prompts
more extensively. Furthermore, the lack of fine-tuning the model on the
defined tasks may have an effect on its performance. In future research, by
fine-tuning the model, we can explore its impact on improving the output
quality.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating Summarization Evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.
  * Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023.  Large language models are state-of-the-art evaluators of translation quality.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ , pages 193–203, Tampere, Finland. European Association for Machine Translation.
  * Leiter et al. (2023) Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. 2023.  The eval4nlp 2023 shared task on prompting large language models as explainable metrics.  In _Proceedings of the 4th Workshop on Evaluation and Comparison for NLP systems_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: Nlg evaluation using gpt-4 with better human alignment.
  * Mathur (2023) Pankaj Mathur. 2023.  orca_mini_v3_7b: An explain tuned llama2-7b model.  https://https://huggingface.co/psmathur/orca_mini_v3_7b,.
  * Mukaka (2012) MM Mukaka. 2012.  Statistics corner: A guide to appropriate use of correlation coefficient in medical research.  _Malawi medical journal : the journal of Medical Association of Malawi_ , 24(3):69—71.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023.  Is chatgpt a good nlg evaluator? a preliminary study.
  * Xu et al. (2023) Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023.  Instructscore: Towards explainable text generation evaluation with automatic feedback.
  * Zar (2005) Jerrold H. Zar. 2005.  _Spearman Rank Correlation_. John Wiley & Sons, Ltd.

##  Appendix A Appendix: Prompt Templates

Below, we present our prompt templates utilized in the described experiments
in this paper.

Prompt P2:
---
Score the effectiveness of the summarization by comparing the key points and
overall coherence of the summarized with the main document.
Checked whether the summary captures the main ideas, maintains the intended
tone and style, and provides a concise yet comprehensive overview of the main
document.
Score the summarization with respect to the summarized document on a
continuous scale from 0 to 100, where a score of zero means irrelevant,
factually incorrect and no readability and score of 100 means relevant,
factually correct, good readable summarized.
To calculate Score, first answer the following questions.
Then, according to the answers to the questions, scored the quality between 0
and 100.
1\. Assess the overall length of the summary. Does it provide a concise
representation of the original document without omitting important
information?
2\. Examine the grammatical accuracy and fluency of the summary. Are the
sentences well-structured, free of errors, and coherent?
3\. Evaluate the ranking of information in the summary. Are the most salient
and crucial details given appropriate emphasis and positioned prominently?
4\. Analyze the level of abstraction in the summary. Does it effectively
distill complex ideas and concepts into more accessible and simplified
language?
5\. Consider the contextual understanding exhibited by the summary. Does it
demonstrate an understanding of the original text beyond simple keyword
extraction?
Also explain your process to get this score to summary.
Also please perform error Analysis of given summary.
What should we change to have a better result?",
main document: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Explanation: gen ’explanation’
Table 3: Prompt P2 Prompt P3:
---
Your Task is to score the Samaritan quality. The original document is
collected from English Wikipedia pages created after 15.07.2023.
Score the effectiveness of the summarization by comparing the key points and
overall coherence of the summarized with the main document.
"Checked whether the summary captures the main ideas, maintains the intended
tone and style, and provides a concise yet comprehensive overview of the main
document.
Score the summarization with respect to the summarized document on a
continuous scale from 0 to 100, where a score of zero means irrelevant,
factually incorrect and not readable and score of 100 means relevant,
factually correct, good readability, grammatical correctness, covers the main
topic and key points of the main document article
Source text: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Table 4: Prompt P3 Prompt P4:
---
Score the effectiveness of the summarization by comparing the key points and
overall coherence of the summarized with the main document.
Checked whether the summary captures the main ideas, maintains the intended
tone and style, and provides a concise yet comprehensive overview of the main
document.
Score the summarization with respect to the summarized document, on a
continuous scale from 0 to 100.
Source text: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Table 5: Prompt P4 Prompt P5:
---
Score the summarization with respect to the summarized document on a
continuous scale from 0 to 100, where a score of zero means irrelevant,
factually incorrect and not readable and score of 100 means relevant,
factually correct, good readability.
let’s think step by step.
In other words, this Score should show the similarity between the main
document and the summarized document.
For similarity measurement, It’s possible to compare the main and summarized
document with a similarity measure such as Cosine Similarity.
word2vec, Bert embedding or n-gram are some of the approaches to calculate
similarity.
Source text: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Table 6: Prompt P5 Prompt P6:
---
Consider these example that summarization is graded in scale 0 - 100.
1\. Source text: Usain Bolt will compete at the IAAF/BTC World Relays in the
Bahamas next month, the Jamaica Athletics Administrative Association has
announced. The six-time Olympic gold medallist will compete at the relay
championship on May 2 and 3 as part of the Jamaican team. ’I’m happy to be
part of the Jamaican team for the IAAF / BTC World Relays in the Bahamas. I am
fit, healthy and ready to run,’ said Bolt. Usain Bolt has confirmed he will be
part of Jamaica’s team at the World Relays in the Bahamas Bolt reacts as he
wins 4x100m gold at the London Olympic Games in 2012 ’I hear the meet was a
lot of fun last year and there was a great atmosphere. Jamaica has a long and
successful tradition in relays and when we put on the national colours we
always do our best to make the country proud,’ he added. JAAA General
Secretary Garth Gayle commented, ’We were extremely pleased that Usain was
available for selection and that the world’s fastest man will be running for
Jamaica. We can expect some sprint magic on the track in the Bahamas on 2nd
and 3rd May.’ The full Jamaican team list for the competition will be
announced shortly. Bolt insists he always does ’his best to make his country
proud’ while wearing Jamaica colours.
1\. Summary: Jamaican sprinter Usain Bolt has confirmed he will be part of the
Jamaican team at the IAAF/BTC World Relays in the Bahamas.
1. Score : 95,
2\. Source text: Referee Mark Clattenburg has been named to take charge of the
Manchester derby on Sunday, despite having sent off three players from United
and City this season. City captain Vincent Kompany was dismissed for two
bookable offences during Belgium’s narrow 1-0 defeat of Israel in their Euro
2016 qualifier on March 31, meaning he is now suspended for the match against
Wales in June. And, although Clattenburg has been accused of favouring Louis
van Gaal’s side in the past, it’s worth noting that the 40-year-old has only
sent off two players season in the Premier League this season and both have
been from United; Tyler Blackcett in the 5-3 defeat by Leicester and Luke Shaw
in the 1-1 draw with West Ham. Mark Clattenburg will officiate the Manchester
derby between United and City at Old Trafford The English referee sent off
City and Belgium captain Vincent Kompany during the international break
Leicester 5-3 Manchester United West Ham 1-1 Manchester United Manchester
United 3-0 Tottenham Manchester City 3-1 West Ham Liverpool 2-1 Manchester
City Chelsea 1-1 Manchester City Clattenburg has courted controversy during
his career but is generally regarded as one of the Premier League’s leading
referees alongside Michael Oliver. The champion’s shock 2-1 loss to Crystal
Palace on Monday saw United move a point above their local rivals to add extra
incentive for both sides ahead of the derby at Old Trafford, which could
ultimately decide who finishes second behind expected winners Chelsea. While
Manuel Pellegrini’s side have struggled since the turn of the year, turning
from title challengers to fourth place chases, United are coasting on
confidence having won their last five consecutive league games. Clattenburg
will be joined on Sunday by assistants Simon Beck and Jake Collin, while
Jonathan Moss will serve as the fourth official. Clattenburg has shown only
two red cards this season, both to United players including Luke Shaw
(centre).
2\. Summary: United’s win over Liverpool was their first league win since the
3-0 win over Leicester on March 31 City’s win over West Ham was their first
league win since the 3-0 win over Chelsea on March 31 Manchester City’s win
over West Ham was their first league win since the 3-0 win over Chelsea on
March 31 Manuel Pellegrini’s side are top of the Premier League table, four
points clear of Chelsea, who have a game.
2\. Score : 26.666666666
following these examples, please score the following input.
Source text: {main document},
Summary: {Summary}",
Score: gen ’score’ pattern=’(100|[1-9]?[0-9])’,
Table 7: Prompt P6


# Models of reference production: How do they withstand the test of time?

Fahime Same$\heartsuit$, Guanyi Chen$\spadesuit$ Kees van Deemter$\spadesuit$
$\heartsuit$Department of Linguistics, University of Cologne
$\spadesuit$Department of Information and Computing Sciences, Utrecht
University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In recent years, many NLP studies have focused solely on performance
improvement. In this work, we focus on the linguistic and scientific aspects
of NLP. We use the task of generating referring expressions in context (REG-
in-context) as a case study and start our analysis from GREC, a comprehensive
set of shared tasks in English that addressed this topic over a decade ago. We
ask what the performance of models would be if we assessed them (1) on more
realistic datasets, and (2) using more advanced methods. We test the models
using different evaluation metrics and feature selection experiments. We
conclude that GREC can no longer be regarded as offering a reliable assessment
of models’ ability to mimic human reference production, because the results
are highly impacted by the choice of corpus and evaluation metrics. Our
results also suggest that pre-trained language models are less dependent on
the choice of corpus than classic Machine Learning models, and therefore make
more robust class predictions.

##  1 Introduction

NLP research can have different aims. Some NLP research focuses on developing
new algorithms or building practical NLP applications. Another line of NLP
work constructs computational models that aim to explain human language and
language use; this line of work has been dubbed NLP-as-Science (van Deemter,
2023). Among other things, NLP-as-Science demands that we ask ourselves to
what extent NLP research findings generalise along a range of dimensions.

In addition to the practical applications of Referring Expression Generation
(REG, Reiter, 2017), REG is also one of the typical tasks in NLP-as-Science,
where REG algorithms are built to model and explain the reference production
of human beings Krahmer and van Deemter (2012); van Deemter (2016). In the
computational linguistics and cognitive science community, REG can be divided
into two distinct tasks: _one-shot REG_ , finding a referring expression (RE)
to single out a referent from a set, and _REG-in-context_ , generating an RE
to refer to a referent at a given point in a discourse.

In a classic setup, REG-in-context is often approached in two steps: The first
is to decide on the form of an RE at a given point in the discourse, and the
second is to decide on its content. Many researchers have been interested in
the first sub-task, referential form selection: the task to decide which
referential form (e.g., pronoun, proper name, description, etc.) an RE takes
(McCoy and Strube, 1999; Henschel et al., 2000; Kibrik et al., 2016). Nearly
15 years ago, Belz et al. (2008) introduced the GREC shared tasks and a number
of English REG corpora with two goals: (1) assessing the performance of
computational models of reference production (Belz et al., 2009), and (2)
understanding the contribution of linguistically-inspired factors to the
choice of referential form (Greenbacker and McCoy, 2009b; Kibrik et al., 2016;
Same and van Deemter, 2020).

15 years have passed since the GREC challenge was organised, and many new
models and corpora have been proposed in the meantime (e.g., Castro Ferreira
et al. (2018); Cunha et al. (2020), and Same et al. (2022)). We, therefore,
decided that it was time to ask, in the spirit of NLP-as-Science, how well the
lessons that GREC once taught our research community hold up when scrutinised
in light of all these developments. In other words, we will investigate to
what extent the findings from GREC can be generalised to other corpora and
other models.

To this end, we pursue the following objectives: (1) We extend GREC by testing
its REG algorithms not only on the GREC corpora but also on a corpus that was
not originally considered and that has a different genre, namely the Wall
Street Journal (WSJ) portion of OntoNotes (Hovy et al., 2006; Weischedel et
al., 2013); (2) We fine-tune pre-trained language models on the task of REG-
in-context and assess them in the GREC framework.

In Section 2, we detail the GREC shared tasks and introduce the corpora used
in GREC. Section 3 spells out our research questions. In Section 4 and Section
5, we introduce the algorithms and corpora that we use. Section 6 reports the
performance of each algorithm on each corpus, followed by analyses in Section
7. Section 8 will discuss our findings and draw some lessons.

##  2 The GREC Shared Tasks

In this section, we summarise the GREC task, the corpora used by GREC, and its
conclusions.

###  2.1 The GREC Task and its Corpora

According to Belz et al., “ _the GREC tasks are about how to generate
appropriate references to an entity in the context of a piece of discourse
longer than a sentence_ " (2009, p. 297). The main task was to predict the
referential form, namely whether to use a pronoun, proper name, description or
an empty reference at a given point in discourse.

The GREC challenges use two corpora, both created from the introductory
sections of Wikipedia articles: (1) GREC-2.0 (henceforth msr, as it was used
in the GREC-MSR shared tasks of 2008 and 2009) consists of 1941 introductory
sections of the articles across five domains (people, river, mountain, city,
and country); and (2) GREC-People (henceforth neg as it was used in the GREC-
NEG shared task in 2009) contains 1000 introductory sections from Wikipedia
articles about composers, chefs, and inventors. Here is an example from neg:

David Chang (born 1977) is a noted American chef. He is chef/owner of Momofuku
Noodle Bar, Momofuku Ko and Momofuku Ssäm Bar in New York City. Chang attended
Trinity College, where he majored in religious studies. In 2003, Chang opened
his first restaurant, Momofuku Noodle Bar, in the East Village.

A key difference between msr and neg lies in their RE annotation practices. In
msr, only those REs that refer to the main topic of the article are annotated,
while in neg, mentions of all human referents are annotated. For instance, in
a document about David Chang, msr will only annotate REs referring to David
Chang, while neg will include annotations for all human referents, including
David Chang and others.

###  2.2 REG Algorithms Submitted to GREC

Various REG algorithms were submitted to the GREC challenges. These consist of
feature-based ML algorithms: CNTS (Hendrickx et al., 2008), ICSI (Favre and
Bohnet, 2009), IS-G (Bohnet, 2008), OSU Jamison and Mehay (2008) and UDel
Greenbacker and McCoy (2009a), and an algorithm that mixes feature-based ML
and rules: JUNLG (Gupta and Bandopadhyay, 2009). Table 1 presents the details
of each model, including the ML method, and the original reported accuracy on
msr (cf. Belz et al. (2009) for details).

Name | GREC ST | ALG | Acc
---|---|---|---
UDel |  msr ’09 | C5.0 | 77.71
ICSI |  msr ’09 | CRF | 75.16
CNTS |  msr ’08 | MBL | 72.61
IS-G |  msr ’08 | MLP | 70.78
OSU |  msr ’08 | MaxEnt | 69.82
JUNLG |  msr ’09 | Rule | 75.40
Table 1:  An overview of the algorithms submitted to GREC. The first column
contains the name of the respective algorithm. The column GREC ST presents the
name of the msr shared task to which the algorithm was submitted. The third
column, ALG, lists the algorithms used, where abbreviations from top to bottom
are C5.0 decision tree, conditional random field, memory-based learning,
multi-layer perceptron, maximum entropy, and frequency-based rules. The fourth
column, Acc, reports the original accuracy of the algorithms, as reported in
Belz et al. (2009). Note that UDel, ICSI, and JUNLG were submitted to both the
msr ’08 and msr’09 shared tasks, and we only present the newest results here.

###  2.3 Feature Selection

The GREC Tasks were designed to find out _what kind of information is useful
for making choices between different kinds of referring expressions in
context_ (Belz et al., 2009, p. 297). However, the original paper does not
consider the factors that contributed to the RE choice in the systems
submitted to GREC. In a follow-up study, Greenbacker and McCoy (2009b)
conducted a feature selection study informed by psycholinguistics. They
experimented with various feature subsets derived from their system, known as
UDel, which had previously been submitted to the GREC. Additionally, they
incorporated selected features from another REG system, CNTS (Hendrickx et
al., 2008), into their study. They show that features motivated by
psycholinguistic studies and certain sentence construction features have a
positive impact on the performance of REG models. Follow-up feature-selection
studies including Kibrik et al. (2016) and Same and van Deemter (2020) also
emphasise the contribution of factors such as recency and grammatical role to
the choice of RE form.

##  3 Research Questions

15 years after the GREC shared tasks, we were curious to know to what extent
the conclusions from GREC still “stand”. We, therefore, came up with the
following research questions.

In the first place, we are interested in _the impact of the choice of corpus
on the performance of REG algorithms_ ($\mathcal{R}_{1}$). GREC uses only the
introductory part of Wikipedia articles (see Section 2), which represents only
one genre of human language use. Considering that a good REG algorithm needs
to model the general use of reference, a better evaluation framework should
include texts from multiple genres. Therefore, we also include the WSJ corpus
in the study (see Section 5 for more details) and conduct a correlation
analysis to quantify how the choice of corpus impacts the evaluation results.

Second, previous studies suggested that classic machine learning (ML) based
REG algorithms perform on par with most recent neural methods (Same et al.,
2022). However, their study has three limitations: (1) they did not
incorporate pre-trained language models (PLMs); (2) they focused on the
surface forms of REs, which partly depend on the performance of surface
realisation; (3) they did not assess the models based on the intuition that a
model with good explanatory power should be less influenced by the choice of
corpus. Therefore, we adopt PLMs to the task of REG-in-context (see Section 4
for more details) and investigate _how good is the explanatory power of PLM-
based REG models compared to classic ML-based models_ ($\mathcal{R}_{2}$)
using the enhanced GREC framework.

Finally, as previously mentioned, one of the primary theoretical objectives of
GREC was to computationally explore the contribution of factors that originate
from linguistic studies to the choice of referential forms. It is reasonable
to expect that such contributions may change depending on the choice of
corpus. In this study, we conduct an importance analysis to investigate
_whether the importance ranking of linguistic factors changes when we use
different corpora_ ($\mathcal{R}_{3}$).

##  4 REG Algorithms

In what follows, we introduce the REG algorithms that are considered in this
study.

###  4.1 ML-based REG

For this study, we have narrowed our focus to feature-based ML algorithms that
predict the type of RE. Consequently, we reconstruct five ML-based REG
algorithms, namely UDel, ICSI, CNTS, IS-G, and OSU, along with their
respective feature sets, while excluding JUNLG. Note that we implement CNTS
slightly differently from Hendrickx et al. (2008). Concretely, Hendrickx et
al. (2008) have mentioned that they have used the TiMBL package (Daelemans et
al., 2007) for implementing the Memory Based Learning algorithm. Instead, we
implemented the k-Nearest Neighbors algorithm. According to Daelemans et al.
(2007), Memory Based Learning is the direct descendant of k-Nearest Neighbors.
More information on the implementation of these models can be found in
Appendix B.

###  4.2 PLM-based REG

Figure 1: Illustration of the PLM-based REG Algorithm.

Deep learning approaches have been used in many previous works on REG (Castro
Ferreira et al., 2019; Cao and Cheung, 2019; Cunha et al., 2020; Chen et al.,
2021). Different from previous work111Note that Chen et al. (2021, 2023) also
leveraged a PLM, but did not fine-tune it. Instead, they used the word
representations from the PLM as static inputs to an RNN and made predictions
using the RNN., we fine-tune PLMs on REG corpora in this study.

To fine-tune PLMs on REG corpora, we began by pre-processing each corpus using
the same paradigm as described by Cunha et al. (2020). More precisely, each
referent in a given document was replaced with its corresponding proper name.
For example, all underlined REs in Example (1) were replaced by “David Chang”.
Subsequently, as depicted in Figure 1, we fed the data into a PLM, and, for
each referent (e.g., “David Chang” ), we extracted the representations of its
first token and its last token and summed them. The final representations were
then sent to a fully connected layer for predicting the RE forms. In this
study, we use BERT and RoBERTa (see section 6.1 for more details).

##  5 REG Corpora

In the following, we explain the corpora used in this work. These corpora are
English-language corpora.

###  5.1 The msr and neg Corpora

In the current study, we only use the articles from the training sets of these
corpora (see the number of documents in Table 2). Following the same approach
as Castro Ferreira et al. (2018), we created a version of the GREC corpora for
the End-to-end (E2E) REG modelling. For the classic ML models, we reproduced
the models using the feature sets from the studies mentioned in Section 2.2.

###  5.2 The wsj Corpus

As mentioned earlier, the WSJ portion of the OntoNotes corpus (Weischedel et
al., 2013) is our third data source.222We used Ontonotes 5.0 licensed by the
Linguistic Data Consortium (LDC) https://catalog.ldc.upenn.edu/LDC2013T19. We
use the version of the corpus that Same et al. (2022) developed for E2E REG
modeling.333Note that wsj was used in Same et al. (2022), but no corpus
analysis or comparison was provided. Since empty pronouns are not annotated in
wsj, we decided to also exclude them from the two GREC corpora and focus on a
3-label classification task. The labels considered in this study are _pronoun_
, _description_ , and _proper name_. Table 2 presents a detailed overview of
these corpora.

| msr | neg | wsj
---|---|---|---
number of documents | 1655 | 808 | 582
word/doc (mean) | 148 | 129 | 530
sent/doc (mean) | 7.1 | 5.8 | 25
par/doc (mean) | 2.3 | 2.2 | 10.8
referent/doc (mean) | 1 | 2.6 | 15
number of RE | 11705 | 8378 | 25400
description % | 13.84% | 4% | 38.29%
proper name % | 38.09% | 40.79% | 34.57%
pronoun % | 41.79% | 48.75% | 27.14%
empty % | 6.28% | 6.47% | -
Table 2:  Comparison of the msr, neg, and wsj corpora in terms of their
length-related characteristics and distribution of REs. Doc, sent and par
stands for documents, sentences and paragraphs.

#### Data Splits.

We have made a document-wise split of the data. We split the wsj data in
accordance with the CoNLL 2012 Shared Task (Pradhan et al., 2012). Our wsj
training, development, and test sets contain 20275, 2831, and 2294 samples,
respectively. We did an 85-5-10 split of the GREC datasets in accordance with
Belz et al. (2009). After excluding empty pronouns, the msr training,
development, and test sets contain 9413, 519, 1038 instances, and the neg
training, development, and test sets contain 6681, 259, 896 instances.

#### Proportion of Referring Expressions

As shown in Table 2, pronouns and proper names make up 80% and 89.5% of the
referential instances in msr and neg, respectively. This implies that the
other two referential forms, namely descriptions and empty references, account
for approximately 20% of the cases in msr and about 10% in neg. Given this
imbalance in the frequency of different forms within the two corpora, we
question its potential effect on algorithm performance. Specifically, we are
wondering if forms with lower frequencies are accurately predicted by the
algorithms.

##  6 Evaluation

| msr | neg | wsj
---|---|---|---
| Acc. | F1 | wF1 | Acc. | F1 | wF1 | Acc. | F1 | wF1
UDel | 66.86 | 56.76 | 64.3 | 80.80 | 55.45 | 77.9 | 63.74 | 64.23 | 63.2
ICSI | 71.19 | 64.73 | 70.4 | 80.36 | 64.53 | 78.6 | 64.62 | 64.15 | 63.4
CNTS | 68.59 | 61.39 | 67.2 | 78.68 | 61.62 | 76.8 | 64.31 | 64.59 | 64.4
OSU | 68.02 | 60.28 | 66.6 | 79.24 | 57.04 | 76.5 | 69.20 | 69.63 | 68.9
IS-G | 67.05 | 58.83 | 65.3 | 77.34 | 59.52 | 75.6 | 69.15 | 69.35 | 69.2
BERT | 71.68 | 66.70 | 71.4 | 77.79 | 72.87 | 77.7 | 80.95 | 80.93 | 80.9
RoBERTa | 70.91 | 67.53 | 70.7 | 80.80 | 77.29 | 80.7 | 82.61 | 82.70 | 82.6
Average | 69.19 | 62.32 | 67.99 | 79.29 | 64.05 | 77.69 | 70.65 | 70.80 | 70.37
Table 3:  Overall accuracy (Acc.), macro-averaged F1 (F1), and weighted-macro
F1 (wF1) scores of the algorithms depicted in Section 4. For instance, msr-
UDel refers to a C5.0 classifier trained on the msr corpus, using the feature
set mentioned in Greenbacker and McCoy (2009a).

In this section, we introduce the evaluation protocol and report the
performance of the models.

###  6.1 Implementation Details

For BERT and RoBERTa, we used bert-base-cased and roberta-base, both from
Hugging Face. For fine-tuning, we set the batch size to 16, the learning rate
to 1e-3, the dropout rate to 0.5, and the size of the output layer to 256. We
ran each model for 20 epochs and used the one that achieved the highest F1
score on the development set. The implementation details of the classic ML-
based models can be found in Appendix B.

###  6.2 Evaluation Protocol

The main evaluation metric in the GREC-MSR shared tasks was accuracy. In
addition to accuracy, we also report macro-F1 and weighted-macro F1. We argue
that different metrics evaluate algorithms from different perspectives and
provide us with different meaningful insights. For pragmatic tasks like REG,
it makes sense to ask how well an algorithm performs on naturally distributed
data which is often imbalanced. For these cases, reporting accuracy and
weighted F1 are logical. Furthermore, analogous to other classification tasks,
minority categories should not be overlooked. Take as an example the class
_description_ in the neg corpus, which occurs only 4%. If a model fails to
produce this class, the produced document might sound unnatural. Therefore, it
is important to ensure that an algorithm is not over- or under-generating
certain classes. Looking into accuracy and macro-F1 together provides insights
into such cases.

###  6.3 Performance of the Models

The overall accuracy of the models, their macro F1, and their weighted-macro
F1 are presented in Table 3. We also present the ranking of the models based
on these scores in Appendix A.

#### PLM-based Models.

The best-performing models across all corpora and metrics are PLM-based
models. In six out of nine rankings, BERT and RoBERTa are ranked as the top
two models. The sole exception is neg, where BERT is the second worst model.
The benefit of using PLMs is the largest on the wsj corpus. For example,
RoBERTa improves the macro F1 score from 69.63 (i.e., the performance of the
best ML-based model) to 82.70.

#### ML-based Models.

In contrast to the robust performance of the PLM models, the performance of
the classic ML models is more corpus-dependent. In the case of msr and neg,
ICSI is the best-performing model, while in the case of wsj, it is at the
bottom section of the rankings. Another interesting observation is the
performance of the UDel models. In terms of accuracy, UDel has the highest
performance in neg, while it has the lowest performance in both msr and wsj.
In terms of macro-F1 rankings, the neg UDel model dropped from first to last
place, whereas BERT improved from penultimate place to second place. In
general, our ML models yielded lower scores than the original models used in
the GREC study (Belz et al., 2009). This could be attributed to a variety of
factors, including differences in feature engineering and model parameters.

#### Comparing Different Metrics.

Upon comparing average scores across the three metrics, we observe that for
msr and neg, PLMs are clear winners only when macro-F1 is the metric in
question. However, for wsj, PLMs are winners on all three metrics. This may be
because the distribution of categories in wsj is much more balanced than in
the other two corpora.

##  7 Analysis

To further compare the different models and investigate the impact of the
choice of corpus, we conduct (1) a Bayes Factor (BF) analysis to determine
whether the accuracy rates reported in Section 6 come from similar or
different distributions, (2) a per-class evaluation of predictions to assess
the success of each model in predicting individual classes, (3) a correlation
analysis to quantify how the evaluation results change with respect to the
choice of a corpus, and (4) a feature selection study to check how the
importance of each feature changes as a function of the choice of corpus.

###  7.1 Bayes Factor Analysis

|  | msr | neg | wsj
---|---|---|---|---
Model | Category | P | R | F | P | R | F | P | R | F
Udel | description | 55.36 | 19.38 | 28.71 | 0.00 | 0.00 | 0.00 | 60.29 | 62.95 | 61.59
name | 72.39 | 62.21 | 66.92 | 76.65 | 80.32 | 78.44 | 60.42 | 49.44 | 54.38
pronoun | 64.53 | 88.51 | 74.64 | 84.06 | 92.14 | 87.91 | 71.00 | 83.44 | 76.72
ICSI | description | 51.69 | 38.12 | 43.88 | 100.00 | 17.74 | 30.13 | 81.92 | 40.53 | 54.22
name | 80.33 | 66.82 | 72.95 | 81.85 | 73.14 | 77.25 | 55.12 | 86.40 | 67.37
pronoun | 69.41 | 87.39 | 77.37 | 79.05 | 94.76 | 86.19 | 72.17 | 69.61 | 70.86
CNTS | description | 53.68 | 31.88 | 40.00 | 75.00 | 14.52 | 24.33 | 64.31 | 63.67 | 63.30
name | 76.79 | 61.75 | 68.45 | 77.84 | 72.87 | 75.27 | 60.34 | 66.75 | 63.38
pronoun | 66.16 | 88.51 | 75.72 | 79.32 | 92.14 | 85.25 | 71.90 | 62.54 | 66.89
OSU | description | 53.57 | 28.12 | 36.88 | 100.00 | 4.84 | 9.23 | 72.70 | 56.91 | 63.84
name | 69.39 | 68.43 | 68.91 | 79.01 | 72.07 | 75.38 | 63.56 | 73.30 | 68.08
pronoun | 69.20 | 81.98 | 75.05 | 79.27 | 95.20 | 86.51 | 73.43 | 80.87 | 76.97
ISG | description | 57.97 | 25.00 | 34.93 | 77.78 | 11.29 | 19.72 | 73.88 | 63.41 | 68.25
name | 71.46 | 65.21 | 68.19 | 71.77 | 79.79 | 75.57 | 62.19 | 76.64 | 68.66
pronoun | 65.10 | 84.01 | 73.36 | 82.30 | 84.28 | 83.28 | 75.36 | 67.36 | 71.14
BERT | description | 52.86 | 46.25 | 49.33 | 62.71 | 59.68 | 61.16 | 82.63 | 79.37 | 80.97
name | 74.35 | 72.81 | 73.57 | 77.32 | 75.27 | 76.28 | 79.64 | 82.69 | 81.14
pronoun | 74.84 | 79.73 | 77.21 | 80.04 | 82.31 | 81.16 | 80.48 | 80.87 | 80.67
RoBERTa | description | 56.33 | 55.62 | 55.97 | 76.47 | 62.90 | 69.02 | 86.19 | 77.40 | 81.56
name | 76.50 | 64.52 | 70.00 | 78.70 | 80.59 | 79.63 | 77.22 | 89.25 | 82.80
pronoun | 71.40 | 82.66 | 76.62 | 83.04 | 83.41 | 83.22 | 86.47 | 81.19 | 83.75
Table 4:  Per-class precision, recall and F1 score of each label. The results
report on training seven different algorithms on three corpora for predicting
three labels, namely description, name, and pronoun.

Given that the accuracy scores are provided for all GREC systems in Belz et
al. (2009), we chose to focus our analysis on the raw distributions of these
scores. Our aim is to determine if there are significant differences between
the accuracies of our models by comparing these distributions. We conduct a
Bayes Factor analysis with a beta distribution of 0.01 (henceforth: the
threshold). This analysis aims to assess, for each pair of accuracies, how
strong the evidence is that they come from a common distribution, or from
different ones. A difference below the threshold indicates that accuracy rates
come from similar distributions; whereas, a difference above the threshold
indicates that they come from different distributions, thus signalling that
they differ evidentially. We interpret the strength of the evidence in favour
of/against similar/different distributions according to Kass and Raftery
(1995). Therefore, based on this approach, we expect that the raw accuracy
distributions of the best- and worst-performing models for each corpus differ
evidentially.

For msr, the comparison between the best- and worst-performing models, namely
BERT and UDel, provides no evidence that their accuracy rates are evidentially
different from each other (BF = 1.4). The same holds for neg, where the
comparison of the best (UDel and RoBERTa) and worst (IS-G) models appear to
have similar probability distributions; therefore, these models are not
evidentially different from each other. Conversely, in the case of wsj, the BF
analysis provides strong evidence that the accuracy distributions of the top-
performing models, BERT and RoBERTa, are different from those of the classic
ML models.

To summarise, we only observed significant differences in the wsj-based
models; the GREC models show more or less the same accuracy distributions. A
reason might be that the aggregated calculation of accuracy loses the
specificity of the classes being calculated.

###  7.2 Per-class Evaluation

As mentioned earlier, the neg models demonstrate high accuracy (e.g. the
highest average accuracy), but we observe a sharp decline in their macro-F1
values. In this analysis, we want to investigate whether the accuracy scores
reported in Table 3 truly reflect the success of these algorithms or if they
are merely the by-product of over-generating the dominant label or under-
generating the less frequent label. Table 4 presents the per-class precision,
recall, and F1 scores of these models.

Upon comparing the F1 scores for the class description across the three
corpora, we observe that the wsj models consistently achieve the highest
scores, with all algorithms exceeding an F1 score of 50. In contrast, the F1
scores for both msr and neg are considerably lower than those of wsj. The F1
scores for neg are particularly low, with two notable instances, UDel and OSU,
scoring 0 and below 10 respectively. The poor prediction of the class
description by the classic ML neg models is likely due to an insufficient
number of instances in the training dataset, thereby hindering the proper
training of the algorithms. In contrast, the two PLM models demonstrate
acceptable performance in predicting the class description (BERT = 61.16 &
RoBERTa = 69.02). This could indicate that pre-trained language models are
advantageous where there is a class imbalance.

Another interesting observation concerns the high recall of the “pronoun"
prediction in the neg models. Four of the classic models have a recall of over
92. In the case of OSU, for example, the recall is 95, which means that of all
the cases that are pronouns, 95% are labelled correctly. This is possibly an
indication that pronouns have been over-generated in this system. In the PLM
models, the recall is below 84.

In sum, the results of our per-class evaluation show the difficulties that the
classic ML-based neg models had in predicting the class _description_. The msr
models also had poor performance in predicting descriptions, yet they were
more successful than neg. These results tentatively suggest that feature-based
classification models need to be trained on an adequate and relatively
balanced number of instances to reliably predict all classes. The results of
this study suggest that the PLM models are less dependent on the choice of
corpus, and therefore predict classes more robustly.

###  7.3 Correlation Analysis

|  | acc | F1 | wF1
---|---|---|---|---
msr/neg | $r_{s}$ | -0.1081 | 0.9643 | 0.4643
$p$ | 0.8175 | 0.0005 | 0.2939
msr/wsj | $r_{s}$ | 0.2857 | 0.5357 | 0.4643
$p$ | 0.5345 | 0.2152 | 0.2939
neg/wsj | $r_{s}$ | -0.1261 | 0.5000 | -0.0357
$p$ | 0.7876 | 0.2532 | 0.9394
Table 5: Spearman correlation coefficient $r_{s}$ and the p-value between
every pair of corpora in terms of accuracy, macro-averaged F1, and weighted
F1.

To quantify how the evaluation results change with respect to corpora, we
compute the Spearman correlation coefficient between every pair of corpora,
indicating how the rank of the models changes. Table 5 shows the computed
coefficients along with the p-values of the tests. It is noteworthy that only
the results evaluated by the macro-weighted F1 on msr and neg are
significantly correlated ($p<.001$).

The lack of correlation between the results on msr/wsj and those on neg/wsj
suggests that using a corpus of a different genre could greatly influence the
ranking of the models and, therefore, make the conclusions difficult to
generalise. Additionally, these results are in line with the fact that msr and
neg are from the same source, both being the introductory part of Wikipedia
articles, and a higher correlation is to be expected. Also, we may conclude
that macro-averaged F1 is a more reliable evaluation metric (see the
discussions in Section 6, Section 7.1, and Section 7.2).

###  7.4 Feature Selection Study

We performed a feature importance analysis to check whether the contribution
of linguistic factors changes depending on the choice of the corpus. We used
XGBoost from the family of Gradient Boosting trees (Chen and Guestrin, 2016)
and then computed the permutated variable importance for each model. Data were
analysed in two ways: firstly, we used the complete dataset, as outlined in
Section 5; secondly, we excluded first-mention REs to concentrate only on
subsequent mentions. Considering that the choice of a referent’ first mention
is less context-dependent, we only report on the latter dataset below:

Figure 2: Different rankings of the features in msr, neg, and wsj OSU models.

As expected, the ranking of feature importance varies across different
corpora. However, a substantial overlap is observed when considering the most
important features across the three corpora. An example is the semantic
category of the REs that is used in various msr and wsj REG models.444Only
human referents are annotated in neg; therefore, this feature is not
applicable. In the case of msr, the REs belong to five semantic categories:
human, city, country, river, and mountain. In the case of wsj, the REs are
annotated for a wide range of categories including human, city, country,
organisation, objects, etc. Notably, in every model that employs semantic
category information, this feature has either the highest or second-highest
importance ranking. A plausible explanation could be that humans use different
referencing strategies to refer to different categories of referents.

In addition to the semantic category, the grammatical role of the RE and the
categorical sentential distance to the antecedent consistently have a high
importance ranking. The grammatical role marks the distinction between
subject, object, and determiner roles. The categorical distance in the number
of sentences provides information on how far an RE is to its nearest
coreferential antecedent. For instance, whether they are both in the same
sentence or are separated by one or more sentences. Figure 2 illustrates the
importance rankings of the OSU features in the three corpora. Other importance
ranking graphs are available in Appendix C. For a comprehensive description of
all features employed in classic ML models and the feature importance
analysis, refer to Same and van Deemter (2020).

##  8 Discussion

In this paper, we have conducted a series of reproductions, evaluations, and
analyses to check whether the conclusions of GREC are still true after 15
years. Below, we summarise and discuss our findings in accordance with our
three research questions in Section 3. We also report our post-hoc
observations on the choice of evaluation metric.

#### Performance of REG Algorithms.

To answer research question $\mathcal{R}_{2}$, we extended the GREC by
introducing a corpus of a different genre, wsj, and two pre-trained (PLM-
based) REG models. We found that, on msr, PLM-based and ML-based models
perform similarly, as confirmed by both the BF and per-class analyses. With
regards to neg, PLM-based and ML-based models have similar accuracy scores, as
confirmed by the BF analysis, but there are large differences when micro-F1 is
used, as confirmed by the per-class evaluation (i.e., ML-based models have
difficulty predicting descriptions). On wsj, PLM-based models are the clear
winners.

These results suggest that, in terms of explanatory power, PLM-based models
have good performance and good “direct support”, i.e., a good ability to
generalise to different contexts (see van Deemter (2023) for further
discussion). Whether they have good “indirect support” (e.g., whether their
predictions are in line with linguistic theories) needs to be investigated in
further probing studies.

#### Impact of the Choice of Corpus.

As our evaluations and analyses demonstrate, the choice of corpus plays a
crucial role in assessing REG algorithms. This role is twofold. Firstly, the
choice of corpus strongly influences the evaluation results, pertaining to the
research question $\mathcal{R}_{1}$. Secondly, in addition to the score
differences discussed in Section 6, we found that: (1) the difference between
PLM-based and ML-based models on wsj is larger (and evidentially different)
than on msr and neg models (as evidenced by the BF analysis); (2) the
correlations of the evaluation results between wsj and both msr and neg are
not significant.

For $\mathcal{R}_{3}$, we conducted feature selection analyses across the
three corpora, discovering that the importance of the features ranks
differently for each corpus. This suggests that when investigating the
“indirect support” for a model, one needs to aggregate findings from multiple
corpora with different genres.

#### The Use of Evaluation Metrics.

As we discussed in Section 6.2, different metrics evaluate different aspects
of a model. This was further ascertained by the inconsistency of the BF
analysis and per-class analysis. One lesson we have learned is that it is not
enough to report or do analyses on a single metric. Another lesson is that the
evaluation results by macro-F1 are more reliable than other metrics because
(1) they are consistent across corpora with similar genres (i.e., msr and neg;
see the Correlation analysis results); (2) the differences identified by using
macro-F1 can be confirmed by the per-class evaluation.

##  9 Conclusion

We are now in a position to address the question that we raised in the
Introduction: Can the conclusions from the GREC shared tasks still be trusted?
By examining a wider class of corpora, models, and evaluation metrics than
before, we found that the answer to this question is essentially negative
since the GREC conclusions are prone to drastic change once a different corpus
or a different metric is employed.

Perhaps this should come as no surprise. According to a widely accepted view
of scientific progress (e.g., Jaynes (2002); applied to NLP in (van Deemter,
2023)), theories should be updated again and again in light of new data (i.e.,
indirect Support), and when new models are proposed, the plausibility of
existing models should be compared against the plausibility of these new
models (as well as pre-existing ones). New metrics deserve a place in this
story as well, even though they are often overlooked. In other words, what we
have seen in the present study is nothing more than science in progress –
something we are bound to see more of as the enterprise called NLP-as-Science
matures.

#### Ethics Statement:

Regarding potential biases, in addition to the biases present in text-based
datasets, biases can also be introduced by the pre-trained language models
Bender et al. (2021) used in this work. In other words, the REG algorithms we
developed in this study may make different predictions with respect to
different genders, for instance. In the future, we plan to investigate this
phenomenon and find ways to mitigate it.

#### Supplementary Materials Availability Statement:

All associated data, source code, output files, scripts, documentation, and
other relevant material to this paper are publicly available and can be
accessed on our GitHub repository: https://github.com/fsame/REG_GREC-WSJ, DOI:
10.5281/zenodo.8182689.

#### Acknowledgements:

We thank the anonymous reviewers for their helpful comments. Fahime Same is
supported by the German Research Foundation (DFG)– Project-ID 281511265 – SFB
1252 “Prominence in Language”.

## References

  * Belz et al. (2008) Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. 2008.  The GREC challenge 2008: Overview and evaluation results.  In _Proceedings of the Fifth International Natural Language Generation Conference_ , pages 183–193, Salt Fork, Ohio, USA. Association for Computational Linguistics.
  * Belz et al. (2009) Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. 2009.  Generating referring expressions in context: The GREC task evaluation challenges.  In _Empirical methods in natural language generation_ , pages 294–327. Springer.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big.  _Proceedings of FAccT_.
  * Bohnet (2008) Bernd Bohnet. 2008.  IS-G: The comparison of different learning techniques for the selection of the main subject references.  In _Proceedings of the Fifth International Natural Language Generation Conference_ , pages 192–193, Salt Fork, Ohio, USA. Association for Computational Linguistics.
  * Cao and Cheung (2019) Meng Cao and Jackie Chi Kit Cheung. 2019.  Referring expression generation using entity profiles.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3163–3172, Hong Kong, China. Association for Computational Linguistics.
  * Castro Ferreira et al. (2018) Thiago Castro Ferreira, Diego Moussallem, Ákos Kádár, Sander Wubben, and Emiel Krahmer. 2018.  NeuralREG: An end-to-end approach to referring expression generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1959–1969, Melbourne, Australia. Association for Computational Linguistics.
  * Castro Ferreira et al. (2019) Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019.  Neural data-to-text generation: A comparison between pipeline and end-to-end architectures.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 552–562, Hong Kong, China. Association for Computational Linguistics.
  * Chen et al. (2021) Guanyi Chen, Fahime Same, and Kees van Deemter. 2021.  What can neural referential form selectors learn? In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 154–166, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * Chen et al. (2023) Guanyi Chen, Fahime Same, and Kees van Deemter. 2023.  Neural referential form selection: Generalisability and interpretability.  _Computer Speech & Language_, 79:101466.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 785–794, New York, NY, USA. Association for Computing Machinery.
  * Cunha et al. (2020) Rossana Cunha, Thiago Castro Ferreira, Adriana Pagano, and Fabio Alves. 2020.  Referring to what you know and do not know: Making referring expression generation models generalize to unseen entities.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2261–2272, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Daelemans et al. (2007) Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2007\.  Timbl: Tilburg memory-based learner.
  * Favre and Bohnet (2009) Benoit Favre and Bernd Bohnet. 2009.  ICSI-CRF: The generation of references to the main subject and named entities using conditional random fields.  In _Proceedings of the 2009 Workshop on Language Generation and Summarisation (UCNLG+Sum 2009)_ , pages 99–100, Suntec, Singapore. Association for Computational Linguistics.
  * Greenbacker and McCoy (2009a) Charles Greenbacker and Kathleen McCoy. 2009a.  UDel: Generating referring expressions guided by psycholinguistc findings.  In _Proceedings of the 2009 Workshop on Language Generation and Summarisation (UCNLG+Sum 2009)_ , pages 101–102, Suntec, Singapore. Association for Computational Linguistics.
  * Greenbacker and McCoy (2009b) Charles F Greenbacker and Kathleen F McCoy. 2009b.  Feature selection for reference generation as informed by psycholinguistic research.  In _Proceedings of the CogSci 2009 Workshop on Production of Referring Expressions (PRE-Cogsci 2009)_.
  * Gupta and Bandopadhyay (2009) Samir Gupta and Sivaji Bandopadhyay. 2009.  Junlg-msr: A machine learning approach of main subject reference selection with rule based improvement.  In _Proceedings of the 2009 Workshop on Language Generation and Summarisation_ , pages 103–104. Association for Computational Linguistics.
  * Hendrickx et al. (2008) Iris Hendrickx, Walter Daelemans, Kim Luyckx, Roser Morante, and Vincent Van Asch. 2008.  CNTS: Memory-based learning of generating repeated references.  In _Proceedings of the Fifth International Natural Language Generation Conference_ , pages 194–95, Salt Fork, Ohio, USA. Association for Computational Linguistics.
  * Henschel et al. (2000) Renate Henschel, Hua Cheng, and Massimo Poesio. 2000.  Pronominalization revisited.  In _Proceedings of the 18th conference on Computational linguistics-Volume 1_ , pages 306–312. Association for Computational Linguistics.
  * Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006.  Ontonotes: the 90% solution.  In _Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers_ , pages 57–60.
  * Jamison and Mehay (2008) Emily Jamison and Dennis Mehay. 2008.  OSU-2: Generating referring expressions with a maximum entropy classifier.  In _Proceedings of the Fifth International Natural Language Generation Conference_ , pages 196–197, Salt Fork, Ohio, USA. Association for Computational Linguistics.
  * Jaynes (2002) E.T. Jaynes. 2002.  _Probability Theory: The Logic of Science_.  Cambridge University Press, Cambridge, UK.
  * Kass and Raftery (1995) Robert E Kass and Adrian E Raftery. 1995.  Bayes factors.  _Journal of the american statistical association_ , 90(430):773–795.
  * Kibrik et al. (2016) Andrej A Kibrik, Mariya V Khudyakova, Grigory B Dobrov, Anastasia Linnik, and Dmitrij A Zalmanov. 2016.  Referential choice: Predictability and its limits.  _Frontiers in psychology_ , 7(1429).
  * Krahmer and van Deemter (2012) Emiel Krahmer and Kees van Deemter. 2012.  Computational generation of referring expressions: A survey.  _Computational Linguistics_ , 38(1):173–218.
  * Kuhn et al. (2018) Max Kuhn, Steve Weston, Mark Culp, Nathan Coulter, and Ross Quinlan. 2018.  Package ‘c50’.
  * McCoy and Strube (1999) Kathleen E. McCoy and Michael Strube. 1999.  Generating anaphoric expressions: Pronoun or definite description? In _The Relation of Discourse/Dialogue Structure and Reference_.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 1–40, Jeju Island, Korea. Association for Computational Linguistics.
  * Reiter (2017) Ehud Reiter. 2017.  A commercial perspective on reference.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 134–138, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Same et al. (2022) Fahime Same, Guanyi Chen, and Kees Van Deemter. 2022.  Non-neural models matter: a re-evaluation of neural referring expression generation systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5554–5567, Dublin, Ireland. Association for Computational Linguistics.
  * Same and van Deemter (2020) Fahime Same and Kees van Deemter. 2020.  A linguistic perspective on reference: Choosing a feature set for generating referring expressions in context.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4575–4586, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * van Deemter (2016) Kees van Deemter. 2016.  _Computational models of referring: a study in cognitive science_.  MIT Press.
  * van Deemter (2023) Kees van Deemter. 2023.  Dimensions of explanatory value in nlp models.  _Computational Linguistics_.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013.  Ontonotes release 5.0 ldc2013t19.  _Linguistic Data Consortium, Philadelphia, PA_ , 23.

##  Appendix A Ranking of the Models

#### Accuracy-based Ranking

msr: BERT $>$ ICSI $>$ RoBERTa $>$ CNTS $>$ OSU $>$ IS-G $>$ UDel

neg: UDel $=$ RoBERTa $>$ ICSI $>$ OSU $>$ CNTS $>$ BERT $>$ IS-G

wsj: RoBERTa $>$ BERT $>$ OSU $>$ IS-G $>$ ICSI $>$ CNTS $>$ UDel

#### Macro-F1 Ranking

msr: RoBERTa $>$ BERT $>$ ICSI $>$ CNTS $>$ OSU $>$ IS-G $>$ UDel

neg: RoBERTa $>$ BERT $>$ ICSI $>$ CNTS $>$ IS-G $>$ OSU $>$ UDel

wsj: RoBERTa $>$ BERT $>$ OSU $>$ IS-G $>$ CNTS $>$ UDel $>$ ICSI

#### Macro-weighted F1 Ranking

msr: BERT $>$ RoBERTa $>$ ICSI $>$ CNTS $>$ OSU $>$ IS-G $>$ UDel

neg: RoBERTa $>$ ICSI $>$ UDel $>$ BERT $>$ CNTS $>$ OSU $>$ IS-G

wsj: RoBERTa $>$ BERT $>$ IS-G $>$ OSU $>$ CNTS $>$ ICSI $>$ UDel

##  Appendix B Implementation Details for ML-based Models

The R programming language was used mostly for running the classic ML models.
The specification of the models can be found below:

#### Conditional Random Field [CRF].

The R Package CRF (https://cran.r-project.org/web/packages/crfsuite/) was used
to train these models. The iterations are set to 3000, and the learning method
is Stochastic Gradient Descent with L2 regularization term (l2sgd).

#### Decision Tree [C5.0].

The R Package C5.0 (Kuhn et al., 2018) was used to build the decision trees.
The number of boosting iterations (trials) is set to 3, and the splitting
criterion is information gain (entropy).

#### Memory-Based Learning [MBL].

As mentioned before, we implemented the k-Nearest Neighbors [KNN] algorithm
instead of MBL. The R package caret with the method KNN was used to implement
this model.

#### Maximum Entropy [MaxEnt].

The multinom algorithm from the nnet R package was used to implement this
model.

#### Multi-Layer Perceptron [MLP].

The Keras package was used to implement MLP. The model consists of two hidden
layers with 16 and 8 units, respectively. The hidden layers use the rectified
linear activation function (ReLU), and the output layer uses the Sigmoid
activation function. The model is fitted for 50 training epochs. In addition,
50 samples (batch size) are propagated through the network.

#### eXtreme Gradient Boosting [XGBoost].

XGBoost was used for the feature selection experiments. We used the R packages
xgboost and DALEXtra for the analysis. We set the learning rate to 0.05, the
minimum split loss to 0.01, the maximum depth of a tree to 5, and the sub-
sample ratio of the training instances to 0.5.

##  Appendix C Feature Importance Rankings

The graphs in Figure 3 show the rankings across msr, wsj, and wsj. A maximum
number of eight features is depicted in the graphs.

Figure 3: Importance ranking of the features in msr, neg, and wsj models.