#  _DialogSum Challenge_ : Results of Dialogue Summarization Shared Task

Yulong Chen  1, Naihao Deng ∗ 2, Yang Liu, Yue Zhang 1,3
1 School of Engineering, Westlake University
2 School of Computer Science and Engineering, University of Michigan
3 Institute of Advanced Technology, Westlake Institute for Advanced Study
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal Contribution.

###### Abstract

We report the results of _DialogSum Challenge_ , the shared task on
summarizing real-life scenario dialogues at INLG 2022. Four teams participate
in this shared task and three submit their system reports, exploring different
methods to improve the performance of dialogue summarization. Although there
is a great improvement over the baseline models regarding automatic evaluation
metrics, such as Rouge scores, we find that there is a salient gap between
model generated outputs and human annotated summaries by human evaluation from
multiple aspects. These findings demonstrate the difficulty of dialogue
summarization and suggest that more fine-grained evaluatuion metrics are in
need.

Figure 1: An example in the public test set of the _DialogSum_ dataset. Tokens
highlighted in red represent tokens that only appear in the summary but not in
the dialogue text, requiring the model to summarize with a high level semantic
understanding. Tokens hightlighted in blue represent the information that
spans across turns. Tokens highlighted in green show the corresponding
information in the dialogue text and the summary. Note that such information
scatters in various places in the dialogue.

##  1 Introduction

With the power of Pretrained Language Models (PLMs), research on text
summarization has made great progresses Liu and Lapata (2019). However,
previous research focuses on monologue summarization, such as news articles
Paulus et al. (2018); Gehrmann et al. (2018); Liu and Lapata (2019); Liu et
al. (2021), patents Pilault et al. (2020) and academic papers Koncel-
Kedziorski et al. (2019). However, as an important communicative channel
Bender and Koller (2020), dialogues receive less attention from the community.

To this end, we propose _DialogSum Challenge_ to encourage researchers to
investigate different solutuons for real-life summarization Chen et al.
(2021b). Different from previous dialogue summarization tasks Carletta et al.
(2005); Gliwa et al. (2019), _DialogSum Challenge_ focuses on diverse real-
life scenarios such as schooling, work, medication, shopping, leisure, travel
with large scale data.

The challenges of _DialogSum_ can be stated from three perspectives. First,
_DialogSum_ include a variety of topics, requiring models to process text with
different real-life scenarios. Second, compared with well-structured
monologues, dialogues have unique discourse structures and language styles
Grosz et al. (1995b). The structures and use of languages differ from
monologues, for instance, the key information spans over the context Grosz et
al. (1995a), which makes dialogues more difficult to encode. Third, compared
with monologue summarization, dialogue summaries are written from a different
perspective, usually including speakers’ intents and actions Chen et al.
(2021a). Therefore, dialogue summarization is abstractive in nature and
requires a high level understanding beyond text semantics Chopra et al.
(2016); Khandelwal et al. (2019). Figure 1 shows an example in _DialogSum_.
Apart from the research challenges, _DialogSum Challenge_ incentivizes
summarization systems that can help end users. Summarizing daily spoken
dialogues can help archive the important information in business and personal
communication. This presumably lifts the burden of manually taking notes,
liberating human beings from the tedious work.

Responding to our calls, four teams participate in the shared task, and three
of them submit their system reports. The submitted systems typically employ
PLMs, such as BART Lewis et al. (2020) and Pegasus Zhang et al. (2020a). In
addition, they explore diverse methods to improve the performance, including
integrating additional features, modifying the decoding process for better
summary generation, multi-task tuning the model with auxiliary tasks and using
data from other sources.

To evaluate the performance, we construct a hidden test set that contains 100
manually annotated samples, and evaluate models on hidden and public test sets
using both automatic and manual evaluations. For automatic evaluation, we use
Rouge scores and BERTScore. For manual evaluation, we follow _DialogSum
Challenge_ Chen et al. (2021b) and evaluate model outputs from multiple
aspects. Results show that tuning models on CNN/Daily News corpus Hermann et
al. (2015) or AMI dataset Carletta et al. (2005), and incorporating topics in
summary generation process can improve the model performance. However, there
are still rooms for models to improve the metric scores, as well as the
quality of the generated summaries in terms of identifying the interlocutors’
intents, capturing the discourse relation, etc. Besides, we observe the
mismatch between BERTScore and human scores, which aligns with the findings by
Hanna and Bojar (2021).

Full details of the shared task description and logistics, as well as the
dataset can be found at https://cylnlp.github.io/dialogsum-challenge/.

##  2 Task

Given a piece of dialogue text as input, the task is to ask a model to
generate a summary that conveys the key information of the dialogue. The
output summary should be concise, coherent, consistent and be written from a
listener’s perspective.

##  3 Data

| Public Test Set | Hidden Test Set
---|---|---
Model | R1 | R2 | RL | BERTScore | R1 | R2 | RL | BERTScore
Human | 53.35 | 26.72 | 50.84 | 92.63 | - | - | - | -
GoodBai | 47.61 | 21.66 | 45.48 | 92.72 | 49.66 | 26.03 | 48.55 | 91.69
UoT | 47.29 | 21.65 | 45.92 | 92.26 | 49.75 | 25.15 | 46.50 | 91.76
IITP-CUNI | 47.26 | 21.18 | 45.17 | 92.70 | 45.89 | 21.88 | 43.16 | 91.13
Table 1: Scores by automatic metrics for each submission and human results. We embolden the top scores among models, as well as the human score if it is the highest among all the scores. | Public Test Set | Hidden Test Set
---|---|---
Model | R1 | R2 | RL | BERTScore | R1 | R2 | RL | BERTScore
TCS_WITM | 47.02 | 21.20 | 44.90 | 90.13 | 50.32 | 25.59 | 47.40 | 91.81
Table 2: Scores by automatic metrics for the submission from TCS_WITM. The
model submitted by TCS_WITM predicts 3 summaries based on the 3 topics in the
public test set. We take the highest score among the 3 summaries to calculate
the scores.

#### Data Sources

We use the train, dev and public test data from _DialogSum_. Additionally, we
collect 100 summaries as hidden test set from the same website where
_DialogSum_ crawls the data 111http://tingroom.com. We follow the exact same
procedure as the annotation for the original _DialogSum_ dataset Chen et al.
(2021a). We remove the non-English characters, correct typos and grammatical
errors, and filter out duplicated dialogues based on text similarity 222We
compute the Rouge scores between two dialogues and filter out dialogues that
have more than $80\%$ Rouge-1 scores.. The annotators are instructed to write
the summaries for each dialogue by: (1) conveying the salient information in
the dialogue and; (2) keeping the summary short and; (3) writing from the
observer perspective and in formal language. Additionally, we ask annotators
to keep tense consistency, preserve important discourse relations, explicitly
describe emotion and speaker’s intents. Also, annotators are instructed to
provide a short topic for each dialogue. Table 3 shows the statistics for the
data in _DialogSum Challenge_.

Train | Dev | Test${}_{\text{public}}$ | Test${}_{\text{hidden}}$ | Total
---|---|---|---|---
12,460 | 500 | 500 | 100 | 13,560
Table 3: Number of dialogues in each split for _DialogSum Challenge_.

##  4 Evaluation Set-Ups

#### Automatic Evaluation

We adopt two metrics, Rouge Lin (2004) and BERTScore Zhang et al. (2020b) for
automatic evaluation. We use RoBERTa Liu et al. (2019) large as the backbone
to calculate BERTScore.

#### Manual Evaluation

Furthermore, we conduct manual evaluation from various aspects, including
standard summarization metrics Kryscinski et al. (2019, 2020), coreference
information, intent identification, discourse relation following Chen et al.
(2021a), as well as objectiveness (whether the summary is insusceptible to
subjectivity such as subjective assumptions in the dialogues). Besides,
annotators give an overview score for the predicted summary.

##  5 Submissions

###  5.1 IITP-CUNI

The model submitted by Indian Institute of Technology Patna and Charles
University employs a multi-task learning set-up to improve model performance.
In their experiments, they explore several auxiliary tasks including
extractive summarization to classify whether a given sentence belongs to the
summary or not, novelty detection Ghosal et al. (2022) of whether the given
summaries correspond to the same dialogue, as well as a masked language
modeling Devlin et al. (2019) task to recover summaries. They find that the
BART Lewis et al. (2020) large model tuned with the auxiliary task of the
extractive summarization task with data from AMI Carletta et al. (2005) corpus
performs the best.

###  5.2 UoT

The participants from the University of Tübingen use the pre-trained BART
model which is further tuned on CNN/Daily News corpus Hermann et al. (2015).
Besides, they penalize generating longer summaries in the decoding part of the
model, and post-process the summaries to resolve generation errors (e.g.
replacing speakers’ names who do not appear in the dialogue with #Person_1# or
#Person_2#, and fixing duplicated labels such as #Person_1#Person_1# to
#Person_1#).

They also explore methods such as intermediate task transfer learning with
training on commonsense reasoning task or other summarization task first and
then tune the model on _DialogSum_ , transforming dialogue structures to news
with structures similar to what BART model is trained on, as well as data
augmentation by using data from SAMSum Gliwa et al. (2019) dataset. However,
they do not find any improvement using these techniques.

###  5.3 TCS_WITM

The model from TCS research adopts the pre-trained PEGASUS Zhang et al.
(2020a) large model which is further fine-tuned on CNN/Daily News. They also
incorporate the topics provided in _DialogSum_ dataset and feed the topic
together with the dialogue text to their model. The extra information from the
topics boost up the model performance compared to the baseline performance of
simply feeding dialogues to the model.

Dialogue:
---
#Person1#: Excuse me, could you tell me how to get to the school clinic? I’ve
lost my way.
#Person2#: Yes. Go straight ahead till you come to the traffic lights, turn
left there and it’s the first turning on the right.
#Person1#: Straight ahead to the traffic lights, left and then right.
#Person2#: That’s it. It’ll take you about five minutes.
#Person1#: Thank you very much.
Gold:
Summary: #Person1# is lost on the way to the school clinic. #Person2# shows
#Person1# the correct direction.
IIPT-CUNI:
Prediction:  #Person2# shows #Person1# the way to the school clinic.
UoT:
Prediction:  #Person2# shows #Person1# the way to the school clinic.
TCS_WITM:
Prediction:  #Person2# tells #Person1# how to get to the school clinic.
GoodBai:
Prediction: #Person2# tells #Person1# how to get to the school clinic.
Figure 2: An example where all the predicted summaries miss the the context
#Person1# is lost, while all of the gold summaries contain this context.
However, all the predicted summaries successfully capture the event of
#Person2# shows #Person1# the direction.

##  6 Results

Table 1 and Table 2 show the results of human agreement and submissions from
participants in _DialogSum Challenge_ by the automatic metrics of Rouge-1
(R1), Rouge-2 (R2), Rouge-L (RL), and BERTScore. We do not compare TCS_WITM
with other models because TCS_WITM uses gold topic information.

In general, scores of the submissions are higher than the baseline models in
the original _DialogSum_ paper, demonstrating the effort from the
participants. Both submissions from UoT and IITP-CUNI tune their models on
other dataset then on _DialogSum_. In particular, UoT tunes their model on
CNN/Daily News corpus, while IITP-CUNI tunes their model on AMI dataset. The
reported results show that the model by UoT outperforms the model by IITP-
CUNI. This might be attributed to the different in training size, where AMI
Carletta et al. (2005) has 137 meetings, while CNN/Daily News corpus Hermann
et al. (2015) has 312,000 articles. Thus, the model tuned on CNN/Daily News
corpus might have better generalization ability. The model by TCS_WITM which
adopts such a method achieves 50.32 in Rouge-1 score for the hidden test case,
showing that generating the summary with the given topic can also help the
performance.

However, even the best-performed model underperforms humans by a margin larger
than 5.0 in terms of all the Rouge scores. This indicates that _DialogSum_ is
challenging and there is still a large room for future improvement.

Although existing works on summarization adopt BERTScore Gabriel et al.
(2021), we observe that the BERTScore deviates from the human scores. For
instance, though GoodBai achieves the best BERTScore on the public test set,
it is TCS_WITM with a lower BERTScore that achieves the best human scores
(Overview score in Table 4). Same phenomenon also exists for the hidden test
set. This observation aligns with the finding from Hanna and Bojar (2021) that
BERTScore performance still deviates from human. Thus, the BERTScore is still
not perfect to serve as the ultimate metric for summarization tasks, and our
community might come up with a better automatic metric that aligns with human
scores.

| Public Test Set | Hidden Test Set
---|---|---
Model | CoRef | Dis | Obj | Intent | Summ | Over | CoRef | Dis | Obj | Intent | Summ | Over
Perfect Score | 1.00 | 1.00 | 1.00 | 1.00 | 5.00 | 5.00 | 1.00 | 1.00 | 1.00 | 1.00 | 5.00 | 5.00
GoodBai | 0.96 | 0.86 | 1.00 | 0.72 | 4.12 | 3.96 | 0.90 | 0.90 | 1.00 | 0.70 | 4.20 | 4.15
UoT | 0.98 | 0.92 | 1.00 | 0.80 | 4.18 | 4.08 | 0.75 | 0.75 | 1.00 | 0.80 | 4.00 | 3.70
IITP-CUNI | 0.88 | 0.66 | 0.96 | 0.76 | 3.94 | 3.64 | 0.75 | 0.85 | 1.00 | 0.70 | 3.80 | 3.70
Table 4: Prediction results by one of the annotators of the _DialogSum_ dataset. “CoRef”, “Dis”, “Obj”, “Intent”, “Summ”, “Over” indicates coreference information, discourse relation, objective description, intent identification, standard summarization metrics and overall scores, respectively. We embolden the best scores for each column. | Public Test Set | Hidden Test Set
---|---|---
Model | CoRef | Dis | Obj | Intent | Summ | Over | CoRef | Dis | Obj | Intent | Summ | Over
TCS_WITM | 0.88 | 0.90 | 1.00 | 0.82 | 4.20 | 4.10 | 0.90 | 0.80 | 0.84 | 0.70 | 3.95 | 3.80
Table 5: Prediction results by one of the annotators of the _DialogSum_
dataset for TCS_WITM.

##  7 Human Analysis

We randomly sample 50 examples for the public test set and 20 examples for the
hidden test set to conduct manual analysis. As discussed in our proposal, we
include the metrics of coreference information, discourse relation, objective
description, intent identification, standard summarization metrics and
overview scores. We annotate -1, 0, 1 for the metrics of coreference
information, discourse relation, objective description, intent identification
where 1 means all correct, 0 means partially correct and -1 means all
incorrect. We annotate from 1 to 5 for the standard summarization metrics and
overview scores. The higher, the better.

#### Coreference Information

Whether the summary aligns interlocutors and their conversation actions or
contents.

#### Discourse Relation

Whether the summary captures important relations between main events,
identifying discourse relations and using appropriate phrases to express such
relations.

#### Objective Description

Whether the summary employs objective languages to describe dialogues.

#### Intent Identification

Whether the summary captures the interlocutors’ intents.

#### Standard Summarization Metrics

Kryscinski et al. (2019, 2020) Whether the summary is fluent, consistent,
relevant and coherent. However, in practice, we find that summaries generated
by PLMs are mostly fluent, sometimes better than human annotated summary. And
we have already evaluate consistent and coherent with more fine-grained
metrics (Coreference Information, Objective Description, etc). Thus, we focus
on relevance and judge whether the generated summary is informative and
relevant.

#### Overview Scores

Overview score of the summary with the aforementioned metrics.

Table 4 and Table 5 report the scores from the aforementioned metrics. There
is not a universal model which performs the best across all of these metrics,
instead, each model excels at certain metrics. Overall, TCS_WITM achieves the
best overview score on the public test set, while GoodBai achieves the best
overview score on the hidden test set.

##  8 Error Analysis

Dialogue:
---
#Person1#: What time is it, Tom?
#Person2#: Just a minute. It’s ten to nine by my watch.
#Person1#: Is it? I had no idea it was so late. I must be off now.
#Person2#: What’s the hurry?
#Person1#: I must catch the nine-thirty train.
#Person2#: You’ve plenty of time yet. The railway station is very close. It
won’t take more than twenty minutes to get there.
Gold:
Summary: #Person1# is catching a train. Tom asks #Person1# not to hurry.
IITP-CUNI:
Prediction: #Person1# and Tom are in a hurry to catch the nine-thirty train.
Figure 3: An error example where the model fails to distinguish the intent
between #Person1# and Tom (#Person2#).

Table 1, 2, 4, 5 show that the submitted models underperform human beings.
Here we analyze some examples where the models make mistakes or fail to
capture certain information.

Figure 3 shows an example where the model from IITP-CUNI makes a factual error
and fails to reason about who is in the hurry 333We only include one of the
gold summaries for demonstration purpose.. In order to capture the correct
relationship, the model needs to reason that “Tom” is the name of #Person2#,
and Tom (#Person2#) is asking #Person1# not to hurry by saying “You’ve plenty
of time yet”. However, the model from IITP-CUNI fails in such reasoning
processes. This suggests that reasoning about information across the dialogue
discourse is challenging.

Figure 2 shows an example where all of the model predictions deviate from the
gold summaries. All of the 3 summaries annotated by human beings include the
context of #Person1# being lost on the way. In contrast, none of the model
predictions include this context. This is plausible as the majority of the
dialogue is dedicated to #Person2# showing #Person1# the correct direction,
and the model might only capture such salient information in the dialogue.
However, the general pattern when human beings summarize is to lay out the
cause (context) of an event before telling the event, which is demonstrated in
the gold summaries. Thus, there is still a large room for improvement for the
model to generate human-like summaries.

Appendix A gives more examples of models making mistakes in terms of the
metrics from § 7.

##  9 Conclusion

We host _DialogSum Challenge_ of summarizing daily dialogue conversation at
INLG 2021. Our dataset possesses characteristics distinguished from the
existing datasets and poses new challenge to the summarization community.
There are 4 teams who submit their models during the challenges. An overview
of their methods is provided in this report. We evaluate their predictions by
automatic metrics and human analysis. Results show that there are still rooms
for models to improve the Rouge scores, as well as the quality of the
generated summaries in terms of identifying interlocutors’ intents, capturing
the discourse relation, etc. Besides, we observe the mismatch between
BERTScore and human scores, which aligns with the findings by Hanna and Bojar
(2021). Therefore, we advocate to our community to explore automatic metrics
that can better align with human scores.

## Acknowledgement

Yue Zhang is the corresponding author. We would like to thank Yi Zhang for her
help in the human analysis as well as the collection of the hidden test set.
This work receives a support from the Tencent AI Lab Rhino- Bird Focused
Research Program.

## References

  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Carletta et al. (2005) Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005.  The ami meeting corpus: A pre-announcement.  In _International workshop on machine learning for multimodal interaction_ , pages 28–39. Springer.
  * Chen et al. (2021a) Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021a.  DialogSum: A real-life scenario dialogue summarization dataset.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 5062–5074, Online. Association for Computational Linguistics.
  * Chen et al. (2021b) Yulong Chen, Yang Liu, and Yue Zhang. 2021b.  DialogSum challenge: Summarizing real-life scenario dialogues.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 308–313, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 93–98, San Diego, California. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gabriel et al. (2021) Saadia Gabriel, Antoine Bosselut, Jeff Da, Ari Holtzman, Jan Buys, Kyle Lo, Asli Celikyilmaz, and Yejin Choi. 2021.  Discourse understanding and factual consistency in abstractive summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 435–447, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Ghosal et al. (2022) Tirthankar Ghosal, Tanik Saikh, Tameesh Biswas, Asif Ekbal, and Pushpak Bhattacharyya. 2022.  Novelty detection: A perspective from natural language processing.  _Computational Linguistics_ , 48(1):77–117.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Grosz et al. (1995a) Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995a.  Centering: A framework for modeling the local coherence of discourse.  _Computational Linguistics_ , 21(2):203–225.
  * Grosz et al. (1995b) Barbara J Grosz, Aravind K Joshi, and Scott Weinstein. 1995b.  Centering: A framework for modelling the local coherence of discourse.
  * Hanna and Bojar (2021) Michael Hanna and Ondřej Bojar. 2021.  A fine-grained analysis of BERTScore.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 507–517, Online. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019.  Sample efficient text summarization using a single pre-trained transformer.  _ArXiv preprint_ , abs/1905.08836.
  * Koncel-Kedziorski et al. (2019) Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019.  Text Generation from Knowledge Graphs with Graph Transformers.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2284–2293, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2021) Yang Liu, Sheng Shen, and Mirella Lapata. 2021.  Noisy self-knowledge distillation for text summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 692–703, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Pilault et al. (2020) Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020.  On extractive and abstractive neural document summarization with transformer language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9308–9319, Online. Association for Computational Linguistics.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.  PEGASUS: pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhang et al. (2020b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A More Error Examples

Figure 4, 5, 6, 7, 8 show examples of predicting the wrong coreference
information, wrong discourse relation, summarizing with description that is
not objective, wrong intent of the interlocutors and with a low standard
summarization score as well as a low overview score, respectively.

Dialogue:
---
#Person1#: How long does it take to get to downtown from here?
#Person2#: It is 15 minutes ’ drive.
#Person1#: What companies do we have in our neighborhood?
#Person2#: Mitsubishi, HP, IBM and many other famous corporations.
#Person1#: Does the 7th floor belong to our company too?
#Person2#: Yes, our company has two floors, the 6th and 7th floor in this
building.
#Person1#: It ’ s such a large firm. Do we have our own staff restaurant?
#Person2#: Yes, at the end of the hall.
Gold:
Summary: #Person2# tells #Person1# information about their company and its
surroundings.
UoT:
Prediction: #Person2# asks #Person2# about the distance to downtown and the
companies in their neighborhood.
Figure 4: An error example for predicting the wrong coreference information.
Dialogue:
---
#Person1#: Hi! How are things going with you?
#Person2#: I am doing well. What’s up with you?
#Person1#: Believe it or not, the company I worked for closed down, so I’m out
of a job.
#Person2#: The same with me. Have you given much thought to what you want to
do next?
#Person1#: I am not being all that particular right now because I just need to
keep a roof over my head.
#Person2#: How about interviews? Have you been on any of those yet?
#Person1#: I wish that I could get the opportunity to be interviewed.
#Person2#: How about the electrician program that they have listed over there?
#Person1#: I read about that, and the position sounded great!
#Person2#: Let’s go see how we can apply for those positions.
Gold:
Summary: #Person1# and #Person2# are both unemployed. #Person2# suggests
applying for the electrician program and #Person1# agrees.
IITP-CUNI:
Prediction: #Person1# tells #Person2# #Person1# is out of a job because
#Person1# needs to keep a roof over #Person1#’s head. #Person2# suggests
applying for electrician positions.
Figure 5: An error example for predicting the wrong discourse relation.
Dialogue:
---
#Person1#: John dates her seven times a week.
#Person2#: Really? That’s a straws in the wind.
#Person1#: I think so. Maybe he’s fallen for her.
#Person2#: Yeah. They suit each other. A perfect match between a man and a
girl.
#Person1#: Right.
Gold:
Summary: #Person1# and #Person2# think that John and the girl are a perfect
match.
IITP-CUNI:
Prediction: #Person1# and #Person2# talk about John and the girl he loves.
Figure 6: An error example for summarization with description that is not
objective. Dialogue:
---
#Person1#: Can I help you?
#Person2#: I need some stamps for this letter.
#Person1#: What kind of stamps do you want?
#Person2#: How much do I need for this letter?
#Person1#: I must weigh it first. Err… It’s five grams over weigh, Do you want
to send it as an ordinary or registered letter?
#Person2#: I want it registered. How much is it then?
#Person1#: Registration plus overnight… err… seven dollars in all.
#Person2#: Here’s a 10 - dollar bill.
#Person1#: Now, your receipt, and the change.
#Person2#: Thanks. Good-bye.
Gold:
Summary: #Person2# wants to send a letter. #Person1# says it’s five grams
overweight plus overnight so seven dollars in all.
GoodBai:
Prediction: #Person1# helps #Person2# buy some stamps for a registered letter.
Figure 7: An error example for predicting the wrong intent of the
interlocutors. Gold:
---
Summary: #Person1# and #Person2# are talking about the terrorist attacks on
9-11, which was nightmarish for #Person1#’s family.
UoT:
Prediction: #Person1# and #Person2# talk about where they were for the
terrorist attacks on 9-11. #Person1# was at home with #Person1#’s parents in
New York City and #Person1# didn’t see the crash itself but
Figure 8: An example with a low standard summarization score and a low
overview score. The prediction generated by the model seems unfinished.


BRX TH-6709

CALT-TH 2022-028

All identically conserved gravitational tensors are metric variations of
invariant actions

S. Deser

Walter Burke Institute for Theoretical Physics,
California Institute of Technology, Pasadena, CA 91125;
Physics Department, Brandeis University, Waltham, MA 02454
<EMAIL_ADDRESS>
###### Abstract

I prove an old unsolved conjecture, the hard — necessary — part of its obvious
sufficiency, namely that all identically conserved symmetric $2$-tensors are
necessarily metric variations of invariant actions, thus sparing the world
more alternative gravity theories. The proof is reasonably simple, if perhaps
a “physicist’s”.

##  1 Introduction

The possible existence of covariantly conserved gravitational 2-tensors, i.e.,
those depending only on the metric and curvatures (and their covariant
derivatives) that do not stem from an action would seem to have been been
raised and solved over a century ago, but to my knowledge was only discussed
recently [1]. Given its physical importance — adding a whole new set of
alternate gravity models — this is a bit surprising, to say the least. Here I
fill this gap, showing that they are indeed excluded, at least at a
“physicist’s proof” level.

##  2 Proof

We divide all conserved tensors into two disjoint categories, those that do
not contain terms with one or more leading covariant derivatives, and those
that do. The former consists of two types. They are, of course, the Einstein
tensor $G^{\mu\nu}$ and (in $D>4$) the Lovelock (double density) tensor [2]

$L^{\mu\nu}=\epsilon^{\mu...}\epsilon^{\nu...}g_{..}\,\,\,g_{..}R_{....}\,\,R_{....}$

Its covariant conservation is manifest, by the Bianchi identities, but what is
most relevant is that it is the metric variation of $\int
1/\sqrt{-g}g_{\mu\nu}L^{\mu\nu}$ because the metric variation of the
curvatures vanishes by Bianchi as well. For Einstein, I need not belabor the
same result — here the variation of the curvature’s contribution vanishes
since it consists of two covariant derivatives $\sim DD\delta g$, to which the
explicit metric in $\int\sqrt{-g}g^{\mu\nu}R_{\mu\nu}$ is transparent. It was
proved in [2] that those are the only two conserved tensors containing second
derivatives (but no higher) of the metric. This means they have NO explicit
$D$ since they can only act on curvatures, thus making them of higher than 2
derivative order, and subject to part 2. That takes care of part 1.

There is one intermediate, Part 1.5, example, in $D=3$ only, namely, the third
derivative Chern-Simons action, whose leading $T$-term is the Cotton tensor’s
(see [3] for details). Likewise, there is an infinite variety from the basic
forms, for example just insert powers of d’Alembertians in the middle of any
actions etc. The universal point is that all $T$ must start as projectors by
the linearized argument and as can also be seen from the action’ s variations
— varying curvatures, even if there are many derivatives on them, the
outermost factor is always the projector. The apparent exceptions, Einstein
and Lovelock, are not — it’s just that there the projectors annihilate their
subjects: see e.g., the first term in (4).

##  3 Part 2 — higher derivatives

In the linearized limit, where covariant derivatives become ordinary, the most
general identically conserved symmetric tensor is
$\partial^{2}_{\alpha\beta}H^{\mu\alpha\nu\beta}$ where $H$ has the algebraic
symmetries of the Riemann tensor. There are two sub-categories, namely the
operator acting on a tensor $h_{\alpha\beta}$ to give linearized Einstein and
the still more special $P=(\partial^{2}_{\mu\nu}-\eta_{\mu\nu}\partial^{2})$
acting on a scalar $X$. These are the transverse projectors at linearized
level; there, one could also allow all sorts of ordinary derivatives standing
before these, but that is not allowed at full level because, as can easily be
seen, no $Z$ in (1) can compensate for the ensuing leading $R$ terms in their
divergences.

Now back to the full theory. The most general candidate $T^{\mu\nu}$ is the
sum of the above three projectors, each acting on its (tensor or scalar) $X$
plus a correction term $Z^{\mu\nu}$. Let me focus on the simplest case — bad
enough — to show how it goes. We have

$T^{\mu\nu}=P^{\mu\nu}X+Z^{\mu\nu}$ (1)

with the “scalar” $P$ now made from covariant derivatives; hence the
divergence of (1) begins with $R^{\mu\nu}D_{\nu}X$. What $Z$ can cancel this?
It must begin with $R^{\nu\nu}Z^{\prime}$, whose div is
$1/2RD_{\mu}Z^{\prime}+R^{\mu\nu}D_{\nu}Z^{\prime}$. Can this be done? Yes,
but we shall see that only if $T$ is action-derived. Let’s divide $Z$ into
$g^{\mu\nu}Y+R^{\mu\nu}W$ (we could have included a term proportional to
Riemann, but it is clearly useless here). The div of (1) then becomes

$D_{\nu}T^{\mu\nu}=0=R^{\mu\nu}D_{\nu}X+R^{\mu\nu}D_{\nu}W+1/2(D^{\mu}R)W+D^{\mu}Y.$
(2)

So $W$ can cancel $X$, but the last two terms must cancel internally. So the
$W$ part must be a total divergence, which is possible only if it depends on
$R$ alone, but not Ricci or Riemann, as must then $Y$, and hence $X$ as well!
That does it: The action and its variations are

$I=\int\sqrt{-g}F(R),\,\,\,\,\,\delta I=\int V\sqrt{-g}\delta
g_{\mu\nu}[(P^{\mu\nu}+R^{\mu\nu})F^{\prime}-1/2g^{\mu\nu}F],$ (3)

in agreement with (2). Note first that the linearized limit is OK, since the
explicit $R$ term is higher order, while the last term is absent since the
flat metric replaces $\sqrt{-g}$. Also, arbitrary covariant derivatives inside
$F$ are allowed, by following the reasoning below (2). There remains one final
term, one that vanishes at linear order, namely $R_{,\mu}R_{,\nu}V(R)$, whose
divergence is

$(D^{2}RR_{,\mu}+D_{\mu\nu}RR^{\,\,\,\nu}_{,})V+(DRDR)V_{,\mu}.$ (4)

Precisely the same chain that led from (2) to (3) shows that this term must
come from the action

$I=\int\sqrt{-g}R_{...}R_{,\mu}g^{\mu\nu}R_{,\nu...}$ (5)

so this time corrections to $PX$ in (1) are also required.

The moral is that conservation plus linearized limit conservation suffice to
constrain any identically conserved $T$ to be the variation of an action, at
least for the simplest category (1); strictly speaking, I have not proven this
for the slightly more general case where the $R_{,}R_{,}$ involve more
derivatives and the $V$ is a tensor. This is a physicist’s proof: I will not
push my luck nor test the reader’s patience by proving this for the mixed
cases: the reasoning is rather clear and life is short.

This completes our proof: the second category, the one with explicit leading
$D$-terms, so of higher derivative order, necessarily begins with what would
be transverse projectors at linearized level plus corrections as in our
simplest example, and also must stem from an action or else there are no
possible $Z$-correction terms.

##  4 Conclusion

I have given a physicist’s proof that all identically conserved gravity two-
tensors are necessary derivable from actions, as has always been tacitly
assumed to date. Physicist’s because I have only treated one, if generic,
case, rather than all three possible, if similar, ones.

## Acknowledgements

I thank my old collaborator, Y. Pang for useful correspondence, a referee for
insisting on better details, and J. Franklin for tech help. This work was
supported by the U.S. Department of Energy, Office of Science, Office of High
Energy Physics, Award Number de-sc0011632.

## References

  * [1] S. Deser and Y. Pang Phys Lett B790, 533(2019), arXiv 1811.03124; S.Deser ibid B795,482(2019), arXiv 1905.12831.
  * [2] D. Lovelock, J Math Phys 12, 498(1971).
  * [3] S. Deser, R. Jackiw and S.Templeton, Ann Phys 140, 372(1982).


#  RAGE Against the Machine: Retrieval-Augmented LLM Explanations

Joel Rorseth  University of Waterloo
<EMAIL_ADDRESS>Parke Godfrey  York University
<EMAIL_ADDRESS>Lukasz Golab  University of Waterloo
<EMAIL_ADDRESS>Divesh Srivastava  AT&T Chief Data Office
<EMAIL_ADDRESS>Jaroslaw Szlichta  York University
<EMAIL_ADDRESS>
###### Abstract

This paper demonstrates RAGE, an interactive tool for explaining Large
Language Models (LLMs) augmented with retrieval capabilities; i.e., able to
query external sources and pull relevant information into their input context.
Our explanations are counterfactual in the sense that they identify parts of
the input context that, when removed, change the answer to the question posed
to the LLM. RAGE includes pruning methods to navigate the vast space of
possible explanations, allowing users to view the provenance of the produced
answers.

##  I Introduction

Motivation. Artificial Intelligence (AI) has seen remarkable growth in terms
of both capability and popularity, exemplified by recent large language models
(LLMs) such as OpenAI’s ChatGPT, Microsoft’s Copilot, and Google’s Gemini. The
rapid progress in LLM capability is driven by scale, as AI researchers train
increasingly complex models with increasingly large datasets using enormous
computational resources. Within a short span, state-of-the-art models have
progressed from training millions, to billions, and now to trillions of
internal parameters. However, this increased complexity further obscures the
underlying decision-making process of LLMs, making it challenging to
rationalize or troubleshoot their outputs. As LLMs are adopted in critical
sectors, it is imperative that verifiable explanations accompany their
outputs, to build trust.

The unique enhancements in capability that distinguish LLMs from previous
language models amplify their explainability concerns. Of particular relevance
is retrieval-augmented generation (RAG), a popular prompt engineering strategy
that leverages a powerful new LLM capability known as in-context learning.
With RAG, an LLM augments its trained knowledge by learning from external
knowledge sources, supplied directly via the LLM’s input context (prompt). RAG
has been pivotal for LLMs in reducing their tendency to hallucinate plausible
yet incorrect outputs. This complex process, however, obfuscates the
provenance of the produced answers.

Background. Due to the recency of LLMs and their emergent capabilities, few
efforts have been made to explain their phenomena. Under the umbrella of
mechanistic interpretability, low-level analyses have sought to understand the
mechanisms behind transformer-based language models, and capabilities like in-
context learning, by analyzing circuits that form amongst attention heads, or
by assessing an LLM’s ability to override trained knowledge [1]. Our
explainability focus—which aims to trace the provenance of LLM answers during
RAG—instead demands high-level explanations of RAG in simple terms. While
specific prompting strategies such as chain-of-thought (CoT) prompting could
serve as interpretable explanations, RAG has yet to receive dedicated focus in
the explainability literature. RAG is a leading prompting strategy for the use
of modern LLMs in question answering (QA), as CoT and other prompting
techniques are less applicable and require specialized examples. Specific
concerns about RAG, such as the lack of provenance in LLM answers, or the
“lost in the middle” context position bias observed in recent LLMs [2],
warrant dedicated study under an explainability lens.

Contributions. To fill this gap, we demonstrate $\sf{RAGE}$,111A video is
available at https://vimeo.com/877281038.  an interactive tool designed to
enable RAG Explainability for LLMs.222The tool is available at http://lg-
research-2.uwaterloo.ca:8092/rage.  Our tool deduces provenance and salience
for external knowledge sources used during RAG, exposing the in-context
learning behaviors of the LLM. Motivated by our prior work using
counterfactual explanations for information retrieval [3], we derive
provenance counterfactually by identifying minimal context perturbations that
change an LLM’s output. Our contributions are summarized as follows.

  1. 1.

Answer Origin Explainability. We introduce a novel framework to assess the
origin of LLM answers, with respect to context knowledge sources, by
evaluating counterfactual source combinations and permutations.

  2. 2.

Pruning Strategies. We present inference pruning strategies to reduce the
space of possible counterfactual explanations, by prioritizing the evaluation
of important context perturbations.

  3. 3.

Interactive Demo. Participants will pose questions to an LLM augmented with
knowledge sources from real datasets. RAGE will display explanations for RAG
scenarios where answers are ambiguous, sourced from inconsistent external
knowledge, or traced through a chronological sequence. Participants will see
how subjective questions, such as determining the greatest professional tennis
player, can be answered differently by an LLM, depending on the combination
and order of context sources.

##  II System Description

###  II-A Problem Description

Open-book question answering is a task where a system determines an answer to
a given question using common knowledge about the topic and a provided set of
sources. In $\sf{RAGE}$, we explain how an LLM performs this task, using its
own pre-trained knowledge and retrieved knowledge sources. A user initiates
the process by posing a search query $q$ to a retrieval model $M$. Given an
index of knowledge sources and a relevance threshold $k$, the retrieval model
$M$ scores and ranks the $k$ most relevant sources from the index with respect
to query $q$. The resulting ordering of sources, denoted as $D_{q}$, forms a
sequence we refer to as the context.

We combine $D_{q}$ and $q$ to form a natural language prompt $p$ for the LLM
$L$. This prompt instructs $L$ to answer question $q$ using the information
contained within the set of delimited sources from $D_{q}$. Although $p$
serves as the final and sole input to the LLM, we denote the answer $a$
produced by the LLM for a given query $q$ and the sequence of knowledge
sources $D_{q}$ as $a=L(q,D_{q})$. We also define $S(q,d,D_{q})$ as the
relative relevance score of a source $d$ $\in$ $D_{q}$ with respect to the
query $q$ and other sources within $D_{q}$. To derive explanations, we assess
the answers generated across various combinations or permutations of the
sources in $D_{q}$. We refer in general to these two methods as context
perturbations.

###  II-B Architecture

Figure 1: The architecture of $\sf{RAGE}$.

$\sf{RAGE}$ is an interactive Python web application developed using the
Plotly Dash web framework. We installed the 7B Llama 2 Chat LLM [4] (meta-
llama/Llama-2-7b-chat-hf) through the Hugging Face Transformers library. Our
software is, however, fully compatible with any similar transformer-based LLM.
All knowledge sources (documents) are retrieved from our locally-configured
document indexes, using a BM25 retrieval model from the Pyserini retrieval
toolkit [5].

We run our application on an Ubuntu 22.04 server, with an Intel Core i9-7920x
CPU, 128GB of DDR4 RAM, and an NVIDIA RTX 4090 GPU with 24GB of memory. We use
PyTorch’s CUDA library to run LLM operations on the GPU.

In $\sf{RAGE}$, users can generate explanations in terms of two complementary
perturbations: source combinations or source permutations. Combinations
elucidate how the presence of sources affects the LLM’s predicted answer,
while permutations elucidate the effect of their order. Alongside
counterfactual explanations for each answer, $\sf{RAGE}$ presents a pie chart
to visualize the distribution of answers, a list of perturbation-answer rules,
and a table associating different answers with the perturbations that produced
them.

###  II-C RAG Explanations

In generating counterfactuals, $\sf{RAGE}$ aims to identify minimal
perturbations to the context that lead to a change in the LLM’s predicted
answer. Combination-based counterfactual explanations, which can serve as
citations, may be generated using a top-down or bottom-up search. A top-down
counterfactual must remove a combination of sources (subset of $D_{q}$) to
flip the full-context answer to a target answer. On the other hand, a bottom-
up counterfactual must retain sources to flip the empty-context answer to the
target answer.

In either case, the candidate solution search space is defined as the set of
all combinations of the given sources. We propose an iterative algorithm that
tests combinations in increasing order of subset size. Specifically, we
evaluate all combinations containing $k$ sources before moving on to those
with $k+1$ sources. Since there may be multiple combinations of equal size, we
iterate through these equal-size combinations in order of their estimated
relevance. This is calculated as the sum of the relative relevance scores of
all sources within the combination, which can be expressed as $\sum_{d\in
D_{q}}S(q,d,D_{q})$.

To estimate the relative relevance of a source $d\in D_{q}$, the user can
select from two scoring methods $S$. In the first method, we aggregate the
LLM’s attention values, summing them over all internal layers, attention
heads, and tokens corresponding to a combination’s constituent sources. In the
second method, we sum the relevance scores produced by the retrieval model for
each source. Since we only compare scores for combinations of equal size,
there is no need to normalize combination scores by the number of sources.

To generate permutation-based counterfactual explanations, $\sf{RAGE}$
searches for the most similar source permutation (with respect to their given
order) such that the LLM responds with a different answer. These explanations
quantify the stability of the LLM’s answer with respect to the order of the
context sources, thus revealing any unexpected context position bias. Our
algorithm generates all length-$k$ permutations for the $k$ sources, then
computes Kendall’s Tau rank correlation coefficient for each permutation (with
respect to their given order in $D_{q}$). Once generated and measured, the
permutations are subsequently sorted and evaluated in decreasing order of
similarity, based on decreasing Kendall’s Tau.

For both combinations and permutations, our algorithm continues until it finds
a perturbation that changes the answer, or until a maximum number of
perturbations have been tested. Before comparing against the original answer,
we convert answers to lowercase, remove punctuation, and trim whitespace.

To supplement this counterfactual analysis, we analyze the answers over a
selected set of perturbed sources. To obtain a set of combinations,
$\sf{RAGE}$ considers all combinations of the retrieved sources $D_{q}$, or
draws a fixed-size random sample of $s$ combinations. Based on the user’s
original question, a prompt is created for each selected combination, which is
then used to retrieve corresponding answers from the LLM. After analyzing the
answers, $\sf{RAGE}$ renders a table that groups combinations by answer, along
with a pie chart illustrating the proportion of each answer across all
combinations. A rule is determined for each answer, when applicable,
identifying sources that appeared in all combinations leading to this answer.

In a similar manner, the user can instruct $\sf{RAGE}$ to analyze answers from
a selected set of source permutations. The table and pie chart illustrating
associations between answers and permutations resemble those of the
combination case, with the rule calculation adopting a unique definition. For
each answer, we determine a rule that identifies any context positions for
which all permutations leading to this answer shared the same source. Users
may again choose to analyze all permutations, or a fixed-size random sample of
$s$ permutations.

For the latter, a naive solution might generate all $k!$ permutations of the
$k$ sources, then uniformly sample $s$ permutations, resulting in $O(k!)$ time
complexity. To improve the efficiency, we propose an implementation using the
Fisher–Yates shuffle algorithm [6], which produces an unbiased random
permutation of any finite sequence in $O(k)$. In our approach, we invoke the
Fisher-Yates algorithm $s$ times to generate $s$ random permutations,
resulting in an efficient $O(ks)$ solution.

$\sf{RAGE}$ also allows the user to analyze the most optimum permutations. As
observed in recent works [2], LLMs often exhibit a context position bias,
paying more attention to sources appearing at the beginning and end of the
context than those in the middle. As a result, sources that are important for
obtaining a given answer may not receive sufficient consideration by the LLM.
Given a distribution of the expected attention paid to each position, this
“lost in the middle” bias can be counteracted by positioning important sources
in high-attention positions. By requesting “optimal permutations” from
$\sf{RAGE}$, the user can analyze a set of permutations with optimum placement
of relevant sources in high-attention positions.

To estimate the relevance of a source, the user can choose to use either the
LLM’s attention scores or the retrieval model’s assessed relevance score. If
desired, the user can calibrate the expected distribution of LLM context
position attention by selecting a predefined V-shaped distribution. Optimal
permutations aim to maximize both the relevance and attention of their
constituent sources. A naive $O(k!)$ solution might generate all $k!$
permutations, scoring each by summing the product of each source’s relevance
and attention, then sorting and selecting the $s$ highest-scoring
permutations. Recognizing that optimal permutations must maximize both the
relevance and attention of their constituent sources, we propose an efficient
solution by formulating this problem as an instance of the _assignment
problem_ in combinatorics.

Numerous algorithms have been proposed to solve this problem, which aim to
find the most optimal assignment of all $k$ sources to all $k$ context
positions. Since $\sf{RAGE}$ allows the user to request the top-$s$ optimal
permutations, our formulation adopts a variant of the assignment problem that
seeks the $s$ assignments with minimal cost. We use the algorithm proposed by
Chegireddy and Hamacher [7], which allows us to calculate the $s$ optimal
permutations in $O(sk^{3})$.

Figure 2:  Combination insights for the query about The Big Three.

##  III Demonstration Plan

Conference participants will explore the provenance of information included in
retrieval-augmented LLM responses. They will then reinforce these findings by
evaluating the importance of relative position among sources.

###  III-A Categorization of Use Cases

The explanations generated by $\sf{RAGE}$ are applicable across countless
domains. Use cases can be categorized based on various factors, such as
whether knowledge sources form a timeline, or when questions are subjective,
leading to ambiguous answers. In the former case, $\sf{RAGE}$ identifies
salient periods in time. In the latter case, it procures evidence to support
various answers. Knowledge sources may differ in terms of their consistency.
Our tool can identify consistent and inconsistent sources. Sources may or may
not share semantic dependencies, and may or may not share syntactic formats.
$\sf{RAGE}$ will highlight source agreement and disagreement.

In the following subsections, we introduce several use cases that highlight
the axes of this categorization. We begin by exploring the possibility of an
ambiguous answer, which requires efficient evaluation over a large sample of
knowledge source combinations. Next, we present a scenario in which sources
are slightly inconsistent, testing $\sf{RAGE}$’s ability to identify minor
differences that can change the LLM’s answer. Last, we provide an example in
which the sources form a timeline, requiring $\sf{RAGE}$ to strategically
navigate alternate timelines by minimally combining and permuting the sources.

###  III-B Use Case #1: Ambiguous Answers

The user asks an LLM to determine the best tennis player among “The Big Three”
of Novak Djokovic, Roger Federer, and Rafael Nadal. The user does not have any
specific comparison metric in mind, so they use the system to retrieve a set
of related documents, each containing a different ranking of The Big Three
based on different metrics (e.g., total number of match wins and number of
weeks ranked first). The user expects that Novak Djokovic, who recently
surpassed Rafael Nadal and Roger Federer in total Grand Slam wins, might be
the LLM’s choice. But when asked with the combination of all retrieved
documents, the LLM’s answer is “Roger Federer.”

Curious about why the LLM chose Federer, the user poses the same query and
documents to $\sf{RAGE}$, requesting combination insights. As illustrated in
Figure 2, $\sf{RAGE}$ analyzes the answers generated by the LLM using various
combinations of the given documents, and discovers that the first document led
the LLM to produce this answer. This document ranks various tennis players
based on total match wins, with Federer ranking first at 369. $\sf{RAGE}$’s
answer rules formalize this explicitly, asserting that this document was
included in every combination for which the LLM answered “Roger Federer.”

The user now comprehends why the LLM chose Federer but remains curious about
the document’s relative significance. Reviewing the original ranking, they
notice that this document has prominent placement at the beginning of the
context. To investigate the impact of this position, the user requests
permutation-based explanations for the same inputs. Surprisingly, $\sf{RAGE}$
reveals that moving the document to the second position altered the answer to
“Novak Djokovic.” In short, these explanations have enabled the user to
promptly identify the document that influenced the LLM’s answer, and to
understand the impact of its relative position.

###  III-C Use Case #2: Inconsistent Sources

The user turns to an LLM for help in determining the most recent winner of the
US Open women’s tennis championship. A small set of documents is retrieved,
each containing relevant statistics about US Open championships. The documents
share similar format, but some may be more current than others. Hoping that
the LLM will pinpoint the most recent winner across all documents, the user
requests combination insights in $\sf{RAGE}$, and observes how the combination
containing all sources produces the response “Coco Gauff.” With no further
explanation, the user aims to verify this result by identifying the source
document behind the answer, and discovers that the last context document
recognizes Gauff as the 2023 champion.

Curious whether other out-of-date documents could have been mistakenly sourced
for an incorrect answer, the user asks $\sf{RAGE}$ to derive permutation
insights. By reordering the context documents in various configurations and
analyzing the resulting answers, $\sf{RAGE}$ discovers that the LLM
incorrectly identifies the 2022 champion “Iga Swiatek” whenever the last
document is moved towards the middle of the sequence. Using $\sf{RAGE}$, the
user has identified the up-to-date document that offers the correct answer,
and has gleaned insights about out-of-date documents and their ability to
confuse the LLM.

###  III-D Use Case #3: Timelines

The user consults an LLM to determine how many times Novak Djokovic won the
Tennis Player of the Year award between 2010 and 2019. The user gathers
relevant documents from the system, each corresponding to one year’s winner.
Collectively, the documents form a timeline for the three winners: Rafael
Nadal (2010, 2013, 2017, 2019), Novak Djokovic (2011, 2012, 2014, 2015, 2018),
and Andy Murray (2016).

The user poses their question to $\sf{RAGE}$, which reports that the LLM
produces the expected answer of 5 when incorporating the combination of all
retrieved documents. To validate the LLM’s response, the user expects an
explanation listing each year Djokovic won the award, along with a citation to
a supporting document. To achieve this, the user reviews the combination
counterfactual generated by $\sf{RAGE}$ to determine the minimal set of
documents (and thus the exact years) required to infer the correct answer.
$\sf{RAGE}$ cites five separate documents from those provided, each
documenting a different year in which Djokovic won Player of the Year.

Hoping to ensure that the LLM has not overlooked any time period covered by
the documents, the user asks $\sf{RAGE}$ to derive permutation insights over
the same inputs. By analyzing a sample of permutations, the user is presented
with a pie chart and answer table that indicate a consistent answer of 5. The
user observed that no rules were found, and concludes that the LLM
consistently comprehends the entire timeline of the twenty-tens, regardless of
the specific order of the timeline’s constituent documents. Through
$\sf{RAGE}$, the user has successfully discovered which segments of the
timeline were crucial in determining the correct answer.

## References

  * [1] J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou _et al._ , “Larger language models do in-context learning differently,” _arXiv preprint arXiv:2303.03846_ , 2023.
  * [2] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” 2023, arXiv:2307.03172.
  * [3] J. Rorseth, P. Godfrey, L. Golab, M. Kargar, D. Srivastava, and J. Szlichta, “Credence: Counterfactual explanations for document ranking,” in _ICDE_ , 2023, pp. 3631–3634.
  * [4] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv:2307.09288_ , 2023.
  * [5] J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, and R. Nogueira, “Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations,” in _SIGIR_ , 2021, pp. 2356–2362.
  * [6] R. A. Fisher and F. Yates, _Statistical tables for biological, agricultural aad medical research_. Edinburgh: Oliver and Boyd, 1938.
  * [7] C. R. Chegireddy and H. W. Hamacher, “Algorithms for finding k-best perfect matchings,” _Discrete Applied Mathematics_ , vol. 18, no. 2, pp. 155–165, 1987.