# An Exam-based Evaluation Approach
Beyond Traditional Relevance Judgments

Naghmeh Farzi<EMAIL_ADDRESS>University of New HampshireUSA and  Laura
Dietz<EMAIL_ADDRESS>University of New HampshireUSA

###### Abstract.

Current IR evaluation is based on relevance judgments, created either manually
or automatically, with decisions outsourced to Large Language Models (LLMs).
We offer an alternative paradigm, that never relies on relevance judgments in
any form. Instead, a text is defined as relevant if it contains information
that enables the answering of key questions. We use this idea to design the
EXAM Answerability Metric to evaluate information retrieval/generation systems
for their ability to provide topically relevant information.

We envision the role of a human judge to edit and define an exam question bank
that will test for the presence of relevant information in text. We support
this step by generating an initial set of exam questions. In the next phase,
an LLM-based question answering system will automatically grade system
responses by tracking which exam questions are answerable with which system
responses. We propose two evaluation measures, the recall-oriented EXAM Cover
metric, and the precision-oriented EXAM Qrels metric, the latter which can be
implemented with trec_eval. This paradigm not only allows for the expansion of
the exam question set post-hoc but also facilitates the ongoing evaluation of
future information systems, whether they focus on retrieval, generation, or
both. 111Data and code available at https://github.com/laura-
dietz/flan-t5-exam-appendix

LLM

    Large Language Model

##  1\. Introduction

Large Language Models are capable of generating and/or retrieving responses
for search queries, resulting in a plethora of systems that combine
traditional retrieval with neural ranking and natural language generation.
Ideally the systems’ responses cover relevant information content while being
concise and complete. However, there is a need for convincing evaluation
metrics to assess the accuracy and completeness of the information content in
responses. This should be accomplished in a repeatable and reusable manner and
without resorting to expensive human judgments.

In the remainder of the paper we will focus on the evaluation task of the
following process:

##### Task Statement.

An information retrieval/generation system is given a search _query_ to
produce a relevant _system response_. The response can take the form of a
passage ranking, a set of extractive summaries, or a single (generated) text.
Given system responses across multiple queries from multiple systems, the task
is to assign each system an _evaluation score_ that represents the quality of
the information content provided in the system response.

A popular evaluation approach is to ask LLMs whether a passage is relevant for
a query. Empirically this has been shown to work well (Faggioli et al., 2023;
MacAvaney and Soldaini, 2023; Thomas et al., 2023, –inter alia). However,
skepticism remains about whether LLMs can be trusted to completely replace
humans in the judgment process (Faggioli et al., 2023).

A solution may be to integrate a human oversight into the evaluation process.
While using human judges to verify automatic relevance labels is a common
suggestion, it can be a tedious task without additional context.

. The common suggestion is to employ judges to verify automatic relevance
labels. However, without further context, this remains a tedious task. While
LLMs can be asked to produce a rationale to support their labeling decision,
studies indicate that human judges might overly trust rationales generated by
LLMs (Fok and Weld, 2023).

Acknowledging that judging passage-level relevance is a daunting task—being
simultaneously cognitively challenging and boring—we propose a more “human-
friendly” way to leverage human labor for defining what constitutes relevant
information content. In this paper we focus on the empirical evaluation of an
alternative automatic approach, leaving human-subject studies for future work.

Figure 1. EXAM approach. Left: The only information available to the system
that is being evaluated are queries and, optionally, a text corpus. The system
response can be a ranking of passages, or a generated response that will be
segmented into passages (blue). Passages from all systems will be pooled for
assessment, and additional passages can be added at a later if needed. Right:
The EXAM evaluation system uses three phases detailed in Section 3. For each
query, an exam question bank is developed, which can be modified later in an
iterative fashion (purple). All passages from the system response (e.g., p1,
p2, p3) are graded based on which questions (q1?, q2?, …, q5?) can be
correctly answered with the passage text (red). We support two modes: one
where answers are verified against an answer key (depicted as check marks), or
by having an LLM self-rate the answerability on a scale from 0 to 5. The EXAM
evaluation scores are derived from these grades (green). The EXAM Cover score
is based on how many questions are covered, as binary verification or via a
minimum self-rating level. For EXAM Qrels a relevance file for trec_eval is
derived, which is based on the coverage or best self-rating obtained by this
passage in isolation. We provide a worked example in Section 4.5

Our goal is to develop an evaluation approach that,

  1. (1)

is amenable to integrating a human into-the-loop in a less daunting way,

  2. (2)

never requires manual passage-level relevance judgments,

  3. (3)

benefits from latest advances in Large Language Models

  4. (4)

yields reusable test collections that can evaluate (future) systems,
especially those that employ language generation

  5. (5)

allows to expand the test collection post-hoc to reveal differences between
systems

Of course, such an evaluation approach has to agree with traditional
evaluation paradigms, as quantified by rank correlation of system
leaderboards.

We propose that, for every query, information retrieval generation systems
take an exam for which they did not study. Using a query-specific question
bank of exam questions, the EXAM Answerability Metric determines whether
important information content is provided in the system response to that
query.

As part of topic development, exam questions need to be designed for each
piece of desired information content. Whenever it is possible to answer this
question with the system’s response, we consider this piece of information
content covered.

The idea was first introduced by Sander and Dietz (2021), who envision that
(1) human judges would manually create the exam question bank, while (2) an
automated question answering system performs the daunting task of grading each
system response. The system’s EXAM evaluation score (3) is the higher, the
more questions can be correctly answered based on the system’s response.

##### Contributions.

We significantly expand on the EXAM idea with our proposed approach
FLAN-T5-EXAM, depicted in Figure 1.

  * •

FLAN-T5-EXAM supports development of exam question banks with prompt-based
generation,

  * •

modernizes the question answering system with the recently released FLAN-T5
family (Longpre et al., 2023),

  * •

proposes an EXAM-variant that is inter-operable with commonly used evaluation
tools (e.g. trec_eval)

  * •

expands the empirical evaluation to additional datasets from TREC DL 2019 and
2020 and baselines from automated relevance prompting (Sun et al., 2023;
Faggioli et al., 2023; Thomas et al., 2023).

A strength of this approach is that, in contrast to work LLM-based relevance
grading, we can readily integrate humans into the evaluation (Goal 1). Based
on the long history of classroom education and exams, we argue it is more
natural for human judges to control the design of exam questions than to
directly provide relevance judgments. These could be developed manually from
scratch, on the basis of system responses, or in dialog with a chat-LLM. In
this work we focus on the two extreme cases: fully manually created exam
questions and fully automatically generated exam questions—leaving human
subject studies to future work.

By virtue of automating the grading of system responses, human judges are
never required to perform passage-level relevance assessments (Goal 2). The
evaluation approach yields reusable test collections (Goal 4) that can be
expanded by modifying the question bank (Goal 5) at any point in the
evaluation process, as the remaining pipeline is fully automated. The impact
of a question bank modifications can be directly observed by listing passages
whose relevance grade would change.

At the same time, humans are fully in control of defining which information
content is relevant via the exam question bank.

##  2\. Related Work

We focus on an approach that does not require passage-level relevance
judgments or source texts. Our work is unique in this regard, but aspects
relate to many active branches of research, which we detail below.

###  2.1. LLM-based Relevance Assessment

While our approach does not attempt to imitate the relevance-judgment process,
several recent methods studied this approach.

Sun et al. (2023) rerank passages using a simple LLM prompt “does the passage
answer the query?” We use their prompting technique as a baseline “Sun” for
comparison.

Faggioli et al. (2023) conduct an early experiment by asking an LLM to judge
the relevance of a passage. They design a simple prompt “Fag” and a more
elaborate multi-relevance few-shot prompt developed for TREC DL.

Thomas et al. (2023) compare the ability of LLMs to perform document-level
relevance judgments in comparison to different groups of human annotators.
They find that especially the label quality of crowd-workers is inferior to
fully automatic LLM-based relevance labels. Thomas et al. are using a very
detailed prompt (Figure 1 in (Thomas et al., 2023)), clarifying the role and
query description and asking the LLM to comment on the query intent and
trustworthiness. As the full prompt well exceeds the 512 token boundary of
FLAN-T5-large models, we use an abridged prompt “Thom”.222All baseline prompts
are provided in our online appendix.

In 1SLs, MacAvaney and Soldaini (2023) focus on evaluating passages with a
DuoPrompt, that instructs an LLM to indicate which of two passages is more
relevant for a query. In contrast, our approach does not fall in the paradigm
of pair-wise ranking approaches.

Several voices raised critiques about using LLM’s for relevance labels in
general. Faggioli et al. (2023) elaborates a wide-range of theoretical
concerns, centered on questions of trustworthiness and reliability of LLMs now
and in the future. Wang et al. (2023) empirically demonstrate that LLMs
exhibit unfair positional bias towards candidates displayed for evaluation.
Fok and Weld (2023) studies general issues of human over-reliance and under-
reliance on LLMs. They elaborate why rationales produced by LLMs for human
verification do generally not lead to improvements.

###  2.2. Nugget-based Evaluation

There is a long history in the IR community to evaluate the relevance of
documents by breaking down the information need into a set of “nuggets” (also
called query intents, facts, or SCUs) that can each be evaluated independently
(Lin and Demner-Fushman, 2006). The common definition of a nugget is “the
smallest portion of text that constitutes relevant information in and of
itself” (Pavlu et al., 2012). While text is defined as relevant as soon as it
matches any nugget, the nugget document score is based on the sum of shingle
match scores in any document—to be considered as relevant when surpassing a
threshold.

With the advent of LLMs, nugget-style evaluation is being revamped, most
recently in the TREC Crisis Facts track (McCreadie and Buntain, 2023): Judges
are asked identify atomic “facts” (similar to nuggets). System responses are
analysed for mentions of these facts, either via a boolean OR or with an
embedding-based method.

Our proposal is related in that our questions are intended to capture nuggets.
A hurdle in using nugget-based evaluation method is to reliably match nuggets
to text content. In contrast, with FLAN-T5-EXAM, we are leaning on decades-
long research of factoid question answering—which is now considered a solved
task.

###  2.3. Evaluation with Test Questions

The idea of basing an evaluation on a bank of test questions has been widely
discussed in literature on summarization (Clarke and Lapata, 2010), recently
with automated question answering methods. Eyal et al. (2019) suggest a system
evaluation score that is based on the number of questions that a question
answering system can correctly answer using the system response—a principle
that both the original EXAM method and our approach follows.

In contrast to our approach, most approaches use a Cloze-style idea to
generate questions from a given gold summary or source text. Questions can be
in the form of multiple-choice questions (Huang et al., 2020), free text
questions with exact-match answer verification (Deutsch et al., 2020), or be
derived from extracted entities and relations (Wang et al., 2020; Eyal et al.,
2019). A full survey is provided by Mulla and Gharpure (2023).

As it pertains to information retrieval evaluation, the problem with
generating questions from a given text or gold summary, is that (1) such a
gold summary is usually not available and (2) it is unclear which of these
questions relate to relevant information in the gold summary (or source text).

The original EXAM method avoids this problem altogether by asking a human to
design questions that address the search query. In contrast, we propose to
automatically generate questions directly from the query, building on the
world-knowledge of ChatGPT—with the intention of employing manual labor to
verify or weight the question set.

###  2.4. LLMs, Passages, and Questions

Many approaches integrate passages, questions, and LLMs in some form. This
includes improving question answering via retrieval-augmentation (Sachan et
al., 2022; Li et al., 2023). Improve factuality in fact verification, by
breaking each claim down into several questions (Zhang and Gao, 2023).
Exploiting the self-verification ability of LLMs to improve the reasoning
(Weng et al., 2023). Evaluating the quality of LLMs with multiple tests (Liang
et al., 2022; Chang et al., 2023).

Arabzadeh et al. (2024) develop an approach to evaluate retrieved passages via
questions in a very different way: For search queries that resemble questions
(as in TREC DL), LLMs are asked to generate a correct answer and an incorrect
“liar” answer. These responses as used as positive/negative gold answers for
retrieved passages: If the embedding-based similarity of the passage is closer
to the correct answer than the incorrect answer, the passage is deemed
relevant.

###  2.5. Difference to Original EXAM (Sander and Dietz, 2021)

Our contribution differs from the original EXAM method in several important
ways:

1\. Obtaining an Exam Question Bank::

To obtain exam questions,

  * •:

The original EXAM method is based on manually created multiple-choice exam
questions.

  * •:

We propose to semi-automatically generate free-text questions for each query,
as described in Section 3.1.1.

2\. Grading System Responses::

To grade each passage via the answerability of exam question,

  * •:

The original EXAM method uses a pre-neural multiple-choice question answering
system with answer verification.

  * •:

First, we propose to modernize the question answering system with an LLM-based
approach (Section 3.2.2).

  * •:

Second, we explore the ability of LLMs to self-rate the answerability of a
question with given context, without directly verifying the correctness of
answer (Section 3.2.3).

3\. EXAM Cover Evaluation::

To evaluate each system,

  * •:

With EXAM Cover, we follow the original EXAM method by evaluating systems
according to the number of answerable exam questions (Section 3.3.1).

  * •:

To improve adoption, we add a variant “EXAM Qrels” that implements a related
idea so that it is inter-operable with the popular evaluation tool trec_eval
(Section 3.3.2).

##  3\. Approach

The EXAM Cover metric quantifies the information content of retrieved,
generated, or retrieval-augmented generated information. Using a question bank
of exam questions, the systems are graded based on the set of exam questions
that can be correctly answered based on the system’s response: The more
questions can be answered, the higher the EXAM score of the system.

Our EXAM evaluation system assumes the following inputs:

  1. (1)

A set of queries, optionally with query subtopics.

  2. (2)

A set of system responses, which can come in the form of a passage ranking or
as generated content.

Note that the exam questions are secret, and not available to the retrieval
generation system.

The EXAM evaluation approach is structured into the following three phases
that we detail in the remainder of the section, and depict in Figure 1.

1\. Obtaining an Exam Question Bank::

A process of creating an exam of questions, with or without gold standard
answer keys.

2\. Grading System Responses::

All passages in system responses are graded using an automated question
answering system. The set of answerable questions is tracked for each passage.

3\. EXAM Evaluation::

The system’s evaluation score that is the higher, the more exam questions can
be answered with any of the first $k$ passages of the system’s response.

Table 1.  Question generation prompts (GenQ).  Question Generation: TREC DL Prompt |  Question Generation: TREC CAR Y3 Prompt
---|---
Break the query ’{query_title}’ into concise questions that must be answered. Generate 10 concise insightful questions that reveal whether information relevant for ’{query_title}’ was provided, showcasing a deep understanding of the subject matter. Avoid basic or introductory-level inquiries. Keep the questions short and in a Python list format. |  Explore the connection between ’{query_title}’ with a specific focus on the subtopic ’{query_subtopic}’. Generate insightful questions that delve into advanced aspects of ’{query_subtopic}’, showcasing a deep understanding of the subject matter. Avoid basic or introductory-level inquiries. Give the question set in a Python list format.
Table 2.  Grading prompts.  Grading: Question Answering Prompt
---
provide a complete and concise answer to the question based on the context.
Question: {question} Context: {context}
Grading: Self-rating Prompt
Can the question be answered based on the available context? choose one: \- 5:
The answer is highly relevant, complete, and accurate. \- 4: The answer is
mostly relevant and complete but may have minor gaps or inaccuracies. \- 3:
The answer is partially relevant and complete, with noticeable gaps or
inaccuracies. \- 2: The answer has limited relevance and completeness, with
significant gaps or inaccuracies. \- 1: The answer is minimally relevant or
complete, with substantial shortcomings. \- 0: The answer is not relevant or
complete at all. Question: {question} Context: {context}

###  3.1. EXAM Question Banks

####  3.1.1. Generating EXAM Question Banks (GenQ)

We employ a generative LLM to automatically generate free-text- questions for
each query.

In our experiment, we use ChatGPT to obtain question banks. The prompt is
designed to elicit concise, insightful questions based on the query, tailored
to specific goals of the IR task and domain. In application to TREC CAR Y3, we
ask for questions that explore the connection between the query with a
specific focus on each subtopic. For TREC DL we ask to break the query into
concise questions. The complete prompts used in the experimental evaluation
are listed in Table 1.

We obtain free-text questions without answer key from these prompts, such as
the following for TREC CAR Y3 query tqa2:L_0384:

> Query title: The Integumentary System
> Query subtopic: Structure of the Skin
>
>   * q1?
>
> How does the epidermis, dermis, and hypodermis work together to provide
> protection, sensation, and regulation for the body?
>
>   * q2?
>
> Can the integumentary system be compromised by diseases and conditions, and
> if so, how does this impact the health of the skin?
>
>   * q3?
>
> How does the skin act as a barrier against pathogens and other foreign
> substances?
>
>

####  3.1.2. Manual Verification and Refinement of EXAM Question Banks

Our method facilitates the inclusion of a human-in-the-loop when constructing
the exam question bank. For example, the human judge could verify that the
question bank is covering essential information content for the query.
Questions can be changed, added, or deleted at any time: during topic
development and during a post-evaluation analysis. As better systems are being
developed, more difficult questions can be added to reveal quality differences
between systems.

While our evaluation system is designed to support the human-in-the-loop, we
leave the user study to future work.

####  3.1.3. Manual Question Banks

Alternatively, question banks can be created from scratch, as suggested for
the original EXAM method. If questions are associated with a gold answer key,
it can be used for answer verification. However, we acknowledge that the
requirement of an answer key renders the topic development more difficult. TQA
collection (Kembhavi et al., 2017) provides an exam question bank of multiple-
choice tests for topics in TREC CAR Y3.

###  3.2. EXAM Grading

In this phase, we conduct a secret exam using the system responses as
background information. We track which of the questions are answerable with
which passage.

The original EXAM method uses a pre-neural question answering system from AI2,
which was designed to answer multiple-choice questions with given context. The
answer is verified as correct when the question answering system responds with
the correct choice. The downside is that this question answering system was
difficult to set up,333We were unable to install the question answering system
used in the original EXAM method. and was specifically designed for questions
in the style of the ARC and TQA datasets. A goal of our work is to modernise
this question answering system and allow for broader applicability.

####  3.2.1. Pre-processing System Responses

To initialize the grading phase, we pre-process system responses. Longer
responses are segmented into paragraph-sized passages, each approximately 400
tokens in length. In case of submitted passage rankings, each passage will
graded individually. We convert each passage into plain text and assign a
unique passage_id for later use.

####  3.2.2. LLM-based question answering with Answer Checking

To take advantage of the latest development in LLMs, we propose to use an LLM-
based question answering system. While any LLM could be used, in this work we
experiment with the FLAN-T5 family that became recently available. In
particular we focus on the FLAN-T5-large model which can complete annotations
within a reasonable time on an NVIDIA A40 GPU.

##### Question prompt.

We perform some initial prompt engineering444A version of FLAN-T5-large fine-
tuned on the SQuAD2 was also considered. to ensure that the LLM is answering
the question with the provided context, and not from its own memory. For the
FLAN-T5-question answering system we use a prompt to instruct the LLM to
“provide a complete and concise answer to the question based on the context”.
The full prompt is given in Table 2.

Using separate requests to avoid position bias, we obtain answers each exam
question on all passages. Whenever the prompt exceeds the 512 token boundary,
we truncate the given context but ensure that the question remains intact.

##### Answer verification.

When a known correct answer is available for the question bank, we verify the
predicted answer with a heuristic matching function: Both the correct and
predicted answers are normalized by lower-casing, stopword removal, and
stemming. A match is considered when the edit distance between these strings
is less than 20% of the length of the longer string. This heuristic has been
manually verified to yield accurate matches, effectively reducing false
negatives compared to strict exact matching.

####  3.2.3. EXAM Grading by Self-rating

Since our focus is on whether a question is answerable within a given context,
rather than the answer itself, we explore a way to avoid difficulties in
producing gold answers and the verification of predicted answers altogether.
We lean on the ability of modern LLMs to match language patterns and observed
self-verification behavior (Weng et al., 2023) and ask the LLM self-rate the
answerability of a question with the context. We use the prompt given in Table
1.

We find that in the vast majority of cases, FLAN-T5 indeed responds with a
numerical code between 0 and 5. In the remaining cases, we assign a rating of
1 by default or 0 when expressions of unanswerability are
encountered.555“unanswerable” ,“no” ,“no answe” ,“not enough information”
,“unknown” ,“it is not possible to tell” ,“it does not say” , or “no relevant
information”.

###  3.3. EXAM Evaluation

####  3.3.1. EXAM Cover Evaluation

We follow the suggestion of Sander and Dietz and Eyal et al. to evaluate
systems by the number of questions $Q$ that can be answered with any of its
passages $P$.

(1) $\text{EXAM Cover{}}(P)=\frac{1}{|Q|}\left|\bigcup_{p\in
P}\left\\{q|\text{correct}(q,p),\forall q\in Q\right\\}\right|$

Depending on the grading method chosen in Section 3.2, for a passage $p$, the
question $q$ is considered ”correct” when

  * •

the answer was verified as correct

  * •

the self-rating is above a threshold (e.g. $\geq 1$ or $\geq 4$)

The goal of the exam is to assess the comprehensiveness of the information
presented in the entire ranking. Hence, EXAM Cover monitors the set of
questions that can be accurately answered using any passage in the ranking.
EXAM Cover constitutes a recall-oriented evaluation metric that does not
reward system for providing redundant content. Instead, it encourages to cover
the breadth of relevant information content with the passage set $P$.

####  3.3.2. EXAM Qrels Evaluation: EXAM for trec_eval

Recognizing the benefits of integrating novel evaluation metrics into
established evaluation tool-chains, we propose a variant of EXAM that can be
implemented with trec_eval. We can convert EXAM grades to a relevance “qrels”
file. We mark any passage as relevant whenever it allows answering an exam
question that was generated for a query (or query-subtopic). This yields a
binary relevance label for each passage $p$:

(2) $\text{relevance-label}(p)=\exists q\in Q:\text{correct}(q,p)$

The ranking can be evaluated with trec_eval using this EXAM Qrels file using
any traditional evaluation metric. In the empirical evaluation we use
Precision@k, which reflects the proportion of passages in the top $k$ with
relevant information content.

This approach avoids the problem of unjudged passages (also called “holes”
(MacAvaney and Soldaini, 2023)) in test collections. The EXAM grading pipeline
can be applied to update the qrels file whenever new passages appear in a
system response.

##### Self-rated Answerability.

We can extend the EXAM Qrels idea to multi-graded relevance labels based on
the self-rated confidence $\text{rating}(q,p)$ that question $q$ is answerable
with passage $p$.

For every passage, we identify the highest self-rating that a passage obtained
on any exam question in the question bank $Q$.

(3) $\text{relevance-label}(p)=\max_{q\in Q}\text{rating}(q,p)$

This relevance label is then incorporated into the qrels file as a multi-
relevance scale. Setting a threshold for multi-graded relevance666set with
trec_eval option --level_for_rel considers only those passages as relevant
that meet or exceed the specified self-rating level.

While EXAM Qrels does not discourage redundant information in the ranking,
unlike EXAM Cover, it still demonstrates strong rank correlation with official
leaderboards in our experiments. This is particularly evident when the
official evaluation metrics themselves do not penalize redundant information
in the ranking, as observed in the cases of TREC CAR Y3 and TREC DL.

##  4\. Experimental Evaluation (of the Evaluation Metric)

###  4.1. Experimental Setup

We study our approach in three experiments.

TREC CAR Y3 / TQA::

Replicating the experimental setup of Sander et al by using queries and
systems from TREC CAR Y3 (Dietz and Foley, 2019) which we evaluate with exam
questions from the TQA dataset (Kembhavi et al., 2017).

TREC CAR Y3 / GenQ::

Switching to our prompt-based generated question banks, we demonstrate that
similar results are obtained when systems from TREC CAR Y3 are evaluated with
a new set of generated questions.

TREC DL 2019 & 2020 / GenQ::

To demonstrate generalization to other datasets we use queries and systems
from two years of TREC DL (Craswell et al., 2020, 2021) using a generated
question bank for each test collection.

For all four experiments we use rankings of systems submitted to the
respective TREC track (test data set) along with manually produced official
TREC judgments.

The goals for the evaluation metric are:

Leaderboard rank-correlation::

The leaderboard of systems under the EXAM Cover and EXAM Qrels metric should
be similar to the official leaderboard. This is evaluated with two rank
correlation measures: Spearman’s rank correlation coefficient, measures
differences in the leaderboard rank, and Kendall’s $\tau$ rank correlation
which penalizes swaps of two systems on the leaderboard.

Inter-annotator agreement::

High passage-level agreement between official judgments and our predicted
relevance labels. We provide count statistics and Cohen’s $\kappa$ inter-
annotator agreement which corrects for leniency vs strictness.

For question generation we use gpt-3.5-turbo-instruct; for question
verification and self-rating we use google/flan-t5-large with the text2text-
generation pipeline from Hugging
Face.777https://huggingface.co/google/flan-t5-large We also explored fine-
tuning the FLAN-T5-large on the SQuAD2 dataset, but results were slightly
worse.

We obtain ten questions for each of the 721 query-subtopics in TREC CAR Y3,
and each query in TREC DL. We graded all passages in official judgments and
the top 20 of all run submissions with EXAM. Across 131 queries in TREC CAR Y3
we grade, 85,329 passages. Across 43 queries in TREC DL 2019, we grade 9,260
passages. Across 54 queries in TREC DL 2020, we grade 11,386 passages.

In EXAM Qrels we use trec_eval’s Precision@20 metric for simplicity. For self-
rated approaches, we define a minimum rating of 4 as relevant (“strict” binary
relevance scale) unless otherwise specified.

Baselines. We demonstrate that our approach is outperforming the original EXAM
Answerability Metric (Sander and Dietz, 2021). Since Sander’s work
demonstrated that ROUGE metrics are uncorrelated with leaderboard rankings, we
omit the comparison here.

Although prompt-based relevance judgments differ significantly in spirit from
out approach, we provide several baselines as upper-bound reference for the
TREC DL experiments: Sun (Sun et al., 2023), Fag (Faggioli et al., 2023), Thom
(abridged prompt, grade 1 or better counted as relevant) (Thomas et al.,
2023). Each prompt is used to directly grade passages for relevance with
FLAN-T5-Large. We also explored suggested few-shot prompts but found that all
of them lead to a drastically decreased quality—for example, for Sun a
Spearman correlation of 0.4 instead of 0.96, and Kendall’s $\tau$ of -0.01
instead of 0.24.

Significance testing. We perform a standard-error bar overlap test on all
results and only describe significant improvements in the text. The standard
error in all experiments ranges between 0.01 and 0.015. For brevity they are
omitted here, but are available in the online appendix and illustrated in 2.

###  4.2. Results on TREC CAR Y3 / TQA

We first reproduce the experimental setup of the original EXAM method, using
the TREC CAR Y3 collection and exam question banks from the TQA dataset.

As part of the exam grading, the FLAN-T5-large LLM is prompted to answer each
exam question with every passage from the system response, which is then
verified against the gold standard answer.

The EXAM Cover leaderboard generated using the TQA-based EXAM Cover approach
is very similar to the official CAR-Y3 leaderboard (Table 3). With the
exception of the ECNU_ReRank1 method, the top 10 of the leaderboard is in
exactly the same order. The Spearman rank measure is 0.94, Kendall $\tau$ is
0.84. Our approach outperforms the original EXAM metric significantly,
obtaining a correlation comparable to MAP vs NDCG (cf. Table 1 in (Sander and
Dietz, 2021)). This is visually confirmed in Figure 2. While several methods
obtain similar EXAM scores, these are indeed very similar systems with exactly
the same score on the official leaderboard.

Given the distinct nature of our evaluation paradigm, a high inter-annotator
agreement on the passage-level is not anticipated. Indeed, as shown in Table
4, the per-passage inter-annotator agreement is very low ($\kappa\approx
0.08$). In contrast to the leaderboard correlation, this demonstrates that a
high agreement is not necessary to evaluate system quality correctly.

Table 3. TREC CAR Y3 leaderboards, ordered by EXAM Cover on TQA questions. In terms of both Spearman’s and Kendall’s $\tau$ rank correlation coefficients, all our EXAM approaches obtain a better correlation with the official leaderboard than the original EXAM implementation. Note that some submitted systems were not included on the official leaderboard as they had not been assessed. CAR-Y3 |  TQA EXAM Cover |  GenQ EXAM Cover |  GenQ EXAM Qrels |  official rank |  TQA (Sander and Dietz, 2021) rank
---|---|---|---|---|---
_overall_ | 0.388 | 0.811 | 1 |  |
dangnt-nlp | 0.300 | 0.609 | 0.555 | 1 | 2
ReRnak3_BERT | 0.288 | 0.620 | 0.594 | 2 | 5
ECNU_ReRank1 | 0.285 | 0.572 | 0.512 | 8 | 12
ReRnak2_BERT | 0.281 | 0.623 | 0.593 | 3 | 1
IRIT1 | 0.279 | 0.569 | 0.540 | 5 | 7
IRIT2 | 0.279 | 0.569 | 0.540 | 5 | 4
IRIT3 | 0.279 | 0.569 | 0.540 | 5 | 9
ECNU_BM25 | 0.278 | 0.604 | 0.588 |  |
ECNU_BM25_1 | 0.278 | 0.602 | 0.586 | 8 | 11
ICT-BM25 | 0.278 | 0.598 | 0.576 |  |
Bert-ConvKNRM-50 | 0.277 | 0.605 | 0.571 | 9 | 6
UNH-bm25-rm | 0.276 | 0.545 | 0.535 |  |
bm25-populated | 0.275 | 0.547 | 0.502 | 10 | 8
UNH-qee | 0.271 | 0.614 | 0.568 |  |
UNH-bm25-ecmpsg | 0.267 | 0.522 | 0.487 | 11 | 10
Bert-ConvKNRM | 0.248 | 0.503 | 0.395 |  |
Bert-DRMMTKS | 0.247 | 0.366 | 0.251 | 12 | 3
ICT-DRMMTKS | 0.237 | 0.369 | 0.247 | 16 | 13
UvABottomUp2 | 0.148 | 0.251 | 0.289 | 15 | 16
UvABM25RM3 | 0.147 | 0.260 | 0.300 | 13 | 15
UvABottomUpCh. | 0.145 | 0.255 | 0.283 | 14 | 14
UvABottomUp1 | 0.142 | 0.209 | 0.233 |  |
spearman | 0.937 | 0.869 | 0.865 |  | 0.75
kendall | 0.841 | 0.687 | 0.738 |  | 0.57
Figure 2. Comparing TREC CAR Y3 leaderboards, our EXAM metrics show strong correlation with the official TREC leaderboard. Methods are ordered by their official leaderboard rank (filled circles). Methods without official ranks are placed according to their TQA EXAM Cover scores. Standard error bars given. For context, Sander’s EXAM leaderboard (depicted as crosses) does not show as strong a correlation. Table 4. Inter-annotator agreement on TREC CAR Y3 / TQA question banks using question answering with answer verification. Boxed cells indicate mappings from label to judgment. Largest count per column is depicted in bold.  BINARY | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 1+2+3 | 0 |  |
1 | 553 | 452 | 1005 | 0.078
0 | 2237 | 3110 | 5347 | 0.079
Table 5. Inter-annotator agreement on TREC CAR Y3 / GenQ question banks using
self-ratings. Top: Graded ratings-vs-judgments. Rating 4 is most frequently
associated with a positive grade. Middle: Lenient relevance for ratings $\geq
1$. Bottom: Strict relevance when ratings $\geq 4$.

GRADED | Label | Judgments | Total
---|---|---|---
| 3 | 2 | 1 | 0 |
5 | 11 | 52 | 13 | 20 | 96
4 | 430 | 1121 | 472 | 1425 | 3448
3 | 5 | 33 | 19 | 98 | 155
2 | 17 | 76 | 27 | 217 | 337
1 | 5 | 57 | 23 | 160 | 245
0 | 52 | 238 | 139 | 1642 | 2071

LENIENT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 1+2+3 | 0 |  |
1+2+3+4+5 | 2361 | 1920 | 4281 | 0.3
0 | 429 | 1642 | 2071 | 0.29

STRICT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 1+2+3 | 0 |  |
4+5 | 2099 | 1445 | 3544 | 0.35
0+1+2+3 | 691 | 2117 | 2808 | 0.34

###  4.3. Results on TREC CAR Y3 with GenQ

When switching to generated question banks, which do not have a correct answer
key, we obtain comparable (albeit slightly lower) results.

The leaderboard correlation (Table 3, columns marked with GenQ) still obtain a
very good correlation of Spearman $\approx 0.86$. According to Table 2 (green
lines), both the EXAM Cover and EXAM Qrels methods follow the trend of the
official leaderboard. Interestingly, the system ICT-DRMMTKS denotes a positive
outlier under both exam question banks.

###  4.4. Results on TREC DL with GenQ

Across both years of TREC DL we find that our FLAN-T5-EXAM approach supports
the conclusions discussed above.

We find that the EXAM Qrels method correlates exceptionally well with the
official leaderboard, obtaining spearman rank correlations of 0.93 in TREC DL
2019 and 0.96 on TREC DL 2020, and kendall $\tau$ correlation of 0.8 and 0.84
respectively. This is slightly better than the relevance-labeling prompt Thom
(Thomas et al., 2023) and slightlyworse than the prompt Sun (Sun et al.,
2023).

The EXAM Cover method correlates slightly less. This is likely because the
EXAM Cover focuses on coverage more than precision.

As mentioned before, a high inter-annotator agreement is not necessary for
good leaderboard correlation. Nevertheless, our EXAM approach obtains an
inter-annotator agreement score (Cohen’s $\kappa$) that is in the range of
relevance-prompting methods.

Table 6. Inter-annotator agreement on TREC DL 2019 / GenQ question banks using
self-ratings. Top: Lenient relevance. Bottom: Strict relevance.

LENIENT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 2+3 | 0+1 |  |
1+2+3+4+5 | 1627 | 1809 | 3436 | 0.34
0 | 874 | 4950 | 5824 | 0.34

STRICT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 2+3 | 0+1 |  |
4+5 | 1439 | 1356 | 2795 | 0.36
0+1+2+3 | 1062 | 5403 | 6465 | 0.36

Comparison $\kappa$: Sun 0.39 / Fag 0.44 / Thom 0.08

Table 7. Inter-annotator agreement on TREC DL 2020 / GenQ question banks using
self-ratings. Top: Lenient relevance. Bottom: Strict relevance.

LENIENT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 2+3 | 0+1 |  |
1+2+3+4+5 | 1203 | 3313 | 4516 | 0.22
0 | 463 | 6407 | 6870 | 0.22

STRICT | Label | Judgments | Total | Cohen’s $\boldsymbol{\kappa}$
---|---|---|---|---
| 2+3 | 0+1 |  |
4+5 | 1029 | 2437 | 3466 | 0.25
0+1+2+3 | 637 | 7283 | 7920 | 0.25

Comparison $\kappa$: Sun 0.24 / Fag 0.29 / Thom 0.08

###  4.5. A Worked Example

To illustrate how our FLAN-T5-EXAM method functions, we present a detailed
example from the TREC CAR Y3 dataset for query_id tqa2:L_0384. The passage
presented below was retrieved at rank 1 by the dangnt-nlp method and was
assessed by TREC judges as ’MUST be mentioned’.

> Query title: The Integumentary System
> Query subtopic: Structure of the Skin
> Passage:
> ID: b95bf325b7fdacac183b1daf7c118be407f52a3a
>
> The skin is the largest organ in the human body. Skin is made up of three
> layers, the epidermis, dermis and the fat layer, also called the hypodermis.
> The epidermis is the outer layer of skin that keeps vital fluids in and
> harmful bacteria out of the body. The dermis is the inner layer of skin that
> contains blood vessels, nerves, hair follicles, oil, and sweat glands.
> Severe damage to large areas of skin exposes the human organism to
> dehydration and infections that can result in death.
> TREC judgment: 3 (MUST be mentioned)
>

The TQA question NDQ_007535 “Outer layer of the skin?” was correctly answered
as “epidermis”. Under the self-rating prompt, FLAN-T5 responds that question
can be answered in a mostly relevant way but may have minor gaps (rating 4).

A generated exam questions, “How does the epidermis, dermis, and hypodermis
work together to provide protection, sensation, and regulation for the body?”,
also lead to a rating of 4. While we cannot automatically confirm the
correctness without an answer key, we manually highlight the relevant text
spans in the passage. To incorporate a human-in-the-loop, one would ask
FLAN-T5 to produce an answer for the generated question, to be highlighted in
a verification interface.

Table 8. Leaderboard for TREC DL 2019, using ratings $\geq 3$ as relevant.  DL2019 |  EXAM Cover |  EXAM Qrels |  Official Rank
---|---|---|---
_overall_ | 0.900 | 1 |
idst_bert_p2 | 0.707 | 0.759 | 2
idst_bert_p3 | 0.707 | 0.756 | 3
p_exp_rm3_bert | 0.700 | 0.755 | 4
idst_bert_p1 | 0.702 | 0.752 | 1
idst_bert_pr2 | 0.695 | 0.751 | 6
p_bert | 0.695 | 0.745 | 5
idst_bert_pr1 | 0.688 | 0.745 | 7
p_exp_bert | 0.700 | 0.744 | 8
TUW19-p1-f | 0.716 | 0.741 | 14
ms_duet_passage | 0.684 | 0.740 | 22
runid3 | 0.695 | 0.726 | 12
runid4 | 0.691 | 0.721 | 11
TUW19-p2-f | 0.714 | 0.720 | 17
srchvrs_ps_run2 | 0.712 | 0.713 | 19
TUW19-p3-f | 0.726 | 0.712 | 13
TUW19-p3-re | 0.688 | 0.705 | 15
TUW19-p1-re | 0.686 | 0.704 | 16
TUW19-p2-re | 0.695 | 0.702 | 20
ICT-CKNRM_B50 | 0.693 | 0.688 | 23
runid2 | 0.674 | 0.685 | 29
bm25base_p | 0.677 | 0.679 | 33
srchvrs_ps_run3 | 0.716 | 0.670 | 24
bm25tuned_prf_p | 0.665 | 0.659 | 25
bm25tuned_ax_p | 0.665 | 0.651 | 27
srchvrs_ps_run1 | 0.681 | 0.644 | 34
ICT-CKNRM_B | 0.686 | 0.635 | 21
ICT-BERT2 | 0.686 | 0.635 | 18
test1 | 0.684 | 0.617 | 9
bm25base_ax_p | 0.667 | 0.617 | 26
runid5 | 0.681 | 0.586 | 30
TUA1-1 | 0.684 | 0.585 | 10
bm25tuned_p | 0.663 | 0.584 | 35
bm25base_prf_p | 0.660 | 0.145 | 28
|  |  |  Sun / Fag / Thom
Spearman correlation | 0.72 | 0.93 | 0.96 / 0.96 / 0.86
Kendall correlation | 0.54 | 0.80 | 0.85 / 0.85 / 0.67
Table 9. Leaderboard for TREC DL 2020, using ratings $\geq 4$ as relevant. Some methods omitted to not violate page margins. DL2020 |  EXAM Cover |  EXAM Qrels |  Official Rank
---|---|---|---
_overall_ | 0.911 | 1.000 |
NLE_pr3 | 0.748 | 0.786 | 12
pash_f2 | 0.744 | 0.786 | 5
pash_f3 | 0.741 | 0.786 | 3
pash_f1 | 0.741 | 0.785 | 4
NLE_pr2 | 0.737 | 0.783 | 16
CoRT-electra | 0.731 | 0.781 | 9
pash_r3 | 0.739 | 0.778 | 1
NLE_pr1 | 0.743 | 0.777 | 17
p_d2q_bm25_duo | 0.756 | 0.776 | 6
pinganNLP2 | 0.741 | 0.776 | 13
pinganNLP1 | 0.741 | 0.776 | 15
pash_r2 | 0.739 | 0.776 | 2
RMIT-Bart | 0.737 | 0.776 | 10
pash_r1 | 0.735 | 0.775 | 11
p_d2q_rm3_duo | 0.752 | 0.773 | 7
rr-pass-roberta | 0.741 | 0.769 | 22
pinganNLP3 | 0.741 | 0.769 | 14
bigIR-T5-BERT-F | 0.720 | 0.766 | 26
bcai_bertl_pass | 0.748 | 0.762 | 23
nlm-ens-bst-2 | 0.743 | 0.759 | 28
fr_pass_roberta | 0.733 | 0.758 | 20
bigIR-BERT-R | 0.728 | 0.758 | 19
bigIR-T5-R | 0.750 | 0.755 | 24
1 | 0.728 | 0.755 | 18
p_bm25rm3_duo | 0.739 | 0.754 | 8
nlm-ens-bst-3 | 0.735 | 0.753 | 29
nlm-bert-rr | 0.722 | 0.750 | 30
bigIR-T5xp-T5-F | 0.735 | 0.745 | 27
2 | 0.730 | 0.744 | 25
bigIR-DCT-T5-F | 0.733 | 0.741 | 21
relemb_mlm_0_2 | 0.724 | 0.739 | 31
nlm-prfun-bert | 0.730 | 0.728 | 32
TUW-TK-2Layer | 0.739 | 0.722 | 34
p_d2q_bm25 | 0.713 | 0.699 | 35
p_d2q_bm25rm3 | 0.693 | 0.693 | 36
bert_6 | 0.726 | 0.691 | 37
TUW-TK-Sparse | 0.700 | 0.682 | 33
CoRT-bm25 | 0.694 | 0.664 | 38
indri-fdm | 0.709 | 0.652 | 43
bl_bcai_mdl1_vt | 0.707 | 0.650 | 40
terrier-DPH | 0.689 | 0.650 | 52
indri-lmds | 0.702 | 0.649 | 47
terrier-InL2 | 0.693 | 0.649 | 44
bcai_class_pass | 0.694 | 0.647 | 41
indri-sdm | 0.698 | 0.642 | 48
terrier-BM25 | 0.683 | 0.641 | 45
DLH_d_5_t_25 | 0.650 | 0.641 | 46
$\dots$ | $\dots$ | $\dots$ | $\dots$
DoRA_Small | 0.285 | 0.182 | 57
DoRA_Large | 0.278 | 0.171 | 59
DoRA_Med | 0.270 | 0.171 | 58
|  |  |  Sun / Fag / Thom
Spearman Correlation | 0.88 | 0.96 | 0.96 / 0.95 / 0.95
Kendall Correlation | 0.70 | 0.84 | 0.88 / 0.85 / 0.84

##  5\. conclusion

With FLAN-T5-EXAM we are proposing an alternative evaluation approach that
does not make any use of manual passage-level relevance judgments—and does not
attempt to mimic relevance judgment process. Instead, an exam question bank is
created as part of topic development, envisioning that each question addresses
one important piece of information content. As a result, whenever such
questions are answerable with text from an retrieval generation system, we
conclude that the system provides relevant information.

Using three TREC data sets, we demonstrate that (1) our proposed approach can
reproduce official TREC leaderboards nearly perfectly; and (2) it is a strong
contender in comparison to other recent LLM-based relevance label predictors
(Sun et al., 2023; Faggioli et al., 2023; Thomas et al., 2023). In contrast,
FLAN-T5-EXAM offers a clear path towards integrating a human-in-the-loop, by
supporting the refinement of the exam question banks, which is how relevant
information content is defined.

We believe that further research will improve the question bank generation, as
well as study the positive effects of this approach on the quality, cost, and
satisfaction of human judges.

We hope that by integrating EXAM evaluation metric into trec_eval, we offer a
system that can be easily adopted by future IR evaluation tracks, offering
organizers an avenue to reduce assessment costs, obtain reusable test
collections for generative information systems.

###### Acknowledgements.

This material is based upon work supported by the National Science Foundation
under Grant No. 1846017. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the National Science Foundation.

## References

  * (1)
  * Arabzadeh et al. (2024) Negar Arabzadeh, Amin Bigdeli, and Charles LA Clarke. 2024\.  Adapting Standard Retrieval Benchmarks to Evaluate Generated Answers.  _arXiv preprint arXiv:2401.04842_ (2024).
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023\.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_ (2023).
  * Clarke and Lapata (2010) James Clarke and Mirella Lapata. 2010.  Discourse Constraints for Document Compression.  _Computational Linguistics_ 36, 3 (2010).
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021.  Overview of the TREC 2020 deep learning track.  _arXiv preprint arXiv:2102.07662_ (2021).
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_ (2020).
  * Deutsch et al. (2020) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2020\.  Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary.  _arXiv preprint arXiv:2010.00490_ (2020).
  * Dietz and Foley (2019) Laura Dietz and John Foley. 2019.  TREC CAR Y3: Complex Answer Retrieval Overview. In _Proceedings of Text REtrieval Conference (TREC)_.
  * Eyal et al. (2019) Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.  Question Answering as an Automatic Evaluation Metric for News Article Summarization. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 3938–3948.  https://doi.org/10.18653/v1/N19-1395
  * Faggioli et al. (2023) Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023\.  Perspectives on large language models for relevance judgment. In _Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval_. 39–50.
  * Fok and Weld (2023) Raymond Fok and Daniel S Weld. 2023.  In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making.  _arXiv preprint arXiv:2305.07722_ (2023).
  * Huang et al. (2020) Luyang Huang, Lingfei Wu, and Lu Wang. 2020.  Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ (2020).  https://doi.org/10.18653/v1/2020.acl-main.457
  * Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2017), 5376–5384.
  * Li et al. (2023) Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023\.  LLatrieval: LLM-Verified Retrieval for Verifiable Generation.  _arXiv preprint arXiv:2311.07838_ (2023).
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022\.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ (2022).
  * Lin and Demner-Fushman (2006) Jimmy Lin and Dina Demner-Fushman. 2006.  Will pyramids built of nuggets topple over?. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_. 383–390.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023\.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ (2023).
  * MacAvaney and Soldaini (2023) Sean MacAvaney and Luca Soldaini. 2023.  One-Shot Labeling for Automatic Relevance Estimation.  _arXiv preprint arXiv:2302.11266_ (2023).
  * McCreadie and Buntain (2023) Richard McCreadie and Cody Buntain. 2023.  _CrisisFACTS: Buidling and Evaluating Crisis Timelines_.  Technical Report. Univerity of Glasgow.
  * Mulla and Gharpure (2023) Nikahat Mulla and Prachi Gharpure. 2023.  Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications.  _Progress in Artificial Intelligence_ 12, 1 (2023), 1–32.
  * Pavlu et al. (2012) Virgil Pavlu, Shahzad Rajput, Peter B Golbus, and Javed A Aslam. 2012.  IR system evaluation using nugget-based test collections. In _Proceedings of the fifth ACM international conference on Web search and data mining_. 393–402.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022\.  Improving passage retrieval with zero-shot question generation.  _arXiv preprint arXiv:2204.07496_ (2022).
  * Sander and Dietz (2021) David P Sander and Laura Dietz. 2021.  EXAM: How to Evaluate Retrieve-and-Generate Systems for Users Who Do Not (Yet) Know What They Want.. In _DESIRES_. 136–146.
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023\.  Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent.  _arXiv e-prints_ (2023), arXiv–2304.
  * Thomas et al. (2023) Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023.  Large language models can accurately predict searcher preferences.  arXiv:2309.10621 [cs.IR]
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 5008–5020.  https://doi.org/10.18653/v1/2020.acl-main.450
  * Wang et al. (2023) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_ (2023).
  * Weng et al. (2023) Yixuan Weng, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, Jun Zhao, et al. 2023\.  Large Language Models are Better Reasoners with Self-Verification. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Zhang and Gao (2023) Xuan Zhang and Wei Gao. 2023.  Towards llm-based fact verification on news claims with a hierarchical step-by-step prompting method.  _arXiv preprint arXiv:2310.00305_ (2023).

  *[LLMs]: Large Language Model