c

# DORIS-MAE:
Scientific Document Retrieval using Multi-level Aspect-based Queries

Jianyou Wang &Kaicheng Wang 11footnotemark: 1 &Xiaoyue Wang &Prudhviraj Naidu
&&Leon Bergen &Ramamohan Paturi 22footnotemark: 2 &Laboratory for Emerging
Intelligence
University of California, San Diego
La Jolla, CA 92093
{jiw101, kaw036, xiw027, prnaidu, lbergen, rpaturi} @ ucsd.edu Equal
ContributionEqual Contribution

###### Abstract

In scientific research, the ability to effectively retrieve relevant documents
based on complex, multifaceted queries is critical. Existing evaluation
datasets for this task are limited, primarily due to the high cost and effort
required to annotate resources that effectively represent complex queries. To
address this, we propose a novel task, Scientific DOcument Retrieval using
Multi-level Aspect-based quEries (DORIS-MAE), which is designed to handle the
complex nature of user queries in scientific research. We developed a
benchmark dataset within the field of computer science, consisting of 100
human-authored complex query cases. For each complex query, we assembled a
collection of 100 relevant documents and produced annotated relevance scores
for ranking them. Recognizing the significant labor of expert annotation, we
also introduce Anno-GPT, a scalable framework for validating the performance
of Large Language Models (LLMs) on expert-level dataset annotation tasks. LLM
annotation of the DORIS-MAE dataset resulted in a 500x reduction in cost,
without compromising quality. Furthermore, due to the multi-tiered structure
of these complex queries, the DORIS-MAE dataset can be extended to over 4,000
sub-query test cases without requiring additional annotation. We evaluated 17
recent retrieval methods on DORIS-MAE, observing notable performance drops
compared to traditional datasets. This highlights the need for better
approaches to handle complex, multifaceted queries in scientific research. Our
dataset and codebase are available at https://github.com/Real-Doris-Mae/Doris-
Mae-Dataset.

##  1 Introduction

Scientists often have complex questions that require thorough exploration
within various parts of their field (Figure 1). Finding relevant scientific
literature, one of many challenges in this process, can be especially
difficult when dealing with multi-faceted queries. These queries typically
encompass numerous interconnected topics and require an information retrieval
(IR) system capable of recognizing and responding to this level of complexity.

Information retrieval, especially query-based document retrieval [78, 3, 34],
is integral to many applications, from search engines [9, 13, 29] and content
recommendations [58, 31, 41], to open-domain question answering (QA) [82, 33,
15]. A persistent challenge, however, is the low accuracy in processing
complex and multi-intent user queries. Despite advanced search engines using
semantic understanding and user behavior data in addition to keyword matching
[13], these systems still fall short when dealing with complex, multi-intent
queries. Neural Information Retrieval (NIR) models [63, 85, 75, 23, 66, 85,
47] are primarily trained on relatively simple queries [57, 17, 64, 61, 87].
Moreover, MacAvaney et al. [49] creates a variety of diagnostic probes,
revealing NIR models’ instability when processing out-of-distribution textual
inputs, potentially because they do not comprehend the deeper semantics of
text. These limitations can lead to inadequate performance with more complex
queries. While open-domain QA models like ChatGPT [11] have shown significant
capability, they frequently produce incorrect or even fabricated responses [4,
46, 50, 39, 89], and are prohibitively expensive if directly applied to an
entire corpus of scientific documents.

Figure 1: Example from the DORIS-MAE dataset. Each query is broken down into
aspects and sub-aspects. Aspects are semantically distinct components of the
query, and sub-aspects are minimal requirements that can be extracted from the
aspects.

To address these challenges, we introduce a novel task, Scientific Document
Retrieval for Multi-level Aspect-based quEries (DORIS-MAE). DORIS-MAE extends
query-based and example-based IR paradigms [20, 44], aiming to give users more
control in formulating queries using natural language. In order to advance
research in this area, we present the DORIS-MAE dataset, comprising 100 unique
complex queries in the computer science domain, paired with ranked pools of
relevant CS article abstracts. Each query is organized into a hierarchical
structure of aspects/sub-aspects, which aids annotation.

A distinguishing feature of the DORIS-MAE dataset is its aspect-based
hierarchical structure shown in Figure 1. This feature aids automation of the
annotation process, expands the test case volume without additional
annotation, and opens up exploration into the usefulness of aspect information
for retrieval methods.

Furthermore, we introduce Anno-GPT, a pipeline for validating Large Language
Model (LLM) annotations in a statistically sound manner. Our tests reveal that
LLM annotations achieve quality comparable to those produced by human
annotators, but with considerable savings in both time and cost. Additionally,
the design of our pipeline lends itself to easy adaptation for different
domains.

In our experiments, we evaluated 17 IR/NIR (Information Retrieval/Neural
Information Retrieval) methods using the DORIS-MAE dataset. The methods have
worse performance on DORIS-MAE compared to traditional document retrieval
datasets, highlighting the complexity of DORIS-MAE, and the need for more
sophisticated retrieval methods.

Our main contributions are three-fold. First, by formulating the DORIS-MAE
task we are shedding a new light on tackling complex, multi-faceted queries
during scientific research. Second, we propose Anno-GPT, a procedure for
rigorously evaluating the ability of LLMs to replace human experts for
challenging annotation tasks. Third, we demonstrate the value of breaking
complex queries down to an Multi-level Aspect-based hierarchical structure,
for both annotation accuracy and potential improvements in retrieval methods.

##  2 Related Work

A range of methods have been developed for document retrieval and re-ranking.
Classic retrieval methods like TF-IDF [67] and BM25 [72] utilize keyword
matching between queries and documents, and fall short when the necessary key
phrases are not known to users. To address this, researchers have applied deep
learning techniques to develop NIR models, including RocketQA-v2 [63],
ColBERT-v2 [66], SimLM [75], and SPLADE-v2 [23]. These models generate latent
vector representations for queries and documents, which are effective for many
retrieval tasks, but may not be able to handle textual inputs that are outside
of their training distributions [49].

Other models such as SPECTER [18] and aspect-based ASPIRE [54] focus on
calculating document-level similarity. These models are designed to retrieve
given an existing paper as input, which makes them less suited for open-ended
queries. Other retrieval models, such as text-embedding models (ada-002 by
OpenAI [28] and E5-large-v2 [74]) and sentence-embedding model Sentence-BERT
[62], all face similar limitations as NIR methods in that they struggle to
simultaneously represent multiple aspects of a query. Despite these
challenges, they tend to perform better on our dataset than specialized NIR
models for efficient dense passage retrieval, indicating their potential for
complex tasks like DORIS-MAE.

Large-scale traditional IR datasets [57, 86, 40, 66, 69] primarily contain
simple, web-based queries. Models performing well on these datasets often
struggle on DORIS-MAE. In contrast, more specialized IR datasets [56, 14, 53]
require human expert annotation, and consequently are more limited in their
scale. Mysore et al. [53] and Chan et al. [14] introduce the concept of
"aspect" in document retrieval tasks. These datasets provide pre-defined
categories for aspects such as "background" or "method". DORIS-MAE extends
this work by allowing for open-ended aspects based on the context of the
queries. In Narrative-Driven Recommendation (NDR) research [6, 2, 55], user
queries are descriptions that capture a range of users’ needs. These queries
are related to everyday tasks, such as finding restaurants and entertainment.

Progress in LLMs [11] and prompt-engineering [24, 45, 59, 79, 65] has made it
feasible to leverage LLMs such as ChatGPT for annotating NLP tasks [90, 21],
even outperforming crowd-workers in some cases [60, 70, 27]. However, these
tasks do not necessitate domain-specific knowledge, and the annotations
produced may not measure up to expert annotations. Faggioli et al. [22],
MacAvaney and Soldaini [48] explored the notion of using LLMs to assist human
in relevance judgements, and evaluated on TREC-DL datasets [19], which have
single-faceted queries. Our work seeks to extend these efforts to new
annotation tasks requiring domain expertise, while introducing a separate
pipeline stage and a hypothesis testing stage.

##  3 Dataset Description

The DORIS-MAE task uses a dataset of 100 complex, human-written queries, each
containing between 95 to 226 words. Each complex query is broken down into a
hierarchy of aspects and sub-aspects, with aspects representing significant
semantic components of the original query, typically a sentence or a few
sentences. Sub-aspects further decompose an aspect into simpler, verifiable
requirements. Both aspects and sub-aspects are generally one sentence long,
though their semantic complexity varies. A complex query can have up to 9
aspects, and each aspect can further contain up to 6 sub-aspects. Figure 1
shows an example.

For each complex query $Q$, we created a pool $\mathcal{P}$ of approximately
100 potentially relevant scientific abstracts to evaluate the re-ranking
performance of various retrieval/reranking methods. Within the context of $Q$
and its $\mathcal{P}$, the collection of aspects and sub-aspects is together
denoted as (with slight abuse of notation) $Q:=\\{a_{i}\\}$. For any aspect or
sub-aspect $a_{i}\in Q$, and any paper abstract $p_{j}\in\mathcal{P}$, they
form a question pair $(a_{i},p_{j})$. We generated a total of 165,144 question
pairs from the 100 queries, together denoted as
$\mathcal{D}:=\\{(a_{i},p_{j})\\}$.

To compute the relevance of a paper $p_{j}$ for a query $Q$, we first break
the query down into its aspects. We then compute the relevance score
$S(a_{i},p_{j})$, which measures the relevance of paper $p_{j}$ for aspect
$a_{i}$. The relevance score $S(p_{j}|Q)$ for the query $Q$ is the sum of its
aspect relevance scores. An average score $\frac{S(p_{j}|Q)}{|Q|}\geq 1$
indicates $p_{j}$ is a relevant abstract for $Q$.

$S(p_{j}|Q):=\sum_{a_{i}\in Q}S(a_{i},p_{j})$ (1)

Thus, for each complex query and candidate pool, we provide a complete ranking
for the list of abstracts. Additionally, we can choose a combination of
aspects within a complex query $Q$ and concatenate the corresponding sentences
to form a sub-query $q\subset Q$, and can calculate the relevance score
between a paper abstract $p_{j}$ and a sub-query $q$.

$S(p_{j}|q):=\sum_{a_{i}\in q}S(a_{i},p_{j})$ (2)

Hence, for any sub-query $q\subset Q$ and the candidate pool $\mathcal{P}$ of
$Q$, we can also provide a complete ground-truth ranking order for
$\mathcal{P}$. This allows the dataset to extend to over 4000 sub-query test
cases at no additional annotation cost.

Candidate pool abstracts are taken from a database of 360,000 computer science
papers between 2011-2021 sourced from arXiv111https://arxiv.org/. We
complemented each arXiv paper with its corresponding citation information by
cross-matching it on Semantic Scholar [37].

###  3.1 Query Formation

The DORIS-MAE task aims to mirror real-world scenarios where a researcher has
an incomplete concept for a research project and needs to explore the breadth
of existing research to establish a solid starting point. The 100 complex
queries in our dataset simulate this scenario.

Each query is based on one or more existing research papers. We randomly
selected 140 papers from AI, NLP, ML, and CV categories on arXiv. We examined
each selected paper’s motivation, background, related work, methodology, and
experimental results. Using this information, we reverse-engineered a complex
query designed to reflect the early thought process of the paper’s authors.
Overall, DORIS-MAE contains 80 queries derived from single paper abstracts,
with 20 queries from each of AI, NLP, ML and CV. We created an additional 20
composite queries, each integrating ideas from 2-3 abstracts. These composite
queries are designed to simulate more interdisciplinary and unexplored
research ideas than the other 80 queries.

###  3.2 Decomposing Queries to Aspects

In this section, we discuss our process of decomposing queries into a list of
aspects and further breaking down aspects into sub-aspects. The guiding
principles for determining aspects are as follows:

  1. (i)

Each aspect must correspond to a prominent and semantically meaningful
component in the query (refer to Figure 1 for this correspondence).

  2. (ii)

Each aspect should contain sufficient context to make sense independently,
eliminating potential ambiguities.

  3. (iii)

Each aspect must be semantically distinct and unrelated to others, ensuring
their contents are disjoint and do not overlap.

Some aspects may fall into broader facets such as background, method, or
result as defined in [53]. Regarding decomposing aspects to sub-aspects, our
criteria are:

  1. (i)

Each sub-aspect should not contain more information than its parent aspect.

  2. (ii)

Each sub-aspect should represent a semantic segment of its parent aspect

  3. (iii)

Different sub-aspects may overlap semantically but each should pose a distinct
question.

  4. (iv)

Each sub-aspect should be as simple as possible.

Given the inherent difficulty in recalling the full details of a query while
ranking a large candidate pool, the aspect-based hierarchical structure of our
complex queries provides a systematic, efficient, and interpretable approach
for annotation. This leads to a more precise ranking of the candidate pool. As
each aspect corresponds to a specific part of the original query $Q$,
combining several aspects is equivalent to concatenating their corresponding
parts in the query, forming semantically coherent sub-queries $q\subset Q$.

###  3.3 Candidate Pooling

To create a candidate pool $\mathcal{P}$ of approximately 100 paper abstracts
for each query $Q$, we used a variety of IR and NIR methods, similar to the
approach in [53]. More specifically, we utilized popular IR search algorithms
TF-IDF [67] and BM25 [72] at different granularities (i.e., sentence and
paragraph-level for each query) to retrieve around 80 paper abstracts. We then
employed OpenAI’s text embedding model, ada-002 [28], to extract 20 more paper
abstracts. We also use citation signals from semantic scholar [37] to included
any papers that directly cite or are cited by any of the original papers used
to create that specific query. Lastly, to prevent bias towards lexicon-
matching retrieval methods, we excluded from $\mathcal{P}$ any papers that
authors referenced during query $Q$’s formulation. See Appendix C.6 for a
sensitivity analysis of our candidate pool construction procedure.

##  4 Anno-GPT Framework

We propose Anno-GPT, a framework for developing a statistically sound
annotation pipeline. We use chatgpt-3.5-turbo-0301 to annotate all 165,144
question pairs $\mathcal{D}:=\\{(a_{i},p_{j})\\}$ where $a_{i}$ is an
aspect/sub-aspect of a query, and $p_{j}$ is an abstract in the query’s
candidate pool. This strategy minimizes human annotation efforts. The key to
this approach lies in breaking down complex queries into simpler aspects and
sub-aspects, ensuring an objective and manageable annotation task. Without
this structured approach, we found that evaluating an abstract’s relevance to
a complex query was challenging, due to variability in how partial relevance
was assessed. However, the scenario changes significantly when dealing with
question pairs $(a_{i},p_{j})$, which only look at a single aspect or sub-
aspect of the query. In such cases, assigning a coarse-scale relevance score
between 0-2 becomes feasible, maintaining a reasonable degree of objectivity.

The performance of the annotation pipeline may be influenced by several
factors: the procedure for breaking down queries into aspects, the criteria
used for scoring query relevance, and the LLM prompt selection [24, 45, 59,
79, 65]. In order to avoid overfitting in this pipeline, our methodology
comprises two distinct development and testing stages. The development stage
involves optimizing all stages of the pipeline, and using feedback and
observed outcomes to iteratively refine this strategy. The testing stage uses
a prespecified hypothesis test. The null hypothesis is that there is no
significant difference between the agreement levels of ChatGPT and humans, and
those among humans themselves. After the pipeline has been optimized, we
evaluate it on a test set. We compute the difference between ChatGPT-human
agreement and human-human agreement. If this difference is sufficiently close
to 0, with a small confidence interval, we can use the optimized prompt $\pi$
to annotate the full dataset.

Below we give a summary of the Anno-GPT framework:

  1. i

Construct aspect-document question pairs, $\mathcal{D}:=\\{(a_{i},p_{j})\\}$.

  2. ii

Select development set $S\textsubscript{dev}\subset\mathcal{D}$, and use human
annotators to score $S\textsubscript{dev}$.

  3. iii

Select test set $S\textsubscript{test}\subset\mathcal{D}\setminus
S\textsubscript{dev}$ based on desired power, and use human annotators to
score $S\textsubscript{test}$.

  4. iv

Optimize prompting strategy $\pi$ and other hyperparameters on
$S\textsubscript{dev}$. Fix $\pi$.

  5. v

Evaluate fixed $\pi$ on $S\textsubscript{test}$.

  6. vi

If satisfactory performance is achieved on $S\textsubscript{test}$, proceed to
use $\pi$ to annotate the entire dataset $\mathcal{D}$.

  7. vii

Otherwise, repeat steps iii, iv, v for $S\textsubscript{dev}\leftarrow
S\textsubscript{dev}\cup S\textsubscript{test}$ and new
$S\textsubscript{test}$.

It is important to note that automated annotation for DORIS-MAE only used a
single development/testing cycle, and therefore did not go into step vii.
Therefore, there was no risk of inflated estimates of annotation accuracy due
to multiple comparisons [30, 83, 1]. If multiple cycles are necessary, then
the new $S\textsubscript{test}$ must be sufficiently large to avoid these
problems.

###  4.1 Annotation Guidelines

Our team of annotators consists of three graduate students in computer
science, all with at least two years of research experience in NLP, CV, ML,
and AI. Annotators, both human and ChatGPT, are asked to score each question
pair using a 3-point grading scale (0-2):

  * •

Level 0: The abstract is unrelated or does not provide any help to the key
components of the aspect or sub-aspect.

  * •

Level 1: The abstract answers some (or all) key components (either explicitly
or within one natural inferential step), but at least one key component is not
answered explicitly.

  * •

Level 2: The abstract directly answers all the key components of the aspect or
sub-aspect.

We decided to include both direct and indirect coverage under Level 1,
acknowledging that distinguishing between these cases could be challenging and
potentially subjective. The detailed guidelines for human annotation can be
found in Appendix B.

###  4.2 Optimization of Annotation Pipeline

For the development stage, two annotators independently annotated a randomly
selected set $S\textsubscript{dev}$ of 90 question pairs from the complete set
$\mathcal{D}$ of 165,144 pairs. This annotated development set then served as
the basis for refining the prompting strategies for ChatGPT.

The quality of annotations was evaluated using three metrics: macro F1 score,
exact accuracy (agreement), and Spearman’s rank correlation coefficient
(Spearman’s $\rho$). These metrics measure the agreement level between
annotators, and have been used successfully in similar tasks [53]. After
satisfactory agreement levels were achieved between ChatGPT and human
annotations on the development set, we transitioned to the hypothesis testing
stage, with all three annotators involved. In this stage, we employed the
fixed finalized prompting strategy, ensuring no overfitting or leakage from
the test set.

Our prompting strategy development involved experimenting with recent
methodologies such as few-shot in-context learning (ICL) [16, 11, 80, 51, 91]
and chain of thought (CoT) [38, 81, 77, 32]. We found that the CoT approach
offered the most robust and optimal results for the task of annotating
question pairs $(a_{i},d_{j})\in\mathcal{D}$. A comprehensive description of
the prompt engineering process can be found in Appendix B.

###  4.3 Annotation Evaluations

Hypothesis testing was conducted using a sample of 250 question pairs
$S\textsubscript{test}$ from $\mathcal{D}$, distinct from the development set.
The selected pairs were independently annotated by three human annotators. We
used bootstrapping to estimate the 95% confidence intervals for the macro F1
score, accuracy (agreement), and Spearman’s $\rho$. Though the sampling
temperature is fixed at zero, recognizing small randomness introduced by GPU
non-determinism [52], the ChatGPT annotations are run twice and the pairwise
comparisons with human annotators are averaged across these runs.

In addition to these prespecified analyses, we conducted post-hoc analyses
using an adjudication procedure to create a more stable set of human
annotations [53]. We use majority voting [8] between the three human
annotators to decide the adjudicated annotation.

The results presented in Table 1 show that the rate of ChatGPT-human agreement
is within range of human-human agreement. Specifically, ChatGPT’s performance
is comparable to that of human annotators as measured by F1 and exact
agreement (accuracy). The average agreement level for ChatGPT is numerically
lower than average human agreement level as measured by Spearman’s $\rho$. We
note that the lowest Spearman’s $\rho$ among two humans is $46.51\%$, which is
comparable to the average ChatGPT/human’s $\rho=46.61\%$, suggesting ChatGPT’s
performance is still within the range of human-level agreement. All $p$-values
are larger than the $\alpha=0.05$ criterion.

In a post-hoc analysis, we found that comparing ChatGPT to adjudicated human
annotations numerically increased the rate of agreement. This provides
qualitative evidence for ChatGPT’s performance relative to that of human
experts. We further analyzed instances where ChatGPT’s annotations diverged
from those of humans. Interestingly, the nature of these discrepancies was
similar to those found between humans, with differences largely revolving
around the interpretation of key components in aspect/sub-aspects. For
example, ChatGPT occasionally differed from human annotators in determining
the importance of a given component. Detailed examples of ChatGPT’s reasoning
and a comprehensive error analysis can be found in Appendix B.

Table 1: Annotation agreement between humans and ChatGPT. H is human, G is
ChatGPT, A is Adjudication, CI is 95% confidence interval. $p$-values
correspond to the null hypothesis that there is no difference between avg. H&H
and avg. H&G. Higher $p$-values indicate less evidence of a difference between
ChatGPT and humans.
Metrics | G&A | avg. H&H | avg. G&H | H&H CI | G&H CI |  $p$-value
---|---|---|---|---|---|---
F1-score (macro) | 64.17 | 58.33 | 57.46 | (52.33, 63.46) | (50.93, 62.79) | 0.74
Accuracy | 67.40 | 64.13 | 62.07 | (59.73, 68.80) | (57.67, 66.13) | 0.41
Spearman’s $\rho$ | 52.63 | 54.31 | 46.61 | (46.87, 61.56) | (38.67, 54.41) | 0.07

###  4.4 Scalability of Annotations

The hypothesis testing results support the use of ChatGPT for annotation. At
deployment, the pipeline annotated all 165,144 aspect-paper pairs within a
span of 24 hours, at a cost under $150. By contrast, human experts typically
require approximately 4 minutes per question pair, resulting in an estimated
11,146 hours to annotate the entire dataset. The deployment resulted in a time
reduction by a factor of 500, and a cost reduction by a factor of 1,000,
without sacrificing annotation quality. Upon completion of the annotation
process, we utilized Equations 1 and 2 to compile the results and compute the
final rankings for both full-query and sub-query test cases. Anno-GPT could
potentially utilize any high-performance LLM to replace ChatGPT and can be
adapted for other expert-level tasks, given the availability of a small set of
domain expert annotations for validation.

##  5 Retrieval Results

This section presents the results of testing 17 models discussed in Section 2
on the DORIS-MAE dataset. When available, we trained the model on our CS
corpus and denoted the best version as trained in domain (ID), see full
training details in Appendix D.1. To contextualize their performance on DORIS-
MAE, we compare the results with these models’ previously reported
performances on various IR datasets, including MS MARCO [57], LoTTE [66], NQ
[40], and Wiki-QA [86].

###  5.1 DORIS-MAE Benchmarking Results

In our benchmarking process for DORIS-MAE, we use complex queries as inputs to
these models. We employ a variety of metrics common in the IR/NIR literature
for the evaluation, including R@5, R@20, R-Precision (RP),
$\textrm{NDCG}^{\textrm{exp}}_{10\%}$, MRR@10, and MAP. For fairness, we
adopted an alternative approach for models like RocketQA-v2 [63] and
ColBERT-v2 [66] that were not designed to handle long queries. For these
cases, we allow the models to process the input as either a single text string
or sentence-by-sentence, and report the maximum performance achieved. Uniquely
among the models that we consider, the ASPIRE models (TSApire/OTAspire) are
designed to handle multi-aspect queries. For brevity, we only report the
higher number among these two options for models in Table 2. For more detailed
results, refer to Appendix C.

Table 2: Query level performance on full DORIS-MAE. Standard errors are estimated by bootstrapping. ID means a model is trained in domain. Method | R@5 | R@20 | RP | $\textrm{NDCG}^{\textrm{exp}}_{{10\%}}$ | MRR@10 | MAP
---|---|---|---|---|---|---
Random | 4.41 | 18.48 | 16.29 | 7.31 | 3.59 | 19.63
E5-L-v2[74] |  16.51 $\pm$ 2.05 | 43.77 $\pm$ 2.14 |  37.46 $\pm$ 2.44 | 25.90 $\pm$ 2.15 | 14.85 $\pm$ 2.73 |  40.49 $\pm$ 2.32
RocketQA-v2[63] | 15.63 $\pm$ 1.88 |  45.41 $\pm$ 2.43 | 34.36 $\pm$ 2.32 |  30.30 $\pm$ 2.26 |  20.87 $\pm$ 3.12 | 40.18 $\pm$ 2.23
ada-002[28] | 15.38 $\pm$ 1.95 | 42.84 $\pm$ 2.53 | 35.81 $\pm$ 2.67 | 27.46 $\pm$ 2.48 | 19.88 $\pm$ 3.21 | 40.37 $\pm$ 2.55
SimCSE[25] | 14.90 $\pm$ 1.89 | 42.62 $\pm$ 2.40 | 35.27 $\pm$ 2.34 | 26.88 $\pm$ 2.36 | 21.19 $\pm$ 3.47 | 39.02 $\pm$ 2.35
SPLADE-v2[25] | 14.78 $\pm$ 1.89 | 40.14 $\pm$ 2.33 | 31.65 $\pm$ 2.38 | 26.08 $\pm$ 2.00 | 17.82 $\pm$ 2.99 | 37.23 $\pm$ 2.26
SPECTER-v2[18] | 14.50 $\pm$ 2.15 | 43.36 $\pm$ 2.50 | 33.41 $\pm$ 2.33 | 25.65 $\pm$ 2.23 | 17.19 $\pm$ 2.96 | 37.12 $\pm$ 2.10
SPECTERID | 13.32 $\pm$ 1.76 | 42.52 $\pm$ 2.37 | 31.55 $\pm$ 2.28 | 21.27 $\pm$ 2.03 | 14.48$\pm$ 2.78 | 36.02 $\pm$ 2.19
TSAspire[54] | 14.26 $\pm$ 1.80 | 41.25 $\pm$ 2.40 | 33.81 $\pm$ 2.47 | 26.63 $\pm$ 2.05 | 15.59 $\pm$ 2.59 | 37.00 $\pm$ 2.29
SentBERT[62] | 14.09 $\pm$ 1.88 | 44.69 $\pm$ 2.47 | 33.79 $\pm$ 2.41 | 21.88 $\pm$ 2.07 | 13.23 $\pm$ 2.69 | 37.75 $\pm$ 2.28
OTAspire[54] | 13.34 $\pm$ 1.56 | 42.25 $\pm$ 2.53 | 33.63 $\pm$ 2.38 | 25.52 $\pm$ 2.29 | 14.18 $\pm$ 2.66 | 36.70 $\pm$ 2.22
ANCEFirstP[85] | 13.21 $\pm$ 2.02 | 34.54 $\pm$ 2.20 | 30.51 $\pm$ 2.50 | 20.30 $\pm$ 2.02 | 13.87 $\pm$ 2.64 | 34.53 $\pm$ 2.35
SPLADE-v2[23] | 11.80 $\pm$ 1.86 | 36.59 $\pm$ 2.12 | 29.90 $\pm$ 2.20 | 21.35 $\pm$ 2.12 | 14.30 $\pm$ 2.77 | 33.98 $\pm$ 2.23
LLAMA[71] | 12.74 $\pm$ 1.82 | 34.51 $\pm$ 2.36 | 28.33 $\pm$ 2.14 | 16.65 $\pm$ 1.68 | 11.78 $\pm$ 2.45 | 31.29 $\pm$ 1.99
SimLM[75] | 12.68 $\pm$ 1.77 | 35.67 $\pm$ 2.49 | 28.90 $\pm$ 2.42 | 18.91 $\pm$ 1.86 | 11.29 $\pm$ 2.44 | 33.06 $\pm$ 2.34
BM25[72] | 8.47 $\pm$ 1.80 | 30.50 $\pm$ 2.38 | 21.94 $\pm$ 2.03 | 13.23 $\pm$ 1.97 | 9.19 $\pm$ 2.46 | 25.99 $\pm$ 1.68
ColBERT-v2[66] | 8.45 $\pm$ 1.46 | 27.86 $\pm$ 2.29 | 22.33 $\pm$ 2.01 | 12.57 $\pm$ 1.71 | 6.69 $\pm$ 2.15 | 25.80 $\pm$ 1.83
TF-IDF[67] | 10.71 $\pm$ 1.48 | 29.22 $\pm$ 2.25 | 24.79 $\pm$ 2.06 | 18.25 $\pm$ 2.01 | 12.41 $\pm$ 2.53 | 28.77 $\pm$ 1.81
ERNIE[47] | 6.49 $\pm$ 0.94 | 22.58 $\pm$ 1.72 | 20.18 $\pm$ 1.82 | 9.66 $\pm$ 1.18 | 3.77 $\pm$ 1.06 | 22.71 $\pm$ 1.65
SciBERT[5] | 5.13 $\pm$ 1.25 | 17.99 $\pm$ 1.69 | 17.13 $\pm$ 1.88 | 7.50 $\pm$ 1.34 | 3.41 $\pm$ 1.57 | 20.34 $\pm$ 1.64

To better interpret the results, we compare against a random ranking baseline.
In general, the models show consistent behavior, with larger and more general-
purpose models (like E5-Large-V2, RocketQA-v2, ada-002, Specter-v2) faring
better than the smaller and more specialized ones (like SciBERT [5],
ColBERT-v2, BM25, TF-IDF). Though the Aspire models were designed for multi-
aspect queries, they do not have strong performance on the complex queries in
DORIS-MAE.

When we compare the DORIS-MAE performance of these models with their reported
results on traditional retrieval datasets MS MARCO (in Table 4) and NQ (in
Table 4), we observe a significant reduction in their performances on DORIS-
MAE. We choose metrics for comparison based on what is available in previously
published work. The results highlight the challenges posed by DORIS-MAE and
suggest gaps in the ability of existing methods to generalize well to complex
query retrieval.

Finally, in Table 5, we make a comparison with the model performances on
specialized retrieval datasets such as CSFCube [53], RELISH [10], and
TRECCOVID [73]. The comparison reveals a consistent level of difficulty
between DORIS-MAE and these completely human-annotated datasets, indicating
that DORIS-MAE presents a similarly challenging retrieval task.

Table 3: Comparison with MS MARCO. Stats collected from [85, 66, 63, 75, 23].
| MS MARCO | DORIS
---|---|---
Ranking Method | MRR@10 | MRR@10
ANCE | 33.0 | 13.87
ColBERT-v2 | 39.7 | 6.69
RocketQA-v2 | 41.9 | 20.87
SimLM | 41.1 | 11.29
SPLADE-v2 | 36.8 | 14.30

Table 4: Comparison with NQ. Stats collected from [66].
| NQ | DORIS
---|---|---
Ranking Method | R@20 | R@20
ANCEFirstP | 81.9 | 34.54
BM25 | 59.1 | 30.50
RocketQA-v2 | 83.7 | 45.41
SimLM | 85.2 | 35.67

Table 5: Comparison with CSFCube, TRECCOVID, and RELISH. Results are from [53,
54].
| CSFCube | TRECCOVID | RELISH | DORIS-MAE
---|---|---|---|---
Ranking Method | RP | MAP | R@20 | MAP | MAP | RP | MAP | R@20
TSAspire | - | 40.26 | - | 26.24 | 61.29 | 33.81 | 37.00 | 41.25
OTAspire | - | 40.79 | - | 30.92 | 62.57 | 33.63 | 36.70 | 42.25
Specter-v2 | 18.32 | - | 52.12 | 28.24 | 60.62 | 33.41 | 37.12 | 43.36
TF-IDF | 14.59 | - | 39.69 | - | - | 24.79 | 28.77 | 29.22
BM25 | 13.50 | - | 42.73 | - | - | 21.94 | 25.99 | 30.50

###  5.2 Additional Experiments

Up until now, the hierarchical aspect-based structure that Anno-GPT utilizes
has been hidden from all the evaluated models because of inability of existing
methods to break down a complex query automatically. Even though these
structures are not readily available for real-life retrieval methods, they may
still hold value once this query decomposition process can be fully automated.
To explore this potential, we conduct an experiment where instead of using the
original query, each retrieval method had access to a concatenated string of
all aspects within a query, excluding sub-aspects. The results, as illustrated
in Table 6, show that Sentence-BERT performs best on four metrics: R@5/20, RP,
and MAP. These findings suggest that the use of aspect information could be
potentially beneficial to guide retrieval methods.

The hierarchical structure of our dataset can be used to create additional,
less complex tasks involving only parts of the query. For instance, by pulling
out parts of the query corresponding to 2 Aspects, we are able to generate
over 1000 test cases. For this task, we found a significant increase in the
number of relevant abstracts. After re-evaluating all models for the sub-query
DORIS-MAE, we observed model performance consistent with those seen in
previous benchmarks, as indicated in Table 7. Comparing Table 7 with Table 2,
we observed noticeably higher numbers for metrics such as RP,
$\textrm{NDCG}^{\textrm{exp}}_{{10\%}}$, MRR and MAP, which are indicators of
better model performances on this sub-query task. Since models remain
unchanged, this suggest the sub-query task is less challenging than full-query
task. This is intuitive since sub-queries are less complex and contain fewer
aspects. Overall, the creation of these sub-query tasks underscores the
adaptability of our dataset, which could accommodate a range of task
complexities under the setting of DORIS-MAE.

Table 6: Ranking performance given model access to aspects. Full table is in Appendix C Method | R@5 | R@20 | RP | $\textrm{NDCG}^{\textrm{exp}}_{{10\%}}$ | MRR@10 | MAP
---|---|---|---|---|---|---
ada-002 | 14.09 | 42.23 | 33.56 | 26.54 | 20.20 | 37.62
SentBERT | 17.73 | 45.34 | 35.67 | 25.00 | 15.52 | 39.87
RocketQA-v2 | 13.83 | 43.81 | 32.59 | 27.45 | 16.08 | 37.90
Table 7: Ranking performance on sub-query (2 Aspects) task. Full table is in Appendix C Method | R@5 | R@20 | RP | $\textrm{NDCG}^{\textrm{exp}}_{{10\%}}$ | MRR@10 | MAP
---|---|---|---|---|---|---
ada-002 | 13.49 | 40.24 | 47.35 | 39.00 | 24.33 | 51.67
SentBERT | 12.15 | 36.71 | 45.08 | 34.78 | 20.71 | 48.96
RocketQA-v2 | 12.79 | 39.19 | 46.47 | 38.78 | 23.72 | 50.81

###  5.3 Supervised Learning on DORIS-MAE

To assess the utility of our dataset for training IR models, we conducted an
experiment where we allocated 40 queries for training and the remaining 60 for
testing. Using supervised contrastive learning (SCL), we optimized a margin-
based triplet loss as presented in Equation 3. Each triplet, represented as
$(A,P,N)$, consisted of a query and two abstracts. The higher-ranked abstract
in the training data served as the positive instance (i.e., $P$), with the
other functioning as the negative instance (i.e., $N$). Given each query’s
candidate pool size of $\geq 100$, we derived multiple triplets. This process
yielded 3,000 triplets from the designated 40 training queries. Subsequently,
we fine-tuned an E5-L-v2 model, a RoBERTa-based text embedding variant with
355 million parameters, over a single epoch.

$L(A,P,N)=\max\Bigg{(}\frac{<A,N>}{||A||\cdot||N||}-\frac{<A,P>}{||A||\cdot||P||}+m,0\Bigg{)}\quad
A,P,N\in\mathbb{R}^{n},m>0$ (3)

When evaluating the model on the 60 test queries, we noted a marked
improvement across all metrics with the SCL-trained model in contrast to the
pre-trained baseline. This comparison can be found in Table 8. These positive
outcomes, achieved using supervised contrastive learning on DORIS-MAE,
underscore the utility of our train/test split for model fine-tuning.

Note that Table 2 reports models performance on the full DORIS-MAE dataset. We
also report models performance on our proposed test set of 60 queries in
Appendix C.4.

Table 8: Comparison of SCL vs pretrained. Standard errors are estimated by bootstrapping. Method | R@5 | R@20 | RP | $\textrm{NDCG}^{\textrm{exp}}_{{10\%}}$ | MRR@10 | MAP
---|---|---|---|---|---|---
SCL-trained E5-v2 | 19.57 $\pm$ 2.33 | 52.45 $\pm$ 3.17 | 44.47 $\pm$ 3.11 | 34.67 $\pm$ 3.17 | 23.16 $\pm$ 4.28 | 49.15 $\pm$ 3.14
pretrained E5-v2 | 14.70 $\pm$ 1.72 | 42.38 $\pm$ 2.59 | 38.24 $\pm$ 2.94 | 26.31 $\pm$ 2.94 | 14.53 $\pm$ 3.69 | 40.62 $\pm$ 2.85

##  6 Conclusion and Future Work

This paper introduces a novel task, Scientific DOcument Retrieval using Multi-
level Aspect-based quEries (DORIS-MAE), aimed at modeling the process of
information retrieval in the context of scientific research. We also present a
dataset for DORIS-MAE generated using the Anno-GPT framework.

The results show room for improvement in the performance of current retrieval
methods when dealing with DORIS-MAE. Future studies may explore modifications
to model architectures and training procedures to better address complex,
multifaceted queries.

An understanding of how noise in aspect annotation affects the overall task is
an interesting point for future investigation, as it can shed light on how
errors in the automated annotations may affect the final candidate pool
ranking in DORIS-MAE.

The hierarchical structure of complex queries, as exemplified in DORIS-MAE, is
an area that warrants further attention. Future work might include the
development of more sophisticated automated query breakdown methods,
potentially drawing from advances in question decomposition [36, 92],
sequence-to-sequence modelling [42, 43] and semantic parsing [76, 88, 68].

Limitations: DORIS-MAE currently contains queries and abstracts from the
computer science domain. Consequently, models trained on this dataset may not
generalize well to other disciplines. An extended, multi-domain version of
DORIS-MAE is a logical direction for future work. The task of determining
aspect relevance is challenging due to the complexity of the abstracts.
Improved annotation guidelines and training (for both humans and models) may
address this challenge. Finally, while we harnessed LLMs to streamline the
annotation, the generation of queries and their aspect decomposition remains
manual. We found the development of a reliable, automated query generation
pipeline difficult, but anticipate that advances in LLMs might bridge this gap
in the near future.

## References

  * Abdi et al. [2007] H. Abdi et al.  Bonferroni and šidák corrections for multiple comparisons.  _Encyclopedia of measurement and statistics_ , 3(01):2007, 2007.
  * Afzali et al. [2021] J. Afzali, A. M. Drzewiecki, and K. Balog.  Pointrec: A test collection for narrative-driven point of interest recommendation.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2478–2484, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380379.  doi: 10.1145/3404835.3463243.  URL https://doi.org/10.1145/3404835.3463243.
  * Akkalyoncu Yilmaz et al. [2019] Z. Akkalyoncu Yilmaz, W. Yang, H. Zhang, and J. Lin.  Cross-domain modeling of sentence-level evidence for document retrieval.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3490–3496, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1352.  URL https://aclanthology.org/D19-1352.
  * Bang et al. [2023] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_ , 2023.
  * Beltagy et al. [2019] I. Beltagy, K. Lo, and A. Cohan.  Scibert: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, 2019.
  * Bogers and Koolen [2017] T. Bogers and M. Koolen.  Defining and supporting narrative-driven recommendation.  In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ , RecSys ’17, page 238–242, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450346528.  doi: 10.1145/3109859.3109893.  URL https://doi.org/10.1145/3109859.3109893.
  * Bowman et al. [2015a] S. Bowman, G. Angeli, C. Potts, and C. D. Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, 2015a.
  * Bowman et al. [2015b] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal, Sept. 2015b. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1075.  URL https://aclanthology.org/D15-1075.
  * Brin and Page [1998] S. Brin and L. Page.  The anatomy of a large-scale hypertextual web search engine.  _Computer networks and ISDN systems_ , 30(1-7):107–117, 1998.
  * Brown et al. [2019] P. Brown, R. Consortium, and Y. Zhou.  Large expert-curated database for benchmarking document similarity detection in biomedical literature search.  _Database_ , 2019, 10 2019.  ISSN 1758-0463.  doi: 10.1093/database/baz085.  URL https://doi.org/10.1093/database/baz085.  baz085.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Burges et al. [2005] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender.  Learning to rank using gradient descent.  In _Proceedings of the 22nd International Conference on Machine Learning_ , ICML ’05, page 89–96, New York, NY, USA, 2005. Association for Computing Machinery.  ISBN 1595931805.  doi: 10.1145/1102351.1102363.  URL https://doi.org/10.1145/1102351.1102363.
  * Buttcher et al. [2016] S. Buttcher, C. L. Clarke, and G. V. Cormack.  _Information retrieval: Implementing and evaluating search engines_.  Mit Press, 2016.
  * Chan et al. [2018] J. Chan, J. C. Chang, T. Hope, D. Shahaf, and A. Kittur.  Solvent: A mixed initiative system for finding analogies between research papers.  _Proc. ACM Hum.-Comput. Interact._ , 2(CSCW), nov 2018.  doi: 10.1145/3274300.  URL https://doi.org/10.1145/3274300.
  * Chen and Yih [2020] D. Chen and W.-t. Yih.  Open-domain question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pages 34–37, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-tutorials.8.  URL https://aclanthology.org/2020.acl-tutorials.8.
  * Chowdhery et al. [2022] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Clark et al. [2018] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Cohan et al. [2020] A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. S. Weld.  Specter: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, 2020.
  * Craswell et al. [2021] N. Craswell, B. Mitra, E. Yilmaz, and D. Campos.  Overview of the trec 2020 deep learning track, 2021.
  * Dimitriadou et al. [2014] K. Dimitriadou, O. Papaemmanouil, and Y. Diao.  Explore-by-example: An automatic query steering framework for interactive data exploration.  In _Proceedings of the 2014 ACM SIGMOD international conference on Management of data_ , pages 517–528, 2014.
  * Ding et al. [2022] B. Ding, C. Qin, L. Liu, L. Bing, S. Joty, and B. Li.  Is gpt-3 a good data annotator?  _arXiv preprint arXiv:2212.10450_ , 2022.
  * Faggioli et al. [2023] G. Faggioli, L. Dietz, C. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, and H. Wachsmuth.  Perspectives on large language models for relevance judgment, 2023.
  * Formal et al. [2021] T. Formal, C. Lassance, B. Piwowarski, and S. Clinchant.  Splade v2: Sparse lexical and expansion model for information retrieval.  _arXiv preprint arXiv:2109.10086_ , 2021.
  * Gao et al. [2021a] T. Gao, A. Fisch, and D. Chen.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, 2021a.
  * Gao et al. [2021b] T. Gao, X. Yao, and D. Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic, Nov. 2021b. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.552.  URL https://aclanthology.org/2021.emnlp-main.552.
  * Gebru et al. [2020] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumeé III, and K. Crawford.  Datasheets for Datasets.  _arXiv:1803.09010 [cs]_ , Jan. 2020.
  * Gilardi et al. [2023] F. Gilardi, M. Alizadeh, and M. Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_ , 2023.
  * Greene et al. [2022] R. Greene, T. Sanders, L. Weng, and A. Neelakantan.  New and improved embedding model.  https://openai.com/blog/new-and-improved-embedding-model/, 2022.  Accessed: 2023-06-03.
  * Guha et al. [2003] R. Guha, R. McCool, and E. Miller.  Semantic search.  In _Proceedings of the 12th International Conference on World Wide Web_ , WWW ’03, page 700–709, New York, NY, USA, 2003. Association for Computing Machinery.  ISBN 1581136803.  doi: 10.1145/775152.775250.  URL https://doi.org/10.1145/775152.775250.
  * Hsu [1996] J. Hsu.  _Multiple comparisons: theory and methods_.  CRC Press, 1996.
  * Hu et al. [2008] Y. Hu, Y. Koren, and C. Volinsky.  Collaborative filtering for implicit feedback datasets.  In _2008 Eighth IEEE international conference on data mining_ , pages 263–272. Ieee, 2008.
  * Huang et al. [2022] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR, 2022.
  * Jin et al. [2022] Q. Jin, Z. Yuan, G. Xiong, Q. Yu, H. Ying, C. Tan, M. Chen, S. Huang, X. Liu, and S. Yu.  Biomedical question answering: A survey of approaches and challenges.  _ACM Comput. Surv._ , 55(2), jan 2022.  ISSN 0360-0300.  doi: 10.1145/3490238.  URL https://doi.org/10.1145/3490238.
  * Karpukhin et al. [2020] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, 2020.
  * Khattab and Zaharia [2020] O. Khattab and M. Zaharia.  Colbert: Efficient and effective passage search via contextualized late interaction over bert.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_ , pages 39–48, 2020.
  * Khot et al. [2021] T. Khot, D. Khashabi, K. Richardson, P. Clark, and A. Sabharwal.  Text modular networks: Learning to decompose tasks in the language of existing models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1264–1279, 2021.
  * Kinney et al. [2023] R. M. Kinney, C. Anastasiades, R. Authur, I. Beltagy, J. Bragg, A. Buraczynski, I. Cachola, S. Candra, Y. Chandrasekhar, A. Cohan, M. Crawford, D. Downey, J. Dunkelberger, O. Etzioni, R. Evans, S. Feldman, J. Gorney, D. W. Graham, F. Hu, R. Huff, D. King, S. Kohlmeier, B. Kuehl, M. Langan, D. Lin, H. Liu, K. Lo, J. Lochner, K. MacMillan, T. Murray, C. Newell, S. Rao, S. Rohatgi, P. L. Sayre, Z. Shen, A. Singh, L. Soldaini, S. Subramanian, A. Tanaka, A. D. Wade, L. M. Wagner, L. L. Wang, C. Wilhelm, C. Wu, J. Yang, A. Zamarron, M. van Zuylen, and D. S. Weld.  The semantic scholar open data platform.  _ArXiv_ , abs/2301.10140, 2023.
  * Kojima et al. [2022] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa.  Large language models are zero-shot reasoners.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf.
  * Kuhn et al. [2023] L. Kuhn, Y. Gal, and S. Farquhar.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_ , 2023.
  * Kwiatkowski et al. [2019] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_ , 2019.
  * Lee et al. [2013] J. Lee, K. Lee, and J. G. Kim.  Personalized academic research paper recommendation system.  _arXiv preprint arXiv:1304.5457_ , 2013.
  * Li et al. [2018] Z. Li, J. Cai, S. He, and H. Zhao.  Seq2seq dependency parsing.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3203–3214, 2018.
  * Lin et al. [2022] B. Lin, Z. Yao, J. Shi, S. Cao, B. Tang, S. Li, Y. Luo, J. Li, and L. Hou.  Dependency parsing via sequence generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7339–7353, 2022.
  * Lissandrini et al. [2019] M. Lissandrini, D. Mottin, T. Palpanas, and Y. Velegrakis.  Example-based search: A new frontier for exploratory search.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 1411–1412, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450361729.  doi: 10.1145/3331184.3331387.  URL https://doi.org/10.1145/3331184.3331387.
  * Liu et al. [2023] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35, 2023.
  * Liu et al. [2022] T. Liu, Y. Zhang, C. Brockett, Y. Mao, Z. Sui, W. Chen, and B. Dolan.  A token-level reference-free hallucination detection benchmark for free-form text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6723–6737, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.464.  URL https://aclanthology.org/2022.acl-long.464.
  * Lu et al. [2022] Y. Lu, Y. Liu, J. Liu, Y. Shi, Z. Huang, S. F. Y. Sun, H. Tian, H. Wu, S. Wang, D. Yin, and H. Wang.  Ernie-search: Bridging cross-encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval, 2022.
  * MacAvaney and Soldaini [2023] S. MacAvaney and L. Soldaini.  One-shot labeling for automatic relevance estimation.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’23, page 2230–2235, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9781450394086.  doi: 10.1145/3539618.3592032.  URL https://doi.org/10.1145/3539618.3592032.
  * MacAvaney et al. [2022] S. MacAvaney, S. Feldman, N. Goharian, D. Downey, and A. Cohan.  ABNIRML: Analyzing the behavior of neural IR models.  _Transactions of the Association for Computational Linguistics_ , 10:224–239, 2022.  doi: 10.1162/tacl_a_00457.  URL https://aclanthology.org/2022.tacl-1.13.
  * Manakul et al. [2023] P. Manakul, A. Liusie, and M. J. Gales.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_ , 2023.
  * Min et al. [2022] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.759.
  * Morin and Willetts [2020] M. Morin and M. Willetts.  Non-determinism in tensorflow resnets.  _arXiv preprint arXiv:2001.11396_ , 2020.
  * Mysore et al. [2021] S. Mysore, T. O' Gorman, A. McCallum, and H. Zamani.  Csfcube - a test collection of computer science research articles for faceted query by example.  In J. Vanschoren and S. Yeung, editors, _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran, 2021.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/20f07591c6fcb220ffe637cda29bb3f6-Paper-round2.pdf.
  * Mysore et al. [2022] S. Mysore, A. Cohan, and T. Hope.  Multi-vector models with textual guidance for fine-grained scientific document similarity.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4453–4470, 2022.
  * Mysore et al. [2023] S. Mysore, A. McCallum, and H. Zamani.  Large language model augmented narrative driven recommendations, 2023.
  * Neves et al. [2019] M. Neves, D. Butzke, and B. Grune.  Evaluation of scientific elements for text similarity in biomedical publications.  In _Proceedings of the 6th Workshop on Argument Mining_ , pages 124–135, Florence, Italy, Aug. 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-4515.  URL https://aclanthology.org/W19-4515.
  * Nguyen et al. [2016] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng.  Ms marco: A human generated machine reading comprehension dataset.  _choice_ , 2640:660, 2016.
  * Ostendorff [2020] M. Ostendorff.  Contextual document similarity for content-based literature recommender systems.  _arXiv preprint arXiv:2008.00202_ , 2020.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Qin et al. [2023] C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang.  Is chatgpt a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_ , 2023.
  * Reddy et al. [2019] S. Reddy, D. Chen, and C. D. Manning.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, 2019.
  * Reimers and Gurevych [2019] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Ren et al. [2021] R. Ren, Y. Qu, J. Liu, W. X. Zhao, Q. She, H. Wu, H. Wang, and J.-R. Wen.  Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2825–2835, 2021.
  * Saeidi et al. [2018] M. Saeidi, M. Bartolo, P. Lewis, S. Singh, T. Rocktäschel, M. Sheldon, G. Bouchard, and S. Riedel.  Interpretation of natural language rules in conversational machine reading.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2087–2097, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1233.  URL https://aclanthology.org/D18-1233.
  * Sanh et al. [2022] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. Le Scao, A. Raja, et al.  Multitask prompted training enables zero-shot task generalization.  In _ICLR 2022-Tenth International Conference on Learning Representations_ , 2022.
  * Santhanam et al. [2022] K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia.  Colbertv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, 2022.
  * Sparck Jones [1972] K. Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_ , 28(1):11–21, 1972.
  * Sun et al. [2020] Y. Sun, L. Zhang, G. Cheng, and Y. Qu.  Sparqa: skeleton-based semantic parsing for complex questions over knowledge bases.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 8952–8959, 2020.
  * Thakur et al. [2021] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych.  Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Törnberg [2023] P. Törnberg.  Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.  _arXiv preprint arXiv:2304.06588_ , 2023.
  * Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Trotman et al. [2014] A. Trotman, A. Puurula, and B. Burgess.  Improvements to bm25 and language models examined.  In _Proceedings of the 2014 Australasian Document Computing Symposium_ , pages 58–65, 2014.
  * Voorhees et al. [2021] E. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang.  Trec-covid: Constructing a pandemic information retrieval test collection.  _SIGIR Forum_ , 54(1), feb 2021.  ISSN 0163-5840.  doi: 10.1145/3451964.3451965.  URL https://doi.org/10.1145/3451964.3451965.
  * Wang et al. [2022a] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei.  Text embeddings by weakly-supervised contrastive pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022a.
  * Wang et al. [2022b] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei.  Simlm: Pre-training with representation bottleneck for dense passage retrieval.  _arXiv preprint arXiv:2207.02578_ , 2022b.
  * Wang et al. [2023] S. Wang, A. Shrivastava, and S. Livshits.  Treepiece: Faster semantic parsing via tree tokenization.  _arXiv preprint arXiv:2303.17161_ , 2023.
  * Wang et al. [2022c] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022c.
  * Wang et al. [2022d] Y. Wang, Y. Hou, H. Wang, Z. Miao, S. Wu, Q. Chen, Y. Xia, C. Chi, G. Zhao, Z. Liu, X. Xie, H. Sun, W. Deng, Q. Zhang, and M. Yang.  A neural corpus indexer for document retrieval.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 25600–25614. Curran Associates, Inc., 2022d.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/a46156bd3579c3b268108ea6aca71d13-Paper-Conference.pdf.
  * Wei et al. [2021] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Wei et al. [2022a] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. [2022b] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates, Inc., 2022b.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf.
  * Weissenborn et al. [2017] D. Weissenborn, G. Wiese, and L. Seiffe.  Making neural QA as simple as possible but not simpler.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 271–280, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics.  doi: 10.18653/v1/K17-1028.  URL https://aclanthology.org/K17-1028.
  * Weisstein [2004] E. W. Weisstein.  Bonferroni correction.  _https://mathworld. wolfram. com/_ , 2004.
  * Williams et al. [2018] A. Williams, N. Nangia, and S. Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, 2018.
  * Xiong et al. [2020] L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ , 2020.
  * Yang et al. [2015] Y. Yang, W.-t. Yih, and C. Meek.  Wikiqa: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 2013–2018, 2015.
  * Yang et al. [2018] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, 2018.
  * Zhang et al. [2019] H. Zhang, J. Cai, J. Xu, and J. Wang.  Complex question decomposition for semantic parsing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4477–4486, 2019.
  * Zhang et al. [2023a] M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith.  How language model hallucinations can snowball.  _arXiv preprint arXiv:2305.13534_ , 2023a.
  * Zhang et al. [2023b] Y. Zhang, Z. Wang, and J. Shang.  Clusterllm: Large language models as a guide for text clustering.  _arXiv preprint arXiv:2305.14871_ , 2023b.
  * Zhao et al. [2021] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR, 2021.
  * Zhou et al. [2022] B. Zhou, K. Richardson, X. Yu, and D. Roth.  Learning to decompose: Hypothetical question decomposition based on comparable texts.  _arXiv preprint arXiv:2210.16865_ , 2022.

Appendix: Table of Contents

  1. A.

DatasetA

    1. 1.

Dataset License and Code LicenseA.1

    2. 2.

Dataset Hosting, Accessibility and MaintenanceA.2

    3. 3.

Dataset OverviewA.3

    4. 4.

DORIS-MAE StructureA.4

    5. 5.

Statistics of DatasetA.5

  2. B.

AnnotationB

    1. 1.

Annotation GuidelinesB.1

    2. 2.

Examples for AnnotationsB.2

    3. 3.

Prompt EngineeringB.3

    4. 4.

Error Analysis for ChatGPT’s AnnotationB.4

    5. 5.

Annotation ScalabilityB.5

  3. C.

Experiment ResultsC

    1. 1.

MetricsC.1

    2. 2.

Query Level Benchmark ResultsC.2

    3. 3.

Using Aspects for RetrievalC.3

    4. 4.

Models Performance on DORIS-MAE Test SetC.4

    5. 5.

Two-Aspect Sub-Query Level Benchmark ResultsC.5

    6. 6.

Candidate Pool Sensitivity AnalysesC.6

  4. D.

Experiment SetupD

    1. 1.

Retraining DetailsD.1

    2. 2.

Model SpecificationsD.2

  5. E.

Author StatementE

  6. F.

Datasheet for DORIS-MAEF

##  Appendix A Dataset

###  A.1 Dataset License and Code License

The DORIS-MAE dataset is made available under the Creative Commons
Attribution-NonCommercial (CC-BY-NC) license. This license allows users to
share and adapt the dataset under the condition that they provide appropriate
credit to the original creators and do not use it for commercial purposes. A
copy of the full license can be found at https://github.com/Real-Doris-
Mae/Doris-Mae-Dataset/blob/main/dataset_license.md.

The code used in this paper is released under the MIT License. The MIT License
is a permissive open-source license that allows for the free use,
modification, and distribution of the code, as long as the original license is
included with any derivative work. A copy of the full license can be found at
https://github.com/Real-Doris-Mae/Doris-Mae-Dataset/blob/main/code_license.md.

###  A.2 Dataset Hosting, Accessibility and Maintenance

The DORIS-MAE dataset with its meta-data is released and can be accessed
freely at (https://doi.org/10.5281/zenodo.8299749) under the CC-BY-NC license.
We commit to regularly maintain the dataset and codebase by incorporating user
feedback. We will potentially introduce more features as part of future work
in the next version of DORIS-MAE. We confirm that the current version of
DORIS-MAE would always remain accessible at the same link.

###  A.3 Dataset Overview

The DORIS-MAE dataset is comprised of four main sub-datasets, each serving
distinct purposes.

The Query dataset contains 100 human-crafted complex queries spanning across
five categories: ML, NLP, CV, AI, and Composite. Each category has 20
associated queries. Queries are broken down into aspects (ranging from 3 to 9
per query) and sub-aspects (from 0 to 6 per aspect, with 0 signifying no
further breakdown required). For each query, a corresponding candidate pool of
relevant paper abstracts, ranging from 99 to 138, is provided.

The Corpus dataset is composed of 363,133 abstracts from computer science
papers, published between 2011-2021, and sourced from arXiv. Each entry
includes title, original abstract, URL, primary and secondary categories, as
well as citation information retrieved from Semantic Scholar. A masked version
of each abstract is also provided, facilitating the automated creation of
queries.

The Annotation dataset includes generated annotations for all 165,144 question
pairs, each comprising an aspect/sub-aspect and a corresponding paper abstract
from the query’s candidate pool. It includes the original text generated by
ChatGPT explaining its decision-making process, along with a three-level
relevance score (e.g., 0,1,2) representing ChatGPT’s final decision.

Finally, the Test Set dataset contains human annotations for a random
selection of 250 question pairs used in hypothesis testing. It includes each
of the three human annotators’ final decisions, recorded as a three-level
relevance score (e.g., 0,1,2). Note, the development set contains 90 annotated
question pairs. Since the annotations in the development set come from only 2
annotators and since the development set is strictly not a part of the
hypothesis testing stage, we do not include the development set in the
officially released DORIS-MAE dataset. Instead, we provide the development set
as part of the codebase on https://github.com/Real-Doris-Mae/Doris-Mae-
Dataset/.

It is important to note that each of these datasets serves a specific function
in our overall process, contributing to the optimization, validation, and
benchmarking of our proposed approach.

###  A.4 DORIS-MAE Structure

  1. -

aspect2aspect_id: A dictionary that maps aspects/sub-aspects to their
aspect_id.

  2. -

aspect_id2aspect: The inverse mapping of aspect2aspect_id.

  3. -

Query: contains 50 complex queries.

    1. •

Query 1:

      1. -

query_text: The text of the query.

      2. -

query_type: One of the [ML, NLP, CV, AI, Composite].

      3. -

idea_from: The id of the referenced abstract when creating the query

      4. -

candidate_pool: The list of abstract_ids in the candidate pool of Query 1.

      5. -

sent2aspect_id: A dictionary that maps individual sentence to its
aspect_id(s).

      6. -

aspect_id2sent: The inverse mapping of sent2aspect_id.

      7. -

aspects: contains all aspects ids excluding sub-aspects for Query 1.

        1. •

aspect1_id: [sub-aspect1_id, sub-aspect2_id,…]

        2. •

aspect2_id:

        3. •

…

    2. •

Query 2:

    3. •

…

  4. -

Corpus: contains 363,133 Computer Science(CS) paper abstracts.

    1. •

Abstract 1:

      1. -

original_abstract: The text of the abstract.

      2. -

masked_abstract: The text of the abstract with specialized topic words masked.

      3. -

title: Title of the paper.

      4. -

url: Link to the paper.

      5. -

primary_category: Primary category of the paper, assigned by arXiv.

      6. -

categories: Full list of the categories of the paper, assigned by arXiv.

      7. -

ss_id: The paper’s semantic scholar id.

      8. -

incoming_citations: List of abstract_ids citing this paper.

      9. -

outgoing_citations: List of abstract_ids cited by this paper.

      10. -

abstract_id: the index 1 is the abstract_id for this paper.

    2. •

Abstract 2:

    3. •

…

  5. -

Annotation: contains 83,591 annotated question pairs.

    1. •

Annotation 1: The first question pair.

      1. -

aspect_id: The id of the aspect/sub-aspect in the question pair.

      2. -

abstract_id: The id of the abstract in the question pair.

      3. -

gpt_response: The reasoning and decision by ChatGPT for this question pair.

      4. -

score: The relevance score of the question pair (e.g. 0,1,2).

    2. •

Annotation 2:

    3. •

…

  6. -

Test Set: contains 250 humanly annotated question pairs.

    1. •

Test Set Question Pair 1:

      1. -

aspect_id: The id of the aspect/sub-aspect in the question pair.

      2. -

abstract_id: The id of the abstract in the question pair.

      3. -

human_annotation: contains the relevance score assigned by all 3 annotators.

        1. •

annotator_1: Score of annotator_1.

        2. •

annotator_2: Score of annotator_2.

        3. •

annotator_3: Score of annotator_3.

    2. •

Test Set Question Pair 2:

    3. •

…

###  A.5 Statistics of Dataset

Table 9: Number of relevant abstracts for each query type. Comp means composite queries. Sub-query stands for two-aspect sub-queries. |  | Min | Max | Mean | Std
---|---|---|---|---|---
Query | ML | 2 | 84 | 24.35 | 19.47
NLP | 0 | 41 | 17.65 | 11.98
CV | 0 | 52 | 18.55 | 13.29
AI | 1 | 51 | 11.60 | 12.35
Comp | 0 | 61 | 9.95 | 14.63
All | 0 | 84 | 16.42 | 15.49
Sub-query | ML | 1 | 105 | 36.40 | 25.34
NLP | 0 | 83 | 29.86 | 18.56
CV | 1 | 99 | 28.52 | 16.57
AI | 0 | 79 | 20.96 | 16.16
Comp | 0 | 81 | 23.09 | 21.91
All | 0 | 105 | 28.14 | 21.07

Table 10: Normalized score of abstracts for each query type. Compo means composite queries. Sub-query stands for two-aspect sub-queries. |  | Min | Max | Mean | Std
---|---|---|---|---|---
Query | ML | 0 | 1.778 | 0.665 | 0.377
NLP | 0 | 1.786 | 0.609 | 0.361
CV | 0 | 1.615 | 0.623 | 0.339
AI | 0 | 1.667 | 0.510 | 0.337
Comp | 0 | 1.909 | 0.483 | 0.332
All | 0 | 1.909 | 0.578 | 0.356
Sub-query | ML | 0 | 2.000 | 0.679 | 0.478
NLP | 0 | 2.000 | 0.638 | 0.453
CV | 0 | 2.000 | 0.642 | 0.434
AI | 0 | 2.000 | 0.533 | 0.419
Comp | 0 | 2.000 | 0.544 | 0.425
All | 0 | 2.000 | 0.610 | 0.448

(a) Normalized scores of abstracts in candidate pool per query.

(b) Aggregated normalized scores of abstracts per query type.

Figure 2: Each line in (a) represents a query. Each line in (b) represents a
query type. The color of the line represents the type of the query. The
candidate abstracts are in descending order w.r.t. their normalized scores.

To better interpret the results of our evaluations and guide future
improvements, we provide several analyses of the distribution and relevance of
the abstracts in our DORIS-MAE dataset.

We used Equations 1 and 2 to determine the relevance of an abstract to a
specific query. We then examined the distribution of the number of relevant
abstracts for different queries and considered how this might impact various
evaluation metrics.

Table 10 presents the distribution of relevant abstracts categorized by query
type (ML, NLP, CV, AI, and Composite) in our dataset. For each category, we
determined the count of relevant abstracts for its respective queries and
provided summary statistics of these counts. We find that composite queries
have a lower average number of relevant abstracts, implying that such queries
might encompass aspects or sub-aspects not extensively covered by papers in
their associated pools. This observation, along with the varied number of
relevant abstracts per query (ranging from 0 to 84), suggests that the DORIS-
MAE dataset encompasses queries of diverse difficulty levels. It’s noteworthy
that four queries lack any relevant abstracts and were consequently omitted
from our evaluation.

The lower portion of Table 10 presents the distribution of relevant abstracts
for 1003 sub-queries with two aspects. As the table shows, the number of
relevant abstracts increases across all five types, suggesting that the sub-
query DORIS-MAE task is less challenging than the full-query DORIS-MAE task.

Next, we examined the distribution of normalized scores (defined as
$\frac{S(p_{j}|Q)}{|Q|}$) for each abstract given a query. We compiled the
normalized scores for each abstract in the candidate pools for each query type
and report the aggregate statistics in Table 10. Composite queries have the
lowest average relevance scores.

Figure 2(a) shows the normalized scores within a candidate pool (ranked in
descending order) for a given query. Figure 2(b) aggregates the data from
Figure 2(a). As seen in Figure 2(a), there are only a few plateaus (i.e.,
abstracts with tied scores), which means the normalized scores provide a fine-
grained ranking for the candidate pool. This granularity is valuable when
evaluating reranking models.

##  Appendix B Annotation

###  B.1 Annotation Guidelines

In Section 4, we introduced the grading scale (0-2) for evaluating the
relationship between an abstract and an aspect/sub-aspect in a question pair.
To facilitate this annotation process, we expand the definition of Level 1
into two sub-categories, creating a four-level grading scale during the
annotation phase. Afterward, these levels are mapped back onto the original
three-level scale.

The four-level grading scale and their corresponding mappings are as follows:

  1. 1.

DISAGREE: This corresponds to Level 0 in the main paper. The abstract is
unrelated or does not provide any help to the key components of the aspect or
sub-aspect.

  2. 2.

DISPUTE: This is a sub-category of Level 1. The abstract answers some key
components and is adaptable to fulfill the aspect/sub-aspect. However, the
abstract does not directly address the aspect/sub-aspect.

  3. 3.

AGREE: This is another sub-category of Level 1. The abstract directly
addresses only a portion of the aspect/sub-aspect, but not in its entirety.

  4. 4.

CONCUR: This corresponds to Level 2. The abstract directly answers all the key
components of the aspect or sub-aspect.

These labels "DISAGREE", "DISPUTE", "AGREE", and "CONCUR" serve to add
semantic meaning to the grading levels, aiding annotators during the
annotation process.

For consistency, "DISPUTE" and "AGREE" are treated as equal in weight,
representing different scenarios under Level 1, but neither is considered
stronger or weaker than the other.

During the annotation process, annotators must follow a set of guidelines
designed to maintain consistency and minimize bias. These guidelines cover
aspects like time management, communication among annotators (which is
explicitly forbidden), use of additional resources, the process of question
analysis, and the identification of key components. Details of these
guidelines can be found in the following section.

  1. 1.

Resource usage: Given the technical nature of the abstracts, annotators may
occasionally need to look up unfamiliar terms. They can use traditional search
engines for these instances. However, they are strictly prohibited from using
any large language model (LLM) interfaces or search engines powered by LLMs.

  2. 2.

Annotation process: Annotators are required to read the entire abstract and
question at least once before making a decision. They need to refresh their
understanding of the four grading levels in their minds before finalizing
their decision. On average, this process takes about 4 minutes, but no strict
time limit is enforced. During the annotation process, annotators are strictly
and explicitly prohibited from communicating in any way.

  3. 3.

Key component identification: The grading scale revolves around the concept of
"key components". It’s up to the annotators, leveraging their research
experience, to identify what constitutes as a key component in an aspect/sub-
aspect. If there are multiple key components (e.g., 2 or 3), they are expected
to monitor all of these components and verify whether the abstract mentions
them.

  4. 4.

Inference rules: Annotators can leverage their expert knowledge to draw
reasonable inferences from the abstract’s content with respect to the key
components. However, they should avoid unnecessary complications in the
correlation between the aspect and the abstract. Each inference made must have
a clear, straightforward justification. The aim here is to ensure accuracy
without over-complicating the annotation process.

###  B.2 Examples for Annotations

In this section, we provide examples of aspect/abstract pairs with different
relevance scores.

####  B.2.1 Examples with relevance score 0

  * > Aspect/Sub-aspect: The paper mentions the possibility that the labeling process introduces unwanted randomness or noise.
>
>
> Abstract: Recent approaches in literature have exploited the multi-modal
> information in documents (text, layout, image) to serve specific downstream
> document tasks. However, they are limited by their - (i) inability to learn
> cross-modal representations across text, layout and image dimensions for
> documents and (ii) inability to process multi-page documents. Pre-training
> techniques have been shown in Natural Language Processing (NLP) domain to
> learn generic textual representations from large unlabelled datasets,
> applicable to various downstream NLP tasks. In this paper, we propose a
> multi-task learning-based framework that utilizes a combination of self-
> supervised and supervised pre-training tasks to learn a generic document
> representation applicable to various downstream document tasks.
> Specifically, we introduce Document Topic Modelling and Document Shuffle
> Prediction as novel pre-training tasks to learn rich image representations
> along with the text and layout representations for documents. We utilize the
> Longformer network architecture as the backbone to encode the multi-modal
> information from multi-page documents in an end-to-end fashion. We showcase
> the applicability of our pre-training framework on a variety of different
> real-world document tasks such as document classification, document
> information extraction, and document retrieval. We evaluate our framework on
> different standard document datasets and conduct exhaustive experiments to
> compare performance against various ablations of our framework and state-of-
> the-art baselines.

> Human reasoning: The above abstract is not related to the aspect.

  * > Aspect/Sub-aspect: The document should use semi-definite programming to estimate error bound.
>
>
> Abstract: We develop methodology for estimation and inference using machine
> learning to enrich economic models. Our framework takes a standard economic
> model and recasts the parameters as fully flexible nonparametric functions,
> to capture the rich heterogeneity based on potentially high dimensional or
> complex observable characteristics. These "parameter functions" retain the
> interpretability, economic meaning, and discipline of classical parameters.
> Deep learning is particularly well-suited to structured modeling of
> heterogeneity in economics. We show how to design the network architecture
> to match the structure of the economic model, delivering novel methodology
> that moves deep learning beyond prediction. We prove convergence rates for
> the estimated parameter functions. These functions are the key inputs into
> the finite-dimensional parameter of inferential interest. We obtain
> inference based on a novel influence function calculation that covers any
> second-stage parameter and any machine-learning-enriched model that uses a
> smooth per-observation loss function. No additional derivations are
> required. The score can be taken directly to data, using automatic
> differentiation if needed. The researcher need only define the original
> model and define the parameter of interest. A key insight is that we need
> not write down the influence function in order to evaluate it on the data.
> Our framework gives new results for a host of contexts, covering such
> diverse examples as price elasticities, willingness-to-pay, and surplus
> measures in binary or multinomial choice models, effects of continuous
> treatment variables, fractional outcome models, count data, heterogeneous
> production functions, and more. We apply our methodology to a large scale
> advertising experiment for short-term loans. We show how economically
> meaningful estimates and inferences can be made that would be unavailable
> without our results.

> Human reasoning: The above abstract is not related to the aspect.

####  B.2.2 Examples with relevance score 1

  * > Aspect/Sub-aspect: The paper should introduce a system that can represent the possible options and consequences in decision making using actions and states.
>
>
> Abstract: Decision-making is often dependent on uncertain data, e.g. data
> associated with confidence scores or probabilities. We present a comparison
> of different information presentations for uncertain data and, for the first
> time, measure their effects on human decision-making. We show that the use
> of Natural Language Generation (NLG) improves decision-making under
> uncertainty, compared to state-of-the-art graphical-based representation
> methods. In a task-based study with 442 adults, we found that presentations
> using NLG lead to 24% better decision-making on average than the graphical
> presentations, and to 44% better decision-making when NLG is combined with
> graphics. We also show that women achieve significantly better results when
> presented with NLG output (an 87% increase on average compared to graphical
> presentations).

> Human reasoning: The above abstract explicitly discuss decision-making.
> However, the abstract does not mention anything about actions and states.
> Thus, some key components are missing. More specifically, it belongs to the
> class "Agree" under Level 1.

  * > Aspect/Sub-aspect: The document should introduce a system that can retrieve fine-grained video of actions from longer videos and match these actions to the text description or instruction.
>
>
> Abstract: In this paper, we propose a new approach for retrieval of video
> segments using natural language queries. Unlike most previous approaches
> such as concept-based methods or rule-based structured models, the proposed
> method uses image captioning model to construct sentential queries for
> visual information. In detail, our approach exploits multiple captions
> generated by visual features in each image with ‘Densecap’. Then, the
> similarities between captions of adjacent images are calculated, which is
> used to track semantically similar captions over multiple frames. Besides
> introducing this novel idea of ’tracking by captioning’, the proposed method
> is one of the first approaches that uses a language generation model learned
> by neural networks to construct semantic query describing the relations and
> properties of visual information. To evaluate the effectiveness of our
> approach, we have created a new evaluation dataset, which contains about 348
> segments of scenes in 20 movie-trailers. Through quantitative and
> qualitative evaluation, we show that our method is effective for retrieval
> of video segments using natural language queries.

> Human reasoning: The above abstract mentions retrieval of video segments
> using natural language queries, which is related to the aspect. However,
> another key component of the abstract is "action". Although it is very
> natural to infer that the system in the document can retrieve video of
> actions, it is not explicitly mentioned in the abstract. More specifically,
> it belongs to the class "Dispute" under Level 1.

####  B.2.3 Examples with relevance score 2

  * > Aspect/Sub-aspect: The paper should mention chemistry or chemistry related knowledge.
>
>
> Abstract: Modern astronomical surveys are observing spectral data for
> millions of stars. These spectra contain chemical information that can be
> used to trace the Galaxy’s formation and chemical enrichment history.
> However, extracting the information from spectra, and making precise and
> accurate chemical abundance measurements are challenging. Here, we present a
> data-driven method for isolating the chemical factors of variation in
> stellar spectra from those of other parameters (i.e. \teff, \logg, eh). This
> enables us to build a spectral projection for each star with these
> parameters removed. We do this with no ab initio knowledge of elemental
> abundances themselves, and hence bypass the uncertainties and systematics
> associated with modeling that rely on synthetic stellar spectra. To remove
> known non-chemical factors of variation, we develop and implement a neural
> network architecture that learns a disentangled spectral representation. We
> simulate our recovery of chemically identical stars using the disentangled
> spectra in a synthetic APOGEE-like dataset. We show that this recovery
> declines as a function of the signal to noise ratio, but that our neural
> network architecture outperforms simpler modeling choices. Our work
> demonstrates the feasibility of data-driven abundance-free chemical tagging.

> Human reasoning: The above abstract explicitly mention "recovery of
> chemically identical stars" which is related to chemistry.

  * > Aspect/Sub-aspect: The paper should provide robust safeguards to sensitive patient data.
>
>
> Abstract: In survival analysis, regression models are used to understand the
> effects of explanatory variables (e.g., age, sex, weight, etc.) to the
> survival probability. However, for sensitive survival data such as medical
> data, there are serious concerns about the privacy of individuals in the
> data set when medical data is used to fit the regression models. The closest
> work addressing such privacy concerns is the work on Cox regression which
> linearly projects the original data to a lower dimensional space. However,
> the weakness of this approach is that there is no formal privacy guarantee
> for such projection. In this work, we aim to propose solutions for the
> regression problem in survival analysis with the protection of differential
> privacy which is a golden standard of privacy protection in data privacy
> research. To this end, we extend the Output Perturbation and Objective
> Perturbation approaches which are originally proposed to protect
> differential privacy for the Empirical Risk Minimization (ERM) problems. In
> addition, we also propose a novel sampling approach based on the Markov
> Chain Monte Carlo (MCMC) method to practically guarantee differential
> privacy with better accuracy. We show that our proposed approaches achieve
> good accuracy as compared to the non-private results while guaranteeing
> differential privacy for individuals in the private data set.

> Human reasoning: The above abstract explicitly discuss differential privacy"
> and sensitive private medical data, which satisfies all components in the
> abstract.

###  B.3 Prompt Engineering

The prompt strategy was optimized on the development set, and was designed
with an emphasis on clarity and conciseness. It aimed to elicit detailed
reasoning and a definitive final decision from ChatGPT.

Key to the final prompt design was the selection of labels for the grading
options: "Disagree," "Dispute," "Agree," and "Concur." These labels were
chosen to reflect the different grading scale levels defined in Section B.1.

These labels, along with their detailed definitions, are included in the
prompt. They serve to guide ChatGPT in its decision-making process by
providing consistent terminology for each grading level. The final prompt used
in the hypothesis testing stage is displayed below.

> You are a reference librarian. A user wants to find a scientific paper that
> satisfies a particular requirement. Based on the abstract of the paper, you
> will determine whether the requirement is met.
>
> Here is the user requirement: {aspect}
>
> Here is the abstract: {abstract}
>
> Explain whether the abstract satisfies the user requirement. Think step by
> step, showing all of your reasoning. Use the following *rules* to determine
> whether the requirement is satisfied:
>
> Disagree means the abstract does not address the user requirement;
>
> Dispute means the abstract does not directly address the user requirement,
> but is adaptable to fulfill the user’s need;
>
> Concur means the abstract directly and completely addresses the user
> requirement;
>
> Agree means the abstract directly addresses only a portion of the user
> requirement, but not in its entirety.
>
> Think carefully about the aforementioned *rules* to decide whether you
> [DISAGREE, DISPUTE, AGREE, CONCUR] based on the definition of these 4
> options. After you have provided your reasoning, state your final selection
> in *all capital* letters at the *end* of your response.

This prompt produced the best results on the development set, with the
distribution of its responses closely matching that of human annotators. Based
on these results, we proceeded to use this prompt in the testing process.

###  B.4 Error Analysis for ChatGPT’s Annotation

Though the final prompt yielded results comparable to human annotators, we
aimed to identify potential inconsistencies in ChatGPT’s responses that
slightly deviate from humans’ answers. We have organized them into several
categories:

####  B.4.1 Definition Inconsistency

Test Case Prompt:

> You are a reference librarian. A user wants to find a scientific paper that
> satisfies a particular requirement. Based on the abstract of the paper, you
> will determine whether the requirement is met.
>
> Here is the user requirement: The model in the document should have the
> ability to discover the geometry and global structure of the face
>
> Here is the abstract: Facial image super-resolution (SR) is an important
> preprocessing for facial image analysis, face recognition, and image-based
> 3D face reconstruction. Recent convolutional neural network (CNN) based
> method has shown excellent performance by learning mapping relation using
> pairs of low-resolution (LR) and high-resolution (HR) facial images.
> However, since the HR facial image reconstruction using CNN is
> conventionally aimed to increase the PSNR and SSIM metrics, the
> reconstructed HR image might not be realistic even with high scores. An
> adversarial framework is proposed in this study to reconstruct the HR facial
> image by simultaneously generating an HR image with and without blur. First,
> the spatial resolution of the LR facial image is increased by eight times
> using a five-layer CNN. Then, the encoder extracts the features of the up-
> scaled image. These features are finally sent to two branches (decoders) to
> generate an HR facial image with and without blur. In addition, local and
> global discriminators are combined to focus on the reconstruction of HR
> facial structures. Experiment results show that the proposed algorithm
> generates a realistic HR facial image. Furthermore, the proposed method can
> generate a variety of different facial images.
>
> Explain whether the abstract satisfies the user requirement. Think step by
> step, showing all of your reasoning. Use the following *rules* to determine
> whether the requirement is satisfied: disagree means the abstract does not
> address the user requirement; dispute means the abstract does not directly
> address the user requirement, but is adaptable to fulfill the user’s need;
> concur means the abstract directly and completely addresses the user
> requirement; agree means the abstract directly addresses only a portion of
> the user requirement, but not in its entirety. Think carefully about the
> aforementioned *rules* to decide whether you [DISAGREE, DISPUTE, AGREE,
> CONCUR] based on the definition of these 4 options. After you have provided
> your reasoning, state your final selection in *all capital* letters at the
> *end* of your response.

ChatGPT’s Reasoning:

> The user requirement is that the model in the document should have the
> ability to discover the geometry and global structure of the face. The
> abstract discusses a proposed algorithm for facial image super-resolution
> using an adversarial framework that generates a realistic HR facial image.
> The algorithm uses local and global discriminators to focus on the
> reconstruction of HR facial structures. Based on this information, I CONCUR
> that the abstract satisfies the user requirement. The proposed algorithm is
> designed to reconstruct the HR facial image by simultaneously generating an
> HR image with and without blur. The local and global discriminators are
> combined to focus on the reconstruction of HR facial structures, which
> implies that the model has the ability to discover the geometry and global
> structure of the face. Therefore, the abstract directly and completely
> addresses the user requirement.

##### Error Analysis:

Despite the abstract’s implied ability to discern facial structures, it
doesn’t state this explicitly. However, even after recognizing the ability as
"implied," ChatGPT marked the abstract as directly satisfying the requirement.
This indicates occasional misalignment between ChatGPT’s reasoning and the
labeling criteria.

####  B.4.2 Inference Inconsistency

Test Case Prompt:

> You are a reference librarian. A user wants to find a scientific paper that
> satisfies a particular requirement. Based on the abstract of the paper, you
> will determine whether the requirement is met.
>
> Here is the user requirement: The document should introduce a model that can
> take human faces with stripe patterns and accurately remove the stripe
> pattern.
>
> Here is the abstract: In this paper, we propose a robust 3D face recognition
> system which can handle pose as well as occlusions in real world. The system
> at first takes as input, a 3D range image, simultaneously registers it using
> ICP(Iterative Closest Point) algorithm. ICP used in this work, registers
> facial surfaces to a common model by minimizing distances between a probe
> model and a gallery model. However the performance of ICP relies heavily on
> the initial conditions. Hence, it is necessary to provide an initial
> registration, which will be improved iteratively and finally converge to the
> best alignment possible. Once the faces are registered, the occlusions are
> automatically extracted by thresholding the depth map values of the 3D
> image. After the occluded regions are detected, restoration is done by
> Principal Component Analysis (PCA). The restored images, after the removal
> of occlusions, are then fed to the recognition system for classification
> purpose. Features are extracted from the reconstructed non-occluded face
> images in the form of face normals. The experimental results which were
> obtained on the occluded facial images from the Bosphorus 3D face database,
> illustrate that our occlusion compensation scheme has attained a recognition
> accuracy of 91.30
>
> Explain whether the abstract satisfies the user requirement. Think step by
> step, showing all of your reasoning. Use the following *rules* to determine
> whether the requirement is satisfied: disagree means the abstract does not
> address the user requirement; dispute means the abstract does not directly
> address the user requirement, but is adaptable to fulfill the user’s need;
> concur means the abstract directly and completely addresses the user
> requirement; agree means the abstract directly addresses only a portion of
> the user requirement, but not in its entirety. Think carefully about the
> aforementioned *rules* to decide whether you [DISAGREE, DISPUTE, AGREE,
> CONCUR] based on the definition of these 4 options. After you have provided
> your reasoning, state your final selection in *all capital* letters at the
> *end* of your response.

ChatGPT’s Reasoning:

> The abstract DISAGREES with the user requirement. The paper proposes a 3D
> face recognition system that can handle pose and occlusions in real-world
> scenarios. The system takes a 3D range image as input, registers it using
> the ICP algorithm, and extracts occlusions by thresholding the depth map
> values of the 3D image. However, the paper does not introduce a model that
> can remove stripe patterns from human faces. Therefore, the abstract does
> not address the user requirement.

##### Error Analysis:

The human annotators labeled this question as "Dispute" due to the mention of
occlusion detection, removal, and restoration in the abstract, inferring that
"stripe patterns" could fall under the broader category of "occlusions".
However, ChatGPT did not make the same inference, indicating a different style
of reasoning.

####  B.4.3 Key Component Inconsistency

Test Case Prompt:

> You are a reference librarian. A user wants to find a scientific paper that
> satisfies a particular requirement. Based on the abstract of the paper, you
> will determine whether the requirement is met.
>
> Here is the user requirement: The navigation agent model in the document
> should have the ability to be trained on data with varying levels of
> augmentation
>
> Here is the abstract: Localization is the problem of estimating the location
> of an autonomous agent from an observation and a map of the environment.
> Traditional methods of localization, which filter the belief based on the
> observations, are sub-optimal in the number of steps required, as they do
> not decide the actions taken by the agent. We propose "Active Neural
> Localizer", a fully differentiable neural network that learns to localize
> accurately and efficiently. The proposed model incorporates ideas of
> traditional filtering-based localization methods, by using a structured
> belief of the state with multiplicative interactions to propagate belief,
> and combines it with a policy model to localize accurately while minimizing
> the number of steps required for localization. Active Neural Localizer is
> trained end-to-end with reinforcement learning. We use a variety of
> simulation environments for our experiments which include random 2D mazes,
> random mazes in the Doom game engine and a photo-realistic environment in
> the Unreal game engine. The results on the 2D environments show the
> effectiveness of the learned policy in an idealistic setting while results
> on the 3D environments demonstrate the model’s capability of learning the
> policy and perceptual model jointly from raw-pixel based RGB observations.
> We also show that a model trained on random textures in the Doom environment
> generalizes well to a photo-realistic office space environment in the Unreal
> engine.
>
> Explain whether the abstract satisfies the user requirement. Think step by
> step, showing all of your reasoning. Use the following *rules* to determine
> whether the requirement is satisfied: disagree means the abstract does not
> address the user requirement; dispute means the abstract does not directly
> address the user requirement, but is adaptable to fulfill the user’s need;
> concur means the abstract directly and completely addresses the user
> requirement; agree means the abstract directly addresses only a portion of
> the user requirement, but not in its entirety. Think carefully about the
> aforementioned *rules* to decide whether you [DISAGREE, DISPUTE, AGREE,
> CONCUR] based on the definition of these 4 options. After you have provided
> your reasoning, state your final selection in *all capital* letters at the
> *end* of your response.

ChatGPT’s Reasoning:

> The abstract does not directly address the user requirement. The abstract
> discusses a neural network model for localization of an autonomous agent in
> various environments. The model is trained with reinforcement learning and
> is able to learn the policy and perceptual model jointly from raw-pixel
> based RGB observations. However, there is no mention of the navigation agent
> model being trained on data with varying levels of augmentation. Therefore,
> I DISAGREE that the abstract satisfies the user requirement.

##### Error Analysis:

Human annotator 1 assigned the label "Dispute" and human annotator 2 assigned
the label "Disagree", while ChatGPT assigned the label "Disagree". The
difference between ChatGPT and human annotator 1 in this case comes from their
different definition of key components in the aspect. Human annotator 1
believes the navigation agent model itself is a key component, thus by
mentioning the localization problem of an autonomous agent, the abstract
already indirectly implies the navigation agent model. However, both ChatGPT
and annotator 2 believes navigation agent that can be trained on data with
varying levels of augmentation is the only key component in the aspect.
Therefore, since data with varying level of augmentation is not mentioned in
the abstract, both of them assigned "Disagree" indicating the abstract is
unrelated to the only key component in the aspect. This form of inconsistency
is the most common form of inconsistencies we found in ChatGPT’s reasoning
versus some human’s reasoning. However, since humans also disagree on what
constitutes as key components occasionally, this form of key component
inconsistency cannot be completely avoided even when ChatGPT exhibits
comparable performance as humans.

###  B.5 Annotation Scalability

The Anno-GPT framework, as detailed in Section 4, employs ChatGPT for
automated annotations, contingent on its performance being comparable to human
experts during the hypothesis testing stage. This system scales linearly with
the number of complex queries, aspects/sub-aspects, and the size of candidate
pools per query, enabling larger datasets for training and testing purposes.
For instance, with ten API keys, a multi-threaded program could create a
training-sized dataset of 10,000 complex queries, each with a candidate pool
of 100 abstracts, in approximately 240 hours.

Creating a DORIS-MAE dataset for another field, such as quantum computing or
biomedical engineering, involves several preparatory stages. Initial complex
queries (e.g., 50 queries) need to be either expertly crafted or gathered from
online technical discussions in the field. Next, the complex queries must be
broken down into a list of aspects and sub-aspects. This process could be
carried out by an expert in the field or a proficient annotator.

Finally, it is critical to fine-tune the final prompt and verify ChatGPT’s
competence in the new field of research. As per our annotation guidelines,
this stage involves three expert annotators, each dedicating an average of 20
hours. This step constitutes the largest fixed cost of applying Anno-GPT to a
new field, but it’s a necessary investment for ensuring dataset quality.
However, once a language model such as ChatGPT is confirmed to perform at a
comparable level to human experts, the Anno-GPT framework can then be used to
create larger datasets in the domain.

##  Appendix C Retrieval Results

This section presents the full results of benchmarking sixteen models (ada-002
[28], E5-Large-v2 [74], LLAMA [71], Text-Supervised (TS)-ASPIRE [54], Optimal-
Transport (OT)-ASPIRE [54], SPECTER [18], Sentence-BERT [62], RocketQA-v2
[63], ANCE FirstP [85], SimLM [75], SPLADE-v2 [23], ColBERT-v2 [66], SciBERT
[5], ERNIE-Search [47], BM25 [72], TF-IDF [67]) discussed in Section 2 on the
DORIS-MAE dataset. For SciBERT and SPECTER, we trained them on our 360k CS
corpus with citation signals (i.e. in domain) and denoted their best trained
version as SciBERTID and SPECTERID.

###  C.1 Metrics

In our experiments, we used several metrics which are commonly used in the
IR/NIR literature for evaluating ranking and retrieval models. These include
Recall@5 (R@5), Recall@20 (R@20), R-Precision (RP), NDCG10%,
$\textrm{NDCG}^{\textrm{exp}}_{10\%}$ [12], Mean Reciprocal Rank@10 (MRR@10),
and Mean Average Precision (MAP). An abstract was categorized as relevant when
its score from Equation 1 $\frac{S(p_{j}|Q)}{|Q|}\geq 1$. Note that NDCG10% is
excluded from the main paper due to an atypically high random baseline value
of 48.71%.

###  C.2 Query Level Benchmark Results

####  C.2.1 Whole Query Embedding

In Table 11, we showcase the results obtained when models (ada-002,
E5-Large-v2, LLAMA, SPECTER, SentBERT, RocketQA-v2, ANCE, SPLADE-v2,
ColBERT-v2, SciBERT, ERNIE, BM25, TF-IDF) embed the entire query as a single
vector and embed the entire abstract as a single vector. The relevance between
the query and the abstract is determined by calculating the cosine similarity
or L2 distance (with SPECTER using L2 distance as recommended by Cohan et al.
[18]) between the query vector and the abstract vector.

In Table 11, we see the general-purposed text embedding models (ada-002,
E5-Large-v2, LLAMA, SPECTER) (that are not restricted by the size of context
window) typically outperform specialized NIR models (RocketQA-v2, SimLM,
SPLADE-v2, ColBERT-v2). In-domain fine-tuning improves performance for SciBERT
but shows mixed results for SPECTER.

Table 11: Models’ query-level ranking performance (embedding query and abstract as single vectors) Method | R@5 | R@20 | RP |  NDCG10% | $\textrm{NDCG}^{\textrm{exp}}_{10\%}$ | MRR@10 | MAP
---|---|---|---|---|---|---|---
random | 4.41 | 18.48 | 16.29 | 48.71 | 7.31 | 3.59 | 19.63
ada-002 | 15.38 $\pm$ 1.95 | 42.84 $\pm$ 2.53 | 35.81 $\pm$ 2.67 |  72.48 $\pm$ 1.10 |  27.46 $\pm$ 2.48 |  19.88 $\pm$ 3.21 | 40.37 $\pm$ 2.55
E5-L-v2 |  16.51 $\pm$ 2.05 |  43.77 $\pm$ 2.14 |  37.46 $\pm$ 2.44 | 70.11 $\pm$ 0.97 | 25.90 $\pm$ 2.15 | 14.85 $\pm$ 2.73 |  40.49 $\pm$ 2.32
LLAMA | 12.74 $\pm$ 1.82 | 34.51 $\pm$ 2.36 | 28.33 $\pm$ 2.14 | 64.75 $\pm$ 1.04 | 16.65 $\pm$ 1.68 | 11.78 $\pm$ 2.45 | 31.29 $\pm$ 1.99
SPLADE-v2[25] | 14.78 $\pm$ 1.89 | 40.14 $\pm$ 2.33 | 31.65 $\pm$ 2.38 | 71.29 $\pm$ 0.89 | 26.08 $\pm$ 2.00 | 17.82 $\pm$ 2.99 | 37.23 $\pm$ 2.26
SPECTER-v2 | 14.50 $\pm$ 2.15 | 43.36 $\pm$ 2.50 | 33.41 $\pm$ 2.33 | 70.19 $\pm$ 0.98 | 25.65 $\pm$ 2.23 | 17.19 $\pm$ 2.96 | 37.12 $\pm$ 2.10
SPECTER | 13.34 $\pm$ 1.92 | 40.25 $\pm$ 2.60 | 32.66 $\pm$ 2.55 | 68.06 $\pm$ 1.15 | 24.43 $\pm$ 2.35 | 18.23 $\pm$ 2.99 | 35.75 $\pm$ 2.46
SPECTERID | 13.32 $\pm$ 1.76 | 42.52 $\pm$ 2.37 | 31.55 $\pm$ 2.28 | 69.34 $\pm$ 1.09 | 21.27 $\pm$ 2.03 | 14.48 $\pm$ 2.78 | 36.02 $\pm$ 2.19
SentBERT | 14.39 $\pm$ 1.97 | 42.42 $\pm$ 2.45 | 32.63 $\pm$ 2.60 | 69.70 $\pm$ 1.11 | 22.94 $\pm$ 2.21 | 13.28 $\pm$ 2.71 | 37.47 $\pm$ 2.34
RocketQA-v2 | 10.59 $\pm$ 1.20 | 33.72 $\pm$ 2.21 | 28.42 $\pm$ 2.18 | 66.17 $\pm$ 1.17 | 18.57 $\pm$ 2.01 | 11.51 $\pm$ 2.49 | 32.57 $\pm$ 2.14
ANCEFirstP | 11.10 $\pm$ 1.53 | 32.09 $\pm$ 2.09 | 28.60 $\pm$ 2.14 | 64.81 $\pm$ 0.99 | 17.18 $\pm$ 1.74 | 9.07 $\pm$ 1.93 | 30.72 $\pm$ 2.02
SimLM | 12.68 $\pm$ 1.77 | 35.67 $\pm$ 2.49 | 28.90 $\pm$ 2.42 | 66.31 $\pm$ 1.10 | 18.91 $\pm$ 1.86 | 11.29 $\pm$ 2.44 | 33.06 $\pm$ 2.34
SimCSE | 14.90 $\pm$ 1.89 | 42.62 $\pm$ 2.40 | 35.27 $\pm$ 2.34 | 70.18 $\pm$ 1.01 | 26.88 $\pm$ 2.36 | 21.19 $\pm$ 3.47 | 39.02 $\pm$ 2.35
ColBERT-v2 | 8.45 $\pm$ 1.46 | 27.86 $\pm$ 2.29 | 22.33 $\pm$ 2.01 | 59.81 $\pm$ 1.16 | 12.57 $\pm$ 1.71 | 6.69 $\pm$ 2.15 | 25.80 $\pm$ 1.83
SciBERT | 5.13 $\pm$ 1.25 | 17.99 $\pm$ 1.69 | 17.13 $\pm$ 1.88 | 47.28 $\pm$ 1.17 | 7.50 $\pm$ 1.34 | 3.41 $\pm$ 1.57 | 20.34 $\pm$ 1.64
SciBERTID | 12.86 $\pm$ 1.86 | 35.53 $\pm$ 2.23 | 27.54 $\pm$ 2.29 | 64.55 $\pm$ 1.17 | 17.24 $\pm$ 1.64 | 8.57 $\pm$ 1.79 | 31.50 $\pm$ 2.02
ERNIE | 6.49 $\pm$ 0.94 | 22.58 $\pm$ 1.72 | 20.18 $\pm$ 1.82 | 54.91 $\pm$ 1.17 | 9.66 $\pm$ 1.18 | 3.77 $\pm$ 1.06 | 22.71 $\pm$ 1.65
BM25 | 8.47 $\pm$ 1.80 | 30.50 $\pm$ 2.38 | 21.94 $\pm$ 2.03 | 51.46 $\pm$ 1.40 | 13.23 $\pm$ 1.97 | 9.19 $\pm$ 2.46 | 25.99 $\pm$ 1.68
TF-IDF | 10.71 $\pm$ 1.48 | 29.22 $\pm$ 2.25 | 24.79 $\pm$ 2.06 | 58.31 $\pm$ 1.12 | 18.25 $\pm$ 2.01 | 12.41 $\pm$ 2.53 | 28.77 $\pm$ 1.81

####  C.2.2 Sentence-by-Sentence Embedding

For fairness, we adopted an alternative approach for models like RocketQA-v2
[63] and ColBERT-v2 [66] that are trained on shorter queries such as in MS
MARCO [57] and NQ [40]. In Table 12, we show the results when these models
process each sentence in the query independently. Furthermore, we also note
that TSAspire and OTAspire by their design can only encode both the query and
the abstract into multiple vectors, as they utilize contextualized sentence
embedding vectors for "aspects". Note, they do not process each sentence
independently. These two models employ the distance function described in [54]
to assess the relationship between the query vectors and the abstract vectors.

To calculate relevance between the query $Q$, which is encoded into a list of
vectors $\\{\vec{Q_{1}},\dots,\vec{Q_{n}}\\}$ and the abstract $p$, which is
encoded into a list of vectors $\\{\vec{p_{1}},\dots,\vec{p_{m}}\\}$, we
employ the standard max-sim operation used in [35], specified by Equation 4.
We report our results in Table 12.

$\textrm{Rel}(p|Q)=\frac{1}{n}\sum_{i=1}^{n}\max_{1\leq j\leq
m}\frac{<\vec{Q_{i}},\vec{p_{j}}>}{||\vec{Q_{i}}||\cdot||\vec{p_{j}}||}$ (4)

We can see RocketQA-v2 benefited the most from processing the text sentence by
sentence, reaching the highest recorded performance across 6 metrics so far,
which suggests a potential reason for the weakness of other NIR models, that
they are unsuitable to process longer and more complex queries.

As mentioned before, in Table 2 in the main paper, we took the best recorded
metrics for each model over these two options (Table 11 and Table 12).

Table 12: Models’ query-level ranking performance (embedding query and abstract into multiple vectors) Method | R@5 | R@20 | RP |  NDCG10% | $\textrm{NDCG}^{\textrm{exp}}_{10\%}$ | MRR@10 | MAP
---|---|---|---|---|---|---|---
random | 4.41 | 18.48 | 16.29 | 48.72 | 7.31 | 3.59 | 19.63
SentBERT | 14.16 $\pm$ 1.95 | 44.81 $\pm$ 2.54 | 33.84 $\pm$ 2.50 | 69.58 $\pm$ 0.96 | 22.03 $\pm$ 2.06 | 13.39 $\pm$ 2.63 | 37.89 $\pm$ 2.34
RocketQA |  15.55 $\pm$ 1.93 |  45.38 $\pm$ 2.43 |  34.43 $\pm$ 2.39 |  73.07 $\pm$ 1.02 |  30.27 $\pm$ 2.29 |  20.79 $\pm$ 3.06 |  40.25 $\pm$ 2.28
SPLADE-v2 | 12.84 $\pm$ 1.83 | 37.86 $\pm$ 2.41 | 29.30 $\pm$ 2.40 | 68.06 $\pm$ 0.98 | 23.78 $\pm$ 2.13 | 16.53 $\pm$ 2.71 | 34.67 $\pm$ 2.16
TSAspire | 14.26 $\pm$ 1.80 | 41.25 $\pm$ 2.40 | 33.81 $\pm$ 2.47 | 71.63 $\pm$ 1.11 | 26.63 $\pm$ 2.05 | 15.59 $\pm$ 2.59 | 37.00 $\pm$ 2.29
OTAspire | 13.34 $\pm$ 1.56 | 42.25 $\pm$ 2.53 | 33.63 $\pm$ 2.38 | 70.43 $\pm$ 1.08 | 25.52 $\pm$ 2.29 | 14.18 $\pm$ 2.66 | 36.70 $\pm$ 2.22
ANCE | 13.17 $\pm$ 1.89 | 34.55 $\pm$ 2.11 | 30.33 $\pm$ 2.50 | 66.83 $\pm$ 1.01 | 20.27 $\pm$ 1.91 | 13.83 $\pm$ 2.45 | 34.37 $\pm$ 2.33
SimLM | 13.77 $\pm$ 1.87 | 38.11 $\pm$ 2.26 | 30.17 $\pm$ 2.30 | 67.35 $\pm$ 1.14 | 23.61 $\pm$ 2.14 | 16.63 $\pm$ 2.88 | 34.94 $\pm$ 2.14
ColBERT-v2 | 9.19 $\pm$ 1.27 | 30.28 $\pm$ 2.03 | 23.77 $\pm$ 1.97 | 61.99 $\pm$ 1.05 | 13.79 $\pm$ 1.48 | 7.20 $\pm$ 2.13 | 28.36 $\pm$ 1.84
SciBERT | 5.37 $\pm$ 0.83 | 22.34 $\pm$ 2.01 | 19.44 $\pm$ 1.86 | 53.19 $\pm$ 1.24 | 8.01 $\pm$ 1.17 | 4.08 $\pm$ 1.38 | 22.54 $\pm$ 1.62
SciBERTID | 12.46 $\pm$ 1.88 | 35.40 $\pm$ 2.17 | 27.24 $\pm$ 2.27 | 64.67 $\pm$ 1.18 | 18.42 $\pm$ 2.08 | 12.74 $\pm$ 2.63 | 32.01 $\pm$ 2.03
ERNIE | 6.71 $\pm$ 1.39 | 24.24 $\pm$ 1.96 | 21.39 $\pm$ 2.08 | 54.27 $\pm$ 1.02 | 9.92 $\pm$ 1.50 | 6.05 $\pm$ 2.02 | 24.00 $\pm$ 1.81
SimCSE | 13.21 $\pm$ 1.86 | 35.83 $\pm$ 2.31 | 29.41 $\pm$ 2.23 | 65.20 $\pm$ 1.17 | 20.62 $\pm$ 1.84 | 13.59 $\pm$ 2.58 | 33.89 $\pm$ 2.17

###  C.3 Using Aspects for Retrieval

In Section 5.2, we explored using the hierarchical structure of aspects and
sub-aspects as an alternative to traditional queries. For this analysis, we
concatenated the aspects into a paragraph to compute query embeddings for
various models. However, this resulted in a verbose and repetitive paragraph,
which did not resemble a real query. Given that some methods are trained
specifically for sentence-level tasks, we offered the option of using a multi-
vector representation for the aspects, with one vector per aspect. Similar to
Sections C.2.1 and C.2.2, we considered two ways to process queries and
abstracts: Approach 1 concatenates aspects and computes a single embedding;
Approach 2 embeds aspects and abstracts sentence-by-sentence (this is denoted
with a * in the table). For clarity, we only record the best metrics for each
model between these two options in Table 13.

When we compared Table 13 with Tables 12 and 11, we did not observe any clear
effect of using aspects for retrieval. Sentence-BERT and SPLADE-v2 did achieve
stronger results in this setting, but the effect did not generalize to other
models.

Table 13: Models’ ranking performance using concatenated aspects as query Method | R@5 | R@20 | RP |  NDCG10% | $\textrm{NDCG}^{\textrm{exp}}_{10\%}$ | MRR@10 | MAP
---|---|---|---|---|---|---|---
random | 4.60 | 18.30 | 16.48 | 48.79 | 7.41 | 3.65 | 19.73
SentBERT* |  17.71 $\pm$ 2.44 |  45.26 $\pm$ 2.66 |  35.63 $\pm$ 2.48 | 70.47 $\pm$ 0.97 | 24.94 $\pm$ 2.19 | 15.54 $\pm$ 3.00 |  39.79 $\pm$ 2.42
RocketQA* | 13.79 $\pm$ 1.48 | 43.69 $\pm$ 2.38 | 32.63 $\pm$ 2.17 |  72.21 $\pm$ 0.93 |  27.39 $\pm$ 2.04 | 16.03 $\pm$ 2.52 | 37.90 $\pm$ 2.13
ANCE* | 15.35 $\pm$ 2.02 | 38.36 $\pm$ 2.29 | 29.45 $\pm$ 2.07 | 67.27 $\pm$ 1.08 | 20.46 $\pm$ 1.82 | 14.54 $\pm$ 2.75 | 33.59 $\pm$ 1.89
SPLADE-v2 | 15.07 $\pm$ 1.99 | 41.20 $\pm$ 2.42 | 34.26 $\pm$ 2.45 | 71.34 $\pm$ 0.91 | 24.91 $\pm$ 1.90 | 18.30 $\pm$ 3.00 | 38.72 $\pm$ 2.33
ColBERT-v2* | 7.37 $\pm$ 1.11 | 26.64 $\pm$ 1.94 | 22.28 $\pm$ 1.82 | 58.70 $\pm$ 1.14 | 12.17 $\pm$ 1.30 | 5.78 $\pm$ 1.65 | 25.24 $\pm$ 1.69
ERNIE* | 5.34 $\pm$ 1.23 | 19.96 $\pm$ 1.58 | 18.02 $\pm$ 1.64 | 50.33 $\pm$ 1.13 | 7.14 $\pm$ 1.03 | 3.14 $\pm$ 1.26 | 21.38 $\pm$ 1.50
SciBERT* | 5.41 $\pm$ 0.78 | 21.36 $\pm$ 1.86 | 18.50 $\pm$ 1.72 | 51.27 $\pm$ 1.15 | 6.84 $\pm$ 0.92 | 2.29 $\pm$ 1.04 | 21.66 $\pm$ 1.57
SciBERTID* | 8.88 $\pm$ 1.44 | 30.81 $\pm$ 2.15 | 22.56 $\pm$ 2.00 | 59.86 $\pm$ 1.01 | 13.82 $\pm$ 1.64 | 10.24 $\pm$ 2.28 | 26.53 $\pm$ 1.77
ada-002 | 14.12 $\pm$ 2.00 | 42.13 $\pm$ 2.73 | 33.60 $\pm$ 2.44 | 71.52 $\pm$ 0.98 | 26.57 $\pm$ 2.57 |  20.22 $\pm$ 3.49 | 37.63 $\pm$ 2.29
E5-L-v2 | 14.32 $\pm$ 1.97 | 41.55 $\pm$ 2.21 | 34.21 $\pm$ 2.54 | 69.37 $\pm$ 1.00 | 22.51 $\pm$ 2.16 | 13.87 $\pm$ 2.68 | 37.31 $\pm$ 2.33
TSAspire | 14.30 $\pm$ 1.80 | 42.75 $\pm$ 2.25 | 33.63 $\pm$ 2.45 | 70.96 $\pm$ 1.10 | 25.29 $\pm$ 2.14 | 16.26 $\pm$ 2.90 | 37.31 $\pm$ 2.23
OTAspire | 14.45 $\pm$ 1.81 | 40.99 $\pm$ 2.30 | 32.53 $\pm$ 2.43 | 69.74 $\pm$ 1.14 | 23.81 $\pm$ 2.07 | 15.38 $\pm$ 2.76 | 36.09 $\pm$ 2.18
SimLM | 16.18 $\pm$ 2.19 | 39.24 $\pm$ 2.36 | 32.20 $\pm$ 2.22 | 67.56 $\pm$ 1.03 | 23.36 $\pm$ 2.14 | 16.61 $\pm$ 2.84 | 36.24 $\pm$ 2.12
SimCSE | 12.41 $\pm$ 1.84 | 35.02 $\pm$ 2.52 | 27.67 $\pm$ 2.27 | 64.19 $\pm$ 1.24 | 20.52 $\pm$ 1.93 | 15.27 $\pm$ 2.57 | 30.83 $\pm$ 2.12
SPECTER | 11.56 $\pm$ 1.76 | 35.30 $\pm$ 2.34 | 28.43 $\pm$ 2.49 | 63.55 $\pm$ 1.30 | 19.69 $\pm$ 2.33 | 15.07 $\pm$ 3.12 | 31.39 $\pm$ 2.22
SPECTERID | 12.01 $\pm$ 1.77 | 35.79 $\pm$ 2.27 | 28.71 $\pm$ 2.17 | 66.22 $\pm$ 1.05 | 19.70 $\pm$ 2.04 | 13.44 $\pm$ 2.67 | 32.69 $\pm$ 1.97
SPECTER-v2 | 12.97 $\pm$ 1.78 | 41.45 $\pm$ 2.50 | 31.30 $\pm$ 2.35 | 68.37 $\pm$ 1.13 | 23.16 $\pm$ 2.20 | 14.59 $\pm$ 2.66 | 35.77 $\pm$ 2.09
BM25 | 6.25 $\pm$ 1.00 | 25.11 $\pm$ 2.09 | 20.51 $\pm$ 1.88 | 49.65 $\pm$ 1.31 | 10.33 $\pm$ 1.33 | 5.72 $\pm$ 1.62 | 23.74 $\pm$ 1.57
TF-IDF | 7.62 $\pm$ 1.40 | 23.16 $\pm$ 2.10 | 20.14 $\pm$ 2.09 | 50.73 $\pm$ 1.52 | 12.55 $\pm$ 1.81 | 7.72 $\pm$ 2.15 | 24.25 $\pm$ 1.91
* : Approach 2

###  C.4 Model Performance on DORIS-MAE Test Set

In Appendix section C.2, we evaluated zero-shot performance of embedding
models on the full DORIS-MAE dataset. We now present zero-shot results on the
DORIS-MAE test set, which consists of 60 queries. For these queries, we run
all IR models and show their performances in Table 14. All models use their
individual optimal embedding method (i.e. processing query/abstract sentence
by sentence or as an entire paragraph). Results from this table serve as a
baseline for future work that trains models on the DORIS-MAE training set.

Table 14: Models’ query-level ranking performance on DORIS-MAE test set Method | R@5 | R@20 | RP |  NDCG10% | $\textrm{NDCG}^{\textrm{exp}}_{10\%}$ | MRR@10 | MAP
---|---|---|---|---|---|---|---
random | 4.58 | 17.91 | 17.34 | 50.22 | 7.36 | 3.32 | 20.69
ada-002 |  15.80 $\pm$ 1.92 | 40.87 $\pm$ 3.09 | 37.64 $\pm$ 3.17 |  73.92 $\pm$ 1.29 | 28.47 $\pm$ 3.22 | 23.29 $\pm$ 4.52 |  41.72 $\pm$ 3.12
SimCSE | 14.40 $\pm$ 1.71 | 41.94 $\pm$ 3.01 | 36.29 $\pm$ 2.86 | 71.31 $\pm$ 1.25 | 29.08 $\pm$ 3.16 |  25.00 $\pm$ 4.85 | 39.65 $\pm$ 2.86
RocketQA | 13.26 $\pm$ 1.29 |  44.35 $\pm$ 2.80 | 35.32 $\pm$ 2.97 | 73.87 $\pm$ 1.26 |  29.83 $\pm$ 2.88 | 20.61 $\pm$ 4.01 | 40.66 $\pm$ 2.84
SPECTER-v2 | 13.33 $\pm$ 2.26 | 41.83 $\pm$ 3.07 | 35.98 $\pm$ 3.16 | 71.46 $\pm$ 1.15 | 27.67 $\pm$ 3.26 | 20.70 $\pm$ 4.47 | 38.46 $\pm$ 2.80
TSAspire | 13.31 $\pm$ 1.67 | 43.57 $\pm$ 2.97 | 36.51 $\pm$ 2.96 | 72.78 $\pm$ 1.26 | 27.72 $\pm$ 2.81 | 18.22 $\pm$ 3.81 | 38.50 $\pm$ 2.87
E5-L-v2 | 14.59 $\pm$ 1.79 | 42.06 $\pm$ 2.47 |  38.16 $\pm$ 2.93 | 71.96 $\pm$ 1.18 | 26.04 $\pm$ 2.76 | 14.25 $\pm$ 3.52 | 40.51 $\pm$ 2.87
OTAspire | 13.12 $\pm$ 1.61 | 42.65 $\pm$ 3.22 | 35.35 $\pm$ 2.97 | 71.01 $\pm$ 1.36 | 25.69 $\pm$ 2.72 | 15.38 $\pm$ 3.22 | 37.83 $\pm$ 2.86
SPLADE-v2 | 10.99 $\pm$ 1.18 | 36.21 $\pm$ 2.57 | 32.50 $\pm$ 2.96 | 71.01 $\pm$ 1.18 | 25.10 $\pm$ 2.90 | 19.44 $\pm$ 4.38 | 36.44 $\pm$ 2.81
SentBERT | 12.18 $\pm$ 1.72 | 43.23 $\pm$ 2.98 | 34.38 $\pm$ 3.07 | 69.63 $\pm$ 1.31 | 19.37 $\pm$ 2.47 | 10.84 $\pm$ 3.34 | 37.72 $\pm$ 2.93
SimLM | 10.49 $\pm$ 1.27 | 32.19 $\pm$ 2.72 | 29.73 $\pm$ 3.14 | 66.73 $\pm$ 1.36 | 17.50 $\pm$ 2.52 | 11.90 $\pm$ 3.66 | 32.28 $\pm$ 3.04
ANCEFirstP | 10.88 $\pm$ 1.83 | 32.03 $\pm$ 2.72 | 29.02 $\pm$ 3.04 | 66.48 $\pm$ 1.32 | 17.92 $\pm$ 2.28 | 11.11 $\pm$ 2.79 | 32.90 $\pm$ 2.83
LLAMA | 9.60 $\pm$ 1.36 | 32.34 $\pm$ 2.86 | 28.43 $\pm$ 2.69 | 65.15 $\pm$ 1.24 | 14.71 $\pm$ 1.84 | 9.36 $\pm$ 2.67 | 31.16 $\pm$ 2.60
TF-IDF | 9.53 $\pm$ 1.42 | 26.77 $\pm$ 2.51 | 25.27 $\pm$ 2.50 | 58.14 $\pm$ 1.59 | 17.03 $\pm$ 2.39 | 11.59 $\pm$ 3.36 | 28.56 $\pm$ 2.28
BM25 | 6.86 $\pm$ 1.52 | 26.43 $\pm$ 2.27 | 22.45 $\pm$ 2.51 | 51.55 $\pm$ 1.76 | 12.29 $\pm$ 2.55 | 8.65 $\pm$ 3.49 | 26.00 $\pm$ 2.16
ColBERTv2 | 6.49 $\pm$ 1.21 | 23.99 $\pm$ 2.78 | 21.58 $\pm$ 2.66 | 59.46 $\pm$ 1.64 | 12.92 $\pm$ 2.49 | 7.15 $\pm$ 2.97 | 25.24 $\pm$ 2.41
ERNIE | 7.43 $\pm$ 1.26 | 24.92 $\pm$ 2.23 | 22.43 $\pm$ 2.50 | 57.52 $\pm$ 1.50 | 10.83 $\pm$ 1.62 | 3.64 $\pm$ 1.28 | 24.94 $\pm$ 2.24
SciBERT | 4.57 $\pm$ 1.08 | 17.35 $\pm$ 1.84 | 16.83 $\pm$ 2.27 | 49.42 $\pm$ 1.56 | 7.13 $\pm$ 1.19 | 2.00 $\pm$ 1.06 | 20.61 $\pm$ 1.91

###  C.5 Two-Aspect Sub-Query Level Benchmark Results

The dataset’s hierarchical structure provides an opportunity to formulate
simpler tasks involving only parts of a complex query. By extracting parts
corresponding to two aspects, we generated over 1000 test cases, significantly
increasing the number of relevant abstracts as outlined in Section A.5. The
full results for this task are given in Table 15. As the length of these sub-
queries is similar to a long sentence, we did not differentiate between
embedding the entire query as a single vector or multiple vectors sentence-by-
sentence. Except for OTAspire and TSAspire, we applied a single-vector
embedding for all other models, including Sentence-BERT and ANCE.

We observed that ada-002 outperformed other models across all categories. All
models exhibited slight increases in R-Precision,
$\textrm{NDCG}^{\textrm{exp}}_{10\%}$, and MAP. These minor improvements
correlate with similar increases in their random baselines, suggesting that
they might be due to shifts in the random baseline rather than the models’
retrieval capabilities. Meanwhile, the small decreases in the random baselines
of R@5 and R@20 indicate a drop in the retrieval models’ Recall@k performance
on the sub-query level. This corresponds to an increase in the number of
relevant abstracts per sub-query as shown in Table 10. As Recall@k is
inversely related to the number of relevant abstracts per sub-query, and as
the retrieval models do not appear to effectively retrieve relevant abstracts
for these simpler sub-queries, it is unsurprising that their R@5 and R@20
metrics dropped substantially.

Table 15: Models’ ranking performance on 2-aspect sub-queries Method | R@5 | R@20 | RP |  NDCG10% | $\textrm{NDCG}^{\textrm{exp}}_{10\%}$ | MRR@10 | MAP
---|---|---|---|---|---|---|---
random | 4.60 | 18.76 | 21.55 | 41.10 | 15.06 | 5.92 | 24.71
ada-002 |  13.47 $\pm$ 0.47 | 40.22 $\pm$ 0.64 | 47.34 $\pm$ 0.77 |  69.78 $\pm$ 0.39 |  38.97 $\pm$ 0.64 |  24.29 $\pm$ 1.07 |  51.65 $\pm$ 0.77
LLAMA | 8.06 $\pm$ 0.31 | 29.77 $\pm$ 0.61 | 36.49 $\pm$ 0.69 | 56.85 $\pm$ 0.44 | 23.59 $\pm$ 0.50 | 9.80 $\pm$ 0.66 | 39.01 $\pm$ 0.65
E5-L-v2 | 11.62 $\pm$ 0.37 | 38.26 $\pm$ 0.58 | 45.11 $\pm$ 0.78 | 65.82 $\pm$ 0.42 | 35.12 $\pm$ 0.59 | 21.00 $\pm$ 0.95 | 48.73 $\pm$ 0.76
TSAspire | 13.15 $\pm$ 0.69 | 43.47 $\pm$ 1.04 | 41.46 $\pm$ 1.24 | 65.54 $\pm$ 0.71 | 36.31 $\pm$ 0.97 | 23.54 $\pm$ 1.58 | 45.85 $\pm$ 1.20
OTAspire | 12.56 $\pm$ 0.70 |  43.66 $\pm$ 1.09 | 40.60 $\pm$ 1.23 | 65.06 $\pm$ 0.69 | 35.54 $\pm$ 0.91 | 22.71 $\pm$ 1.49 | 45.53 $\pm$ 1.16
SPECTER | 10.51 $\pm$ 0.85 | 35.99 $\pm$ 1.07 | 35.40 $\pm$ 1.13 | 55.71 $\pm$ 0.67 | 26.26 $\pm$ 0.95 | 14.06 $\pm$ 1.32 | 38.27 $\pm$ 1.05
SPECTER-v2 | 11.14 $\pm$ 0.41 | 36.09 $\pm$ 0.60 | 43.73 $\pm$ 0.80 | 64.80 $\pm$ 0.41 | 32.68 $\pm$ 0.55 | 18.02 $\pm$ 0.90 | 47.26 $\pm$ 0.79
SPECTERID | 8.05 $\pm$ 0.31 | 30.97 $\pm$ 0.56 | 38.11 $\pm$ 0.77 | 57.19 $\pm$ 0.43 | 23.96 $\pm$ 0.45 | 10.09 $\pm$ 0.64 | 40.23 $\pm$ 0.74
SentBERT | 12.79 $\pm$ 0.42 | 40.31 $\pm$ 0.62 |  48.34 $\pm$ 0.80 | 69.38 $\pm$ 0.42 | 38.52 $\pm$ 0.61 | 22.98 $\pm$ 1.00 | 52.34 $\pm$ 0.80
RocketQA | 13.37 $\pm$ 0.61 | 42.42 $\pm$ 1.00 | 42.65 $\pm$ 1.19 | 67.20 $\pm$ 0.70 | 37.33 $\pm$ 1.04 | 24.19 $\pm$ 1.65 | 46.39 $\pm$ 1.21
ANCEFirstP | 11.23 $\pm$ 0.43 | 35.12 $\pm$ 0.58 | 42.49 $\pm$ 0.77 | 64.00 $\pm$ 0.43 | 31.64 $\pm$ 0.62 | 17.70 $\pm$ 0.96 | 45.78 $\pm$ 0.77
SimLM | 12.61 $\pm$ 0.70 | 37.73 $\pm$ 1.02 | 37.90 $\pm$ 1.08 | 63.18 $\pm$ 0.69 | 31.59 $\pm$ 0.95 | 19.18 $\pm$ 1.46 | 41.82 $\pm$ 1.10
SPLADE-v2 | 12.95 $\pm$ 0.46 | 39.55 $\pm$ 0.62 | 46.63 $\pm$ 0.75 | 68.26 $\pm$ 0.39 | 37.33 $\pm$ 0.60 | 22.80 $\pm$ 1.00 | 50.71 $\pm$ 0.75
ColBERT-v2 | 8.13 $\pm$ 0.50 | 27.22 $\pm$ 0.87 | 28.38 $\pm$ 1.02 | 53.32 $\pm$ 0.71 | 22.72 $\pm$ 0.79 | 11.84 $\pm$ 1.19 | 32.03 $\pm$ 0.98
ERNIE | 3.25 $\pm$ 0.21 | 15.96 $\pm$ 0.53 | 20.05 $\pm$ 0.87 | 39.26 $\pm$ 0.69 | 12.83 $\pm$ 0.56 | 3.29 $\pm$ 0.66 | 23.27 $\pm$ 0.79
SciBERT | 2.43 $\pm$ 0.20 | 14.59 $\pm$ 0.71 | 17.56 $\pm$ 0.78 | 33.29 $\pm$ 0.58 | 9.65 $\pm$ 0.42 | 1.35 $\pm$ 0.24 | 21.23 $\pm$ 0.72
SciBERTID | 9.41 $\pm$ 0.35 | 32.43 $\pm$ 0.56 | 39.28 $\pm$ 0.72 | 60.19 $\pm$ 0.42 | 27.91 $\pm$ 0.51 | 13.63 $\pm$ 0.80 | 42.19 $\pm$ 0.70
BM25 | 9.14 $\pm$ 0.62 | 31.68 $\pm$ 0.99 | 31.22 $\pm$ 1.00 | 51.05 $\pm$ 0.71 | 25.52 $\pm$ 0.84 | 14.74 $\pm$ 1.18 | 33.94 $\pm$ 0.94
TF-IDF | 9.69 $\pm$ 0.58 | 32.56 $\pm$ 0.93 | 33.32 $\pm$ 1.05 | 56.58 $\pm$ 0.67 | 28.79 $\pm$ 0.91 | 19.57 $\pm$ 1.52 | 36.41 $\pm$ 0.97
SimCSE | 10.98 $\pm$ 0.38 | 34.57 $\pm$ 0.58 | 41.40 $\pm$ 0.74 | 63.00 $\pm$ 0.43 | 31.19 $\pm$ 0.56 | 17.22 $\pm$ 0.90 | 44.96 $\pm$ 0.74

###  C.6 Candidate Pool Sensitivity Analyses

In this section, we perform a sensitivity analysis on the candidate pools in
DORIS-MAE. The original candidate pools were primarily constructed using
keyword-based methods. To evaluate how this may have influenced our results,
we generated a second set of candidate pools for the first 50 queries in
DORIS-MAE. These new "embedding-based candidate pools" were generated with two
text-embedding models, E5-v2 and SPECTER-v2, which were not used during the
original candidate pool creation.

For each query, we collect the top-ranking 150 abstracts that are not in the
query’s original candidate pool (75 from E5-v2 and 75 from SPECTER-v2). We
applied the same LLM annotation procedure to estimate the relevance of each
abstract in the candidate pool given the query.

Relevance Analysis: We first evaluated whether the new pool contains documents
which are more relevant than those in the original pool. For each of the 50
queries, and for each candidate pool (i.e. the original pool and the extended
embedding pool), we record the highest relevance score. For each query, we
then compare the highest relevance score from each pooling method. On average,
the most relevant documents in the original pool scored higher than those in