1 billion pages. More detailed statistics are reported in Table 4.3. IQAD
contains 2 different test sets that differ in the period in which data were
gathered and in the annotation procedure. The first set contains 2.2K
questions, with about 17 answer candidates per question, annotated manually by
crowd-workers. The second test split contains 2K questions instead and about
16 answer candidates per question annotated by crowd-workers following strict
fact verification guidelines. A manual analysis of a subsample of 100
questions for each dataset showed higher annotation quality in the second set.
In the experimental Sections, we refer to the first test set as “IQAD Bench 1”
while the second is named “IQAD Bench 2”. In summary, IQAD is a very
challenging dataset because it reflects real user questions, which may be
vague, ambiguous or malformed.

###### WQA

WQA (Web Question Answering) is yet another AS2 dataset created sampling
questions asked to the Alexa virtual assistant in 2019 [Zhang et al., 2021;
Hsu et al., 2021]. Questions are non-representative and are anonymized to
remove connections to the original users and geographical locations. Moreover,
questions are not filtered by topic, so they span several different arguments.
For each question, 500 pages were extracted from an index of more than 100M
documents crawled from the web using ElasticSearch. Then, answer candidates
has been ranked with a state-of-the-art AS2 system and the top 100 were
annotated by crowd-workers. Statistics for this dataset are given in Table
4.3. Similarly to IQAD, WQA is a challenging corpus because questions mirror
real Alexa customer queries, which usually are noisy, malformed or misleading.

Dataset | Train set | Development set | Test set
---|---|---|---
Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q
IQAD | 221334 | 3894129 | 17.6 | - | 2434 | 43369 | 17.8 | - | 2252 | 38587 | 17.1 | -
2088 | 33498 | 16.0 | -
WQA | 9984 | 149513 | 15.0 | 0.23 | 5000 | 74805 | 15.0 | 0.21 | 5000 | 74712 | 14.9 | 0.21

Table 4.3: Statistics for IQAD and WQA.

###### NewsAS2

NewsAS2555We release NewsAS2 here:
https://huggingface.co/datasets/lucadiliello/news_as2 is a dataset derived
from NewsQA [Trischler et al., 2017], which was originally designed for
Machine Reading.

NewsQA is a dataset composed of over 100K question-answer pairs generated by
crowd-workers. First, the authors extracted 12.7K articles from the CNN/Daily
Mail corpus with random selection. Those articles cover several topics, such
as politics and business. Then, crowd-workers have been divided into three
groups. The first group task was to generate questions by letting the workers
see only the article’s title and a short summary, thus promoting curiosity
about the article’s content. The second group was then asked to highlight
answers in the documents, if present, or to reject questions when nonsensical
or malformed. A positive label was assigned to answer on which at least two
crowd-workers agreed. Finally, to improve the dataset quality, the authors
asked the third group of workers to review the questions without an agreement
on the answer and to make the final decision, or to reject them.

We downloaded NewsQA from the MRQA [Fisch et al., 2019] competition
repository666https://github.com/mrqa/MRQA-Shared-Task-2019 and split the
development set in both dev and test because the original test set in not
labeled. The conversion from MR to Answer Sentence Selection has been
performed with the following algorithm. First, each sample $(q,{\bm{D}})$ from
the original dataset was transformed in a tuple
$(q,\\{s_{1},s_{2},\dots,s_{n}\\})$ by splitting the document ${\bm{D}}$ into
multiple sentences with the NLTK tokenizer [Bird et al., 2009]. Then, each
sentence $s_{i}$ was labeled as positive if it entirely contained one of the
original answer spans, as negative otherwise. NewsAS2 contains more than 70K
questions and 1.8M question-answer pairs. The number of questions without at
least a positive answer candidate is 1.5% in the development set and 1.1% for
the test set. We remove those questions from the corresponding sets (clean
setting). Statistics for NewsAS2 are reported in Table 4.4.

###### TriviaAS2

TriviaAS2777We release TriviaAS2 here:
https://huggingface.co/datasets/lucadiliello/trivia_as2 is a dataset for AS2
derived from TriviaQA [Joshi et al., 2017], which was originally designed for
Machine Reading.

TriviaQA is a challenging dataset for Reading Comprehension composed of
questions and answers retrieved from 14 trivia and quiz-league websites. In
particular, about 95K trivia questions authored by enthusiasts were extracted,
and for each question, an average of 6 documents were crawled using the Bing
search engine. Questions were filtered to have a minimum length of 4 tokens
while documents were filtered when containing keywords like trivia, question
and answer or when they contained ill-formatted text, such as PDFs. In
addition, for each query, the authors extracted an additional set of about 2
documents from Wikipedia by searching the entities contained in the questions.
Finally, documents were filtered when not containing the original answer to
promote learning with distant supervision.

As for NewsQA, we download TriviaQA from MRQA and split the development into
both development and test to have non-hidden annotations. The conversion from
MR to AS2 is performed the same algorithm used for NewsAS2. Statistics for the
resulting TriviaAS2 are given in Table 4.4.

Dataset | Train set | Development set | Test set
---|---|---|---
Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q | Q | QA | A/Q | +/Q
NewsAS2 | 71561 | 1840533 | 25.7 | 1.7 | 2070 | 50895 | 24.6 | 1.7 | 2059 | 50842 | 24.7 | 1.7
TriviaAS2 | 61688 | 1843349 | 29.9 | 1.7 | 1201 | 37470 | 31.2 | 5.3 | 1190 | 37691 | 31.7 | 5.4

Table 4.4: Statistics for NewsAS2 and TriviaAS2.

####  4.2.2 Fact Verification

Fact Verification is the task of predicting whether a claim is entailed,
neutral or contradicted by a series of evidence sentences. Given the nature of
this task, which requires the model to reason over multiple sentences, it is
well suited for our Multi-Sentence Inference architecture, see Section 6.

###### FEVER

FEVER [Thorne et al., 2018], an acronym for “Fact Extraction and
VERification”, is a challenging dataset composed of about 185K claims and 920K
evidence sentences extracted from Wikipedia.

The data has been generated with the following procedure. First, the initial
section of 50K popular pages from Wikipedia has been extracted and cleaned.
Then, a set of crowd-workers was tasked to generate claims containing a single
statement related to the page topic. To avoid the creation of too complex
claims, the crowd-workers were provided with a limited set of words collected
from linked pages that could be used for the generation. Moreover, crowd-
workers were allowed to modify existing claims, thus changing whether they
were supported by the available evidence or not.

In the second phase, crowd-workers were provided with claims and were asked to
find a set of evidence texts to support or refute the claim. If the claim
could not be supported or contradicted, the workers should have provided
evidence for their decision.

Since FEVER’s original task involves both Information Retrieval of the
evidence sentences and Recognizing Textual Entailment, we used the evidence
collected by Thorne et al. [2018] using a BERT-based DocIR888We release our
version of the FEVER dataset here:
https://huggingface.co/datasets/lucadiliello/fever. The final dataset
statistics are reported in Table 4.5. On average, for each claim, a set of 5
evidence candidates is available to predict the target label. To evaluate our
models, we submitted them to the FEVER shared task
website999https://competitions.codalab.org/competitions/18814 because labels
for the test set are hidden.

Dataset | Train set | Development set | Test set
---|---|---|---
C | E | E/C | C | E | E/C | C | E | E/C
FEVER | 145405 | 722473 | 4.97 | 19998 | 98915 | 4.94 | 19998 | 98839 | 4.94

Table 4.5: Statistics for the FEVER dataset. “C” and “E” are the total number
of claims and evidence sentences, while “E/C” is the average number of
evidence sentences per claim.

####  4.2.3 Summarization

The statistics for all datasets about Abstractive Summarization used as
benchmarks in this Thesis are reported in Table 4.6.

###### CNN/DailyMail

The CNN/DailyMail dataset is a widely used benchmark dataset for Abstractive
Summarization which was developed by Hermann et al. [2015]. The dataset is
composed of news articles scraped from the CNN and The Daily Mail news
websites between 2010 and 2015. Every article of the dataset is paired with
summaries that were written by humans, providing a concise and informative
representation of the article’s main points. The dataset covers a wide range
of topics, including politics, technology, science, sports, and entertainment.
The articles are written in English and have a mix of short and long
sentences, making the dataset challenging to summarize.

###### XSum

XSum is a large-scale dataset for text Summarization that was created by
Narayan et al. [2018]. The XSum dataset consists of approximately 227K news
articles and their corresponding summaries. The articles were sourced from a
wide range of news outlets, including The Guardian, The Independent, and The
Daily Mail and cover a variety of topics, including politics, business and
sports. The summaries in the XSum dataset are typically one or two sentences
in length and provide a concise description of the main points of the article.
The dataset was built with a two-stage process. First, a group of human
annotators wrote a summary for each article. Then, a second group of human
annotators reviewed the summaries and edited them to ensure they provided an
accurate and comprehensive summary of the article, thus ensuring high-quality
summaries.

###### Samsum

The Samsum dataset is a corpus for conversations Summarization that was
introduced by Liu [2019]. The dataset consists of over 11,000 conversations
between two participants with the scope of scheduling a meeting. Each
conversation contains a set of messages exchanged between the participants,
and the goal of the Summarization task is to produce a concise summary of the
conversation that captures its most important aspects.

Conversational Summarization poses several challenges that are not present in
traditional document Summarization. Conversations are often highly
contextualized and contain a large amount of implicit information, such as the
participants’ intentions, preferences, and expectations. As a result,
summarizing a conversation requires not only identifying important content but
also understanding the underlying discourse and social interactions.

In the Samsum dataset, a set of small summary sentences were produced by human
annotators. Each summary is intended to capture the key points of the
conversation and provide a concise overview of its content.

###### Gigaword

Gigaword is a large-scale text corpus for Abstractive Summarization created by
the Linguistic Data Consortium (LDC) [Graff et al., 2003]. The dataset
contains over one billion words and covers a wide range of topics and domains
over more than two decades of events, from 1994 to 2015. The documents are
news articles extracted from a variety of sources, such as the North American
News, the AQUAINT corpus and the Agence France-Presse. In the Gigaword
dataset, every article is paired with a very short summary that captures its
key points. These summaries are typically one to three sentences long and were
written by human editors.

Dataset | Set | Examples | Document Length | Summary Length
---|---|---|---|---
Mean | Std | Min | Max | Mean | Std | Min | Max
CNN/DailyMail | Train | 287113 | 4033.6 | 1954.3 | 48 | 15925 | 294.7 | 120.1 | 14 | 7388
Dev | 13368 | 3923.8 | 1992.3 | 245 | 11412 | 328.0 | 146.2 | 52 | 8541
Test | 11490 | 3967.0 | 2024.7 | 293 | 11991 | 311.9 | 128.0 | 51 | 3410
XSum | Train | 204017 | 2202.4 | 1795.5 | 15 | 174045 | 125.4 | 30.2 | 1 | 399
Dev | 11327 | 2174.7 | 1664.9 | 26 | 27345 | 125.5 | 30.4 | 5 | 500
Test | 11333 | 2214.5 | 1817.6 | 14 | 63937 | 125.3 | 30.5 | 17 | 410
Samsum | Train | 14732 | 511.1 | 402.6 | 0 | 5492 | 110.1 | 60.8 | 3 | 300
Dev | 819 | 499.3 | 403.3 | 57 | 2950 | 110.1 | 61.3 | 17 | 300
Test | 818 | 521.5 | 409.1 | 49 | 2793 | 108.7 | 58.4 | 17 | 300
Gigaword | Train | 3803957 | 181.5 | 47.3 | 35 | 636 | 51.0 | 14.7 | 6 | 265
Dev | 189651 | 181.5 | 47.3 | 40 | 520 | 51.8 | 14.4 | 7 | 158
Test | 1951 | 170.8 | 56.4 | 3 | 409 | 52.2 | 19.3 | 7 | 137

Table 4.6: Statistics for all Summarization datasets employed in this work. We
report the number of examples for each split and statistics about the length
of documents and summaries (in terms of characters, including whitespaces). As
a rule of thumb, the number of characters per token is $\sim 4$ for most
Transformer models.

###  4.3 Benchmark Suites

Dataset | Task type | Classes | Train set |  | Development set |  | Test set
---|---|---|---|---|---|---|---
Examples | Examples/Class (%) |  | Examples | Examples/Class (%) |  | Examples
CoLA | Classification | 2 | 8551 | 29.5/70.5 |  | 1043 | 30.9/69.1 |  | 1063
MNLI-m | Classification | 3 | 392702 | 33.3/33.3/33.4 |  | 9815 | 35.5/31.8/32.7 |  | 9796
MRPC | Classification | 2 | 3668 | 32.6/67.4 |  | 408 | 31.6/68.4 |  | 1725
QNLI | Classification | 2 | 104743 | 50.0/50.0 |  | 5463 | 49.5/50.5 |  | 5463
QQP | Classification | 2 | 363846 | 63.1/36.9 |  | 40430 | 63.2/36.8 |  | 39095
RTE | Classification | 2 | 2490 | 50.2/49.8 |  | 277 | 52.7/47.3 |  | 3000
SST-2 | Classification | 2 | 67349 | 44.2/55.8 |  | 872 | 49.1/50.9 |  | 1821
STS-B | Regression | - | 5749 | - |  | 1500 | - |  | 1379

Table 4.7: Statistics for all datasets inside the GLUE benchmark suite.
“%Examples/Class” shows the percentage of examples for each label class. The
labels of the test set are hidden, thus we do not show statistics about their
class distribution.

###### GLUE

GLUE (General Language Understanding Evaluation) [Wang et al., 2018] is a
collection of datasets for testing systems on a variety of tasks. It includes
CoLA, a benchmark for Language Acceptability [Warstadt et al., 2019]; QNLI,
MNLI and WNLI, three datasets for Natural Language Inference [Wang et al.,
2018; Williams et al., 2018; Levesque et al., 2012]; RTE, to test performance
in Textual Entailment [Giampiccolo et al., 2007];
QQP101010https://quoradata.quora.com/First-Quora-Dataset-Release-Question-
Pairs and MRPC for paraphrasing [Dolan and Brockett, 2005]; STS-B for sentence
similarity [Cer et al., 2017]; SST-2, to test models in Sentiment Analysis
[Socher et al., 2013].

We do not include WNLI in our experiments because it is hard to beat even a
trivial majority classifier [Devlin et al., 2019]. As for MNLI, we use the
_matched_ version, usually indicated with MNLI-m. Statistics for each dataset
are reported in Table 4.7. A more fine-grained analysis of each dataset
contained in GLUE is given in Appendix B.1. Notice that recently, a new
version of this benchmark called SuperGLUE [Wang et al., 2019a] has been
released, and we encourage its usage for future research.

##  Chapter 5 Alternative Efficient Objectives

The first part of this work focuses on Transformers pre-training, which is a
core part of building Artificial General Intelligence. Transformers pre-
training is a task that requires (i) costly and large computational
infrastructures and a lot of energy, (ii) high-level skills to manage
distributed training and peer-to-peer efficient communications, (iii)
collection and cleaning/filtering of large amounts of raw data and (iv) well-
tested code implementations to avoid nodes failure [Strubell et al., 2019;
Brown et al., 2020].

In this Chapter, we address the first issue mentioned before by proposing new
pre-training tasks and objectives that require shorter pre-training to reach
the same level of final performance. The other issues are generally addressed
by large organizations which develop high-end frameworks to help researchers
and developers. We mention: (i) PyTorch [Paszke et al., 2019] and Tensorflow
[Abadi et al., 2015], which allow for fast ANN development and training
through automatic back-propagation; (ii) DeepSpeed [Rasley et al., 2020] and
ColossalAI [Bian et al., 2021] that increment training speed and reduce
inference time by sharding model’s weights and through efficient communication
protocols between nodes in distributed settings; (iii) CommonCrawl [Wang,
2023] and The Pile [Gao et al., 2020], which are large sources of cleaned text
freely available and (iv) PyTorch-Lightning [Falcon and The PyTorch Lightning
team, 2019] and HuggingFace [Wolf et al., 2020], which help developers by
proving frameworks to abstract from the underlying hardware and repositories
with thousands of ready-to-be-used models.

Many recent works focused on improving pre-training efficiency through the
implementation of innovative objectives or by improving the model architecture
[Lan et al., 2020; Sanh et al., 2020; Turc et al., 2019]. For example, the
authors of ELECTRA trained a BERT model as a discriminator instead of a
generator [Clark et al., 2020]. Instead of masking a random subset of input
tokens, ELECTRA exploits a small Language Model trained with MLM to create
challenging replacements. Then, the discriminator is tasked to predict which
tokens are original and which are fakes introduced by the smaller generator
network.

ELECTRA introduces many innovations, such as more efficient training and the
missing discrepancy between pre-training and fine-tuning because no special
[MASK] token is used in the discriminator training. However, it is not clear
whether the improvements of ELECTRA over BERT derive more from the generator-
discriminator architecture or the longer and more costly pre-training.

We provide an extensive description of the works related to this research in
Section 3.3.1. Notice that they are orthogonal to our proposed methodology,
which emphasizes the efficiency of the pre-training objective and the
classification head size. Therefore, they can be combined with our alternative
pre-training objectives. Additionally, works related to the models and the
tasks we use in this Chapter can be found in Section 3.1 and 3.2,
respectively.

We study a batch of new pre-training strategies designed to reduce training
costs while maintaining the same level of final performance. We provide a
thoughtful study of both theoretical cost and experimental training time since
the two do not always align well due to underlying hardware optimization
techniques and different levels of memory usage. Our contributions can the
summarized as the following three self-supervised objectives [Di Liello et
al., 2022a]:

  * •

Random Token Substitution (RTS): an efficient alternative to ELECTRA that
drops the expensive generator network and works by randomly replacing some
tokens with others and by tasking the model at recognizing which are originals
and which are not;

  * •

Cluster-based Random Token Substitution (C-RTS): a technique that improves
over RTS using previous predictions to find more challenging replacements
utilizing a simple and efficient statistical approach;

  * •

Swapped Language Modeling (SLM): an alternative to BERT’s MLM which randomly
replaces tokens into others and asks the model to predict the original values.

The next Sections will cover a description of our proposed objectives and a
comparison of the computational complexity with the SOTA pre-training tasks
for Transformers. Then, we provide details about the experiment setting we
used to evaluate our claims and results obtained over many different
benchmarks. We also report negative findings in Appendix C.2 to help future
research.

(a) Masked Language Modeling

(b) Token Detection

Figure 5.1: BERT’s MLM and ELECTRA’s TD objectives examples.

(a) Random Token Substitution

(b) Cluster-based Random Token Substitution

(c) Swapped Language Modeling

Figure 5.2: Examples for RTS, C-RTS and SLM objectives.

###  5.1 Alternative Objectives

In this Section, we illustrate our proposed objectives, highlighting the
advantages compared to state-of-the-art methods.

####  5.1.1 Random Token Substitution

RTS is a variant of the ELECTRA architecture in which the expensive generator
network is substituted with a straightforward random sampling approach.
Examples of how ELECTRA and RTS work are depicted in Figure 5.1(b) and 5.2(a),
respectively. RTS uses a very small classification head on top of the
Transformer for binary classification, which reduces computational load and
memory usage, allowing for larger batch sizes.

In RTS, we randomly select the 15% of the input tokens and we change them into
others sampled with uniform distribution from the vocabulary. Then, the model
is tasked to recognize which tokens are original and which have been replaced
with others.

###### Advantages

RTS necessitates a minimal classification head, and predictions are not
carried out across the entire vocabulary. Some might argue that removing the
MLM head would not result in significant improvements since the embedding
layer, which shares weights with the language modeling classification head,
still acts as a bottleneck. However, this is not accurate, as the embedding
layer functions by indexing the weights matrix to extract only the
representation corresponding to each input token. In contrast, the language
modeling head computes probabilities for all tokens in the vocabulary, making
it computationally much more expensive by several orders of magnitude.

Another important advantage of RTS is that there is no pre-training/fine-
tuning discrepancy because the [MASK] token is not used in both pre-training
and fine-tuning [Clark et al., 2020].

###### Drawbacks

RTS works well with Transformer models containing up to 100M parameters,
however, we do not suggest its usage for larger models because it may lead to
overfitting, as shown in the successive sections. When the model size
increases, the RTS task can be solved with very high accuracies because most
replaced tokens are easy to discover, leading to weak error signals and lower
model quality.

####  5.1.2 Cluster-based Random Token Substitution

C-RTS is an improvement of RTS that targets the overfitting issue by
exploiting the history of previous predictions to find more challenging
replacements. The main idea of C-RTS is to model the miss-classification
probability of the model for each token independently through a simple
statical approach.

More formally, given a Transformer model $\mathcal{M}$ and sequence of tokens
$[t_{1},t_{2},\dots,t_{L}]$ in input, $\mathcal{M}$ outputs a sequence of
predictions ${\bm{\mathrm{p}}}$ for the labels ${\bm{\mathrm{y}}}$, where
$y_{i}=0$ means that $t_{i}$ is original and $y_{i}=1$ indicates that $t_{i}$
was replaced with some other token $t_{i}^{\prime}$ ($t_{i}\rightarrow
t_{i}^{\prime}$). Our objective is to maximize:

$P(p_{i}=0\ |\ t_{i}\rightarrow t_{i}^{\prime})$ (5.1)

since we want to find challenging replacements that are hard to be detected by
$\mathcal{M}$.

Equation 5.1 could be estimated by counting the number of successes and
failures of the model $\mathcal{M}$ at recognizing false input tokens in the
previous training steps. An important difference between C-RTS and ELECTRA is
that we do statistics over individual tokens, while ELECTRA’s generator uses
the whole input context to find challenging alternatives.

###### Clustering

Since storing a matrix containing counts for every pair of tokens would lead
to a very large memory footprint (vocabulary sizes range from 30K [Devlin et
al., 2019] entries to over 120K [He et al., 2020]), we split the vocabulary
into $n$ clusters with similar size. We perform the clustering based on token
similarity. However, since we do not have an already-trained embedding layer
yet, we train a _word2vec_ [Mikolov et al., 2013] model to obtain vectorial
representations. We run word2vec on the same data that we use for pre-
training, to avoid any performance improvement stemming from the usage of
additional text.

In particular, we configure word2vec to use a context size of 2 words on both
sides of the target token and we set the embedding size to 300. We use the
PyTorch implementation of word2vec [Paszke et al., 2019] and the whole
training takes less than 10 minutes on our machine. Given the negligible
overhead of this step when compared to the pre-training time of Transformers,
we omit those 10 minutes from the final training-time comparisons.

Once word embeddings are computed, we used the K-means algorithm [Lloyd, 1982]
to group tokens into clusters. We implemented the script by exploiting the
implementation of K-means in the _scikit-learn_ [Pedregosa et al., 2011]
library. To choose the best number of clusters $n$, we first run the
clustering algorithm with 20 random restarts for each value of
$n\in\\{30,100,300,1000\\}$. This experiment took about 20 minutes on the
machine described in Section 1.2. Then, we search for the best number of
clusters by running a short pre-training of 5000 steps and by selecting the
value that gives the smallest accuracy of C-RTS on the pre-training dev set.

###### Statistic-based replacement

Once we partition the tokens of the vocabulary ${\mathbb{V}}$ into clusters
$\\{{\mathbb{C}}_{1},\dots,{\mathbb{C}}_{n}\\}$, we estimate the probability
$P(p_{i}=0\ |\ t_{i}\rightarrow t_{i}^{\prime})$ with the following
methodology. First, we use a matrix ${\bm{F}}\in{\mathbb{Z}}^{n\times n}$ to
count the number of successes and failures of the model. At each training
step, if $t_{i}\in{\mathbb{C}}_{a}\rightarrow
t_{i}^{\prime}\in{\mathbb{C}}_{b}$, then ${\bm{F}}_{a,b}$ is increased by 1 if
$p_{i}=0$ and decreased by 1 otherwise.

As in RTS, we select 15% of the input tokens for replacement. If token
$t_{i}\in{\mathbb{C}}_{a}$ is chosen, we maximize the miss-classification
probability by estimating $P(p_{i}=0\ |\ t_{i}\in{\mathbb{C}}_{a}\rightarrow
t_{i}^{\prime}\in{\mathbb{C}}_{b})$ with ${\bm{F}}$. Probability of replacing
$t_{i}$ to $t_{i}^{\prime}$ is computed with:

$P(t_{i}\in{\mathbb{C}}_{a}\rightarrow
t_{i}^{\prime}\in{\mathbb{C}}_{b})=P(t_{i})\ P(t^{\prime}_{i}\ |\
{\mathbb{C}}_{b})\ P({\mathbb{C}}_{b}\ |\ {\mathbb{C}}_{a})$ (5.2)

where:

  * •

$P(t_{i})$ is 15% as in ELECTRA’s TD or BERT’s MLM;

  * •

$P(t^{\prime}_{i}\ |\ {\mathbb{C}}_{b})$ is equal to
$\frac{1}{|{\mathbb{C}}_{b}|}$ because we pick a target token $t_{i}^{\prime}$
with uniform probability from ${\mathbb{C}}_{b}$;

  * •

$P({\mathbb{C}}_{b}\ |\ {\mathbb{C}}_{a})$ is computed taking advantage of
history about previous predictions, which are stored in the matrix ${\bm{F}}$.

In particular, the latter is computed by selecting the $a$-th row of
${\bm{F}}$ and by considering it as a multinomial distribution over target
clusters. To obtain values from ${\bm{F}}_{a}$ interpretable as probabilities,
we first apply the $min-max$ normalization:

${\bm{F}}_{a}^{\prime}=\frac{{\bm{F}}_{a}-\min({\bm{F}}_{a})}{\max({\bm{F}}_{a})-\min({\bm{F}}_{a})}$
(5.3)

to restrict the values into the $[0,1]$ interval. Then, we use _$\gamma$
-softmax_ function to smooth the distribution, which allows us to control the
temperature:

${\bm{F}}_{a}^{\prime\prime}=\frac{e^{{\bm{F}}_{a,b}^{\prime}/\gamma}}{\sum_{k=1}^{n}e^{{\bm{F}}_{a,k}^{\prime}/\gamma}}$
(5.4)

The final vector ${\bm{F}}_{a}^{\prime\prime}$ is then used as multinomial
distribution to estimate $P({\mathbb{C}}_{b}\ |\ {\mathbb{C}}_{a})$. An
example of C-RTS architecture is depicted in Figure 5.2(b).

####  5.1.3 Swapped Language Modeling

The SLM objective is computationally equivalent to MLM, yet it diverges in its
approach by omitting the use of a special [MASK] token, resulting in superior
performance. In the MLM task, 15% of the input tokens are selected and then
treated in three different ways: (i) in 80% of the cases, they are substituted
with the special [MASK] token, (ii) 10% of the time they are replaced by other
tokens, and (iii) 10% of the time they remain unchanged. SLM can be viewed as
a variation of BERT’s MLM, where tokens are consistently transformed into
others. The remaining aspects of the model’s architecture and loss functions
remain unchanged. Figure 5.2(c) provides an illustrative example of SLM.

###  5.2 Complexity Analysis

Objective | MLM | RTS | C-RTS | SLM | MLM + TD
---|---|---|---|---|---
LM head complexity | $O(|{\mathbb{V}}|\cdot d)$ | $O(d)$ | $O(d)$ | $O(|{\mathbb{V}}|\cdot d)$ |  $O(|{\mathbb{V}}|\cdot d)$ \+ $O(d)$

Table 5.1: Time complexity in number of FLOPS of the classification head
employed by different objectives.

In this Section, we analyze the space and time complexity of state-of-the-art
objectives for Transformers used in the literature and of our proposals. We
focus especially on the size of the language model head because the remaining
part of the Transformer is shared across all architectures (ours
comprehended). For more details about each objective, refer to Section
2.2.2.1. Table 5.1 summarizes the complexity of the language modeling head for
MLM, TD and our proposed objectives.

###### Masked Language Modeling & Swapper Language Modeling

MLM and SLM predict the original value or masked or replaced tokens. Figure
5.1(a) shows an example of input data and targets for MLM while in Figure
5.2(c) we depict an example of the SLM objective. Both objectives require a
classification head that spans the whole vocabulary ${\mathbb{V}}$. Given the
final hidden state ${\bm{\mathrm{h}}}_{i}\in{\mathbb{R}}^{d}$ in output from
the Transformer stack for each token $t_{i}$, the predictions for each class
are performed by multiplying ${\bm{\mathrm{h}}}_{i}$ with a learnable matrix
${\bm{W}}_{LM}\in{\mathbb{R}}^{|{\mathbb{V}}|\times d}$:

${\bm{\mathrm{p}}}_{i}=\text{softmax}\left({\bm{W}}_{LM}{\bm{\mathrm{h}}}_{i}\right)$
(5.5)

Thus, the computational complexity of the language modeling head for each
token is $O(|{\mathbb{V}}|\times d)$. We ignore the complexity of the softmax
because it is linear in the hidden size $d$.

###### Token Detection & Random Token Detection

ELECTRA’s architecture is composed of two blocks: a generator trained with MLM
and a discriminator trained with TD. The complexity of MLM is studied in the
previous Section. Token Detection is a binary prediction task, thus the
language modeling head is composed of a small matrix
${\bm{W}}_{LM}\in{\mathbb{R}}^{2\times d}$ followed by a softmax function. We
can derive that the complexity of TD is $O(d)$. RTS and C-RTS have the same
complexity as ELECTRA’s discriminator since they are similar but the generator
is substituted with our algorithms for selecting the token replacements.

###  5.3 Experiments

In this Section, we describe the testing environment we use to measure the
accuracy of the reference and proposed objectives. First, we provide details
about the pre-training setting, the datasets and the best hyper-parameters we
found for each objective. Then, fine-tuning details such as datasets, training
parameters and the hardware used to run the experiments are given. Finally, we
present results as tables and plots, including comparisons against models
released by the authors of BERT when available.

####  5.3.1 Architectures

Every objective has been tested by applying it to the same Transformer
architecture, equivalent to that of BERT [Devlin et al., 2019] or RoBERTa [Liu
et al., 2019b]. For every model and architecture, we use an uncased
vocabulary, which means WordPiece converts text to lowercase before
tokenizing. We experiment with models of two different sizes: Base and Small.

###### Base

The Base architecture is equivalent to that of BERTBase, which has a
vocabulary ${\mathbb{V}}$ containing 30522 tokens based on WordPiece (see
Paragraph A.3.2), a hidden size $d$ of 768, an intermediate size equal to
3072, 512 positional embeddings, 12 attention heads and 12 layers.

###### Small

The Small architecture we employ is similar to that of ELECTRASmall [Clark et
al., 2020], which in prior experiments resulted to be better than BERTSmall
[Turc et al., 2019]. ELECTRA demonstrated that it is more effective to reduce
the _width_ (the hidden size) of the Transformer instead of the _height_ (the
number of layers). We use a vocabulary ${\mathbb{V}}$ containing 30522 tokens
based again on WordPiece, a hidden size $d$ of 256, an intermediate size of
1024, 512 positional embeddings 12 attention layers and 4 attention heads.

Models | FLOPS | Time | Memory | # Parameters | Objective
---|---|---|---|---|---
Total | LM Head
BERTBase \+ MLM | $1.61\cdot 10^{19}$ | 3d 14h | 13.2GiB | 109M | 23M | MLM (1.0)
BERTBase \+ RTS | $1.54\cdot 10^{19}$ | 2d 22h | 9.9GiB | 109M | 1536 | RTS (50.0)
BERTBase \+ C-RTS | $1.54\cdot 10^{19}$ | 2d 22h | 9.9GiB | 109M | 1536 | C-RTS (50.0)
BERTBase \+ SLM | $1.61\cdot 10^{19}$ | 3d 14h | 13.2GiB | 109M | 23M | SLM (1.0)
ELECTRABase \+ MLM/TD | $1.98\cdot 10^{19}$ | 3d 18h | 18.0GiB | 143M | 23M + 1536 | MLM (1.0) + TD (50.0)

Table 5.2: FLOPS, time, memory, number of parameters and objectives to pre-
train Base models on two A100 GPUs of our machine. Notice that the number of
parameters of MLM/SLM and RTS/C-RTS is the same because the weights of the
embedding layer and language modeling head are shared. However, the memory
usage of MLM/SLM is higher because gradients over the language modeling head
are much larger in memory than gradients over the embedding layer. Notice also
that FLOPS and training time of C-RTS differ slightly from those reported in
Di Liello et al. [2022a] because we offloaded computations over clusters to
the CPU. Finally, we report the objectives along with their weight when
computing the total loss.

####  5.3.2 Pre-Training

In order to provide a meaningful and fair comparison between the objectives,
we use the same pre-training dataset for all experiments. In more detail, we
train all models on the same dump of the English Wikipedia and on the
BookCorpus, which are cleaned before training from HTML tags and tables, from
lists, indexes and everything not interpretable as raw text. Notice that the
original BookCorpus used by Devlin et al. [2019] is not available anymore, and
we do not know whether the version we use is of the same quality. For more
details about the pre-training corpora, see Section 4.1.

We copied the training set from Devlin et al. [2019], in which the authors
train the models for 900K steps with a maximum sequence length of 128 tokens
and 100K additional steps with a sequence length of 512. This allows us to
save a lot of pre-training time without a significant loss in performance. The
increased efficiency derives from the fact that the attention module
complexity is quadratic in the sequence length. Thus, for the first 900K
steps, the model is theoretically 4 times faster111Actual speed-up is
generally slightly lower because of the embedding layer and the language
modeling head whose complexities are linear in the sequence length. than when
training with the full sequence length. However, larger and heavy-trained
models such as RoBERTa [Liu et al., 2019b] showed that attention weights and
positional embeddings quality may suffer from pre-trainings with shorter
sequence lengths due to the lack of computation of long-range dependencies
between tokens. Nevertheless, we pre-train with the setting described by
Devlin et al. [2019] because the comparison is still fair and because of our
limited computational budget.

Regarding the training hyper-parameters, we apply a weight decay of 0.01 to
every parameter apart from those in biases and LayerNorms modules and we set
$\epsilon=10^{-8}$, $\beta_{1}=0.9$ and $\beta_{2}=0.999$ in the optimizer. We
use a triangular learning rate scheduler, with a peak of $1\cdot 10^{-4}$
(Base) or $2\cdot 10^{-4}$ (Small) and 10K warmup steps. The probability in
drop-out layers is set to 0.1. Finally, we use a batch size of 256 examples
for Base models and 1024 for the Small versions, for optimal utilization of
the GPU memory.

We train also a Base and a Small ELECTRA architecture for comparison. Notice
that the ELECTRA requires more FLOPS because of the presence of the additional
generator network. FLOPS are a general and architecture-independent indicator
of model requirements that measure the total number of floating point
operations required for training. To address this issue, and train all models
with the same compute budget, we reduce the total number of steps of ELECTRA
from 900K@128 + 100K@512 to 689K@128 + 77K@512 for both Small and Base models.
The generator size is set to $\frac{1}{3}$ and $\frac{1}{4}$ for Base and
Small models as in the original work, respectively222Applies to the hidden
size, the intermediate size and the number of attention heads.. Inspired by
Clark et al. [2020], we use FLOPS as a measure of compute requirements.
However, actual training time is sometimes misaligned with FLOPS because of
hardware acceleration such as NVIDIA Tensor Cores, mixed-precision training,
low precision _float32_ matrix multiplication and other optimization
techniques. Tables 5.2 and 5.3 summarize the training time and FLOPS required
by the Base and Small models.

Each experiment is executed on two GPUs of our machine (which is described in
Section 1.2), using mixed-precision training.

Models | FLOPS | Time | Memory | # Parameters | Objective
---|---|---|---|---|---
Total | LM Head
BERTSmall \+ MLM | $3.04\cdot 10^{18}$ | 1d 7h | 12.5GiB | 13.5M | 3.9M | MLM (1.0)
BERTSmall \+ RTS | $2.73\cdot 10^{18}$ | 17h | 6.3GiB | 13.5M | 512 | RTS (20.0)
BERTSmall \+ C-RTS | $2.73\cdot 10^{18}$ | 17h | 6.3GiB | 13.5M | 512 | C-RTS (20.0)
BERTSmall \+ SLM | $3.04\cdot 10^{18}$ | 1d 7h | 12.5GiB | 13.5M | 3.9M | SLM (1.0)
ELECTRASmall \+ MLM/TD | $4.87\cdot 10^{18}$ | 1d 16h | 17.4GiB | 18.2M | 3.9M + 512 | MLM (1.0) + TD (20.0)

Table 5.3: FLOPS, time, memory and number of parameters required to pre-train
Small models on the same hardware. For more details, see Table 5.2.

####  5.3.3 Fine-Tuning

After pre-training every model on the same amount of data and with the same
hyper-parameters, we evaluate them on different benchmarks.

###### GLUE

First, for a general performance evaluation, we use the GLUE benchmark suite,
which contains datasets for many different tasks such as Questions Similarity
and Natural Language Inference. For more details about the GLUE dataset, refer
to Section 4.3. As in Devlin et al. [2019] and Clark et al. [2020], we omit
WNLI from the GLUE datasets because even a trivial majority classifier is
better than most Base or Small pre-trained models.

As a starting point, we copy the fine-tuning hyper-parameters of Liu et al.
[2019b] for every GLUE task. After that, we evaluate affine configurations to
ensure good final accuracy of the models. The final hyper-parameters are
described in Table 5.4. Other parameters have been kept constant for all
experiments: no weight decay, triangular learning rate with a warmup phase of
10% of the total fine-tuning steps and a maximum sequence length of 128.
Regarding the testing, we take the best model based on the evaluation metric
of each GLUE dataset. Moreover, we apply early stopping with a patience of 5
and we validate at the end of each training epoch.

Hyper-Parameter | CoLA | MNLI-m | MRPC | QNLI | QQP | RTE | SST-2 | STS-B
---|---|---|---|---|---|---|---|---
Batch Size | 16 | 32 | 16 | 32 | 32 | 16 | 32 | 16
Learning Rate | $1\cdot 10^{-5}$ | $1\cdot 10^{-5}$ | $1\cdot 10^{-5}$ | $1\cdot 10^{-5}$ | $1\cdot 10^{-5}$ | $2\cdot 10^{-5}$ | $1\cdot 10^{-5}$ | $2\cdot 10^{-5}$
Max epochs | 20 | 8 | 10 | 10 | 8 | 40 | 10 | 40

Table 5.4: GLUE fine-tuning hyper-parameters.

Regarding the evaluation metrics, we use those proposed by the original
authors of GLUE [Wang et al., 2018] and similar works [Devlin et al., 2019;
Liu et al., 2019b; Clark et al., 2020]. In particular, we use accuracy for all
tasks apart from CoLA and STS-B, which are evaluated with Matthew and Spearman
correlation coefficients, respectively.

Finally, we repeat every experiment with 5 random seeds (which influence the
classification head initialization and the dataset shuffling), and we send the
results of the best-performing checkpoints on the dev set to the GLUE
Leaderboard for evaluation on the test
set333https://gluebenchmark.com/leaderboard/.

###### ASNQ

We use ASNQ to evaluate models performance on large Answer Selection datasets.
ASNQ contains more than 20M question-answer pairs in the training set, making
it the largest dataset for AS2 to date. More details about ASNQ are given in
Section 4.2.1. We use a batch size of 2048 examples for each model and a
learning rate equal to $1\cdot 10^{-5}$. The values are obtained with a greedy
search: we try all the combinations of batch sizes in
$\\{128,256,512,1024,2048,4096\\}$ and learning rates in $\\{2\cdot
10^{-6},5\cdot 10^{-6},1\cdot 10^{-5},2\cdot 10^{-5}\\}$. Since most of the
question-answer pairs in the dataset are shorter than 128 tokens, we set the
maximum sequence length to this value. We allow models to train up to 10
epochs, and we do early stopping on the development set if MAP does not
improve for 4 validations in a row. Other training parameters are the same of
the fine-tuning on GLUE. Experiments are repeated 3 times with different
initialization seeds to show average and standard deviation in the results.

We measure performance using Mean Average Precision (MAP) and Mean Reciprocal
Rank (MRR). The first is a soft metric that measures the Average Precision at
different levels of recall, while the second is a hard metric that measures
only the reciprocal rank of the positive answer with the highest score. For
more details about Answer Selection ranking, see Section 2.1.1.2.

###### WikiQA, TREC-QA and Transfer-Learning

As a last step, we evaluate the pre-trained models on two small datasets for
Answer Sentence Selection: WikiQA and TREC-QA. We first fine-tune the pre-
trained models directly on those two datasets. Then, as in TandA [Garg et al.,
2019], we measure performance in Transfer-Learning by taking the checkpoints
of the models after a transfer step on ASNQ and fine-tuning them on WikiQA and
TREC-QA. We indicate the two transfer steps and subsequent adaptations to the
final task with ASNQ $\rightarrow$ WikiQA and ASNQ $\rightarrow$ TREC-QA.

Fine-tuning hyper-parameters are selected again with greedy search. For the
batch size, we try values in $\\{32,64,128\\}$, while for the learning rate,
we search in $\\{2\cdot 10^{-6},5\cdot 10^{-6},1\cdot 10^{-5},2\cdot
10^{-5}\\}$. The maximum sequence length is fixed to 128 tokens while the
number of epochs is limited to 40. Again, we use early stopping to spare
resources, with a patience of 5.

###  5.4 Results

####  5.4.1 Pre-Training

We start the comparison of the results with a discussion about pre-training
times. For each Base model, Table 5.2 shows training time and the total number
of FLOPS. RTS and C-RTS use 20% less training time to complete the same number
of pre-training steps. The difference in terms of FLOPS is instead smaller,
around 5%. Notice that RTS and C-RTS use 45% GPU memory, thus training times
could be reduced even further by doubling the batch size. However, we keep the
same training parameters for all models to provide a fair comparison.

With Small models, the gains in training time are even larger. RTS and C-RTS
allow to reduce pre-training time by 45% and FLOPS by 10%. Gaps are larger
than Base models because the embedding layer and the language modeling head
are bound to the vocabulary size $|{\mathbb{V}}|$. We do not shrink the
vocabulary size because recent works show significant drops in performance
when reducing it [Turc et al., 2019] and because BERT’s vocabulary size is
already one of the smallest among recently released pre-trained models. RTS
and C-RTS have about $\frac{1}{3}$ of the memory footprint of MLM or SLM. As
with Base models, we could have increased the batch size to speed up the
training of RTS and C-RTS even further.

####  5.4.2 Fine-Tuning

In this Section, we present the comparison on different tasks of the models we
pre-trained. The statistical significance comparison in the tables if
performed compared to BERTBase \+ MLM and BERTSmall \+ MLM. Results of the
original BERTBase \+ MLM/NSP [Devlin et al., 2019] derive from our fine-tuning
on the various tasks in the same setting as the other models, because the
original authors do best model selection on the development set.

#####  5.4.2.1 Base Models

Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | AVG
---|---|---|---|---|---|---|---|---|---
BERTBase \+ MLM/NSP [Devlin et al., 2019] | 57.6 (1.8) | 84.3 (0.4) | 82.3 (1.3) | 91.0 (0.7) | 91.0 (0.2) | 68.9 (1.4) | 92.6 (0.1) | 89.1 (0.3) | 82.1
ELECTRABase \+ MLM/TD | 63.4 (1.3) | 83.7 (0.2) | 87.2 (0.8) | 90.4 (0.1) | 91.2 (0.1) | 74.6 (1.4) | 91.4 (0.4) | 88.5 (0.2) | 83.8
BERTBase \+ MLM | 58.1 (1.0) |  83.4 (0.2) | 87.5 (0.5) |  90.2 (0.3) | 90.9 (0.1) | 67.4 (1.2) |  92.2 (0.3) |  87.8 (0.3) | 82.2
BERTBase \+ RTS | 58.1 (1.1) | 82.7 (0.2) |  87.6 (1.0) | 89.4 (0.3) | 90.9 (0.1) | 68.5 (1.4) | 91.5 (0.3) | 86.6 (0.4) | 81.9
BERTBase \+ C-RTS | 57.4 (0.7) | 82.0 (0.3) | 84.2 (0.4) | 89.6 (0.2) | 90.6 (0.1) | 66.6 (2.4) | 91.5 (0.2) | 87.0 (0.2) | 81.1
BERTBase \+ SLM |  59.6 (1.0) |  83.4 (0.2) | 87.5 (0.4) | 89.9 (0.2) |  91.0 (0.1) |  69.2 (1.2) | 92.1 (0.1) | 87.6 (0.3) | 82.5

Table 5.5: Results on GLUE development set for Base models. We do not employ
ensemble models. We measure performance with Accuracy on all tasks apart from
CoLA and STS-B, in which we use Matthews correlation coefficient and
Spearman’s rank correlation coefficient, respectively. We use bold formatting
for our proposals and the best results. We underline statistically significant
improvements over BERTBase \+ MLM.

Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | AVG
---|---|---|---|---|---|---|---|---|---
BERTBase \+ MLM/NSP [Devlin et al., 2019] | 53.0 | 82.7 | 82.8 | 89.6 | 88.2 | 62.8 | 91.2 | 80.6 | 78.9
ELECTRABase \+ MLM/TD | 60.6 | 83.6 | 84.1 | 90.2 | 89.0 | 69.1 | 92.4 | 85.5 | 81.8
BERTBase \+ MLM | 53.7 | 83.6 | 82.6 | 89.9 | 89.1 | 63.1 | 92.3 | 83.6 | 79.7
BERTBase \+ RTS | 57.3 | 82.6 | 81.3 | 89.3 | 88.9 | 66.2 | 91.7 | 82.2 | 79.9
BERTBase \+ C-RTS | 57.3 | 82.8 | 81.9 | 89.4 | 88.6 | 60.9 | 91.0 | 82.2 | 79.3
BERTBase \+ SLM | 57.0 | 83.3 | 83.1 | 89.7 | 88.9 | 65.0 | 92.3 | 83.8 | 80.4

Table 5.6: Results on the GLUE test set for Base models. For more details
about the metrics, see Table 5.5. For the evaluation on the GLUE leaderboard,
we select the best models on the development set.

Results of Base models over the GLUE benchmarks are presented in Table 5.5 for
the development set (as in Clark et al. [2020]) and in Table 5.6 for the test
set. Results on the development set allow the comparison of the standard
deviation in results. This is not possible on the test set because of the
limited number of submissions allowed by the GLUE Leaderboard.

On GLUE, the models pre-trained with RTS and C-RTS objectives obtain
comparable results with all the other techniques, except for ELECTRA-TD.
However, while RTS and C-RTS require about 20% less training time than a
similar MLM-based model, ELECTRA-TD requires more time to complete the pre-
training and consumes much more memory due to the presence of the additional
generator network, thus reducing scalability. In particular, ELECTRA uses 36%
more memory than a similar MLM- or SLM-based model and almost doubles the
memory required by RTS or C-RTS. The SLM objective we designed outperforms MLM
in most tasks while being trained with the same computational budget. In
particular, SLM achieves an average GLUE score higher than our MLM-based BERT
by 0.7 points and also higher than the original BERT, which was trained with
MLM and NSP (+1.5). SLM shows also that the pre-training/fine-tuning
discrepancy is a major issue of MLM-based models such as BERT or RoBERTa.

We compare also with the original BERT model released by Devlin et al. [2019],
which uses an additional sentence-level objective to improve performance.
Notice that even if Liu et al. [2019b] demonstrated that dropping the NSP
objective does not affect final accuracy, we believe this claim is valid only
with very long pre-trainings. RoBERTa [Liu et al., 2019b] is trained for a
full day on 1024 GPUs, which is about 53 more times the computational
requirements of BERT [Devlin et al., 2019]. Thus, with models pre-trained with
an academic budget and over tasks where the model has to work on two input
sequences, such as AS2, our results show that NSP is still beneficial because
it improves the representation of the CLS token, which is used for
classification.

We do not compare against the ELECTRA models released by Clark et al. [2020]
because they were pre-trained on 20 times the total tokens seen by our models.

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR |  | MAP | MRR |  | MAP | MRR
BERTBase \+ MLM/NSP [Devlin et al., 2019] | 66.8 (0.1) | 73.2 (0.2) |  | 82.8 (0.9) | 84.2 (1.0) |  | 87.2 (0.9) | 92.9 (1.0)
ELECTRABase \+ MLM/TD | 64.9 (0.3) | 71.7 (0.4) |  | 81.8 (1.6) | 83.2 (1.6) |  | 86.8 (1.4) | 92.2 (1.5)
BERTBase \+ MLM | 65.5 (0.2) | 72.2 (0.3) |  | 79.9 (1.1) | 81.2 (1.2) |  | 86.4 (0.7) | 91.5 (0.7)
BERTBase \+ RTS | 64.6 (0.1) | 71.1 (0.2) |  | 78.5 (2.5) | 80.1 (2.5) |  | 86.6 (1.5) | 91.8 (1.5)
BERTBase \+ C-RTS | 64.7 (0.1) | 71.5 (0.1) |  | 79.0 (1.9) | 80.6 (1.6) |  | 87.0 (0.8) | 91.8 (1.1)
BERTBase \+ SLM |  65.8 (0.3) |  72.7 (0.3) |  |  80.2 (1.5) |  81.7 (1.6) |  |  87.3 (1.3) |  92.1 (1.5)

Table 5.7: Results on ASNQ, WikiQA and TREC-QA for Base models. The results on
ASNQ are on the dev. set. We use bold formatting for our proposals and the
best results. We underline statistically significant improvements over
BERTBase \+ MLM.

Model |  ASNQ $\xrightarrow{}$ WikiQA |  |  ASNQ $\xrightarrow{}$ TREC-QA
---|---|---|---
MAP | MRR |  | MAP | MRR
BERTBase \+ MLM/NSP [Devlin et al., 2019] | 87.9 (0.4) | 89.3 (0.4) |  | 89.1 (0.4) | 93.4 (0.5)
ELECTRABase \+ MLM/TD | 88.4 (0.4) | 89.8 (0.4) |  | 88.9 (0.3) | 92.0 (0.5)
BERTBase \+ MLM |  88.9 (1.0) |  90.2 (1.0) |  | 87.6 (0.9) | 90.6 (1.1)
BERTBase \+ RTS | 87.9 (0.5) | 89.3 (0.5) |  | 88.5 (0.5) |  93.4 (0.4)
BERTBase \+ C-RTS | 87.1 (0.6) | 88.3 (0.6) |  |  88.7 (0.4) |  92.9 (0.7)
BERTBase \+ SLM | 87.7 (0.8) | 89.3 (0.7) |  | 87.9 (0.6) | 91.0 (0.6)

Table 5.8: Results on WikiQA and TREC-QA for Base models after the transfer
step on ASNQ.

Regarding the results in Answer Sentence Selection, which are provided in
Table 5.7 and 5.8, RTS and C-RTS show comparable performance with MLM and SLM
in most tasks. On WikiQA, results are slightly lower on average, while they
are superior on TREC-QA. On ASNQ, RTS and C-RTS are below MLM and SLM by less
than 1 point in MAP, while being pre-trained for 20% less time. With the
transfer step over ASNQ before the final fine-tuning on WikiQA and TREC-QA,
relative differences between objectives do not change. Results are higher on
average because the models are accustomed to performing Answer Selection on a
large dataset before being fine-tuned. Notice also that the transfer step on
ASNQ benefits more WikiQA than TREC-QA. We argue that the reason is that both
ASNQ and WikiQA are created from documents sourced from Wikipedia, while TREC-
QA is based on human-generated answers. On the other hand, SLM shows his
superiority over MLM in 4 tasks out of 5. SLM outperforms a similar MLM-based
model by 1.9 points in MAP on WikiQA, by 0.9 on TREC-QA, by 0.3 on ASNQ
$\rightarrow$ WikiQA and by 0.3 on ASNQ, while requiring the same
computational budget to pre-train. These results highlight again the advantage
of removing the special [MASK] token from the pre-training.

#####  5.4.2.2 Small Models

Here we present the results of applying our effective objectives to Small
Transformer models. Table 5.9 shows the results over the GLUE development set,
while in Table 5.10 we provide the results over 3 AS2 datasets and the average
GLUE score on the test set. We do not show performance on Transfer-Learning
from ASNQ to either WikiQA or TREC-QA because ASNQ is so large that after the
transfer step, the differences in accuracy between the objectives are minimal
and not statistically significant.

On the GLUE dev set, Small models trained with RTS and C-RTS outperform a
similar MLM-based model on every task. The average GLUE score of RTS and C-RTS
is respectively 2.0 and 2.1 points higher than MLM and even higher than SLM on
some individual datasets. The only drawback of RTS and C-RTS is that the
standard deviation is higher across most tasks, indicating a higher
sensitivity to parameters’ initialization. On the GLUE test set, RTS and C-RTS
perform accordingly to the dev set, outperforming MLM by 1.3 and 1.6 points.
We can conclude that RTS and C-RTS are well suited for Small models, since
they require almost half of the pre-training time and deliver superior
performance compared to MLM, SLM and ELECTRA’s TD.

The AS2 dataset results exhibit a trend resembling those of GLUE: RTS and
C-RTS consistently outperform MLM in all tasks. For instance, WikiQA showcases
RTS’s superiority over MLM by 2.1 points in Mean Average Precision, while ASNQ
shows a 1.2-point difference in favor of RTS. While results between MLM and
RTS on TREC-QA are quite similar, it is noteworthy that BERT with RTS
surpasses ELECTRASmall trained with TD in all tasks, despite the latter
requiring twice the pre-training time and three times the GPU memory.

Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | AVG
---|---|---|---|---|---|---|---|---|---
ELECTRASmall \+ MLM/TD | 51.2 (0.1) | 76.8 (0.2) | 85.5 (0.3) | 86.9 (0.0) | 88.4 (0.0) | 63.9 (0.3) | 87.0 (0.3) | 85.5 (0.1) | 78.1
BERTSmall \+ MLM | 42.2 (0.9) | 78.8 (0.2) | 80.8 (0.9) | 85.7 (0.6) | 88.6 (0.1) | 58.6 (1.3) | 89.4 (0.2) | 84.4 (0.2) | 76.1
BERTSmall \+ RTS |  51.2 (1.8) |  79.9 (0.2) | 81.5 (0.7) |  87.9 (0.1) |  89.4 (0.1) |  61.2 (0.6) | 88.4 (0.4) |  85.2 (0.2) | 78.1
BERTSmall \+ C-RTS |  51.6 (0.9) |  79.8 (0.1) | 81.3 (1.0) |  87.0 (0.4) |  89.4 (0.1) |  61.6 (1.4) |  89.5 (0.2) |  85.3 (0.3) | 78.2
BERTSmall \+ SLM |  45.8 (0.5) |  79.1 (0.2) |  83.3 (0.4) | 86.3 (0.3) |  89.0 (0.1) |  60.8 (1.4) | 88.6 (0.4) |  86.0 (0.2) | 77.4

Table 5.9: Results on GLUE development set for Small models. For more details
about the metrics, see Table 5.6. We use bold formatting for our proposals and
the best results. We underline statistically significant improvements over
BERTSmall \+ MLM.

Model | WikiQA |  | TREC-QA |  | ASNQ |  | GLUE
---|---|---|---|---|---|---|---
MAP | MRR |  | MAP | MRR |  | MAP | MRR |  | AVG
ELECTRASmall \+ MLM/TD | 75.8 (1.6) | 77.5 (1.5) |  | 83.3 (1.0) | 88.2 (1.6) |  | 59.2 (0.0) | 66.3 (0.0) |  | 75.5
BERTSmall \+ MLM | 74.5 (1.9) | 75.8 (1.9) |  | 85.3 (1.3) | 89.9 (1.2) |  | 58.6 (0.3) | 66.0 (0.4) |  | 74.1
BERTSmall \+ RTS |  76.6 (0.7) |  78.4 (0.9) |  | 85.2 (2.4) | 90.5 (2.8) |  |  59.8 (0.1) |  66.9 (0.2) |  | 75.4
BERTSmall \+ C-RTS | 74.0 (1.1) | 75.8 (1.3) |  | 86.1 (0.6) |  92.5 (2.0) |  |  59.6 (0.2) |  67.1 (0.2) |  | 75.7
BERTSmall \+ SLM | 73.6 (0.8) | 75.2 (0.9) |  |  86.3 (1.4) | 91.7 (1.8) |  |  59.2 (0.2) | 66.5 (0.3) |  | 75.7

Table 5.10: Results of our Small models pre-trained with RTS, C-RTS and SLM
compared with MLM on AS2 datasets and the GLUE test set.

Regarding C-RTS, its performance, on average, is slightly lower than RTS in
AS2 but slightly better on the GLUE test set, with differences in only a few
decimal points. However, in the next Section, we will demonstrate that C-RTS
exhibits clear advantages over RTS when trained for a longer duration, likely
due to the slower convergence of the more challenging pre-training task.

Finally, SLM provides good performance on the GLUE test set, matching the
average score of C-RTS and surpassing the expensive ELECTRASmall architecture
and the MLM-based BERT. On the other Answer Sentence Selection tasks, SLM
performance is better than those of BERT with MLM on 2 datasets out of three.
Notice that BERT with SLM and MLM share the exactly same architecture, pre-
training time and memory footprint.

Model | WikiQA | TREC-QA | ASNQ | MRPC | QNLI
---|---|---|---|---|---
BERTSmall \+ RTS | 76.6 (0.9) | 85.5 (1.5) | 59.9 (0.2) | 81.5 (1.5) |  86.9 (0.3)
BERTSmall \+ C-RTS |  77.4 (1.0) |  86.4 (0.9) |  60.7 (0.1) |  84.2 (0.1) |  86.9 (0.1)

Table 5.11: Comparison between RTS and C-RTS. Results for WikiQA and TREC-QA
are computed over the test set, while we report the best results over the dev.
set for the others. We compare results with MAP for WikiQA, TREC-QA and ASNQ,
while we use Accuracy for the others. We underline statistically significant
improvements over BERTSmall \+ RTS.

####  5.4.3 RTS vs C-RTS

The results of the previous Section show that C-RTS achieves accuracies
similar to RTS in most benchmarks. To fully demonstrate the potential of a
harder training task, we compared RTS and C-RTS when pre-trained for longer on
the same data as before. In order to accomplish that, we use a new pre-
training configuration in which the maximum sequence length is not reduced for
the first steps (e.g.we use always 512 tokens). Then, we set the batch size to
1024 examples and train both models for 200K steps. After every 10K pre-
training steps, we evaluate over 5 different benchmarks: ASNQ, WikiQA, TREC-
QA, QNLI and MRPC. We selected those datasets because they cover 3 different
domains and the sizes range from very few examples (WikiQA) to more than 20M
(ASNQ). We exploit Small models for this experiment due to the cost of pre-
training Base Transformers.

In Table 5.11 we report the results in the last pre-training step of the
comparison described above. C-RTS obtains better results than RTS on 4 tasks
out of 5 thanks to the harder training task. On the other hand, both
objectives perform comparably from a statistical viewpoint on QNLI. This
demonstrates empirically that selecting challenging replacements forces the
model to find deeper relations between tokens. We provide the full plots of
the comparison in Appendix C.1

Model | GLUE | FLOPS |  | Model | GLUE | FLOPS
---|---|---|---|---|---|---
BERTSmall \+ RTS | 75.4 |  $\times$ 0.10 |  |  BERTLarge \+ MLM/NSP [Devlin et al., 2019] | 83.3 |  $\times$ 12
BERTSmall \+ MLM | 74.1 |  $\times$ 0.12 |  |  ELECTRABase \+ MLM/TD [Clark et al., 2020] | 85.7 |  $\times$ 21
BERTSmall \+ SLM | 75.7 |  $\times$ 0.12 |  |  RoBERTaBase \+ MLM [Liu et al., 2019b] | 86.3 |  $\times$ 53
BERTBase \+ RTS | 79.9 |  $\times$ 0.81 |  |  ELECTRALarge \+ MLM/TD [Clark et al., 2020] | 88.6 |  $\times$ 194
BERTBase \+ MLM | 79.7 |  $\times$ 1 |  |  RoBERTaLarge \+ MLM [Liu et al., 2019b] | 88.8 |  $\times$ 200
BERTBase \+ SLM | 80.4 |  $\times$ 1 |  |  ALBERTLarge \+ MLM/SOP [Lan et al., 2020] | 90.0 |  $\times$ 1937

Table 5.12: Comparison of computational requirements for state-of-the-art
large language models. We report the average GLUE score as well as an
estimation of the FLOPS used for the pre-training, normalized on BERTBase \+
MLM.

###  5.5 Pre-Training Computational Trends

Our goal is to develop models that achieve comparable performance to MLM-based
models while requiring less computational resources. Our findings indicate
that the performance improvement follows a logarithmic pattern as the size of
the pre-training dataset increases and models are trained for longer.
Additionally, our computationally lighter models, such as RTS and C-RTS,
demonstrate no statistically significant difference compared to more expensive
models on most tasks. A comparison in Table 5.12 reveals that the top-
performing architectures only outperform MLM-based models when trained on
significantly larger amounts of data and for much more time. For instance,
ELECTRABase utilizes 21 times more resources than BERTBase, and RoBERTa uses
53 times more. It is worth noting that achieving a score of 90.0 on the GLUE
benchmark requires a model to be trained for 2000 times the duration of the
original BERT model training, which is quite remarkable. For those reasons, we
believe research in finding more efficient architectures, objectives or
distillation techniques is extremely important to reduce the carbon footprint
of models pre-training.

##  Chapter 6 Self-supervised Objectives for Multi-Sentence Inference

In this Chapter, we propose pre-training objectives designed specifically to
improve accuracy in multiple-sentence classification. Many tasks such as AS2,
Fact Verification or NLI are solved by training a sentence pair classifier,
for example by feeding the Transformer model with a question-answer or a
claim-evidence pair. Recent studies [Zhang et al., 2021; Tymoshenko and
Moschitti, 2021] showed that modeling multiple candidate sentences helps in
learning long-range dependencies and allows the model to compare them, instead
of scoring them individually.

A major problem is that off-the-shelf Transformer models struggle at modeling
multiple related sentence candidates together. For this reason, we propose a
new pre-training technique that exploits weak supervision of pre-training
corpora to adapt the model at reasoning over multiple input sentences taken
from the same paragraph. We call _Jointwise_ the class of models trained with
objectives that target relations among multiple input sentences. At the same
time, we call _Pairwise_ models the class of architectures designed to reason
over 1 or 2 input sequences.

Regarding the related works, additional information about the RoBERTa
architecture [Liu et al., 2019b] can be found in Section 3.1.2. Additionally,
Sections 3.2.1 and 3.2.2 describe the actual state-of-the-art for Answer
Sentence Selection and Fact Verification. In Section 3.3.2 instead, a
description of works about multi-sentence inference is provided, while in
Section 3.3.3 we analyze related research that exploits the weak structure of
large corpora for supervision.

To demonstrate the superiority of our approach, we evaluate models pre-trained
with our objectives on three datasets for Answer Sentence Selection (ASNQ,
WikiQA and TREC-QA) and one dataset for Fact Verification (FEVER). Statistics
and details about each dataset are given in Section 4.2.1. We select
RoBERTaBase [Liu et al., 2019b] as the starting point because it provides good
accuracy on a wide range of tasks and does not require expensive hardware to
be fine-tuned. We compare against a baseline pre-trained only with common
objectives (e.g. MLM) as well as models pre-trained over data formatted for
our task but without explicit supervision.

###  6.1 Limits of Pairwise Models

Examples of Pairwise models comprehend Transformers either (i) pre-trained
with sentence-level objectives such as NSP [Devlin et al., 2019] or SOP [Lan
et al., 2020] or (ii) fine-tuned on Pairwise tasks such as Natural Language
Inference, Answer Sentence Selection and Fact Verification.

When applied to downstream tasks, Pairwise models perform effectively in
comparing two input sequences, thanks to the Cross-Attention of Transformers.
In the AS2 task, they receive a question and a single answer candidate and
produce a score indicating the degree of correctness. However, this approach
has limitations as the model cannot efficiently compare and resolve references
among multiple answer candidates to select the best answer.

Conversely, in Fact Verification, the current state-of-the-art approach
involves providing the model with a claim and a set of evidence sentences. The
model then determines whether the claim is supported, neutral, or refuted by
the evidence sentences [Liu et al., 2020; Tymoshenko and Moschitti, 2021].
Nevertheless, Transformers are typically pre-trained with token-level and
sentence-level objectives, focusing on at most two spans of text. Although
models like RoBERTa are trained on very long sequences containing multiple
sentences, they are not explicitly trained to identify connections and
relations between these sentences. Consequently, they may struggle to learn
meaningful dependencies between the candidates during fine-tuning, making them
less adaptable to tasks requiring reasoning over multiple inputs.

###  6.2 Multi-Sentence Transformer Model

In this Section, we present the Transformer architecture and the pre-training
objective we developed for the multi-sentence training and inference tasks.

####  6.2.1 Joint Encoder Architecture

We created an architecture specifically designed to process and reason over
multiple input sequences. We indicate with $k+1$ the maximum number of text
spans that can be provided to the model. The first input sentence is called
_pivot_ and is of special importance because we study its relation with the
other $k$ input sequences. For example, in AS2 we place the question in the
first position and the model should predict its relation with the other $k$
answer candidates. Similarly, in Fact Verification we set the claim as the
pivot and we predict whether the other $k$ evidence sentences support or
reject the claim.

We experimented with 2 different configurations: the first, referred to as
_Fixed_ (Figure 6.1), and the second, named _Flexible_ (Figure 6.2), both
consisting of a RoBERTa Encoder [Liu et al., 2019b]. The main differences
between the Fixed and Flexible variants are how inputs are formatted before
being fed to the model and how multiple predictions are performed from the
last hidden state.

#####  6.2.1.1 Fixed Model

Figure 6.1: Example of Jointwise Fixed architecture.

In the Fixed variant of our architecture, each of the $k+1$ input text spans
has the same Fixed length $L$, measured in the number of tokens. We crop or
pad on the right all the input sequences whose length is different than $L$.
Notice that this may lead to pad tokens between different input sentences.
This is not an issue because the attention mask will be aware of those special
tokens and will disable attention scores in the corresponding positions. We
add positional embeddings and sequence id embeddings to the word embeddings.
Positional embeddings help the model in computing attention between pairs of
tokens in different positions while sequence id embeddings are useful to let
the model know to which sequence each token belongs. We are aware that having
both positional ids and sequence ids in a Fixed architecture is redundant,
however, we keep both embeddings because the overhead is negligible and we can
use a single implementation for both Fixed and Flexible models. We also
experimented using the same positional ids $0,\dots,L-1$ for every sentence,
but we didn’t measure significant gains. Notice that in this case, token type
ids played an important role and couldn’t be removed.

Jointwise Fixed models perform $k+1$ sentence-level predictions for each step,
employing a classification head over output embeddings
$[{\bm{\mathrm{o}}}_{0},{\bm{\mathrm{o}}}_{1},\dots,{\bm{\mathrm{o}}}_{k}]$,
which correspond to the first token $<$s$>$ of each input sentence111$<$s$>$
is RoBERTa’s “begin of sentence” token, equivalent to BERT’s [CLS].. In
particular, given the last hidden state ${\bm{H}}\in{\mathbb{R}}^{(k+1)L\times
d}$, the output embeddings
${\bm{\mathrm{o}}}_{0},{\bm{\mathrm{o}}}_{1},\dots,{\bm{\mathrm{o}}}_{k}\in{\mathbb{R}}^{d}$
correspond to the rows $0,L,\dots,kL$ of ${\bm{H}}$.

#####  6.2.1.2 Flexible Model

Figure 6.2: Example of Jointwise Flexible architecture.

The Flexible architecture is similar to the Fixed version but allows input
sequences to have a variable length. In Figure 6.2, the length of each input
sequence is indicated with $L_{0},\dots,L_{k}$. The only constraint in
Flexible models is:

$\sum_{i=0}^{k}L_{i}\leq L$ (6.1)

If the input does not match the constraint, we iteratively remove one token at
a time from the longest sequence until we do not match the requirement. We use
again positional and sequence ids. In particular, preliminary experiments
demonstrated that sequence ids are very important in this architecture because
they help the model to better understand the context for each prediction. In
Flexible models, we employ only 2 sequence ids instead of $k+1$. We assign a
sequence id equal to 0 to the pivot and 1 to all the other $k$ sentence
candidates. Predictions with Flexible models are simpler because we consider
always only the output embedding ${\bm{\mathrm{o}}}_{0}$ of the first token to
perform single or multi-label predictions.

####  6.2.2 Multi-Sentence Inference Tasks

Here, we describe how we adapt AS2 and Fact Verification for the multi-
sentence inference task. We denote with $k$ the maximum number of predictions
performed by our Jointwise model.

###### AS2

We extend the definition of AS2 given in Section 2.1.1.2 for multi-sentence
inference. In particular, we partition the candidates ${\mathbb{C}}$ in
disjoint subsets ${\mathbb{C}}_{0},\dots,{\mathbb{C}}_{m}$ of size $k$ with
random selection. Then, we feed the model $\mathcal{M}$ with the question and
one subset at a time, obtaining $k$ predictions over the candidates. If one
subset ${\mathbb{C}}_{i}$ is smaller than $k$, we add padding on the right
until we match the sequence length of the other elements in the batch and we
do not predict and back-propagate errors from the corresponding classification
heads. Finally, we select the best answer among all subsets as the one having
the highest overall score, as in the original task.

Our intuition is that by providing $\mathcal{M}$ with $k$ different
candidates, it can model interrelated information to find the best answer $a$
to $q$ [Zhang et al., 2021].

###### Fact Verification

Let’s consider a claim $c$ and a set of evidence sentences
${\mathbb{E}}=\\{e_{1},\dots,e_{n}\\}$ retrieved using a BERT-based DocIR [Liu
et al., 2020]. In the classic definition of the task, $c$ should be
supported/refuted by at least one piece of evidence $e_{i}\in{\mathbb{E}}$ to
be labeled accordingly. More details about Fact Verification are given in
Section 2.1.2.

As for Joint AS2, we provide the model $\mathcal{M}$ with the claim $c$ and
the evidence ${\mathbb{E}}$, such that different evidence sentences can be
modeled jointly and complement each other [Tymoshenko and Moschitti, 2021].

####  6.2.3 Training and Inference using Joint Encoders

We describe how we exploit output embeddings
${\bm{\mathrm{o}}}_{0},\dots,{\bm{\mathrm{o}}}_{k}$ of Fixed and
${\bm{\mathrm{o}}}_{0}$ of Flexible models to perform predictions. We
differentiate between AS2, that requires to predict a label for each
candidate, and Fact Verification, which requires to predict a single label for
every input example.

#####  6.2.3.1 Predict a single label

We experiment with two different strategies to predict a single label from the
output embeddings of the model.

###### IE1

We place a linear layer ${\bm{W}}\in{\mathbb{R}}^{2\times d}$ only on top of
${\bm{\mathrm{o}}}_{0}$, that corresponds to the embedding of the first input
sentence (similar to BERT [Devlin et al., 2019]). This classification head can
be used for both Fixed and Flexible models. We call this method Individual
Evidence inference head and we obtain predictions $p$ with:

$p={\bm{W}}{\bm{\mathrm{o}}}_{0}$ (6.2)

###### AE1

Here, we apply a single linear layer ${\bm{W}}\in{\mathbb{R}}^{2\times d}$ to
the average of all the output embeddings
${\bm{\mathrm{o}}}_{1},\dots,{\bm{\mathrm{o}}}_{k}$ to gather information from
all input candidates. By definition, this classification head can be used only
with Fixed models. We refer to this technique as Aggregated Evidence
classification head:

$p={\bm{W}}\left(\frac{1}{k}\sum_{i=1}^{k}{\bm{\mathrm{o}}}_{i}\right)$ (6.3)

#####  6.2.3.2 Predict $k$ labels

To predict $k$ different labels, we experimented with 3 techniques (2 for
Fixed and 1 for Flexible models).

###### IEk

A linear layer with shared weights ${\bm{W}}\in{\mathbb{R}}^{2\times d}$ is
applied to each output embedding ${\bm{\mathrm{o}}}_{i}$ to obtain
predictions. By definition, this head works only with Fixed models. We refer
to this method as $k$-candidate Individual Evidence inference head:

$p_{i}={\bm{W}}{\bm{\mathrm{o}}}_{i}\ \ \forall i\in 1,\dots,k$ (6.4)

###### AEk

In this experiment, let the linear layer be a matrix of shared weights
${\bm{W}}\in{\mathbb{R}}^{2\times 2d}$. We feed the linear layer with the
concatenation of the first output embedding ${\bm{\mathrm{o}}}_{0}$ with each
candidate embedding (${\bm{\mathrm{o}}}_{1},\dots,{\bm{\mathrm{o}}}_{k}$).
This head is yet again specifically designed only for Fixed models. We call
this technique $k$-candidate Aggregated Evidence (AEk) classification head
(“$\cdot$” means vectors concatenation):

$p_{i}={\bm{W}}({\bm{\mathrm{o}}}_{0}\cdot{\bm{\mathrm{o}}}_{i})\ \ \forall
i\in 1,\dots,k$ (6.5)

###### REk

This last classification head is designed only for Flexible models. Let
$\\{{\bm{W}}_{i}\in{\mathbb{R}}^{2\times d},i\ \in\ 1,\dots,k\\}$ be a set of
$k$ linear layers. We apply all of them independently on the output embedding
${\bm{\mathrm{o}}}_{0}$ of the first input token to obtain $k$ predictions. We
name this method $k$-candidate Reduced Evidence (REk). Predictions are
computed as follows:

$p_{i}={\bm{W}}_{i}{\bm{\mathrm{o}}}_{0}\ \ \forall i\in 1,\dots,k$ (6.6)

In figure 6.3 we depict every classification head to better catch the
differences between them. Furthermore, Table 6.1 provides a summary of the
applicable classification heads for each model and task.

Figure 6.3: Inference heads for Joint Transformer models.

Task | Fixed |  | Flexible
---|---|---|---
IE1 |  IEk |  AE1 |  AEk |  REk |  |  IE1 |  IEk |  AE1 |  AEk |  REk
MSPP Pre-Training |  | ✓ |  | ✓ |  |  |  |  |  |  | ✓
Answer Sentence Selection |  | ✓ |  | ✓ |  |  |  |  |  |  | ✓
Fact Verification | ✓ |  | ✓ |  |  |  | ✓ |  |  |  |

Table 6.1: Compatibility matrix between models, classification heads and
downstream tasks.

####  6.2.4 Continuous Pre-Training of Joint Encoders

Figure 6.4: Example of MSPP objective.

In long documents, paragraphs are used to explore the document’s topic from
various perspectives. Despite this rich source of information, most
Transformer pre-training strategies fail to take advantage of it, leaving the
unsupervised pre-training phase without any guidance. To improve the ability
of Transformer models to understand the relationships between multiple
sentences, we introduce a new pre-training task called Multi-Sentences in
Paragraph Prediction (MSPP). In this task, the model is given $k+1$ sentences
$\\{s_{0},\dots,s_{k}\\}$ and is asked to predict whether sentences
$\\{s_{1},\dots,s_{k}\\}$ belong to the same paragraph $P$ as $s_{0}$, in the
document ${\bm{D}}$. We use the IEk and AEk prediction heads for Fixed models
and REk with Flexible models to perform the $k$ predictions. More formally, we
create a label $y_{i}$ for each sentence $s_{i}$ with the following rule:

$y_{i}=\begin{cases}1\ \ \text{if}\ s_{0},s_{i}\in P\ \text{in}\ {\bm{D}}\\\
0\ \ \text{otherwise}\\\ \end{cases}\forall\ i\in\\{1,\dots,k\\}$ (6.7)

During the pre-training phase, we randomly select a sentence from a paragraph
as $s_{0}$. We then select $k_{1}$ other sentences from the same paragraph as
positive examples, $k_{2}$ sentences from other paragraphs in the same
document as hard negative examples, and $k_{3}$ sentences from documents other
than ${\bm{D}}$ as easy negatives. We ensure $k_{1}+k_{2}+k_{3}=k$ by sampling
more easy negatives ($k_{3}$) if paragraph $P$ is not large enough to sample
$k_{2}$ hard negatives. This method tasks the model to learn the relationship
between sentences in a paragraph, improving its ability to understand the
underlying structure of long documents. An example of MSPP is given in Figure
6.4.

###  6.3 Experiments

To assess the performance of our joint Transformers, we conduct experiments on
three AS2 datasets and one Fact Verification dataset. We do not use commonly
used language model benchmarks for our evaluation, such as GLUE, because they
only involve Pairwise classification tasks, which is not sufficient for our
study.

####  6.3.1 Continuous Pre-Training

In our experiments, we want to avoid improvements stemming from the usage of
additional data. For this reason, we use the same pre-training corpora of
RoBERTa: English Wikipedia, BookCorpus, OpenWebText and CC-News. For more
details about the datasets, see Section 4.1. RoBERTa was also trained on the
STORIES dataset, however, we omit it because it is not publicly available
anymore222https://github.com/tensorflow/models/tree/archive/research/lm_commonsense#1-download-
data-files.

In all of our experiments, we set $k=5$ as in Zhang et al. [2021]; Tymoshenko
and Moschitti [2021] because it aligns well with the number of
candidates/evidence sentences in the datasets and because it does still allow
an efficient training with a reasonable input sequence length. We set also
$L=64$ in the experiments. Thus, Fixed models will have at most 6 input
sequences of length 64 while Flexible models will also accept up to 6 input
sequences whose total length is at most 384 tokens. However, since Flexible
models do not apply internal padding, we found that setting the maximum
sequence length $L$ to $256$ does not affect performance but increases
training speed.

The pre-training dataset is created with the MSPP strategy by setting
$k_{1}=1,k_{2}=2,k_{3}=2$. With these settings, we have a good balance of soft
and hard negative examples. The resulting dataset contains about 180M
examples. We do continuous pre-training starting from a RoBERTaBase checkpoint
because training from scratch would require a very large computational
budget333RoBERTaBase was trained on 1024 NVIDIA V100 GPUs for 1 day, which
corresponds to an expense of more than 100,000$ on Amazon Web Services or
similar cloud computing providers.. The original checkpoint is only augmented
with sequence-level embeddings and a small classification head for binary
predictions. While doing continuous pre-training, we found it beneficial to
keep the original MLM objective along with MSPP. We train the model over the
dataset described above for 100K steps with a batch size of 4096. We used a
triangular learning rate with a peak value of $5\cdot 10^{-5}$ and 10K warmup
steps. We set the optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and
$\epsilon=10^{-8}$ to update the weights of the model. We also apply a weight
decay of $0.01$ and dropout with probability $0.1$ to regularize model
training. Since we trained in mixed-precision (_fp16_), we did not use
gradient clipping because gradients are already scaled by the NVIDIA Apex
algorithm automatically. For the pre-training, we experimented with 2 Fixed
and 1 Flexible model: the first trained with IEk, the second with AEk and the
Flexible with REk. As in BERT and ALBERT [Devlin et al., 2019; Lan et al.,
2020], we use an equal weight for the MLM and MSPP to compute the final loss
value. The training takes about 2.2 days (Fixed) and 1.7 days (Flexible) on
our machine, which is described in Section 1.2, using mixed precision and the
parameters described above. In terms of efficiency, our continuous pre-
trainings account for about a 7% (Fixed) and a 5% (Flexible) increase in
computation compared to the original RoBERTa pre-training.

####  6.3.2 Fine-Tuning

We evaluate the 3 continuously pre-trained checkpoints on ASNQ, WikiQA, TREC-
QA and FEVER, using the correct classification heads, i.e. IEk, AEk or REk for
AS2 and IE1 or AE1 for Fact Verification.

We compare against many baselines from other works. On AS2, we compare with
(i) RoBERTaBase, used as a Pairwise Cross-Encoder, which is the standard
practice [Garg et al., 2019] and (ii) RoBERTaBase with our classification
heads but without MSPP pre-training. The latter allows us to evaluate how
RoBERTaBase without the specialized pre-training objective adapts at
processing $k+1$ input sequences.

On FEVER instead, we compare with GEAR [Zhou et al., 2019], KGAT [Liu et al.,
2020] and Transformer-XH [Zhao et al., 2020]. We also include 3 models from
Tymoshenko and Moschitti [2021]: (i) a RoBERTaBase for multi-sentence
classification with IE1 prediction head, (ii) two Pairwise Cross-Encoder
RoBERTaBase with heads that do max-pooling and weighted-sum to gather the
context for the classification.

We optimize our models with a maximum sequence length of $384$ tokens for all
experiments, which is $64$ multiplied by the number of sentences $k+1$.
Regarding ASNQ, we train the model for a maximum of 10 epochs using a learning
rate of $10^{-5}$ and a batch size of 512 examples, with the FuseAdam
optimizer. We warmup for only 5000 steps, and we use early stopping based on
the Mean Average Precision (MAP) on the development set. For WikiQA and TREC-
QA, we train the model using a learning rate of $2\cdot 10^{-6}$ and a batch
size of 32, with 1000 warmup steps. We train for up to 40 epochs and use early
stopping based on the MAP metric on the development set. Finally, for the
FEVER dataset, we use a batches of 64 example, a learning rate of $10^{-5}$,
and 1000 warmup steps. We use early stopping based on the Accuracy metric on
the development set.

We evaluate models’ performance on Answer Sentence Selection with Mean Average
Precision (MAP), Mean Reciprocal Rank (MRR) and Precision@1 (P@1), as in
similar works [Garg et al., 2019]. For the FEVER dataset, we measure
performance with Label Accuracy, which is a standard metric in Fact
Verification [Liu et al., 2020; Tymoshenko and Moschitti, 2021; Zarharan et
al., 2021]. It evaluates the accuracy in predicting whether a claim is
supported/refuted/neutral to the set of retrieved evidence sentences.

###  6.4 Results

In this Section, we report results obtained by fine-tuning the baselines and
our MSPP pre-trained models on AS2 and Fact Verification benchmarks. We report
results from original papers when available. On AS2, we analyze the results
after a standard fine-tuning as well as the results of chaining our Jointwise
models with a common Pairwise re-ranker. The statistical significance in AS2
is compared to RoBERTaBase [Liu et al., 2019b].

####  6.4.1 Answer Sentence Selection

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
RoBERTaBase Fixed (IEk) | 62.1 (0.8) | 65.6 (1.2) | 57.4 (1.3) |  | 39.4 (1.6) | 40.3 (1.8) | 19.7 (1.9) |  | 41.9 (2.4) | 50.8 (3.9) | 30.9 (5.4)
RoBERTaBase Fixed (AEk) | 62.3 (0.7) | 66.0 (1.0) | 57.7 (1.1) |  | 39.0 (2.8) | 39.7 (2.9) | 18.7 (3.9) |  | 42.3 (3.2) | 49.2 (5.0) | 29.7 (6.9)
RoBERTaBase Flexible (REk) | 63.7 (0.3) | 70.2 (0.4) | 58.9 (0.6) |  | 77.8 (2.3) | 79.6 (2.4) | 69.5 (3.4) |  | 84.8 (0.7) | 91.3 (1.7) | 84.8 (3.1)
RoBERTaBase [Liu et al., 2019b] | 66.9 (0.1) | 73.1 (0.1) | 61.8 (0.2) |  | 85.3 (0.9) | 86.5 (1.0) | 77.1 (2.1) |  | 89.3 (0.9) | 93.1 (1.0) | 87.9 (2.2)
RoBERTaBase Fixed (IEk) + MSPP |  67.2 (0.2) |  73.7 (0.2) |  63.0 (0.3) |  |  88.5 (1.5) |  89.0 (1.5) |  82.7 (2.2) |  |  91.1 (0.5) |  95.2 (1.3) |  91.7 (2.2)
RoBERTaBase Fixed (AEk) + MSPP |  67.3 (0.2) |  73.7 (0.2) |  63.0 (0.3) |  |  87.9 (1.4) |  89.0 (1.5) |  81.9 (2.6) |  | 90.1 (1.0) | 93.6 (0.6) | 88.7 (0.8)
RoBERTaBase Flexible (REk) + MSPP |  69.4 (0.0) |  75.7 (0.0) |  65.3 (0.1) |  |  88.7 (0.8) |  90.0 (0.8) |  83.5 (1.2) |  |  91.0 (0.7) |  95.0 (0.8) |  91.2 (1.2)
TandA RoBERTaBase [Garg et al., 2019] | - | - | - |  | 88.9 | 90.1 | - |  | 91.4 | 95.2 | -

Table 6.2: Results on ASNQ, WikiQA and TREC-QA. IEk, AEk and REk are the
prediction heads we use both for MSPP pre-training and fine-tuning. We use
bold formatting for our proposals and the best results. We underline
statistically significant improvements over RoBERTaBase.

Results on all the AS2 datasets are reported in Table 6.2. We experiment using
all multi-label heads defined in Section 6.2.3, i.e. IEk, AEk and REk. Results
are reported against a Pairwise baseline which uses the same Transformer
encoder and has the same parameters as our Joint models (which in some cases
have only a slightly larger classification head, with negligible effects on
fine-tuning/inference time). The Table shows two main discoveries: (i) models
fine-tuned directly on the downstream tasks in a multi-sentence prediction
setting do not achieve high re-ranking scores; (ii) MSPP pre-training is
fundamental to accustom the model to reason over multiple sentences.

Regarding the first point, the low performance of models fine-tuned directly
on the multi-sentence tasks can be explained by considering that attention
scores are not pre-trained to compare many sentences. Most pre-training
objectives focus on token-level dependencies (e.g. MLM, TD, SLM) while
sentence-level objectives such as NSP or SOP consider at most 2 input
sequences. This is further confirmed by the results on ASNQ, which is a large
dataset, and those on WikiQA and TREC-QA, which contain only a few thousand
examples. Results on ASNQ are higher in the standard fine-tuning setting
because the model can leverage the large quantity of annotated examples (about
4M, each containing 1 question and $k=5$ answer candidates) to learn multi-
sentence relations.

Regarding the Fixed models, both checkpoints pre-trained with MSPP and fine-
tuned with either IEk and AEk improve over the Pairwise baseline by a large
margin on WikiQA and TREC-QA. Specifically, the Joint Fixed model with IEk
classification head improves by 5.6 points in Precision@1 on WikiQA and by 3.8
points on TREC-QA. Similarly, the Joint model with AEk head outperforms the
baseline by 4.8 in Precision@1 on WikiQA and by 0.8 in TREC-QA. On ASNQ,
improvements are modest because the dataset is larger and reduces the
differences seen in pre-training, but are still consistent. In particular, the
Joint model with IEk and AEk show a gain in MAP of 0.3 and 0.4 points
respectively. When compared with Precision@1 instead, improvements are larger
and equal to 1.2 points for both models.

The Flexible model we present reaches the best results on ASNQ and WikiQA,
with a gap of 3.5 points in P@1 over the baseline on ASNQ and 6.4 points over
the WikiQA baseline. On TREC-QA, MAP and P@1 are aligned with the other Joint
models, with gains over the Pairwise baseline of 1.7 points in MAP and 3.3 in
Precision@1. To the best of our knowledge, results on ASNQ are the actual
state-of-the-art when compared with other models with a similar number of
parameters444In Chapter 7 we will show that large models can achieve even
higher scores.. This shows that using a Flexible architecture is overall the
best choice to design a multi-sentence inference model for Answer Sentence
Selection.

Finally, we compare with TandA, which is the actual state-of-the-art in AS2
for WikiQA and TREC-QA. Our methods demonstrate similar performance on both
datasets, with a maximum difference of 0.3 points when measured with MAP and
MRR. Notice that TandA exploits a large labeled dataset for AS2 (ASNQ) to
perform a pre-fine-tuning step and transfer knowledge into smaller datasets.
We use instead only raw data from large corpora without manual annotation.

#####  6.4.1.1 Re-Rankers Chain

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
RoBERTaBase [Liu et al., 2019b] | 66.9 (0.1) | 73.1 (0.1) | 61.8 (0.2) |  | 85.3 (0.9) | 86.5 (1.0) | 77.1 (2.1) |  | 89.3 (0.9) | 93.1 (1.0) | 87.9 (2.2)
RoBERTaBase Fixed (IEk) + MSPP |  71.3 (0.4) | 73.1 (0.5) |  63.9 (0.8) |  |  88.5 (1.4) |  89.0 (1.5) |  82.7 (3.0) |  |  93.5 (1.2) |  95.4 (1.4) |  92.2 (1.8)
RoBERTaBase Fixed (AEk) + MSPP |  71.5 (0.3) |  73.4 (0.3) |  64.3 (1.1) |  |  87.9 (0.8) |  88.7 (0.7) |  82.1 (1.1) |  |  93.5 (1.1) |  94.9 (1.2) |  91.2 (1.4)
RoBERTaBase Flexible (REk) + MSPP |  70.7 (0.5) | 72.5 (0.6) |  64.1 (1.0) |  |  88.1 (1.1) |  89.1 (1.2) |  82.8 (1.9) |  | 90.7 (0.7) | 92.2 (0.8) | 89.7 (1.2)
Superior limit | 86.5 |  | 97.9 |  | 100.0

Table 6.3: Results on ASNQ, WikiQA and TREC-QA for Joint Fixed and Flexible
models when chained after the RoBERTaBase Pairwise re-ranker. In the last row,
we show the maximum achievable performance based on the Hit-Rate at $k=5$
(HR@5) of the Pairwise re-ranker. We use bold formatting for our proposals and
the best results. We underline statistically significant improvements over
RoBERTaBase.

In this Section, we study whether Jointwise models can be chained with a
Pairwise Cross-Encoder to improve the performance of the latter. More
specifically, we use a RoBERTaBase Cross-Encoder to find the top $k$ more
relevant answers for each question. After that, we apply the Jointwise model
to better re-rank the top $k$ answers retrieved by the baseline Cross-Encoder.

Results are reported in Table 6.3. The last row of the Table shows the upper
limit to Jointwise model performance. The limit is calculated as the Hit-
Rate@k of the Pairwise re-ranker. This comparison is needed because if the
Pairwise baseline is not able to rank at least one positive candidate in the
top $k$, Jointwise encoders will process only negative examples and will
receive a bad score by the evaluation metrics.

In this experiment, Fixed models provide better overall performance,
especially on the TREC-QA dataset. On the other hand, the Flexible model
provides similar performance to Fixed models on ASNQ and WikiQA, but falls
short behind on TREC-QA, still providing a better re-ranking of the first $k$
candidates than the baseline Pairwise Cross-Encoder. On average, performance
is lower than applying Jointwise models directly to the whole datasets: we
suppose that our models work better when applied to all data because more
candidates can be compared with each other. Moreover, the accuracy of
Jointwise models when applied after a Pairwise re-ranker is limited by the
latter’s performance. This is further confirmed by checking the results on
TREC-QA, in which most Jointwise models perform better than the previous
setting because the Pairwise re-ranker was able to score always at least a
positive candidate in the top $k$ (HR@k = 100.0%).

####  6.4.2 Fact Verification

Model |  | FEVER
---|---|---
| L.A. Dev | L.A. Test
GEAR [Zhou et al., 2019] |  | 70.69 | 71.60
DOMLIN [Stammbach and Neumann, 2019] |  | 71.54 | -
KGAT with RoBERTaBase [Liu et al., 2020] |  | 78.29 | 74.07
Transformer-XH [Zhao et al., 2020] |  | 78.05 | 72.39
RoBERTaBase Pairwise + MaxPool [Tymoshenko et al., 2021] |  | 79.82 | -
RoBERTaBase Pairwise + WgtSum [Tymoshenko et al., 2021] |  | 80.01 | -
DOMLIN++ [Stammbach and Ash, 2020-10] |  | 77.48 | 76.60
DREAM [Zhong et al., 2020] |  | 79.16 | 76.85
RoBERTaBase Fixed (IE1) |  | 78.92 (0.46) | 73.01
RoBERTaBase Fixed (AE1) |  | 78.85 (0.36) | 72.94
RoBERTaBase Flexible (IE1) |  | 79.25 (0.27) | 73.56
RoBERTaBase Fixed (IE1) + MSPP (IEk) |  | 81.21 (0.24) | 74.39
RoBERTaBase Fixed (AE1) + MSPP (AEk) |  | 81.21 (0.16) | 74.13
RoBERTaBase Flexible (IE1) + MSPP (REk) |  |  82.10 (0.10) | 75.36

Table 6.4: Results on FEVER dev and test sets. We use the prediction heads IE1
and AE1 for fine-tuning and either IEk, AEk or REk for the MSPP pre-training.
‘-’ means results are not reported in the original paper. We use bold
formatting for our proposals and the best results.

We present results on the Fact Verification task by comparing our models
against many baselines from the literature. Table 6.4 reports the Label
Accuracy on the development and test sets of the FEVER dataset.

Flexible models without our ad-hoc pre-training task reach remarkable
performance on both the development and test set, obtaining a Label Accuracy
higher than some previous state-of-the-art systems. On the other hand, when
our models are further trained with the MSPP objective, we obtain state-of-
the-art results among models based on a Base architecture (about 120M of
parameters). In particular, by testing models on the Fever Shared Task
Leaderboard, we obtain a significant gain of 1.29 points when using our
Jointwise Flexible model with MSPP. On the development set, all of our Joint
models outperform the baselines by a margin of at least 1.20 points in Label
Accuracy.

Finally, notice that our Joint Flexible model achieves a score slightly below
the actual state-of-the-art on FEVER: DREAM and DOMLIN++. However, those
models use a better retrieval system (we are limited by the performance of the
BERT-based Doc-IR) and larger models for the Fact Verification task. In
particular, DREAM Fact Verification systems is based on an XLNetLarge [Yang et
al., 2019] model while DOMLIN++ uses a RoBERTaLarge [Liu et al., 2019b]
architecture. Both models contain about 360M parameters, which is much more
than the 124M parameters of our Jointwise RoBERTaBase.

###  6.5 Latency Analysis

Although our Joint model allows for longer input sequences in the Transformer,
it also decreases the number of forward passes required by an equivalent
Pairwise Cross-Encoder. To provide a simplified analysis of the latency in
AS2, let $L$ be the length of the longest sentence. In the case of the
Pairwise Cross-Encoder, $k$ forward passes (one for each candidate $c_{i}$) of
the Transformer are needed with a sequence length of $2L$. However, our Joint
model only requires a single forward pass over the Transformer with an input
length of $L(k+1)$ (one pass with the query and $k$ answer candidates).

Since the Transformer’s Self-Attention has a quadratic relationship with the
input sequence length, we can expect the inference time of our Joint model to
be approximately $O((k+1)^{2}L^{2})$, and $O(4L^{2}k)$ for the Pairwise Cross-
Encoder. Thus, the ratio between the inference time of our Joint model and the
Pairwise Cross-Encoder is:

$r=\frac{O((k+1)^{2}L^{2})}{O(4L^{2}k)}=O(k)$ (6.8)

However, $k$ and $L$ are usually small numbers and the Transformer contains
also many affine transformations and an embedding layer that scale linearly in
the input sequence length. We observe empirically that when fine-tuning on the
AS2 datasets with a batch size of 32, there is only a slight increase in the
average inference latency equal to 14.1% for Fixed and 9% for Flexible models.
Flexible models are faster because they allow to shrink batches dynamically
when the input examples are all shorter than the allowed maximum sequence
length.

##  Chapter 7 Self-Supervised Objectives for AS2

In this Chapter, we study pre-training sentence-level objectives specifically
designed for Answer Sentence Selection. Annotating large datasets is time-
consuming and expensive (because of the large set of possible answer
candidates retrieved for each question). For this reason, we aim at pre-
training Transformer models that could be adapted to various AS2 datasets even
if they contain a reduced number of training examples.

We propose three novel self-supervised sentence-level objectives that could be
used in pre-training along with MLM to gather paragraph-level semantics from
single or multiple documents into the model. The objectives we propose do not
need additional expensive labeled datasets (such as in Garg et al. [2019]),
but leverage semi-structured knowledge from large pre-training corpora such as
Wikipedia or the CommonCrawl. In order to align with the sentence-pair
structure of the Answer Sentence Selection task, our pre-training objectives
are specifically designed to function with a pair of input text sequences.

The first objective can be summarized as predicting whether two random spans
of text are extracted from the same paragraph of a document. The second
objective instead provides the model with a paragraph and a sentence and
should predict whether the sentence belongs to that paragraph in the original
document. Finally, the third objective tasks the model to predict whether two
paragraphs are extracted from the same original document.

We provide a detailed overview of the Transformer models used in this Chapter
in Section 3.1. Moreover, works related to the task we address here can be
found in Section 3.2.1, while for similar techniques that exploit the weak
supervision of raw documents, the reader should refer to Section 3.3.3.

By doing continuous pre-training starting from different pre-trained
checkpoints such as RoBERTa [Liu et al., 2019b] and ELECTRA [Clark et al.,
2020], we show that raw data can be exploited to increase final accuracy on
different AS2 datasets. The next Section examines the reason why common pre-
training tasks are not effective at structuring information from large pre-
training corpora. Next, we describe our alternative objectives in detail.
Then, we present the experimental setting and the results of our objectives
when compared with common Transformer baselines. To help future research, we
report a set of negative results in Appendix D.1. Finally, we show how we
achieve new state-of-the-art results on some datasets by continuously pre-
training larger models with our newly proposed tasks.

###  7.1 The issue with common Objectives

Several Transformer models, including BERT, RoBERTa, ELECTRA, ALBERT, DeBERTa,
and others, are trained using token-level objectives, sometimes supplemented
with sentence-level tasks. BERT, RoBERTa, ALBERT, and DeBERTa primarily employ
the Masked Language Modeling (MLM) objective as their main training strategy,
while ELECTRA focuses on the Token Detection task. BERT also incorporates the
Next Sentence Prediction (NSP) objective to enhance performance, particularly
in classification tasks involving two sentences, such as Natural Language
Inferencing, AS2, or Textual Entailment. However, Liu et al. [2019b]
demonstrated that by extending the training duration, NSP can be omitted
without compromising final accuracy. Additionally, Lan et al. [2020] indicated
that NSP is a trivial task with easily attainable high accuracies, resulting
in weak error signals for the model. Consequently, they proposed Sentence
Order Prediction (SOP) as a more challenging alternative to NSP.

In this Chapter, we demonstrate that token-level objectives are good at
capturing short-range dependencies in the input text, but are not optimal for
reasoning over inputs composed of separated spans of text, e.g., a question
and an answer. Most masked tokens in MLM or replaced tokens in TD can be
correctly predicted by looking at the surrounding context [Zaheer et al.,
2020; Beltagy et al., 2020]. Moreover, we show that NSP and SOP are not
challenging enough to train the model at reasoning over two input spans of
text and do not align well with the final downstream AS2 task.

###  7.2 Alternative Objectives for AS2

Large sources of data such as Wikipedia, the CommonCrawl or the Pile are
collections of documents, and each document is divided into a list of
paragraphs by humans. The paragraphs of a single document generally describe
the same argument from different perspectives. The objectives we propose in
this Chapter exploit this weak supervision to create 3 self-supervised tasks
that do not need manually annotated data.

We define pre-training corpus ${\mathbb{C}}$ in the following way:

  * •

the corpus is a set containing millions of documents:
${\mathbb{C}}=\\{{\bm{D}}_{1},\dots,{\bm{D}}_{k}\\}$;

  * •

each document is a sequence of paragraphs: ${\bm{D}}_{i}=[P_{1},\dots,P_{n}]$,
where $n$ is the number of paragraphs in ${\bm{D}}_{i}$;

  * •

paragraphs are lists of sentences: $P_{j}=[s_{1},\dots,s_{m}]$, where $m$ is
the number of sentences in $P_{j}$.

We use the notation $s_{ab}$ to indicate the subsequence of a paragraph
$P_{j}$ that starts from sentence $a$ and ends with sentence $b$,
$s_{ab}=s_{a}\cdot\ldots\cdot s_{b}$ (we use “$\cdot$”to denote text
concatenation). Then, we define our objectives as follows.

####  7.2.1 Spans in the Same Paragraph (SSP)

Figure 7.1: Example of SSP objective.

The first task consists in detecting whether two spans of text $s_{ab}$ and
$s_{cd}$ are extracted from the same paragraph $P_{j}\in{\bm{D}}_{i}$. More
formally, given two subsequences $s_{ab},s_{cd}$ such that $s_{ab}\cap
s_{cd}=\emptyset$, the task is to predict whether $s_{ab},s_{cd}\in P_{j}$. We
create positive example pairs by sampling two disjoint sequences $s_{ab}$ and
$s_{cd}$ from the same paragraph $P_{j}$ of some document ${\bm{D}}_{i}$.
Then, we create $k_{1}$ hard negatives by sampling $s_{ab}\in P_{j}$ and
$s_{cd}\in P_{l}$ from two different paragraphs in the same document, i.e
$P_{j},P_{l}\in{\bm{D}}_{i}$ and $j\neq l$. This forces the model at
recognizing that the general topic of the text spans is the same but described
from different viewpoints. Finally, we create $k_{2}$ easy negatives by
sampling $s_{ab}\in P_{j}$ and $s_{cd}\in P_{l}$ such that
$P_{j}\in{\bm{D}}_{i},P_{l}\in{\bm{D}}_{f}$ and $i\neq f$.

The task is designed to mimic the input of Transformer models for the Answer
Sentence Selection task, which is composed of a question and a candidate
answer. The question corresponds to $s_{ab}$ while the answer is replaced by
$s_{cd}$. Since $s_{ab}$ and $s_{cd}$ are two sequences of sentences in some
paragraphs, we describe the sampling method we use to define their length.
Questions in most cases are composed of a single sentence, with rare peaks of
up to three sentences. Thus, we sample the length of $s_{ab}$ in the interval
$[1,3]$, with weighted probabilities of 70%, 20% and 10%, respectively. Then,
since most answers in the AS2 datasets are composed of one to five sentences,
we randomly sample the length of $s_{cd}$ in $[1,5]$ with probabilities of 14%
for 1 and 5, and 24% for the other lengths.

Regarding the number of hard and easy negatives sampled for each positive
pair, we try to obtain $k_{1}=2$ hard negatives from within the same paragraph
and $k_{2}=2$ easy negatives from paragraphs in other documents. When there
are not enough sentences in the paragraph to create $k_{1}$ hard negatives, we
force $k_{1}+k_{2}=4$ by sampling more easy negatives. An example of the SSP
objective is depicted in Figure 7.1.

####  7.2.2 Spans in Paragraph (SP)

Figure 7.2: Example of SP objective.

SP is a task that challenges the model at recognizing whether a span of text
$s_{ab}$ is extracted from some paragraph $P_{j}\in{\bm{D}}_{i}$. More
specifically, we divide every paragraph $P_{j}$ in the corpus into two subsets
$s_{ab}$ and $P_{j}\setminus s_{ab}$. Then, we create positive example pairs
by sampling $s_{ab}$ and $P_{j}\setminus s_{ab}$, with $s_{ab}\in P_{j}$. For
the hard negatives, we use the same $s_{ab}\in P_{j}$ but paired with other
reduced paragraphs $P_{l}\setminus s_{cd}$ in the same document, such that
$P_{j},P_{l}\in{\bm{D}}_{i}$ and $j\neq l$. This should force the model at
recognizing that the topic is the same but addressed from different
perspectives. Finally, we create easy negatives by pairing $s_{ab}\in P_{j}$
with reduced paragraphs $P_{l}\setminus s_{cd}$ sampled from other documents,
i.e. $P_{j}\in{\bm{D}}_{i},P_{l}\in{\bm{D}}_{f}$ such that $i\neq f$.

As in SSP, we define a sampling strategy for the length of $s_{ab}$. Since
$s_{ab}$ represents the question when fine-tuning on the downstream task, we
apply the same rule of SSP, by sampling the length of $s_{ab}$ in the interval
$[1,3]$, with weighted probabilities of 70%, 20% and 10%. The second part is
always the text that remains in paragraph $P_{i}$ after the split, and its
length may vary considerably from one corpus to another. For example,
Wikipedia has longer paragraphs than CC-News and OpenWebText on average,
leading to longer pairs. We do not set any constraint on the length of the
right part of the pair because the tokenizer will take care of truncating
sequences that exceed the maximum sequence length allowed.

As in SSP, we sample up to $k_{1}=2$ hard negatives from within the same
document and then we sample $k_{2}$ easy negatives until $k_{1}+k_{2}=4$. An
example of the SP objective is depicted in Figure 7.2.

####  7.2.3 Paragraphs in the Same Document (PSD)

The last new pre-training task we propose asks the model to classify whether
two entire paragraphs are sampled from the same document. Formally, positive
pairs are created by sampling two paragraphs $P_{j}$ and $P_{l}$ such that
$P_{j},P_{l}\in{\bm{D}}_{i}$ and $j\neq l$. In PSD there is no distinction
between easy and hard negatives because there is no way of creating negatives
from within the same document. We create negative pairs by sampling $P_{j}$
and $P_{l}$ from different documents, i.e. $P_{j}\in{\bm{D}}_{i}$ and
$P_{l}\in{\bm{D}}_{f}$ given $i\neq f$.

We do not crop paragraphs to reduce their length because this task will be
performed by the tokenizer, by iteratively removing one token from the longest
sequence in the pair until the total length is within the limit. For each
positive pair, we create $k=4$ additional negative examples. An example of PSD
is given in Figure 7.3.

Figure 7.3: Example of PSD objective.

###  7.3 Experiments

In this Section, we describe the experimental setting used to evaluate the
effectiveness of our pre-training objectives.

####  7.3.1 Continuous Pre-Training

Model | Steps | LR | BS | MSL | # Tokens | Effort | Objectives
---|---|---|---|---|---|---|---
RoBERTaBase \+ SSP | 400K | $1\cdot 10^{-4}$ | 4096 | 128 | 210B | +2.5% | MLM (1.0) + SSP (1.0)
RoBERTaBase \+ SP | 200K | $1\cdot 10^{-4}$ | 4096 | 256 | 210B | +5.0% | MLM (1.0) + SP (1.0)
RoBERTaBase \+ PSD | 200K | $1\cdot 10^{-4}$ | 4096 | 256 | 210B | +5.0% | MLM (1.0) + PSD (1.0)
RoBERTaBase \+ ALL | 400K | $1\cdot 10^{-4}$ | 4096 | 128 | 210B | +2.5% | MLM (1.0) + ALL (1.0)
ELECTRABase \+ SSP | 400K | $1\cdot 10^{-4}$ | 4096 | 128 | 210B | +5.0% | MLM (1.0) + TD (50.0) + SSP (1.0)
ELECTRABase \+ SP | 200K | $1\cdot 10^{-4}$ | 4096 | 256 | 210B | +10.0% | MLM (1.0) + TD (50.0) + SP (1.0)
ELECTRABase \+ PSD | 200K | $1\cdot 10^{-4}$ | 4096 | 256 | 210B | +10.0% | MLM (1.0) + TD (50.0) + PSD (1.0)
ELECTRABase \+ ALL | 400K | $1\cdot 10^{-4}$ | 4096 | 128 | 210B | +5.0% | MLM (1.0) + TD (50.0) + ALL (1.0)
DeBERTaBase \+ ALL | 100K | $1\cdot 10^{-4}$ | 4096 | 128 | 52B | +1.25% | MLM (1.0) + SP (1.0)
DeBERTaLarge \+ ALL | 100K | $1\cdot 10^{-4}$ | 4096 | 128 | 52B | +1.25% | MLM (1.0) + ALL (1.0)
DeBERTaV3Base \+ ALL | 100K | $1\cdot 10^{-4}$ | 4096 | 128 | 52B | +0.625% | MLM (1.0) + TD (50.0) + ALL (1.0)
DeBERTaV3Large \+ ALL | 100K | $1\cdot 10^{-4}$ | 4096 | 128 | 52B | +0.625% | MLM (1.0) + TD (50.0) + ALL (1.0)

Table 7.1: Hyper-parameters used for the continuous pre-training of several
Transformer models with our specialized objectives. The “Effort” column
indicates the amount of additional FLOPS required compared to the original
pre-training. “MSL” is the maximum allowed input length. “LR” is the learning
rate while “BS” refers to the training batch size. The last column describes
the used training tasks along with their weight (in round brackets).

We do continuous pre-training starting from RoBERTa, ELECTRA, DeBERTa and
DeBERTaV3 checkpoints. We apply all of our objectives to RoBERTa and ELECTRA,
while for DeBERTa and DeBERTaV3, since they are more expensive to be
continuously pre-trained, we apply only the objectives that show more
consistent performance across all tasks, even though it may not always be the
best. We also experiment by mixing the objectives and challenging the model to
automatically identify the task and predict the correct label. We mix the
datasets for the continuous pre-training at the example level, thus a batch
can contain samples from both SSP, SP or PSD, with equal probability. The
other pre-training parameters are the same as SSP. Those experiments are
indicated with “ALL” in the following Sections.

To avoid improvements deriving from the usage of more data, we exploit the
same corpora used for the original pre-training of the models. In particular,
we do continuous pre-training with SSP, SP and PSD on English Wikipedia,
BookCorpus, OpenWebText and CC-News. For ELECTRA, the BookCorpus is not
included in the original pre-training, however, it accounts for less than 0.5%
of the training examples. After the pre-processing of the corpora with the
techniques described above, we obtain about 420M examples for SP, 260M for SP
and 200M for PSD.

In Table 7.1, we report the hyper-parameters used for each continuous pre-
training experiment. For every model, we combine our objectives with the
original tasks, i.e. MLM for RoBERTa and DeBERTa and TD for ELECTRA and
DeBERTaV3. We found this beneficial since our long continuous pre-training may
reduce models’ generalization by only performing binary classification on the
output embedding of the first token.

Regarding the length of the continuous pre-training, we also report the total
number of tokens provided to each model. For reference, RoBERTaBase and
DeBERTaV3Base were pre-trained over about 2000B tokens, ELECTRABase over more
than 500B tokens and DeBERTaBase over about 1000B tokens. DeBERTaLarge and
DeBERTaV3Large were trained over the same amount of tokens of the
corresponding Base versions. For more details about the original pre-trained
models, refer to Section 3.1. Notice that for DeBERTaV3 models, we couldn’t
exploit the original generators for Token Detection, as they were not
released. We used DeBERTaSmall and DeBERTaV3Base instead as the generators for
DeBERTaV3Base and DeBERTaV3Large, respectively. We don’t think this issue has
noticeable effects on final accuracy. The column “Effort” of the table shows
the effective amount of computational FLOPS required for our continuous pre-
training compared to the original model training. This ratio is computed by
taking into account the number of total training steps, the batch size and the
maximum sequence length. Notice that the Attention Mechanism complexity is
quadratic in the input length. Thus, by training with a limit of 128 or 256,
we reduce the computational complexity, and thus training time, by
$\frac{1}{16}$ and $\frac{1}{4}$ respectively (original models were all
trained with a sequence length of 512). To further demonstrate that our
achievements do not stem from this additional short pre-training, we compare
models continuously pre-trained with and without our objectives in Section
7.4.3.

Regarding the optimization and the hardware, we set $\beta_{1}=0.9$ and
$\beta_{2}=0.99$, the weight decay to $0.01$ and we use a triangular learning
rate with 10% warmup steps. We trained all models on our machine (see Section
1.2 for more details), and each experiment required about 2.5 days of
continuous pre-training.

#####  7.3.1.1 Comparison of objectives hardness

Here we compare common sentence-level objectives with our proposals in terms
of final accuracy. As explained before, we state that NSP [Devlin et al.,
2019], SOP [Lan et al., 2020], SSO [Wang et al., 2019b] and other sentence-
level tasks are trivial and do not provide meaningful signals to the model. In
Table 7.2 we report the accuracy on the development set after the continuous
pre-training with our proposed objectives (SSP, SP and PSD) and the
performance of other sentence-level tasks from the literature. Since we don’t
have access to the pre-training development sets of BERT, ALBERT and
StructBERT, the performance of NSP, SOP and SSO are measured over a small
random shard of OpenWebText and CC-News, which were not used in the pre-
training of those two models. Thus, we expect slightly higher performance on
the original data because here they are tested out of domain.

The results indicate that performing sentence-level predictions is more
challenging for SSP and SP compared to NSP, SOP and SSO. Surprisingly, solving
PSD proves to be even more difficult, despite having access to two large
paragraphs for predictions, unlike SSP and SP, which only rely on two small
text spans. Notably, we find it intriguing that the accuracy of SSP and SP is
similar. This similarity can be attributed to the fact that the two tasks are
not significantly different in practice, considering that the average
paragraph length in the pre-training corpora is 1.4 sentences. Consequently,
after filtering the paragraphs that are too small, it frequently occurs that
the remaining portion of the paragraph in SP, after removing the text span, is
merely another short text span, similar to the case in SSP.

Model | Accuracy | F1
---|---|---
RoBERTaBase \+ SSP | 91.8 | 83.1
ELECTRABase \+ SSP | 90.4 | 79.9
RoBERTaBase \+ SP | 91.3 | 83.3
ELECTRABase \+ SP | 89.9 | 80.1
RoBERTaBase \+ PSD | 83.5 | 61.4
ELECTRABase \+ PSD | 82.3 | 57.1
BERTBase [Devlin et al., 2019] (NSP) | 96.9 | 97.1
ALBERTBase [Lan et al., 2020] (SOP) | 93.7 | 94.7
StructBERT [Wang et al., 2019b] (SSO) | 92.3 | 93.4

Table 7.2: Accuracy and F1-score of several objectives on the pre-training
validation set.

####  7.3.2 Fine-Tuning

To demonstrate the superiority of our approach, we fine-tune all the models
listed in Table 7.1 on 6 different datasets for Answer Sentence Selection.
Specifically, we use ASNQ, WikiQA, TREC-QA, WQA, NewsAS2 and TriviaAS2, which
are all described in Section 4. We use the same hyper-parameters applied in
the continuous pre-training apart from the learning rate, the batch size, the
number of warmup steps of the triangular learning rate and the total number of
epochs. In Table 7.3 we provide a summary of the search space of all hyper-
parameters divided by dataset. For every experiment, the number of warmup
steps is computed to match the number of training steps in the first epoch. We
optimize the hyper-parameters using the development set and using only the
baselines, thus the best hyper-parameters for the models continuously pre-
trained with our objectives may vary slightly. We do not conduct an extensive
grid search because it is too expensive.

Performance of each model is measured with common metrics for AS2: Mean
Average Precision, Mean Reciprocal Rank and Precision@1, as in related works
[Lauriola and Moschitti, 2021; Garg et al., 2019]. In particular, in the
description of the results we focus on Precision@1 because this is the rate at
which a correct answer is returned to the user.

Parameter | ASNQ | WikiQA & TREC-QA | WQA | NewsAS2 & TriviaAS2
---|---|---|---|---
Batch Size | $\\{1024,2048\\}$ | $\\{16,32,64,128\\}$ | $\\{64,128,256,512\\}$ | $\\{128,256,512\\}$
Learning Rate | $\\{1\cdot 10^{-5},2\cdot 10^{-5}\\}$ | $\\{5\cdot 10^{-6},1\cdot 10^{-5},2\cdot 10^{-5}\\}$ | $\\{5\cdot 10^{-6},1\cdot 10^{-5},2\cdot 10^{-5}\\}$ | $\\{1\cdot 10^{-5},2\cdot 10^{-5}\\}$
Max epochs | 6 | 40 | 20 | 10

Table 7.3: Hyper-parameters search space for the fine-tuning over several AS2
datasets.

###  7.4 Results

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
TandA RoBERTaBase ♣ | - | - | - |  | 88.5 (0.8) | 89.9 (0.8) | 83.0 (1.3) |  | 90.1 (0.4) | 94.1 (0.6) | 89.7 (0.7)
CT RoBERTaBase [Soldaini et al., 2020] | 66.3 | 65.4 | 53.2 |  | 89.9 | 91.0 | - |  | 92.4 | 96.7 | -
TandA RoBERTaBase [Garg et al., 2019] | - | - | - |  | 88.9 | 90.1 | - |  | 91.4 | 95.2 | -
RLAS-BIABC [Gharagozlou et al., 2022] | - | - | - |  | 88.8 (3.6) | 89.1 (1.7) | - |  | 89.8 (1.5) | 90.6 (9.2) | -
RoBERTaBase [Liu et al., 2019b] | 66.9 (0.1) | 73.1 (0.1) | 61.8 (0.1) |  | 85.8 (1.3) | 87.2 (1.3) | 78.3 (2.8) |  | 89.7 (0.7) | 94.4 (1.1) | 90.0 (1.9)
RoBERTaBase \+ SSP |  68.1 (0.2) |  74.5 (0.3) |  64.1 (0.3) |  |  88.7 (0.3) |  89.9 (0.4) |  82.9 (0.7) |  | 89.3 (0.7) | 93.6 (0.6) | 88.5 (1.2)
RoBERTaBase \+ SP |  68.3 (0.1) |  74.5 (0.2) |  64.1 (0.2) |  |  87.7 (0.3) |  88.9 (0.4) | 81.0 (0.8) |  | 90.1 (0.8) | 94.7 (1.3) |  90.9 (2.6)
RoBERTaBase \+ PSD |  67.7 (0.2) |  73.7 (0.3) |  62.6 (0.4) |  | 86.4 (1.1) | 88.0 (1.0) | 80.5 (1.6) |  |  90.3 (0.5) |  95.1 (0.7) | 90.3 (1.3)
RoBERTaBase \+ ALL |  68.0 (0.1) |  74.1 (0.2) |  63.9 (0.4) |  |  88.2 (0.4) |  89.5 (0.4) |  82.5 (0.9) |  | 89.3 (0.7) | 93.4 (0.6) | 87.9 (1.2)
TandA ELECTRABase ♣ | - | - | - |  | 90.2 (0.8) | 91.4 (0.7) | 85.6 (1.1) |  | 91.6 (0.7) | 95.5 (0.7) | 92.6 (1.5)
ELECTRABase [Clark et al., 2020] | 67.5 (0.2) | 73.6 (0.2) | 62.4 (0.4) |  | 85.0 (2.6) | 86.5 (2.7) | 77.1 (4.0) |  | 89.9 (0.4) | 94.0 (0.9) | 90.3 (1.7)
ELECTRABase \+ SSP |  69.7 (0.2) |  75.7 (0.2) |  65.3 (0.3) |  |  88.6 (1.4) |  90.0 (1.4) |  82.5 (2.0) |  | 89.6 (0.7) | 93.5 (0.9) | 88.5 (1.9)
ELECTRABase \+ SP |  69.0 (0.1) |  75.1 (0.1) |  65.0 (0.2) |  |  88.1 (1.5) | 89.5 (1.5) | 81.8 (2.3) |  | 90.3 (0.7) | 94.6 (0.7) | 91.2 (1.5)
ELECTRABase \+ PSD |  68.9 (0.3) |  75.1 (0.3) |  65.3 (0.4) |  | 85.6 (0.7) | 87.3 (0.6) | 78.6 (0.7) |  | 87.9 (1.1) | 92.2 (1.1) | 85.9 (2.2)
ELECTRABase \+ ALL |  69.3 (0.2) |  75.2 (0.2) |  65.0 (0.2) |  |  87.3 (1.2) | 88.7 (1.1) | 80.8 (1.9) |  |  90.4 (0.4) |  95.5 (1.0) |  92.6 (1.8)

Table 7.4: Results of our RoBERTaBase and ELECTRABase models on ASNQ, WikiQA
and TREC-QA. We report several baselines such as TandA, Cascade Transformer
(CT) and RLAS-BIABC. $\clubsuit$: our versions of TandA. We use bold
formatting for our proposals and the best results. We underline statistically
significant improvements over either RoBERTaBase or ELECTRABase.

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
DeBERTaBase [He et al., 2020] | 68.2 (0.1) | 74.4 (0.1) | 64.4 (0.3) |  | 88.0 (2.1) |  89.3 (1.9) | 81.6 (3.1) |  | 89.9 (1.2) | 94.1 (1.2) | 89.7 (2.5)
DeBERTaBase \+ ALL |  69.9 (0.2) |  76.3 (0.2) |  66.9 (0.3) |  |  88.2 (1.0) |  89.3 (0.9) |  82.4 (1.7) |  |  91.1 (0.9) |  94.6 (1.1) |  90.7 (2.2)
DeBERTaV3Base [He et al., 2021] | 70.5 (0.1) | 77.1 (0.1) | 67.8 (0.2) |  | 88.9 (2.3) | 90.1 (2.1) | 83.0 (3.6) |  | 90.0 (0.3) | 93.2 (0.6) | 89.1 (0.8)
DeBERTaV3Base \+ ALL |  71.0 (0.2) |  77.2 (0.2) |  68.2 (0.3) |  |  89.5 (1.4) |  90.7 (1.5) |  84.3 (2.2) |  |  91.3 (0.9) |  94.1 (1.2) |  91.5 (1.2)

Table 7.5: Results of our DeBERTaBase and DeBERTaV3Base models on ASNQ, WikiQA
and TREC-QA. We underline statistically significant improvements over the
corresponding baselines.

Model | WQA |  | NewsAS2 |  | TriviaAS2
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
TandA RoBERTaBase ♣ | 69.1 (0.1) | 78.0 (0.2) | 66.2 (0.2) |  | 82.3 (0.2) | 85.1 (0.3) | 76.1 (0.5) |  | 77.3 (0.5) | 83.0 (0.6) | 75.5 (0.6)
CT RoBERTaBase [Soldaini et al., 2020] | 57.8 | 76.9 | 67.5 |  | - | - | - |  | - | - | -
RoBERTaBase [Liu et al., 2019b] | 68.4 (0.1) | 77.3 (0.2) | 65.4 (0.2) |  | 82.4 (0.2) | 85.2 (0.3) | 76.4 (0.6) |  | 76.9 (0.6) | 82.2 (0.5) | 73.1 (0.5)
RoBERTaBase \+ SSP |  69.2 (0.1) |  77.7 (0.1) |  65.7 (0.2) |  | 82.3 (0.3) | 85.3 (0.3) | 76.5 (0.4) |  |  77.5 (0.3) |  82.7 (0.4) |  74.3 (0.6)
RoBERTaBase \+ SP |  69.4 (0.1) |  78.0 (0.2) |  66.0 (0.4) |  |  82.7 (0.4) |  85.6 (0.4) |  77.2 (0.7) |  | 77.1 (0.2) |  82.7 (0.3) |  74.0 (0.5)
RoBERTaBase \+ PSD |  69.4 (0.1) |  78.0 (0.1) |  66.0 (0.2) |  | 82.4 (0.2) | 85.0 (0.2) | 76.3 (0.3) |  | 76.0 (0.1) | 81.5 (0.1) | 72.2 (0.1)
RoBERTaBase \+ ALL |  69.5 (0.2) |  78.2 (0.1) |  66.2 (0.3) |  | 82.2 (0.4) | 85.1 (0.3) | 76.2 (0.7) |  | 77.0 (0.3) | 82.4 (0.5) | 73.6 (0.6)
TandA ELECTRABase ♣ | 70.7 (0.1) | 79.5 (0.1) | 68.2 (0.1) |  | 82.3 (0.4) | 85.1 (0.5) | 76.4 (0.9) |  | 74.2 (0.2) | 80.1 (0.1) | 70.6 (0.2)
ELECTRABase [Clark et al., 2020] | 70.1 (0.2) | 78.6 (0.2) | 66.9 (0.3) |  | 82.0 (0.2) | 84.8 (0.2) | 76.0 (0.2) |  | 73.3 (0.7) | 79.1 (1.1) | 68.9 (1.3)
ELECTRABase \+ SSP |  70.6 (0.0) |  79.2 (0.1) |  67.5 (0.2) |  | 82.2 (0.2) | 85.0 (0.2) | 76.2 (0.4) |  |  74.7 (0.5) |  80.4 (0.2) |  70.8 (0.3)
ELECTRABase \+ SP |  70.6 (0.1) |  79.2 (0.2) |  67.5 (0.3) |  |  82.4 (0.1) |  85.2 (0.2) | 76.5 (0.3) |  |  74.5 (0.3) | 80.0 (0.4) | 70.3 (0.8)
ELECTRABase \+ PSD |  70.7 (0.1) |  79.2 (0.2) |  67.8 (0.4) |  |  82.5 (0.2) |  85.4 (0.2) |  76.8 (0.3) |  |  74.1 (0.3) | 79.8 (0.3) | 70.2 (0.3)
ELECTRABase \+ ALL |  70.7 (0.0) |  79.4 (0.0) |  67.6 (0.1) |  |  82.5 (0.3) |  85.5 (0.3) |  76.9 (0.5) |  |  74.8 (0.5) |  80.7 (0.6) |  71.4 (0.9)

Table 7.6: Results of our RoBERTaBase and ELECTRABase models on WQA, NewsAS2
and TriviaAS2. $\clubsuit$: our versions of TandA. We use bold formatting for
our proposals and the best results. We underline statistically significant
improvements over either RoBERTaBase or ELECTRABase.

Model | WQA |  | NewsAS2 |  | TriviaAS2
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
DeBERTaBase [He et al., 2020] | 69.5 (0.0) | 78.5 (0.2) | 66.6 (0.3) |  |  82.9 (0.5) | 85.5 (0.5) | 77.0 (0.8) |  | 77.8 (0.2) | 83.3 (0.4) | 75.1 (0.4)
DeBERTaBase \+ ALL |  69.7 (0.1) |  78.6 (0.1) |  68.0 (0.2) |  |  82.9 (0.3) |  85.7 (0.4) |  77.3 (0.6) |  |  79.5 (0.3) |  84.2 (0.5) |  76.5 (0.8)
DeBERTaV3Base [He et al., 2021] | 70.7 (0.2) | 78.9 (0.1) | 67.8 (0.1) |  | 82.8 (0.4) | 85.6 (0.3) | 77.2 (0.5) |  | 80.4 (0.2) | 85.1 (0.4) | 77.2 (0.5)
DeBERTaV3Base \+ ALL |  71.1 (0.1) |  79.4 (0.1) |  68.4 (0.1) |  |  83.1 (0.2) |  85.8 (0.2) |  77.8 (0.6) |  |  81.1 (0.3) |  86.0 (0.6) |  79.5 (0.8)

Table 7.7: Results of our DeBERTaBase and DeBERTaV3Base models on WQA, NewsAS2
and TriviaAS2. We underline statistically significant improvements over the
corresponding baselines.

We divide results across the different models we used to improve readability.
We begin by describing results on Base models and then we show the accuracy of
Large architectures on the same tasks. Then, we demonstrate empirically that
the improvements shown in this Chapter do not stem from the additional pre-
training but rather from our harder tasks. Lastly, we offer insights into the
reasons behind the varying performance of different objectives on different
datasets. In the tables, the statistical significance is computed always with
respect to the vanilla model of each block. We report results from the
original papers when available, otherwise we fine-tuned the models on the
different tasks.

####  7.4.1 Base Models

The results over Base models are reported in Tables 7.4 and 7.5 for ASNQ,
WikiQA and TREC-QA, and in Tables 7.6 and 7.7 for WQA, NewsAS2 and TriviaAS2.
Results are split across several tables for better readability. This Section
proceeds by describing the results for each objective we propose.

###### SSP

Our Spans in the Same Paragraph objective improves models’ accuracy on all
tasks compared to the baselines, apart from the combination with RoBERTa and
ELECTRA on TREC-QA. The advantages are more marked on small datasets, which
confirms that our SSP objective is particularly effective when fine-tuning
data is scarce. More on this in Section 7.4.4. For example, SSP with either
RoBERTa or ELECTRA improves the Precision@1 by 2.9 points on ASNQ and by 4.6
points on WikiQA. In the second table, SSP shows minor improvements in
Precision@1 on WQA, NewsAS2 and TriviaAS2 of 0.6, 0.2 and 1.9 points
respectively.

###### SP

The Spans in Paragraph objective provides significant improvements across all
6 datasets and combined with every architecture. For example, when applied to
RoBERTa or ELECTRA, it improves the Precision@1 by up to 3.6 points on ASNQ,
4.7 on WikiQA and 1.4 on TriviaAS2. As before, the general pattern shows
discrete improvements on large datasets and very high gains on small datasets
such as WikiQA. We believe this is the most flexible objective, and we
empirically confirm this claim, because it is composed of short spans of text
that mimic the question and paragraphs gathered from multiple documents thst
mimic the answer.

###### PSD

Models trained with Paragraph is the Same Document outperform the baselines on
all datasets apart from the combination with ELECTRA on TREC-QA111The results
of ELECTRA+PSD on TREC-QA are significantly lower than the average, we suspect
there was an issue with the model initialization for this experiment.. The
improvements are consistent compared to the baselines, however, the average
gains are smaller when compared with SSP or SP. We suspect that this is the
consequence of PSD working at a higher level, by performing predictions over
entire paragraphs, which may not align optimally with the AS2 task. Among the
most significant results, we cite the 3.1 points of improvements in
Precision@1 on ASNQ and the gains of 0.9 and 1.3 on WQA and TriviaAS2
respectively.

###### ALL

In this experiment, we show the results of combining all the proposed
objectives for a single continuous pre-training. When applied to RoBERTa,
ELECTRA and DeBERTa, the results show improvements across all datasets and
more agnostic results to the target tasks, which means that this objective
should be the choice when testing on new datasets. Regarding the two DeBERTa
architectures, both models augmented with the combination of SSP, SP and PSD
outperform the baselines by a significant margin on all datasets. In
particular, we report the gains in Precision@1 of 2.5 points on ASNQ, 2.4 on
TREC-QA and 2.3 points on TriviaAS2.

Finally, notice that we match the performance of TandA on several datasets
such as WikiQA, WQA and NewsAS2, even outperforming it in some scenarios.
TandA uses ASNQ to transfer knowledge about the Answer Sentence Selection task
before fine-tuning on smaller datasets, leading to higher final accuracy. We
match the performance of TandA in many experiments while using only unlabeled
data that do not require expensive annotation processes. We compare our models
also with the Cascade Transformer [Soldaini and Moschitti, 2020]. Notice that
this technique performs an additional training step over ASNQ before fine-
tuning over the target datasets, as TandA does. We surpass the Cascade
Transformer by a large margin on ASNQ with all of our continuously pre-trained
models. Moreover, we surpass the performance of RLAS-BIABC with our
DeBERTaV3Base \+ ALL model on both WikiQA and TREC-QA by a significant margin
of 0.7 and 1.5 MAP points, respectively.

####  7.4.2 Large Models

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
MAP | MRR | P@1 |  | MAP | MRR | P@1 |  | MAP | MRR | P@1
TandA BERTLarge [Garg et al., 2019] | - | - | - |  | 90.4 | 91.2 | - |  | 91.2 | 96.7 | -
TandA RoBERTaLarge [Garg et al., 2019] | - | - | - |  | 92.0 | 93.3 | - |  | 94.3 | 97.4 | -
DeBERTaLarge [He et al., 2020] | 72.6 (0.2) | 78.5 (0.2) | 68.8 (0.3) |  | 89.5 (0.3) | 90.8 (0.4) | 84.0 (0.7) |  | 91.9 (0.2) | 94.7 (1.1) | 90.7 (1.7)
DeBERTaLarge \+ ALL |  73.4 (0.1) |  78.9 (0.1) |  69.5 (0.1) |  |  90.1 (0.1) | 91.2 (0.2) |  85.1 (0.5) |  | 91.8 (1.1) | 94.6 (0.7) | 90.7 (1.7)
TandA DeBERTaLarge \+ ALL | - | - | - |  |  91.4 (1.0) |  92.4 (0.8) |  86.8 (1.6) |  |  92.3 (0.6) |  94.9 (1.6) |  93.1 (1.7)
DeBERTaV3Large [He et al., 2021] | 73.8 (0.1) | 79.0 (0.1) | 69.5 (0.1) |  | 90.9 (1.2) | 92.0 (1.1) | 86.2 (2.0) |  | 91.7 (0.7) | 94.7 (0.5) | 91.8 (0.8)