# Bridging Topic, Domain, and Language Shifts:
An Evaluation of Comprehensive Out-of-Distribution Scenarios

Andreas Waldis 1,2, Iryna Gurevych1
1Ubiquitous Knowledge Processing Lab (UKP Lab)
Department of Computer Science and Hessian Center for AI (hessian.AI)
Technical University of Darmstadt
2Information Systems Research Lab
Department of Computer Science, Lucerne University of Applied Sciences and
Arts
www.ukp.tu-darmstadt.de www.hslu.ch
* Corresponding author<EMAIL_ADDRESS>
###### Abstract

Language models (LMs) excel in in-distribution (ID) scenarios where train and
test data are independent and identically distributed. However, their
performance often degrades in real-world applications like argument mining.
Such degradation happens when new topics emerge, or other text domains and
languages become relevant. To assess LMs’ generalization abilities in such
out-of-distribution (OOD) scenarios, we simulate such distribution shifts by
deliberately withholding specific instances for testing, as from the social
media domain or the topic Solar Energy.

Unlike prior studies focusing on specific shifts and metrics in isolation, we
comprehensively analyze OOD generalization. We define three metrics to
pinpoint generalization flaws and propose eleven classification tasks covering
topic, domain, and language shifts. Overall, we find superior performance of
prompt-based fine-tuning, notably when train and test splits primarily differ
semantically. Simultaneously, in-context learning is more effective than
prompt-based or vanilla fine-tuning for tasks when training data embodies
heavy discrepancies in label distribution compared to testing data. This
reveals a crucial drawback of gradient-based learning: it biases LMs regarding
such structural obstacles.

##  1 Introduction

Previous evaluations (Wang et al., 2019b, a) of fine-tuned language models
(LMs) (Devlin et al., 2019; Liu et al., 2019b; Radford et al., 2019; He et
al., 2021b) assume training and testing data are independent and identically
distributed - referred to as in-distribution (ID). However, out-of-
distribution (OOD) evaluations address more practical scenarios when these
assumptions do not, introducing specific distribution shifts between training
and testing data. For example, we expect LMs to generalize to new topics, text
domains, and languages when used in argument mining tasks (Slonim et al.,
2021). Although OOD generalization is well-studied in NLP, existing evaluation
studies have primarily focused on a single type of distribution shift (K et
al., 2020; Yang et al., 2022b; Yuan et al., 2023), such as text domains by
training and testing on reviews from different product groups (Blitzer et al.,
2007a) or statements from social media or debating platforms (Hardalov et al.,
2021). Owing to this reliance on single shift types, OOD generalization
encompassing multiple such shift types (as shown in Figure 1) remains
understudied. As a result, methodological and analytical advancements have
been limited; for instance, existing methods do not necessarily transfer to
other shift types because of their overreliance on shift-specific features
(Chen et al., 2021; Liang et al., 2022).

Topic Shift $\rightarrow$ He states that solar energy can replace soil in
cities.
$\rightarrow$ PiS supports free health care, provided by the state.

Domain Shift $\rightarrow$ Remember, #God has it all worked out. $\rightarrow$
Calli, you killed the debate.

Language Shift $\rightarrow$ Dies muss jede Person für sich selbst
entscheiden. $\rightarrow$ Même si cela ne va pas régler le fond du problème.

Figure 1: Example of the topic (Solar Energy vs. Universal Health Care from
Shnarch et al. (2018)), domain (social media vs. debating platform from
Mohammad et al. (2016); Walker et al. (2012)), and language shifts (German vs.
French from Vamvas and Sennrich (2020))

To address this research gap, we consider tasks covering multiple types of
distribution shifts to evaluate OOD generalization comprehensively and fine-
grained evaluation metrics allowing the detection of crucial generalization
flaws. Starting with the latter and keeping known generalization issues in
mind (Reimers and Gurevych, 2017; Mosbach et al., 2021), we define three
relevant metrics (§ 2): absolute performance (Applicability), such as $F_{1}$
score; correlation between dev loss and performance (Reliability); deviations
across multiple runs (Stability). Next, and unlike previous work (Yang et al.,
2022b; Yuan et al., 2023), we go beyond solely considering domain shifts and
introduce eleven diverse OOD classification tasks covering topic, domain, and
language shifts (§ 3). These tasks originate from fields in which such shifts
are commonly treated: argument mining, stance detection, sentiment analysis,
and text entailment (Blitzer et al., 2007a; Vamvas and Sennrich, 2020; Slonim
et al., 2021; Hardalov et al., 2021; Yang et al., 2022b). During our
experiments (§ 4), we explore and analyze a variety of LMs and different
learning paradigms, such as vanilla (FT) and prompt-based fine-tuning (P+FT),
or in-context learning (ICL) using ChatGPT. Overall, we make the following
contributions:

1) We introduce eleven diverse OOD classification tasks that cover topic,
domain, and language distribution shifts. With this, we establish for the
first time a basis for comprehensively evaluating the variety of OOD scenarios
beyond domain shifts.

2) We propose three fine-grained evaluation metrics and show that they allow
us to identify crucial generalization flaws, such as the misalignment of loss
and performance.

3) We conduct an extensive evaluation of the introduced tasks. We demonstrate
the superior performance of P+FT in terms of Stability ($+2.3$) and
Reliability ($+5.8$). Additionally, ICL with ChatGPT lags behind gradient-
based methods on average but excels particularly when distribution shifts
induce structural generalization obstacles, such as differences in label
distributions.

4) We carry out an in-depth analysis and discover that P+FT and ICL provide
more robust predictions and P+FT retains significantly more semantics in its
encoder layers than FT.

Unlike studies focusing exclusively on domain shifts (Yuan et al., 2023),
fine-tuning methods outperform ICL using ChatGPT on average when considering
different shifts - even with minimal trainable parameters. While recognizing
the out-of-the-box effectiveness for domain shifts, it is essential to note
that information from testing data in simulated OOD scenarios might already
have been incorporated during the pre-training of such large LMs.
Consequently, assessing the challenges posed by distribution shifts in the
context of specific LMs is essential when comparing them and drawing
conclusions regarding real-world OOD performance with truly unseen data.

##  2 Generalization

In the following, we define the investigated generalization scenarios (§ 2.1)
and elaborate on metrics for fine-grained generalization measurement (§ 2.2).

###  2.1 Generalization Scenarios

To comprehensively analyze out-of-distribution (OOD) generalization, we
compare OOD and in-distribution (ID) scenarios. While we assume that train and
test instances are independent and identically distributed in the ID setting,
we deliberately introduce different covariate distribution shifts for OOD.
More precisely, we withhold instances of specific topics (like Nuclear
Energy), domains (such as social media), and languages (like French) for
testing. By considering these shift types, we aim to examine crucial
capabilities of language models (LMs) for real-world scenarios, such as
generalization towards upcoming topics, changing text domains, and different
languages. Ideally, no gap between ID and ODD exists, and training data does
not bias LMs apart from relevant features to solve the tasks. Such unwanted
bias can originate from semantic features about specific topics or differently
distributed labels. However, we anticipate noticeable performance gaps based
on prior work (Stab et al., 2018; Kumar et al., 2022a).

###  2.2 Generalization Success

Although generalization success is often assessed using a single metric (like
$F_{1}$ score), previous works tend to ignore known stability issues, such as
apparent deviations regarding randomness (Mosbach et al., 2021). To identify
such generalization flaws and provide a comprehensive evaluation, we define
three fine-grained measurements that consider multiple runs with different
folds and seeds:

#### Applicability

evaluates how well LMs perform on tasks measured as the average performance
($\mu_{F_{1}}$) across the multiple runs.

#### Reliability

quantifies how consistent predictions of LMs are, measuring the average
Kendall correlation $\mu_{\tau}$ between loss and $F_{1}$ for dev instances.
Ideally, we expect an inverse linear correlation with $\mu_{\tau}$ equal to
-1. However, given that we determine final labels as the class with the
highest probability, we allow dev performance and loss to increase
simultaneously. When class probabilities change from $(95\%,5\%)$ to
$(90\%,10\%)$, the cross-entropy changes from $0.074$ to $0.15$. While
predicting the same class ($\hat{y}=k_{0}$), we infer that the LM is becoming
less sure about the prediction. This becomes particularly relevant for OOD
generalization, where overfitting to distributional properties of training
data, such as unique vocabulary, is likely to introduce uncertainty when
processing test data.

#### Stability

examines the impact of data and randomness on the Applicability and
Reliability properties. As recommended by Reimers and Gurevych (2017), we
measure the standard deviation of these two properties ($\sigma_{F_{1}}$ and
$\sigma_{\tau}$) based on a set of multiple runs.

##  3 Out-of-Distribution Tasks

In this section, we outline the selection (§ 3.1) and composition (§ 3.2) of
the tasks used to verify OOD capabilities comprehensively. Additionally, we
characterize (§ 3.3) the induced distribution shifts regarding semantic,
surface, and label properties.

###  3.1 Task Selection

Our primary focus lies on datasets from argument mining, stance detection, and
sentiment analysis, which meet our criteria of having annotations for at least
one type of shift: topic, domain, or language. These fields traditionally
explore the influence of these properties in their studies. Moreover, in
contrast to English-only datasets, multilingual datasets often provide
multiple annotations — such as language and domain (x-review) or topic and
language (x-stance). This enables us to formulate two tasks to address both
types of shifts. In total, we consider eleven tasks (§ 3.1) that cover shifts
in topic, domain, and/or language:

#### Argument Quality (arg-qua)

Toledo et al. (2019) provides 9,100 argument pairs annotated whether the first
or second has the higher quality, covering 22 topics.

#### Argument Similarty (arg-sim)

Reimers et al. (2019) annotated 3,595 arguments pairs of 28 topics, whether
they are similar or not.

#### Argument Classification (arg-cls)

Stab et al. (2018) annotated arguments for their argumentative stance (pro,
con, neutral) regarding one of eight topics.

#### Evidence Classification (evi-cls)

Shnarch et al. (2018) presented 5,785 sentences annotated as relevant or not
for one out of 118 topics.

#### Sentiment Classification (review)

(Blitzer et al., 2007a) collected 8,000 reviews annotated as positive or
negative for four domains (Amazon product groups).

#### Multi-Dataset Stance Detection (stance)

Following Hardalov et al. (2021), we use the semeval (Mohammad et al., 2016),
emergent (Ferreira and Vlachos, 2016), iac dataset (Walker et al., 2012) to
evaluate stance detection across three domains (social media, news, and
debating). All of them are annotated with the same labels (pro, con, neutral).

#### Multi-Dataset Entailment (entail)

Following Yang et al. (2022b), we consider three medium-sized datasets (rte
(Wang et al., 2018), SciTail (Khot et al., 2018), hans (McCoy et al., 2019))
to evaluate textual-entailment across three domains.

#### Multi-Lingual Stance Detection (x-stance)

This dataset (Vamvas and Sennrich, 2020) includes around 63,000 comments
annotated either as favor or against regarding 12 topics and covering three
languages (de, fr, it).

#### Multi-Lingual Sentiment Classification (x-review)

Prettenhofer and Stein (2010) presents a set of 43,000 positive or negative
reviews covering four languages (de, en, fr, jp) and three domains (Amazon
product groups).

| Shift | Separability |  $\Delta$ Flesch |  $\Delta$ Words | KL
---|---|---|---|---|---
arg-qua | Top. | 78.6 | 1.5 | 2.2 | 0.1
arg-sim | Top. | 75.8 | 4.6 | 0.27 | 0.4
arg-cls | Top. | 28.7 | 2.0 | 0.6 | 1.6
evi-cls | Top. | 56.3 | 2.4 | 0.7 | 7.1
1-6 review | Dom. | 52.7 | 6.5 | 60.5 | 0.0
stance | Dom. | 86.7 | 2.7 | 60.8 | 70.8
entail | Dom. | 40.4 | 5.1 | 31.2 | 12.8
1-6 x-stance | Lang./Top. | 0.05/19.8 | 16.6/1.3 | 6.6/0.3 | 0.6/0.4
x-review | Lang./Dom. | 0.07/72.4 | 11.0/1.8 | 60.0/6.5 | 0.0/0.0

Table 1:  Distribution shift characteristics between train and test splits of
the eleven tasks - averaged across all folds: Separability (Sun et al., 2022),
differences in Flesch ((Flesch, 1948)) score and word number between train and
test instances, and how the class distribution of these splits differ (KL
divergence).

###  3.2 Task Composition

We enforce distribution shifts for OOD evaluation by composing train/dev/test
splits which include instances with distinct distributional properties, such
as unique topics or text domains (Figure 1). We utilize multiple folds to
ensure each distinct distributional property (like a unique topic) is tested
precisely once. To ensure a fair comparison, we create the same number of ID
folds with the same-sized train/dev/test splits as for OOD.

###  3.3 Distribution Shifts

The nature of induced distribution shifts influences the difficulties and,
consequently, the amount of generalization expected to solve a specific OOD
classification task. We describe induced shifts in terms of semantic, surface,
and label properties in § 3.1. For all metrics, a higher value indicates
increased difficulty. We start analyzing the semantic separability of train
and test instances. Following Sun et al. (2022), we first embed111We follow
Reimers and Gurevych (2019) and embed instance with paraphrase-multilingual-
mpnetbase-v2. all instances. Subsequently, we employ k-means clustering
(Lloyd, 1982; MacQueen, 1967) to identify two clusters. We then measure the
alignment between the identified clusters and the train/test assignments using
the adjusted rand index (Hubert and Arabie, 1985). A higher score indicates a
more significant semantic divergence between train and test instances.

Next, we examine surface-level text features to identify generalization
challenges of the text beyond semantics. To this end, we calculate the
differences in average readability (Flesch, 1948) and word count ($\Delta$
Flesch, $\Delta$ Words) between train and test instances. Lastly, we assess
distributional disparities between the class labels of train and test
instances using Kullback-Leibler (KL) divergence (Boyd and Vandenberghe,
2004). A higher KL value denotes more significant imbalances, adding an extra
layer of complexity, as LMs tend to overfit the training label distribution.

Based on these metrics, we categorize tasks into different groups. First, arg-
qua, arg-sim, and stance exhibit high semantic differences with separability
scores ranging from $75.8$ to $86.7$. Next, review, stance, entail, and
x-review embody surface-level difficulties due to the varying readability
($\Delta$ Flesch) or text lengths ($\Delta$ Word Count). Finally, evi-cls,
stance, and entail exhibit unequal label distributions, resulting in high KL
divergence values, thereby posing additional challenges. Among these tasks,
stance appears to be the most challenging, featuring distinct semantic
domains, text structure variations, and unequal class distributions.

##  4 Experiments

In this section, we start by outlining our experimental setup (§ 4.1) before
discussing results (§ 4.2) and conclude with an in-depth analysis (§ 4.3).

###  4.1 Setup

#### Models

We primarily experiment with base-sized LMs, including BERT (Devlin et al.,
2019), RoBERTa (Liu et al., 2019b), and DeBERTa-v3 (He et al., 2021b) and
their multilingual counterparts (Devlin et al., 2019; Conneau et al., 2020a;
He et al., 2021b). For additional experiments, we consider ALBERT (Lan et al.,
2020), DeBERTa (He et al., 2021a), ELECTRA (Clark et al., 2020), and
gpt-3.5-turbo (Ouyang et al., 2022) denoted as ChatGPT.

#### Learning Paradigms

We assess the generalization capabilities of LMs under various learning
paradigms. These include linear probing (LP), vanilla fine-tuning (FT),
prompting (P), prompt-based fine-tuning (P+FT), and in-context learning (ICL).
In LP and FT, we use newly initialized classification heads atop the LM, which
remains either frozen (LP) or trainable (FT)222We use [SEP] to concatenate the
input with its topic, if available. For P and P+FT, we rely on the pre-trained
MLM head and keep the LM frozen (P) or trainable (P+FT). Finally, we employ
in-context learning (ICL) to verify the capabilities of large LMs. Further
details on the ICL and P+FT paradigm can be found in Appendix § A.5 and § A.4,
respectively.

#### Evaluation

We conduct evaluations on all tasks and learning paradigms and take LP and P
as a lower bound and ID fine-tuning (FT-ID) as an upper bound. To account for
data variability and randomness, we use three different seeds and employ a
multifold setup, either three- or four-fold. Using these runs, we use fine-
grained performance reporting including average Stability ($\mu_{F_{1}}$),
Reliability ($\mu_{\tau}$), and the Stability
($\sigma_{F_{1}}$,$\sigma_{\tau}$) - as defined in § 2.2.

| arg-qua | arg-sim | arg-cls | evi-cls | review | stance | entail | x-stance | x-review | Applicability | Reliability
---|---|---|---|---|---|---|---|---|---|---|---
| Top. | Top. | Top. | Top. | Dom. | Dom. | Dom. | Lang./Top. | Lang./Dom. | $\mu_{F_{1}}\pm\sigma_{F1}$ | $\mu_{\tau}\pm\sigma_{\tau}$
LPBERT | 48.4 | 57.1 | 42.7 | 65.6 | 81.0 | 27.9 | 46.3 | 52.5/56.7 | 67.5/73.3 | $56.3\pm{0.8}$ | $-58.4\pm{6.2}$
PBERT | 40.5 | 50.4 | 40.1 | 49.2 | 72.9 | 25.0 | 41.2 | 34.5/48.6 | 45.6/54.5 | $45.7\pm{0.2}$ | -
FTBERT | 75.5 | 68.4 | 57.5 | 74.7 | 89.3 | 31.1 | 50.7 |  62.0/63.9 | 77.7/84.4 | $66.8\pm{0.9}$ | $-56.8\pm{12.3}$
P+FTBERT | 76.2 | 66.0 | 59.8 | 75.7 | 89.3 | 28.5 | 48.0 | 59.5/63.6 |  79.6/83.9 | $66.4\pm{1.1}$ | $-61.7\pm{12.4}$
1-12 FT-IDBERT | 87.9 | 76.4 | 67.3 | 78.9 | 90.4 | 61.1 | 93.6 | 67.6 | 87.0 | $78.9\pm{0.4}$ | $-96.1\pm{6.5}$
LPDeBERTa-v3 | 53.0 | 70.0 | 55.1 | 67.9 | 88.6 | 23.4 | 58.0 | 55.4/59.7 | 78.7/83.6 | $63.0\pm{0.5}$ | $-64.3\pm{4.3}$
PDeBERTa-v3 | 54.2 | 58.6 | 40.3 | 57.2 | 61.9 | 26.5 | 54.6 | 51.1/51.2 | 49.5/52.0 | $50.6\pm{1.0}$ | -
FTDeBERTa-v3 | 78.4 | 75.4 | 64.0 | 77.3 | 93.4 | 29.6 | 55.6 |  69.8/69.3 | 91.3/90.9 | $72.3\pm{1.1}$ | $-72.6\pm{13.4}$
P+FTDeBERTa-v3 | 78.5 | 79.1$\dagger$ | 74.6 | 78.6 | 94.2$\dagger$ | 33.0 | 60.2 | 69.7/69.9 |  91.8/ 91.4 | $74.6\pm{0.9}$ | $-78.4\pm{8.4}$
1-12 FT-IDDeBERTa-v3 | 89.0 | 78.4 | 75.2 | 80.6 | 93.9 | 63.3 | 95.4 | 72.2 | 92.1 | $82.2\pm{0.4}$ | $-97.7\pm{6.5}$
LPRoBERTa | 51.8 | 55.3 | 41.6 | 62.5 | 85.7 | 28.7 | 39.2 | 55.1/57.5 | 82.8/82.5 | $58.4\pm{0.6}$ | $-56.3\pm{6.2}$
PRoBERTa | 48.3 | 55.3 | 42.9 | 51.8 | 80.5 | 24.0 | 40.9 | 42.4/48.7 | 67.2/73.4 | $52.3\pm{0.0}$ | -
FTRoBERTa | 70.9 | 73.0 | 56.9 | 77.5 | 92.2 | 30.0 | 51.3 | 62.2/66.8 | 89.6/90.1 | $69.1\pm{2.5}$ | $-69.7\pm{10.4}$
P+FTRoBERTa | 77.6 | 74.3 | 66.0 | 77.9 | 92.0 | 29.1 | 52.4 |  67.4$\dagger$/67.5$\dagger$ |  89.7/90.0 | $71.3\pm{0.5}$ | $-75.5\pm{8.1}$
1-12 FT-IDRoBERTa | 84.0 | 79.4 | 71.0 | 80.9 | 92.9 | 64.7 | 94.1 | 66.3 | 91.0 | $80.5\pm{1.9}$ | $-96.6\pm{4.7}$

Table 2: Results for BERT, DeBERTa-v3, and RoBERTa using linear probing (LP),
prompting (P), fine-tuning (FT), and prompt-based fine-tuning (P+FT), as well
as ID fine-tuning (FT-ID). We report average Applicability ($\mu_{F_{1}}$),
Reliability ($\mu_{\tau}$), Stability ($\sigma_{F_{1}},\sigma_{\tau}$). Best
OOD performance within one LM are underlined, bold highlights best OOD
performance across LMs, and $\dagger$ indicates better OOD than ID
performance.

###  4.2 Results

In the following, we report and discuss the results of evaluating the proposed
OOD tasks for different LMs and learning paradigms. First, we present these
results aggregated before entering the detailed level. Eventually, we focus on
model specialties, parameter-efficient tuning methods, and compare gradient-
based methods with in-context learning using ChatGPT covering the English-only
tasks.

#### Aggregated Results

Figure 2 shows the aggregated results of evaluating all eleven tasks for three
LMs in different evaluation scenarios. Overall, we note superior
generalization capabilities of DeBERTa-v3 compared to BERT and RoBERTa.
Comparing ID and OOD vanilla fine-tuning (FT) using all three generalization
measurements reveals a fine-grained picture of their generalization gap and
potential flaws. OOD shows degraded Applicability ($F_{1}$ score), lower
Reliability (correlation $\tau$ between loss and $F_{1}$), and lower Stability
(higher $\sigma_{F_{1}}$ and $\sigma_{\tau}$) than ID. In particular, we see
the misalignment of loss and performance - a violation of a fundamental
generalization assumption - attributed crucially to these gaps.
Simultaneously, we observe for DeBERTa-v3 and RoBERTa that prompt-based fine-
tuning (P+FT) not only improves the absolute performances (Applicability) but
also leads to better Reliability and fewer deviations (Stability).

Figure 2: Analysis of generalization gaps between ID and OOD (FT) and OOD
prompt-based fine-tuning (P+FT) across eleven tasks (§ 3) on three models
regarding Applicability ($F_{1}$), Reliablity ($\tau$), and Stability \-
deviation of $F_{1}$ and $\tau$.

#### Detailed Results

After finding fundamental generalization flaws considering FT on an aggregated
level, we continue to analyze results in more detail considering additional
learning paradigms (§ 4.1). Comparing them, LP provides better results than P
for all LMs, while both underperform the others. As in the aggregated results,
P+FT performs better than FT regarding all metrics for DeBERTa-v3 and RoBERTa,
where DeBERTa-v3 provides $2.3$ better Applicability ($\mu_{F_{1}}$), $5.8$
higher Reliability ($\mu_{\tau}$), and better Stability with $0.2$
($\sigma_{F_{1}}$) and $5.0$ ($\sigma_{\tau}$). At the same time, it
outperforms FT in ten out of eleven tasks and reaches ID performance for two
tasks (arg-sim and review). In contrast, BERT using P+FT performs slightly
worse compared to FT. Note that we will discuss the role of the selected LM in
more detail in the next paragraph.

Regarding the different tasks, we see stance and entail are the most difficult
OOD tasks because of the most significant gap between OOD results (FT and
P+FT) and FT-ID. Further, a small performance gap between LP and P is a good
indicator for identifying such cases. These difficulties correlate with their
previously measured high KL divergences (§ 3.1). Nevertheless, P+FT with
DeBERTa-v3 provides improvements of $3.4$ (stance) and $4.6$ (entail) compared
to FT. Overall, we observe higher gains of P+FT and bigger differences between
ID and OOD for topic and domain than language shifts.

#### Model Specialities

Previously, we found that P+FT is highly effective with DeBERTa-v3 and RoBERTa
but not with BERT. To gain more insights into the influence of model
specialties like pre-training objectives, we evaluate additional LMs on the
English-only tasks involving topic and domain shift. From Figure 3, we note
that mixed pre-training (token and sentence objectives) make LMs less suited
for P+FT as ALBERT exhibits no clear gains ($0.3$). Next, considering DeBERTa
and ELECTRA does not provide ground to confirm that the ELECTRA-style
objective of DeBERTa-v3 attributes to its superiority. Instead, since DeBERTa
outperforms RoBERTa, we see the disentangled attention as a crucial factor in
the success of DeBERTa-v3. By comparing with DeBERTa, we assume the
significantly more extensive vocabulary of 120k tokens is another reason for
the success of DeBERTa-v3. Overall, DeBERTa-v3 is the best choice for P+FT due
to its token-level-only pre-training objectives and its various other
specialties.

Figure 3: Average Applicability of comparing various LMs tuned on the English-
only tasks using FT or P+FT.

#### Parameter-Efficient Tuning

Next, we turn to parameter-efficient tuning - LoRA (Hu et al., 2022), P-Tuning
(Liu et al., 2021b), and Prompt-Tuning (Lester et al., 2021) \- to evaluate
the importance of full-model tuning. Following Table 3, we see that LoRA with
$r=4$ performing on par as full-parameter tuning and outperforms P-Tuning and
Prompt-Tuning on most tasks. These results indicate that OOD fine-tuning still
leaves a large potential of LMs unexploited.

#### In-Context Learning

Concluding, we evaluate in Figure 4 the English-only tasks using in-context
learning (ICL) with ChatGPT333Please find details in the Appendix (§ A.5). On
average, ICL does not reach the performance level of gradient-based (FT and
P+FT) approaches. Regarding the different types of shifts, ICL underperforms
them on topic shifts while performing on par or better for tasks exhibiting a
domain shift. From the results of these latter tasks, we distinguish between
two situations where ICL contemporary outperforms gradient-based learning.
First, if we assume that a task was already well covered within the enormous
pre-training of ChatGPT, such as for sentiment analysis (review). In addition,
ICL succeeds when the nature of the distribution shifts prevents gradient-
based paradigms from adequately learning - like for stance or entail where the
shifts induce heavy differences in class distributions (high KL divergence).
This is also visible when considering the relatively small gains of prompt-
based fine-tuning (P+FT) over the prompting-only paradigm (P) - from § 4.1
$+6.5$ for stance and $+5.6$ for entail with DeBERTa-v3 compared to $+34.3$
for arg-cls.

| arg-qua | arg-sim | arg-cls | evi-cls | review | stance | entail | Applicability | Reliability
---|---|---|---|---|---|---|---|---|---
P+FTDeBERTa-v3 | 78.5 | 79.1 | 74.6 | 78.6 | 94.2 | 33.0 | 60.2 | 71.2$\pm$1.3 | -84.4$\pm$8.3
+P-Tuning | 56.3 | 54.9 | 38.1 | 54.7 | 53.7 | 33.7 | 43.5 | 47.8$\pm$1.2 | -22.0$\pm$20.3
+Prompt-Tuning | 54.8 | 54.0 | 38.2 | 54.6 | 53.4 | 32.3 | 43.2 | 47.2$\pm$0.7 | -6.0$\pm$30.5
+LoRA | 78.1 | 78.8 | 73.4 | 77.9 | 94.9 | 33.1 | 60.8 | 71.0$\pm$1.0 | -75.5$\pm$4.9

Table 3: Results of parameter efficient approaches for DeBERTa-v3 compared to
plain P+FT and LoRA. Figure 4: Comparison of ICL using ChatGPT and P+FT & FT
with DeBERTa-v3

###  4.3 Analysis

After observing the success of P+FT, we analyze how it differs from FT in more
detail. Therefore, we rely on DeBERTa-v3 and arg-cls where differences between
P+FT and FT are the most prominent.

#### Surface-Level Correlations

First, we analyze how correct and wrong predictions are biased regarding
surface-level features (Flesch score and word count). Considering Figure 5, we
observe correct predictions from any setup are biased towards shorter (lower
word count) but more complex (higher Flesch) inputs compared to the dataset
average (dashed line) - and vice versa for wrong predictions. However, prompt-
based fine-tuning (P+FT) relies less on surface correlations and shows similar
patterns as FT-ID. These results provide further evidence that P+FT leads to
better generalization capabilities. In addition, ICL using ChatGPT shows
fewer, but not none, deviations from the dataset average. Therefore, we assume
a general bias of LMs regarding such surface features.

Figure 5: Comparison of the correlation of correct and wrong predictions with
word count and Flesch score across different learning paradigms.

#### Model Insights

While we previously considered LMs as black boxes, we next focus on the inners
of LMs tuned using FT and P+FT. From Table 4, we see that P+FT predicts with a
similar certainty as ID fine-tuning (FT-ID) and a higher one than FT. Thus, we
see P+FT being less confused by the distribution shift than FT. Further, we
note a more evident correlation between certainty and surface features (Flesch
score or word counts) for FT than P+FT. For example, FT tends to be less
confident for input with a higher length or a lower Flesch score. This
observation aligns with the previous analysis regarding the dependence on
surface features.

Next, we analyze token-level attributions of LMs. Following (Kobayashi et al.,
2020), we calculate the attribution between input and classification tokens -
CLS for FT and FT-ID, and MASK for P+FT. Note that we consider only the
attribution of the input text and make embeddings of CLS and MASK comparable
using the Euclidean norm. We see that, on average, FT-ID and FT have a higher
average attribution than P+FT. Therefore, attributions are more evenly
distributed for P+FT. We can observe this pattern already when we compare the
token attribution of the pre-trained model (raw vs. P+raw). Apparent
differences are also visible when we compare how attributions of correct or
wrong predicted instances differ. While P+FT shows maximum 0.4 differences
(P+FT), this rises to 1.0 for FT-ID. With these results, we assume LMs applied
in prompt-based or vanilla fine-tuning fundamentally differ in how inputs are
processed. An analysis of the layer-wise embeddings of the CLS and MASK tokens
further supports this assumption. As in Figure 6, we observe that by relying
on the MASK token as classification proxy, P+FT does not touch semantic
information (about topics) until the last layers. In contrast, FT eliminates
this information across all layers during training.

| FT-ID | FT | P+FT | raw | P+raw
---|---|---|---|---|---
Average Certainty | 97.6 | 95.9 | 97.8 | - | -
Certainty$\times$Flesch | 5.1 | 8.6 | 4.1 | - | -
Certainty$\times$Word Count | -10.3 | -13.2 | -6.3 | - | -
Average Attribution | 16.2 | 15.5 | 13.0 | 16.3 | 13.2
Correct Attribution | 16.4 | 15.8 | 13.1 | - | -
Wrong Attribution | 15.2 | 14.9 | 12.7 | - | -

Table 4: analysis of models certainty and attribution for FT-ID, FT, and P+FT
of DeBERTa-v3. With raw and P+raw, we analyze how the model attributes input
tokens after pre-training. Figure 6: Overview of the T-SNE (van der Maaten and
Hinton, 2008) reduced embeddings of the classification proxy tokens for every
second lay. We use the CLS token for FT and MASK for P+FT.

##  5 Related Work

#### Out-of-Distribution Generalization

Mitigating generalization gaps between OOD and ID is a broadly studied aspect
in Computer Vision (Zhou et al., 2023), covering data augmentation (Liu et
al., 2019a; Yao et al., 2022), representation learning (Khosla et al., 2012;
Laskin et al., 2020; Zhang et al., 2022), optimizing model structure (Wortsman
et al., 2022; Balaji et al., 2018), or specialized benchmark datasets (Koh et
al., 2021; Yang et al., 2022a). Simultaneously, generalization for OOD
scenarios is less studied in NLP. Works primarily focus on robustness of PLMs
(Hendrycks et al., 2019; Jin et al., 2020; Zhou et al., 2020; Wang et al.,
2021) or OOD detection (Zhou et al., 2021; Koner et al., 2021; Cho et al.,
2023). Similar to Computer Vision (Tseng et al., 2020), work in NLP consider
single types of distribution shift like either domains (Blitzer et al., 2007b;
Yang et al., 2022b; Yuan et al., 2023), languages (K et al., 2020; Conneau et
al., 2020b), or topics (Stab et al., 2018; Allaway and McKeown, 2020). Such
approaches (Liang et al., 2022; Xu et al., 2018; Peng et al., 2018; Rietzler
et al., 2020) often rely on the specific distribution shift (like the topic
shift) to optimize the training procedure or the model architecture.
Therefore, they are not universally applicable to OOD tasks covering other
distribution shifts - like generalization towards upcoming topics and unseen
languages.

#### Prompting

A common practice in NLP is to tune pre-trained language models by providing a
natural language input and optimize them to predict an arbitrary label (Devlin
et al., 2019; Liu et al., 2019b; He et al., 2021b; Radford et al., 2019).
Thus, we use pre-training abilities for processing the input but not for final
predictions. Prompting Liu et al. (2021a) allows us to rely upon acquired
competencies during pre-training for both stages. To do so, we reformulate a
task as a cloze to solve it with a textual label. This allows the use of
language models without any further tuning by embedding inputs in task-
specific templates for zero- or few-shot setting (Brown et al., 2020;
Chowdhery et al., 2022; Wang et al., 2022; Biderman et al., 2023). Further,
prompting allowed base-sized language models to reach comparable performance
to their large-sized counterparts (Schick and Schütze, 2021, 2021) with fine-
tuning them with the same limited data as in few-shoot settings. Apart from
such few-shot scenarios, little work examined prompt-based fine-tuning when
using complete datasets. As an expectation, Raman et al. (2023) shows their
robustness against adversarial attacks.

Unlike previous work, we comprehensively address OOD generalization with a
heterogeneous set of eleven tasks covering topic, domain, and language shifts.
At the same time, we point out the importance of fine-grained performance
measurement to identify crucial generalization flaws.

##  6 Discussion

We discuss the reported results and the insights of the conducted in-depth
analysis in the following. The exhaustive evaluations across the proposed OOD
tasks demonstrate the superior performance and robustness of prompt-based
learning methods (P+FT and ICL) compared to traditional fine-tuning (FT).444
These findings extend to ID scenarios — see Appendix § B.1. We see relying on
the pre-trained MLM head and avoiding freshly initialized layers allows for a
more focused learning process and significantly contributes to the success of
P+FT. This is also evident when considering the fact that P+FT tuning
preserves more semantic information in the encoder layers during P+FT tuning.
However, even such approaches still leave latent potentials of LMs untapped,
as LoRA-based tuning using a fraction of all parameters archives almost as
good results as training all parameters of LM. Therefore, we still see their
impossibility to not bias regarding these structural obstacles within training
data as one crucial drawback when applying them for OOD scenarios. This is
particularly visible in the small performance gains when comparing linear
probing and fine-tuning for tasks embodying heavy discrepancies between
training and testing labels. For such scenarios, including tasks with domain
shifts, ICL using a large LM outperforms P+FT while underperforming on
average.

##  7 Conclusion

With this work, we comprehensively address OOD generalization from multiple
perspectives. We emphasize the necessity of utilizing fine-grained performance
metrics, as they expose key weaknesses in generalization. We lay a foundation
for comprehensively evaluating OOD generalization by proposing eleven OOD
classification tasks covering text, domain, and language shifts. Finally, we
show that more recent methods, like prompt-based fine-tuning, narrow the
generalization gap between ID and OOD evaluation and provide more robust and
confident predictions.

#### Outlook

With our findings and the increasing prevalence of large LMs, it is
indispensable to work towards assessing challenges induced by deliberate
distribution shifts in the context of specific LMs. For instance, despite the
strong out-of-the-box performance of ICL using ChatGPT, we anticipate that
certain aspects (like semantic or task-specific information) of testing
instances in a simulated OOD scenario have been part of the pre-training of
these large LMs. Quantifying such aspects enables us to compare LMs and draw
more profound and practical conclusions about the suitability of LMs to handle
truly unseen data in future applications.

## Acknowledgements

We thank Cecilia Liu, Thy Thy Tran, and Kexin Wang for their valuable
feedback. This work has been funded by the Hasler Foundation Grant No. 21024.

## Ethical Considerations and Limitations

###  7.1 Higher Input Length

By embedding the input into a prompt, we sacrifice potential input tokens.
Since the used tasks have relatively short inputs, this is not crucial for
this work. However, this can be an essential limitation for other tasks when
inputs get longer.

###  7.2 Efficiency

We always refer to efficient fine-tuning when discussing efficient methods in
this work. Therefore, we did not consider efficient methods to make inferences
on larger LMs more feasible. We see this as another crucial and essential
aspect of real-world applications. Simultaneously, we think performance-wise
and efficiency-wise will alternate in the future. Therefore, we keep that for
future work.

###  7.3 Large Language Models

We show the competitive performance of ChatGPT compared to gradient-based
approaches by only relying on four demonstrations and without any tuning.
Simultaneously, we need to assume that the pre-training corpus of ChatGPT
leaks crucial aspects - like broadly covers controversially discussed topics
like Nuclear Energy or includes instances of popular datasets (like RTE (Wang
et al., 2018) or SemEval2016 (Mohammad et al., 2016)) word-by-word. When we
have in mind that we use OOD to verify generalization capabilities required
for upcoming scenarios, we need to examine the performance of ChatGPT
carefully and whether it was able to learn the task or just remembered some
semantic aspects of the pre-training.

## References

  * Allaway and McKeown (2020) Emily Allaway and Kathleen McKeown. 2020.  Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8913–8931, Online. Association for Computational Linguistics.
  * Balaji et al. (2018) Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018.  Metareg: Towards domain generalization using meta-regularization.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 1006–1016.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _CoRR_ , abs/2304.01373.
  * Blitzer et al. (2007a) John Blitzer, Mark Dredze, and Fernando Pereira. 2007a.  Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In _ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic_. The Association for Computational Linguistics.
  * Blitzer et al. (2007b) John Blitzer, Mark Dredze, and Fernando Pereira. 2007b.  Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 440–447, Prague, Czech Republic. Association for Computational Linguistics.
  * Boyd and Vandenberghe (2004) Stephen P Boyd and Lieven Vandenberghe. 2004.  _Convex optimization_.  Cambridge university press.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2021) Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2021.  Data augmentation for cross-domain named entity recognition.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5346–5356, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cho et al. (2023) Hyunsoo Cho, Choonghyun Park, Junyeop Kim, Hyuhng Joon Kim, Kang Min Yoo, and Sang goo Lee. 2023.  Probing out-of-distribution robustness of language models with parameter-efficient transfer learning.  _ArXiv_ , abs/2301.11660.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _CoRR_ , abs/2204.02311.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2020b) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020b.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ferreira and Vlachos (2016) William Ferreira and Andreas Vlachos. 2016.  Emergent: a novel data-set for stance classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1163–1168, San Diego, California. Association for Computational Linguistics.
  * Flesch (1948) Rudolph Flesch. 1948.  A new readability yardstick.  _Journal of applied psychology_ , 32(3):221.
  * Hardalov et al. (2021) Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021.  Cross-domain label-adaptive stance detection.  In _EMNLP_.
  * He et al. (2021a) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _CoRR_ , abs/2111.09543.
  * He et al. (2021b) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2019) Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019.  Using pre-training can improve model robustness and uncertainty.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2712–2721. PMLR.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Hubert and Arabie (1985) Lawrence Hubert and Phipps Arabie. 1985.  Comparing partitions.  _Journal of classification_ , 2:193–218.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is BERT really robust? A strong baseline for natural language attack on text classification and entailment.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8018–8025. AAAI Press.
  * K et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020.  Cross-lingual ability of multilingual BERT: an empirical study.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Khosla et al. (2012) Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A. Efros, and Antonio Torralba. 2012.  Undoing the damage of dataset bias.  In _Computer Vision - ECCV 2012 - 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I_ , volume 7572 of _Lecture Notes in Computer Science_ , pages 158–171. Springer.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 5189–5197. AAAI Press.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Online. Association for Computational Linguistics.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran S. Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. 2021.  WILDS: A benchmark of in-the-wild distribution shifts.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5637–5664. PMLR.
  * Koner et al. (2021) Rajat Koner, Poulami Sinhamahapatra, Karsten Roscher, Stephan Günnemann, and Volker Tresp. 2021.  Oodformer: Out-of-distribution detection transformer.  In _32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021_ , page 209. BMVA Press.
  * Kumar et al. (2022a) Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022a.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Kumar et al. (2022b) Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022b.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.
  * Laskin et al. (2020) Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2020.  CURL: contrastive unsupervised representations for reinforcement learning.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5639–5650. PMLR.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 3045–3059. Association for Computational Linguistics.
  * Liang et al. (2022) Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022\.  JointCL: A joint contrastive learning framework for zero-shot stance detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 81–91, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2021a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _CoRR_ , abs/2107.13586.
  * Liu et al. (2019a) Shikun Liu, Edward Johns, and Andrew J. Davison. 2019a.  End-to-end multi-task learning with attention.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 1871–1880. Computer Vision Foundation / IEEE.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b.  GPT understands, too.  _CoRR_ , abs/2103.10385.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Lloyd (1982) Stuart P. Lloyd. 1982.  Least squares quantization in PCM.  _IEEE Trans. Inf. Theory_ , 28(2):129–136.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * MacQueen (1967) J MacQueen. 1967.  Classification and analysis of multivariate observations.  In _5th Berkeley Symp. Math. Statist. Probability_ , pages 281–297. University of California Los Angeles LA USA.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016.  SemEval-2016 task 6: Detecting stance in tweets.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 31–41, San Diego, California. Association for Computational Linguistics.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Peng et al. (2018) Minlong Peng, Qi Zhang, Yu-gang Jiang, and Xuanjing Huang. 2018.  Cross-domain sentiment classification with target domain specific information.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2505–2513, Melbourne, Australia. Association for Computational Linguistics.
  * Prettenhofer and Stein (2010) Peter Prettenhofer and Benno Stein. 2010.  Cross-language text classification using structural correspondence learning.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 1118–1127, Uppsala, Sweden. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raman et al. (2023) Mrigank Raman, Pratyush Maini, J. Zico Kolter, Zachary C. Lipton, and Danish Pruthi. 2023.  Model-tuning via prompts makes NLP models adversarially robust.  _CoRR_ , abs/2303.07320.
  * Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017.  Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers et al. (2019) Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019.  Classification and clustering of arguments with contextualized word embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 567–578, Florence, Italy. Association for Computational Linguistics.
  * Rietzler et al. (2020) Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020.  Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification.  In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , pages 4933–4941. European Language Resources Association.
  * Schick et al. (2020) Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020.  Automatically identifying words that can serve as labels for few-shot text classification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5569–5578, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 255–269. Association for Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 2339–2352. Association for Computational Linguistics.
  * Shnarch et al. (2018) Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018.  Will it blend? blending weak and strong labeled data in a neural network for argumentation mining.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 599–605, Melbourne, Australia. Association for Computational Linguistics.
  * Slonim et al. (2021) Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, et al. 2021.  An autonomous debating system.  _Nature_ , 591(7850):379–384.
  * Stab et al. (2018) Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018.  Cross-topic argument mining from heterogeneous sources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3664–3674, Brussels, Belgium. Association for Computational Linguistics.
  * Sun et al. (2022) Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022.  Out-of-distribution detection with deep nearest neighbors.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 20827–20840. PMLR.
  * Toledo et al. (2019) Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019.  Automatic argument quality assessment - new datasets and methods.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5625–5635, Hong Kong, China. Association for Computational Linguistics.
  * Tseng et al. (2020) Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. 2020.  Cross-domain few-shot classification via learned feature-wise transformation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Vamvas and Sennrich (2020) Jannis Vamvas and Rico Sennrich. 2020.  X -stance: A multilingual multi-target dataset for stance detection.  In _Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing, SwissText/KONVENS 2020, Zurich, Switzerland, June 23-25, 2020 [online only]_ , volume 2624 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey E. Hinton. 2008.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9:2579–2605.
  * Walker et al. (2012) Marilyn A. Walker, Jean E. Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012\.  A corpus for research on deliberation and debate.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23-25, 2012_ , pages 812–817. European Language Resources Association (ELRA).
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2021) Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021.  Infobert: Improving robustness of language models from an information theoretic perspective.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2022.  Robust fine-tuning of zero-shot models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 7949–7961. IEEE.
  * Xu et al. (2018) Chang Xu, Cécile Paris, Surya Nepal, and Ross Sparks. 2018.  Cross-target stance classification with self-attention networks.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 778–783, Melbourne, Australia. Association for Computational Linguistics.
  * Yang et al. (2022a) Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. 2022a.  Openood: Benchmarking generalized out-of-distribution detection.  _CoRR_ , abs/2210.07242.
  * Yang et al. (2022b) Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2022b.  GLUE-X: evaluating natural language understanding models from an out-of-distribution generalization perspective.  _CoRR_ , abs/2211.08073.
  * Yao et al. (2022) Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. 2022.  Improving out-of-distribution robustness via selective augmentation.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 25407–25437. PMLR.
  * Yuan et al. (2023) Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023.  Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and llms evaluations.  _arXiv preprint arXiv:2306.04618_.
  * Zhang et al. (2022) Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P. Xing. 2022.  Towards principled disentanglement for domain generalization.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 8014–8024. IEEE.
  * Zhou et al. (2023) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2023.  Domain generalization: A survey.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 45(4):4396–4415.
  * Zhou et al. (2020) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley, Ke Xu, and Furu Wei. 2020\.  BERT loses patience: Fast and robust inference with early exit.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Zhou et al. (2021) Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021.  Contrastive out-of-distribution detection for pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 1100–1111. Association for Computational Linguistics.

##  Appendix A Additional Details of the Experiments

###  A.1 Training Setup

For all our experiments, we use NVIDIA RTX A6000 GPUs with CUDA (11.7), python
(3.8.10), transformers (4.28.0), PyTorch (1.13.1), and openprompt (1.0.1).

###  A.2 Hyperparameters

We use for the experiments fixed hyperparameters; AdamW (Loshchilov and
Hutter, 2019) as optimizer; a batch size of 16; a learning rate of 0.00002; a
dropout rate of 0.1; a warmup rate of 10% of the steps; random seeds:
$[0,1,2]$. In the case of parameter-efficient tuning, we use a learning rate
of a learning rate of 0.0002. Moreover, we use the following tags from the
huggingface model hub:

  * •

albert-base-v2

  * •

bert-base-uncased

  * •

aajrami/bert-mlm-base

  * •

microsoft/deberta-base

  * •

microsoft/deberta-v3-base

  * •

roberta-base

  * •

google/electra-basediscriminator

###  A.3 Dataset Examples

As a part of this work, we propose eleven different OOD classification tasks
based on 13 different datasets. Note, for multi-lingual datasets, we compose
two tasks covering once the language shift and once the topic (x-stance) or
domain shift (x-review). Table 5 shows an overview of these tasks and examples
for every task. In addition, Table 6 lists the prompt templates used for all
tasks and languages.

| Dataset | Description | Distribution Shift
---|---|---|---
arg-qua |  Argument Quality (Toledo et al., 2019) |  Choose which argument out of two has the higher quality: TOPIC: we should ban fossil fuels ARG-1: fossil fuels pollute and cause a lot of diseases ARG-2: fossil fuel companies often have incredibly bad and dangerous working conditions LABEL: ARG-1 |  Topical (22 topics)
arg-sim |  Argument Similarity (Reimers et al., 2019) |  Decide whether two arguments are similar or not-similar: TOPIC: organ donating ARG-1: One organ and tissue donation can save or enhance the lives of nearly 100 people ARG-2: By donating your organs after you die, you can save or improve as many as 50 lives LABEL: similar |  Topical (28 topics)
arg-cls |  Argument Classification (Stab et al., 2018) |  Classify an argument as pro, con, or no-argument given a topic: TOPIC: abortion ARG: Now our nonprofit really needs your help LABEL: similar |  Topical (8 topics)
evi-cls |  Evidence Classification (Shnarch et al., 2018) |  Decide whether a text is relevant evidence for a topic or not-relevant TOPIC: we should limit executive compensation TEXT: On April 7, 2009, Blankfein recommended guidelines to overhaul executive compensation LABEL: not-relevant |  Topical (118 topics)
review |  Sentiment Classification (Blitzer et al., 2007a) |  Classify product review as positive or negative: DOMAIN: dvd REVIEW: If you don’t own this dvd … my opinion it is the best american animated film ever released LABEL: positive |  Domain books, dvd, electronics, kitchen & housewares
stance | Stance Detection |  Classify a text as either pro, con, or neutral regarding a topic: TOPIC: climate change is a real concern TEXT: Be kind to the earth beneath your feet. #environment LABEL: pro |  Domain News (Ferreira and Vlachos, 2016) Debating (Walker et al., 2012) Social Media (Mohammad et al., 2016)
entail | Entailment |  Predict whether two sentences do entail or not-ential each other: DOMAIN: RTE SENTENCE-1: No Weapons of Mass Destruction Found in Iraq Yet SENTENCE-2: Weapons of Mass Destruction Found in Iraq LABEL: not entail |  Domain RTE (Wang et al., 2018) SciTail (Khot et al., 2018) HANS (McCoy et al., 2019)
x-review |  Multilingual Sentiment Classification (Prettenhofer and Stein, 2010) |  Classify product review as positive or negative: DOMAIN: books LANGUAE: de REVIEW: Ich war vor 5 Jahren in Indien … Ich kann dieses Buch nur empfehlen. LABEL: positive |  Domain books, dvd, music Lingual de, en, fr, jp
x-stance |  Multilingual Stance Detection (Vamvas and Sennrich, 2020) |  Classify a text as either favor, or against regarding a given topic: TOPIC: encomonmy LANGUAE: it TEXT: Non penso che tale ampliamento sia necessario, né urgente. LABEL: against |  Domain books, dvd, music Lingual de, fr, it

Table 5: Overview and examples of the used datasets and information about the
enforced distribution shift.

Task | Prompt
---|---
arg-qua | ARG-1 is MASK than ARG-2 regarding TOPIC
arg-sim | ARG-1 is MASK than ARG-2 regarding TOPIC
arg-cls | The attitude of ARG is MASK regarding TOPIC
evi-cls | TEXT is MASK evidence regarding TOPIC
review | The sentiment of REVIEW is MASK
stance | The attitude of TEXT is MASK regarding TOPIC
entail | SENTENCE-1? MASK, SENTENCE-2
x-stance |  de: Die Haltung von ARG ist MASK zu TOPIC fr: L’attitude de ARG est MASK envers TOPIC it: L’atteggiamento di ARG MASK verso TOPIC
x-review |  de: Die Stimmung von REVIEW ist MASK en: The sentiment of REVIEW is MASK fr: Le sentiment de REVIEW est MASK jp: REVIEW の感情は MASK です

Table 6: Overview of the used prompt templates for all tasks and lanugages.

###  A.4 Prompt-Based Fine-Tuning

In this paper, we adopt prompt-based fine-tuning (P+FT) (Schick and Schütze,
2021) as an alternative approach to vanilla fine-tuning (FT). Unlike FT, P+FT
relies on the pre-trained masked-language modeling (MLM) head and avoids using
new classification heads.

Figure 7:  An exemplary overview of prompt-based fine-tuning (P+FT) for the
sentiment classification task, following Schick and Schütze (2021): re-
formulating the task as cloze (prompt); gathering relevant tokens (verbalizer)
for the specific classes - positive (green) or negative (red); finding the
final prediction by summing up the probability of relevant tokens per class;
backpropagating the error through the MLM head and the LM.

#### Forward Pass

We show in Figure 7 an exemplary overview of P+FT for sentiment analysis given
two classes $K=\\{\textit{positive},\textit{negative}\\}$. In detail, we wrap
the review with a cloze template and add a masking token as the prediction
proxy. Next, the LM processes this prompt and outputs the most probable tokens
$T$ along with their log-probabilities $L$ using the MLM head. Then, the
verbalizer selects the relevant tokens $A$ within $L$ and assigns them a class
mapping - like positive (green) or negative (red). In contrast to other
probing work (Schick and Schütze, 2021; Raman et al., 2023), we automatically
select indicative tokens Schick et al. (2020) using the likelihood ratio
regarding every class $k$ in $K$ based on the train instances. With these
token-class mappings, we sum up the log-probabilities for every class $k$ in
$K$ as $w_{k}=\sum_{a\in A(k)}L(a)$ and apply the softmax (Equation 1) to find
$\hat{y}$.

$\hat{y}=\operatorname*{arg\,max}_{k\in
K}\dfrac{exp(w_{k})}{\sum_{k^{\prime}\in K}exp(w_{k^{\prime}})}$ (1)

#### Backward Pass

While the forward pass represents the prompting paradigm (P), we analyze and
evaluate LMs without any parameter optimization. However, to fine-tune the LM
and MLM head, we calculate the cross-entropy loss $\mathcal{L}_{CE}$ and
update the weights through back-propagation. Note, we initialize the automatic
verbalizer before the training and do not update it anymore afterward.

###  A.5 In-Context Learning Setup

As reported in § 4.2, we evaluated ChatGPT using in-context learning (ICL). In
detail, we provide four demonstration samples from the training instances for
every test instance. For a fair comparison with gradient-based approaches (FT,
P+FT), we allow to sample these demonstration instances from the entire
training set. We use BM25 to calculate the similarity between the test and
train instances. Afterward, we use the top-4 most similar train instances as a
demonstration for a given test instance.

##  Appendix B Additional Results

In addition to results shown in the main paper (§ 4.2), we show in the
following the effectiveness of P+FT for ID scenarios (§ B.1) and that other
methods to prevent freshly initialized classification heads underperforms
prompt-based fine-tuning (§ B.2).

###  B.1 In-Distribution Results

Table 7 shows the superior performance of prompt-based fine-tuning transfers
to ID scenarios.

| arg-qua | arg-sim | arg-cls | evi-cls | review | stance | entail | x-stance | x-review | Applicability | Reliability
---|---|---|---|---|---|---|---|---|---|---|---
| Top. | Top. | Top. | Top. | Dom. | Dom. | Dom. | Lang./Top. | Lang./Dom. | $\mu_{F_{1}}\pm\sigma_{F1}$ | $\mu_{\tau}\pm\sigma_{\tau}$
LPBERT | 55.7 | 69.9 | 58.5 | 70.4 | 85.5 | 55.3 | 72.1 | 58.9 | 76.3 | $67.0\pm 0.2$ | $-71.3\pm 3.8$
PBERT | 47.7 | 50.4 | 36.5 | 54.7 | NaN | 44.7 | 49.2 | 49.2 | 57.2 | $48.7\pm 0.0$ | -
FTBERT | 87.9 | 76.4 | 67.3 | 78.9 | 90.4 | 61.1 | 93.4 | 67.6 | 87.0 | $78.9\pm 0.4$ | $-83.7\pm 6.5$
P+FTBERT | 88.0 | 76.1 | 67.7 | 79.1 | 90.4 | 62.8 | 93.4 | 67.0 | 87.0 | $79.1\pm 0.3$ | $-78.7\pm 8.1$
LPDeBERTa-v3 | 55.1 | 72.5 | 60.3 | 71.1 | 89.3 | 53.2 | 87.1 | 59.6 | 85.5 | $70.4\pm 0.1$ | $-74.6\pm 3.0$
PDeBERTa-v3 | 55.1 | 60.5 | 41.7 | 61.5 | 63.3 | 46.1 | 57.8 | 52.2 | 53.4 | $54.6\pm 0.5$ | -
FTDeBERTa-v3 | 89.0 | 78.4 | 75.2 | 80.6 | 93.9 | 63.3 | 96.7 | 72.5 | 92.1 | $82.4\pm 0.4$ | $-92.3\pm 6.5$
P+FTDeBERTa-v3 | 90.3 | 81.5 | 78.9 | 81.5 | 94.8 | 70.1 | 96.5 | 71.4 | 92.3 | $84.1\pm 0.3$ | $-91.0\pm 7.0$
LPRoBERTa | 54.0 | 67.0 | 57.8 | 69.6 | 88.4 | 53.3 | 73.0 | 59.4 | 86.2 | $67.6\pm 0.2$ | $-80.6\pm 3.1$
PRoBERTa | NaN | 57.1 | 45.3 | 54.7 | 79.5 | 46.3 | 55.6 | 49.2 | 76.1 | $58.0\pm 0.0$ | -
FTRoBERTa | 84.0 | 79.4 | 71.0 | 80.9 | 92.9 | 64.7 | 94.9 | 58.6 | 91.0 | $79.7\pm 1.9$ | $-90.0\pm 4.7$
P+FTRoBERTa | 88.2 | 79.6 | 72.5 | 80.8 | 92.7 | 67.0 | 95.2 | 69.8 | 91.1 | $81.9\pm 0.3$ | $-85.7\pm 6.3$

Table 7: In-distribution (ID) results for BERT, DeBERTa-v3, and RoBERTa using
linear probing (LP), prompting (P), fine-tuning (FT), and prompt-based fine-
tuning (P+FT). We report average Applicability ($\mu_{F_{1}}$), Reliability
($\mu_{\tau}$), Stability ($\sigma_{F_{1}},\sigma_{\tau}$). Best OOD
performance within one LM are underlined and bold highlights best OOD
performance across LMs.

###  B.2 Classification Head Pre-Initialisation

In addition to prompt-based fine-tuning (P+FT), we experimented with pre-
initializing the classification head using a linear probe (LP+FT) following
Kumar et al. (2022b). As reported in Table 8, we did not find a positive
effect of using LP+FT.

| arg-qua | arg-sim | arg-cls | evi-cls | review | stance | entail | x-stance | x-review | Applicability | Reliability
---|---|---|---|---|---|---|---|---|---|---|---
| Top. | Top. | Top. | Top. | Dom. | Dom. | Dom. | Lang./Top. | Lang./Dom. | $\mu_{F_{1}}\pm\sigma_{F1}$ | $\mu_{\tau}\pm\sigma_{\tau}$
FTBERT | 75.5 | 68.4 | 57.5 | 74.7 | 89.3 | 31.1 | 50.7 | 62.0/63.9 | 77.7/84.4 | $66.8\pm 0.9$ | $-56.8\pm 12.3$
LP+FTBERT | 75.7 | 66.5 | 57.3 | 74.1 | 89.3 | 34.2 | 50.4 | 60.8/64.1 | 77.1/84.0 | $66.7\pm 1.4$ | $-56.5\pm 14.3$
P+FTBERT | 76.2 | 66.0 | 59.8 | 75.7 | 89.3 | 28.5 | 48.0 | 59.5/63.6 | 79.6/83.9 | $66.4\pm 1.1$ | $-61.7\pm 12.4$
FTDeBERTa-v3 | 78.4 | 75.4 | 64.0 | 77.3 | 93.4 | 29.6 | 55.6 | 69.8/69.3 | 91.3/90.9 | $72.3\pm 1.1$ | $-72.6\pm 13.4$
LP+FTDeBERTa-v3 | 78.4 | 75.6 | 63.7 | 76.5 | 93.6 | 30.1 | 54.7 | 69.6/69.1 | 91.1/91.1 | $72.1\pm 1.3$ | $-70.8\pm 11.9$
P+FTDeBERTa-v3 | 78.5 | 79.1 | 74.6 | 78.6 | 94.2 | 33.0 | 60.2 | 69.7/69.9 | 91.8/91.4 | $74.6\pm 0.9$ | $-78.4\pm 8.4$
FTRoBERTa | 70.9 | 73.0 | 56.9 | 77.5 | 92.2 | 30.0 | 51.3 | 62.2/66.8 | 89.6/90.1 | $69.1\pm 2.5$ | $-69.7\pm 10.4$
LP+FTRoBERTa | 76.0 | 73.9 | 54.3 | 77.2 | 92.1 | 27.3 | 47.6 | 62.3/67.0 | 89.1/89.2 | $68.7\pm 1.7$ | $-71.0\pm 11.7$
P+FTRoBERTa | 77.6 | 74.3 | 66.0 | 77.9 | 92.0 | 29.1 | 52.4 | 67.4/67.5 | 89.7/90.0 | $71.3\pm 0.5$ | $-75.5\pm 8.1$

Table 8: Comparing vanilla (FT), linear-probing fine-tuning afterward (LP+FT),
and prompt-based fine-tuning (P+FT) for BERT, DeBERTa-v3, and RoBERTa. We
report average Applicability ($\mu_{F_{1}}$), Reliability ($\mu_{\tau}$),
Stability ($\sigma_{F_{1}},\sigma_{\tau}$).