# Ahead-of-Time P-Tuning

Daniil Gavrilov, Nikita Balagansky
Tinkoff
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we propose Ahead-of-Time (AoT) P-Tuning, a novel parameter-
efficient fine-tuning method for pre-trained Language Models (LMs) that adds
input-dependent bias before each Transformer layer. We evaluate AoT P-Tuning
on GLUE and SuperGLUE benchmarking datasets using RoBERTa and DeBERTa models,
showing that it outperforms BitFit and is comparable or better than other
baseline methods for efficient fine-tuning. Additionally, we assess the
inference overhead of AoT P-Tuning and demonstrate that it introduces
negligible overhead compared to established baseline methods. Our method
enables multi-task inference with a single backbone LM, making it a practical
solution for real-world applications.

##  1 Introduction

As pre-trained Language Models (LMs) grow in size (Radford et al., 2019;
Devlin et al., 2019; Raffel et al., 2020), developing new methods for handling
these large models becomes increasingly important. One potential solution is
parameter-efficient fine-tuning, where only a subset of the model’s parameters
are optimized from scratch or using existing weights (Liu et al., 2021b, a;
Houlsby et al., 2019; Hu et al., 2022; Ben Zaken et al., 2022; Lester et al.,
2021).

Typically, the success of these parameter-efficient fine-tuning methods is
judged based on the number of optimized parameters and the model’s performance
on downstream tasks. However, in this paper, we also consider two additional
factors: the inference overhead that results from fine-tuning and the model’s
ability to perform multi-task inference. The latter refers to using a single
LM to handle multiple tasks during inference, which is advantageous for real-
world applications (Lester et al., 2021).

Most current methods, such as P-Tuning (Liu et al., 2021b, a; Lester et al.,
2021), Adapters (Houlsby et al., 2019), and LoRA (Hu et al., 2022), have a
trade-off between introducing inference overhead and the ability to perform
multi-task inference. BitFit (Ben Zaken et al., 2022) does not introduce any
overhead and can handle multi-task inference, but its performance is lacking
compared to other methods.

In this paper, we propose a simple parameter-efficient fine-tuning method for
LMs called Ahead-of-Time (AoT) P-Tuning. This method involves adding input-
dependent bias before each Transformer layer (Vaswani et al., 2017).
Furthermore, AoT P-Tuning can be used in multi-task inference setups with a
single backbone LM.

Our contributions in this paper can be summarized as follows:

  1. 1.

We introduce Ahead-of-Time (AoT) P-Tuning, a parameter-efficient fine-tuning
method that adds input-dependent bias before each Transformer layer.

  2. 2.

We test the proposed method on GLUE and SuperGLUE benchmarking datasets (Wang
et al., 2018, 2019) using RoBERTa (Liu et al., 2019) and DeBERTa (He et al.,
2020) models. Our experiments show that AoT P-Tuning outperforms BitFit and is
comparable or better than other baseline methods for efficient fine-tuning.

  3. 3.

We measure the inference overhead of AoT P-Tuning and demonstrate that it
introduces negligible overhead compared to established baseline methods.

Figure 1: Schematic comparison of P-Tuning v2 (left), and AoT P-Tuning (right). Since the sequence length is not increased, AoT P-Tuning takes significantly less time to evaluate, only requiring the overhead of adding biases to the input sequence (See Section 4.4 for experiments with inference speed). Method | Parameter Efficient | Zero-Cost | Multi-Task Inference
---|---|---|---
Fine-Tuning |  |  |
LoRA |  |  |
LoRA Fused |  |  |
Adapters |  |  |
BitFit |  |  |
P-Tuning v1/v2 |  |  |
AoT P-Tuning (ours) |  |  |
Table 1: Schematic comparison of recent fine-tuning methods with AoT P-Tuning.
Recent fine-tuning approaches either allow inference with no computational
overhead or multi-task inference. See Section 3.1 for details.

##  2 Recent Works

Currently, a wide range of different methods could be referenced with
P-Tuning. Liu et al. (2021b) proposed to add soft prompts to the embeddings of
GPT-2’s input sequence (Radford et al., 2019) to train it on classification
tasks. Lester et al. (2021) proposed a scheme similar to the one used in Liu
et al. (2021b), but trained a T5 model (Raffel et al., 2020) with P-Tuning to
show how the performance of the method changes with the increased scale of the
backbone model.

Recently, Qin & Eisner (2021); Li & Liang (2021); Liu et al. (2021a) proposed
to add prefixes not only to input embeddings but also at each layer of the
Transformer model. In addition, Liu et al. (2021a) suggested training a linear
classification head on top of the backbone model instead of utilizing a LM
head to obtain classification results.

Due to this range of similar methods, we will follow the naming used by Liu et
al. (2021a) and refer to Prompt-Tuning (adding soft prompts to the input
embeddings) as P-Tuning v1 and to Prefix-Tuning (adding soft prefixes at each
layer of Transformer backbone) as P-Tuning v2.

Hu et al. (2022) proposed to train low-rank changes of attention weights,
while Houlsby et al. (2019) fine-tuned additional model layers, which can also
be considered parameter-efficient. Ben Zaken et al. (2022) proposed to fine-
tune only bias terms of the model.

##  3 Ahead-of-Time P-Tuning

###  3.1 Motivation

Parameter-efficient fine-tuning methods are crucial for reducing computational
and memory requirements, enabling deploying of advanced AI models in resource-
constrained environments and real-world applications. In discussions of
parameter-efficient fine-tuning methods, recent research often refers to the
number of parameters utilized for optimization (Liu et al., 2021b, a; Houlsby
et al., 2019; Hu et al., 2022; Ben Zaken et al., 2022; Lester et al., 2021). A
method can be considered superior to another if it uses fewer trained
parameters and achieves better performance in terms of metrics on downstream
tasks.

In this study, we categorize methods based on two additional characteristics:
the inference overhead produced by the parameter-efficient fine-tuning method
and the capability to perform multi-task inference after fine-tuning multiple
models for distinct tasks. While minimizing inference overhead is beneficial
for faster model evaluation, the practicality of multi-task inference is
debatable. However, multi-task inference is desirable for high-demand systems
deployed in real-world scenarios. It can enable more efficient resource
allocation, reduce the overall number of models required, and facilitate
dynamic workload adjustment based on different tasks as needed. Using the same
backbone model for various tasks promotes more streamlined serving, as all
workers share the same model in memory. This approach enables dynamic workload
adjustment based on different tasks as needed. In contrast, without multi-task
inference, some workers might be underutilized depending on external
conditions, resulting in suboptimal resource allocation that could have been
used for other tasks.

Table 1 compares recent methods concerning these properties. P-Tuning v1/v2
has inference overhead because the length of sequences passed to the model
increases, while adapters introduce overhead by adding additional layers to
the model computation. LoRA’s analysis is more complex, as it can be evaluated
in two modes: one in which we fuse the weights of LoRA (Hu et al., 2022),
resulting in no inference overhead since the model is now identical to the
original pre-trained model, and another where we do not fuse weights, leading
to inference overhead due to the need for additional matrix multiplications
during evaluation.

In terms of multi-task inference capability, P-Tuning v1/v2 allows for this,
as different prompts can be stacked in a batch and propagated through the
backbone model. For Adapters, weights can also be stacked in a batch and
evaluated using batched matrix multiplication. However, it is important to
note that these layers must have the same shape, which is a tunable
hyperparameter. While multi-task inference with a fused LoRA is theoretically
possible, similar to Adapters, it demands a substantial amount of GPU memory.
For example, for the DeBERTa-XL model with layers of hidden size d = 1024 and
l = 48, passing a batch with b sequences necessitates storing 1024 * 1024 * 48
* 4 * b parameters for the model, where 4 represents the number of parameter
matrices of the Attention module. Notably, with b = 4, the number of original
parameters in the model is already exceeded, which is impractical. To address
this issue, one can opt not to fuse LoRA weights (to work with factorized
weights with low rank, thus reducing the total number of parameters passed as
a batch) for multi-task inference, although this can introduce computational
overhead.

In contrast, BitFit (Ben Zaken et al., 2022) is advantageous in terms of both
criteria: it only modifies biases of the model, thereby not altering its
structure or introducing computational overhead, and can easily be
parallelized for multi-task inference by stacking biases for different tasks
in a batch. However, based on our experimental results in subsequent sections,
its performance is inferior to other methods.

In light of these considerations, this paper aims to answer the following
research question: Can we develop a parameter-efficient fine-tuning method
that combines the rapid evaluation and multi-task inference capabilities of
BitFit while outperforming it on downstream tasks?

###  3.2 Proposed Mechanism

With AoT P-Tuning, we propose to augment each Transformer layer with a simple
procedure. We define trainable matrices
${\bm{P}}^{i}\in\mathbb{R}^{|{\bm{V}}|\times d}$ for each layer. Then, before
the evaluation of the $i$-th layer, we modify the hidden states as follows:

$\begin{split}&{\bm{H}}^{\prime
i}={\bm{H}}^{i}+{{\bm{P}}^{i}_{x_{1}},\dots,{\bm{P}}^{i}_{x_{n}}}\in\mathbb{R}^{n\times
d},\end{split}$ (1)

where ${\bm{P}}^{i}_{x_{j}}\in\mathbb{R}^{d}$ is a lookup of $x_{j}$-th prompt
embedding from ${\bm{P}}^{i}$. Such a scheme saves us significant time during
evaluation since AoT P-Tuning does not imply an increase in sequence length
and involves only adding biases during the evaluation, which introduces
negligible computational overhead.

While ${\bm{P}}^{i}$ in naive implementation will require much memory to store
parameters, we describe reparametrizations that make training more tractable
in the following subsections.

Note that AoT P-Tuning could be evaluated in parallel with several tasks in a
batch due to the fact that performing look-up from ${\bm{P}}$ can be easily
parallelized. As for other methods for parameter-efficient fine-tuning, we
only optimize parameters of ${\bm{P}}$ and Classification Head during fine-
tuning.

###  3.3 On the Parameter Efficiency of AoT P-Tuning

It is notable that, in most cases, it is not feasible to optimize the weight
${\bm{P}}\in\mathbb{R}^{|{\bm{V}}|\times d}$ for each layer. If we consider
training RoBERTa-Large with such a scheme (which has $|{\bm{V}}|=50265$,
$d=1024$ and $l=24$), then storing all biases ${\bm{P}}$ will exceed $1.2$B
parameters, while the model itself has roughly $350$M parameters.

To overcome this limitation, we propose two reparametrizations of ${\bm{P}}$
so that it can use fewer parameters during training.

The first is based on the Kronecker product (namely, Kronecker AoT P-Tuning).
More specifically, we reparametrize ${\bm{P}}$ as

${\bm{P}}=({\bm{W}}_{L}\otimes{\bm{W}}_{M}){\bm{W}}_{R},$ (2)

where ${\bm{W}}_{L}\in\mathbb{R}^{a\times r}$,
${\bm{W}}_{M}\in\mathbb{R}^{b\times r}$,
${\bm{W}}_{R}\in\mathbb{R}^{r^{2}\times d}$, $a$ and $b$ are selected in such
a way so $a*b=|{\bm{V}}|$, $r$ is the factorization rank which is a
hyperparameter to tune, and $\otimes$ denotes the Kronecker product.

With this reparametrization, training AoT P-Tuning becomes tractable. E.g.,
for RoBERTa-Large, with $a=256$, $b=200$, and $r=20$, ${\bm{P}}$ will contain
roughly $10$M parameters, which is less than $3\%$ of the total number of
parameters in the model111One may note that $256*200=51200\neq 50265$.
However, $50265$ is difficult to factorize efficiently since
$50265=1117*3^{2}*5$. Because of this, we chose to mostly factorize ${\bm{P}}$
in such a way as to make it slightly larger than the original vocabulary size.
Doing so allows us to select more appropriate $a$ and $b$ from the perspective
of parameter and computational efficiency..

The second approach to work with ${\bm{P}}$, which we used in our experiments,
is based on passing the embeddings matrix ${\bm{E}}$ through a learnable Fully
Connected network (namely, FC AoT P-Tuning). Thus, we reparametrize ${\bm{P}}$
as

${\bm{P}}=f({\bm{E}}{\bm{W}}_{1}+{\bm{b}}_{1}){\bm{W}}_{2}+{\bm{b}}_{2},$ (3)

where ${\bm{W}}_{1}\in\mathbb{R}^{d\times r}$,
${\bm{b}}_{1}\in\mathbb{R}^{r}$, ${\bm{W}}_{2}\in\mathbb{R}^{r\times d}$,
${\bm{b}}_{2}\in\mathbb{R}^{d}$, $f$ is a non-linearity, and $r$ is the
mapping rank, which is also a hyperparameter to tune, same as for Kronecker
AoT P-Tuning.

With FC AoT P-Tuning, we utilize knowledge stored in the pre-trained
embeddings matrix ${\bm{E}}$, which should hypothetically perform better than
training ${\bm{P}}$ from scratch as Kronecker AoT P-Tuning.

Note that for both Kronecker and FC AoT P-Tuning, we can evaluate only
specific rows $\\{{\bm{P}}_{x_{i}},\dots,{\bm{P}}_{x_{n}}\\}$ for input
sequence $\\{x_{1},\dots,x_{n}\\}$, making training more efficient.

For both reparametrizations, ${\bm{P}}$ could be fused once training is
complete, and thus the rank of factorization $r$ does not affect inference
speed. During the evaluation, there is no need to store the full ${\bm{P}}$ in
GPU memory. Instead, it could be stored in RAM, and only rows of these
matrices should be placed in GPU memory to be added to the hidden states
before each layer.

From a certain perspective, choosing between AoT P-Tuning and P-Tuning is a
trade-off between evaluation speed and RAM consumption during inference. If
RAM is limited, then usual P-Tuning could be used at the cost of slower
inference. In other cases, AoT P-Tuning is viable if there is enough RAM and
inference speed is crucial. Although, in most cases, ${\bm{P}}$ matrices for
different tasks could be easily stored in the RAM. For RoBERTa-Large, a single
task parameter will require roughly $2.4$Gb if stored in half-precision.

###  3.4 Intuition Behind AoT P-Tuning and Connection to Other Methods

Having ${\bm{H}}^{\prime}$, after passing through ${\bm{W}}_{Q}$,
${\bm{W}}_{K}$, and ${\bm{W}}_{V}$ we obtain ${\bm{Q}}^{\prime}$,
${\bm{K}}^{\prime}$, and ${\bm{V}}^{\prime}$. Note that
${\bm{V}}^{\prime}={\bm{H}}{\bm{W}}_{V}+\\{{\bm{P}}_{x_{1}},\dots,{\bm{P}}_{x_{n}}\\}{\bm{W}}_{V}\stackrel{{\scriptstyle\text{def}}}{{=}}{\bm{V}}+{\bm{P}}_{x}{\bm{W}}_{V}$.

The result of evaluating Attention with AoT P-Tuning could be seen as:

$\begin{split}{\bm{A}}_{i}^{\prime}=&\sum_{j=1}^{n}{\bm{a}}_{j}({\bm{Q}}_{i}^{\prime},{\bm{K}}^{\prime}){\bm{P}}_{x_{j}}{\bm{W}}_{V}+\sum_{j=1}^{n}{\bm{a}}_{j}({\bm{Q}}_{i}^{\prime},{\bm{K}}^{\prime}){\bm{V}}_{j}.\end{split}$
(4)

From such a perspective, there is a clear connection between AoT P-Tuning
(Equation 4) and P-Tuning v2 (Equation 9) with the following changes:

  1. 1.

For AoT P-Tuning, attention weights ${\bm{a}}_{j}$, $j\in\overline{1,l}$ are
used for both terms in Equation 4.

  2. 2.

For AoT P-Tuning, attention is evaluated on modified ${\bm{Q}}^{\prime}$. In
addition, there is a difference in the form of dependency of
${\bm{K}}^{\prime}$ and ${\bm{V}}^{\prime}$ on prefix weight. For AoT
P-Tuning, we add prefixes to ${\bm{K}}$ and ${\bm{V}}$, while for P-Tuning v2,
prefixes are concatenated to these matrices.

  3. 3.

For AoT P-Tuning, the first term of Equation 4 implies evaluation of Attention
with a prompt which is dependent on the input text, while for P-Tuning v2, the
prompt ${\bm{P}}_{V}$ is constant.

Considering Equation 4, AoT can be seen as a form of the P-Tuning method, for
which we embed prefixes before evaluating the attention layer222It is possible
to think of AoT P-Tuning as a method which adds bias after the evaluation of
the Transformer layer. In this case, it could be seen as a method that
directly models the result of the evaluation of P-Tuning v2 with a slightly
different computation order. However, we believe that this way is more
difficult to consider..

Also, one may note that AoT P-Tuning is highly related to BitFit (Ben Zaken et
al., 2022). Both methods are identical in their practical properties as they
are zero-cost during the inference and allow multi-task inference (See Table
1). Although, there is a clear difference in the form of biases added during
model evaluation: BitFit adds constant bias to each element in the sequence as
${\bm{H}}^{\prime i}={\bm{H}}^{i}+\\{{\bm{b}},\dots,{\bm{b}}\\}$. Such bias
leads to implied Attention results in the form of:

$\begin{split}{\bm{A}}_{i_{BitFit}}^{\prime}={\bm{b}}{\bm{W}}_{V}+\sum_{j=1}^{n}{\bm{a}}_{j}({\bm{Q}}_{i}^{\prime},{\bm{K}}^{\prime}){\bm{V}}_{j},\end{split}$
(5)

which no longer modifies the attention map with input-dependent bias. In the
latter sections, we will show that such discrepancy leads to poor performance
of BitFit during fine-tuning compared to AoT P-Tuning.

##  4 Experiments

###  4.1 Experimental Details

We compared AoT P-Tuning (Kronecker and FC reparametrizations of ${\bm{P}}$)
with other fine-tuning methods capable of performing multi-task inference:
P-Tuning v1, P-Tuning v2 on GLUE and SuperGLUE (Wang et al., 2018, 2019)
Benchmarking Datasets. We also evaluated plain fine-tuning, LoRA, Adapters,
and BitFit for reference. For each fine-tuning approach, we experimented with
the RoBERTa-Base, RoBERTa-Large, and DeBERTa-XL backbone models.

For each task, we performed a grid hyperparameter search (see Appendix Table 2
for hyperparameter ranges). For RoBERTa models, we evaluated each
hyperparameter set with $5$ different seed values and reported median and std
score values for each task. For DeBERTa-XL, we used to assess each
hyperparameter assignment with a single seed due to longer training time. See
Appendix Table 1 for a list of metrics used for each task.

(a)

(b)

Figure 2: SuperGLUE macro score for RoBERTa-Base (a) and DeBERTa-XL (b)
models. See Section 4.2 for details.

We used the Adam (Kingma & Ba, 2015) optimizer with a constant learning rate
for each task. We stopped training once the validation metric stopped
increasing (see the "patience" parameter in Appendix Table 4).

For Kronecker AoT P-Tuning with RoBERTa models, we parametrized the matrix
${\bm{P}}=({\bm{W}}_{L}\otimes{\bm{W}}_{M}){\bm{W}}_{R}$ with $a=256$, and
$b=200$, while for DeBERTa, we used $a=b=360$. ${\bm{W}}_{L}$ and
${\bm{W}}_{M}$ were initialized randomly, while ${\bm{W}}_{R}$ was initialized
as a zero matrix. For FC AoT P-Tuning, we initialized ${\bm{W}}_{1}$ randomly,
while ${\bm{W}}_{2}$, ${\bm{b}}_{1}$, and ${\bm{b}}_{2}$ were initialized with
zeros. For Kronecker AoT P-Tuning, we applied dropout (Srivastava et al.,
2014) to the ${\bm{P}}_{x}$ with a fixed probability equal to $0.1$. In
contrast, for FC AoT P-Tuning, we applied dropout to ${\bm{E}}$ before
multiplying it with ${\bm{W}}_{1}$.

Each experiment was run on a single NVIDIA A100 GPU with a total computation
time of roughly $1200$ days.

RoBERTa-Large
---
Model | RTE | COPA | WSC | WiC
Fine-Tuning | 88.1 ± 1.5 | 87.0 ± 10.2 | 80.8 ± 6.3 | 73.8 ± 1.6
Adapters | 87.7 ± 18.5 | 89.0 ± 10.1 | 77.9 ± 9.8 | 73.5 ± 1.0
LoRA | 87.4 ± 1.6 | 91.0 ± 8.5 | 79.8 ± 10.6 | 71.9 ± 1.2
BitFit | 87.7 ± 0.8 | 91.0 ± 2.3 | 71.2 ± 6.7 | 71.3 ± 9.5
P-Tuning v1 | 62.8 ± 2.3 | 75.0 ± 4.3 | 66.3 ± 1.3 | 64.1 ± 0.9
P-Tuning v2 | 87.4 ± 1.5 | 87.0 ± 6.3 | 75.0 ± 7.7 | 70.8 ± 1.5
Kron. AoT P-Tuning (ours) | 84.8 ± 1.3 | 72.0 ± 9.1 | 67.3 ± 3.0 | 71.0 ± 1.0
FC AoT P-Tuning (ours) | 88.4 ± 0.9 | 85.0 ± 10.1 | 79.8 ± 4.1 | 72.1 ± 1.5
| MultiRC | CB | BoolQ | Macro
Fine-Tuning | 83.3 ± 1.1 | 97.3 ± 2.8 | 85.6 ± 0.3 | 85.1
Adapters | 83.7 ± 20.3 | 100.0 ± 0.0 | 85.7 ± 10.6 | 85.4
LoRA | 75.7 ± 17.4 | 100.0 ± 2.6 | 84.6 ± 0.6 | 84.3
BitFit | 82.5 ± 0.6 | 100.0 ± 0.7 | 85.4 ± 1.0 | 84.2
P-Tuning v1 | 54.3 ± 2.9 | 81.4 ± 3.0 | 64.3 ± 1.2 | 66.9
P-Tuning v2 | 82.4 ± 0.6 | 100.0 ± 0.8 | 85.0 ± 0.6 | 83.9
Kron. AoT P-Tuning (ours) | 82.8 ± 0.8 | 97.3 ± 2.3 | 84.8 ± 0.5 | 80.0
FC AoT P-Tuning (ours) | 82.7 ± 19.3 | 100.0 ± 0.0 | 85.5 ± 10.3 | 84.8
DeBERTa-XL
Model | RTE | COPA | WSC | WiC
Fine-Tuning | 89.9 | 96.0 | 76.9 | 75.9
Adapters | 90.3 | 96.0 | 89.4 | 77.3
LoRA | 90.3 | 97.0 | 89.4 | 75.5
BitFit | 89.2 | 97.0 | 86.5 | 73.7
P-Tuning v1 | 78.3 | 90.0 | 67.3 | 66.8
P-Tuning v2 | 90.6 | 97.0 | 89.4 | 76.5
Kron. AoT P-Tuning (ours) | 88.8 | 96.0 | 87.5 | 71.8
FC AoT P-Tuning (ours) | 91.0 | 98.0 | 94.2 | 74.1
| MultiRC | CB | BoolQ | Macro
Fine-Tuning | 84.3 | 98.4 | 86.7 | 86.9
Adapters | 86.7 | 97.3 | 88.9 | 89.4
LoRA | 86.0 | 100.0 | 88.3 | 89.5
BitFit | 85.2 | 100.0 | 86.5 | 88.3
P-Tuning v1 | 82.1 | 93.8 | 79.4 | 79.7
P-Tuning v2 | 87.1 | 97.3 | 87.0 | 89.3
Kron. AoT P-Tuning (ours) | 86.3 | 83.1 | 87.3 | 85.8
FC AoT P-Tuning (ours) | 86.5 | 92.3 | 88.1 | 89.2
Table 2: Results on the SuperGLUE Dev set. For RoBERTa-Large, each result is
median and std across several seeds, and the Macro column is a mean score
across all tasks. For DeBERTa-XL, we evaluated each hyperparameter assignment
with a single seed and reported its metric score. We bolded the best results
and underlined the second best results. Fine-tuning is omitted from comparison
with other methods and was not bolded for visibility. See Section 4.2 for
details.

###  4.2 Results

Refer to Table 2 and Appendix Table 3 for the trained model results.

Our observations indicate that FC AoT P-Tuning outperforms Kronecker AoT
P-Tuning. This outcome mainly results from FC reparametrization using a pre-
trained embedding matrix rather than learning biases from scratch.

We also observed that FC AoT P-Tuning performs better or is similar to
P-Tuning v1/v2 and LoRA measured by Macro score. Additionally, FC AoT P-Tuning
primarily yields comparable results to Adapters. In cases where Adapters
outperform AoT P-Tuning, the gains are usually marginal based on our
experiments. Furthermore, AoT P-Tuning surpasses BitFit in most instances.
Only for the RoBERTa-Base backbone evaluated with GLUE Macro score, BitFit
performs on par with AoT P-Tuning. We noticed that both AoT P-Tuning
reparametrizations predominantly exhibit a lower variance of metrics across
different seeds.

Note that the results presented in Table 2 and Appendix Table 3 were obtained
using different training hyperparameters, including distinct factorization
ranks for LoRA and Adapters. This implies that if we consider having the same
rank for different tasks for LoRA or Adapters, which is essential for multi-
task inference (see Section 3.1 for details), these results could represent an
upper bound of performance. Refer to Figure 2 for SuperGLUE macro scores with
varying prefix lengths $p$, and prefix ranks $r$ for different methods. Also,
remember that while we included AoT P-Tuning in these figures for reference,
AoT P-Tuning does not require the same rank for multi-task inference since
weights are fused during evaluation, and $r$ no longer affects any shape. When
considering the same rank for Adapters and LoRA, these methods only marginally
outperform BitFit. We also observed that LoRA struggled with large $r$.

Based on our experimental results, if the ability to perform multi-task
inference without computational overhead is essential, then AoT P-Tuning will
not significantly diminish performance compared to LoRA and Adapters and could
be utilized.

(a)

(b)

(c)

Figure 3: Speed measurements for baseline methods with sequence length equal
to $384$ for DeBERTa-XL. See Appendix Figures 5, 6 for results with other
sequence lengths and Section 4.4 for details.

With per-task Expected Validation Performance (EVP) (Dodge et al., 2019), we
observed that AoT P-Tuning highly depends on the number of hyperparameter
assignments (see Appendix Figures 2, 4). Although, in most cases, using less
than $100$ hyperparameter assignments for AoT P-Tuning is enough for it to
outperform P-Tuning v2, which is not crucial in most cases.

###  4.3 Analysis of Trained Weights

We investigated trained ${\bm{P}}$ matrices for WSC, COPA, CB, and RTE tasks
with the DeBERTa-XL model. Since FC AoT P-Tuning performed better than
Kronecker factorization, we selected this reparametrization method to report
the results.

More specifically, we sorted rows of ${\bm{P}}$ matrices for each layer
measured by the $L_{2}$ norm and reported the appropriate tokens for these
rows. See Appendix Tables 5, 6, 7, 8 for results.

For the WSC task, there is a clear interpretation of trained rows for
${\bm{P}}$, since rows with a large $L_{2}$ norm represent tokens responsible
for pronouns and names, which is crucial for solving WSC. For the COPA task,
we observed that the model tends to assign large norms for verb tokens. For
the RTE and CB tasks, ${\bm{P}}$ also assigns large norms for name tokens,
which often occur in the training data, while CB primarily modifies adverbs
for later layers.

###  4.4 Inference Speed Overhead

In Figure 3 and Appendix Figures 5, 6, we investigated the computational
overhead of AoT P-Tuning compared to other baselines.

We estimated inference time for RoBERTa-Base, RoBERTa-Large, and DeBERTa-XL
models with batch sizes $\in[1,16,64]$ and sequence lengths $\in[64,128,384]$.
For batch size equal to $1$, we evaluated the model $300$ times, and $100$
times for other values. For evaluation, we used A100 80Gb GPU. We report mean
values of inference time normalized by the inference time of the vanilla model
(i.e., plain fine-tuning).

We evaluated AoT P-Tuning on two setups. For the first setup, we fused
${\bm{P}}$ so that the model can perform on its top speed. We did not perform
fusing for the second setup. While a lack of fusing is not required to use AoT
P-Tuning, we report these results for reference. We also report LoRA results
for the experiments in which we did not fuse weights. This setup makes it
possible to use LoRA in a multi-task setup (see Section 3.1 for details).

We observed that the proposed method performed differently depending on the
experiment’s parameters. AoT performed with computational overhead for a small
model (i.e., RoBERTa-Base), small sequence length, and small batch size. We
observed $12\%$ overhead on inference speed compared to plain fine-tuning for
RoBERTa-Base with batch size $1$ and sequence length $64$. However, AoT
P-Tuning still performed faster than other baselines by a large margin (i.e.,
LoRA for RoBERTa-Base with batch size $1$ and sequence length $384$ added
$50-70\%$ computational overhead compared to fine-tuning).

Once model size or input size is increased, we observed that the overhead of
adding biases for AoT P-Tuning becomes negligible. The proposed method
performed the same as plain fine-tuning (for some experiments, we observed
$1-2\%$ overhead, while for others, AoT P-Tuning performed even faster than
fine-tuning, which is a variation in inference time measurements). For the
most practical setup with large models (DeBERTa-XL), small batch size ($=1$)
and long sequence length ($=384$), AoT P-Tuning performed slightly faster than
fine-tuning, while other methods performed $12-25\%$ slower. LoRA and Adapters
showed $10\%$ overhead for larger batch sizes, while AoT P-Tuning still
performed with negligible overhead.

##  5 Conclusion and Future Work

This paper focused on parameter-efficient fine-tuning methods emphasizing
inference speed and multi-task capabilities. We observed that widely used
methods typically involve a trade-off between fast and multi-task inference,
while those that can perform both tend to struggle in downstream task
evaluations. To address this challenge, we introduced a novel method called
AoT P-Tuning, which has negligible inference overhead while maintaining multi-
task performance.

Our experiments demonstrated that AoT P-Tuning outperforms BitFit, which also
does not introduce any overhead and could be evaluated in a multi-task manner.
Furthermore, AoT P-Tuning is competitive with other methods across various
downstream tasks and pre-trained models, offering faster inference times.

Although we investigated two reparametrizations based on the Kronecker product
and fully connected (FC) network, further exploration of alternative
reparametrizations for weight ${\bm{P}}$ may enhance the performance of our
proposed method. Moreover, while our approach is straightforward, implementing
various architectural modifications could improve AoT P-Tuning’s performance
and decrease the need for hyperparameter tuning.

## References

  * Ben Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 1–9, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-short.1.  URL https://aclanthology.org/2022.acl-short.1.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith.  Show your work: Improved reporting of experimental results.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2185–2194, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1224.  URL https://aclanthology.org/D19-1224.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  _CoRR_ , abs/2006.03654, 2020.  URL http://dblp.uni-trier.de/db/journals/corr/corr2006.html#abs-2006-03654.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 2790–2799. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/houlsby19a.html.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _ICLR 2022_ , April 2022.  URL https://www.microsoft.com/en-us/research/publication/lora-low-rank-adaptation-of-large-language-models/.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.243.  URL https://aclanthology.org/2021.emnlp-main.243.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 4582–4597, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.353.  URL https://aclanthology.org/2021.acl-long.353.
  * Liu et al. (2021a) Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _CoRR_ , abs/2110.07602, 2021a.  URL https://arxiv.org/abs/2110.07602.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  Gpt understands, too.  _arXiv:2103.10385_ , 2021b.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.  URL http://arxiv.org/abs/1907.11692.  cite arxiv:1907.11692.
  * Qin & Eisner (2021) Guanghui Qin and Jason Eisner.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 5203–5212, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.410.  URL https://aclanthology.org/2021.naacl-main.410.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pp. 5998–6008, 2017.
  * Wang et al. (2018) Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018.  URL http://arxiv.org/abs/1804.07461.  cite arxiv:1804.07461Comment: https://gluebenchmark.com/.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf.

##  Appendix A Background

###  A.1 Evaluation of Transformer

Having an input sequence ${\bm{x}}=\\{x_{1},\dots,x_{n}\\}$, where $x_{i}$ is
token index, the embeddings of input texts are evaluated as
${\bm{H}}^{0}=\\{{\bm{E}}_{x_{1}},\dots,{\bm{E}}_{x_{n}}\\}$, where
${\bm{E}}\in\mathbb{R}^{|{\bm{V}}|\times d}$ is the embeddings matrix,
$|{\bm{V}}|$ is the vocabulary size, $d$ is the size of the hidden state of
the model, and ${\bm{E}}_{x_{i}}$ is an embedding of the token $x_{i}$. Hidden
states ${\bm{H}}^{i}$ are then passed to the $(i+1)$-th layer of the
Transformer to evaluate ${\bm{H}}^{i+1}$ with a total $l$ number of layers. To
do so, ${\bm{H}}^{i}$ are first mapped through three matrices ${\bm{W}}_{Q}$,
${\bm{W}}_{K}$, ${\bm{W}}_{V}\in\mathbb{R}^{d\times d}$ to get ${\bm{Q}}$,
${\bm{K}}$ and ${\bm{V}}$, which are then used to evaluate the attention
layer’s results as:

$\begin{split}{\bm{A}}&=\text{attention}({\bm{Q}},{\bm{K}},{\bm{V}})=\text{softmax}(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d}}){\bm{V}}\in\mathbb{R}^{n\times
d}.\end{split}$ (6)

After ${\bm{A}}$ is evaluated, it is passed through the remaining layers333In
fact, Transformer architecture implies evaluation of multi-head Attention. We
omit this in this paper for simplicity since all derivations could be easily
extended on the multi-head case., including residual connections and FC layers
to get ${\bm{H}}^{i+1}$. Here and later, we omit the layer index $i$ for
attention result ${\bm{A}}$ for visibility.

###  A.2 P-Tuning v1

Having a pre-trained Transformer LM with parameters $\Theta$, instead of fine-
tuning all parameters of this model on a downstream task, it is possible to
define soft prompts ${\bm{P}}\in\mathbb{R}^{p\times d}$, where $p$ is the
length of prompt. ${\bm{P}}$ is then concatenated to input sequence embeddings
as:

${\bm{H}}^{\prime
0}=\text{concat}({\bm{P}},{\bm{H}}^{0})\in\mathbb{R}^{(p+n)\times d}.$ (7)

Then, only ${\bm{P}}$ and Classification Head are fine-tuned on a downstream
task, while $\Theta$ remains frozen444Original implementation of P-Tuning v1
implied utilizing the LM Head of a pre-trained model instead of training a
Classification Head.. Such parametrization of fine-tuning makes it possible to
perform multi-task inference.

###  A.3 P-Tuning v2

Instead of concatenation of a single prompt ${\bm{P}}$ to the ${\bm{H}}^{0}$,
P-Tuning v2 proposed to concatenate soft prefixes at each layer of the
Transformer model. To apply P-Tuning v2, soft prefixes
${\bm{P}}_{K},{\bm{P}}_{V}\in\mathbb{R}^{p\times d}$ are defined for each
layer and concatenated to the ${\bm{K}}$ and ${\bm{V}}$ matrices before
evaluating the attention
${\bm{K}}^{\prime}=\text{concat}({\bm{P}}_{K},{\bm{K}})$,
${\bm{V}}^{\prime}=\text{concat}({\bm{P}}_{V},{\bm{V}})$. Then, Attention is
evaluated as follows:

$\begin{split}&{\bm{A}}^{\prime}=\text{attention}({\bm{Q}},{\bm{K}}^{\prime},{\bm{V}}^{\prime}),\end{split}$
(8)

where $i$-th component of ${\bm{A}}^{\prime}$ could be then written as:

$\begin{split}{\bm{A}}_{i}^{\prime}=&\sum_{j=1}^{p}{\bm{a}}_{j}({\bm{Q}}_{i},{\bm{K}}^{\prime}){\bm{P}}_{{\bm{V}}_{j}}+\sum_{k=1}^{n}{\bm{a}}_{k+p}({\bm{Q}}_{i},{\bm{K}}^{\prime}){\bm{V}}_{k}.\end{split}$
(9)

Note that ${\bm{a}}\in\mathbb{R}^{p+n}$ are attention weights for the $i$-th
token (we omit the $i$-th index for simplicity) and thus
$\sum_{j=1}^{p+n}{\bm{a}}_{j}=1$.

As for P-Tuning v1, only parameters of soft prefixes
${\bm{P}}_{K},{\bm{P}}_{V}$ and Classification Head are optimized on a
downstream task while freezing the parameters of a backbone model.

Task | Metric | Task | Metric
---|---|---|---
CoLA | Mattews Correlation | BoolQ | Accuracy
MRPC | $\frac{\text{Accuracy}+\text{F}1}{2}$ | CB | $\frac{\text{Accuracy}+\text{F}1}{2}$
RTE | Accuracy | RTE | Accuracy
SST-2 | Accuracy | COPA | Accuracy
MNLI | Accuracy | MultiRC | $\frac{\text{Accuracy}+\text{F}1}{2}$
QNLI | Accuracy | WSC | Accuracy
QQP | $\frac{\text{Accuracy}+\text{F}1}{2}$ | WiC | Accuracy
STSB | $\frac{\text{Pearson}+\text{Spearman}}{2}$ |  |
Table 3: Metrics used in our experiments for each task. See Section 4 for more
details.

Parameter | Range
---|---
All Tasks, except RTE
P-Tuning v1/v2/AoT
batch size |  $16$, $64$
learning rate |  $1\mathrm{e}{-4}$, $5\mathrm{e}{-4}$, $5\mathrm{e}{-3}$, $1\mathrm{e}{-3}$
$p$ |  $5$, $10$, $20$, $50$, $100$
LoRA $r$ |  $2$, $4$, $16$, $32$, $64$
Adapters $r$ |  $16$, $32$, $64$, $128$, $256$
Kron. $r$ |  $5$, $10$, $25$, $30$, $50$
FC $r$ |  $32$, $64$, $128$, $256$, $512$
Fine-Tuning
learning rate |  $1\mathrm{e}{-5}$, $5\mathrm{e}{-5}$, $1\mathrm{e}{-4}$, $5\mathrm{e}{-4}$, $5\mathrm{e}{-3}$
RTE
batch size |  $16$, $32$, $64$, $128$
learning rate |  $1\mathrm{e}{-5}$, $5\mathrm{e}{-5}$, $1\mathrm{e}{-4}$, $5\mathrm{e}{-4}$, $5\mathrm{e}{-3}$, $1\mathrm{e}{-3}$, $2\mathrm{e}{-3}$, $1\mathrm{e}{-2}$

Parameter | Range
---|---
P-Tuning v1/v2/AoT
batch size |  $16$, $32$, $64$
learning rate |  $5\mathrm{e}{-5}$, $1\mathrm{e}{-4}$, $3\mathrm{e}{-4}$, $5\mathrm{e}{-4}$, $1\mathrm{e}{-3}$, $2\mathrm{e}{-3}$, $5\mathrm{e}{-3}$
$p$ |  $5$, $10$, $20$, $50$, $100$
LoRA $r$ |  $2$, $4$, $16$, $32$, $64$
Adapters $r$ |  $16$, $32$, $64$, $128$, $256$
Kron. $r$ |  $5$, $10$, $25$, $30$, $50$
FC $r$ |  $32$, $64$, $128$, $256$, $512$
Fine-Tuning
learning rate |  $1\mathrm{e}{-5}$, $5\mathrm{e}{-5}$, $1\mathrm{e}{-4}$, $5\mathrm{e}{-4}$, $5\mathrm{e}{-3}$

Table 4: Hyperparameter ranges used in experiments with GLUE and SuperGLUE
benchmarking datasets for RoBERTa (left) and DeBERTa (right) models. $p$ is
the prompt length used for P-Tuning v1/v2, and $r$ is the rank of weight
factorization used for AoT P-Tuning (See Section 3.3). For GLUE experiments,
each hyperparameter set was evaluated with different seed values. See Section
4 for more details. RoBERTa-Base
---
Model | STS-B | SST-2 | RTE | QQP |
Fine-Tuning | 90.6 ± 0.3 | 95.0 ± 0.2 | 81.2 ± 0.7 | 89.6 ± 0.2 |
Adapters | 90.7 ± 0.2 | 94.4 ± 0.3 | 80.5 ± 2.0 | 89.2 ± 0.1 |
LoRA | 90.1 ± 0.3 | 94.3 ± 0.5 | 80.5 ± 1.8 | 86.3 ± 0.3 |
BitFit | 90.3 ± 0.1 | 94.5 ± 0.5 | 80.9 ± 1.4 | 85.5 ± 0.6 |
P-Tuning v1 | 86.9 ± 0.9 | 94.0 ± 0.3 | 60.3 ± 2.4 | 82.2 ± 1.5 |
P-Tuning v2 | 89.2 ± 0.3 | 94.6 ± 0.2 | 80.5 ± 3.4 | 86.4 ± 3.3 |
Kron. AoT P-Tuning (ours) | 89.7 ± 0.2 | 94.0 ± 0.2 | 77.6 ± 1.4 | 88.2 ± 0.1 |
FC AoT P-Tuning (ours) | 90.0 ± 0.2 | 94.4 ± 0.3 | 78.0 ± 1.3 | 87.9 ± 0.2 |
| QNLI | MRPC | MNLI | CoLA | Macro
Fine-Tuning | 92.4 ± 0.1 | 90.8 ± 0.5 | 87.0 ± 0.3 | 63.8 ± 1.4 | 86.3
Adapters | 92.4 ± 0.2 | 91.1 ± 1.1 | 86.8 ± 0.1 | 63.0 ± 1.3 | 86.0
LoRA | 91.6 ± 0.3 | 90.6 ± 0.7 | 84.7 ± 0.3 | 60.3 ± 1.0 | 84.8
BitFit | 90.9 ± 0.5 | 90.5 ± 1.7 | 85.0 ± 0.1 | 60.4 ± 1.2 | 84.7
P-Tuning v1 | 88.3 ± 0.5 | 82.0 ± 1.7 | 80.8 ± 0.6 | 45.8 ± 27.1 | 77.5
P-Tuning v2 | 91.9 ± 1.6 | 89.1 ± 1.1 | 85.3 ± 0.2 | 60.7 ± 2.6 | 84.7
Kron. AoT P-Tuning (ours) | 90.7 ± 0.4 | 89.5 ± 1.1 | 84.6 ± 0.1 | 59.3 ± 1.2 | 84.2
FC AoT P-Tuning (ours) | 91.3 ± 0.4 | 90.3 ± 0.3 | 85.4 ± 0.1 | 60.3 ± 2.2 | 84.7
RoBERTa-Large
Model | STS-B | SST-2 | RTE | QQP |
Fine-Tuning | 91.9 ± 0.2 | 96.1 ± 0.4 | 88.1 ± 1.5 | 90.3 ± 0.2 |
Adapters | 92.1 ± 0.2 | 96.3 ± 0.4 | 90.0 ± 0.1 | 94.2 ± 0.1 |
LoRA | 91.4 ± 0.3 | 95.9 ± 0.2 | 87.2 ± 18.7 | 93.7 ± 23.7 |
BitFit | 91.8 ± 0.2 | 96.2 ± 0.4 | 87.7 ± 0.8 | 87.2 ± 0.6 |
P-Tuning v1 | 75.5 ± 6.3 | 94.4 ± 0.4 | 62.8 ± 2.3 | 76.9 ± 2.5 |
P-Tuning v2 | 91.0 ± 0.4 | 96.1 ± 0.3 | 87.4 ± 1.5 | 86.6 ± 0.6 |
Kron. AoT P-Tuning (ours) | 91.1 ± 0.8 | 96.2 ± 0.2 | 84.8 ± 1.3 | 89.4 ± 0.1 |
FC AoT P-Tuning (ours) | 91.7 ± 0.4 | 96.7 ± 0.1 | 88.4 ± 0.9 | 88.7 ± 0.2 |
| QNLI | MRPC | MNLI | CoLA | Macro
Fine-Tuning | 94.3 ± 0.2 | 91.6 ± 0.6 | 89.9 ± 0.2 | 68.1 ± 1.9 | 88.8
Adapters | 91.3 ± 0.4 | 90.1 ± 0.2 | 67.2 ± 1.3 | 87.7 ± 18.5 | 88.6
LoRA | 91.0 ± 7.2 | 88.9 ± 24.0 | 66.3 ± 1.9 | 87.4 ± 1.6 | 87.7
BitFit | 94.1 ± 0.4 | 91.0 ± 1.0 | 89.4 ± 0.1 | 69.8 ± 3.1 | 88.4
P-Tuning v1 | 79.1 ± 2.4 | 79.0 ± 1.1 | 75.9 ± 18.3 | 24.7 ± 17.6 | 71.0
P-Tuning v2 | 94.0 ± 1.1 | 91.2 ± 0.9 | 89.4 ± 0.7 | 66.9 ± 1.5 | 87.8
Kron. AoT P-Tuning (ours) | 94.2 ± 0.1 | 89.7 ± 0.9 | 89.3 ± 0.1 | 65.5 ± 1.9 | 87.5
FC AoT P-Tuning (ours) | 94.1 ± 0.2 | 91.6 ± 0.8 | 89.6 ± 0.1 | 69.2 ± 0.9 | 88.8
Table 5: Results on the GLUE Dev set. Each result is median and std across several seeds, and the Macro column is a mean score across all tasks. We bolded the best results and underlined the second best results. Fine-tuning is omitted from comparison with other methods and was not bolded for visibility. See Section 4 for details. |  RTE |  MNLI, QQP |  QNLI |  Other Tasks |  WiC |  CB, COPA, WSC |  MultiRC |  Other Tasks
---|---|---|---|---|---|---|---|---
Epochs | $200$ | $5$ | $10$ | $100$ | $500$ | $500$ | $10$ | $100$
Patience | $20$ | $2$ | $2$ | $10$ | $20$ | $100$ | $4$ | $10$
Table 6: The number of maximum epochs used for each GLUE and SuperGLUE Task.
Once the Dev score stopped increasing for "patience" steps, training was
halted. See Section 4 for more details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 4: Per-task GLUE Benchmarking Dataset results for a different number of
trained parameters of P-Tuning v2 and AoT P-Tuning with RoBERTa-Base (a-h) and
RoBERTa-Large (i-p). We also provide results of plain fine-tuning for
reference. See Section 4 for more details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 5: Expected Validation Performance of trained models with GLUE
Benchmarking Datasets for RoBERTa-Base (a-h) and RoBERTa-Large (i-p). See
Section 4.2 for more details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

Figure 6: Per-task SuperGLUE Benchmarking Dataset results for a different
number of trained parameters of P-Tuning v2 and AoT P-Tuning with RoBERTa-
Large (a-g) and RoBERTa-Large (h-n). We also provide results of plain fine-
tuning for reference. See Section 4.2 for more details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

Figure 7: Expected Validation Performance of trained models with SuperGLUE
Benchmarking Datasets for RoBERTa-Base (a-g) and RoBERTa-Large (h-n). See
Section 4.2 for more details.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8: Speed measurements for baseline methods with sequence length equal
to $384$ for different back-bone models. See Section 4.4 for details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

Figure 9: Speed measurement for baseline methods with sequence length $\in[16,64]$ for different backbone models. See Section 4.4 for details. $l$# | Tokens $x$ with largest norm $||{\bm{P}}_{x}||_{2}$
---|---
0 |  likes, a, is, loves, was, to, as, wants, s, ., pony, eded, himself, Man, were, and, I, has, I, are, Frank, ., hates, As, A, A, like, It, crop, Frank, After, ,, joins, As, Eric, Likes, It, just, would, onna, him, To, behaving, after, in, because, behaves, Is, We, Like
5 |  ,, ., narrower, doorway, backdoor, window, lousy, shortest, nicer, checkpoint, knob, thinner, narrowing, oub, quieter, BAD, ;, VID, rectangle, tighter, crappy, intruder, tongues, fing, rimination, blocker, and, raiding, detector, unmarked, sharper, knife, coolest, thicker, hoops, DOWN, lightsaber, asshole, millisec, KEY, sharp, token, slashing, Defenders, jug, Donna, slider, wedge, dding, kb
10 |  her, Her, herself, him, above, she, out, hers, him, HER, She, she, care, HER, above, Her, bold, CARE, cared, over, harder, louder, Above, smarter, sooner, her, cares, better, Out, vind, stronger, She, taller, tougher, Him, ahead, so, HIM, Susan, happier, up, Harry, aloud, higher, Above, SHE, could, apart, barking, inem
22 |  ., there, dry, for, There, Her, her, sword, the, arse, wy, dry, duc, The, it, took, cr, Rig, og, There, landing, the, wide, centrally, red, grass, sw, oa, above, engine, FT, spir, cd, Coun, Ross, there, ws, guy, starter, mans, aniel, green, freely, d, wide, stall, far, artz, THERE, didn
32 |  it, me, olit, Polit, Pat, him, Private, Susan, pat, he, her, Self, Ins, Doc, Coun, Ang, Aut, Sil, ochond, me, Nob, IT, Senator, Professional, Dri, itized, Je, Capt, Hillary, Whe, He, Kid, Registered, itious, Michelle, Political, It, Shut, Phot, BIT, Politics, Bit, Jacob, ruct, Young, HE, Tu, them, Mot, itu
37 |  him, they, it, they, her, them, their, his, he, it, its, hers, theirs, was, he, Susan, old, older, THEY, They, ITS, forth, Georg, Thom, Tom, erved, Carl, Anna, nob, anos, itans, to, Eric, itcher, Harry, Tim, Jen, them, Kid, Jeremy, JOHN, Jennifer, hands, Todd, put, Thomas, she, Dan, Michelle, s
46 |  chased, erved, house, houses, life, chasing, market, self, chase, raised, chester, hunt, castle, HOU, atics, Singer, western, ogenous, rounded, stretched, esian, essed, omorphic, horse, SER, central, ledge, hole, asio, Self, Self, iverse, oker, Judd, DF, aday, paced, ourced, erness, Barkley, scape, sey, ationally, owned, landers, ded, study, directed, OWN, produced
Table 7: Tokens with the largest $L_{2}$ norm of ${\bm{P}}$ entries for the WSC task. See Section 4.3 for more details. $l$# | Tokens $x$ with largest norm $||{\bm{P}}_{x}||_{2}$
---|---
0 |  fit, Loud, as, Air, Upon, Sets, Bound, Apart, scratched, sets, fit, Upon, hosted, Shot, Unt, Host, fitt, Sight, atri, Ocean, ceed, ashore, set, enture, underwent, planes, boats, Waves, Ali, shi, Active, Set, Atmosp, Airways, Host, chat, Endless, pelled, rew, ached, unct, fitted, Proud, flu, itable, anson, Bound, Assets, host, sets
5 |  set, Set, sets, Set, SET, Setting, setting, SET, set, padd, Setting, Sets, sets, bed, Cause, setting, the, cause, tread, itch, paddle, cause, thirsty, Khe, he, anned, this, ?, of, Cause, What, bidding, This, This, what, What, a, his, lic, The, wish, fugitive, they, Bed, Air, wake, conscience, ., crowd, Let
10 |  ?, ?, ?!, ??, ., ?", "?, )?, ???, ’?, set, .?, !?, ?), !, ????, …?, set, ?’, ,, Set, Set, ed, to, ??, ????????, as, ?????, ?)., setting, ?,, ?’", ?], sets, …, -, lt, —, :, lic, ???, led, ur, …, punching, of, t, ?"., sets, um
22 |  What, set, What, out, Set, sets, on, to, ’?, what, Set, in, WHAT, Sets, Setting, WHAT, from, Setting, dropped, of, Dig, Got, ?’, set, Exper, Gets, Ground, …?, happened, Whatever, Your, decom, Getting, Got, overlooked, Crack, )?, He, police, !?, happens, Suc, sets, what, Detective, GOT, Whatever, SET, Getting, Flying
32 |  glued, hid, melted, ., sent, breaths, etz, breath, Breath, baptized, watch, putting, tongues, braces, put, hid, bleach, icating, burying, aver, lifting, Illuminati, orneys, melting, withdrawing, numb, radios, inserts, amins, avert, breathing, puts, informants, lifting, hide, conscience, recommending, withdrawn, ransom, catch, Gael, Vern, roth, ears, Put, gins, breathed, attorneys, loss, biblical
37 |  hid, dropped, raided, fought, ungle, Hide, destruct, smuggled, abandoned, looted, attacked, barric, slid, dodged, drop, shut, drowned, hide, destruct, buggy, battled, shutdown, Hide, Attack, hid, rawl, inaccessible, avalanche, slipped, deleted, rawling, encrypted, withdrawn, Killed, dug, dropping, hoard, weapon, swallowed, defensive, destroy, exited, destroy, fight, Fighting, lost, deny, suppress, encrypt, aggressively
45 |  hopped, chats, pumped, paints, backed, spun, tread, coached, reefs, privately, noodles, buddies, malls, whisper, endorsements, squeezed, pals, blush, comed, edits, rallies, gigs, recol, mocked, curs, Bare, bubbles, warmed, chat, profiles, emails, Dreams, pads, chalk, interviewed, sneakers, rocked, Gloves, hubs, docs, shaved, Rise, primaries, listened, shy, essays, whispers, leeve, girlfriends, socks
Table 8: Tokens with the largest $L_{2}$ norm of ${\bm{P}}$ entries for the COPA task. See Section 4.3 for more details. $l$# | Tokens $x$ with largest norm $||{\bm{P}}_{x}||_{2}$
---|---
0 |  gression, rium, History, orer, aic, history, oration, ré, orative, amic, history, version, ural, osa, avage, ory, lia, range, History, rica, nation, root, USE, á, ination, ulation, mentation, issance, state, rum, adal, idden, jection, oly, ó, esis, orean, discovery, ria, ada, uration, entry, ord, verse, inations, ugal, itus, olics, ESSION, ativity
5 |  ., to, in, for, and, of, ,, s, the, be, ’s, by, on, or, from, at, or, with, :, ly, a, an, ;, on, -, ., in, under, an, as, I, and, !, about, er, In, but, ?, A, is, ed, a, that, o, ers, S, ing, now, ), -
10 |  ., of, and, for, morph, votes, elector, with, uild, igraph, tatt, Assignment, as, contribut, advant, are, hod, Voters, matically, Init, rede, olon, on, rehabilit, neum, mog, looted, req, by, Claim, the, ynchron, dule, promot, socio, portfolios, goto, vulner, vote, setup, nominate, anism, s, subscrib, iop, lihood, slot, elist, ramid, ysc
22 |  in, In, in, be, In, .", being, Straw, -, its, a, Majority, of, a, Latest, the, Jack, ine, latest, it, Lawyers, Watts, "., "-, Massachusetts, their, .’, been, ure, Till, ’.", Signs, .’", Seventh, ?", Taxes, Atlanta, !", electric, at, IN, ide, Current, Ladies, KP, Jersey, Students, Knights, it, Anders
32 |  Se, Hum, Brazil, Mur, Hur, aver, Hum, Yugoslavia, Mour, jud, a, Hawai, Pag, Kant, ibal, Malaysia, EFF, Hur, .", adj, mur, Islam, and, Guinea, Britain, Sadd, Def, Niger, ,, Holland, amus, Hay, Ma, Appro, Mur, Countries, Wid, Asians, Nor, else, Calendar, Hed, Ved, ldom, english, Hind, mur, bury, Ded, hol
37 |  [SEP], +., Sk, Ble, Gre, cloud, Else, ., +,, "., uran, cs, Ever, 2048, Ble, Keefe, Hyp, athan, Lib, Fra, Exp, bro, Edit, Ros, Bean, Bo, Beck, Shell, sit, !., Saud, Phys, -, shell, Ol, BLIC, -, Over, ea, orthy, Shot, pn, pas, ester, Reviewed, Spe, sell, 2024
45 |  Chance, Sw, chance, Nine, Shares, Chance, Scientists, Tw, Besides, Prof, chances, Sn, sw, TW, EFF, J, IJ, Besides, chance, Between, icist, GU, SW, pan, Ja, Psy, tw, Between, xon, Bj, Conj, Shares, Moh, UTH, Prediction, science, intend, Science, iov, Nine, jp, dds, NJ, Jr, y, Nin, etsy, Ibid, ymm, Reporting
Table 9: Tokens with the largest $L_{2}$ norm of ${\bm{P}}$ entries for the RTE task. See Section 4.3 for more details. $l$# | Tokens $x$ with largest norm $||{\bm{P}}_{x}||_{2}$
---|---
0 |  .”, didn, doesn, don, ”., ,”, Didn, Doesn, didn, doesn, Don, wouldn, Wouldn, Does, ”, couldn, DON, Did, hadn, But, Don, Isn, ).", DON, shouldn, “, Obviously, Obviously, Isn, don, hasn, ))., Does, "?, ].", wasn, Did, ],", ,., Naturally, …", ),", Would, “, But, ”;, Naturally, ]., ,), DOES
5 |  ,, ’t, !,, .,, ?,, didn, , not, to, +,, *,, ),, the, ., ],, considered, doesn, in, , ,), a, /,, ,[, you, don, ,,, ,., shouldn, (),, hasn, ;, for, thought, weren, hadn, thought, wasn, NOT, hair, ’, .;, aren, ‘,, Said, ,, couldn, isn, .—, idered
10 |  Shant, Georg, Expect, Led, Assistant, Amph, Registered, Ear, McA, THEIR, Prev, Emb, -,, Called, Gw, Alc, Until, Rhod, Introduced, that, Lat, Unt, Ul, Sv, Gh, to, of, Fernand, ,, elta, jac, unch, Ov, Sebast, apologised, JOHN, !"., Ll, hid, Somewhere, Been, Recently, and, Somebody, Fram, Coh, ’)., Sty, Elsewhere, Unt
22 |  ’s, ’re, A, ’ve, the, a, her, ’, be, A, a, have, DOES, LIKE, ?", "?, ’d, )?, ?, s, ABOUT, ", Like, Pant, didnt, ’m, ’?, E, The, doesnt, Was, re, :, ie, Surely, ’ll, Corinth, At, Across, your, their, ?,, THEY, …?, or, Fra, HOW, )/
32 |  I, ’?, he, "?, ?’, )?, ’t, ”., .?, …?, .”, ?"., ’:, He, !?, ?,, He, I, and, ?’", ?!", ?", ?)., !’, he, .:, ?!, she, +., )!, ’., .’, !?", ,”, ’)., ???, !., ).", we, CLOSE, ‘., "!, .], .–, ????, ’/, ’re, .’"
37 |  ., ’s, ’t, ?, ’, :, I, -, ’, ,, of, he, B, B, in, I, and, ’m, -, s, ", ’d, by, for, ;, b, on, you, !, ", He, to, /, ’ve, y, ’re, ed, with, ., ’ll, a, back, the, b, she, He, E, C
45 |  ’t, not, NOT, the, not, Not, never, Not, ’s, ’re, NOT, ’ve, you, of, [SEP], t, nt, Never, The, in, NEVER, he, to, the, [CLS], hardly, never, neither, I, ,, ’m, cannot, no, The, annot, it, Their, me, didnt, He, and, doesnt, Ear, a, ., Never, none, if, on, nobody
Table 10: Tokens with the largest $L_{2}$ norm of ${\bm{P}}$ entries for the
CB task. See Section 4.3 for more details.