# Image-to-image Mapping with Many Domains by Sparse Attribute Transfer

Matthew Amodio
Yale University
&Rim Assouel
Université de Montréal, Mila
&Victor Schmidt
Université de Montréal, Mila
&Tristan Sylvain
Université de Montréal, Mila
&Smita Krishnaswamy
Yale University
&Yoshua Bengio11footnotemark: 1
Université de Montréal, Mila
Equal contribution.

###### Abstract

Unsupervised image-to-image translation consists of learning a pair of
mappings between two domains without known pairwise correspondences between
points. The current convention is to approach this task with cycle-consistent
GANs: using a discriminator to encourage the generator to change the image to
match the target domain, while training the generator to be inverted with
another mapping. While ending up with paired inverse functions may be a good
end result, enforcing this restriction at all times during training can be a
hindrance to effective modeling. We propose an alternate approach that
directly restricts the generator to performing a simple sparse transformation
in a latent layer, motivated by recent work from cognitive neuroscience
suggesting an architectural prior on representations corresponding to
consciousness. Our biologically motivated approach leads to representations
more amenable to transformation by disentangling high-level abstract concepts
in the latent space. We demonstrate that image-to-image domain translation
with many different domains can be learned more effectively with our
architecturally constrained, simple transformation than with previous
unconstrained architectures that rely on a cycle-consistency loss.

##  1 Introduction

In the task of unsupervised image-to-image translation, we seek a mapping
between two distributions of images where we do not have a known pairwise
correspondence between points. Implicit in the task is the assumption that the
variation in a system with two distributions can be broken down into: 1.
variation separating the source and target domains and 2. variation intrinsic
to just the source or just the target domain. The goal in image-to-image
translation is to remove variation along the first axis while preserving all
of the other variation along the axes in the second category.

Since their advent, the field has been dominated by generative adversarial
networks (GANs) based on the cycle-consistency assumption [16, 1, 6, 10, 15].
With the cycle-consistency assumption, two networks are trained simultaneously
(going in both directions between the two domains) and they are enforced to be
inverses of each other. With a cycle-consistent GAN, the removal of the
variation separating the source and target domains (from 1. above) is
accomplished by the discriminator distinguishing between the target domain
images and the generated images. To avoid changing other sources of variation
(from 2. above), the cycle-consistency loss is employed. While invertibility
may be a desirable property in some mappings, it prohibits other mappings from
being learned (e.g. many-to-one relationships cannot be inverted losslessly).
Moreover, empirical evidence shows that training under cycle-consistency
produces unwanted restrictions, like generating a set of images with highly
correlated pairwise distances to the original set of images, as measured in
pixel space [4, 2]. These limitations shed light on the seemingly arbitrary
difference between domains where cycle-consistent models succeed impressively
versus those where they utterly fail [2].

Figure 1: A conceptual figure of the consciousness prior in the context of
image-to-image translation.

The problems of cycle-consistent training are especially exacerbated when
mapping between many domains, for example using one model to map between
domains represented by all attributes in the canonical CelebA dataset [11]. In
the context of many domains, one generator network performs all of the
mappings, distinguished by conditional labels. With one generator network
having to be its own inverse in pixel space between many different domains,
with just conditional labels to distinguish them, it comes as no surprise that
existing cycle-consistency models struggle.

We offer an alternative approach: rather than a completely unrestricted
architecture coupled with a cycle-consistency constraint, we directly restrict
the generator architecture.

Our approach, which is called semantic attribute transfer, is built on the
previously proposed notion of the consciousness prior [5]. Based off of ideas
from cognitive neuroscience, the consciousness prior hypothesizes that high-
level abstract concepts are stored sparsely in deep representations [3, 13,
7]. Rather than performing transformations directly in pixel space, this prior
supposes transformations between high-level traits can be done in a latent
layer with relatively simple functions performed on sparse coordinates chosen
with an attention mechanism.

Semantic attribute transfer bridges the gap between the consciousness prior,
previously detailed in the context of natural language processing, and image-
to-image translation. In the consciousness prior work, consciousness is
defined as a process involving the restriction of the attention of a network
to changing one feature at a time. While they view this as consciousness over
time, we modify this as consciousness in abstract representation feature
space.

We restrict the translation to be a consciousness process in the following
way: we eschew the cycle-consistency loss term, but instead of using an
entirely unconstrained generator architecture, we only allow it to perform a
single-feature transformation. We train the generator to autoencode, i.e. when
given an image as input, it produces the same image as output. Jointly, we
allow an attention mechanism to select a single neuron in the middle layer of
the generator, perform a simple parametric change to its value, and finish the
feedforward to produce a generated image (Figure 1).

This construction places a prior on the representation of high-level concepts
in the latent layer of the generator, corresponding to each domain being
stored as independent factors at this level. As proposed in the previous work
on the consciousness prior, this allows each transformation to constitute a
single “conscious thought”. With this, an analogy can be made to how a human
would focus consciousness on one main difference between the two
distributions, and ignore all other minor differences.

Such a strictly constrained transformation function ensures only the dominant
axis of variation between domains is changed, as it can only change the amount
of information that can be stored in a single neuron (while also being able to
autoencode from the representation). Furthermore, this obviates the need for
heuristic losses frequently used in image-to-image translation like the cycle-
consistency loss or the loss that enforces a network not to change input that
is already in the target domain [16].

The main experimental findings are as follows:

  1. 1.

Our semantic attribute transfer model improves performance on image-to-image
translation tasks with many different domains trained simultaneously, using
traits from CelebA and CUB [11, 14].

  2. 2.

The strength of our representation can be improved by adding more domains to
the mapping problem due to increasingly disentangled representations. This
makes our model more scalable to increasing numbers of domains than previous
work.

  3. 3.

Our results indicate that requiring cycle-consistency in training is an overly
restrictive heuristic: without it our model learns inverse functions when
domains are inherently one-to-one and not when otherwise.

##  2 Related Work

#### Consciousness Prior

The consciousness prior hypothesizes a biologically motivated representation
constraint on the manipulation of high-level, abstract characteristics in the
data [5]. In previous work, it was explained in terms of consciousness over
time in a recurrent natural language processing model. This framed the memory
of a model at time $t$ in terms of its memory at time $t-1$, using a
consciousness process $C$:

$\displaystyle c_{t}=C(c_{t-1},a_{t},f_{t})$

By restricting $a_{t}$ to be sparse and $f_{t}$ to be simple, we can simulate
the model focusing on only one feature at each step.

#### StarGAN

The StarGAN extended image-to-image translation work by considering many
domains [6]. To avoid the combinatorial growth of having to train distinct
models for every pair of domains, they use one generator network and one
discriminator network that are conditioned upon the target domain. Then, using
the original domain label, a cycle-consistency loss is imposed upon both
directions. Otherwise, their architectures are the completely unconstrained
ones commonly used in other image-to-image translation models.

##  3 Model

Figure 2: The generator network architecture, simultaneously trained to
produce the identity (blue path) and a transformed image (red path), with only
a single neuron change in the middle layer different between the two.

Here we consider the multi-domain image-to-image translation problem. Let
$\\{x_{ij}\\}_{j=1}^{n}\in X_{i}$ be a finite sample from domain
$i=1...n_{domain}$. We seek mappings $G_{i}:X_{i}^{c}\rightarrow X_{i}$ that
take points outside of domain $i$ (in the complement) and map them into domain
$i$. This terminology emphasizes that we want a separate mapping for each
domain $i$ that maps into domain $i$ from anywhere else in the space. We note
that the domains may not form a partition as they may overlap. We employ the
generative adversarial network formulation that trains these mappings by
pitting a generator against a discriminator in the standard alternating
optimization process:

$\displaystyle\underset{G}{min}\>\underset{D}{max}\>L_{GAN}=\mathbb{E}_{x_{i}\sim
p_{X_{i}}}[logD(x_{i}|i)]+\mathbb{E}_{x_{i}^{c}\sim
p_{X_{i}^{c}}}[log(1-D(G(x_{i}^{c}|i)|i)]$

Each mapping $G_{i}$, which maps into domain $i$, shares the same network
weights. The target domain to map into is selected via a conditioning label
$i$. We choose this rather than using separate networks for each generator
$G_{i}$ and each discriminator $D_{i}$ to facilitate scaling to large numbers
of domains.

In our image-to-image domain translation context, we modify the previous
definition of a consciousness process $C$:

$\displaystyle c_{i}=C(c_{j},a_{i},f_{i})$

Instead of operating on a time axis, it is now a function that operates on a
low-level representation of an image in domain $j$, and parameterized by a
sparse attention mechanism $a_{i}$ and a simple parameterized edit $f_{i}$,
produces a low-level representation of an image in domain $i$.

To create a generator that uses such a consciousness process, we create a
novel architecture for the generator function $G(x|i)$. The conditioning label
is provided to the generator by concatenating a one-hot vector to the input
channel-wise. We decompose our generator into a downsampling encoder $E$ and
an upsampling decoder $H$. We run our generator in two distinct modes
(optimized jointly), and in the first it is simply the composition of the
encoder and the decoder. In this part of the total loss we train it as an
autoencoder with the usual reconstruction loss, which will result in an
approximate identity function.

$\displaystyle L_{recon}$ $\displaystyle=||x-G(x|i)||^{2}$

Simultaneously, we run the generator in another form designed to perform the
transformation. To do so, we place a transform $t(v)$ in between the encoder
and decoder:

$\displaystyle G_{t}(x|i)=H(t(E(x)))$

Crucially, by enforcing $G$ to be the identity and restricting $t$ properly,
we ensure that the generating function $G_{t}$ conforms to the notion of the
consciousness prior, by only allowing the “consciousness” of the generator to
focus on a single element, selected via a sparse attention mechanism, and
change it with a simple edit.

The transform $t(v)$ consists of two aspects: an attention mechanism which
selects a feature dimension to edit, and a simple quadratic edit. We chose the
simplest transformation that had sufficient expressiveness to produce domain
adaptation results (an affine transformation was insufficiently expressive).
Let $v\in\mathbb{R}^{d}$ be a feature vector representation in the internal
layer of the generator. A domain-specific attention vector $\tilde{a_{i}}$
selects one of the dimensions to edit, and domain-specific parameters
$s_{1i},s_{2i},b_{i}$ are used in the simple quadratic edit:

$\displaystyle a_{i}=softmax(\tilde{a_{i}})$ $\displaystyle
t(v)=a_{i}\cdot(s_{2i}\cdot v^{2}+s_{1i}\cdot v+b)+(1-a_{i})\cdot v$

where $\cdot$ denotes element-wise multiplication. To ensure that the
attention mechanism is sparse, we add an entropy penalty to the loss:

$\displaystyle L_{entropy}=-\underset{ij}{\Sigma}\left(a_{ij}\cdot
log(a_{ij})+(1-a_{ij})\cdot log(1-a_{ij})\right)$

When adapting the consciousness prior concept from natural language to images,
we are faced with the additional complication of points whose feature depth is
distributed across spatial locations. Thus, here we observe that while the
transform $t(x)$ performs the domain translation in a sparse way with respect
to the feature depth, it does not have spatial awareness. To address this, we
introduce a skip connection network to act as a mask and select the spatial
coordinates for the transform to apply to. This skip connection network has a
one-channel sigmoid output mask $M$ and is used in the final generator model
in the following way:

$\displaystyle G(x|i)=M\cdot G(x|i)+(1-M)\cdot x$

The total objectives for the generator and the discriminator, respectively,
are thus:

$\displaystyle L_{G}$ $\displaystyle=L_{GAN}+\lambda_{recon}\cdot
L_{recon}+\lambda_{entropy}\cdot L_{entropy}$ $\displaystyle L_{D}$
$\displaystyle=L_{GAN}$

with coefficients $\lambda_{recon},\lambda_{entropy}$ controlling how much
each term weighs relatively in the total.

##  4 Experiments

There are many challenges to building models in the multi-domain image-to-
image context. Most significantly, many different mappings must be learned at
once. In a typical image-to-image task, only two domains are considered, and
there is one generator entirely dedicated to going in each direction. That
means each generator’s weights only need to be directed towards one set of
pixel combinations (e.g. learning weights that can only produce red coloring,
thus never erroneously producing yellow coloring, because the network just
tunnels all output towards the red part of the space). In this context,
though, many different sets of pixel combinations all need to be generated at
different times. Beyond just the challenges in generation, the adversarial
discriminator also faces a more difficult task in having to distinguish
between many different domains, as opposed to just one.

Our architecture, based on the U-Net framework, consists of three stride-two
downsampling layers with Leaky ReLU activations and three upsampling layers
with regular ReLU activations, and skip connections between downsampling
layers and upsampling layers of the same dimensionality [12]. We use the Adam
optimizer to train with a learning rate of $0.0002$ and momentum parameters
$\beta_{1}=0.5,\beta_{2}=0.9$ [9]. Resolution $128\times 128$ is used for
inputting the images. The loss function is the standard sigmoid-based GAN
loss. The discriminator’s architecture is identical to the downsampling half
of the generator except with a flattening, conditional projection, and
classification layer added on top of the last layer.

We use several baseline models as comparisons. The most relevant method we
compare to for multi-domain image-to-image translation is the StarGAN [6]. The
StarGAN relies on a cycle-consistency loss, but is architecturally equivalent
to ours, allowing us to test this form of loss against our approach which
follows the consciousness prior. We compare to it with varying choices for
their hyperparameters, the coefficient of the cycle-consistency and the
coefficient of the identity loss, with StarGAN1 being 10 and 5, StarGAN2 being
10 and 0, and StarGAN3 being 1 and 0, respectively. We then additionally
compare to two other multi-modal cycle-consistency models, namely the MUNIT
and NoiseAugmented models [8, 1]. We note that these models are designed for
producing multiple outputs within one target domain, but do not natively
handle multiple domains like we have in our context. We thus use the same
approach as in the StarGAN of giving the model the target label to learn a
conditional output distribution.

###  4.1 CUB

In our first experiment, we perform image-to-image translation using primary
color of birds in the CUB dataset [14]. We use each primary color with more
than 100 image (eight total domains).

Figure 3: Samples from our model on the CelebA and CUB datasets. Individual
examples are held constant as the target distribution used for the
conditioning label is changed.

Samples from our model with changing domains are shown in Figure 3. We
emphasize that each image is produced by changing just one neuron in the
middle layer away from the identity function. Through this, we see that the
concept of primary color has been stored sparsely and simply, conforming to
the hypothesis of the consciousness prior.

To quantify the performance, we utilize the traditional measure of generative
distributional accuracy, the FID score. In image-to-image translation, it is
insufficient to measure image quality, as a real image is provided to the
generator as input. Instead, a distributional score is necessary to measure
how well the input images match the images in the target domain. For this
reason, FID score, which measures distributional distance to the target
distribution, is the most appropriate metric. In Table 1, we report the
average FID score across all domains from our model and the baselines. We
found our model performed significantly better than all baselines. In Figure
5, we plot the FID score per domain. Our model has a consistently low FID in
each domain, with only its worst domain FID being higher than any other
models’ best domain FID.

###  4.2 CelebA

In the previous experiment, there were only eight domains and they were
characterized by color changes only. From here we move on to a harder test of
more domains that are characterized by more complicated transformations in the
CelebA dataset. Because the attributes in the dataset are not necessarily
mutually exclusive (e.g. an image can both be “black-hair” and “glasses”), we
expand the number of attributes $n_{a}$ into $2\cdot n_{a}$ domains, where
each attribute has a domain corresponding to “has the attribute” and a domain
corresponding to “does not have the attribute”.

In Figure 3, we see samples from our model with changing target attributes,
again only different from the identity function by a single simple change to
one neuron in the middle layer. For example, by being able to produce two
images, one without glasses and with glasses, that only differ in their latent
representation by one neuron, this provides strong empirical evidence that the
abstract concept of “glasses” has indeed been stored sparsely and simply. As
before, in Figure 5 we show the FID score per domain for each model, and in
Table 2 we report the mean FID across all domains per model. In this case as
in the previous experiment, we see our model significantly outperform the
cycle-consistent models across all domains.

We next perform a series of experiments on this dataset to more thoroughly
examine our model to better understand how it outperforms the best alternative
StarGAN model.

#### Increasing number of domains effect

We first evaluate the effect of increasing the number of domains that must be
mapped between, by designing an experiment that isolates this effect, which we
call the “crowding effect”. As an example of the crowding effect, a model
might be able to generate high quality “glasses” images if that is the only
output domain, because it can simply learn to ignore any part of the image
that is far from the eyes. However, if a model must both generate different
hair colors given one condition label and glasses given another condition
label, the performance on generating glasses may suffer.

To quantify this effect, we train a model on just $Domain_{i}$, and measure
its FID. Then, we train a new model on both $Domain_{i}$ and $Domain_{j}$, and
compare the FID on $Domain_{i}$ from the second model to its FID in the first
model. We keep doing this, adding more domains in powers of two. This gives a
series of FID scores for each domain corresponding to increasing numbers of
total domains in the model. We build a simple linear regression on these
scores and look at the fitted slope of the regression line: an estimate of the
change in FID for a given domain from adding another domain to the total
mapping model.

In other words, these scores represent the change in FID for a given
$Domain_{i}$ from making a $n$-domain mapping problem a $2n$-domain mapping
problem. We report the average of these scores for each model in Table 4.2.
While the cycle-consistency model has an FID that worsens as the number of
domains increases, our model actually improves its FID as the number of
domains increases.

#### Latent space disentanglement

We hypothesize one reason for our model’s better generative performance is
that the representation it learns and operates on better disentangles the
attribute it is transforming. To investigate this, we look at the domains
corresponding to “having” and “not having” each attribute, and calculate the
maximal mean discrepancy (MMD) as a distributional distance between them in
the deepest latent layer of each model. The distances are reported in Table 4,
with the mean and standard deviation across all domains. We see that our model
does indeed produce significantly more separation between points with an
attribute and points without that attribute, providing evidence that our model
has more effectively disentangled this variation from other underlying
variation in the data.

Figure 4: FID scores on the CUB dataset, per domain. The domains are ranked by
FID per model.

Figure 5: FID scores on the CelebA dataset, per domain. The domains are ranked
by FID per model.

| Mean FID
---|---
Ours | 50.710 +/- 13.316
MUNIT | 79.694 +/- 25.865
NoiseAugmented | 86.381 +/- 27.798
StarGAN1 | 84.580 +/- 7.166
StarGAN2 | 76.433 +/- 9.181
StarGAN3 | 107.725 +/- 3.626

Table 1: FID scores on the CUB dataset.

| Mean FID
---|---
Ours | 37.588 +/- 13.347
MUNIT | 58.821 +/- 18.132
NoiseAugmented | 66.358 +/- 15.998
StarGAN1 | 52.927 +/- 17.592
StarGAN2 | 51.680 +/- 17.448
StarGAN3 | 51.937 +/- 18.914

Table 2: FID scores on the CelebA dataset.

| Change in FID
---|---
Our Model | -1.622 +/- 1.023
StarGAN | 2.404 +/- 1.212
Table 3: Crowding effects: the change in FID for $Domain_{i}$ if an $n$-domain
mapping problem becomes a $2n$-domain mapping problem. For our model the FID
actually improves.

| MMD
---|---
Our Model | 0.440 +/- 0.074
StarGAN | 0.118 +/- 0.081

Table 4: MMD between the distribution of points with an attribute and the
distribution without the attribute, in the deepest latent layer for each
model. Our model better disentangles the attribute and separates them.

| Correlation
---|---
Our Model | 0.847 +/- 0.163
StarGAN | 0.976 +/- 0.009
Table 5: Pairwise distance correlations between learned transformations on all
possible domains on the CelebA dataset. Some transformations require not-near-
perfect correlations.

| Mean FID
---|---
Full Model | 37.588 +/- 13.347
No recon loss | 39.429 +/- 14.484
No entropy loss | 41.445 +/- 14.885

Table 6: The ablation study results with the mean FID across all domains. The
full model with all loss terms performs better than models without all of
them.

#### Pairwise distance correlation

Previous work has shown that cycle-consistent mappings are constrained to
learning transformations that preserve pairwise distances between points in
ambient pixel space [4, 2]. Here we perform an experiment to investigate
whether this extended to the multi-domain setting. As this restriction may not
always be desirable, we evaluate whether it holds in our model without a
cycle-consistency loss. We take a fixed set of images, and then transform them
to each target domain in turn. For each pair of domains, we calculate the
correlation between the pairwise distances of points in the first domain with
their representations in the other domain. Table 4.2 shows the StarGAN
produces high correlations in every domain (the minimum between any pair was
0.968). In our model, some domains had high correlations, while others did not
(the minimum was 0.484).

#### Cycle-consistency in our model

While we do not enforce cycle-consistency in our model, we can evaluate
whether our fully-trained model has the property. We map to and from each
domain and report the average mean-squared error (MSE) between the original
image and composite output after the cycle. To compare, we note that the
cycle-consistent baseline had an average MSE of $0.024$, but the standard
deviation was just $0.003$: this value is almost identical for every pair of
domains. This makes sense, as the model is explicitly trying to minimize this
term. In our model, the average MSE was $0.010$, and the standard deviation
was drastically higher: $0.027$. For some domains, the reconstruction error
was almost zero, while for other domains the reconstruction error was much
higher than the baseline’s highest. Not all desirable transformations are one-
to-one and invertible: our model produces cycle-consistent mappings in some
cases, and not in others.

Figure 6 shows an example of a mapping that is cycle-consistent in domains
where it is natural (black hair to brown hair and back to black hair) and an
example of a mapping that is not cycle-consistent but still reasonable. An
image with facial hair is mapped to the “no facial hair” domain and then re-
mapped to the “facial hair” domain. The resulting image doesn’t have the same
facial hair as the original (it has a beard instead of the original soul
patch), but is a perfectly reasonable mapping. For the StarGAN, it handles the
first transformation but for the second one it is prevented from changing the
input at all by the restriction of cycling back to exactly the image it
started with.

Figure 6: Some domains are inherently one-to-one and our model is cycle-
consistent (despite training not with it). Other domains are many-to-one and
our model is not forced to learn arbitrary one-to-one mappings. The StarGAN is
prevented from changing the input at all in the latter case by the cycle-
consistency restraint.

#### Ablation study

Lastly, we perform an ablation study to measure the effect of the different
loss terms in our model: the reconstruction loss which forces the generator to
produce the identity without the semantic attribute transform and the entropy
loss which forces the attention mechanism in the transform to be sparse. In
Table 6, we report the mean across all domains. Each loss term contributes to
the ultimate performance, as the full model achieves the best score.

##  5 Discussion

Here we presented a novel framework that conforms with the notions from the
recent theoretical advances in cognitive neuroscience, motivating future work
in biologically motivated architectures.

##  6 Broader Impact

Our proposed method has impact on the research community, but we believe it is
safe from concerns about negative ethical or societal consequences. As an
architectural advance in image-to-image translation, our model does not
implicate any negative consequences moreso than any other image-to-image
translation model like the CycleGAN or the StarGAN. Moreover, by using pre-
determined attributes in canonical datasets, we do not isolate any attribute
that is not used ubiquitously in machine learning research. Many useful
applications of this kind of work exist, like mapping between healthy and
diseased medical imaging results.

One potential positive impact of our work is the furthering of progress
towards biologically plausible and biologically motivated architectures. We
believe the previous work of the consciousness prior, along with ours here,
can offer the opportunity for the deep learning community to become more
involved with the cognitive neuroscience community. Such a grounding in a
biological science would stand to benefit the broader community in our
opinion.

## References

  * [1] A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. Courville.  Augmented cyclegan: Learning many-to-many mappings from unpaired data.  arXiv preprint arXiv:1802.10151, 2018.
  * [2] M. Amodio and S. Krishnaswamy.  Travelgan: Image-to-image translation by transformation vector learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8983–8992, 2019.
  * [3] B. J. Baars.  A cognitive theory of consciousness.  Cambridge University Press, 1993.
  * [4] S. Benaim and L. Wolf.  One-sided unsupervised domain mapping.  In Advances in neural information processing systems, pages 752–762, 2017.
  * [5] Y. Bengio.  The consciousness prior.  arXiv preprint arXiv:1709.08568, 2017.
  * [6] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789–8797, 2018.
  * [7] S. Dehaene, H. Lau, and S. Kouider.  What is consciousness, and could machines have it?  Science, 358(6362):486–492, 2017.
  * [8] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 172–189, 2018.
  * [9] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [10] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang.  Diverse image-to-image translation via disentangled representations.  In Proceedings of the European conference on computer vision (ECCV), pages 35–51, 2018.
  * [11] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [12] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [13] R. Van Gulick and E. N. Zalta.  The stanford encyclopedia of philosophy, 2009.
  * [14] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [15] Z. Yi, H. Zhang, P. Tan, and M. Gong.  Dualgan: Unsupervised dual learning for image-to-image translation.  In Proceedings of the IEEE international conference on computer vision, pages 2849–2857, 2017.
  * [16] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.


# MerA: Merging Pretrained Adapters For Few-Shot Learning

Shwai He1 Run-Ze Fan3 Liang Ding2 Li Shen4 Tianyi Zhou1 Dacheng Tao2
1University of Maryland, College Park 2The University of Sydney
3University of Chinese Academy of Sciences 4JD Explore Academy
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Adapter tuning, which updates only a few parameters, has become a mainstream
method for fine-tuning pretrained language models to downstream tasks.
However, it often yields subpar results in few-shot learning. AdapterFusion,
which assembles pretrained adapters using composition layers tailored to
specific tasks, is a possible solution but significantly increases trainable
parameters and deployment costs. Despite this, our preliminary study reveals
that even single adapters can outperform Adapterfusion in few-shot learning,
urging us to propose Merging Pretrained Adapters (MerA) that efficiently
incorporates pretrained adapters to a single model through model fusion.
Extensive experiments on two PLMs demonstrate that MerA achieves substantial
improvements compared to both single adapters and AdapterFusion. To further
enhance the capacity of MerA, we also introduce a simple yet effective
technique, referred to as the "same-track" setting, that merges adapters from
the same track of pretraining tasks. With the implementation of the "same-
track" setting, we observe even more impressive gains, surpassing the
performance of both full fine-tuning and adapter tuning by a substantial
margin, e.g., 3.5% in MRPC and 5.0% in MNLI.

##  1 Introduction

Pretrained language models (PLMs) Devlin et al. (2019); Liu et al. (2019) have
revolutionized the field of natural language processing, with fine-tuning
being a mainstream approach to leverage the power of PLMs. However, with the
ever-increasing number of parameters in PLMs Brown et al. (2020), there is a
need for parameter-efficient fine-tuning techniques He et al. (2022a) to
reduce training costs. One representative technique is adapters tuning Houlsby
et al. (2019), which updates only a subset of parameters. Despite their
advantages, adapter tuning often starts with randomly initialized blocks and
may not perform well in scenarios with limited training data, such as few-shot
learning Moosavi et al. (2022); Bansal et al. (2022).

Figure 1: Comparison between a single adapter and MerA in sentiment analysis
tasks. Left: The dilemma of a single adapter in few-shot learning. Right: the
improvement of merging pretrained adapters.

One possible solution is to transfer knowledge from pretrained adapters to
target tasks Chawla et al. (2021); Zhong et al. (2022); Wang et al. (2022).
AdapterFusion Pfeiffer et al. (2021) has been proposed to assemble pretrained
adapters with composition layers to integrate knowledge. However,
AdapterFusion compromises parameter efficiency He et al. (2022b), primarily
due to the excessive number of trainable parameters in the composition layers.
On the other hand, deploying parallel pretrained adapters also increases
computational costs.

In this work, we turn to explore an efficient approach to leverage pretrained
adapters, raising the following question: Can the current AdapterFusion
framework fully exploit the pretrained adapters under few-shot scenarios? If
not, how to incorporate the pretrained adapters more efficiently? To this end,
we first conduct a series of experiments to compare the performance of single
adapters and that of AdapterFusion under few-shot scenarios, with results
shown in Figure 2. Surprisingly, a single adapter outperforms AdapterFusion
with much few trainable parameters. Such preliminary study prompts us to
directly leverage pretrained adapters to extend the potential of single
adapters.

To achieve this, we propose an approach that merges pretrained adapters into a
single one (MerA). On the one hand, the merge pretrained adapter does not
introduce additional trainable parameters. On the other hand, the knowledge
from pretrained adapters enhances downstream performance, as illustrated in
Figure 1. We first implement two straightforward methods for merging
parameters, including summation (“Sum.”) and averaging (“Avg.”), whereas the
lack of one-to-one correspondences between the parameters of different models
leads to suboptimal performance. Inspired by Solomon et al. (2015); Singh and
Jaggi (2020), we further propose to align adapters’ parameters through optimal
transport based on weights (“Wts.”) and activations (“Acts.”).

Extensive few-shot experiments demonstrate that MerA achieves significant
improvements compared with Adapters, e.g., 2.7% in averaged accuracy. In
addition, we also find that merging adapters from the same track of tasks
further enhances the capacity of MerA. Therefore, we introduce a simple yet
effective technique called the "same-track" setting. With the implementation
of the "same-track" setting, we observe even more impressive gains, surpassing
the performance of full fine-tuning and Adapter tuning by a substantial
margin, e.g., 3.5% in MRPC and 5.0% in MNLI.

##  2 Methodology

AdapterFusion.  Adapters Houlsby et al. (2019) are bottleneck modules plugged
in PLMs, with model dimensions $d$ and reduction factor $r$. In standard
Adapter Tuning, only adapter layers are trainable, while other layers are
frozen. After tuning, adapters contain specific knowledge for a single task.
AdapterFusion Pfeiffer et al. (2021) is proposed to leverage knowledge from
pretrained adapters, which improves the performance in downstream tasks and
prevents catastrophic forgetting for each adapter.

Motivation.  However, the resource constraint scenario challenges the
parameter efficiency of AdapterFusion. For one thing, AdapterFusion requires
composition layers that include additional trainable parameters, e.g., Query,
Key, and Value. The trainable parameters of a composition layer are $3d^{2}$
(ignoring bias terms for the time being), while a single adapter only contains
$\frac{2d^{2}}{r}$. On the other hand, AdapterFusion has to assign parallel
pretrained adapters for each adapter layer, which multiplies the additional
deployment cost.

Figure 2: Comparison between single adapters and AdapterFusion under 10-shot
scenarios.

Despite the parameters budgets, to check whether AdapterFusion is an optimal
choice to improve the task-specific performance under data constraint
scenarios, we conduct few-shot experiments to compare single adapter and
AdapterFusion. As is shown in Figure 2, single adapters consistently
outperform AdapterFusion with fewer parameters, which indicates the
superiority of tuning single adapters. Inspired by the preliminary study, we
intend to merge multiple adapters into a single model to explore the potential
of single adapters.

Table 1: Experimental results of different merging Methods on GLUE benchmark,
where we perform merging with the same pretrained adapters for a fair
comparison. All tasks are evaluated using accuracy. Average scores on all
shots are underlined. The best results are bold.

Method | #Param. | MRPC | SST-2 | MNLI
---|---|---|---|---
(Trained) |  10 |  20 |  30 | Avg. |  10 |  20 |  30 | Avg. |  10 |  20 |  30 | Avg.
Fine-Tune | 100% | 66.2 | 67.6 | 69.1 | 67.6 | 60.7 | 62.3 | 63.9 | 62.3 | 39.4 | 41.1 | 42.2 | 40.9
AdapterFusion | 18% | 65.2 | 66.4 | 67.5 | 66.4 | 60.1 | 61.5 | 63.6 | 61.7 | 39.3 | 40.8 | 41.7 | 40.6
Adapter | 0.8% | 65.4 | 66.8 | 67.7 | 66.6 | 60.2 | 61.1 | 64.1 | 61.8 | 39.1 | 41.0 | 42.0 | 40.7
w/ Sum | 0.8% | 67.4 | 67.9 | 68.6 | 68.0 | 61.2 | 62.1 | 64.2 | 62.5 | 39.2 | 41.4 | 42.0 | 40.9
w/ Avg | 0.8% | 67.2 | 67.7 | 68.4 | 67.8 | 60.6 | 62.6 | 63.7 | 62.3 | 39.4 | 41.7 | 42.1 | 41.1
w/ Wts | 0.8% | 67.4 | 68.4 | 69.6 | 68.5 | 60.7 | 63.3 | 65.4 | 63.1 | 39.9 | 41.8 | 42.4 | 41.4
w/ Acts | 0.8% | 67.6 | 68.8 | 70.3 | 68.9 | 61.6 | 63.2 | 65.6 | 63.5 | 39.7 | 42.1 | 42.6 | 41.5

Adapter Merging.  Due to the limitations of AdapterFusion, we intend to make
more efficient use of the knowledge in the pretrained adapters. Integrating
the parallel adapter into a module can reduce the excess trainable parameters
and improve downstream performance. We first consider two simple methods for
merging the weights in adapters trained on different tasks. The first one is
the summation (“Sum.”):

$\widetilde{W}=\sum\nolimits_{j=1}^{n}W_{\tau_{j}},$ (1)

where we denote $\tau$ as the indices of the tasks and weights of the adapter
trained on task $\tau_{j}$ as $W_{\tau_{j}}$. The second one is averaging
(“Avg.”):

$\widetilde{W}=\frac{1}{n}\sum\nolimits_{j=1}^{n}W_{\tau_{j}}.$ (2)

However, the problem with vanilla summation and averaging is the lack of one-
to-one correspondence between parameters from different adapters. In
particular, the $p$-th neuron of the $i$-th adapter might behave very
differently from the $p$-th neuron of the $j$-th adapter and instead behave
similarly to another neuron. Therefore, aligning the neurons first and then
assembling adapters makes sense. Inspired by Singh and Jaggi (2020); Zan et
al. (2022), we align adapters’ parameters via optimal transport based on
weights (“Wts.”) and activations (“Acts.”).

Given adapters trained from task $\tau_{i}$ and $\tau_{j}$, we plug them into
language models and denote the $l$-th adapter layer’s incoming edges as
$W^{(\ell,\,\ell-1)}_{\tau_{i}}$ and $W^{(\ell,\,\ell-1)}_{\tau_{j}}$,
respectively. We align $W^{(\ell,\,\ell-1)}_{\tau_{j}}$ to
$W^{(\ell,\,\ell-1)}_{\tau_{i}}$ by constructing convex combinations of
previous layer transport matrix $T^{(\ell-1)}$ based on weights (“Wts”) or
activations (“Acts”), normalized appropriately via the inverse of
corresponding column marginals $\beta$:

$\widehat{W}^{(\ell,\,\ell-1)}_{\tau_{j}}\leftarrow
W^{(\ell,\,\ell-1)}_{\tau_{j}}T^{(\ell-1)}\text{diag}(1/\beta^{(\ell-1)}),$
(3)
$\widetilde{W}^{(\ell,\,\ell-1)}_{\tau_{j}}\leftarrow\text{diag}(1/{\beta^{(\ell)}}){T^{(\ell)}}^{\top}\widehat{W}^{(\ell,\,\ell-1)}_{\tau_{j}}.$
(4)

We refer $\widetilde{W}^{(\ell,\,\ell-1)}_{\tau_{j}}$ to the aligned weights
of the adapter from task $\tau_{j}$, which can be directly added with
$W^{(\ell,\,\ell-1)}_{\tau_{i}}$. We carry out this procedure on all
pretrained adapters:

$\widetilde{W}^{(\ell,\,\ell-1)}\leftarrow\frac{1}{n}\sum\nolimits_{j=1}^{n}{W}_{r_{j}}^{(\ell,\,\ell-1)}.$
(5)

##  3 Empirical Evaluation

###  3.1 Setup

We collect pretrained adapters from AdapterHub Pfeiffer et al. (2020), which
are trained on imdb Maas et al. (2011), boolq Clark et al. (2019), scitail
Khot et al. (2018) and winogrande Sakaguchi et al. (2021). The pretraining
tasks cover different NLP tracks, including sentiment analysis, question-
answering, natural language inference, and common sense reasoning. Our
experiments were conducted on the widely-used GLUE benchmark Wang et al.
(2019).

We use Adam Kingma and Ba (2015) as the optimizer with $\beta_{1}$,
$\beta_{2}$ = 0.9, 0.98. We set the weight decay as 0.1 and grid-search the
learning rate and training epochs from {1e-5, 5e-5, 1e-4}, and {5, 10}. The
maximum length is 128. We follow previous works Phang et al. (2018); Lee et
al. (2020); Dodge et al. (2020); He et al. (2023); Zhong et al. (2023) to
fine-tune the pretrained language models, e.g., BERT Devlin et al. (2019), on
the downstream training set and report results on the dev set using the last
checkpoint.

Table 2: Comparison between MerA and adapter architectures under various fine-
tuning strategies. We display the performance of 30 shots tuning and denote
head-based tuning, prompt-based tuning, and prompt-based tuning with
demonstrations as “head”, “prompt”, and “prompt-demo”, respectively.

Method | #Param. | BERT-Base | RoBERTa-Base
---|---|---|---
(Trained) |  head |  prompt |  +demos | Avg. |  head |  prompt |  +demos | Avg.
Fine-Tune | 100% | 63.9 | 71.8 | 72.6 | 69.4 | 71.1 | 79.4 | 79.9 | 76.8
AdapterFusion | 18% | 63.6 | 71.2 | 71.7 | 68.8 | 70.9 | 78.9 | 79.4 | 76.1
Adapter | 0.8% | 64.1 | 71.5 | 72.1 | 69.2 | 70.5 | 78.5 | 79.3 | 76.4
MerA | 0.8% | 65.7 | 73.6 | 73.9 | 71.1 | 73.3 | 81.3 | 82.1 | 78.9

###  3.2 Results

#### Main Results.

In Table 1, we carefully compare MerA (with merging methods above: “Sum.”,
“Avg.”, “Wts”, “Acts”) to the standard adapter Houlsby et al. (2019)
(“Adapter”) and AdapterFusion Pfeiffer et al. (2021) (“AF”) on GLUE benchmark
for pretrained language models BERT Devlin et al. (2019), where we set
training shots to 10, 20, 30, respectively.

MerA achieves significantly better performance than vanilla adapter Houlsby et
al. (2019); Pfeiffer et al. (2021) with the same parameters budget. Compared
to AdapterFusion Pfeiffer et al. (2021), our methods significantly reduce
required trainable parameters and improve performance.

Table 3: Effect of different tracks.  We denote a single pretrained adapter as
“+Single”, and denote MerA with pretrained tracks (e.g., “+QA”). For each
track, the same number of pretrained adapters are chosen for MerA.

Method | MRPC | MNLI
---|---|---
10 |  20 |  10 |  20
Adapter | 65.4 | 67.0 | 39.4 | 40.8
+Single | 65.0 | 67.4 | 40.6 | 42.3
+QA | 66.9 | 68.9 | 39.2 | 40.8
+NLI | 66.4 | 68.6 | $\text{44.2}^{\Uparrow+4.8}$ | $\text{45.8}^{\Uparrow+5.0}$
+Sentiment | 64.2 | 66.9 | 39.8 | 41.9
+Comsense | 65.4 | 68.2 | 39.6 | 41.8
+STS | $\text{68.9}^{\Uparrow+3.5}$ | $\text{70.1}^{\Uparrow+3.1}$ | 40.2 | 42.2

#### MerA with Different Merging Methods

MerA from all merging methods outperforms the standard adapter tuning, where
optimal transport methods outperform two naive methods in all tasks,
reflecting the necessity of weight alignment. Notably, “Acts” based Adapters
achieve up to 2.7% average improvement compared to the standard adapter and
even beat full fine-tuning, so we set “Acts” as the default setting in the
following experiments.

(a) MNLI

(b) SST-2

Figure 3: Effect of MerA on Initialization. “Baseline” represent vanilla BERT.
We denote BERT equipped with adapters as “Adapters” and denote BERT equipped
with “MerA” as “MerA”, respectively.

#### Prompt-Based Finetuning

In addition to head-based fine-tuning, we also investigate the applicability
of MerA in prompt-based fine-tuning, which leverages informative and
indicative prompts to enhance performance Gao et al. (2021). In Table 2, we
consider two strategies including prompt-based finetuning and prompt-based
finetuning with demonstrations. Experimental results demonstrate the
consistent improvement of MerA across various fine-tuning strategies. This
highlights the effectiveness and versatility of MerA in enhancing performance
in various fine-tuning scenarios.

#### Effect on Initialization

To analyze the benefits of merging adapters, we conducted a zero-shot
experiment to examine the effect of MerA on task initialization. In Figure 3,
we plugged MerA into BERT and compared it with BERT models equipped with
randomly initialized adapters (“Adapters”) and without any adapters
(“Baseline”). Compared to “Adapter” and “Baseline”, MerA ensures a superior
initial state for the target tasks and achieves significant accuracy
improvement, e.g., 2.4% in MNLI and 0.98% in SST-2. These findings reveal the
efficacy of MerA in enhancing the initialization of target tasks.

#### Augmenting MerA with Same-Track Setting

One potential approach to further increase the gain of MerA at initialization
is to merge the adapters trained in the same NLP track because of the sharing
of knowledge in a specific track. So we further investigate the roles of
different pretrained tracks. We consider pretrained adapters from five
different tracks, including question-answering (QA), common sense reasoning
(Comsense), natural language inference (NLI), semantic textual similarity
(STS), and sentiment analysis (Sentiment). MRPC and MNLI belong to semantic
textual similarity and natural language inference, respectively. We merge
adapters trained in the same track and validate MerA on MNLI and MRPC. We also
consider directly fine-tuning a single pretrained adapter “+Single”,
experimenting with multiple adapters within the same tracks, and reporting the
best results.

Table 3 compares MerA merged from different tracks, where we can see
significant improvements when the pretraining tasks are on the same track as
the downstream task. In this case, MerA outperforms the standard adapter by
3.5% in MRPC and 5.0% in MNLI. However, when pretrained tasks are on a
different track, MerA may encounter a performance drop. The above findings
inform the importance of the knowledge shared in one track.

##  4 Conclusion

In this work, we systematically investigate adapters and AdapterFusion on few-
shot scenarios. Based on our findings, we propose a plug-in strategy, i.e.,
MerA, for existing adapters. Our empirical results indicate the potential to
make MerA a golden standard efficient few-shot learning strategy for the NLP
community.

##  5 Limitations

Despite the progress we made, there still exist limitations in our work. On
the one hand, we only investigated some classic merging methods and found that
“Acts” performs the best in selected criteria. However, other advanced pruning
methods may exist that can further improve the performance, which deserves
exploration in future work. On the other hand, since we only consider BERT and
RoBERTa in limited tasks, it would be valuable to consider other architecture
families (e.g., XLNET Yang et al. (2019), ELECTRA Clark et al. (2020)) and
tasks (e.g., machine translation).

## Ethics Statement

We take ethical considerations very seriously. This paper focuses on higher
model and data efficiency for Adapters. Both the datasets and models used in
this paper are publicly available and have been widely adopted by researchers.
We ensure that the findings and conclusions of this paper are reported
accurately and objectively.

## References

  * Bansal et al. (2022) Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee, and Andrew McCallum. 2022\.  Meta-adapters: Parameter efficient few-shot fine-tuning through meta-learning.  In _AutoML_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Chawla et al. (2021) Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. 2021.  Data-free knowledge distillation for object detection.  In _WACV_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _ICLR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint_.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _ACL_.
  * He et al. (2022a) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022a.  Towards a unified view of parameter-efficient transfer learning.  In _ICLR_.
  * He et al. (2023) Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, and Dacheng Tao. 2023.  Pad-net: An efficient framework for dynamic networks.  In _ACL_.
  * He et al. (2022b) Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022b.  SparseAdapter: An easy approach for improving the parameter-efficiency of adapters.  In _EMNLP_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _ICML_.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.  In _AAAI_.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Lee et al. (2020) Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020.  Mixout: Effective regularization to finetune large-scale pretrained language models.  In _ICLR_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Others. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _ACL_.
  * Moosavi et al. (2022) Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. 2022\.  Adaptable adapters.  In _NAACL_.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021.  Adapterfusion: Non-destructive task composition for transfer learning.  In _EACL_.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterhub: A framework for adapting transformers.  In _EMNLP Demonstrations_.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman. 2018.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint_.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Commun. ACM_.
  * Singh and Jaggi (2020) Sidak Pal Singh and Martin Jaggi. 2020.  Model fusion via optimal transport.  In _NeurIPS_.
  * Solomon et al. (2015) Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas J. Guibas. 2015.  Convolutional wasserstein distances: efficient optimal transportation on geometric domains.  _ACM Trans. Graph._
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _ICLR_.
  * Wang et al. (2022) Zhenyi Wang, Xiaoyang Wang, Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. 2022.  Meta-learning without data via wasserstein distributionally-robust model fusion.  In _UAI_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _NeurIPS_.
  * Zan et al. (2022) Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022.  On the complementarity between pre-training and random-initialization for resource-rich machine translation.  In _COLING_.
  * Zhong et al. (2022) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022.  Panda: Prompt transfer meets knowledge distillation for efficient model adaptation.  _arXiv preprint_.
  * Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. 2023.  Revisiting token dropping strategy in efficient BERT pretraining.  In _ACL_.