# Exceeding the Limits of Visual-Linguistic Multi-Task Learning

Cameron R. Wolfe  Rice UniversityHouston, TXUSA<EMAIL_ADDRESS>and  Keld T.
Lundgaard  Salesforce EinsteinCambridge, MAUSA

###### Abstract.

By leveraging large amounts of product data collected across hundreds of live
e-commerce websites, we construct 1000 unique classification tasks that share
similarly-structured input data, comprised of both text and images. These
classification tasks focus on learning the product hierarchy of different
e-commerce websites, causing many of them to be correlated. Adopting a multi-
modal transformer model, we solve these tasks in unison using multi-task
learning (MTL). Extensive experiments are presented over an initial 100-task
dataset to reveal best practices for “large-scale MTL” (i.e., MTL with $\geq
100$ tasks). From these experiments, a final, unified methodology is derived,
which is composed of both best practices and new proposals such as DyPa, a
simple heuristic for automatically allocating task-specific parameters to
tasks that could benefit from extra capacity. Using our large-scale MTL
methodology, we successfully train a single model across all 1000 tasks in our
dataset while using minimal task specific parameters, thereby showing that it
is possible to extend several orders of magnitude beyond current efforts in
MTL.

Multi-task learning, deep learning, multi-modal Learning, transformers

Work done while CW was an intern at Salesforce Einstein.

††copyright: none††ccs: Computing methodologies Machine learning
algorithms††ccs: Computing methodologies Neural networks††ccs: Computing
methodologies Computer vision††ccs: Computing methodologies Natural language
processing

##  1\. Introduction

Figure 1. A depiction of the BERT-style, multi-modal classification model that
was used in all experiments. The main parameters of the network (i.e., the
parameters of the transformer) are shared, and a task-specific classification
head is assigned to each task.

Background. The transformer model (Vaswani et al., 2017) has revolutionized
deep learning research (Devlin et al., 2018; Zhou et al., 2019; Dosovitskiy et
al., 2020; Carion et al., 2020). After being proposed for neural machine
translation, transformers were adopted for self-supervised training over large
language corpora with the proposal of BERT (Devlin et al., 2018), allowing
models to be pre-trained, made publicly available, and fine-tuned on
downstream tasks (Devlin et al., 2018; Liu et al., 2019). Due to their
remarkable performance, transformer models became popular in domains beyond
natural language processing (NLP), such as computer vision (Dosovitskiy et
al., 2020; Carion et al., 2020), multi-modal deep learning (Lu et al., 2019a;
Kiela et al., 2019), and even music generation (Huang et al., 2018). The use
cases for transformers are constantly expanding as deep learning practitioners
develop new applications (Lee et al., 2020; Reimers and Gurevych, 2019; Zhang
et al., 2019; Sun et al., 2019).

The current trend in transformer research (and deep learning in general) is
towards larger models and datasets (Brown et al., 2020; Conneau et al., 2019;
Raffel et al., 2019). Despite the widespread moral and practical questioning
of this trend (Sharir et al., 2020; Peng and Sarazen, 2019; Hao, 2019;
Strubell et al., 2019), the deep learning community has continued forward,
having yet to reach an upper bound to model capacity and performance (Brown et
al., 2020). In fact, significantly overparameterized deep learning models have
even been shown to discover high-performing, generalizable solutions (Nakkiran
et al., 2019). These findings illustrate that one may benefit from using a
larger model. However, such benefit comes at the cost of increased inference
and training time, possibly to the point of complete intractability (Sharir et
al., 2020; Brown et al., 2020).

So, where does this leave deep learning practitioners if they want to reap the
performance benefits of larger transformer models? One option would be to
discover low-cost transformer variants that perform well (Wang et al., 2020;
Dai et al., 2020; Kitaev et al., 2020; Wu et al., 2020; Guo et al., 2019; Lan
et al., 2019). Additionally, transformers could be pruned to reduce inference
time (Fan et al., 2019; Goyal et al., 2020; Chen et al., 2020; Gordon et al.,
2020). However, despite the valuable contributions of these methodologies,
transformer models remain computationally expensive to train, especially if
done from scratch. Inspired by these issues, this work explores an orthogonal
direction of research. Namely, if the computational complexity of the
transformer model cannot be completely avoided, _how can we get the most
benefit from training a single model?_

Multi-task learning (MTL) with transformer models has been explored by
previous work (Liu et al., 2019; Cooper Stickland and Murray, 2019; Lu et al.,
2019b). Because BERT models encode redundant data within their representations
(Goyal et al., 2020; Zhou et al., 2020) and can be pruned without performance
deterioration (Fan et al., 2019; Gordon et al., 2020; Chen et al., 2020), such
models intuitively contain sufficient capacity to perform well in the MTL
domain. However, no work has yet fully explored the limits of transformers for
MTL (i.e., most works solve only 10-20 tasks in unison (Liu et al., 2019;
Cooper Stickland and Murray, 2019)).

Our contribution. In this work, a multi-modal, multi-task classification
dataset is constructed using e-commerce product images, textual descriptions,
and categorical attributes. Such e-commerce data allows for the formation of
1000 similar, but unique classification tasks that can be solved with MTL. In
our case, we focus on groups of task that learn the product hierarchy of
different e-commerce websites, which has numerous practical applications in
improving personalized product recommendations. To handle this dataset, a
multi-modal BERT model (Kiela et al., 2019) is adopted and modified to
simultaneously solve hundreds, or thousands, of such classification tasks with
a single model and minimal task-specific parameters. We coin this problem
“large-scale MTL”, which represents MTL in the case of $\geq$ 100 tasks being
solved by a single model. Our work aims to derive a simple, unified approach
for training high-performing models within the large-scale MTL domain.
Although we utilize a previously-proposed model architecture, the novel
contribution of this work lies within the training methodology that is
developed of large-scale MTL.

Some notable contributions of this work are as follows:

  * •

We perform extensive experiments with an MTL dataset of 100 tasks, resulting
in a unified, high-performing methodology for training models in the large-
scale MTL domain.

  * •

We extensively study the behavior of our proposed methodology, providing
numerous ablation experiments for transfer learning, increased model capacity,
smaller-scale MTL datasets, and much more.

  * •

We demonstrate that our proposed methodology is very scalable and capable of
solving 1000 classification tasks simultaneously.

##  2\. Related Work

Multi-task learning has been a popular topic in deep learning for some time.
Formative works in this area explored methods of optimally weighting tasks
within a group or modifying gradients to prevent conflict between tasks
(Kendall et al., 2017; Yu et al., 2020). Following these efforts, numerous
methods of handling objectives comprised of multiple tasks were proposed (Wang
et al., 2020; Clark et al., 2019; Liu et al., 2018; Lee et al., 2017).
However, such methods share a common goal of training a unified model over a
group of tasks that performs well and limits requirements for task-specific
parameters. Multi-task learning approaches have since been applied to numerous
domains, such as forming sentence embeddings (Sanh et al., 2018; Subramanian
et al., 2018), solving computer vision tasks (Kokkinos, 2016), and even
performing multi-modal reasoning (Lu et al., 2019b; Pramanik et al., 2019;
Nguyen and Okatani, 2018). Several, more comprehensive, summaries of
developments in the multi-task learning space are also available (Ruder, 2017;
Worsham and Kalita, 2020).

The introduction of the transformer architecture (Vaswani et al., 2017) and
BERT (Devlin et al., 2018) revolutionized deep learning for NLP (Liu, 2019;
Liu et al., 2019; Houlsby et al., 2019; Conneau et al., 2019) and several
other domains (Carion et al., 2020; Huang et al., 2018; Lee et al., 2020;
Dosovitskiy et al., 2020). Shortly after their introduction, transformer
architectures were applied to multi-modal data, leading to numerous variants
(Huang et al., 2020; Lu et al., 2019a; Zhou et al., 2019; Gao et al., 2020;
Chen et al., 2019a; Qi et al., 2020; Tan and Bansal, 2019; Su et al., 2019; Li
et al., 2019a, b; Alberti et al., 2019; Sun et al., 2019). Such attention-
based approaches for multi-modal learning can be roughly categorized into
single-stream and separate-stream methodologies, based on whether all
modalities are processed separately or as a single sequence. Generally,
single-stream architectures are popular because of their simplicity and
performance (Chen et al., 2019a; Zhou et al., 2019; Singh et al., 2020; Cao et
al., 2020). Many of such single-stream models for multi-modal deep learning
share identical architectures to BERT (Devlin et al., 2018) and can even be
initialized with the same pre-trained weights (Qi et al., 2020; Kiela et al.,
2019; Singh et al., 2020). For multi-modal classification, it has been shown
that BERT-style, single-stream architectures perform well and are easy to
train (Kiela et al., 2019).

Transformer models, especially variants of BERT, have become popular in the
multi-task learning community. Due to their many parameters (e.g., 110M
parameters in BERT base and 340M parameters in BERT large (Devlin et al.,
2018)), these models are capable of learning generalizable representations for
many tasks simultaneously. For example, BERT models have been used to
simultaneously solve multiple GLUE tasks (Wang et al., 2018; Liu et al., 2019;
Cooper Stickland and Murray, 2019), leading to improved generalization across
all tasks. Similarly, multi-task learning was extended to multi-modal
applications, leading to the development of visual-linguistic models trained
on diverse sets of tasks (Nguyen and Okatani, 2018; Lu et al., 2019b). In
these cases, multi-task learning was shown to lead to improved overall model
performance on most tasks, proving information from other tasks enables
positive inductive transfer within a model’s shared representations (Wang et
al., 2020).

##  3\. Methodology

###  3.1. Dataset

Figure 2. Illustrates various properties of the 100-task dataset including: a
histogram of the number of labels for each task (upper left), a histogram of
the number of output classes for each task (upper right), a histogram of the
number of labels for each output class (lower left), and a scatter plot of the
number of labels and output classes for each task (lower right).

Our data is obtained from live e-commerce websites and cannot be made publicly
available. However, no multi-modal, multi-task datasets currently exist that
provide hundreds of classification tasks with similarly-structured input data
(i.e., tasks that can be solved with MTL). Therefore, this private dataset is
the only viable option, to the authors’ knowledge, for the study of large-
scale MTL. Nonetheless, we argue that large-scale MTL is quite applicable in
the industrial setting. In our case, such data arises from a multi-tenency of
e-commerce websites, but a similar setup could exist for many different
software-as-a-service (SaaS) companies with several customers (e.g., web
hosting platforms, human resources platforms, advertisement platforms, etc.).
In these cases, developing unified models that generalize across different
domains and customers can lower costs and ease the use of such models in
practice (i.e., serving a single model for all customers is easier than
serving a separate model for every customer). In the future, large-scale
multi-task datasets could be created or derived from established datasets for
further study in this area.

Each example in our dataset, associated with a single e-commerce product, has
standardized metadata associated with it, including textual data (e.g., the
name or description) and many images. Numerous categorical attributes may also
be assigned to each product, including the “type” and “category”
attributes.111Product type corresponds to the lowest-level classification of a
product (e.g., shirt, hat, etc.). Product category represents a product’s
global position in the product hierarchy of a website (e.g., “Men
$\rightarrow$ Dress $\rightarrow$ Shirt” or “Children $\rightarrow$ Winter
$\rightarrow$ Hat”). The values of these attributes are one-hot encoded to
create the set of output classes for a task. The possible values for these
attributes are not standardized, allowing many unique classification tasks to
be created by predicting type and category attributes across different
websites. Though numerous different attributes exist, we choose to study the
product type and category attributes because they are related to the product
hierarchy of an e-commerce website, which has practical applications in
improving product recommendations. Many of these attribute prediction tasks
are difficult to solve in isolation due to a lack of sufficient data (i.e.,
some websites have very few training examples), making the possibility of
positive inductive transfer via MTL appealing.

The dataset used in the majority of experiments, comprised of over 250,000
products, contains 100 attribute prediction tasks, sampled from 50 unique
websites (i.e., product type and category prediction for each unique website).
As previously mentioned, both of these attributes are related to the product
hierarchy of e-commerce websites. As a result, many of the attribute
prediction tasks in our dataset are correlated (e.g., the product hierarchies
of different apparel-based e-commerce websites are usually similar). Various
metrics and properties of the 100-task dataset are depicted in Fig. 2. As can
be seen, there is significant variability in the properties of each task.
Experiments with up to 1000 tasks are also provided in Section 5.2. These
larger datasets are constructed identically, but with more e-commerce
websites.

###  3.2. Model Architecture

We adopt the model architecture proposed in (Kiela et al., 2019) with some
slight modifications, outlined below. The model uses the BERT base uncased
architecture and is initialized with the associated pre-trained weights
(Devlin et al., 2018). The processing of textual data is identical to that of
BERT. Images are converted to image embeddings with the EfficientNet-B4 model
(Tan and Le, 2019) and projected to the correct dimension with a learnable
linear layer. Image embeddings are created as a preprocessing step, and the
EfficientNet model is not fine-tuned during training. All image embeddings
associated with a product are concatenated with the textual tokens (i.e.,
divided by a “[SEP]” token) to form a single input sequence. Token type and
position embeddings, which are fine-tuned throughout training, are also added
to the input sequence. Image and textual tokens receive different token type
embeddings. The same position embedding is added to all image tokens (i.e.,
the images are not ordered), but position embeddings are incremented at each
position in the textual sequence. BERT parameters are fine-tuned and shared
across tasks. The output of BERT is passed into a task-specific classification
head to perform the final classification, and the correct task must be
specified as part of the forward pass. A detailed illustration of this model
is presented in Fig. 1.

###  3.3. Evaluation

Because our model is trained to simultaneously solve up to 1000 tasks,
evaluating the model’s performance is not trivial. We choose to perform
evaluation with several metrics:

  * •

Mean Accuracy: The accuracy is computed separately for each task, and the mean
of these accuracies is reported.

  * •

T10 Accuracy: This metric considers the 10% of tasks that contain the most
number of data examples. The accuracy is computed separately for each of these
tasks, and the mean of these accuracies is reported (i.e., mean accuracy of
high-resource tasks).

  * •

B10 Accuracy: This metric considers the 10% of tasks that contain the least
number of data examples. The accuracy is computed separately for each of these
tasks, and the mean of these accuracies is reported (i.e., mean accuracy of
low-resource tasks).

All of the above metrics are evaluated on hold-out test sets for each task,
which are constructed with a uniform 80-20 split.

###  3.4. Experimental Details

Experiments on the 100-task dataset are run for 15 total epochs, where an
epoch is defined as a cycle through a number of data examples equal to the
size of the full dataset. For datasets with more tasks (e.g., see Section
5.2), the number of epochs is reduced proportionally, based on the number of
training examples in the dataset. All experiments use a batch size of 64,
which was the maximum size to fit in a single Tesla V100 GPU. Although some of
the larger tests were run on a100 GPUs with more RAM, we maintain the same
batch size across all tests for simplicity. Each batch is comprised entirely
of examples from a single task, and all models are trained with the AdamW
optimizer. Each test was run on a single GPU for simplicity.

##  4\. How to handle lots of tasks

In this section, extensive ablation experiments over the 100-task dataset are
provided. The lessons learned from these experiments allowed us to arrive
step-by-step at our final methodology, which yields superior performance in
the large-scale MTL domain. This training methodology, discovered through
extensive experimentation and tuning, is the main novel contribution of this
work, as no previous work attempts to solve such a large number of tasks
simultaneously using MTL.

###  4.1. Optimization

Figure 3. Mean accuracy on the 100-task dataset with different learning rate
strategies.

Because many tasks are being optimized simultaneously, achieving convergence
with a large-scale MTL model is not trivial. In fact, numerous initial
attempts at training a model on the 100-task dataset diverged completely.
Eventually, it was discovered that convergence could be achieved by i) using a
low learning rate throughout training ii) freezing the shared transformer
weights during initial epochs or iii) utilizing a well-tuned learning rate
warm up schedule.

The fixed learning rate strategy utilizes a learning rate of $10^{-5}$
throughout training. The freezing strategy utilizes a higher learning rate of
$10^{-4}$ during initial epochs, but keeps the weights of the transformer
backbone fixed. The weights of the transformer are unfrozen after four epochs,
at which point the learning rate is reduced by $10\times$. The warm up
schedule is comprised of an initial warm up phase, which increases the
learning rate from $10^{-5}$ to $10^{-4}$ during the first four epochs,
followed by a step schedule that decreases the learning rate $10\times$ at
epochs eight and 12.

A comparison of each of these three optimization strategies is presented in
Fig. 3. Although the fixed learning rate strategy converges, the training
process was slow compared to other methods. The freezing strategy provided a
slight improvement in performance, but the best results were achieved using
the warm up strategy. As shown in Fig. 3, the warm up strategy quickly reaches
a stable plateau in performance, leading us to adopt this strategy in the
following experiments.

###  4.2. Task Sampling Strategies

Figure 4. T10 accuracy achieved by models trained with different task sampling
strategies on the 100-task dataset.

Each batch during training contains examples sampled from a single task.
However, the best approach for sampling tasks at each iteration has yet to be
standardized, especially in the large-scale MTL domain. The naive approach
would be to randomly sample a task with uniform probability at each iteration.
On the other hand, one could sample tasks with a probability that is
proportional to the number of data examples for that task (i.e., referred to
as “data size” sampling). These two strategies can also be interpolated to
yield the probability of sampling task $T$ for a given batch, $P(T)$, as
follows.

$\displaystyle P(T)=\frac{N_{T}^{\alpha}}{\sum_{t}N_{t}^{\alpha}}$

where $N_{T}$ is the number of training examples for task $T$ and
$\alpha\in[0,1]$ is a hyperparameter. The value of $\alpha$ can be simply
understood as an interpolation between uniform ($\alpha=0$) and data size
($\alpha=1$) sampling.

Previous work has shown that annealing schedules for $\alpha$ can yield
performance improvements for MTL (Cooper Stickland and Murray, 2019). To test
the validity of such claims for large-scale MTL, we compare decaying the value
of $\alpha$ throughout training (i.e., linear decay from 1.0 to 0.0) to data
size and uniform task sampling; see Fig. 4. Although data size sampling seems
to provide faster convergence in initial epochs, $\alpha$ decay significantly
outperforms both data size and uniform task sampling. We emphasize that the
choice of task sampling strategy significantly impacts performance in the
large-scale MTL domain (i.e., 10% difference in T10 accuracy). _This
observation inspired an extensive exploration of different $\alpha$ decay
schedules to ensure the best possible task sampling strategy was discovered._

###  4.3. Choosing an $\alpha$ Decay Schedule

Table 1. Model performance for different $\alpha$ decay schedules on the 100-task dataset. $\alpha$ Decay Method |  $\alpha$ Range | Mean Acc. | T10 Acc. | B10 Acc.
---|---|---|---|---
Linear |  1.0 $\rightarrow$ 0.5 |  74.66% $\pm$ 0.42 |  73.56% $\pm$ 0.56 |  50.05% $\pm$ 1.91
|  1.0 $\rightarrow$ 0.1 |  74.97% $\pm$ 0.23 |  72.37% $\pm$ 0.48 |  51.70% $\pm$ 0.95
|  1.0 $\rightarrow$ 0.0 |  75.31% $\pm$ 0.31 |  72.49% $\pm$ 0.74 |  52.98% $\pm$ 0.29
Exponential |  1.0 $\rightarrow$ 0.5 |  74.79% $\pm$ 0.12 |  73.33% $\pm$ 0.31 |  51.69% $\pm$ 0.19
|  1.0 $\rightarrow$ 0.1 | 75.48% $\pm$ 0.13 |  71.69% $\pm$ 0.31 | 54.09% $\pm$ 0.88
|  1.0 $\rightarrow$ 0.0 |  74.86% $\pm$ 0.27 |  69.85% $\pm$ 1.15 |  52.38% $\pm$ 0.72
Cosine |  1.0 $\rightarrow$ 0.5 |  74.63% $\pm$ 0.11 |  73.71% $\pm$ 0.27 |  52.25% $\pm$ 0.27
|  1.0 $\rightarrow$ 0.1 |  74.88% $\pm$ 0.31 |  73.33% $\pm$ 0.50 |  53.02% $\pm$ 0.33
|  1.0 $\rightarrow$ 0.0 |  74.47% $\pm$ 0.35 |  73.28% $\pm$ 0.45 |  51.20% $\pm$ 0.91
Demon |  1.0 $\rightarrow$ 0.5 |  73.20% $\pm$ 1.00 | 73.82% $\pm$ 0.69 |  47.46% $\pm$ 2.90
|  1.0 $\rightarrow$ 0.1 |  74.51% $\pm$ 0.13 |  73.72% $\pm$ 0.41 |  52.01% $\pm$ 1.51
|  1.0 $\rightarrow$ 0.0 |  74.31% $\pm$ 0.10 |  73.84% $\pm$ 0.26 |  52.92% $\pm$ 0.81
Figure 5. Illustration for all different $\alpha$ decay schedules that were
tested. All decay schedules started and ended at the beginning and ending of
training, respectively. The Demon schedule was adopted from (Chen et al.,
2019b).

We empirically compare numerous options for $\alpha$ decay, depicted in Fig.
5. Exponentially decaying $\alpha$ from a value of 1.0 to 0.1 throughout
training was found to consistently achieve balanced performance; see Table 1.
As a result, we adopt this approach in the rest of experiments. Intuitively,
the exponential $\alpha$ decay schedule works well because it performs data
size sampling during the early parts of training, where it is conducive to
faster convergence. However, the schedule quickly decays to lower $\alpha$
values to avoid damaging the model’s performance. Experiments were also
performed using restarts, warm up, and cyclical schedules for $\alpha$, but
none of these more complex schedules improved performance.

###  4.4. How often should a new task be sampled?

Previous work has shown that performing consecutive iterations on the same
task could degrade MTL performance (Cooper Stickland and Murray, 2019; Liu et
al., 2019). We find that the same is true within the large-scale MTL domain.
Interestingly, performing as few as ten consecutive iterations on each task
during training led our model to completely diverge on the 100-task dataset.
Therefore, _sampling a new task at every iteration of training is seemingly
conducive to good performance._

###  4.5. Limiting Task Specific Parameters

Figure 6. Comparison of the performance of attention-based task-specific heads
to FC task-specific heads in terms of mean accuracy.

Within our implementation, each task must have a task-specific classification
head. A “naive” classification head implementation – a fully-connected (FC)
layer followed by a nonlinearity (e.g., ReLU) – consumes many parameters. For
example, FC classification heads consume 58 million parameters for the
100-task dataset.222Fewer parameters could be potentially consumed by
performing a single linear projection from the tokens outputted by BERT to the
output space. However, we found that such an approach performed worse than
utilizing a classification head that contains a non-linearity. Intuitively,
this problem worsens as the methodology scales to hundreds, or thousands, of
tasks. Therefore, _a more parameter-efficient approach had to be developed to
make large-scale MTL more feasible._

Inspired by (Cooper Stickland and Murray, 2019), we formulate each task-
specific classification head as a low-dimension self-attention module. More
specifically, we take the final token sequence outputted from BERT, project
the tokens to some lower dimension $d_{t}$, then perform self-attention at
this lower dimension. After the self-attention layer, tokens are averaged and
passed through a linear classification layer to get the final output. Using
this formulation, we are able to significantly decrease usage of task-specific
parameters (e.g., $d_{t}=64$ uses $10\times$ fewer parameters than FC
classification heads). The performance of FC and attention-based
classification heads (i.e., with $d_{t}=64$) is shown in Fig. 6. As can be
seen, the attention-based task-specific heads, despite using $10\times$ fewer
parameters, maintain similar performance to the FC classification heads,
leading us to use them within the proposed methodology.

####  4.5.1. Investing Task-Specific Parameters Wisely

Though parameter-efficiency makes large-scale MTL more feasible, increasing
the number of task-specific parameters may be useful if performance is
improved. To investigate whether performance can be improved with more task-
specific parameters, we run experiments using task-specific attention modules
with different numbers of heads and hidden dimensions.333Task-specific
attention modules with multiple layers were also tested, but led to no
noticeable performance improvement. The results of these experiments are
presented in Table 2. Assuming a fixed value of $d_{t}$, changing the number
of attention heads does not significantly impact performance. Furthermore,
model performance on high-resource tasks consistently improves as $d_{t}$ is
increased, but B10 accuracy begins to degrade as $d_{t}$ becomes too large,
revealing that larger classification heads cause overfitting on low-resource
tasks. Because the sizes of task-specific heads cannot be tuned by hand in the
large-scale MTL domain (i.e., this would require manually setting $d_{t}$ for
hundreds or thousands of tasks), _there is a need for some heuristic that
automatically allocates task-specific parameters to tasks that need them_.

Table 2. Performance of different sizes of task-specific classification heads on the 100-task dataset. For DyPA, the hidden dimension of task-specific attention heads in the lowest quartile is listed. This dimension increases by $2\times$ in each following quartile. Task-Specific Head Type | # Attn. Heads | $d_{t}$ | Mean Acc. | T10 Acc. | B10 Acc.
---|---|---|---|---|---
Fixed-Size | 2 | 64 | 72.95% | 66.39% | 55.07%
| 4 | 64 | 72.28% | 66.41% | 51.29%
| 2 | 128 | 72.17% | 69.55% | 54.41%
| 4 | 128 | 72.28% | 68.85% | 52.04%
| 2 | 512 | 72.04% | 67.11% | 51.79%
| 4 | 512 | 73.74% | 68.70% | 51.43%
| 2 | 1024 | 73.69% | 69.72% | 51.06%
| 4 | 1024 | 73.69% | 69.72% | 51.05%
DyPA | 2 |  128 ($2\times$) | 74.33% | 69.83% | 54.39%
| 4 |  128 ($2\times$) | 74.18% | 69.77% | 54.00%

###  4.6. Dynamic Parameter Allocation (DyPA)

Dynamic Parameter Allocation (DyPA) is a heuristic we propose to solve the
problem outlined in Section 4.5.1. The intuitive idea behind DyPA is to
dynamically provide more task-specific parameters to tasks that would benefit
from increased capacity (i.e., complex or difficult tasks). In DyPA, the
number of labels associated with each task is used as a proxy for its
complexity. The number of labels for each task is first normalized with
respect to that of all other tasks, providing a normally-distributed
complexity score for each task. Using this score, all tasks are separated into
quartiles. Then, the size of a task’s classification head is determined by the
quartile in which it resides. Tasks in higher quartiles are given larger task-
specific classification heads (i.e., a larger value of $d_{t}$) and vice
versa. We implement DyPA using attention-based classification heads (see
Section 4.5), but the scheme can be trivially extended to other task-specific
module variants.

DyPA introduces a few new hyperparameters, as one must decide the sizes of
task-specific heads to be used in the first and last quartiles.444$d_{t}$ must
be specified for the first and fourth quartiles, then the values of $d_{t}$
for the second and third quartiles can be linearly interpolated. We observe
that T10 accuracy begins to saturate when $d_{t}>1024$. Therefore, we set
$d_{t}=128$ and $d_{t}=1024$ for the first and fourth quartiles, respectively
(i.e., quartiles two and three then have sizes 256 and 512). Using these
settings, DyPA is tested with both two and four attention heads. As can be
seen in Table 2, DyPA improves mean and T10 accuracy in comparison to fixed-
size classification heads, while maintaining comparable B10 accuracy. In other
words, _DyPA improves performance on high-resource tasks without overfitting
on low-resource tasks._ Furthermore, DyPA still uses roughly $3.5\times$ fewer
task-specific parameters in comparison to FC classification heads.

##  5\. Results

Table 3. Performance of the proposed methodology on datasets containing different numbers of tasks. Baseline performance is obtained by training a separate model on each task in the 100-task dataset. All metrics are evaluated over the 100-task dataset. Method | Mean Acc. | T10 Acc. | B10 Acc.
---|---|---|---
Baseline | 61.50% | 72.33% | 34.40%
100 Task MTL |  73.29% $\pm$ 0.43 |  69.30% $\pm$ 0.53 |  54.39% $\pm$ 0.42
500 Task MTL |  72.54% $\pm$ 0.13 |  67.36% $\pm$ 0.13 |  52.80% $\pm$ 1.80
1000 Task MTL | 74.98% $\pm$ 0.11 |  67.02% $\pm$ 0.15 | 57.08% $\pm$ 0.65

In this section, the performance of our final large-scale MTL methodology,
outlined in Section 4, is evaluated over the 100-task dataset and larger
datasets. As a baseline, we train models separately on each task in the
100-task dataset. It should be noted that this baseline is comprised of 100
BERT base models and is nearly infeasible to use in practice. Because our
training methodology for large-scale MTL incorporates best practice of
numerous previously-proposed training methodologies (see Sec. 4), we do not
directly adopt such methods as baselines within this section. Rather, we refer
to the extensive ablation experiments in Sec. 4 for comparisons to alternative
MTL training methodologies and choose not to replicate such comparisons on
larger datasets due to extensive compute requirements. We demonstrate that our
methodology effectively trains models to achieve high performance on all
tasks, _even when the number of tasks is increased to as many as 1000._

###  5.1. 100-Task Dataset

We first evaluate the proposed methodology on the 100-task dataset. As can be
seen in Table 3, MTL improves mean and B10 accuracy by over 10% and 20% in
comparison to the baseline, respectively. Although individually-trained models
achieve slightly improved T10 accuracy, such performance is to be expected, as
high-resource tasks contain sufficient training data to perform well in
isolation (i.e., each task in the T10 accuracy group has $>10$K training
labels). In the low-resource scenario, however, the baseline performance is
quite poor (i.e., $<35\%$ B10 accuracy). The positive inductive transfer
provided by large-scale MTL improves performance on these low-resource tasks
dramatically (i.e., over 20% absolute improvement), while only decreasing T10
accuracy by 3%. Furthemore, the average performance of MTL models across all
tasks in the 100-task dataset is markedly increased in comparison to the
baseline, improving by nearly 12%. Therefore, these results indicate that our
proposed methodology, _in addition to allowing 100 BERT base models to be
compressed into a single model_ , provides a significant performance benefit
on nearly all tasks.

###  5.2. More Tasks

To test the ability of the proposed methodology to generalize to larger
numbers of tasks, we train our model on datasets with up to 1000 tasks. As
explained in Sec. 3.1, these larger datasets are constructed identically to
the 100-task dataset, but with more e-commerce sites. To make the performance
of these models comparable to the baseline, we perform evaluation only over
the original 100 tasks, and use the remaining tasks only for training
purposes.

As seen in Table 3, our proposed large-scale MTL methodology performs well as
the number of tasked is scaled well beyond 100. It should be noted that such
performance is only measured over the original 100 tasks, and these models are
still solving hundreds of tasks on top of the original 100. Although the
500-task model yields slightly degraded performance, all of its validation
metrics are within 2% of those of the 100-task model, revealing that its
results are comparable. Furthermore, the 1000-task model, despite yielding
slightly degraded T10 accuracy, outperforms the 100-task model on other
metrics. In particular, the 1000-task model achieves the best mean and B10
accuracy, yielding 1.7% and 2.7% absolute improvements over the 100-task
model, respectively. Such a result indicates that, even as the number of tasks
is increased to as many as 1000, our proposed methodology can continue to
benefit from positive inductive transfer between tasks. As expected, this
positive inductive transfer seems to be most beneficial for low-resource
tasks, as the 1000-task MTL model is shown to outperform all other models in
B10 accuracy by a significant margin.

Models trained on the larger MTL datasets continue to significantly outperform
the baseline in terms of mean and B10 accuracy. Most notably, the 1000-task
model improves upon the B10 accuracy of the baseline by 22.7%, highlighting
that our methodology’s massive benefit to low-resource tasks improves as the
number of tasks is increased. In addition, the 500 and 1000-task models both
outperform the baseline by over 10% in terms of mean accuracy. As with the
100-task model, the baseline achieves better T10 accuracy in comparison to MTL
models trained on larger datasets. However, it should be noted that, for 500
and 1000-task models, the baseline could not even be constructed, as it would
require the training of several hundred BERT base models. In the 1000-task
case, _such a baseline would consume over 110 billion parameters, while our
model consumes roughly 250 million parameters._

##  6\. Ablation Experiments

###  6.1. Transfer Learning

Table 4. Classification performance for models initialized with either pre-trained BERT weights or large-scale MTL weights, then fine-tuned on a downstream classification task. Initialization Method | Test Accuracy
---|---
BERT Base |  90.27% $\pm$ 0.002
MTL (100 Task) |  90.77% $\pm$ 0.001
MTL (1000 Task) | 91.12% $\pm$ 0.075

Similarly to pre-trained transformer models, our large-scale MTL models are
trained over large datasets to simultaneously perform several tasks.
Intuitively, then, the representations learned through large-scale MTL should
be useful for transfer learning purposes. To test this theory, we fine-tune
models that are “pre-trained” with large-scale MTL on a separate e-commerce
classification dataset. This dataset contains 405,840 examples and 2,196
output classes. The data is formatted as described in Section 3.1, but
consists of examples across numerous e-commerce websites. In other words, it
is a unified ontology of product categories across many e-commerce websites.

The model used for fine-tuning is the same as shown in Fig. 1, but it has only
a single classification head. The transformer backbone of this model is
initialized using either BERT base weights or the parameters of a large-scale
MTL model (i.e., both the 100 and 1000-task models are tested). The entire
model is then fine-tuned for 10 epochs on the downstream ontology
classification task.555Learning rate and weight decay are selected with a grid
search using a hold-out validation set. Then, the model is retrained on the
full dataset with optimal hyperparameters to measure performance on the test
set. The performance of the fine-tuned models on a hold-out test set, averaged
across three separate trials, is shown in Table 4. As can be seen,
initializing the model with weights learned through large-scale MTL yields
improved downstream performance. This performance improvement becomes more
noticeable as the number of tasks used during large-scale MTL training is
increased. In particular, initializing with the 100-task and 1000-task MTL
models yields a 0.50% and 0.85% increase in performance relative to
initializing the model with pre-trained BERT base parameters. Given the
extensive pre-training of BERT base, this consistent, noticeable improvement
in transfer learning performance is quite surprising (i.e., our 100-task model
is trained on a single V100 GPU in less than one day) and demonstrates the
value of representations learned through large-scale MTL.

###  6.2. Increasing Model Capacity

Table 5. Comparison of MTL models trained on the 100-task dataset with both BERT base and BERT large used as transformer backbones. Model Backbone | Mean Acc. | T10 Acc. | B10 Acc.
---|---|---|---
BERT Base |  73.29% $\pm$ 0.43 | 69.30% $\pm$ 0.53 | 54.39% $\pm$ 0.42
BERT Large | 73.39% $\pm$ 0.39 |  68.81% $\pm$ 0.64 |  52.24% $\pm$ 0.93

The proposed model architecture, as shown in Fig. 1, uses BERT base as a
shared backbone between all tasks. Intuitively, increasing the capacity of
this transformer backbone could yield improved model performance. To test
this, the BERT base backbone is replaced with BERT large (Devlin et al.,
2018), and we train this larger model on the 100-task dataset. The results
obtained with the BERT large backbone are shown in Table 5. The larger model
obtains slightly improved mean accuracy, but is outperformed by BERT base on
other metrics. Because increasing the capacity of the transformer backbone
does not yield noticeably improved performance, these experiments clearly
demonstrate the shocking capacity of the BERT base model in the large-scale
MTL domain. Namely, BERT base has seemingly sufficient capacity to perform
well across hundreds of tasks simultaneously, even in comparison to a model
containing $4\times$ more parameters.

###  6.3. Robustness to Larger Tasks

Figure 7. Performance of the proposed methodology when an extra, extremely
large task is added to the 100-task dataset.

Several aspects of our proposed methodology depend upon the distribution over
the number of labels for each task (e.g., $\alpha$-decay and DyPA).
Consequently, one could question whether the model’s performance would be
sensitive to the properties of this distribution. For example, adding an
extra, very large task into the set of possible tasks could scew the
distributions used for sampling and binning within $\alpha$-decay and DyPA,
leading to a noticeable performance degredation. To test this, we add the
ontology classification dataset described in Sec. 6.1 as another task within
our 100-task dataset, which is an order of magnitude larger than any other
task (see Fig. 2). The results obtained by training a large-scale MTL model on
this 101-task dataset are shown in Fig. 7. As can be seen, the proposed
methodology is robust to the addition of this larger task. In fact, the
addition of the ontology classification task results in a 4% absolute
improvement in T10 accuracy. Therefore, it is clear that our large-scale MTL
methodology benefits from the addition of more and larger tasks.

###  6.4. From Small to Large Scale MTL

# Tasks | Use $\alpha$-decay | Use DyPA | Mean Accuracy
---|---|---|---
10 | $\times$ | $\times$ | 60.90%
| $\checkmark$ | $\times$ | 67.46%
| $\checkmark$ | $\checkmark$ | 69.97%
25 | $\times$ | $\times$ | 73.42%
| $\checkmark$ | $\times$ | 76.25%
| $\checkmark$ | $\checkmark$ | 78.12%
50 | $\times$ | $\times$ | 67.71%
| $\checkmark$ | $\times$ | 75.12%
| $\checkmark$ | $\checkmark$ | 75.47%
75 | $\times$ | $\times$ | 71.40%
| $\checkmark$ | $\times$ | 77.34%
| $\checkmark$ | $\checkmark$ | 78.65%
Table 6. Experiments that display the effectiveness of our proposed MTL
methodology at numerous different scales. DyPA and $\alpha$-decay are
selectively eliminated from training to observe their respective performance
benefits.

Although we have shown that the proposed methodology is useful in the large-
scale MTL domain, one may ask whether a similar approach is beneficial with
fewer tasks. To study our methodology’s impact at smaller scales, we measure
performance of MTL models trained across 10, 25, 50, and 75 tasks (i.e.,
smaller sub-samplings of the 100-task dataset). Furthermore, both
$\alpha$-decay and DyPA are selectively eliminated from training to directly
observe their benefit on the model’s performance. For each of these
experiments, only mean accuracy is reported for simplicity.

The results of these smaller-scale MTL experiments are shown in Table 6. As
can be seen, even when fewer tasks are used during training, employing both
$\alpha$-decay and DyPA always leads to the best performance. Interestingly,
the gap in performance between the vanilla model (i.e., trained without
$\alpha$-decay or DyPA) and the proposed methodology is relatively similar at
all scales (i.e., 4.5-9.0% improvement across datasets). As a result, our
methodology is shown to be generally useful for MTL, agnostic of the number of
tasks available for training.

##  7\. Conclusions and Future Work

In this work, we provide proof that overparameterized transformer models are
capable of achieving more than previously thought possible. Using a large-
scale MTL dataset comprised of 100 tasks, extensive experiments are performed
that allow us to understand how multi-modal BERT models can be most easily
trained to simulataneously solve hundreds of tasks. In particular, we propose
a novel DyPA scheme to limit the number of task-specific parameters used by
our model and discover best-practices for task sampling within large-scale
MTL, resulting in a final training methodology for large-scale MTL that is the
main contribution of this work. We then demonstrate that our large-scale MTL
training approach is scalable, capable of being trained across 1000 tasks. By
showing what is possible with a single model, we hope this work inspires
others, both in research and industry, to explore new, creative methods of
using large transformer models efficiently. In the future, we hope to extend
our methodology towards learning different types of tasks and doing zero-shot
learning on new tasks. In addition, we think that it would be very worthwhile
to explore the robustness of large-scale MTL to adversarial tasks, which could
increasingly be a problem as you extend the number and variety of tasks.

## References

  * (1)
  * Alberti et al. (2019) Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. 2019.  Fusion of Detected Objects in Text for Visual Question Answering.  _arXiv e-prints_ , Article arXiv:1908.05054 (Aug. 2019), arXiv:1908.05054 pages.  arXiv:1908.05054 [cs.CL]
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Cao et al. (2020) Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020\.  Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models.  _arXiv e-prints_ , Article arXiv:2005.07310 (May 2020), arXiv:2005.07310 pages.  arXiv:2005.07310 [cs.CV]
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers.  _arXiv preprint arXiv:2005.12872_ (2020).
  * Chen et al. (2019b) John Chen, Cameron Wolfe, Zhao Li, and Anastasios Kyrillidis. 2019b.  Demon: Momentum Decay for Improved Neural Network Training.  _arXiv e-prints_ , Article arXiv:1910.04952 (Oct. 2019), arXiv:1910.04952 pages.  arXiv:1910.04952 [cs.LG]
  * Chen et al. (2020) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020.  The Lottery Ticket Hypothesis for Pre-trained BERT Networks.  _arXiv e-prints_ , Article arXiv:2007.12223 (July 2020), arXiv:2007.12223 pages.  arXiv:2007.12223 [cs.LG]
  * Chen et al. (2019a) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019a.  UNITER: UNiversal Image-TExt Representation Learning.  _arXiv e-prints_ , Article arXiv:1909.11740 (Sept. 2019), arXiv:1909.11740 pages.  arXiv:1909.11740 [cs.CV]
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019.  BAM! Born-Again Multi-Task Networks for Natural Language Understanding.  _arXiv e-prints_ , Article arXiv:1907.04829 (July 2019), arXiv:1907.04829 pages.  arXiv:1907.04829 [cs.CL]
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised Cross-lingual Representation Learning at Scale.  _arXiv e-prints_ , Article arXiv:1911.02116 (Nov. 2019), arXiv:1911.02116 pages.  arXiv:1911.02116 [cs.CL]
  * Cooper Stickland and Murray (2019) Asa Cooper Stickland and Iain Murray. 2019.  BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning.  _arXiv e-prints_ , Article arXiv:1902.02671 (Feb. 2019), arXiv:1902.02671 pages.  arXiv:1902.02671 [cs.LG]
  * Dai et al. (2020) Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. 2020\.  Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing.  arXiv:2006.03236 [cs.LG]
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv e-prints_ , Article arXiv:1810.04805 (Oct. 2018), arXiv:1810.04805 pages.  arXiv:1810.04805 [cs.CL]
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _arXiv e-prints_ , Article arXiv:2010.11929 (Oct. 2020), arXiv:2010.11929 pages.  arXiv:2010.11929 [cs.CV]
  * Fan et al. (2019) Angela Fan, Edouard Grave, and Armand Joulin. 2019\.  Reducing Transformer Depth on Demand with Structured Dropout.  _arXiv e-prints_ , Article arXiv:1909.11556 (Sept. 2019), arXiv:1909.11556 pages.  arXiv:1909.11556 [cs.LG]
  * Gao et al. (2020) Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. 2020.  FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval.  _arXiv e-prints_ , Article arXiv:2005.09801 (May 2020), arXiv:2005.09801 pages.  arXiv:2005.09801 [cs.IR]
  * Gordon et al. (2020) Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020\.  Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning.  _arXiv e-prints_ , Article arXiv:2002.08307 (Feb. 2020), arXiv:2002.08307 pages.  arXiv:2002.08307 [cs.CL]
  * Goyal et al. (2020) Saurabh Goyal, Anamitra R. Choudhury, Saurabh M. Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020.  PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination.  _arXiv e-prints_ , Article arXiv:2001.08950 (Jan. 2020), arXiv:2001.08950 pages.  arXiv:2001.08950 [cs.LG]
  * Guo et al. (2019) Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019\.  Star-Transformer.  _arXiv e-prints_ , Article arXiv:1902.09113 (Feb. 2019), arXiv:1902.09113 pages.  arXiv:1902.09113 [cs.CL]
  * Hao (2019) Karen Hao. 2019\.  Training a single AI model can emit as much carbon as five cars in their lifetimes.  https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-Efficient Transfer Learning for NLP.  _arXiv e-prints_ , Article arXiv:1902.00751 (Feb. 2019), arXiv:1902.00751 pages.  arXiv:1902.00751 [cs.LG]
  * Huang et al. (2018) Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2018.  Music transformer: Generating music with long-term structure. In _International Conference on Learning Representations_.
  * Huang et al. (2020) Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020.  Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers.  _arXiv e-prints_ , Article arXiv:2004.00849 (April 2020), arXiv:2004.00849 pages.  arXiv:2004.00849 [cs.CV]
  * Kendall et al. (2017) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2017\.  Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics.  _arXiv e-prints_ , Article arXiv:1705.07115 (May 2017), arXiv:1705.07115 pages.  arXiv:1705.07115 [cs.CV]
  * Kiela et al. (2019) Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. 2019\.  Supervised Multimodal Bitransformers for Classifying Images and Text.  _arXiv e-prints_ , Article arXiv:1909.02950 (Sept. 2019), arXiv:1909.02950 pages.  arXiv:1909.02950 [cs.CL]
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The Efficient Transformer.  _arXiv e-prints_ , Article arXiv:2001.04451 (Jan. 2020), arXiv:2001.04451 pages.  arXiv:2001.04451 [cs.LG]
  * Kokkinos (2016) Iasonas Kokkinos. 2016\.  UberNet: Training a ‘Universal’ Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory.  _arXiv e-prints_ , Article arXiv:1609.02132 (Sept. 2016), arXiv:1609.02132 pages.  arXiv:1609.02132 [cs.CV]
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  _arXiv e-prints_ , Article arXiv:1909.11942 (Sept. 2019), arXiv:1909.11942 pages.  arXiv:1909.11942 [cs.CL]
  * Lee et al. (2017) Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. 2017\.  Deep Asymmetric Multi-task Feature Learning.  _arXiv e-prints_ , Article arXiv:1708.00260 (Aug. 2017), arXiv:1708.00260 pages.  arXiv:1708.00260 [cs.LG]
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ 36, 4 (2020), 1234–1240.
  * Li et al. (2019a) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. 2019a.  Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training.  _arXiv e-prints_ , Article arXiv:1908.06066 (Aug. 2019), arXiv:1908.06066 pages.  arXiv:1908.06066 [cs.CV]
  * Li et al. (2019b) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  _arXiv e-prints_ , Article arXiv:1908.03557 (Aug. 2019), arXiv:1908.03557 pages.  arXiv:1908.03557 [cs.CV]
  * Liu et al. (2018) Shikun Liu, Edward Johns, and Andrew J. Davison. 2018\.  End-to-End Multi-Task Learning with Attention.  _arXiv e-prints_ , Article arXiv:1803.10704 (March 2018), arXiv:1803.10704 pages.  arXiv:1803.10704 [cs.CV]
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019.  Multi-Task Deep Neural Networks for Natural Language Understanding.  _arXiv e-prints_ , Article arXiv:1901.11504 (Jan. 2019), arXiv:1901.11504 pages.  arXiv:1901.11504 [cs.CL]
  * Liu (2019) Yang Liu. 2019.  Fine-tune BERT for extractive summarization.  _arXiv preprint arXiv:1903.10318_ (2019).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Lu et al. (2019a) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019a.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  _arXiv e-prints_ , Article arXiv:1908.02265 (Aug. 2019), arXiv:1908.02265 pages.  arXiv:1908.02265 [cs.CV]
  * Lu et al. (2019b) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2019b.  12-in-1: Multi-Task Vision and Language Representation Learning.  _arXiv e-prints_ , Article arXiv:1912.02315 (Dec. 2019), arXiv:1912.02315 pages.  arXiv:1912.02315 [cs.CV]
  * Nakkiran et al. (2019) Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2019.  Deep Double Descent: Where Bigger Models and More Data Hurt.  _arXiv e-prints_ , Article arXiv:1912.02292 (Dec. 2019), arXiv:1912.02292 pages.  arXiv:1912.02292 [cs.LG]
  * Nguyen and Okatani (2018) Duy-Kien Nguyen and Takayuki Okatani. 2018.  Multi-task Learning of Hierarchical Vision-Language Representation.  _arXiv e-prints_ , Article arXiv:1812.00500 (Dec. 2018), arXiv:1812.00500 pages.  arXiv:1812.00500 [cs.CV]
  * Peng and Sarazen (2019) Tony Peng and Michael Sarazen. 2019.  The Staggering Cost of Training SOTA AI Models.  https://syncedreview.com/2019/06/27/the-staggering-cost-of-training-sota-ai-models/
  * Pramanik et al. (2019) Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain. 2019.  OmniNet: A unified architecture for multi-modal multi-task learning.  _arXiv e-prints_ , Article arXiv:1907.07804 (July 2019), arXiv:1907.07804 pages.  arXiv:1907.07804 [cs.LG]
  * Qi et al. (2020) Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. 2020\.  ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data.  _arXiv e-prints_ , Article arXiv:2001.07966 (Jan. 2020), arXiv:2001.07966 pages.  arXiv:2001.07966 [cs.CV]
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Ruder (2017) Sebastian Ruder. 2017\.  An Overview of Multi-Task Learning in Deep Neural Networks.  _arXiv e-prints_ , Article arXiv:1706.05098 (June 2017), arXiv:1706.05098 pages.  arXiv:1706.05098 [cs.LG]
  * Sanh et al. (2018) Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2018\.  A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks.  _arXiv e-prints_ , Article arXiv:1811.06031 (Nov. 2018), arXiv:1811.06031 pages.  arXiv:1811.06031 [cs.CL]
  * Sharir et al. (2020) Or Sharir, Barak Peleg, and Yoav Shoham. 2020.  The Cost of Training NLP Models: A Concise Overview.  _arXiv preprint arXiv:2004.08900_ (2020).
  * Singh et al. (2020) Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. 2020\.  Are we pretraining it right? Digging deeper into visio-linguistic pretraining.  _arXiv e-prints_ , Article arXiv:2004.08744 (April 2020), arXiv:2004.08744 pages.  arXiv:2004.08744 [cs.CV]
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019\.  Energy and policy considerations for deep learning in NLP.  _arXiv preprint arXiv:1906.02243_ (2019).
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  _arXiv e-prints_ , Article arXiv:1908.08530 (Aug. 2019), arXiv:1908.08530 pages.  arXiv:1908.08530 [cs.CV]
  * Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018.  Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning.  _arXiv e-prints_ , Article arXiv:1804.00079 (March 2018), arXiv:1804.00079 pages.  arXiv:1804.00079 [cs.CL]
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019\.  VideoBERT: A Joint Model for Video and Language Representation Learning.  _arXiv e-prints_ , Article arXiv:1904.01766 (April 2019), arXiv:1904.01766 pages.  arXiv:1904.01766 [cs.CV]
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  _arXiv e-prints_ , Article arXiv:1908.07490 (Aug. 2019), arXiv:1908.07490 pages.  arXiv:1908.07490 [cs.CL]
  * Tan and Le (2019) Mingxing Tan and Quoc V. Le. 2019.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  _arXiv e-prints_ , Article arXiv:1905.11946 (May 2019), arXiv:1905.11946 pages.  arXiv:1905.11946 [cs.LG]
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _arXiv e-prints_ , Article arXiv:1706.03762 (June 2017), arXiv:1706.03762 pages.  arXiv:1706.03762 [cs.CL]
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  _arXiv e-prints_ , Article arXiv:1804.07461 (April 2018), arXiv:1804.07461 pages.  arXiv:1804.07461 [cs.CL]
  * Wang et al. (2020) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-Attention with Linear Complexity.  arXiv:2006.04768 [cs.LG]
  * Wang et al. (2020) Yuyan Wang, Zhe Zhao, Bo Dai, Christopher Fifty, Dong Lin, Lichan Hong, and Ed H. Chi. 2020.  Small Towers Make Big Differences.  _arXiv e-prints_ , Article arXiv:2008.05808 (Aug. 2020), arXiv:2008.05808 pages.  arXiv:2008.05808 [cs.LG]
  * Worsham and Kalita (2020) Joseph Worsham and Jugal Kalita. 2020.  Multi-task learning for natural language processing in the 2020s: where are we going?  _arXiv e-prints_ , Article arXiv:2007.16008 (July 2020), arXiv:2007.16008 pages.  arXiv:2007.16008 [cs.CL]
  * Wu et al. (2020) Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020.  Lite Transformer with Long-Short Range Attention.  _arXiv e-prints_ , Article arXiv:2004.11886 (April 2020), arXiv:2004.11886 pages.  arXiv:2004.11886 [cs.CL]
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020.  Gradient Surgery for Multi-Task Learning.  _arXiv e-prints_ , Article arXiv:2001.06782 (Jan. 2020), arXiv:2001.06782 pages.  arXiv:2001.06782 [cs.LG]
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019\.  ERNIE: Enhanced language representation with informative entities.  _arXiv preprint arXiv:1905.07129_ (2019).
  * Zhou et al. (2019) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2019\.  Unified Vision-Language Pre-Training for Image Captioning and VQA.  _arXiv e-prints_ , Article arXiv:1909.11059 (Sept. 2019), arXiv:1909.11059 pages.  arXiv:1909.11059 [cs.CV]
  * Zhou et al. (2020) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020\.  BERT Loses Patience: Fast and Robust Inference with Early Exit.  _arXiv e-prints_ , Article arXiv:2006.04152 (June 2020), arXiv:2006.04152 pages.  arXiv:2006.04152 [cs.CL]