# XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

Subhabrata Mukherjee Ahmed Hassan Awadallah Jianfeng Gao
Microsoft Research
{submukhe, hassanam<EMAIL_ADDRESS>
###### Abstract

While deep and large pre-trained models are the state-of-the-art for various
natural language processing tasks, their huge size poses significant
challenges for practical uses in resource constrained settings. Recent works
in knowledge distillation propose task-agnostic as well as task-specific
methods to compress these models, with task-specific ones often yielding
higher compression rate. In this work, we develop a new task-agnostic
distillation framework XtremeDistilTransformers that leverages the advantage
of task-specific methods for learning a small universal model that can be
applied to arbitrary tasks and languages. To this end, we study the
transferability of several source tasks, augmentation resources and model
architecture for distillation. We evaluate our model performance on multiple
tasks, including the General Language Understanding Evaluation (GLUE)
benchmark, SQuAD question answering dataset and a massive multi-lingual NER
dataset with 41 languages.

##  1 Introduction

Large-scale pre-trained models††Code available at:
https://github.com/microsoft/xtreme-distil-transformers
Task-agnostic checkpoints available at:
https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased
https://huggingface.co/microsoft/xtremedistil-l6-h384-uncased
https://huggingface.co/microsoft/xtremedistil-l12-h384-uncased have become the
standard starting point for various natural language processing tasks Devlin
et al. (2019a). Several NLP tasks have achieved significant progress utilizing
these pre-trained models reaching previously unattainable performance Clark et
al. (2020); Liu et al. (2019b). The size of these models have been also
steadily growing to hundreds of millions Devlin et al. (2019a); Yang et al.
(2019) to billions of parameters Raffel et al. (2019); Brown et al. (2020).

The huge size poses significant challenges for downstream applications in
terms of energy consumption and cost of inference Strubell et al. (2019). As
such, it could be a deterrent to using them in practice limiting their usage
in on-the edge scenarios and under constrained computational training or
inference budgets.

Several research directions have considered compressing large-scale models
including work on pruning Gordon et al. (2020), quantization Han et al. (2016)
and distillation Sanh (2019). Knowledge distillation, in particular, has shown
strong results in pre-trained transformer-based language model compression.
With knowledge distillation, we train a student network (with smaller
capacity) to mimic the full output distribution of the teacher network Hinton
et al. (2015). Knowledge distillation has been applied to pre-trained language
model compression in two different settings: (1) before task-specific fine
tuning (i.e. task-agnostic distillation) or (2) after task-specific fine
tuning (i.e. task-specific distillation).

Task-agnostic distillation Sanh (2019); Sun et al. (2019, 2020) has the
advantage that the model needs to be distilled only once and can be reused for
fine-tuning on multiple down-stream tasks. It also allows us to achieve
speedup in both fine-tuning and inference. On the other hand, task-specific
distillation Tang et al. (2019); Jiao et al. (2019); Mukherjee and Hassan
Awadallah (2020) has been shown to achieve significantly higher compression
rate and inference speedup Fu et al. (2020); Mukherjee and Hassan Awadallah
(2020).

In this work, we first study the transferability of pre-trained models across
several source tasks to select the optimal one for transfer. We then aim to
create universally distilled models that can be used with any downstream task
while leveraging the benefits of the techniques and augmentation resources
developed for the source transfer task. We show that distilled models that use
task-specific data transfer to varying degrees and their transferability
depends on choices of the source task, data augmentation strategy and
distillation techniques.

Contributions: More specifically, this work makes the following contributions:

(a) Studies the transferability of several source tasks and augmentation
resources for task-agnostic knowledge distillation.

(b) Develops a distillation framework to learn a massively compressed student
model leveraging deep hidden representations and attention states from
multiple layers of the teacher model with progressive knowledge transfer.

(c) Extensive experiments on several datasets in GLUE benchmark and for
massive multilingual NER demonstrate the effectiveness of task and language
transfer. Finally, we will release the task-agnostic checkpoints for the
distilled models.

##  2 Exploring Tasks for Transfer

###  2.1 Role of Tasks for Distillation

Task-specific distillation assumes the presence of human labeled data to fine-
tune the teacher for the underlying task and provide the student with
corresponding logits for learning. Such techniques have shown massive
compression (e.g., $7.5x$ compression in LRC-BERT Fu et al. (2020) and $35x$
compression in XtremeDistil Mukherjee and Hassan Awadallah (2020)) without
performance loss. An obvious disadvantage is the need to distil for each and
every task which is resource-intensive.

In contrast, task-agnostic methods rely on objectives like masked language
modeling (MLM) and representation transfer over unlabeled data. These do not
require human labels allowing them to learn from massive amounts of text. This
allows the model to retain general information applicable to arbitrary tasks,
but results in much less compression (e.g., $2x$111Considering model
checkpoints with the least performance loss with respect to the big pre-
trained language model. in MiniLM Wang et al. (2020), Sanh (2019) and TinyBERT
Jiao et al. (2019)).

This begs the question of whether we can use more specific tasks, other than
language modeling, that can harness human labels to provide task-specific
logits and representations, while also transferring well to arbitrary tasks.
This would allow us to leverage the relative strength of both of the above
families of techniques to obtain high compression rate of task-specific
methods as well as wide applicability of task-agnostic ones.

Table 1: Transfer performance on fine-tuning BERT on labeled data for several source tasks in the rows (e.g., MNLI), extracting the encoder (e.g., MNLI-BERT) and further fine-tuning on labeled data for several target tasks in the columns. We observe MNLI to obtain the best performance for task transfer on an average. | MRPC | MNLI | RTE | QQP | QNLI | SST-2 | SQuADv1 | Avg
---|---|---|---|---|---|---|---|---
#Labels | 3.7K | 393K | 2.5K | 364K | 108K | 67K | 87K | -
BERT | 83.8 | 84.4 | 66.8 | 91.2 | 91.4 | 92.2 | 88.3 | 85.4
MNLI-BERT | 88.2 | 84.2 | 79.1 | 91.1 | 91.1 | 93.6 | 87.2 | 87.8
QNLI-BERT | 87.0 | 84.8 | 73.3 | 91.0 | 91.6 | 93.0 | 88.1 | 87.0
SST2-BERT | 81.6 | 84.7 | 66.1 | 91.1 | 91.3 | 93.4 | 87.6 | 85.1
SQuADv1-BERT | 86.3 | 84.6 | 69.7 | 87.1 | 91.6 | 92.9 | 88.3 | 85.4

###  2.2 Transferability of Tasks

In order to leverage task-specific distillation techniques, we need to select
a source task that transfers well to other tasks such that: a model distilled
for the source task can obtain a good performance on fine-tuning with labeled
data from arbitrary target tasks. We perform the following analysis with the
pre-trained teacher model with the assumption that the benefits will be
transferred to the student model.

A recent work Chen et al. (2020a) studies the notion of task transferability
for BERT in the context of lottery ticket hypothesis Frankle and Carbin
(2019). Specifically, the authors study if subnetworks obtained for one task
obtained by network pruning transfer to other tasks and if there are universal
subnetworks that train well for many tasks.

The authors observe that while masked language modeling (MLM) is the most
universal task, there are other candidate tasks like natural language
inference and question-answering that allow us to transfer meaningful
representations to other tasks.

Task transfer. Consider a pre-trained neural network model (e.g., BERT)
$f(x;\theta)$ with encoder parameters $\theta\in\mathcal{R}^{d_{1}}$. Given a
source task $\mathcal{S}$ with ground-truth labeled data
$D_{\mathcal{S}}=\\{x,y\\}$, we first fine-tune the pre-trained model
$f(x;\theta,\gamma_{\mathcal{S}})$ by adding task-specific classification
parameters $\gamma_{\mathcal{S}}\in\mathcal{R}^{d_{2}}$. We now extract the
encoder $f(x;\theta_{\mathcal{S}})$ where the parameters
$\theta_{\mathcal{S}}\in\mathcal{R}^{d_{1}}$ have been adapted to the source
task ($\theta\rightarrow\theta_{\mathcal{S}}$). Now, given a target task
$\mathcal{T}$ with labeled data $\mathcal{D_{\mathcal{T}}}$, we further fine-
tune the encoder $f(x;\theta_{\mathcal{S}},\gamma_{\mathcal{T}})$ where
$\gamma_{\mathcal{T}}\in\mathcal{R}^{d_{3}}$ represents task-specific
parameters for the target task.

Selection criteria for the best source task. Given a set of source
$\mathcal{S}$ and target $\mathcal{T}$ tasks, consider
$eval(s\in\mathcal{S}\rightarrow t\in\mathcal{T})$ to be the performance of a
pre-trained language model that is adapted from $s$ to $t$, measured with some
evaluation metric (e.g., accuracy, F1). We define the best source transfer
task as
$argmax_{s\in\mathcal{S}}\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}eval(s\rightarrow
t)$ depicting the best transfer performance obtained on an average on
transferring a pre-trained model from the source to a set of different target
tasks.

While this definition simplifies the transfer problem ignoring task difficulty
(e.g., ternary MNLI is harder than binary SST), domain overlap (SQuAD and QNLI
are both question-answering datasets), task setup (e.g., span extraction in
SQuAD and pairwise-classification in MNLI) and variable amount of training
labels per task, we defer a more controlled study of this problem as future
work.

Candidate tasks for transfer. We consider a subset of source tasks from
lottery ticket hypothesis for BERT Chen et al. (2020a) for which transfer
performance is at least as high as same-task performance on atleast two target
tasks. We ignore MLM since the pre-trained encoder (BERT) is intrinsically
trained with MLM objective that provides no additional information in our
transfer setup.

Transfer evaluation. Table 1 shows the performance of pre-trained BERT-base
with task transfer, where each row depicts a source task $\mathcal{S}$ and
each column represents the target task $\mathcal{T}$. We observe that MNLI as
the source task, followed by QNLI, has the best performance on an average on
transferring to several target tasks, especially those with limited training
labels for fine-tuning. Similar improvements with MNLI for tasks like RTE
(textual entailment) and MRPC (paraphrase) have been reported in recent work
like RoBERTa Liu et al. (2019b). Therefore, we adopt MNLI as the source task
for transfer distillation and evaluate its effectiveness for several target
tasks and languages.

###  2.3 Transfer Set for Knowledge Distillation

Task-agnostic methods can learn from large unlabeled general-purpose text
using self-supervision objectives like MLM. Task-specific distillation, on the
other hand, rely on large-scale task-specific transfer data that is often
difficult to obtain for many tasks. Prior works show large-scale task-specific
transfer data to be instrumental in minimizing the performance gap of the
teacher and student Turc et al. (2019); Mukherjee and Hassan Awadallah (2020).
However, these works primarily explore instance-classification tasks like
sentiment classification (e.g., IMDB and SST2) or topic classification (e.g.,
AG News and Dbpedia) with readily available in-domain transfer data. For
example, sentiment classification in IMDB can benefit from large amounts of
unlabeled user reviews from the forum. However, this is difficult to obtain
for pair-wise classification tasks like NLI. Additionally, NLI being a ternary
classification task (entail / contradict / neutral) requires a transfer set
with a similar label distribution for effective transfer. To address these
issues, we explore techniques to automatically generate large-scale task-
specific transfer sets leveraging a very large bank of web sentences from
Common Crawl in Section 4.1.

##  3 Distillation Framework

Table 2: Contrasting XtremeDistilTransformers with state-of-the-art task-agnostic distilled models. XtremeDistilTransformers leverages embedding factorization, hidden representations and attention states of the teacher from multiple layers with progressive knowledge transfer for distillation while accommodating arbitrary student architecture and languages. | Embedding | Hidden | Attention | Multi- | Progressive | Student-arch. | Multi-
---|---|---|---|---|---|---|---
| Factorization | Representation | State | layer | Transfer | -agnostic | lingual
DistilBERT |  |  |  |  |  |  |
TinyBERT |  | ✓ | ✓ | ✓ |  | ✓ |
MiniLM |  | ✓ | ✓ |  |  | ✓ | ✓
MobileBERT | ✓ | ✓ | ✓ | ✓ | ✓ |  |
XtremeDistilTransformers | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

Overview. Given a pre-trained model fine-tuned on the source task as teacher,
our objective is to distil its knowledge in a compressed (both in terms of
width and depth) student. Given a wide teacher and a narrow student, we employ
embedding factorization to align their widths for knowledge transfer. Given a
deep teacher and a shallow student, we align all the layers of the student to
the topmost layers of the teacher. To this end, we transfer both the hidden
representations as well as attention states from multiple layers of the
teacher to the student with progressive knowledge transfer. The above
techniques in combination allow us to transfer knowledge from any teacher to
any student of arbitrary architecture. Finally, XtremeDistilTransformers
supports for both task and language transfer (refer to Section 4.4) in
contrast to many prior work. Table 2 contrasts XtremeDistilTransformers
against existing distillation techniques, namely, DistilBERT Sanh (2019),
TinyBERT Jiao et al. (2019), MiniLM Wang et al. (2020) and MobileBERT Sun et
al. (2020).

Input Representation. XtremeDistilTransformers uses the tokenizer and special
tokens as used in the teacher model. For instance, it uses Wordpiece
tokenization Wu et al. (2016) with a fixed vocabulary $\mathcal{V}$ (e.g.,
$30k$ tokens) for distilling BERT and adds special symbols “[CLS]” and “[SEP]”
to mark the beginning and end of a text sequence respectively.

Teacher model. Given pre-trained models with variable performance across
tasks, we want to choose the best teacher for the best source transfer task
(i.e. MNLI). We experiment with base and large versions of BERT Devlin et al.
(2019b) and Electra Clark et al. (2020) as teachers. Table 3 shows a
comparison of their performance and parameters on MNLI. Given the same
parameter complexity, we find Electra to be the best on MNLI.

Table 3: Performance of fine-tuning pre-trained teacher models of different sizes on the MNLI task. Model | Params (MM) | Accuracy
---|---|---
BERT-Base | 109 | 84.24
Electra-Base | 109 | 88.21
BERT-Large | 335 | 87.11
Electra-Large | 335 | 90.73

Student model. We compare the performance of state-of-the-art distilled models
in terms of parameters, compression and performance gap (after distillation)
with respect to the teacher (reported in Table 4). We observe MiniLM Wang et
al. (2020) to have the closest performance to the teacher BERT.
Correspondingly, we choose miniature versions of MiniLM (23 MM and 14 MM
parameters) as candidate students. We investigate different student
initialization strategies (including initialization with a task-agnostic
distilled model) and show that their performance can be improved further in
the XtremeDistilTransformers framework. We also study the trade-off between
different architectural aspects (parameters, layers, attention heads and
hidden dimension) against its performance.

In the following section, superscript $\mathcal{T}$ always represents the
teacher and $\mathcal{S}$ denotes the student.

Table 4: Comparing distilled models from prior work based on average GLUE score, parameters (MM) and performance gap with respect to the teacher. Models | GLUE | Params | %Gap
---|---|---|---
BERT-Base | 81.5 | 109 | -
DistilBERT | 75.2 | 66 | 7.73
BERT-Truncated | 76.2 | 66 | 6.50
TinyBERT | 79.1 | 66 | 2.94
MiniLM | 80.4 | 66 | 1.35

###  3.1 Word Embedding Factorization

Our student and teacher model consist of the word embedding layer with
embedding matrices
$\mathcal{W}^{\mathcal{S}}\in\mathcal{R}^{|\mathcal{V}|\times
d^{\mathcal{S}}}$ and
$\mathcal{W}^{\mathcal{T}}\in\mathcal{R}^{|\mathcal{V}|\times
d^{\mathcal{T}}}$, where $d^{\mathcal{S}}<d^{\mathcal{T}}$ depicting a thin
student and a wide teacher.

A large number of parameters reside in the word embeddings of pre-trained
models. For instance, multilingual BERT with WordPiece vocabulary of $V=110K$
tokens and embedding dimension of $D=768$ contains $92MM$ word embedding
parameters. We use a dimensionality reduction algorithm, namely, Singular
Value Decomposition (SVD) to project the teacher word embeddings of dimension
$\mathcal{R}^{|\mathcal{V}|\times d^{\mathcal{T}}}$ to a lower dimensional
space $\mathcal{R}^{|\mathcal{V}|\times d^{\mathcal{S}}}$. Given the teacher
word embedding matrix of dimension $\mathcal{R}^{|\mathcal{V}|\times
d^{\mathcal{T}}}$, SVD finds the best $d^{\mathcal{S}}$-dimensional
representation that minimizes sum of squares of the projections (of rows) to
the subspace.

###  3.2 Hidden Layer Representations

The student and teacher models consist of $L^{\mathcal{S}}$ and
$L^{\mathcal{T}}$ repeated transformer blocks, where
$L^{\mathcal{S}}<L^{\mathcal{T}}$. Considering an input sequence of $n$ tokens
$x=\\{x_{1},x_{2},\cdots x_{n}\\}$, the token embedding ${W}$ is added to the
position ${PE}$ and segment ${SE}$ embeddings as
$z_{i}(x_{i})=W(x_{i})+PE(i)+SE(i)$. The input to the network is given by
$\mathcal{H}^{0}=[z_{1},z_{2},\cdots z_{|x|}]$. In case of the student, the
token embedding is obtained from the SVD-decomposed token embedding of the
teacher model as $\mathcal{W}^{\mathcal{S}}$, whereas the position and segment
embeddings are learnable embeddings of dimension $d^{\mathcal{S}}$.
Transformer blocks repeatedly compute hidden state representations from the
output of the previous layer, where hidden states from the $l$th layer of the
teacher and student are given by,

$\displaystyle\mathcal{H}^{\mathcal{T}}_{l}$
$\displaystyle=Transformer^{\mathcal{T}}_{l}(\mathcal{H}^{\mathcal{T}}_{l-1}),l\in[1,\cdots\mathcal{L}^{\mathcal{T}}]$
(1) $\displaystyle\mathcal{H}^{\mathcal{T}}_{l}$
$\displaystyle=[{h}^{\mathcal{T}}_{l,1},{h}^{\mathcal{T}}_{l,2},\cdots,{h}^{\mathcal{T}}_{l,|x|}],l\in[1,2,\cdots\mathcal{L}^{\mathcal{T}}]$
(2) $\displaystyle\mathcal{H}^{\mathcal{S}}_{l}$
$\displaystyle=Transformer^{\mathcal{S}}_{l}(\mathcal{H}^{\mathcal{S}}_{l-1}),l\in[1,\cdots\mathcal{L}^{\mathcal{S}}]$
(3) $\displaystyle\mathcal{H}^{\mathcal{S}}_{l}$
$\displaystyle=[{h}^{\mathcal{S}}_{l,1},{h}^{\mathcal{S}}_{l,2},\cdots,{h}^{\mathcal{S}}_{l,|x|}],l\in[1,2,\cdots\mathcal{L}^{\mathcal{S}}]$
(4)

###  3.3 Multi-head Self-attention

Transformers view the input representation as a set of key-value pairs
$\\{\mathcal{K},\mathcal{V}\\}$ of dimension same as input sequence length
$|x|$. Each of the key and values are obtained from hidden state
representations of the encoder. Transformers compute the weighted sum of the
values, where the weight for each value is obtained by dot-product of the
query with the key values as
$Attention(\mathcal{Q},\mathcal{K},\mathcal{V})=softmax(\frac{\mathcal{Q}\mathcal{K}^{T}}{\sqrt{n}})\mathcal{V}$.
In the context of multi-head attention with several attention heads, the above
is computed as follows. Consider
$\mathcal{A}_{l,a},a\in[1,2,\cdots\mathcal{AH}]$, where $\mathcal{AH}$ is the
number of attention heads of the teacher and student. Consider the query, key
and values obtained by
$\mathcal{Q}_{l,a}=\mathcal{H}_{l-1}\mathcal{W}^{\mathcal{Q}}_{l,a}$,
$\mathcal{K}_{l,a}=\mathcal{H}_{l-1}\mathcal{W}^{k}_{l,a}$,
$\mathcal{V}_{l,a}=\mathcal{H}_{l-1}\mathcal{W}^{V}_{l,a}$.

Each multi-head attention state of dimension $|x|\times|x|$ from the $l$th
layer is given by:

$\mathcal{A}_{l,a}(\mathcal{Q}_{l,a},\mathcal{K}_{l,a},\mathcal{V}_{l,a})=softmax(\frac{\mathcal{Q}_{l,a}\mathcal{K}_{l,a}^{T}}{\sqrt{n}})\mathcal{V}_{l,a}$
(5)

Since our teacher and student are both transformers with similar multi-head
attention mechanism, we obtain the corresponding attention states:

$\displaystyle\mathcal{A}^{\mathcal{T}}_{l,a},$ $\displaystyle\
l\in[1,2,\cdots\mathcal{L}^{\mathcal{T}}],a\in[1,2,\cdots\mathcal{AH}]$ (6)
$\displaystyle\mathcal{A}^{\mathcal{S}}_{l,a},$ $\displaystyle\
l\in[1,2,\cdots\mathcal{L}^{\mathcal{S}}],a\in[1,2,\cdots\mathcal{AH}]$ (7)

###  3.4 Multi-task Multi-layer Distillation

Multi-layer hidden state transfer. We leverage deep representations from
multiple layers of the teacher that capture different forms of features to aid
the student in learning. In order to align multiple layers of the student to
those of the teacher as a form of mimic learning, we train the student with
the following multi-layer representation loss objective. Given a deep teacher
with $\mathcal{L}^{\mathcal{T}}$ layers and a shallow student with
$\mathcal{L}^{\mathcal{S}}$ layers, where
$\mathcal{L}^{\mathcal{T}}>\mathcal{L}^{\mathcal{S}}$, we align the last
$\mathcal{L}^{\mathcal{S}}$ layers of each model. Given a wide teacher and
narrow student with corresponding dimensions
$d^{\mathcal{T}}>d^{\mathcal{S}}$, we perform a linear transformation to
upscale and align the corresponding hidden state representations of the
student such that
$\widetilde{\mathcal{H}}^{\mathcal{S}}(x)=W^{f}\cdot\mathcal{H}^{\mathcal{S}}(x)+b^{f}$,
where $W^{f}\in R^{d^{\mathcal{T}}\times d^{\mathcal{S}}}$ is the
transformation matrix, $b^{f}\in R^{d^{\mathcal{T}}}$ is the bias.

$layer_{loss}=-\sum_{\begin{subarray}{c}l=1\\\
l^{\prime}=(\mathcal{L}^{\mathcal{T}}-\mathcal{L}^{\mathcal{S}})\end{subarray}}^{\begin{subarray}{c}l^{\prime}=\mathcal{L}^{\mathcal{T}}\\\
l=\mathcal{L}^{\mathcal{S}}\end{subarray}}\sum_{i=1}^{|x|}\frac{||\widetilde{\mathcal{H}_{l}}^{\mathcal{S}}(x_{i})-\mathcal{H}_{l^{\prime}}^{\mathcal{T}}(x_{i})||^{2}}{2\cdot\mathcal{L}^{\mathcal{S}}\cdot|x|}$
(8)

Multi-layer attention transfer. We also leverage the self-attention signals
from the different teacher layers to guide the student. Similar to previous
loss objective, we align the attention states of the last
$\mathcal{L}^{\mathcal{S}}$ layers of the teacher and student from multiple
attention heads with the following objective:

$attn_{loss}=-\sum_{\begin{subarray}{c}l=1\\\
l^{\prime}={(\mathcal{L}^{\mathcal{T}}-\mathcal{L}^{\mathcal{S}})}\end{subarray}}^{\begin{subarray}{c}l^{\prime}=\mathcal{L}^{\mathcal{T}}\\\
l=\mathcal{L}^{\mathcal{S}}\end{subarray}}\sum_{a=1}^{|\mathcal{AH}|}\sum_{i=1}^{|x|}\frac{||\mathcal{A}_{l,a}^{\mathcal{S}}(x_{i})-\mathcal{A}_{{l^{\prime}},a}^{\mathcal{T}}(x_{i})||^{2}}{2\cdot\mathcal{L}^{\mathcal{S}}\cdot|\mathcal{AH}|\cdot|x|}$
(9)

Task-specific logit transfer. Given hidden state representations from last
layer $\mathcal{L}^{\mathcal{S}}$ and $\mathcal{L}^{\mathcal{T}}$ of the
student and teacher, we can obtain the task-specific logits for source
transfer task (e.g., MNLI) from:

$\displaystyle z^{\mathcal{S}}(x)$
$\displaystyle=\mathcal{H}_{\mathcal{L}^{\mathcal{S}}}^{\mathcal{S}}(x)\cdot
W^{\mathcal{S}}$ (10) $\displaystyle z^{\mathcal{T}}(x)$
$\displaystyle=\mathcal{H}_{\mathcal{L}^{\mathcal{T}}}^{\mathcal{T}}(x)\cdot
W^{\mathcal{T}}$ (11)

where $W^{\mathcal{S}}\in R^{d^{\mathcal{S}}\times C}$, $W^{\mathcal{T}}\in
R^{d^{\mathcal{T}}\times C}$, and $C$ is the number of classes. The prior
computations of multi-layer hidden state and attention loss are performed over
large amounts of unlabeled transfer data from the source task. To explicitly
adapt these models to the source task, we leverage some amount of source-task-
specific labeled data to align the logits of the teacher and student. To this
end, we minimize the following task-specific logit loss:

$\displaystyle
logit_{loss}=-\frac{1}{2}||z^{\mathcal{S}}(x)-z^{\mathcal{T}}(x)||^{2}$ (12)

Finally, we fine-tune the student on task-specific labeled data with the
cross-entropy loss:

$ce_{loss}=-\sum_{c=1}^{C}\mathcal{I}(x,c)\ log\
softmax(z_{c}^{\mathcal{S}}(x))$ (13)

where $\mathcal{I}(x,c)$ is a binary indicator ($0$ or $1$) if class label $c$
is the correct classification for $x$ and $z_{c}^{\mathcal{S}}(x)$ is the
predicted logit corresponding to class $c$.

Progressive knowledge transfer. Multi-layer joint optimization of the above
loss functions bears the risk of error propagation from lower layers impacting
the knowledge transfer from upper layers. Recent works Sun et al. (2020);
Mukherjee and Hassan Awadallah (2020) demonstrate the benefit of progressive
knowledge transfer by gradual freezing and unfreezing of neural network layers
for mitigation. We adopt a similar principle in our work with the training
recipe in Algorithm 1.

Instead of jointly optimizing all the loss functions, we first minimize the
multi-layer representation and attention loss to align the last
$\mathcal{L}^{\mathcal{S}}$ layers of the teacher and student. Then we freeze
the student encoder, learn task-specific parameters by optimizing task-
specific logit loss and cross-entropy loss. For any loss function, we freeze
the parameters learned from the previous stage, learn new parameters (e.g.,
softmax for task-specific loss) introduced by a new loss function, and finally
perform end-to-end fine-tuning based on cross-entropy loss. Error propagation
from lower layers is mitigated by freezing lower part of the network while
learning additional task-specific parameters.

Input: (a) Transformer teacher model (e.g., BERT) fine-tuned on hard labels on
task $\mathcal{S}$ (e.g., MNLI). (b) Initial pre-trained student model (e.g.,
MiniLM)
1\. Optimize student params
$\theta=\\{\mathcal{H}_{l}^{\mathcal{S}}\\}_{l=1}^{\mathcal{L}^{\mathcal{S}}},\\{\mathcal{A}_{l,a}^{\mathcal{S}}\\}_{\begin{subarray}{c}l=1\cdots\mathcal{L}^{\mathcal{S}}\\\
a=1\cdots\mathcal{AH}\end{subarray}}$ optimizing losses in Eqn. 8 and 9
2\. Freeze $\theta$ and optimize task-specific parameters $W^{\mathcal{S}}$
optimizing logit loss in Eqn. 12 with soft labels
3\. Update $\theta$ and $W^{\mathcal{S}}$ optimizing logit loss in Eqn. 12
with soft labels
4\. Freeze $\theta$ and optimize task-specific parameters $W^{\mathcal{S}}$
optimizing cross-entropy loss in Eqn. 13 with hard labels
5\. Update $\theta$ and $W^{\mathcal{S}}$ optimizing cross-entropy loss in
Eqn. 13 with hard labels

Algorithm 1 Progressive knowledge transfer.

##  4 Experiments

We first explore several augmentation resources for knowledge transfer. Then
we compare distillation performance and compression rate of
XtremeDistilTransformers with existing models on GLUE Wang et al. (2018),
SQuAD Rajpurkar et al. (2016) and massive multilingual NER on 41 languages in
WikiAnn Pan et al. (2017). All experiments are performed on $4$ Tesla V-$100$
GPUs.

Table 5: Performance of XtremeDistilTransformers on MNLI with different augmentation resources as unlabeled transfer set with Electra-base-discriminator teacher and pre-initialized MiniLM student (parameters in millions). Model | Params | Aug. Data | #Samples | Acc
---|---|---|---|---
Electra | 109 | - | - | 88.12
Xtreme | 22 | MNLI | 392K | 82.56
DistilTransf. |  | SNLI | 550K | 82.57
|  | PAWS | 695K | 82.58
|  | ParaNMT | 5.4MM | 83.70
|  | SentAug | 4.3MM | 84.52
Table 6: Comparing the performance of distilled models DistilBERT Sanh (2019), TinyBERT Jiao et al. (2019), MiniLM Wang et al. (2020) and XtremeDistilTransformers on the development set for several GLUE tasks. R denotes reported published results and HF denotes the performance obtained with our HuggingFace implementations. Models | Params | Speedup | MNLI | QNLI | QQP | RTE | SST | MRPC | SQuADv2 | Avg
---|---|---|---|---|---|---|---|---|---|---
BERT (R) | 109 | 1x | 84.5 | 91.7 | 91.3 | 68.6 | 93.2 | 87.3 | 76.8 | 84.8
BERT-Trun (R) | 66 | 2x | 81.2 | 87.9 | 90.4 | 65.5 | 90.8 | 82.7 | 69.9 | 81.2
DistilBERT (R) | 66 | 2x | 82.2 | 89.2 | 88.5 | 59.9 | 91.3 | 87.5 | 70.7 | 81.3
TinyBERT (R) | 66 | 2x | 83.5 | 90.5 | 90.6 | 72.2 | 91.6 | 88.4 | 73.1 | 84.3
MiniLM (R) | 66 | 2x | 84.0 | 91.0 | 91.0 | 71.5 | 92.0 | 88.4 | 76.4 | 84.9
MiniLM (R) | 22 | 5.3x | 82.8 | 90.3 | 90.6 | 68.9 | 91.3 | 86.6 | 72.9 | 83.3
BERT (HF) | 109 | 1x | 84.4 | 91.4 | 91.2 | 66.8 | 93.2 | 83.8 | 74.8 | 83.7
MiniLM (HF) | 22 | 5.3x | 82.7 | 89.4 | 90.3 | 64.3 | 90.8 | 84.1 | 71.5 | 81.9
XtremeDistilTransf. (HF) | 22 | 5.3x | 84.5 | 90.2 | 90.4 | 77.3 | 91.6 | 89.0 | 74.4 | 85.3
XtremeDistilTransf. (HF) | 14 | 9.4x | 81.8 | 86.9 | 89.5 | 74.4 | 89.9 | 86.5 | 63.0 | 81.7

###  4.1 Unlabeled Augmentation Data

Given the best source transfer task as MNLI, we choose the best teacher from
Table 3 as Electra and the best student from Table 4 as MiniLM for
initializing the student model in XtremeDistilTransformers . We explore the
following augmentation resources;

(a) MNLI Williams et al. (2018): We use training data as the transfer set,
ignoring the labels.

(b) SNLI Bowman et al. (2015) is similar to MNLI with human-written English
sentence-pairs categorized in three classes (entail / contradict / neutral).

(c) PAWS Zhang et al. (2019): This contains human labeled pairs generated from
both word swapping and back translation which feature the importance of
modeling structure, context, and word order for identifying paraphrases.

(d) ParaNMT Wieting and Gimpel (2018) consists of a large number of English-
English sentential paraphrase pairs automatically generated using Neural
Machine Translation to translate the Czech side of a large Czech-English
parallel corpus.

(e) SentAugment Du et al. (2020) uses state-of-the-art sentence embeddings to
encode the information in a very large bank of sentences from CommonCrawl
which can thereafter be used to retrieve in-domain unannotated sentences for
any language understanding task. We leverage the SentAugment Sentence Encoder
(SASE) with sentencepiece tokenization to encode all the sentences in our MNLI
training corpus. Given a pair of NLI sentences $(s_{1},s_{2})$, we leverage
existing code222https://github.com/facebookresearch/SentAugment for fast
K-nearest neighbor search ($K=10$) to retrieve similar sentences for
${\\{s_{1}^{i}\\}}_{i=1}^{K}$ and ${\\{s_{2}^{j}\\}}_{j=1}^{K}$ from pre-
trained FAISS333https://github.com/facebookresearch/faiss indexes built over
the CommonCrawl data. The unlabeled transfer set now consists of all the
nearest neighbor pairs $\\{s_{1}^{i},s_{2}^{j}\\}$.

Table 5 shows the distillation performance with different augmentation
resources as unlabeled transfer set. We observe SentAug to perform the best
for the MNLI task for two reasons.

(a) Amount of transfer data: Given paraphrasical tasks like PAWS and ParaNMT,
we observe ParaNMT to perform better given large-scale transfer data.
Similarly, among textual entailment tasks MNLI, SNLI and SentAug (derived from
MNLI), SentAug performs better given its scale.

(b) Label distribution of source and transfer data. PAWS and ParaNMT being
paraphrasical tasks work with binary labels (e.g., paraphrases or not);
whereas, NLI tasks work with ternary labels (e.g., entail, contradict, or
neutral).

Table 7: Performance comparison on distilling XtremeDistilTransformers of varying capacity with teacher models of different sizes. #TD. and #Tea. denote XtremeDistilTransformers and teacher model parameters in millions. #TD. | Teacher | #Tea. | Speedup | MRPC | MNLI | RTE | QQP | QNLI | SST-2 | Avg
---|---|---|---|---|---|---|---|---|---|---
22 | Electra Base | 109 | 5.3x | 89.0 | 84.5 | 77.3 | 90.4 | 90.2 | 91.6 | 87.2
22 | BERT Base | 109 | 5.3x | 88.7 | 84.2 | 75.5 | 90.5 | 90.4 | 92.3 | 86.9
22 | Electra Large | 335 | 10.6x | 87.8 | 83.8 | 77.6 | 90.5 | 89.8 | 91.2 | 86.8
22 | BERT Large | 335 | 10.6x | 86.5 | 83.3 | 76.9 | 90.5 | 89.7 | 91.1 | 86.3
14 | Electra Base | 109 | 9.4x | 86.5 | 81.8 | 74.4 | 89.5 | 86.9 | 89.9 | 84.8

###  4.2 Distillation Performance in GLUE

Given a model distilled from MNLI, we extract the encoder
$f(x;\theta_{\mathcal{S}})$, add task-specific parameters
$\gamma_{\mathcal{T}}$ and fine-tune
$f(x;\theta_{\mathcal{S}},\gamma_{\mathcal{T}})$ on labeled data for several
tasks with results in Table 6. We report results from both published works and
our results built on top of HuggingFace (HF) Wolf et al. (2020).

We observe XtremeDistilTransformers to obtain the best performance on an
average closely followed by MiniLM and TinyBERT. Since the performance of pre-
trained models vary with careful calibrations, in our implementations with HF
default hyper-parameters (marked as HF in Table 6), we observe
XtremeDistilTransformers initialized with MiniLM to outperform MiniLM by
$4.3\%$ given same hyper-parameters and random seeds, and outperform BERT by
$2\%$ with $5.3x$ inference speedup. In a similar setup, the most compressed
version of XtremeDistilTransformers with $14MM$ parameters performs within
$2.4\%$ of BERT with $9.4x$ inference speedup.

###  4.3 What is more important for distillation, bigger or better teacher?

Table 7 shows the performance of XtremeDistilTransformers distilled from
teachers of different sizes and pre-training schemes. Refer to Table 3 for
teacher performances on MNLI. We observe that a better teacher (e.g., Electra-
base $>$ BERT-base, and Electra-large $>$ BERT-large) leads to a better
student. However, we also observe teacher model complexity to play a
significant role in distillation. For instance, although BERT-large is better
than Electra-base, we observe a slight degradation in distillation performance
when distilled from BERT-large with $3x$ parameters compared to Electra-base —
given the student model of same capacity. We conjecture this to be an artifact
of model capacity as it becomes increasingly difficult for a shallow student
to mimic a much bigger and deeper teacher.

###  4.4 Transfer Distillation for Massive Multilingual Named Entity
Recognition

We experiment with XtremeDistilTransformers distilled from a monolingual task
(MNLI-English) and adapt it for the multilingual setting to perform joint
named entity recognition (NER) on $41$ languages.

Consider XtremeDistilTransformers encoder with hidden layers
$\\{\mathcal{H}_{l}\\}\in\mathcal{R}^{|x|\times d}$, attention states
$\\{\mathcal{A}_{l,a}\\}\in\mathcal{R}^{|x|\times|x|}$ and word embeddings
$\mathcal{W}\in\mathcal{R}^{|\mathcal{V}|\times d}$, where $|x|,d$ and
$|\mathcal{V}|$ denote sequence length, embedding dimension and vocabulary
size. The only factor dependent on the vocabulary size is the word embedding
matrix $\mathcal{W}$. In principle, we can retain learned hidden layers
$\\{\mathcal{H}_{l}\\}$ and attention states $\\{\mathcal{A}_{l,a}\\}$, and
only adapt word embeddings to transfer to other languages with different
vocabulary.

To this end, we leverage word embeddings from multilingual BERT for target
adaptation. Specifically, we use word embedding factorization using Singular
Value Decomposition (SVD) (as outlined in Section 3.1) to project
$\mathcal{R}^{|\mathcal{V}|\times
d^{\mathcal{T}}}\rightarrow\mathcal{R}^{|\mathcal{V}|\times d^{\mathcal{S}}}$
from the mBERT word embedding space to that of XtremeDistilTransformers ,
where $d^{\mathcal{T}}$ and $d^{\mathcal{S}}$ represent the embedding
dimension of the teacher and student.

Now, we switch word embedding parameters in XtremeDistilTransformers
(distilled from English) with the SVD-decomposed mBERT word embeddings while
retaining prior encoder parameters
($\\{\mathcal{H}_{l}\\},\\{\mathcal{A}_{l,a}\\}$) and further distil it on the
multilingual WikiAnn data from $41$ languages in WikiAnn Pan et al. (2017).
Table 8 compares the performance of XtremeDistilTransformers against
multilingual models MMNER Rahimi et al. (2019) and XtremeDistil Mukherjee and
Hassan Awadallah (2020). We observe the most compressed version of
XtremeDistilTransformers with $14$ million parameters to obtain a similar
performance to XtremeDistil but with $2x$ additional compression and within
$4\%$ F1 of mBERT. Note that, in contrast to XtremeDistil, we transfer encoder
parameters from a monolingual distilled model. With progressive knowledge
transfer, we further freeze the word embedding parameters and fine-tune the
encoder parameters on downstream task.

Table 8: Massive multilingual NER average F1 scores across $41$ languages. XtremeDistilTransformers is distilled from MNLI-EN and adapted for multilingual NER. #Compres. denotes compression factor with respect to mBert and Params is number of parameters in millions. Models | Params | #Compres. | F1
---|---|---|---
mBERT-Single | 109*41 | - | 90.76
mBERT | 179 | 1 | 91.86
MMNER | 28*41 | 0.2 | 89.20
XtremeDistil | 28 | 6.4 | 88.64
XtremeDistilTransf. | 22 | 8.1 | 89.38
XtremeDistilTransf. | 14 | 12.8 | 88.18

###  4.5 Ablation Study

Table 9 shows ablation results on removing different components from
XtremeDistilTransformers for multilingual NER. We observe performance
degradation on removing multi-layer attention and hidden state losses from
distillation objective (a and b). When we remove both multi-layer components
in (b), we use hidden-states from only last layer of the teacher. This also
demonstrates the benefit of multi-layer distillation. We observe significant
degradation without embedding factorization using SVD (c) i.e. the student
uses monolingual (English) word embeddings and vocabulary. Without progressive
transfer, and fine-tuning model end-to-end result in some degradation (d).
Finally, we observe that distilling a student model from scratch (i.e.
randomly initialized) without transferring multilingual word embeddings or
encoder parameters (e) result in significant performance loss, thereby,
demonstrating the benefit of transfer distillation.

Table 1 in Appendix shows the variation in performance of
XtremeDistilTransformers with different architecture (number of attention
heads, hidden layers and embedding dimension), compression and performance gap
against multilingual BERT — with the smallest version obtaining $87x$ encoder
compression (or, $1$ million encoder parameters) with $9\%$ F1 gap against
mBERT for NER on $41$ languages.

Table 9: Ablation of XtremeDistilTransformers ($22$MM params) on WikiAnn for NER on $41$ languages. Distillation Features | F1
---|---
All: w/ multi-layer attn. & hidden state, |
embed. factor. w/ freezing | 89.38
(a) w/o multi-layer attn. | 87.86
(b) w/o multi-layer attn., w/o hidden state | 87.61
(c) w/o embed. factor. (monoling. vocab.) | 78.90
(d) w/ embed. factor & w/o freezing | 87.76
(e) init. from scratch | 83.40

##  5 Related Work

Distillation. Prior works on task-specific distillation (Liu et al., 2019a;
Zhu et al., 2019; Tang et al., 2019; Turc et al., 2019) leverage soft logits
from teachers for distilling students. Sun et al. (2019); Sanh (2019); Aguilar
et al. (2019) leverage teacher representations as additional signals. These
methods are often constrained by embedding dimension, width and depth of
models. Some recent works leverage embedding Sun et al. (2020) and shared word
Zhao et al. (2019) projection to address these limitations. Task-agnostic
methods like Jiao et al. (2019); Wang et al. (2020); Sun et al. (2020)
leverage hidden states and attention states from teachers but not task-
specific logits (refer to Table 2 for a contrast). Finally, another line of
work in model compression use quantization (Gong et al., 2014), low-precision
training and network pruning (Han et al., 2016) to reduce the memory
footprint.

Augmentation. Unsupervised contrastive learning techniques like SimCLR Chen et
al. (2020b, c) leverage semantic equivalence of images to train models to
differentiate between images and perturbed versions while obtaining parity
with fully-supervised models. Similarly UDA Xie et al. (2019) leverages
consistency learning between texts and backtranslations to improve few-shot
text classification. Finally, self-training and pre-training with SentAugment
Du et al. (2020) improves text classification with task-specific augmentation.

##  6 Conclusions

We develop a novel distillation framework XtremeDistilTransformers to leverage
the advantages of task-specific distillation for high compression as well as
wide applicability of task-agnostic ones. We study transferability of tasks
for pre-trained models and demonstrate NLI to be a great source task to obtain
a better teacher that transfers well across several tasks. This, in turn, is
used to distil a better student obtaining significant improvements over state-
of-the-art task-agnostic distilled models over several tasks. Finally, we
demonstrate techniques to obtain large-scale task-specific augmentation data
from the web to facilitate this knowledge transfer.

##  7 Appendix

Hyper-parameters. XtremeDistilTransformers is built over HuggingFace with most
of the default hyper-parameters. Please refer to the ReadMe for the attached
code for details.

Table 10: Variation in performance of XtremeDistilTransformers with different architecture (in terms of number of hidden layers (#Layer), attention heads (#Attn) and hidden dimension (#Dim) and parameter compression (#Enc Compress.) with respect to mBERT and corresponding performance gap (F1 Gap%) for massive multilingual NER on 41 languages. #Enc. and #Word Emb. denote the number of encoder and word embedding parameters (in millions, rounded to nearest integer). # Layer | # Attn | # Dim | F1 | #Enc. | #Word | #Enc | F1
---|---|---|---|---|---|---|---
|  |  |  |  | Emb. | Compres. | Gap(%)
2 | 2 | 128 | 82.05 | 1 | 15 | 87 | 11.49
6 | 12 | 192 | 83.29 | 8 | 23 | 11 | 10.15
6 | 12 | 216 | 83.40 | 9 | 26 | 10 | 10.03
2 | 4 | 256 | 84.30 | 2 | 31 | 44 | 9.06
4 | 2 | 128 | 84.43 | 1 | 15 | 87 | 8.92
6 | 2 | 128 | 85.44 | 2 | 15 | 44 | 7.83
4 | 4 | 256 | 86.26 | 3 | 31 | 29 | 6.95
4 | 12 | 312 | 86.69 | 5 | 37 | 17 | 6.48
4 | 12 | 312 | 86.74 | 5 | 37 | 17 | 6.43
6 | 4 | 256 | 86.82 | 5 | 31 | 17 | 6.34
6 | 12 | 384 | 88.00 | 11 | 46 | 8 | 5.07
12 | 12 | 768 | 92.70 | 87 | 92 | 1 | 0

## References

  * Aguilar et al. (2019) Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Edward Guo. 2019\.  Knowledge distillation from internal representations.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chen et al. (2020a) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020a.  The lottery ticket hypothesis for pre-trained bert networks.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1597–1607. PMLR.
  * Chen et al. (2020c) Xiuyi Chen, Fandong Meng, Peng Li, Feilong Chen, Shuang Xu, Bo Xu, and Jie Zhou. 2020c.  Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3426–3437, Online. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2020) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. 2020.  Self-training improves pre-training for natural language understanding.
  * Frankle and Carbin (2019) Jonathan Frankle and Michael Carbin. 2019.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _ICLR_. OpenReview.net.
  * Fu et al. (2020) Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. 2020.  Lrc-bert: Latent-representation contrastive knowledge distillation for natural language understanding.
  * Gong et al. (2014) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014.  Compressing deep convolutional networks using vector quantization.  _CoRR_ , abs/1412.6115.
  * Gordon et al. (2020) Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020.  Compressing bert: Studying the effects of weight pruning on transfer learning.
  * Han et al. (2016) Song Han, Huizi Mao, and William J. Dally. 2016.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _ICLR_.
  * Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  _CoRR_ , abs/1503.02531.
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019.  Tinybert: Distilling bert for natural language understanding.
  * Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a.  Improving multi-task deep neural networks via knowledge distillation for natural language understanding.  _CoRR_ , abs/1904.09482.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mukherjee and Hassan Awadallah (2020) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020.  XtremeDistil: Multi-stage distillation for massive multilingual models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2221–2234, Online. Association for Computational Linguistics.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sanh (2019) Victor Sanh. 2019.  Introducing distilbert, a distilled version of bert.  https://medium.com/huggingface/distilbert-8cf3380435b5.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for bert model compression.
  * Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020\.  MobileBERT: a compact task-agnostic BERT for resource-limited devices.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2158–2170, Online. Association for Computational Linguistics.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling task-specific knowledge from BERT into simple neural networks.  _CoRR_ , abs/1903.12136.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: On the importance of pre-training compact models.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
  * Wieting and Gimpel (2018) John Wieting and Kevin Gimpel. 2018.  ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 451–462, Melbourne, Australia. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, and Quoc V. Le et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _CoRR_ , abs/1609.08144.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2019.  Unsupervised data augmentation for consistency training.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _CoRR_ , abs/1906.08237.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhao et al. (2019) Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019.  Extreme language model compression with optimal subwords and shared projections.
  * Zhu et al. (2019) Wei Zhu, Xiaofeng Zhou, Keqiang Wang, Xun Luo, Xiepeng Li, Yuan Ni, and Guotong Xie. 2019.  PANLP at MEDIQA 2019: Pre-trained language models, transfer learning and knowledge distillation.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 380–388, Florence, Italy. Association for Computational Linguistics.