# SAS: Self-Augmentation Strategy for Language Model Pre-training

Yifei Xu1, Jingqiao Zhang2, Ru He2, Liangzhu Ge2, Chao Yang2, Cheng Yang2,
Ying Nian Wu1 Corresponding author: Cheng Yang.

###### Abstract

The core of self-supervised learning for pre-training language models includes
pre-training task design as well as appropriate data augmentation. Most data
augmentations in language model pre-training are context-independent. A
seminal contextualized augmentation was recently proposed in ELECTRA and
achieved state-of-the-art performance by introducing an auxiliary generation
network (generator) to produce contextualized data augmentation for the
training of a main discrimination network (discriminator). This design,
however, introduces extra computation cost of the generator and a need to
adjust the relative capability between the generator and the discriminator. In
this paper, we propose a self-augmentation strategy (SAS) where a single
network is utilized for both regular pre-training and contextualized data
augmentation for the training in later epochs. Essentially, this strategy
eliminates a separate generator and uses the single network to jointly conduct
two pre-training tasks with MLM (Masked Language Modeling) and RTD (Replaced
Token Detection) heads. It avoids the challenge to search for an appropriate
size of the generator, which is critical to the performance as evidenced in
ELECTRA and its subsequent variant models. In addition, SAS is a general
strategy that can be seamlessly combined with many new techniques emerging
recently or in the future, such as the disentangled attention mechanism from
DeBERTa. Our experiments show that SAS outperforms ELECTRA and other state-of-
the-art models in the GLUE tasks with similar or less computation cost.

## Introduction

In recent years, language model pre-training has achieved enormous success in
various natural language processing (NLP) downstream tasks, largely due to
innovations in pre-training task design as well as associated data
augmentation. Especially, the BERT model (Devlin et al. 2018) is a milestone,
where each input token sequence is augmented by randomly masking a proportion
(typically 15%) of tokens and then a masked language model (MLM) task is
undertook by recovering each masked token out of the vocabulary from the
augmented input. Since its success, BERT has been refined or extended in the
literature by adding or varying pre-training tasks and/or data augmentation.
For example, the RoBERTa (Liu et al. 2019) empirically shows that dynamically
masking tokens boosts BERT’s performance, while pairing sentences and
undertaking a corresponding next sentence prediction (NSP) task does not; the
SpanBERT (Joshi et al. 2019) masks spans of tokens in each input sequence and
then uses two boundary tokens of each masked span to predict each token in the
span; the StructBERT (Wang et al. 2019) shuffles a proportion of unmasked tri-
grams in each input sequence and then re-constructs the right order of these
tokens; the XLNet (Yang et al. 2019) permutes the token index order in each
input sequence and then autoregressively predicts the ending tokens in each
given order; the UNILM (Dong et al. 2019) employs different self-attention
masks to augment input data in three ways so that unidirectional,
bidirectional and sequence-to-sequence LM tasks can be performed together; the
ERNIE (Sun et al. 2019) uses prior knowledge to enhance its data augmentation
by phrase-level masking and entity-level masking; the CLEAR (Wu et al. 2020),
while keeping the MLM task, generates a pair of augmented sentences from each
input sentence using four augmentation methods (words deletion, spans
deletion, synonym substitution and reordering) and then performs a sentence-
level contrastive learning task to minimize the distance between augmented
sentences coming from the same original sentence. In all these models, data
augmentations are manually designed and context-independent without taking
sequence context into account.

Recently, the ELECTRA (Clark et al. 2020b) model is proposed to construct
contextualized data augmentation. It trains an auxiliary generation network to
provide plausible tokens to replace a proportion (typically 15%) of the
original tokens according to the input context, and then utilize a main
discrimination network to classify whether each token in the augmented input
is a replaced token or not. The latter task was named as replaced token
detection (RTD). After pre-training, the generator is discarded and the
discriminator is further fine-tuned for downstream NLP tasks. ELECTRA has
shown impressive advantages over MLM-based models in various downstream tasks
under similar computation cost, especially when a model size is small. Since
the ELECTRA model needs to train two separate neural networks (i.e., generator
and discriminator) side by side, however, it is challenging to maintain an
appropriate level of generator’s ability relative to discriminator’s during
the training. When the generator becomes stronger than what the discriminator
is capable of dealing with, it will prevent effective learning of the
discriminator. In addition, training two separate networks naturally leads to
extra computation cost.

To address the aforementioned disadvantages, we propose a self-augmentation
strategy (SAS) that uses a single neural network to provide contextualized
data augmentation for its own use in subsequent epochs during the training.111
In this paper, we focus on the field of language modeling and note that SAS is
a general strategy that can also be applied to other domains such as computer
vision and pattern recognition. The SAS network includes two light-weight
heads (MLM head and RTD head) on the top of one common transformer encoder
(Vaswani et al. 2017). This network architecture allows automatic adjustment
of the relative difficulty of the generated tokens from the MLM head, while
jointly undertaking both MLM and RTD pre-training task. As a result, our SAS
can achieve higher downstream GLUE scores (Wang et al. 2018) than ELECTRA
using the same or less computation cost. The advantages of SAS can be
summarized as follows:

  * •

Compared to ELECTRA and its subsequent variants that utilize an auxiliary
network to generate contextualized data augmentation, SAS introduces a single
network to jointly generate contextualized data augmentation and perform pre-
training tasks. On one hand, it avoids the challenge to balance the capability
of the generator and that of other model components. On the other hand, it has
both MLM and RTD heads on the top of the same encoder, so that the pre-trained
SAS model have a foundation for both generation and discrimination
capabilities which is beneficial to the fine-tuning of different downstream
tasks. As a comparison, in the case of ELECTRA, the generation network is
discarded and only the discrimination network is used for downstream
applications.

  * •

SAS eliminates the computation cost related to the generation network and
therefore further improves computation efficiency. While ELECTRA has already
shown its computation efficiency (especially for small models), SAS provides
an extra economic pre-training solution for many scenarios where compute
resource is limited .

  * •

SAS is a simple and general strategy so that it is able to easily incorporate
many other techniques emerging recently or in the future. For example, when
SAS incorporates the disentangled attention mechanism proposed in DeBERTa (He
et al. 2020), it can significantly outperform the previous state-of-the-art
models in the GLUE tasks using the same or less computation cost.

## Related Work

Pre-trained Language Models: As introduced in the previous section, many pre-
trained language models have emerged with new performance records in many
downstream NLP tasks in the last several years. Among them, ELECTRA (Clark et
al. 2020b) distinguishes itself from previous models and inspires our SAS
model, since it introduces contextualized data augmentation with an auxiliary
generation network and then train a main discrimination network to perform the
RTD task.

The Electric model is later proposed by Clark et al. (2020a) as an energy-
based model to perform the cloze task (Taylor 1953) using noise-contrastive
estimation (Gutmann and Hyvärinen 2012). It is particularly effective at
producing likelihood scores for text but slightly under-performs ELECTRA on
the GLUE tasks. This model also needs a separate network to provide noise
distribution in order to generate contextualized data augmentation. The MC-
BERT model is proposed by Xu et al. (2020b) to replace the RTD binary
classification task in ELECTRA with a multi-class classification task so that
richer semantic information can be captured. While MC-BERT performs on par
with ELECTRA, it also needs a separate network called meta controller to
generate a multi-class candidate set for each token as contextualized data
augmentation.

COCO-LM (Meng et al. 2021) has been recently proposed to improve ELECTRA by
using two new pre-training tasks called Corrective Language Modelling (CLM)
and Sequence Contrastive Learning tasks. In particular, the CLM refines the
All-Token MLM task studied by Clark et al. (2020b) and jointly learns the RTD
and MLM tasks by a combination of two loss objectives at all output token
positions. This is similar to SAS; however, a key caveat is that the MLM head,
which is computationally much heavier than the RTD head due to its softmax
calculation over the vocabulary, is only conducted at the augmented positions
in SAS. COCO-LM also relies on a separate generation network to generate the
augmented sequence for its two tasks which leads to extra computation. In
contrast, the proposed SAS eliminates the requirement of a separate generator
and achieves higher computation efficiency.

Another recent work is the DeBERTa model (He et al. 2020), which computes the
attention weights among tokens using disentangled matrices on two separate
vectors (content vector and relative position vector) of each token. Because
SAS is a general strategy, it can seamlessly incorporate such a disentangled
attention mechanism to further improve performance, as will be shown in
Section of Experiments.

Teacher-student Mechanism: SAS can be alternatively viewed from the
perspective of a teacher-student mechanism, where the knowledge of a teacher
model is utilized to facilitate the learning process of a student model. In
essence, in the SAS method, we treat the model trained in the current epoch as
a student and that in the previous epoch as a (weak) teacher, since the latter
generates contextualized data augmentation to help the former to learn.

The teacher-student mechanism has been traditionally used in knowledge
distillation for the purpose of label augmentation (Hinton, Vinyals, and Dean
2015), where soft labels generated from a teacher is combined with hard (one-
hot) labels observed from the original data to serve as the ultimate label to
supervise the learning of a student. The size of a student could be either
smaller than or the same as that of a teacher. Sanh et al. (2019) shows that
using the original BERT as a teacher their DistilBERT (student) model is able
to remain 97% of the teacher’s performance in the downstream GLUE tasks, while
the size of the student is only 60% of that of the teacher. When a student has
the same size as its teacher during the knowledge distillation, the advantages
of student over teacher have been empirically shown in recent research. For
example, Kim et al. (2020) show the benefits of their Teacher-free Knowledge
Distillation method by using a pre-trained first-generation model as a
teacher, while Yuan et al. (2020) show the gains of their Self-knowledge
Distillation method by using a (student) model itself in the previous epoch
during the training as a teacher.

In recent years, the teacher-student mechanism has also been used to obtain
state-of-the-art results in semi-supervised settings. Laine and Aila (2016)
propose Temporal Ensembling method to use an EMA (exponential moving average)
of the output predictions of the model in the previous epochs as a teacher
prediction to constitute the consistency loss component for each data
instance, in addition to the cross-entropy loss component for each labeled
data instance. Tarvainen and Valpola (2017) propose to use an EMA of parameter
values (i.e., weights) of (student) model to construct a mean teacher for the
consistency loss component for each data instance.

Curriculum Learning and Self-paced Learning Curriculum learning and self-paced
learning have attracted increasing attention in machine learning. Both of them
generally claim that it helps the learning process to start learning with
easier data and then gradually take more complex or difficult data into
consideration. There are some notable works that intend to use them in various
NLP tasks (Wan et al. 2020; Xu et al. 2020a; Nijkamp et al. 2021). In SAS, we
naturally provides a curriculum for the model itself by incrementally
augmenting the data as MLM improves generation capability over the training
process. In essence, our model can be regarded as a combination of the two
learning strategies which adaptively generate augmented data for better
learning. This kind of idea dates back to the work of Jiang et al. (2015),
which is a pioneer of combining these two strategies.

## Method

In this section, we formulate the self-augmentation strategy as a general
framework that can be applied to various domains, and then detail its usage in
language model pre-training.

### General SAS Framework

In supervised learning, each data instance has both feature $\bm{x}_{i}$ and
its corresponding label $\bm{y}_{i}$. In self-supervised learning such as
language model pre-training, since there is no label in the training dataset,
an input pair of $\bm{\tilde{x}}_{i}$ and label $\bm{\tilde{y}}_{i}$ is
constructed by an augmentation process $q$ from the original data instance of
token sequence $\bm{x}_{i}\sim\mathcal{X}$. The augmentation process can be a
context-independent process (such as random masking with a certain
distribution) or a generation network $f_{g}$ with parameter $\theta_{g}$
which takes the sequence context into account. Let’s consider the latter and
denote the augmented dataset as
$\tilde{{\mathcal{X}}}=\\{[\bm{\tilde{x}}_{i},\bm{\tilde{y}}_{i}]=f_{g}(\bm{x}_{i})|\bm{x}_{i}\sim\mathcal{X};i\in\\{1,\cdots,n\\}\\}$.
As shown in Figure 1(a), the augmented data instance is fed to a heavy-weight
main network with encoder $f_{e}$ with parameter $\theta_{e}$ and a light-
weight head $h_{d}$ with parameter $\theta_{d}$ to conduct pre-training task.
In a forward pass, the auxiliary generator and the main network are used
sequentially. The auxiliary generator $f_{g}$ is usually a heavy-weighted
network.

In SAS, we eliminate the separate generator in augmentation. Instead, we
introduce both generation and discrimination capabilities via a single network
with both MLM and RTD heads, $h_{g}$ and $h_{d}$, and design a light-weight
generation process by using the softmax output of MLM. That is, the MLM head
serves dual purposes for both pre-training and contextualized data
augmentation. In the first epoch $(t=1)$, the augmented data is generated by a
(cold-start) prior distribution $p^{\\{0\\}}$, since there is no prior MLM
output yet. In the $(t+1)$’th epoch for $t\geq 1$, the augmented data is
generated based on MLM’s softmax output of the previous epoch. During the
training, while $h_{g}$ is updated from generation loss and $h_{d}$ from
discrimination loss, the encoder $f_{e}$ is updated from both losses.

Figure 1: The comparison between the traditional self-supervised learning with
model-based augmentation and the self-augmentation strategy.

### SAS in Language Model Pre-training

Now we describe the details of SAS in the context of language model pre-
training. As mentioned above, SAS has a single neural network that includes
two light-weight heads

Figure 2: Workflow of SAS in language model pre-training.

on the top of one common heavy-weight transformer encoder. It undertakes both
the MLM and RTD tasks in each forward pass during the training process. A
workflow of SAS is demonstrated in Figure 2.

For an instance $\bm{x}=(x_{1},x_{2},\cdots,x_{k})$ of a token sequence of
length $k$ in the pre-training dataset $\mathcal{X}$, in the $t$’th epoch of
the pre-training process, we choose a random set $S^{\\{t\\}}$ with $\lceil
0.15\cdot k\rceil$ position indexes, just as BERT and ELECTRA, and create
augmented input instance
$\bm{\tilde{x}}^{\\{t\\}}=(\tilde{x}_{1}^{\\{t\\}},\tilde{x}_{2}^{\\{t\\}},\cdots,\tilde{x}_{k}^{\\{t\\}})$
as follows,

$\left\\{\begin{array}[]{ll}\tilde{x}_{i}^{\\{t\\}}=x_{i}&{i\notin
S^{\\{t\\}}}\\\ \tilde{x}_{i}^{\\{t\\}}\sim p^{\\{t-1\\}}&{i\in
S^{\\{t\\}}}\end{array}\right.$ (1)

In other words, we keep original token $x_{i}$ if $i\notin S^{\\{t\\}}$ and
otherwise replace it by sampling a token based on a generating distribution
$p^{\\{t-1\\}}$ to be detailed below.

In the first epoch ($t=1$), we set $p^{\\{0\\}}$ as a cold-start token
generating distribution, such as a (token-frequency based) unigram
distribution222Our experimental results (in the appendix) show that the
unigram distribution and the uniform distribution work equally well for our
SAS.. Thereafter, in epoch $t+1$ ($t\geq 1$), we leverage the softmax output
$p^{\\{t\\}}_{\theta_{e},\theta_{g}}(x_{i}|\bm{\tilde{x}}^{\\{t\\}})$ of the
MLM head in the $t$’th epoch as a contextualized token generating
distribution. We have

$\begin{split}&p^{\\{t\\}}_{\theta_{e},\theta_{g}}(x_{i}|\bm{\tilde{x}}^{\\{t\\}})=\frac{\exp\\{e(x_{i})^{T}h_{g}(f_{e}(\bm{\tilde{x}}^{\\{t\\}}))_{i}\\}}{\sum_{x^{\prime}\in
V}\exp\\{e(x^{\prime})^{T}h_{g}(f_{e}(\bm{\tilde{x}}^{\\{t\\}}))_{i}\\}},\end{split}$
(2)

where $e(x_{i})$ is the embedding of token $x_{i}$, $f_{e}$ is the encoder
that outputs a sequence of $k$ contextualized representation vectors from the
augmented input $\bm{\tilde{x}}$, $h_{g}$ is the MLM head that generates
representation vector at each augmented position in $S^{\\{t\\}}$, and $V$ is
the vocabulary. Essentially, the SAS model itself in the $t$’th epoch serves
as a teacher to provide data augmentation for its learning in the $(t+1)$’th
epoch.

In practice, with the distribution
$p^{\\{t\\}}_{\theta_{e},\theta_{g}}(x_{i}|\bm{\tilde{x}}^{\\{t\\}})$
generated from the MLM head in the $t$’th epoch, we directly sample
$\tilde{x}_{i}^{\\{t+1\\}}$ from it for every $i\in S^{\\{t+1\\}}$ and then
store the sampled token index in CPU memory for the augmentation usage in the
$(t+1)$’th epoch. This way, we avoid storing the whole distribution with high
memory cost. The loss of the MLM task is calculated as a softmax loss,

$\mathcal{L}_{mlm}^{\\{t\\}}(\bm{x};\theta_{e},\theta_{g})=-\mathbb{E}\left[\sum_{i\in
S^{\\{t\\}}}\log
p^{\\{t\\}}_{\theta_{e},\theta_{g}}(x_{i}|\bm{\tilde{x}}^{\\{t\\}})\right].$
(3)

We adopt the same RTD task as ELECTRA to classify whether each token in the
augmented input $\bm{\tilde{x}}^{\\{t\\}}$ is a replaced token or not. The
corresponding classification probability is

$D(\bm{\tilde{x}}^{\\{t\\}})_{i}=\sigma(h_{d}(f_{e}(\bm{\tilde{x}}^{\\{t\\}}))_{i}),$
(4)

where $\sigma$ is the logistic sigmoid, $h_{d}$ is the RTD head function that
generates a sequence of logits at all the positions. Note that the same
$f_{e}$ is used in Eq. (2) and Eq. (4), indicating that only one forward pass
through the encoder is needed. This largely reduces the computation cost, in
contrast to ELECTRA’s separate forward passes in two neutral networks.

Eq. (4) leads to the loss of the RTD task given the instance $\bm{x}$ in the
$t$’th epoch as

$\begin{split}&\mathcal{L}_{rtd}^{\\{t\\}}(\bm{x};\theta_{e},\theta_{d})=-\mathbb{E}\bigg{[}\sum_{j=1}^{k}y^{\\{t\\}}_{j}\log
D(\bm{\tilde{x}}^{\\{t\\}})_{j}\\\
&+(1-y^{\\{t\\}}_{j})\log(1-D(\bm{\tilde{x}}^{\\{t\\}})_{j})\bigg{]},\end{split}$
(5)

where $y^{\\{t\\}}_{j}=1$ if $\tilde{x}_{j}^{\\{t\\}}=x_{j}$,
$y^{\\{t\\}}_{j}=0$ otherwise.

We train the SAS model by minimizing a combined loss:

$\begin{split}\sum_{\bm{x}\in\mathcal{X}}\mathcal{L}_{mlm}^{\\{t\\}}(\bm{x};\theta_{e},\theta_{g})+&\lambda^{\\{t\\}}\mathcal{L}_{rtd}^{\\{t\\}}(\bm{x};\theta_{e},\theta_{d}),\end{split}$
(6)

where $\lambda^{\\{t\\}}$ is the (relative) weight of the RTD loss which can
be a constant or varied across different epochs. The joint-task learning with
the combines loss, along with the shared encoder for both heads, avoids
learning obstacles from potentially over-challenging replaced tokens.

## Experiments

In this section, we describe the experimental setup and corresponding
experimental results.

### Experimental Setup

Pre-training Details: Our implementation333Code and pretrained model is
available publicly at Github: https://github.com/alibaba/self-augmentation-
strategy. is based on Huggingface Transformers 4.3 framework (Wolf et al.
2020). We include ELECTRA, DeBERTa as well as BERT for comparison. Under the
current constraints of computation resource, we focus on the small and base
models which have been extensively studied and compared by Clark et al.
(2020b), and we set architectures and hyperparameters aligned with ELECTRA.
Please refer to the appendix for the detailed model architecture and pre-
training hyperparameter values. We implement each model by largely re-using
the corresponding code from Huggingface (Wolf et al. 2020), if a pre-trained
checkpoint has not been publicly released by its authors. We use the same pre-
training data as BERT, ELECTRA-Small and ELECTRA-Base, which consists of 3.3
Billion tokens from Wikipedia and BooksCorpus datasets. For fair comparison,
we follow Clark et al. (2020b) to use FLOPs (floating point operations) to
measure computation usage (since FLOPs is a measure agnostic to the particular
hardware and low-level optimizations). We reuse the FLOPs computation
code444See https://github.com/google-
research/electra/blob/master/flops˙computation.py released from Clark et al.
(2020b) so that we essentially take the exactly same assumptions made by Clark
et al. (2020b). Some details of the experimented models are as follows.

  * •

ELECTRA: We train ELECTRA-Base and ELECTRA-Small using the exactly same
hyperparameter values as Clark et al. (2020b), except for larger batch size
and learning rate for ELECTRA-Small to reduce the pre-training time (which is
not reflected in the FLOPs calculation). For ELECTRA-Small model as well as
all other small models, we use batch size 512 and 0.25M pre-training steps,
instead of batch size 128 and 1M steps in Clark et al. (2020b), and double the
learning rate accordingly.555We observe that such a change is able to
significantly reduce the pre-training time without degrading the model
performance. As a reference point, we also include ELECTRA-Small++ whose pre-
trained model checkpoint is publicly released by Clark et al. (2020b). Note
that ELECTRA-Small++ uses 18x training FLOPs compared to ELECTRA-Small because
it is pre-trained much longer with much larger data and its input sequence
length is also quadrupled (Clark et al. 2020b).

  * •

DeBERTa: We implement DeBERTa and adopt its idea of sharing the projection
matrices between relative position embedding and content embedding in all the
attention layers proposed by He et al. (2020). Such a sharing does retain the
model performance on the small and base models while reducing the number of
model parameters.

  * •

BERT: For BERT-Base, we use its model checkpoint publicly released by Devlin
et al. (2018). We implement our BERT-Small and set its embedding size the same
as its hidden size666Clark et al. (2020b) define a different BERT-Small
setting where its embedding size is decreased to half of its hidden size. ,
according to the convention of the BERT models. Please refer to the appendix
for the details about the hyperparameters. Our BERT-Small setting makes its
FLOPs similar to that of ELECTRA-Small when the training steps are the same,
so that a fair comparison of their performance can be made directly.

  * •

SAS: We experiment three SAS settings:

    * –

SAS denotes the setting with a default weight scheduler that epoch-wisely
increases the RTD loss weight $\lambda^{\\{t\\}}$ from 50 to 200 during the
pre-training.

    * –

SASc denotes a setting where RTD loss weight $\lambda^{\\{t\\}}$ is a constant
value 50.

    * –

SAS${}^{\text{DA}}$ denotes a setting that uses the default weight scheduler
and further incorporates the disentangled attention mechanism.

In all the SAS settings, we set the (token-frequency based) unigram
distribution as the cold-start distribution.

Downstream Tasks: We evaluate all models on the General Language Understanding
Evaluation (GLUE) benchmark (Wang et al. 2018). It contains a variety of tasks
covering natural language inference tasks MNLI (Williams, Nangia, and Bowman
2017), RTE (Giampiccolo et al. 2007), and QNLI (Rajpurkar et al. 2016);
semantic similarity tasks MRPC (Dolan and Brockett 2005), QQP (Iyer, Dandekar,
and Csernai 2017), and STS-B (Cer et al. 2017); sentiment classification task
SST-2 (Socher et al. 2013); and linguistic acceptability classification CoLA
(Warstadt, Singh, and Bowman 2019). See the supplementary for more details on
the GLUE tasks.

The evaluation metrics are the average of Spearman correlation and Pearson
correlation for STS-B, Matthews correlation for CoLA, the average of MNLI-
match accuracy and MNLI-mismatch accuracy for MNLI, and accuracy for other
GLUE tasks. We also take the average of metrics of these eight GLUE tasks,
denoted by GLUE Score, as the overall performance metric on these tasks. All
GLUE scores are based on the Dev dataset.

Fine-tuning Procedure: For the fine-tuning of GLUE, we add simple linear
classifiers on top of the encoder of a pre-trained model. Because we observe a
large performance variance in the GLUE tasks with small data sizes (such as
CoLA, MRPC, STS-B and RTE), we adopt the following two methods to reduce the
variance. First, we follow the strategy proposed in the papers (Mosbach,
Andriushchenko, and Klakow 2020; Zhang et al. 2020; Dodge et al. 2020) to
train more epochs with small learning rates for these small tasks. Second, we
fine-tune these small tasks by using multiple random seeds and obtain the
average score across the seeds. Please refer to the supplementary for detailed
fine-tuning hyperparameter settings.

For base models, we pre-train each model once and then use the above fine-
tuning strategy to obtain the score of each GLUE task. Since for some small
models we still observe non-negligible variance of the resulting scores, we
pre-train each small model using five different random seeds. The finally
reported score of each task is the average across the five pre-trained model
checkpoints.

### Overall Comparison Results

Model |  Train FLOPs |  GLUE
mean$\pm$std | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI | QNLI | RTE
---|---|---|---|---|---|---|---|---|---|---
BERT-Small |  1.274e18 |  79.10$\pm$0.07 | 49.69 | 90.14 | 84.64 | 86.04 | 89.51 | 80.00 | 86.84 | 65.94
BERT-Small-1.5x |  1.911e18 |  79.55$\pm$0.09 | 51.80 | 90.37 | 84.07 | 86.04 | 89.80 | 80.20 | 86.58 | 67.51
ELECTRA-Small |  1.294e18 |  80.78$\pm$0.18 | 59.27 | 89.18 | 86.40 | 86.76 | 89.92 | 80.29 | 88.23 | 66.21
ELECTRA-Small++* |  2.403e19 |  82.05 | 58.37 | 91.40 | 87.01 | 87.95 | 90.54 | 82.52 | 88.93 | 69.68
DeBERTa-Small |  1.381e18 |  79.52$\pm$0.43 | 49.51 | 89.91 | 86.68 | 86.29 | 90.26 | 81.51 | 87.78 | 64.26
SASc-Small |  1.279e18 |  81.30$\pm$0.13 | 59.52 | 89.60 | 87.17 | 87.27 | 90.20 | 81.54 | 88.67 | 66.43
SAS-Small |  1.279e18 |  81.61$\pm$0.24 | 60.49 | 90.08 | 87.01 | 87.32 | 90.11 | 81.37 | 88.47 | 68.05
SAS${}^{\text{DA}}$-Small |  1.385e18 |  82.14$\pm$0.22 | 62.35 | 90.55 | 87.55 | 87.52 | 90.60 | 82.20 | 88.71 | 67.65

*: ELECTRA-Small++ is the pre-trained model publicly released by Clark et al. (2020b). It uses 18x training FLOPs compared to ELECTRA-Small.

Table 1: Comparison of small models on the GLUE dev set.

Model |  Train FLOPs | GLUE | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI | QNLI | RTE
---|---|---|---|---|---|---|---|---|---|---
BERT-Base (1M)* |  6.430e19 | 83.06 | 60.07 | 92.09 | 85.29 | 89.22 | 91.27 | 83.99 | 91.43 | 71.12
ELECTRA-Base (766K) |  6.426e19 | 85.46 | 65.53 | 91.28 | 89.95 | 90.33 | 91.65 | 85.49 | 91.85 | 77.62
DeBERTa-Base (1M) |  7.443e19 | 83.97 | 58.46 | 93.23 | 88.97 | 89.36 | 91.37 | 85.53 | 91.52 | 73.29
SAS-Base (959K) |  6.196e19 | 85.20 | 66.24 | 92.20 | 89.22 | 90.30 | 91.46 | 85.54 | 91.95 | 74.73
SAS${}^{\text{DA}}$-Base (833K) |  6.226e19 | 85.60 | 66.56 | 93.35 | 88.73 | 90.05 | 91.73 | 86.49 | 91.74 | 76.17

*: BERT-Base is the model publicly released by Devlin et al. (2018).

Table 2: Comparison of base models on the GLUE dev set.

The performance comparison among the small models is shown in Table 1. In the
table, the second column shows the training FLOPs of each model, and the third
column lists the mean and the standard deviation of GLUE Score for each model
across five independently pre-trained checkpoints.

As for the three competing models, it shows that both ELECTRA-Small and
DeBERTa-Small outperform BERT-Small in terms of GLUE Score. Even when we
increase the pre-training steps of BERT-Small model from 250K to 375K, denoted
by BERT-Small-1.5x in the table, it still is worse than ELECTRA-Small while
being similar to DeBERTa-Small. The GLUE Score of ELECTRA-Small (our
implementation) is about 98.45% of that of ELECTRA-Small++ released by Clark
et al. (2020b) (80.78 vs. 82.05), which is higher than the 97.87% in Table 8
of the original paper Clark et al. (2020b). This verifies the correctness of
our ELECTRA implementation. Note that DeBERTa-Small has much higher MNLI mean
score than ELECTRA-Small, while ELECTRA-Small has higher mean of GLUE Score
which is aligned with the overall advantage of ELECTRA in small models
emphasized by Clark et al. (2020b).

The proposed SAS model further improves the overall performance over ELECTRA
and DeBERTa with fewer FLOPs.777With 1 V100 GPU, the pre-training of
SAS${}^{\text{DA}}$-Small takes 37.5h; both SAS-Small and
SAS${}^{\text{c}}$-Small takes about 24h; and ELECTRA-Small takes about 35h.
The mean of GLUE Score of SASc-Small (with the constant weight for the RTD
loss) is 0.52 point higher than that of ELECTRA-Small, while SAS with the
epoch-wise weight increasing strategy is 0.83 point higher. As for MNLI task,
both SASc-Small and SAS have their mean scores more than 1 point higher than
ELECTRA.

Notably, SAS${}^{\text{DA}}$-Small, the SAS with the disentangled attention
mechanism, achieves a higher GLUE Score than ELECTRA-Small++’s with only 5.76%
of FLOPs consumed by ELECTRA-Small++. The disentangled attention mechanism
embraced by SAS is able to improve the mean of GLUE Score by 0.65% (from 81.61
to 82.14). This is larger than its 0.53% improvement ratio on the basis of
BERT (from 79.10 to 79.52), which shows that SAS can effectively realize the
value of the disentangled attention mechanism. The table also shows that
DeBERTa-Small has the largest standard deviation among all the models. This
might be because DeBERTa-Small needs larger data (such as the data used by He
et al. (2020)) to stably achieve its functionality.

The comparison results on the base models are shown in Table 2. In the first
column of the table, we show the pre-training steps of each model and have
ensured that SAS takes similar or fewer FLOPs than other models. The table
shows that SAS-Base has a slightly lower GLUE Score than ELECTRA-Base, while
SAS${}^{\text{DA}}$-Base888 The pre-training costs 7.7 days by 8 V100 GPUs. is
able to achieve both the highest GLUE Score (ELECTRA-Base is a very close
second) and the highest MNLI score (nearly 1 point higher than the second best
DeBERTa-Base).

### Ablation Study

We design the following variants to investigate the contributions of different
components of SAS:

  * •

Unig-MLM denotes a variant of BERT with random replacement of 15% of tokens
based on the (token-frequency based) unigram distribution. That is, each of
the randomly selected tokens is replaced with a token sampled from the unigram
distribution instead of a [MASK] token.

  * •

Unig-MLM-SAS denotes a variant of SAS with only MLM task instead of two
combined tasks.

  * •

Unig-MLM-RTDc denotes a model that is the same as SASc but use a random
replacement with the unigram distribution in the whole training process.

  * •

Unig-MLM-RTD denotes a model that is the same as SAS but use a random
replacement with the unigram distribution in the whole training process.

Model |  GLUE
mean$\pm$std |  CoLA |  SST-2 |  MRPC |  STS-B |  QQP |  MNLI |  QNLI |  RTE
---|---|---|---|---|---|---|---|---|---
Mask-MLM (BERT) |  79.10 $\pm$0.12 |  49.69 |  90.14 |  84.64 |  86.04 |  89.51 |  80.00 |  86.84 |  65.94
Unig-MLM |  79.00 $\pm$0.07 |  49.61 |  90.06 |  84.97 |  85.43 |  89.42 |  80.11 |  86.93 |  65.46
Unig-MLM-SAS |  79.38 $\pm$0.36 |  49.94 |  89.72 |  86.13 |  85.48 |  89.69 |  80.34 |  87.43 |  66.28
Unig-MLM-RTDc |  80.43 $\pm$0.14 |  59.28 |  89.37 |  85.46 |  85.47 |  89.87 |  80.50 |  88.17 |  65.34
Unig-MLM-RTD |  80.40 $\pm$0.15 |  59.96 |  89.53 |  85.05 |  85.36 |  89.76 |  80.56 |  87.97 |  64.98
SASc |  81.30 $\pm$0.13 |  59.52 |  89.60 |  87.17 |  87.27 |  90.20 |  81.54 |  88.67 |  66.43
SAS |  81.61 $\pm$0.24 |  60.49 |  90.08 |  87.01 |  87.32 |  90.11 |  81.37 |  88.47 |  68.05
Table 3: Ablation study on small models on the GLUE dev set.

Table 3 summarizes the results on the small models. First, it shows slight
drop (0.1 point) in GLUE Score if the random masking of BERT is changed to the
random replacement, which might be among the reasons that BERT uses the
special [MASK] token despite its mismatch with the fine-tuning data. Second,
when we use SAS to conduct the MLM task alone, it only increases the mean of
GLUE Score from 79.10 to 79.38 compared to BERT, which indicates that the MLM
task cannot fully utilize the self-augmented data to learn a better model.
Third, by comparing Unig-MLM to Unig-MLM-RTD (or Unig-MLM-RTDc), we observe
that a big (about 1.4 point) increase in the mean of GLUE Score comes from
adding the RTD task with the MLM task. This increase is mainly due to the
improvement in CoLA, as it alone contributes more than a 1.2 point increase in
the mean of GLUE Score. This indicates that the RTD task along with the data
augmented from the unigram distribution can greatly help the syntactic task
(CoLA) but not other semantic tasks in GLUE. Finally, by comparing Unig-MLM-
RTD (or Unig-MLM-RTDc) to SAS (or SASc), we see another notable increase in
the mean of GLUE Score, which comes from the improvement in both the syntactic
task (CoLA) and other semantic tasks such as MNLI. This shows that SAS can
further improve a variety of different tasks, as its pre-trained model has
both generation and discrimination capabilities with MLM and RTD heads
together. The effect of the weight strategy for the RTD loss can also be seen
from Table 3. By comparing SASc to SAS, we see that CoLA’s score has a further
increase when the epoch-wise weight increasing strategy gradually puts more
emphasis on the RTD task.

### Pre-Training Efficiency

Figure 3: Curve of GLUE Score for SAS-Small model and its competitors with
respect to the pre-training steps on the GLUE dev set. Figure 4: Curve of
MNLI’s average accuracy for SAS-Small model and its competitors with respect
to the pre-training steps on the GLUE dev set.

To investigate the convergence of pre-training, in Figures 3 and 4, we plot
GLUE Score and MNLI score with respect to the number of pre-training steps for
SAS-Small and other models. For each model, we select the median run whose
pre-training random seed achieves the median GLUE Score among the five random
seeds. Then for the selected median run of each model, we save a checkpoint
every 25K pre-training steps (i.e., 0.5 epoch), and fine-tune it on each GLUE
task and finally report GLUE Score across the tasks. In Figure 3, it shows
that SAS-Small is significantly better than BERT-Small during the whole pre-
training process, and starts to outperform all other models after the 1.5-th
epoch. Figure 4 shows that SAS-Small and DeBERTa-Small perform consistently
better than BERT-Small and ELECTRA-Small for MNLI task, while SAS-Small
performs on a par with DeBERTa-Small since the second epoch.

## Conclusion

We propose the self-augmentation strategy (SAS) motivated by ELECTRA’s
contextualized data augmentation for language model pre-trainings. SAS
eliminates a separate generation network used in ELECTRA and uses a single
network to generate the contextualized data augmentation for its own
subsequent training. By this way, it avoids the challenge in controlling the
relative capability of the generator and reduces the computation cost. In
addition, our SAS is a general and open framework so that it can easily
incorporate many other techniques emerging recently or in the future to
further improve its performance. The experimental results show that SAS is
able to outperform the previous state-of-the-art language pre-training models
in the small and base scales with similar or less computation cost. We will
continue to validate SAS’s advantage in larger models when sufficient
computation resources are available, while encouraging more studies on
economic and compute-efficient pre-training approaches.

## References

  * Cer et al. (2017) Cer, D.; Diab, M.; Agirre, E.; Lopez-Gazpio, I.; and Specia, L. 2017.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_.
  * Clark et al. (2020a) Clark, K.; Luong, M.-T.; Le, Q.; and Manning, C. D. 2020a.  Pre-Training Transformers as Energy-Based Cloze Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 285–294. Online: Association for Computational Linguistics.
  * Clark et al. (2020b) Clark, K.; Luong, M.-T.; Le, Q. V.; and Manning, C. D. 2020b.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dodge et al. (2020) Dodge, J.; Ilharco, G.; Schwartz, R.; Farhadi, A.; Hajishirzi, H.; and Smith, N. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_.
  * Dolan and Brockett (2005) Dolan, W. B.; and Brockett, C. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Dong et al. (2019) Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H. 2019.  Unified Language Model Pre-training for Natural Language Understanding and Generation.  _CoRR_ , abs/1905.03197.
  * Giampiccolo et al. (2007) Giampiccolo, D.; Magnini, B.; Dagan, I.; and Dolan, W. B. 2007.  The third pascal recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , 1–9.
  * Gutmann and Hyvärinen (2012) Gutmann, M. U.; and Hyvärinen, A. 2012.  Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics.  _Journal of Machine Learning Research_ , 13(11): 307–361.
  * He et al. (2020) He, P.; Liu, X.; Gao, J.; and Chen, W. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the Knowledge in a Neural Network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Iyer, Dandekar, and Csernai (2017) Iyer, S.; Dandekar, N.; and Csernai, K. 2017.  First quora dataset release: Question pairs.  _data. quora. com_.
  * Jiang et al. (2015) Jiang, L.; Meng, D.; Zhao, Q.; Shan, S.; and Hauptmann, A. 2015.  Self-paced curriculum learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 29.
  * Joshi et al. (2019) Joshi, M.; Chen, D.; Liu, Y.; Weld, D. S.; Zettlemoyer, L.; and Levy, O. 2019.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _CoRR_ , abs/1907.10529.
  * Kim et al. (2020) Kim, K.; Ji, B.; Yoon, D.; and Hwang, S. 2020.  Self-Knowledge Distillation: A Simple Way for Better Generalization.  _CoRR_ , abs/2006.12000.
  * Laine and Aila (2016) Laine, S.; and Aila, T. 2016.  Temporal Ensembling for Semi-Supervised Learning.  _CoRR_ , abs/1610.02242.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Meng et al. (2021) Meng, Y.; Xiong, C.; Bajaj, P.; Tiwary, S.; Bennett, P.; Han, J.; and Song, X. 2021\.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  _arXiv preprint arXiv:2102.08473_.
  * Mosbach, Andriushchenko, and Klakow (2020) Mosbach, M.; Andriushchenko, M.; and Klakow, D. 2020.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  _arXiv preprint arXiv:2006.04884_.
  * Nijkamp et al. (2021) Nijkamp, E.; Pang, B.; Wu, Y. N.; and Xiong, C. 2021.  SCRIPT: Self-Critic Pretraining of Transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5196–5202.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Socher et al. (2013) Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 1631–1642.
  * Sun et al. (2019) Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; and Wu, H. 2019.  ERNIE: Enhanced Representation through Knowledge Integration.  _CoRR_ , abs/1904.09223.
  * Tarvainen and Valpola (2017) Tarvainen, A.; and Valpola, H. 2017.  Mean Teachers Are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, 1195–1204.  ISBN 9781510860964.
  * Taylor (1953) Taylor, W. L. 1953.  “Cloze Procedure”: A New Tool for Measuring Readability.  _Journalism Quarterly_ , 30(4): 415–433.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30, 6000–6010.
  * Wan et al. (2020) Wan, Y.; Yang, B.; Wong, D. F.; Zhou, Y.; Chao, L. S.; Zhang, H.; and Chen, B. 2020\.  Self-Paced Learning for Neural Machine Translation.  _arXiv preprint arXiv:2010.04505_.
  * Wang et al. (2018) Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  _CoRR_ , abs/1804.07461.
  * Wang et al. (2019) Wang, W.; Bi, B.; Yan, M.; Wu, C.; Bao, Z.; Peng, L.; and Si, L. 2019.  StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding.  _CoRR_ , abs/1908.04577.
  * Warstadt, Singh, and Bowman (2019) Warstadt, A.; Singh, A.; and Bowman, S. R. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7: 625–641.
  * Williams, Nangia, and Bowman (2017) Williams, A.; Nangia, N.; and Bowman, S. R. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Wu et al. (2020) Wu, Z.; Wang, S.; Gu, J.; Khabsa, M.; Sun, F.; and Ma, H. 2020.  CLEAR: Contrastive Learning for Sentence Representation.  _arXiv preprint arXiv:2012.15466_.
  * Xu et al. (2020a) Xu, B.; Zhang, L.; Mao, Z.; Wang, Q.; Xie, H.; and Zhang, Y. 2020a.  Curriculum learning for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 6095–6104.
  * Xu et al. (2020b) Xu, Z.; Gong, L.; Ke, G.; He, D.; Zheng, S.; Wang, L.; Bian, J.; and Liu, T. 2020b.  MC-BERT: Efficient Language Pre-Training via a Meta Controller.  _CoRR_ , abs/2006.05744.
  * Yang et al. (2019) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J. G.; Salakhutdinov, R.; and Le, Q. V. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  _CoRR_ , abs/1906.08237.
  * Yuan et al. (2020) Yuan, L.; Tay, F. E.; Li, G.; Wang, T.; and Feng, J. 2020.  Revisiting Knowledge Distillation via Label Smoothing Regularization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3903–3911.
  * Zhang et al. (2020) Zhang, T.; Wu, F.; Katiyar, A.; Weinberger, K. Q.; and Artzi, Y. 2020.  Revisiting few-sample BERT fine-tuning.  _arXiv preprint arXiv:2006.05987_.

Model |  GLUE Score
mean$\pm$std | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI | QNLI | RTE
---|---|---|---|---|---|---|---|---|---
Unig-SASc-Small |  81.30$\pm$0.13 | 59.52 | 89.60 | 87.17 | 87.27 | 90.20 | 81.54 | 88.67 | 66.43
Unig-SAS-Small |  81.61$\pm$0.03 | 60.49 | 90.08 | 87.01 | 87.32 | 90.11 | 81.37 | 88.47 | 68.05
Unif-SASc-Small |  81.49$\pm$0.17 | 59.59 | 89.98 | 87.35 | 87.39 | 90.15 | 81.44 | 88.26 | 67.73
Unif-SAS-Small |  81.57$\pm$0.15 | 60.59 | 90.34 | 87.21 | 87.35 | 90.10 | 81.39 | 88.09 | 67.51
Table 4: Comparison of cold-start token generating distributions of SAS-Small
on the GLUE dev set.

## Appendices

### Study on Cold-start Token Generating Distribution

We experiment with the two common choices for the cold-start token generating
distribution for our SAS: the uniform distribution and the (frequency-based)
unigram distribution. The results are shown in Table 4. We use the prefixes
”Unig” and ”Unif” to denote the unigram distribution and the uniform
distribution respectively. The table clearly shows that there is no
statistically significant difference in the GLUE average scores between the
two distribution choices. They work equally well for our SAS.

### Pre-training Details

The following pre-training details apply to our SAS and its competing methods
include the BERT, the ELECTRA and the DeBERTa. We always use Adam as the
optimizer with weight decay. We mostly use the same hyperparameters as BERT
and ELECTRA. For our SAS, we introduce a scheduler for $\lambda^{\\{t\\}}$,
the (relative) weight for the RTD loss in Eq. 6, which has an epoch-wise
linear increase from 50 to 200 by default. We use the whole-word and dynamic
token masking with the masked positions decided on the fly instead of during
the preprocessing. Our own implementation also does not include the next
sentence prediction (NSP) task proposed in the original BERT, as the recent
works such as Liu et al. (2019) have suggested that it does not improve the
performance. For the investigated DeBERTa baseline, we use the same corpus,
tokenizer, training steps as the BERT for a fair comparison. We searched for
the best learning rate for Small models out of [1e-3, 5e-4] and selected
$\lambda$ out of [1, 25, 50, 100] in early experiments. Otherwise, we did no
hyperparameter tuning beyond the experiments. The full set of hyperparameters
is listed in Table 5.

Hyperparameter |  ELECTRA-Small |  All Other Small Models |  All Base Models
---|---|---|---
Number of layers |  12 |  12 |  12
Hidden size |  256 |  256 |  768
FFN inner hidden size |  1024 |  1024 |  3072
Attention heads |  4 |  4 |  12
Attention head size |  64 |  64 |  64
Embedding size |  128 |  256 |  768
Sequence length |  128 |  128 |  512
Mask percent |  15 |  15 |  15
Learning rate decay |  Linear |  Linear |  Linear
Warmup steps |  10000 |  10000 |  10000
Learning rate |  1e-3 |  1e-3 |  2e-4
Adam $\epsilon$ |  1e-6 |  1e-6 |  1e-6
Adam $\beta_{1}$ |  0.9 |  0.9 |  0.9
Adam $\beta_{2}$ |  0.999 |  0.999 |  0.999
Attention dropout |  0.1 |  0.1 |  0.1
Dropout |  0.1 |  0.1 |  0.1
Weight decay |  0.01 |  0.01 |  0.01
Batch size |  512 |  512 |  256
Train steps |  0.25M |  0.25M |  766K - 1M
Table 5: Pre-training hyperparameters for all the models pre-trained by us. Hyperparameter | Value
---|---
Learning rate | 1e-4, 7.5e-5 for Small; 7.5e-5, 5e-5 for Base
Adam $\epsilon$ | 1e-6
Adam $\beta_{1},\beta_{2}$ | 0.9, 0.999
Layerwise LR decay | None
Learning rate decay | Linear
Warmup fraction | 0.1
Attention dropout | 0.1
Dropout | 0.1
Weight decay | None
Batch size | 16, 32
Train epochs | 20 for CoLA, MRPC, STS-B, RTE; 4 for other tasks
Seeds | 5 for CoLA, MRPC, STS-B, RTE; 3 for QNLI, SST2; 1 for MNLI, QQP
Table 6: Fine-tuning hyperparameters for all the investigated models.

### Fine-tuning Details

We originally fine-tuned all the pre-trained models for 4 epochs. However,
because we observed a large variance in the small tasks in GLUE, following the
advice from Mosbach, Andriushchenko, and Klakow (2020), we increase the fine-
tuning process to 20 epochs and select the best epoch for the four small tasks
including CoLA, MRPC, STS-B and RTE. For Small models, we searched for the
best learning rate out of [1e-4, 7.5e-5]. For Base models, we searched for a
learning rate out of [5e-5, 7.5e-5] without the layer-wise learning-rate decay
proposed by ELECTRA, but otherwise used the same hyperparameters as for small
models. Due to limited computation resource, we adjust the number of
independent fine-tuning runs (with different random seeds) so that we fine-
tune more times for these tasks with smaller data sizes (i.e., with more
variability). The full set of hyperparameters is listed in Table 6. Following
the BERT and the ELECTRA, we do not show results on the WNLI GLUE task for the
Dev set results.

### Details about GLUE

We provide further details about the GLUE benchmark tasks as follows.

CoLA: Corpus of Linguistic Acceptability (Warstadt, Singh, and Bowman 2019).
The task is to determine whether a given sentence is grammatical or not. The
dataset contains 8.5k train examples from books and journal articles on
linguistic theory.

SST-2: Stanford Sentiment Treebank (Socher et al. 2013). The task is to
determine if the sentence is positive or negative in sentiment. The dataset
contains 67k train examples from movie reviews.

MRPC: Microsoft Research Paraphrase Corpus (Dolan and Brockett 2005). The task
is to predict whether two sentences are semantically equivalent or not. The
dataset contains 3.7k train examples from online news sources.

STS-B: Semantic Textual Similarity (Cer et al. 2017). The task is to predict
how semantically similar two sentences are on a 1-5 scale. The dataset
contains 5.8k train examples drawn from news headlines, video and image
captions, and natural language inference data.

QQP: Quora Question Pairs (Iyer, Dandekar, and Csernai 2017). The task is to
determine whether a pair of questions are semantically equivalent. The dataset
contains 364k train examples from the community question-answering website
Quora.

MNLI: Multi-genre Natural Language Inference (Williams, Nangia, and Bowman
2017). Given a premise sentence and a hypothesis sentence, the task is to
predict whether the premise entails the hypothesis, contradicts the
hypothesis, or neither. The dataset contains 393k train examples drawn from
ten different sources.

QNLI: Question Natural Language Inference; constructed from SQuAD (Rajpurkar
et al. 2016). The task is to predict whether a context sentence contains the
answer to a question sentence. The dataset contains 108k train examples from
Wikipedia.

RTE: Recognizing Textual Entailment (Giampiccolo et al. 2007). Given a premise
sentence and a hypothesis sentence, the task is to predict whether the premise
entails the hypothesis or not. The dataset contains 2.5k train examples from a
series of annual textual entailment challenges.


# Trends at NIME – Reflections on Editing “A NIME Reader”

Alexander Refsum Jensenius

Michael J. Lyons

University of Oslo, Dept. of Musicology  Oslo, Norway
<EMAIL_ADDRESS>Ritsumeikan University  Kyoto, Japan
<EMAIL_ADDRESS>
(January, 2016)

###### Abstract

This paper provides an overview of the process of editing the forthcoming
anthology “A NIME Reader—Fifteen years of New Interfaces for Musical
Expression.” The selection process is presented, and we reflect on some of the
trends we have observed in re-discovering the collection of more than 1200
NIME papers published throughout the 15 yearlong history of the conference. An
anthology is necessarily selective, and ours is no exception. As we present in
this paper, the aim has been to represent the wide range of artistic,
scientific, and technological approaches that characterize the NIME
conference. The anthology also includes critical discourse, and through
acknowledgment of the strengths and weaknesses of the NIME community, we
propose activities that could further diversify and strengthen the field.

###### keywords:

NIME, proceedings, history, community, reflection

††conference: NIME’16, July 11-15, 2016, Griffith University, Brisbane,
Australia.

1998H.5.5 [Information Interfaces and Presentation] Sound and Music Computing

##  1 Introduction

Over its 15 years of existence, NIME has become an important conference
series. Each year NIME is the meeting point of researchers, developers,
designers, and artists from all over the world. Even though participants come
from widely different backgrounds, they share a mutual interest in
groundbreaking music and technology.

NIME has grown from a small workshop at CHI in 2001 [11], to a large-scale
conference with up to 500 submissions per year. In fact, more than 1200 papers
have been published through the conference so far, and, staying true to the
open and inclusive atmosphere of the community, all of the papers are freely
available online.111http://www.nime.org/archive/ The archive is great if you
know what to look for, but it has grown to a size that is difficult to handle
for newcomers to the field. Even for long-timers and occasional visitors, it
is difficult to get an overview of the history and development of the
community.

As frequent participants at NIME, but also as co-founder (Lyons) and current
chair of the Steering Committee (Jensenius), we are happy to see that a
growing number of papers focus on historical, theoretical and reflective
studies of the NIME community itself. We believe this is a healthy next step
for the community to move on to become a field in its own right. As the level
of meta-studies started to grow, we began to see the potential for a
collection of articles to broadly represent the conference series. This
thought has now materialized in the anthology “A NIME Reader—Fifteen years of
New Interfaces for Musical Expression,” to be published by Springer Verlag in
2016 [8]. This anthology, which will refer to as Reader in the rest of this
paper, consists of 30 selected papers from the conference series, and with two
new commentaries written for each paper.

The Reader is intended for everyone interested in the field, ranging from
newcomers who just want to get an overview of some of the recent advances in
music technology, including important interfaces, technologies and artistic
outcomes, to researchers interested in knowing about the meta-discussions and
reflections in the field. The selection of papers reflects both the depth and
the width of the publication archive, and the commentaries that follow each
paper have added to the value of the original papers while at the same time
help bring some important underlying discussions alive throughout the book.

The current paper is offered with several intentions. First of all we want to
share openly with the community the selection process we underwent when
choosing the 30 articles for the book as well as our experience going through
the entire NIME archive as part of this process. We also want to highlight
some of the insights gained as part of the selection, review, and peer
commentary processes, some of which are well known, but some of which took us
by surprise. From this we propose some actions that respond to issues and
insights resulting from the editing project. Finally, we want to encourage
others to take up meta-study and anthology projects of their own in order to
“digest” the (already massive) growing literature of NIME.

##  2 The Selection Process

By nature, an anthology is a limited selection of chapters, and for this
project we settled on a limit of 30 articles from the NIME archive. This
number was arrived at by estimating the page count of the resulting volume,
and it also fitted well with the idea of including approximately two published
items for each year of the NIME conference up to and including the most recent
edition in 2015.

When we set out to create this volume, neither of us doubted it would be a
challenging task. After all, how does one select 30 papers from more than 1200
items that have been published in the NIME proceedings, as of 2015? How could
we possibly fairly represent the energetic and creative output in the field of
musical interface research, post-NIME? From the outset, the NIME community has
intentionally prioritized a diversity of research styles and approaches. The
conference has also striven to offer an environment that can attract the
participation not only of researchers working in an academic or institutional
context, but also independent artists, researchers and inventors. Moreover,
each of us has our own subjective interests and tastes as to what we consider
significant prior work.

We started the selection process by each creating an initial list of 50 or so
articles perceived as “influential” by the community. This was done by
identifying the most-cited NIME papers in the Google Scholar index. There are
other bibliometric sources, but we found it important to use an openly
available index. Naturally, older papers tend to have more citations than new
ones, so we also considered a measure of the number of citations-per-year, by
dividing the total citation count by the number of years since publication.
Both measures were used to create our “objective,” or more accurately “multi-
subjective,” initial list of works that have enjoyed impact.

From this first list of well-cited works, which at least to some extent tells
us that other researchers have shown interest in a particular paper, we
systematically examined the content of each one. We each separately created
our lists of papers to include. In many cases, we agreed fully, but for others
discussion, and sometimes multiple discussions, were needed before we reached
agreement. At the same time, we found that the initial list had gaps in the
coverage of some topics and research approaches. For example, work that is
primarily artistic in nature is usually not as highly cited as a report on an
important technological advance. This does not imply that artistic research
papers are less excellent, important, or influential on the direction of
subsequent work, but there is just a lower number of active researchers with a
primarily artistic focus. This led each of us to propose some works that have
not been cited as highly as the others, but which we believe represent some of
the diversity of the NIME corpus. Much discussion was also needed to reach
agreement on which of these were to be included in the final list.

Needless to say, we have not been able to include all of the papers that we
believe deserve a place in a NIME anthology. The final selection reflects a
compromise to keep within the limit of just 30 articles. This is also why it
is consciously titled “ _A_ NIME Reader” and not “ _The_ NIME Reader.” We hope
that there will be other anthology projects drawing on the extensive NIME
proceedings in the future. That is also why we towards the end of this article
suggest projects that may be of interest for future work.

The final selection of papers is presented in Table 1. The list covers the
entire history of the NIME conferences, although we for different reasons were
not able to include exactly two papers for each year.

Table 1: List of NIME papers included in the anthology Year/Item |  Title |  Author(s)
---|---|---
2001a |  Principles for Designing Computer Music Controllers |  Cook
2001b |  Problems and Prospects for Intimate Musical Control of Computers |  Wessel & Wright
2002a |  The importance of parameter mapping in electronic instrument design |  Hunt, Wanderley & Paradis
2002b |  Multimodal Interaction in Music Using the Electromyogram and Relative Position Sensing |  Tanaka & Knapp
2002c |  The Plank: Designing a Simple Haptic Controller |  Verplank, Gurevich & Mathews
2003a |  Contexts of Collaborative Musical Experiences |  Blaine & Fels
2003b |  Sonigraphical Instruments: From FMOL to the reacTable* |  Jordà
2003c |  Designing, Playing, and Performing with a Vision-based Mouth Interface |  Lyons, Haehnel & Tetsutani
2003d |  OpenSound Control: State of the Art 2003 |  Wright, Freed & Momeni
2004a |  The Electronic Sitar Controller |  Kapur, Lazier, Davidson, Wilson & Cook
2004b |  PebbleBox and CrumbleBag: Tactile Interfaces for Granular Synthesis |  O’Modhrain & Essl
2004c |  Toward a Generalized Friction Controller: From the Bowed String to Unusual Musical Instruments |  Serafin & Young
2004d |  On-the-fly Programming: Using Code as an Expressive Musical Instrument |  Wang & Cook
2005a |  Towards a Dimension Space for Musical Devices |  Birnbaum, Fiebrink, Malloch & Wanderley
2005b |  The Overtone Violin |  Overholt
2006a |  Sensemble: A Wireless, Compact, Multi-User Sensor System for Interactive Dance |  Aylward & Paradiso
2006b |  Mobile Music Technology: Report on an Emerging Community |  Gaye, Holmquist, Behrendt & Tanaka
2007a |  Wireless Sensor Interface and Gesture-Follower for Music Pedagogy |  Bevilacqua, Guédy, Schnell, Fléty & Leroy
2007b |  Don’t Forget the Laptop: Using Native Input Capabilities for Expressive Musical Control |  Fiebrink, Wang & Cook
2007c |  Expression and its Discontents: Toward an Ecology of Musical Creation |  Gurevich & Treviño
2007d |  The Acoustic, the Digital and the Body: A Survey on Musical Instruments |  Magnusson & Hurtado
2008 |  Eight Years of Practice on the Hyper-Flute: Technological and Musical Perspectives |  Palacio-Quintin
2009 |  A History of Hemispherical Speakers at Princeton, Plus a DIY Guide |  Smallwood, Cook, Trueman & McIntyre
2011 |  Satellite CCRMA: A Musical Interaction and Sound Synthesis Platform |  Berdahl & Ju
2012a |  The Fingerphone: a Case Study of Sustainable Instrument Redesign |  Freed
2012b |  To be inside someone else’s dream: Music for Sleeping & Waking Minds |  Ouzounian, Knapp, Lyon & DuBois
2012c |  TouchKeys: Capacitive Multi-Touch Sensing on a Physical Keyboard |  McPherson
2013 |  The Web Browser As Synthesizer And Interface |  Roberts, Wakefield & Wright
2014 |  To Gesture or Not? An Analysis of Terminology in NIME Proceedings 2001–2013 |  Jensenius
2015 |  Fourteen Years of NIME: The Value and Meaning of ‘Community’ in Interactive Music Research |  Marquez-Borbon & Stapleton

##  3 Peer-Commentary Process

An important feature of the Reader is the inclusion of not only articles
selected to introduce and represent NIME research, but commentaries on those
articles written both by the authors themselves as well as by other
knowledgeable participants in the NIME community (henceforth “experts” or
“peers.”) Each article is accompanied by one commentary by one or several of
the original authors and one expert commentary. Moreover, authors and peers
had a chance to exchange feedback on each other’s commentary. The feedback
ranged from simple corrections to enthusiastic discussions of philosophical
issues and resulted in revision of the commentary texts in the majority of
cases. We, as editors, took part in this process both by facilitating and
moderating the exchanges, as well as in providing suggestions and commentaries
ourselves. We found that the peer-commentary procedure, inspired by Stevan
Harnad’s thinking and writing on the subject [5], really brought our project
to life: new ideas and insights in the peer commentaries and in the exchanges
between authors and experts showed how the selected articles continue to live,
function, and contribute growth in the research community.

Note however, that the peer-commentary process is not the same thing as peer-
review, that is, the use of single or double blind peer comments aimed at
improving the papers. All of the included articles have already undergone a
peer-review process in order to be published at NIME, so we agreed that the
papers should not be updated or expanded, except for updating/removing broken
URLs and making formatting adjustments. We also found that carrying out peer-
review and selection of the commentaries would be overkill, as the
commentaries were mainly seen as a way to create discussion and engagement.

Readers may wonder about the conditions of the commentary process: how were
the experts selected, and how was the commentary elicited? As for the
selection of articles, the choice of commentators was negotiated between the
two editors, through creating and discussing initial lists. Many, but not all,
of the commentators selected were perceived to be frequent “users ” or
supporters of a given article, for having cited it frequently. In other cases,
commentators were selected to increase the inclusiveness (we admit this is
difficult to objectively define) of the participants in the anthology, or to
involve regular conference participants who were not otherwise involved in the
project. Here again, limitations of the scale of the project meant that we
were not able to involve everyone who came to mind and we recognize that many
who have much to contribute were not included as commentators. By the same
token, some we initially approached were not able to join as commentators for
various reasons.

##  4 Observed Trends

The lengthy process of selecting articles from the extensive NIME conference
proceedings as well as through organizing and moderating the peer-commentary
activity, afforded an opportunity to reflect on trends that the NIME community
has experienced in the past decade and a half. Some of these trends are well-
established, others are still developing, while others again are still in
their infancy. While detailed description and discussion of these is beyond
the scope of the current paper, our intention in this section is to outline
some of the major trends we have observed. In the following, each point is
supported by references to articles from the Reader, as indexed in Table 1.

###  4.1 Toy to Instrument

The NIME community has greatly benefited from the tremendous progress in human
interface technology over the past two decades. Not surprisingly, many of the
articles in the Reader introduce, or make use of, improved sensor technologies
(e.g. 2002c, 2003b,c; 2004a,c; 2005b, 2006a, 2007a, 2012c). By the same token,
interface micro-controllers for data acquisition and device control have
improved in every aspect and become easier to use, while higher resolution,
faster, and more flexible communication protocols were developed,
standardized, and are now widely in use (e.g. 2003, 2011). Instrument designs
have made more effective and engaging use of the richer, cleaner, high
bandwidth data streams available, with more complex motion-to-synthesis
parameter mappings (e.g. 2001b; 2002a; 2003b).

More generally, the approaches to the design of new musical interfaces have
become better informed and employ increasingly sophisticated and theoretically
better grounded design strategies (e.g. 2001a; 2003b; 2004b; 2005a; 2007c, d;
2012a; 2014). Indeed, we now perceive this to be the longest and strongest
“trend” or thread in the NIME conference and it shows no sign of abating. More
strictly formal approaches to instrument designs have, in recent years,
employed rigorous machine learning techniques (e.g. 2007a). We acknowledge
this as a highly significant and growing trend at NIME, but have chosen
consciously not to attempt to cover this topic fully in the Reader, but to
leave it as an “advanced topic” for more specialized reviews. Similarly,
progress in computing hardware and software allows increasingly powerful
approaches to sound synthesis (e.g. 2004d; 2009, 2011).

###  4.2 Buttons to Embodiment

A key motivation behind human-interface research of the past two or more
decades has been to move away from the keyboard-mouse-windows concept of HCI
towards more fluid, full-body interaction. NIME is no exception, and indeed
the rapid expansion and progress in this area was one of the factors to which
the conference owes its existence. Accordingly, research into motion sensing
and music making continues to form a significant thread (e.g. 2003b,c; 2004a;
2005b; 2007a). Even more intimately, new musical interfaces have striven to
use signals inside the body, through biosensing technologies (e.g. 2002b;
2012b), or to capture motion of the whole body (e.g. 2003c; 2006a; 2014).

###  4.3 Individual to Community, Stage to Street

One emerging trend in NIME research has been the move from individual
performance to interfaces that put the focus on collaborative creation (e.g.
2003a; 2006a). Likewise, some researchers have explored technologies and
approaches towards supporting group music-making via laptop orchestras (2007b;
2009), web-based performance (2013), and made the technology development
process itself public and collaborative via live coding activities (2004d).
Another strong thread is the use of mobile technology to create music
anywhere, anytime (2006b). This sub-community of NIME emerged so strongly that
it has also spawned independent conferences and workshops, a process that has
now also begun for the live-coding interest group.

###  4.4 “Hacking” to “Professionalism”

Early NIME conferences had an aspect of “Woodstock-like” gatherings of
tinkerers, hackers, and makers. However, the music technology research
community has grown and matured over the years. While a healthy respect for
adhoc, improvised approaches persists, we also see individuals and groups
engage in more long-term and structured development work. This work is often
focused on development as _process_ , with an acknowledgment of both formal
and informal evaluation of the interfaces as an important part of this
process. Part of such evaluation can be that of a larger commitment to
continued performance with new instruments (e.g. 2008) in contrast to the
early preponderance of “demo and die” interfaces.

As part of the increased “professionalism” of the field, we are happy to see
that new interfaces have often inspired crowd-funding campaigns, which again
have lead to the funding of successful companies. A prominent example of
successful commercialization is Smule, which offers music-making apps for
mobile platforms. Similarly, well-supported open source software and hardware
projects have flourished (e.g. 2011, 2012c) and attracted substantial
participation. Introductory courses [4, 6, 9, 12] and outreach activities [1]
have also increased in frequency.

With the increasing maturity of the field, we see that more papers are also
devoted to topics outside the “doing” of NIME, including: pedagogy, history
and various types of reflection on the technologies, the artistic outcomes and
on the community itself.

###  4.5 The NIME Community

The most recent article to be included in the Reader, from NIME 2015, comes
from Adnan Marquez-Borbon and Paul Stapleton [10]. They offer a critical
analysis of the NIME community, concluding that it more closely resembles a
loosely organized _community of interest_ than a well-focused _community of
practice_. Furthermore, they observe that a lack of a mechanism for judging
what can be considered a “good” NIME performance may inhibit progress in the
field. Their commentary to their own article goes further to suggest that a
perceived lack of “coherent vision and critical reflection” may be
marginalizing or excluding those who are not part of the “dominant social
norm.”

Critical discussions of the NIME conferences (and community) have, of course,
taken place for many years—in informal discussions among members, in the
conference Steering Committee, and also at the annual “town hall” meetings at
the last day of the conference. However, the appearance of formal critical
discourse as an official component of the academic program, such as that of
Marquez-Borbon and Stapleton, marks maturity of the conference and an
increasing self-awareness of NIME. Such considerations suggest that it may be
time to engage in research into various aspects of the NIME community.

Many approaches can be envisaged here, from scientometric analyses of the NIME
proceedings, such as Jensenius’ study of the terminology of gesture [7]. It
can also be interesting to carry out sociological or ethnographic studies,
such as that of Born’s study of IRCAM [2]. Born and Devine have also recently
studied the gender (im)balance in higher music technology education in Britain
[3], a topic which is also highly relevant for the still male-dominated NIME
community. To start with, however, a modest proposal that arose during the
peer commentary process on the Marquez-Borbon and Stapleton article, is that
it would be valuable to survey the members of the community itself about their
experiences to date and expectations for how NIME should further develop as a
community.

##  5 Future Directions: How can NIME Continue to be Relevant?

In closing we would like to raise the broader question of the continuing
relevance of the NIME conference. With the advent of a number of other
conferences, festivals, and venues for presentation of research and artistic
practice related to music technology, we recognize a need for the community
that has emerged around the conference to reflect on what has been
accomplished over the past decade and a half, and to consider how to move
forward:

  * •

What aspects of this community’s identity are strongest?

  * •

What shortcomings and imbalances need to be addressed?

  * •

What interests and viewpoints have been neglected and how can positive action
be taken to cover these?

  * •

How can we organize what has been accomplished in such a form that it is
accessible and useful for new generations of researchers and artists?

  * •

How can we promote the activities of our community more widely and support
long term activities such as project repositories, development platforms, and
entrepreneurship?

These are questions for everyone who participates in NIME and we hope that our
anthology project and this article will encourage everyone to become active in
the process of reflecting on what has been achieved and contributing to
deciding the directions the community will take in the future.

###  5.1 “Digesting” the past

One way in which to focus the development of the community itself is through
projects that “digest” past and ongoing work within the field. Our anthology
is one example of such a project. We would like to suggest projects that go
beyond our Reader. For example, it would be interesting to create a platform
that allows ongoing, open (but moderated), peer-commentary of articles
selected into the NIME proceedings. This would allow dialogue on all works to
continue openly throughout the year, after the intense few days of the annual
conference have ended. Participation could be made voluntary, but we can
imagine that this would be attractive for most researchers if it could be made
to function smoothly. Such ongoing dialogue on a project could help with:

  * •

Nurturing ideas: many NIME papers are fairly terse and have only room to
present one (or a few) core ideas of a larger picture. It would be useful to
create a space in which ideas can be expanded, generating new insights,
suggesting new research directions, and supporting community-building.

  * •

Discovering pre-existing ideas and works: the NIME publication archive is
already an incredibly rich resource for discovering neglected ideas and works,
but augmenting this with peer-commentary would serve to make the material more
transparent and accessible.

  * •

Educating new NIMErs: as a still relatively new community, with few books and
the lack of a long history or consensus, newcomers to the field are often
bewildered as to where to start looking for information. There is clearly a
need to develop material that is useful in education but also for researchers
in other fields visiting the NIME community. Such material could be a useful
spin-off of an ongoing peer-commentary platform.

Another idea is to create an active, living repository of musical interface
designs, which allow creators to share code, hardware specifications, data,
and other material associated with a given project. This would work towards
treating the “demo and die” syndrome, and serve as a valuable resource also
for composers and performers who would like to leverage already developed
technology for artistic purposes.

###  5.2 Anthologies to Come

It is important to stress that even though we have made a collection now, the
aim has never been to create the ultimate NIME paper selection, or a
definitive “canon” of relevant work. We would therefore like to provide some
concrete suggestions for others interested in conducting a review or creating
an anthology:

  * •

Anthologies on more specialized topics, for example on mobile or web
technologies, machine learning, or on the importance of tactility, artistic
use, performance considerations, etc.

  * •

Anthologies based on a particular theoretic thrust, such as gesture, liveness,
community development, pedagogy etc.

  * •

Anthologies based on works that have not received much attention, for example,
least cited works, odd technologies, or comical works?

In conclusion, we hope that our collection will be the first of many. After
all, the NIME proceedings archive is a gold mine of good ideas, and a history
of experience and knowledge gained through the dedicated work of many talented
researchers and artists. These should not be forgotten but rather continue to
be used in different ways.

## References

  * [1] F. Bevilacqua, S. Fels, A. R. Jensenius, M. J. Lyons, N. Schnell, and A. Tanaka.  SIG NIME: Music, technology, and human-computer interaction.  In CHI’13 Extended Abstracts on Human Factors in Computing Systems, pages 2529–2532. ACM, 2013.
  * [2] G. Born.  Rationalizing culture: IRCAM, Boulez, and the institutionalization of the musical avant-garde.  Univ of California Press, 1995.
  * [3] G. Born and K. Devine.  Music Technology, Gender, and Class: Digitization, Educational and Social Change in Britain.  Twentieth-Century Music, 12(02):135–172, 2015.
  * [4] S. Fels and M. Lyons.  Creating new interfaces for musical expression: Introduction to NIME.  In ACM SIGGRAPH 2009 Courses, page 11. ACM, 2009.
  * [5] S. Harnad.  The invisible hand of peer review.  Exploit Interactive, 5(April), 2000.
  * [6] A. R. Jensenius.  An Action-Sound Approach to Teaching Interactive Music.  Organised Sound, 18(2):178–189, 2013.
  * [7] A. R. Jensenius.  To gesture or not? An analysis of terminology in NIME proceedings 2001– 2013.  In Proceedings of the International Conference on New Interfaces For Musical Expression, pages 217–220, London, 2014.
  * [8] A. R. Jensenius and M. Lyons, editors.  A NIME Reader—Fifteen years of New Interfaces for Musical Expression.  Springer, Berlin, Forthcoming.
  * [9] M. J. Lyons, A. Mulder, and S. Fels.  Introduction to designing and building musical interfaces.  In Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems, pages 1015–1016. ACM, 2014\.
  * [10] A. Marquez-Borbon and P. Stapleton.  Fourteen Years of NIME: The Value and Meaning of ‘Community’ in Interactive Music Research.  In Proceedings of the International Conference on New Interfaces for Musical Expression, 2015.
  * [11] I. Poupyrev, M. J. Lyons, S. Fels, and T. Blaine.  New interfaces for musical expression.  In Extended Abstracts of CHI ’01, Seattle, USA, 2001.
  * [12] G. Tzanetakis, S. Fels, and M. Lyons.  Blending the physical and the virtual in music technology: from interface design to multi-modal signal processing.  In Proceedings of the 21st ACM international conference on Multimedia, pages 1119–1120. ACM, 2013.