#  IOT: Instance-wise Layer Reordering for Transformer Structures

Jinhua Zhu1,∗, Lijun Wu2, , Yingce Xia2, Shufang Xie2,
Tao Qin2, Wengang Zhou1, Houqiang Li1, Tie-Yan Liu2
1University of Science and Technology of China;
2Microsoft Research;
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contribution and corresponding authors.

###### Abstract

With sequentially stacked self-attention, (optional) encoder-decoder
attention, and feed-forward layers, Transformer achieves big success in
natural language processing (NLP), and many variants have been proposed.
Currently, almost all these models assume that the _layer order_ is fixed and
kept the same across data samples. We observe that different data samples
actually favor different orders of the layers. Based on this observation, in
this work, we break the assumption of the fixed layer order in Transformer and
introduce instance-wise layer reordering into model structure. Our Instance-
wise Ordered Transformer (IOT) can model variant functions by reordered
layers, which enables each sample to select the better one to improve the
model performance under the constraint of almost same number of parameters. To
achieve this, we introduce a light predictor with negligible parameter and
inference cost to decide the most capable and favorable layer order for any
input sequence. Experiments on $3$ tasks (neural machine translation,
abstractive summarization, and code generation) and $9$ datasets demonstrate
consistent improvements of our method. We further show that our method can
also be applied to other architectures beyond Transformer. Our code is
released at Github111https://github.com/instance-wise-ordered-transformer/IOT.

##  1 Introduction

Transformer (Vaswani et al., 2017) has been the dominant architecture in deep
learning models (Hassan et al., 2018; Ng et al., 2019; Carion et al., 2020;
Radford et al., 2019; Dai et al., 2019; Lee et al., 2019; Devlin et al., 2018;
Yang et al., 2019; Cai & Lam, 2019). A Transformer model is stacked by several
identical blocks, and each block consists of sequentially ordered layers: the
self-attention (SA), encoder-decoder attention (ED) (decoder only) and feed-
forward (FF) layer. Recently, various modifications have been proposed, where
the focus is on replacing or inserting some components (e.g., attention
layer/layer norm/position encoding) in standard Transformer (Wu et al., 2019;
Lu et al., 2019; Shaw et al., 2018; So et al., 2019; Ahmed et al., 2017).

Order | De$\to$En | Ratio
---|---|---
1:SA$\to$ED$\to$FF | $34.64$ |  $17.9$%
2:FF$\to$SA$\to$ED | $34.60$ |  $17.9$%
3:ED$\to$FF$\to$SA | $34.75$ |  $16.5$%
4:ED$\to$SA$\to$FF | $34.67$ |  $17.5$%
5:SA$\to$FF$\to$ED | $34.76$ |  $14.1$%
6:FF$\to$ED$\to$SA | $34.78$ |  $16.1$%
Variance | $0.0045$ | -

Table 1: Results for different decoder orders on IWSLT14 De$\to$En
translation.

Despite these Transformer alternatives have achieved improved performances,
one critical element is almost neglected in current models, which is how to
arrange the components within a Transformer network, i.e., the _layer order_
also matters. As pointed by He et al. (2016b), different orders of ReLU, batch
normalization and residual connection significantly affect the performance of
ResNet (He et al., 2016a). Therefore, we ask: What if we reorder the
sequential layers in Transformer (e.g., SA$\to$FF or FF$\to$SA of encoder,
SA$\to$FF$\to$ED or FF$\to$ED$\to$SA of decoder)? What is the best order for
these different layers?

We first conduct preliminary experiments. We vary the three layers in decoder
with all six variants (each with a unique order of the three layers) and train
these models. Results on IWSLT14 German$\to$English translation are reported
in Table 1. As we can see, their performances are similar and no one is
outstanding. The corpus BLEU variance is only $0.0045$, which means that
simply reordering the layers and training over the whole corpus impacts
little. Press et al. (2019) also reported this for machine translation, but
they stopped here.

This seems to be a negative answer. However, we take a further step and ask
one more question: Does different data favor different ordered layers? That
is, we investigate whether each specific data has its own preference for one
particular order. Intuitively, putting various data patterns in one order
should not be the best choice. For example, harder samples may favor a
particular order while easier ones favor another one. Thus, for each order, we
count the ratio of samples that achieve the best score with that order. In
Table 1, we find they almost lie on a uniform distribution (e.g., $17.9$%
samples achieve the best BLEU with order SA$\to$ED$\to$FF). Besides, we
calculate the BLEU variance for each sample, and average all these variances,
the result is $114.76$, which is much larger than above corpus variance
($0.0045$). These both mean the data indeed has its own preference to
different orders. In Table 2, we present translations from all decoders for on
example with BLEU and TER score to give an evidence.

Reference | and just like that , the iceberg shows you a different side of its personality . | BLEU$\uparrow$ | TER$\downarrow$
---|---|---|---
Order 1 Trans | and just so , the iceberg shows a different side of its personality . | $77.11$ | $18.75$
Order 2 Trans | and just like that , the iceberg shows you a different side of its personality . | $100.00$ | $0.00$
Order 3 Trans | and just so , the iceberg gives you another side of his personality . | $0.00$ | $37.50$
Order 4 Trans | and just like this , the iceberg gives you another side of its personality . | $38.71$ | $25.00$
Order 5 Trans | ans so simply , the iceberg shows another side of his personality . | $30.33$ | $50.00$
Order 6 Trans | and just like this , the iceberg shows you another side of his personality . | $36.61$ | $25.00$

Table 2: Translations (_Trans_) from all ordered decoders of Transformer for
one example sentence.

Motivated by above observations, in this work, we present Instance-wise
Ordered Transformer (IOT), in which the layer order is determined by the
specific data through instance-wise learning. To achieve this, we utilize a
light predictor to predict the confidence for each order, given the
corresponding classification losses as training signals. However, directly
training the predictor with conventional (i.e., NMT) loss tends to quickly
converge to a bad order, and ignore explorations on others. Thus, we introduce
an exploration loss and an exploitation loss to make an effective training
while keeping an unambiguous prediction for each data so that the best order
can be decided during inference.

We evaluate our approach on $3$ sequence generation tasks, including neural
machine translation (NMT), abstractive summarization (ABS) and code generation
(CG). For NMT, we work on $8$ IWSLT and $2$ WMT tasks, both on low-resource
and rich-resource scenarios. Our method can consistently obtain $1.0$ BELU
score improvements over Transformer. For ABS, IOT also outperforms Transformer
and other baselines on Gigaword dataset. For CG tasks, the results on $2$
large-scale real-world code datasets (Java and Python) collected from Github
surpass the state-of-the-art performances. These all demonstrate the
effectiveness of our IOT. Furthermore, we provide detailed studies to verify
that the instance-wise learning and order selection make a reasonable and
necessary modeling.

The contributions of this work can be summarized as follows:

  * •

We are the first to leverage instance-wise learning for layer order selection
in a Transformer model (with shared parameters), and we demonstrate the
instance-wise learning is critical.

  * •

We demonstrate our learning approach can be universally applied to other
structures beside Transformer (e.g., Dynamic Convolutions), as long as there
are multiple different layers.

  * •

Experiments on $3$ sequence generation tasks and $9$ datasets verify the
effectiveness of IOT with consistent performance improvements.

##  2 Related Work

Architecture Exploration Inventing novel architectures by human designing or
automatic searching plays an important role in deep learning. Specific to
Transformer structures, various modifications have been proposed. For example,
human knowledge powered designs include DynamicConv (Wu et al., 2019), Macaron
Network (Lu et al., 2019), Reformer (Kitaev et al., 2020) and others
(Fonollosa et al., 2019; Ahmed et al., 2017; Shaw et al., 2018). As for
automatic searching, neural architecture search can discover networks with
state-of-the-art performances but always with complicated computation, i.e.,
Evolved Transformer (So et al., 2019). The underlying principle is to add or
replace some components of Transformer. For instance, Wu et al. (2019) replace
self-attention with dynamic convolution, So et al. (2019) add a separate
convolution layer in a new branch. Different from them, we, instead, only
focus on the selection of layer orders for each data sample so as to improve
the model performance, without a heavy modification. Besides, our approach is
structure agnostic, which can be universally applied to other structures, only
if multiple different layers exist.

Instance-wise Learning Deep learning models are trained over large-scale
datasets, and data samples are often treated equally without modeling the
difference between them. Some works attempt to weight each data with different
importance (Ren et al., 2018; Hu et al., 2019; Chang et al., 2017) or feed
data with curriculum learning according to its difficulty (Bengio et al.,
2009; Fan et al., 2018). However, they often explicitly manipulate the data
during training only, while no distinction exists in inference, and under one
fixed model. Elbayad et al. (2020) take a step further and propose the depth-
adaptive Transformer, which can forecast different depths of the network by
predicting the required computation for a particular data. Similarly, Liu et
al. (2020) propose a sample-wise adaptive mechanism to dynamically calculate
the number of required layers. They both aim at reducing the computation cost
and speed up the inference. Schwartz et al. (2020), Bapna et al. (2020) and
Shazeer et al. (2017) all leverage conditional computation for each sample to
control the computation and accuracy tradeoff during inference. Instead, we
pay attention to the variant modeling functions and perform instance-wise
order selection in order to boost the Transformer performance.

The most related work is Press et al. (2019), which manually generates
randomly ordered Transformer encoders and finds the _Sandwich Transformer_ can
slightly reduce the perplexity of language modeling. However, they find that
Sandwich Transformer pattern has no effect on NMT task. Besides, it still
performs over the whole corpus without considering each specific data. We,
instead, investigate on various sequence-to-sequence generation tasks and
greatly improve the task performances through instance-wise learning, so as to
discover the optimal ordered Transformer for each particular data.

##  3 Instance-wise Ordered Transformer

The overall framework of IOT is presented in Figure 1. In comparison with the
standard Transformer, IOT only incorporates light-weighted predictors and
reorders the encoder/decoder with weight tying, under the constraint of almost
same number of parameters and exempt from heavy modifications. In this
section, we introduce the details of IOT, including training, inference and
discussions.

Figure 1: The IOT framework. _Pred_ means the light predictor introduced in
3.1 for order selection. We show two ordered encoders/decoders here. After
taking $X_{1},X_{2},X_{3}$, the selected order for $Y_{2},Y_{3}$ is the lower
encoder and upper decoder, while for $Y_{1}$ is the upper encoder and lower
decoder.

Notations Sequence-to-sequence learning aims to map one sequence
$x=[x_{1},x_{2},...,x_{T_{x}}]$ into another sequence
$y=[y_{1},y_{2},...,y_{T_{y}}]$, where $x_{i},y_{j}$ denotes the $i$-th and
$j$-th token of $x$ and $y$, $T_{x}$ and $T_{y}$ are the corresponding
lengths. Given one sentence pair $(x,y)$ and a learning model $\mathcal{M}$,
we can define the training objective as minimizing the cross-entropy loss
$\mathcal{L_{M}}=-\sum_{j=1}^{T_{y}}\log P(y_{j}|y_{<j},x)$. Besides,
$D_{KL}(\mathbb{P}\|\mathbb{Q})$ denotes the Kullback-Leibler (KL) divergence
between distributions $\mathbb{P}$ and $\mathbb{Q}$.

###  3.1 Instance-wise Encoder/Decoder

IOT intends to break the fixed order of layers in Transformer. As shown in
introduction, simply reordering the layers w.r.t the whole corpus impacts
little, while each data has its own preference to orders. Therefore, IOT
incorporates instance-wise learning to adjust the favorable order for each
data.

As shown in Figure 1, both encoder and decoder in IOT consist of several
blocks of SA, ED, FF layer with dynamic order, and we assume there are $M$
(e.g., $M=2$) ordered encoders and $N$ (e.g., $N=6$) ordered decoders (with
shared weights). Inspired by the fact that lower training loss implies the
higher proficiency confidence for candidate orders, we utilize the cross-
entropy loss as signals to learn the confidence. That is, we calculate
confidence $\gamma_{m}$ and $\lambda_{n}$ for each encoder encm, decoder decn
(resulted model $\mathcal{M}^{m,n}$), and use them to weight the training loss
$\mathcal{L_{M}}^{m,n}$. To calculate the confidence, we add a simple and
light predictor to help distinguish the orders.

Training Given one source sequence $x=[x_{1},x_{2},...,x_{T_{x}}]$, we first
map each token into word embedding $e=[e_{1},e_{2},...,e_{T_{x}}]$, where
$e_{i}\in\mathbb{R}^{d}$, and then apply one light encoder predictor
$\pi_{\texttt{enc}}$ to predict the confidence of encoder orders using
sentence embedding $s_{e}=\frac{1}{T_{x}}\sum_{i=1}^{T_{x}}e_{i}$. Concretely,
$\pi_{\texttt{enc}}$ takes $s_{e}$ as input and predicts $\gamma_{m}$ for encm
by Gumbel-softmax (Jang et al., 2016):

$\gamma_{m}=\frac{\exp\left(\left(\log\left(\pi_{\texttt{enc}_{m}}\right)+g_{m}\right)/\tau_{e}\right)}{\sum_{k=1}^{M}\exp\left(\left(\log\left(\pi_{\texttt{enc}_{k}}\right)+g_{k}\right)/\tau_{e}\right)},\quad\pi_{\texttt{enc}}=\operatorname{softmax}\left(s_{e}W_{e}\right),$
(1)

where $g_{m}$ is sampled from Gumbel distribution: $g_{m}=-\log(-\log
U_{m}),U_{m}\sim$ Uniform(0, 1), $W_{e}\in\mathbb{R}^{d\times M}$ is the
weight matrix, $\tau_{e}$ is a constant temperature to control the
distribution to be identical approximation with categorical distribution.
Simultaneously, the token embeddings $e$ will feed to the encoders to get
hidden states $h=[h_{1},h_{2},...,h_{T_{x}}]$, then we can calculate decoder
order confidence $\lambda_{n}$ by one predictor $\pi_{\texttt{dec}}$ in the
same way as $\pi_{\texttt{enc}}$:

$\lambda_{n}=\frac{\exp\left(\left(\log\left(\pi_{\texttt{dec}_{n}}\right)+g_{n}\right)/\tau_{d}\right)}{\sum_{k=1}^{N}\exp\left(\left(\log\left(\pi_{\texttt{dec}_{k}}\right)+g_{k}\right)/\tau_{d}\right)},\quad\pi_{\texttt{dec}}=\operatorname{softmax}\left(s_{d}W_{d}\right),$
(2)

where $s_{d}=\frac{1}{T_{x}}\sum_{i=1}^{T_{x}}h_{i}$ and $W_{d}$ is the weight
matrix. For each ordered path through encm and decn, we can obtain the
training loss $\mathcal{L_{M}}^{m,n}$, and the final cross-entropy loss is
weighted by confidence $\gamma_{m}$ and $\lambda_{n}$ with
$\mathcal{L_{M}}^{m,n}$, formulately as:

$\mathcal{L_{C}}=\sum_{m=1}^{M}\sum_{n=1}^{N}(\gamma_{m}\cdot\lambda_{n})\mathcal{L_{M}}^{m,n}.$
(3)

Inference During inference, we directly replace the Gumbel-softmax used in
training with $\operatorname{argmax}$, in order to choose the most capable
encoder and decoder for each sequence $x$:

$\texttt{enc}=\operatorname{argmax}\left(s_{e}W_{e}\right),\quad\texttt{dec}=\operatorname{argmax}\left(s_{d}W_{d}\right).$
(4)

Discussion The decoding process is almost the same as standard Transformer,
with only little overhead for order predictions. One may concern the training
cost is increased through our training. As we present in Section 5.1, the cost
is actually affordable with a fast convergence. Currently, we reorder the
layers of the encoder/decoder block and stack the same ordered block $L$ times
(see Figure 1). A complex extension is to reorder all $L$ blocks of
encoder/decoder and we take it as future work.

###  3.2 Auxiliary Losses

As we can see, the predictors are trained in an unsupervised way, and we
observe they lean to be lazy so that all samples quickly converge to one same
order during training, without a senseful learning. Thus, to make an effective
training and inference, we introduce exploration and exploitation losses.

(1) Exploration: first, we explore the diverse capability of all orders with
help of a loss $\mathcal{L_{D}}$ to encourage all orders to participate in
training. The spirit is the same as to encourage exploration in reinforcement
learning. The expected $\operatorname{softmax}$ probability
$\mathbb{E}_{x}\left[\pi_{x}\right]$ (encoder/decoder) from the predictor
should approximate the uniform distribution
$\mathbb{Q}=[\frac{1}{N},\frac{1}{N},\ldots,\frac{1}{N}]$ (e.g., decoder
orders), and we achieve this by minimizing KL-divergence between the
statistical average $\mathbb{E}_{x}\left[\pi_{x}\right]$ and $\mathbb{Q}$:

$\mathcal{L_{D}}=D_{KL}(\mathbb{Q}\|\mathbb{E}_{x}\left[\pi_{x}\right])=-\frac{1}{N}\sum_{n=1}^{N}\log(\mathbb{E}_{x}\left[(\pi_{x}\right)_{n}])-\log{N},$
(5)

where $(\pi_{x})_{n}$ is the probability of $n$-th decoder order for data $x$.
For encoder order, it is $(\pi_{x})_{m}$ processed in a same way as decoder.

(2) Exploitation: different from $\mathcal{L_{D}}$ to keep all orders
effectively trained, during inference, the output distribution $\pi_{x}$ for
each data should be able to make an unambiguous $\operatorname{argmax}$
selection. We then introduce another loss $\mathcal{L_{S}}$ to constrain each
$\pi_{x}$ to be far away from the uniform distribution $\mathbb{Q}$.
Concretely, we maximize the KL-divergence between each probability $\pi_{x}$
and $\mathbb{Q}$:

$\mathcal{L_{S}}=-\mathbb{E}_{x}\left[D_{KL}(\mathbb{Q}\|\pi_{x})\right]=-\mathbb{E}_{x}\left[-\frac{1}{N}\sum_{n=1}^{N}\log(\pi_{x})_{n}-\log
N\right].$ (6)

Note that we clamp the value of probability $\pi_{x}$ since the KL value is
theoretically unbounded. With above auxiliary losses, the final training
objective is to minimize:

$\mathcal{L}=\mathcal{L_{C}}+c_{1}\mathcal{L_{D}}+c_{2}\mathcal{L_{S}},$ (7)

where $c_{1}$ and $c_{2}$ are coefficients to make a trade-off between
$\mathcal{L_{D}}$ and $\mathcal{L_{S}}$. In this way, we can achieve effective
training, while keeping the ability to distinguish the favorable order for
each data.

Discussion $\mathcal{L_{D}}$ and $\mathcal{L_{S}}$ aim to keep effective
training and unambiguous inference. There are several alternatives. The first
is to simply decay the temperature $\tau$ in Equation (1) and (2), and remove
the auxiliary losses. However, we do not notice obvious gain. Second is to
linearly decay $c_{1}$ only and remove $\mathcal{L_{S}}$, which is able to
fully train all orders at the beginning and loose this constraint gradually.
We find this is also beneficial, but our two losses method performs better.

##  4 Experiments

We conduct experiments on $3$ sequence generation tasks: neural machine
translation (both low-resource and rich-resource), code generation and
abstractive summarization. The main settings of each experiment are introduced
here, and more details can be found in Appendix A.

###  4.1 Dataset

Neural Machine Translation For the low-resource scenario, we conduct
experiments on IWSLT14 English$\leftrightarrow$German (En$\leftrightarrow$De),
English$\leftrightarrow$Spanish (En$\leftrightarrow$Es), IWSLT17
English$\leftrightarrow$French (En$\leftrightarrow$Fr),
English$\leftrightarrow$Chinese (En$\leftrightarrow$Zh) translations. The
training data includes $160k$, $183k$, $236k$, $235k$ sentence pairs for each
language pair respectively. For the rich-resource scenario, we work on WMT14
En$\to$De and WMT16 Romanian$\to$English (Ro$\to$En) translations. For WMT14
En$\to$De, we filter out $4.5M$ sentence pairs for training and concatenate
newstest2012 and newstest2013 as dev set, newstest2014 as test set. For WMT16
Ro$\to$En, we concatenate the $0.6M$ bilingual pairs and $2.0M$ back
translated data222http://data.statmt.org/rsennrich/wmt16_backtranslations/ro-
en/. for training, newsdev2016/newstest2016 serve as dev/test set.

Code Generation Code generation aims to map natural language sentences to
programming language code. We work on one Java (Hu et al., 2018) and one
Python dataset (Wan et al., 2018), following Wei et al. (2019) to process the
two datasets. The Java dataset is collected from Java projects on Github, and
the Python dataset is collected by Barone & Sennrich (2017). We split each
dataset with ratio $0.8:0.1:0.1$ as training, dev and test set.

Abstractive Summarization Abstractive summarization is to summarize one long
sentence into a short one. The dataset we utilized is a widely acknowledged
one: Gigaword summarization, which is constructed from a subset of Gigaword
corpus (Graff et al., 2003) and first used by Rush et al. (2017). The training
data consists of $3.8M$ article-headline pairs, while the dev and test set
consist of $190k$ and $2k$ pairs respectively.

###  4.2 Model and Optimization

For IWSLT translation tasks, we use transformer_iwslt_de_en setting as model
configuration. The number of block, embedding size and feed-forward network
(FFN) size are $6$, $512$, $1024$. WMT tasks use
transformer_vaswani_wmt_en_de_big configuration, with $6$ blocks, embedding
size $1024$ and FFN size $4096$. Optimization and learning scheduler are the
default settings in Vaswani et al. (2017). For code generation, block
number/embedding size/FFN size are $3$, $256$, $1024$ respectively. Others are
the same as NMT. For summarization, we take transformer_wmt_en_de, with $6$
blocks, embedding size $512$ and FFN size $2048$. Dropout (Srivastava et al.,
2014) is set to be $0.3$. Other settings are also the same as NMT task.
Implementation is developed on Fairseq (Ott et al., 2019). We first grid
search $c_{1}$, $c_{2}$ on IWSLT14 De$\to$En dev set, and then apply them on
other tasks. The best setting is $c_{1}=0.1$, $c_{2}=0.01$, and the importance
study of $c_{1}$, $c_{2}$ is shown in Appendix B.1.

###  4.3 Evaluation

We use multi-bleu.perl to evaluate IWSLT14 En$\leftrightarrow$De and all WMT
tasks for a fair comparison with previous works. For other NMT tasks, we use
sacre-bleu for evaluation. During inference, we follow Vaswani et al. (2017)
to use beam size $4$ and length penalty $0.6$ for WMT14 En$\to$De, beam size
$5$ and penalty $1.0$ for other tasks. For code generation, the evaluation is
based on two metrics, the sentence BLEU computes the n-gram precision of a
candidate sequence to the reference, and the percentage of valid code (PoV)
that can be parsed into an abstract syntax tree (AST). As for summarization,
the generated summarization is evaluated by ROUGE-1/2/L F1 score (Lin, 2004).

###  4.4 Main Results

Table 3: Preliminary results of varied orders on IWSLT14 De$\to$En task.

| De$\to$En
---|---
Transformer | $34.64$
Encoder ($M=2,N=1$) | $35.18$
Decoder ($M=1,N=6$) | $35.60$
Encoder$\times$Decoder ($M=2,N=6$) | $35.24$
Encoder$\times$Decoder ($M=2,N=4$) | $35.30$
Encoder$\times$Decoder ($M=2,N=2$) | $35.25$

(a) Results of encoder, decoder orders, and their combinations.

| De$\to$En
---|---
Transformer | $34.64$
IOT ($M=1,N=6$) | $35.60$
IOT ($M=1,N=5$) | $35.65$
IOT ($M=1,N=4$) | $35.62$
IOT ($M=1,N=3$) | $35.58$
IOT ($M=1,N=2$) | $35.32$

(b) Results of varied number of decoder orders.

Encoder/Decoder Orders Encoder block only contains SA and FF layers, the
resulted max number of encoder layer orders $M$ is $2$, while for decoder, the
max order variants $N$ is $6$. Therefore, we first evaluate the utilization of
encoder orders, decoders orders, and both orders on IWSLT14 De$\to$En
translation, in order to see the impacts of different number of order
candidates and their combinations. In Table 4(b) (a), we can see that $2$
ordered encoders improve the result, and $6$ ordered decoders achieve more
gain. This meets our expectation, since the search space is limited when there
are only $2$ ordered encoders. However, if we train both encoder and decoder
orders (e.g., $M=2,N=6$), the results (e.g., $35.30$) can not surpass the $6$
decoders only ($35.60$). We suspect the search space is too large so that
training becomes hard, and decoder orders play a more important role than
encoder orders for sequence generation. Therefore, we turn to investigate
different decoder order candidates (refer to Appendix A.3 for detailed
combinations) in Table 4(b) (b). Results show that $N=4,5,6$ achieve similar
strong performances (results on other tasks/datasets are in Appendix A.4).
Thus, considering the efficiency and improvements, we utilize $N=4$ ordered
decoders (order $1,2,4,6$ in Table 1) to reduce training cost in later
experiments.

Table 4: BLEU scores of IOT on eight IWSLT low-resource translation tasks.

| En$\to$De | De$\to$En | En$\to$Fr | Fr$\to$En | En$\to$Zh | Zh$\to$En | En$\to$Es | Es$\to$En |
---|---|---|---|---|---|---|---|---|---
Transformer | $28.57$ | $34.64$ | $35.9$ | $36.1$ | $26.3$ | $18.4$ | $39.0$ | $40.6$ |
IOT | $\bf{29.52}$ | $\bf{35.62}$ | $\bf{37.2}$ | $\bf{37.8}$ | $\bf{27.2}$ | $\bf{19.3}$ | $\bf{40.1}$ | $\bf{41.7}$ |

Table 5: Results on IWSLT14 De$\to$En translation task (a), Java and Python
code generation tasks (b).

Method | BLEU
---|---
Transformer | $34.64$
IOT | $\bf{35.62}$
Adversarial MLE (Wang et al., 2019a) | $35.18$
DynamicConv (Wu et al., 2019) | $35.20$
Macaron Network (Lu et al., 2019) | $35.40$
MADL (Wang et al., 2019c) | $35.56$

(a) Results on IWSLT14 De$\to$En translation.

| Java |  | Python
---|---|---|---
Method | BLEU | PoV |  | BLEU | PoV
Transformer | $24.58$ | $74.44\%$ |  | $13.20$ | $61.89\%$
IOT | $\bf{25.51}$ | $\bf{77.30\%}$ |  | $\bf{14.05}$ | $\bf{63.14\%}$
Wei et al. (2019) | $17.17$ | $27.4\%$ |  | $12.09$ | $51.9\%$

(b) Results on Java and Python code generations.

NMT Results BLEU scores on $8$ IWSLT low-resource tasks are shown in Table 4.
As we can see, IOT achieves more than $1.0$ BLEU points improvement on all
tasks (e.g., $1.7$ on Fr$\to$En). The consistent gains on various language
pairs well demonstrate the generalization and effectiveness of our method. We
then present comparison with other works on IWSLT14 De$\to$En task in Table
6(b) (a), and IOT is also better than several human designed networks. The
results of WMT14 En$\to$De and WMT16 Ro$\to$En are reported in Table 6. We
also compare with existing works, such as the unsupervised Ro$\to$En based on
pre-trained cross-lingual language model (Lample & Conneau, 2019).

Method | En$\to$De
---|---
Transformer⋆ | $29.12$
IOT | $\bf{30.03}$
Shaw et al. (2018) | $29.20$
Ott et al. (2018) | $29.30$
Wu et al. (2019) | $29.70$
So et al. (2019) | $29.80$
Method | Ro$\to$En
Transformer⋆ | $37.73$
IOT | $\bf{38.83}$
Sennrich et al. (2016) | $33.90$
Lample & Conneau (2019) | $38.50$

Table 6: WMT14 En$\to$De and WMT16 Ro$\to$En translation results. ⋆stands for
our reproduced result.

Similarly, our method outperforms them and shows our framework can work well
on rich-resource scenario.

Code Generation Results The results are shown in Table 6(b)(b). We can observe
that Transformer obtains better result than the LSTM-based work (Wei et al.,
2019). Compared with Transformer, IOT can further improve the quality of
generated code. Specifically, IOT boosts Transformer with $0.93$ BLEU/$2.86\%$
PoV gain on Java generation and $0.75$ BLEU/$1.25\%$ PoV gain on Python
respectively. Again, these results well demonstrate the effectiveness of our
method.

Abstractive Summarization Results The IOT performances on summarization task
are shown in Table 7. From the results, we can see IOT achieves $0.8$, $0.7$
and $1.0$ scores gain of ROUGE-1, ROUGE-2 and ROUGE-L metrics over standard
Transformer on Gigaword summarization. IOT also surpasses other works such as
reinforcement learning based method (Wang et al., 2018), which again verifies
our approach is simple yet effective.

Method | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
Transformer (Vaswani et al., 2017) | $35.59$ | $17.74$ | $32.98$
IOT | $36.37$ | $\bf{18.46}$ | $33.89$
RNNSearch+MRT (Ayana et al., 2016) | $36.54$ | $16.59$ | $33.44$
Concept pointer+DS (Wang et al., 2019b) | $37.01$ | $17.10$ | $34.87$
RNNSearch+select+MTL+ERAML (Li et al., 2018) | $35.33$ | $17.27$ | $33.19$
CGU (Lin et al., 2018) | $36.30$ | $18.00$ | $33.80$
Reinforced-Topic-ConvS2S (Wang et al., 2018) | $36.92$ | $18.29$ | $34.58$

Table 7: ROUGE-1/2/L F1 scores for Gigaword summarization.

##  5 Study and Analysis

###  5.1 Inference/Training Cost

As discussed before, our approach only increases negligible parameters and
inference time cost. Here we compare the detailed inference time and model
size of our framework to the standard Transformer. The detailed parameter
numbers and inference time on IWSLT14 En$\leftrightarrow$De test set are shown
in Table 8. Since we only add one linear layer and $\operatorname{softmax}$
layer as the predictor, the number of extra parameters is $M\times
hidden\\_size$ (encoder predictor) or $N\times hidden\\_size$ (decoder
predictor), which is negligible compared to other model parameters. Therefore,
IOT introduces more model diversity and improves the performance, but under
the constraint of almost same number of parameters. As for the inference time,
the only difference is from the one-pass order prediction and the cost is
extremely low compared with heavy autoregressive generation process, which can
be seen from Table 8.

| En$\to$ De |  | De$\to$En
---|---|---|---
Method | Inference time (s) |  $\\#$parameters |  | Inference time (s) |  $\\#$parameters
Transformer | $1487.19$ | $36741120$ |  | $1432.03$ | $36741120$
IOT $(N=2)$ | $1496.70$ | $36742144$ |  | $1429.26$ | $36742144$
IOT $(N=3)$ | $1470.40$ | $36742656$ |  | $1480.83$ | $36742656$
IOT $(N=4)$ | $1505.83$ | $36743168$ |  | $1444.74$ | $36743168$
IOT $(N=5)$ | $1491.40$ | $36743680$ |  | $1424.10$ | $36743680$
IOT $(N=6)$ | $1479.20$ | $36744192$ |  | $1464.00$ | $36744192$

Table 8: Inference time and model parameters counted for Transformer and our
framework on IWSLT14 En$\leftrightarrow$De. The study is performed on a single
Tesla P100 GPU card.

|  | Transformer | IOT ($N=2$) | IOT ($N=3$)
---|---|---|---|---
En$\to$De | Epoch Time (s) | $277.1$ | $475.6(1.72\times)$ | $685.4(2.47\times)$
Epoch Number | $67$ | $42(0.63\times)$ | $39(0.58\times)$
Total Time (s) | $18565.7$ | $19975.2(1.08\times)$ | $26730.6(1.44\times)$
En$\to$Fr | Epoch Time (s) | $410.5$ | $715.1(1.73\times)$ | $1024.1(2.49\times)$
Epoch Number | $55$ | $37(0.67\times)$ | $33(0.60\times)$
Total Time (s) | $22577.5$ | $26458.7(1.17\times)$ | $33795.3(1.49\times)$
En$\to$Zh | Epoch Time (s) | $349.9$ | $612.3(1.75\times)$ | $931.4(2.66\times)$
Epoch Number | $59$ | $34(0.58\times)$ | $30(0.51\times)$
Total Time (s) | $20644.1$ | $20818.2(1.01\times)$ | $27942.0(1.35\times)$
En$\to$Es | Epoch Time (s) | $310.1$ | $538.0(1.73\times)$ | $784.4(2.53\times)$
Epoch Number | $69$ | $43(0.62\times)$ | $30(0.43\times)$
Total Time (s) | $21369.9$ | $23134.0(1.08\times)$ | $23532.0(1.10\times)$

Table 9: Training cost analysis for Transformer and our IOT on four IWSLT
translation tasks. The study is performed on a single Tesla P100 GPU card.

Apart from the inference cost, one may concern about the training cost since
IOT trains multiple orders in one model. To see the influence, we provide
several statistics here. Specifically, on the four IWSLT En$\to$X translation
tasks, we analyze the cost by counting the training time for each epoch, the
epoch number when model convergences, and the corresponding total training
time. The numbers are presented in Table 9, and we can have several
observations. Take IWSLT14 En$\to$De translation as an example, (1) jointly
optimizing different orders indeed introduces more training cost for each
epoch. Transformer baseline costs $277.1s$ per epoch training, while our IOT
costs $475.6s$ and $685.4s$ with $N=2$ and $N=3$ orders respectively, the
increased cost ratio is about $1.72\times$ and $2.47\times$ (but less than
$2.0$ and $3.0$). (2) However, we find that with the shared parameters between
these orders, the model convergence also becomes faster. Transformer needs
$67$ epochs when converge, while our IOT only needs $42(0.63\times)$ and
$39(0.58\times)$ epochs for $N=2$ and $N=3$ orders, much fewer than
Transformer. (3) The total training cost actually is not increased much. IOT
($N=2$) and IOT ($N=3$) are about $1.08\times$ and $1.44\times$ training time
compared with Transformer baseline (the ratio for IOT ($N=3$) is only $1.10$
on IWSLT17 En$\to$Es). From these observations, we can see that the increased
training cost is affordable due to the fast convergence.

###  5.2 Case Verification

We perform a study with $N=3$ to verify that IOT has made a necessary
instance-wise order selection. We first split IWSLT14 En$\leftrightarrow$De
dev set into $3$ subsets according to the prediction of $\pi_{\texttt{dec}}$,
and then we decode each subset use all $3$ ordered decoders, and report the
BLEU results. As shown in Figure 2, each subset indeed achieves the best score
on the corresponding predicted order (outperforms other orders by $0.2$-$0.4$
BLEU). We also do the same study on the test set, and the predicted order
outperforms others by $0.7$-$0.8$ BLEU. These well prove that IOT makes a
reasonable prediction.

Besides, we find that the predicted orders correlate to different sentence
difficulties. In our case, the set $1$ sentences belong to decoder $1$ achieve
highest BLEU than other sets, which means set $1$ is relatively simple to
translate, and vice versa for samples in set $2$. These imply that different
difficulty sentences have different structure preferences. We provide
statistics and examples in Appendix B.2.

(a) BLEU scores on IWSLT14 En$\to$De. (b) BLEU scores on IWSLT14 De$\to$En.

Figure 2: BLEU scores of three subsets (divided by predicted decoder) on all
decoders.

###  5.3 Apply on Another Structure (DynamicConv)

As we discussed, our instance-wise layer reordering is structure agnostic. In
this subsection, we evaluate this by applying our approach on DynamicConv
network (Wu et al., 2019) beyond standard Transformer, which replaces the
self-attention with dynamic convolution. We train layer ordered DynamicConv on
$N=2,3,4$ decoders and test the performances. The BLEU score of standard
DynamicConv is $35.20$, and with our instance-wise order learning, we achieve
$35.60,35.82,35.87$ for $N=2,3,4$ ordered decoders respectively (near $0.7$
point gain). Therefore, this study verifies our claim that our approach can be
applied to other structures, as long as multiple different layers exist.

###  5.4 Discussions

Ensemble Models | En$\to$De | De$\to$En
---|---|---
1-model (standard) | $28.57$ | $34.67$
2-model (standard) | $29.71$ | $35.92$
3-model (standard) | $30.08$ | $36.40$
4-model (standard) | $30.18$ | $36.54$
1-model (IOT) | $29.52$ | $35.62$
2-model (IOT) | $30.38$ | $36.80$
3-model (IOT) | $30.93$ | $37.25$
4-model (IOT) | $31.02$ | $37.38$

Table 10: Ensemble performances of standard Transformer and IOT.

Ensemble Since our framework involves multiple orders (with shared
parameters), which is also done in ensemble framework, we make a comparison
with ensemble. The ensemble method trains multiple models with different
parameters separately in an independent way. While our work trains orders in a
joint way with an intention to make them more diverse. More importantly, from
the view of time and memory cost, the ensemble framework increases $N$ times
which is totally different from ours. In this sense, our method can be
combined with ensemble to further boost performance. The competitive results
on IWSTL14 En$\leftrightarrow$De test set are shown in Table 10. We can
clearly conclude that IOT and ensemble are complementary to each other.

Regularization IOT consists of different ordered blocks in a weight tying
method, which may looks like a parameter regularization to some extent.
However, we show that IOT is more than regularization and can be complementary
with other regularization methods. Setting (1): We first train a Transformer
model on IWSLT14 De$\to$En task with all shared decoder orders, but without
instance-wise learning, and test the performance with each order. We find the
BLEU scores on test set are near $34.80$ for each order, much worse than IOT,
which means that simply regularizing the shared parameters for different
orders is not the main contribution to performance improvement, and our
instance-wise learning is critical. Setting (2): Another experiment is that we
train Transformer with LayerDrop (Fan et al., 2019), a dropout technique to
regularize the layer parameters. The test BLEU is $35.40$, which achieves
about $0.8$ score improvement over Transformer. After applying IOT with
LayerDrop, we obtain further gains than IOT only ($35.62$) to reach a BLEU
score $36.13$. Therefore, this demonstrates IOT is not only regularization and
can be smoothly integrated with other regularization methods. More details and
experiments on other tasks are shown in Appendix B.3.

##  6 Conclusion

In this work, we propose Instance-wise Ordered Transformer, which leverages
instance-wise learning to reorder the layers in Transformer for each data.
Compared with standard Transformer, IOT only introduces slightly increased
time cost. Experiments on $3$ sequence generation tasks and $9$ datasets
demonstrate the effectiveness of IOT. We also verify that our approach can be
universally applied to other structures, such as DynamicConv. In future, we
plan to work on more complicated reordering in each block, as well as other
tasks such as multi-lingual translation and text classification.

## References

  * Ahmed et al. (2017) Karim Ahmed, Nitish Shirish Keskar, and Richard Socher.  Weighted transformer network for machine translation.  _arXiv preprint arXiv:1711.02132_ , 2017.
  * Ayana et al. (2016) Shiqi Shen Ayana, Zhiyuan Liu, and Maosong Sun.  Neural headline generation with minimum risk training.  _arXiv preprint arXiv:1604.01904_ , 2016.
  * Bapna et al. (2020) Ankur Bapna, Naveen Arivazhagan, and Orhan Firat.  Controlling computation versus quality for neural sequence models.  _arXiv preprint arXiv:2002.07106_ , 2020.
  * Barone & Sennrich (2017) Antonio Valerio Miceli Barone and Rico Sennrich.  A parallel corpus of python functions and documentation strings for automated code documentation and code generation.  _arXiv preprint arXiv:1707.02275_ , 2017.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pp. 41–48, 2009.
  * Cai & Lam (2019) Deng Cai and Wai Lam.  Graph transformer for graph-to-sequence learning.  _arXiv preprint arXiv:1911.07470_ , 2019.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  _arXiv preprint arXiv:2005.12872_ , 2020.
  * Chang et al. (2017) Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum.  Active bias: Training more accurate neural networks by emphasizing high variance samples.  In _Advances in Neural Information Processing Systems_ , pp. 1002–1012, 2017.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 2978–2988, 2019.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Elbayad et al. (2020) Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli.  Depth-adaptive transformer.  In _International Conference on Learning Representations_ , 2020.
  * Fan et al. (2019) Angela Fan, Edouard Grave, and Armand Joulin.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_ , 2019.
  * Fan et al. (2018) Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu.  Learning to teach.  In _International Conference on Learning Representations_ , 2018.
  * Fonollosa et al. (2019) José AR Fonollosa, Noe Casas, and Marta R Costa-jussà.  Joint source-target self attention with locality constraints.  _arXiv preprint arXiv:1905.06596_ , 2019.
  * Graff et al. (2003) David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda.  English gigaword.  _Linguistic Data Consortium, Philadelphia_ , 4(1):34, 2003.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al.  Achieving human parity on automatic chinese to english news translation.  _arXiv preprint arXiv:1803.05567_ , 2018.
  * He et al. (2016a) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016a.
  * He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In _European conference on computer vision_ , pp. 630–645. Springer, 2016b.
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin.  Summarizing source code with transferred api knowledge.  2018\.
  * Hu et al. (2019) Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing.  Learning data manipulation for augmentation and weighting.  In _Advances in Neural Information Processing Systems_ , pp. 15738–15749, 2019.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_ , 2020.
  * Lample & Conneau (2019) Guillaume Lample and Alexis Conneau.  Cross-lingual language model pretraining.  _arXiv preprint arXiv:1901.07291_ , 2019.
  * Lee et al. (2019) Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International Conference on Machine Learning_ , pp. 3744–3753. PMLR, 2019.
  * Li et al. (2018) Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong.  Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pp. 1430–1441, 2018.
  * Lin (2004) Chin-Yew Lin.  Rouge: A packagefor automatic evaluation of summaries.  In _ProceedingsofWorkshop on Text Summarization Branches Out, Post2Conference Workshop of ACL_ , 2004.
  * Lin et al. (2018) Junyang Lin, Xu Sun, Shuming Ma, and Qi Su.  Global encoding for abstractive summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 163–169, 2018\.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju.  Fastbert: a self-distilling bert with adaptive inference time.  _arXiv preprint arXiv:2004.02178_ , 2020.
  * Lu et al. (2019) Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu.  Understanding and improving transformer from a multi-particle dynamic system point of view.  _arXiv preprint arXiv:1906.02762_ , 2019.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov.  Facebook fair’s wmt19 news translation task submission.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pp. 314–319, 2019.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli.  Scaling neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pp. 1–9, 2018.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_ , 2019.
  * Press et al. (2019) Ofir Press, Noah A Smith, and Omer Levy.  Improving transformer models by reordering their sublayers.  _arXiv preprint arXiv:1911.03864_ , 2019.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Ren et al. (2018) Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In _International Conference on Machine Learning_ , pp. 4334–4343, 2018.
  * Rush et al. (2017) Alexander M Rush, SEAS Harvard, Sumit Chopra, and Jason Weston.  A neural attention model for sentence summarization.  In _ACLWeb. Proceedings of the 2015 conference on empirical methods in natural language processing_ , 2017.
  * Schwartz et al. (2020) Roy Schwartz, Gabi Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A Smith.  The right tool for the job: Matching model and instance complexities.  _arXiv preprint arXiv:2004.07453_ , 2020.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ , 2015.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Edinburgh neural machine translation systems for wmt 16.  _arXiv preprint arXiv:1606.02891_ , 2016.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 464–468, 2018.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * So et al. (2019) David So, Quoc Le, and Chen Liang.  The evolved transformer.  In _International Conference on Machine Learning_ , pp. 5877–5886, 2019.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2818–2826, 2016.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Wan et al. (2018) Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu.  Improving automatic source code summarization via deep reinforcement learning.  In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , pp. 397–407, 2018.
  * Wang et al. (2019a) Dilin Wang, Chengyue Gong, and Qiang Liu.  Improving neural language modeling via adversarial training.  In _International Conference on Machine Learning_ , pp. 6555–6565, 2019a.
  * Wang et al. (2018) Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du.  A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pp. 4453–4460, 2018.
  * Wang et al. (2019b) Wenbo Wang, Yang Gao, He-Yan Huang, and Yuxiang Zhou.  Concept pointer network for abstractive summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3067–3076, 2019b.
  * Wang et al. (2019c) Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, Cheng Xiang Zhai, and Tie Yan Liu.  Multi-agent dual learning.  In _7th International Conference on Learning Representations, ICLR 2019_ , 2019c.
  * Wei et al. (2019) Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin.  Code generation as a dual task of code summarization.  In _Advances in Neural Information Processing Systems_ , pp. 6559–6569, 2019.
  * Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli.  Pay less attention with lightweight and dynamic convolutions.  In _International Conference on Learning Representations_ , 2019.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pp. 5754–5764, 2019.

##  Appendix A Experimental Settings and More Results

###  A.1 Detailed Data Settings

Neural Machine Translation Following the common practice (Ott et al., 2019),
we lowercase all words for IWSLT14 En$\leftrightarrow$De. For IWSLT14
En$\leftrightarrow$De, En$\leftrightarrow$Es, IWSLT17 En$\leftrightarrow$Fr,
we use a joint source and target vocabulary with $10k$ byte-pair-encoding
(BPE) (Sennrich et al., 2015) operations, and for IWSLT17
En$\leftrightarrow$Zh, we use a seperate source and target vocabulary. For all
WMT tasks, sentences are encoded by a joint source and target vocabulary of
$32k$ tokens.

Code Generation In the Java dataset, the numbers of training, validation and
test sequences are $69,708$, $8,714$ and $8,714$ respectively, and the
corresponding numbers for Python are $55,538$, $18,505$ and $18,502$. All
samples are tokenized. We use the downloaded Java dataset without further
processing, and use Python standard AST module to further process the python
code. The source and target vocabulary sizes in natural language to Java code
generation are $27k$ and $50k$, and those for natural language to Python code
generation are 18k and $50k$. In this case, following Wei et al. (2019), we do
not apply subword tokenization like BPE to the sequences.

Abstractive Summarization The Gigaword corpus represents for a headline
generation task, each source article contains about $31.4$ tokens on average,
while the target headline contains near $8.3$ tokens per sentence. The
training data consists of $3.8M$ article-headline pairs, while the validation
and test set consist of $190k$ and $2k$ pairs respectively. We preprocess the
dataset in a same way as NMT task. The words in the source article and target
headline are concatenated to make a joint BPE vocabulary. After preprocessing,
there are $29k$ subword tokens in the vocabulary.

###  A.2 Detailed Model/Training Configurations

Model Configuration The detailed model configurations are as follows:

  * •

transformer_iwslt_de_en setting: $6$ blocks in encoder and decoder, embedding
size $512$, feed-forward size $1024$, attention heads $4$, dropout value
$0.3$, weight decay $0.0001$.

  * •

transformer_vaswani_wmt_en_de_big setting: $6$ blocks in encoder and decoder,
embedding size $1024$, feed-forward size $4096$, attention heads $16$, dropout
value $0.3$, attention dropout $0.1$, relu dropout $0.1$.

  * •

transformer_wmt_en_de_big setting: $6$ blocks in encoder and decoder,
embedding size $0124$, feed-forward size $4096$, attention heads $16$, dropout
value $0.3$.

Optimization We adopt the default optimization setting in Vaswani et al.
(2017). Adam (Kingma & Ba, 2014) optimizer with $\beta_{1}=0.9,\beta_{2}=0.98$
and $\epsilon=10^{-9}$. The learning rate scheduler is inverse_sqrt with
warmup steps $4,000$, default learning rate is $0.0005$. Label smoothing
(Szegedy et al., 2016) is used with value $0.1$. As introduced, to learn the
predictors, we clamp the $\operatorname{softmax}$ output with value $0.05$.

###  A.3 Results of Order Combinations

We show in the paper that different number of orders (e.g., $N=4$ or $N=5$)
have varied performances. Therefore, one necessary point is about the
different combinations of these $N$ decoders. Here, we work on $N=5$ IOT model
to show the results of different order candidates.

We first present each ordered decoder in Table 11 again (same as in Table 1).

Code | $1$ | $2$ | $3$ | $4$ | $5$ | $6$
---|---|---|---|---|---|---
Order | SA$\to$ED$\to$FF | FF$\to$SA$\to$ED | ED$\to$FF$\to$SA | ED$\to$SA$\to$FF | SA$\to$FF$\to$ED | FF$\to$ED$\to$SA

Table 11: Each numbered code for one specific ordered decoder.

For the $N=5$ ordered decoders with IOT model, we show the performances with
$5$ combined orders selected from all six variants on dev set of IWSLT14
De$\to$En and En$\to$De translations. The results are reported in Table 12. We
can see the different combinations achieve similar strong performances, which
shows that our approach is robust towards different order combinations. This
also demonstrate that the importance of IOT is the diversity among order
candidates that can help each data distinguish them.

IOT ($N=5$) | $12345$ | $12346$ | $12356$ | $12456$ | $13456$ | $23456$
---|---|---|---|---|---|---
En$\to$De | $30.64$ | $30.65$ | $30.53$ | $30.58$ | $30.70$ | $30.67$
De$\to$En | $36.71$ | $36.67$ | $36.72$ | $36.79$ | $36.74$ | $36.75$

Table 12: BLEU scores for IWSTL14 De$\to$En and En$\to$De translations on dev
set. ‘$12345$’ represents the combinations of order $1,2,3,4,5$ decoders.

For other $N$ ordered decoders, the patterns are similar. Therefore, here we
only report the $N$ combinations used for IOT experiments in the paper as
follows: IOT ($N=2$) is combined by order $4,6$ (ED$\to$SA$\to$FF and
FF$\to$ED$\to$SA), and IOT ($N=3$) is order $1,4,6$, IOT ($N=4$) is order
$1,2,4,6$, and IOT ($N=5$) is order $1,2,4,5,6$.

###  A.4 Results of Different Number of Decoders

The results of $N=4$ ordered decoders (order $1,2,4,6$) are mainly reported in
the paper. Here, we also show results of other $N$ decoders for all tasks,
along with the Transformer baseline.

The results of different $N$ decoders for WMT14 En$\to$De and WMT16 Ro$\to$En
translations, code generation task, and Gigaword summarization are reported in
Table 14, Table 15 and Table 16 respectively. As we can see, more ordered
decoders can bring better performance, which supports the effectiveness of our
framework and demonstrates the data has its own favor towards different
orders. Considering the efficiency, we do not perform experiments with more
than $4$ decoders for these tasks.

Model | En$\to$De | De$\to$En | En$\to$Fr | Fr$\to$En | En$\to$Zh | Zh$\to$En | En$\to$Es | Es$\to$En |
---|---|---|---|---|---|---|---|---|---
Transformer | $28.57$ | $34.64$ | $35.9$ | $36.1$ | $26.3$ | $18.4$ | $39.0$ | $40.6$ |
IOT ($N=6$) | $29.48$ | $35.60$ | $37.4$ | $37.7$ | $27.1$ | $19.2$ | $40.2$ | $41.5$ |
IOT ($N=5$) | $29.51$ | $35.65$ | $37.2$ | $37.6$ | $27.2$ | $19.2$ | $40.2$ | $41.9$ |
IOT ($N=4$) | $29.52$ | $35.62$ | $37.2$ | $37.8$ | $27.2$ | $19.3$ | $40.1$ | $41.7$ |
IOT ($N=3$) | $29.43$ | $35.58$ | $37.0$ | $37.6$ | $27.0$ | $19.1$ | $39.7$ | $41.5$ |
IOT ($N=2$) | $29.18$ | $35.32$ | $36.6$ | $37.1$ | $26.8$ | $18.9$ | $39.6$ | $41.0$ |

Table 13: BLEU scores on $8$ IWSLT tasks with different $N$ ordered decoders
(with shared weights).

Model | WMT14 En$\to$De | WMT16 Ro$\to$En
---|---|---
Transformer | $29.12$ | $37.73$
IOT ($N=2$) | $29.71$ | $38.57$
IOT ($N=3$) | $29.89$ | $38.79$
IOT ($N=4$) | $30.03$ | $38.83$

Table 14: BLEU scores for WMT14 En$\to$De and WMT16 Ro$\to$En translation
tasks of different $N$ ordered decoders in IOT.

| Java |  | Python
---|---|---|---
Model | BLEU | PoV |  | BLEU | PoV
Transformer | $24.58$ | $74.44\%$ |  | $13.20$ | $61.89\%$
IOT ($N=2$) | $25.44$ | $77.88\%$ |  | $13.97$ | $62.22\%$
IOT ($N=3$) | $25.51$ | $75.83\%$ |  | $14.00$ | $62.04\%$
IOT ($N=4$) | $25.51$ | $77.30\%$ |  | $14.05$ | $63.14\%$

Table 15: BLEU and PoV scores for Java and Python code generation results of
different $N$ ordered decoders in IOT.

Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
Transformer | $35.59$ | $17.87$ | $32.98$
IOT ($N=2$) | $36.14$ | $18.37$ | $33.61$
IOT ($N=3$) | $36.15$ | $18.48$ | $33.81$
IOT ($N=4$) | $36.37$ | $18.46$ | $33.89$

Table 16: ROUGE F1 scores for Gigaword abstractive summarization results of
different $N$ ordered decoders in IOT.

##  Appendix B More Studies

###  B.1 Impact of Weighted Auxiliary Losses

We conduct another study on IWSLT14 De$\to$En dev set to investigate the
impact of our proposed auxiliary losses controlled by weight $c_{1}$ and
$c_{2}$. The values of $c_{1}$ and $c_{2}$ are varied between
$[0.0,0.05,0.1,0.5]$ and $[0.0,0.005,0.01,0.05]$ respectively, and the results
are presented in Table 17. It can be seen that the best configuration is
$c_{1}=0.1$ and $c_{2}=0.01$. Therefore, we report the leading results in the
paper with $c_{1}=0.1,c_{2}=0.01$. The results also clearly demonstrate that
the two additional losses are necessary to make our framework effective.

|  $\mathbf{c_{1}}$/$\mathbf{c_{2}}$ | $\bf{0.0}$ | $\bf{0.005}$ | $\bf{0.01}$ | $\bf{0.05}$
---|---|---|---|---|---
IOT ($N=2$) | $\bf{0.0}$ | $35.97$ | $36.36$ | $36.46$ | $36.42$
$\bf{0.05}$ | $36.35$ | $-$ | $36.58$ | $-$
$\bf{0.1}$ | $36.31$ | $36.56$ | $\bf{36.60}$ | $36.54$
$\bf{0.5}$ | $36.39$ | $-$ | $36.40$ | $-$
IOT ($N=3$) | $\bf{0.0}$ | $36.16$ | $36.44$ | $36.66$ | $36.51$
$\bf{0.05}$ | $36.47$ | $-$ | $36.47$ | $-$
$\bf{0.1}$ | $36.35$ | $36.48$ | $\bf{36.79}$ | $36.57$
$\bf{0.5}$ | $36.44$ | $-$ | $36.62$ | $-$

Table 17: BLEU scores for IWSLT14 De$\to$En dev set. The performances are
varied by different weighted auxiliary losses controlled by $c_{1}$ and
$c_{2}$ value.

###  B.2 Data Examples Verification

Set | $S_{i}$ | $T_{i}$ | $\bf{D_{i}}$ | $\bf{F_{i}}$ | BLEUAvg
---|---|---|---|---|---
$i=1$ | $2,404$ | $56,036$ | $\bf{4,557}$ | $\bf{0.3693}$ | $\bf{32.49}$
$i=2$ | $2,093$ | $49,077$ | $\bf{4,897}$ | $\bf{0.3234}$ | $\bf{27.92}$
$i=3$ | $2,786$ | $66,226$ | $\bf{4,699}$ | $\bf{0.3668}$ | $\bf{30.28}$

Table 18: Statistics of each English valid subset on IWSLT14 En$\to$De
translation. $S_{i}$ is the number of sentences in set $i$. Correspondingly,
$T_{i}$ is the token number, $D_{i}$ is the vocabulary size.
$F_{i}=\sum_{j=1}^{20}f_{ij}$ is the sum of the frequency of top $20$ tokens
in set $i$, where $f_{ij}$ is the frequency for token $j$.

As discussed in Section 5.2, the data split by the corresponding predicted
order is in different pattern. For example, the difficulty of each set is
different. We therefore analyze the split data and calculate some statistics
among these subsets. Specifically, we first count the sentence number $S$, the
tokens $T$, and the distinct vocabulary $D_{i}$ in each subset. We show these
numbers in Table 18, along with corresponding averaged BLEU score (see Figure
2). We can see that the vocabulary size of set $1$ is the smallest, and set
$2$ is the largest, which means there are more distinct words in set $2$. This
leads the generation of set $2$ to be harder than set $1$, which maps the BLEU
score ranking among these sets. Besides, we also calculate the token frequency
$f_{ij}$ for token $j$ in each own subset $i$, and sum the frequency of top
$20$ tokens in each subset, $F_{i}=\sum_{j=1}^{20}{f_{ij}}$, to give another
evidence. The results also show that $F_{1}$ is the highest, which means the
tokens in set $1$ contains most frequent words to make an easy learning, while
set $2$ is harder since $F_{2}$ is small.

We further take a look at the data and find the sentences in set $1$ are
mostly “simple sentences”, and set $2$ contains many “emphatic sentences”,
while set $3$ is somehow mixed. In Table 19, we provide some sentence examples
belong to each subset to give more clarifications.

Set $1$ | i will come to it later .
---|---
i wasn 't very good at reading things .
Set $2$ | this is a very little known fact about the two countries .
it 's why they leave lights on around the house .
Set $3$ | and how can we do that ?
it 's our faith , and we will be lo@@ y@@ al to it .

Table 19: Samples of each English valid subset on IWSLT14 En$\to$De
translation.

###  B.3 Regularization

In Section 5.4, we have provided an example of regularization experiments on
IWLST14 De$\to$En translation, which demonstrates that our IOT is not only
regularization and can be smoothly integrated with other regularization
methods. To give more evidences and more details, we extend the regularization
experiments on all IWSLT translation tasks (IWSLT14 En$\leftrightarrow$De,
IWSLT14 En$\leftrightarrow$Es, IWSLT17 En$\leftrightarrow$Zh, IWSLT17
En$\leftrightarrow$Fr), and WMT16 Ro$\to$En translation. The two specific
settings of the experiments are as follows. Setting (1): The first experiment
is “ordered Transformer” without instance awareness. That is, all the
reordered architectures are trained on the whole same corpus with equal
weights, and the parameters for these reordered architectures are shared. More
specifically, the decoder block has different ways to order SA, ED, and FF
layers (e.g., FF$\to$SA$\to$ED, SA$\to$ED$\to$FF, etc), but the parameters for
the reordered blocks are shared. Mathematically, the loss function is:
$\mathcal{L_{C}}=\sum_{n=1}^{N}(\lambda_{n}\cdot\mathcal{L_{M}}^{n})$, where
$\mathcal{L_{M}}^{n}$ is the model loss function for $n$-th ordered decoder.
Compared with Eqn (3), the weight $\lambda_{n}$ is fixed to be $1$ here. At
inference, we first find out the best order according to the dev performance
and apply it on the test set. We cannot use instance-wise reordered model in
this setting, while our proposed IOT can. The experiments are conducted with
transformer_iwslt_de_en configuration for IWSLT translations, and
transformer_vaswani_wmt_en_de_big configuration for WMT16 Ro$\to$En
translation.

Setting (2): We integrate another regularization technique ‘LayerDrop’ (Fan et
al., 2019) into both the Transformer baseline and our IOT ($N=4$) method,
while other settings remain unchanged. The study results of these two settings
are represented in Table 20.

From the results, we have same conclusions as discussed in Section 5.4. Simply
sharing the parameters of different decoders as a regularization cannot boost
the model performance (“Transformer + (1)” in Table 20), while our IOT can
further improve the performance with other regularization methods.

Model | En$\to$De | De$\to$En | En$\to$Fr | Fr$\to$En |
---|---|---|---|---|---
Transformer | $28.57$ | $34.64$ | $35.9$ | $36.1$ |
IOT ($N=4$) | $29.52$ | $35.62$ | $37.2$ | $37.8$ |
Transformer + (1) | $28.35\pm 0.12$ | $34.68\pm 0.05$ | $35.9\pm 0.23$ | $36.7\pm 0.14$ |
Transformer + (2) | $29.01$ | $35.40$ | $36.2$ | $36.8$ |
IOT ($N=4$) + (2) | $29.94$ | $36.13$ | $37.4$ | $38.1$ |
Model | En$\to$Zh | Zh$\to$En | En$\to$Es | Es$\to$En | Ro$\to$En
Transformer | $26.3$ | $18.4$ | $39.0$ | $40.6$ | $37.73$
IOT ($N=4$) | $27.2$ | $19.3$ | $40.1$ | $41.7$ | $38.83$
Transformer + (1) | $26.4\pm 0.05$ | $18.8\pm 0.17$ | $37.7\pm 0.28$ | $39.6\pm 0.11$ | $37.82\pm 0.05$
Transformer + (2) | $26.8$ | $19.0$ | $39.4$ | $40.8$ | $38.33$
IOT ($N=4$) + (2) | $27.3$ | $19.5$ | $40.6$ | $42.5$ | $38.98$

Table 20: Regularization study experiments on $8$ IWSLT translation tasks and
WMT16 Ro$\to$En translation. We study both setting (1): train Transformer
model with all shared decoders but without instance-wise learning, and setting
(2): add LayerDrop (Fan et al., 2019) regularization technique experiments on
these tasks.

###  B.4 Robustness

An impact of IOT training besides performance gain is that the model can be
more robust compared to one order only. In Table 21, we provide one example to
prove the robustness. We train one Transformer model by decoder order _1_ ,
and to decode the sentences with all orders in inference. Obviously, only
decoding with order _1_ leads to good performance, while other orders can not
achieve reasonable scores since the layer order is changed and the feature
exaction becomes incorrect. As for IOT, the generated sequences remain stable
and high results for each order.

Order | Transformer | IOT
---|---|---
$\emph{1}^{\star}$ | $35.84$ | $36.42$
2 | $27.96$ | $36.45$
3 | $8.05$ | $36.47$
4 | $5.11$ | $36.38$
5 | $1.32$ | $36.46$
6 | $0.35$ | $36.41$

Table 21: Robustness study on IWSTL14 De$\to$En translation task on dev set. ⋆
is the order trained by Transformer.

###  B.5 Visualization

To better understand the difference between IOT and standard Transformer, we
investigate on the training process and provide visualization results about
model optimization and performance improvements. Specifically, we plot the
curve of training loss, validation loss, as well as the validation BLEU score
and test BLEU score along the training epochs on IWSLT14 De$\to$En translation
dataset. The loss curves are visualized in Figure 3, and the BLEU curves are
presented in Figure 4.

From the validation loss curves of Figure 3(b) and 3(c), we can first see that
our IOT ($N=3$) training converges faster than Transformer baseline and shows
the advantage of IOT, which is consistent to our analysis in Section 5.1. The
converged (smallest) validation loss value seems to be similar to Transformer
baseline, but please note that the loss computation of IOT is different from
Transformer baseline. As shown in Eqn (3), the loss function of IOT is a
weighted sum of loss values for each order, while for Transformer, it is only
one order loss. Therefore, when we turn to the comparison of validation BLEU
score, the superiority of our IOT can be clearly verified. From the BLEU score
curves in Figure 4, it is obvious that IOT achieves better BLEU score than
standard Transformer along the whole training epochs, on both validation and
test sets. These visualized results well demonstrate the effectiveness of our
IOT approach.

(a) Training loss

(b) Validation loss ($y$-axis: $4.0$-$9.0$)

(c) Validation loss ($y$-axis: $3.9$-$4.1$)

Figure 3: Comparison of training/validation loss curves along the model
training on IWSLT De$\to$En translation. ‘IOT’ is IOT ($N=3$) and ‘Baseline’
is Transformer. Figure 3(c) is the same curve as 3(b), except the value of
$y$-axis in Figure 3(c) is between $3.9$-$4.1$, while $4.9$-$9.0$ for Figure
3(b).

(a) Validation BLEU

(b) Test BLEU ($y$-axis: $0$-$36$)

(c) Test BLEU ($y$-axis: $33$-$36$)

Figure 4: Comparison of validation/test BLEU curves along the model training
on IWSLT De$\to$En translation. ‘IOT’ is IOT ($N=3$) and ‘Baseline’ is
Transformer. Figure 4(c) is the same curve as 4(b), except the value of
$y$-axis in Figure 4(c) is between $33$-$36$, while $0$-$36$ for Figure 4(b).


11institutetext: Rahul Vigneswaran K 22institutetext: Department of Mechanical
Engineering, Amrita Vishwa Vidyapeetham, Amritapuri, India
Phone: +91-8667527705
22email<EMAIL_ADDRESS>33institutetext: Soman KP 44institutetext:
Center for Computational Engineering and Networking (CEN),
Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, India
55institutetext: Prabaharan Poornachandran 66institutetext: Center for Cyber
Security Systems and Networks, Amrita School of Engineering,
Amrita Vishwa Vidyapeetham, Amritapuri, India.

# A Compendium on Network and Host based Intrusion Detection Systems

Rahul-Vigneswaran K  Prabaharan Poornachandran  Soman KP

###### Abstract

The techniques of deep learning have become the state of the art methodology
for executing complicated tasks from various domains of computer vision,
natural language processing, and several other areas. Due to its rapid
development and promising benchmarks in those fields, researchers started
experimenting with this technique to perform in the area of, especially in
intrusion detection related tasks. Deep learning is a subset and a natural
extension of classical Machine learning and an evolved model of neural
networks. This paper contemplates and discusses all the methodologies related
to the leading edge Deep learning and Neural network models purposing to the
arena of Intrusion Detection Systems.

##  1 Introduction

The field of cybersecurity has recently gained a significant importance due to
the digitalization of developing countries. In several other perspectives, it
is even considered as a new way of warfare, due to which several nations are
pushing towards a more robust firewall for all the nation’s digital assets.
This lead to the development of Intrusion Detection System (IDS), which acts
as the first layer of the system and helps in identifying the inbound threats
which must be apprehended for safety reasons.

IDSs are a strategically placed layer in the system that detects threats and
monitors the exchanged packets. These type if security layers are there in
existence since 1980’s 1 . They use the hard-coded algorithms to analyze the
patterns in the captured data and filter the anomalies. These have proven
effective in certain cases but due to its inherent if-else nature, the problem
of false positive or false negative alarms was a common issue. Due to the
false triggers of alarms by the IDS, network administrators faced increased
difficulty in handling intrusion reports. Also, their inability to tackle
unknown attack types rendered them useless.

Commonly there are two types of IDS that exists, namely :

  * •

Network based Intrusion Detection System (NIDS): NIDS is a platform which is
independent and aims at detecting intrusions by examination of network traffic
and monitoring multiple hosts. These type of system access traffic by behaving
like a parasite to the network hub, a switch specifically designed for port
mirroring. For accuracy and efficiency, the sensors are situated at the
strategically chosen choke locations, often in the demilitarized zone. These
receptors collect all the network data and analyze the contents of packets.

  * •

Host-based Intrusion Detection System (HIDS): It usually has a singular agent
embedded onto a host system that detects intrusions by going through the logs,
calls made by the system, modifications of file-system and other states and
activities. In HIDS, a software agent is a sensor.

Traditionally, these IDS systems take advantage and make use of the signatures
of known attacks to identify the inbound attacks. But due to the rapid
advancement of new types of malware whose signatures are unknown, the
filtering of ever-increasing types of state of the art malware becomes
difficult. This paved way for the conjunction of machine learning and
cybersecurity. IDS in machine learning is a classification problem. Therefore,
possibly an unbiased dataset is used for training the model and that model is
optimized for integrating into the IDS to predict malware more accurately.
This whole prediction principle eliminates the complete dependency of malware
signatures for detection, hence acting as an advantage over traditional IDS.

For the prediction model to work more efficiently, we need to feed it with a
lot of training data. Often these raw data are unlabelled and come with
improper feature engineering. The pitfall with traditional machine learning is
that the training data must be labelled properly for the model to discover
patterns and therefore provide prediction results. But this is not always
possible to the vast size of the data. This is where Deep Learning (DL) has
explicit advantages over conventional algorithms in scenarios of processing
data that is unstructured. Pattern discovery can be done without the ancillary
task of labelling. In classical machine learning theory, engineering features
become an imperative part. DL lacks the feature scrutiny. This becomes vital
in plots where the objective is to segregate fraudulent sequences versus non-
fraudulent sequences and justify how we reached our conclusions. Fortunately,
the ultimate motive of an IDS is this, which makes deep learning a perfect
match for Intrusion Detection Systems.

##  2 Big Data Analytics

Presently, the current networks of computers are generating a huge amount of
data traffic in a fast past. This generation of data is increasing
continuously and the types of data that the network traffic contains is also
different. Current traffic of the public satisfies all the three V’s of big
data namely, volume, velocity, and variety 2 . Big Data is a term implicates
the large volume of data (structured & unstructured) that deluge business
worldwide at a faster rate than anyone can predict. A large pool of data
doesn’t always signify that everything is useful and this calls for a tool
that can extract the essence from it to decide the cardinal step. This is
where the Analytics of the so-called Big Data comes into play. Rapid
digitization has to lead the current world to a point where we might run out
storage capabilities soon. This implicates a much-required need to deduce the
pool of data into useful decisions.

Detection of intrusion often involves analyzing Big data. This is defined as a
research problem which the technologies of computing which are mainstream
cannot cope up with the flow of information. Even an isolated source of the
event can lead to Big data challenges. According to 3 , a 1 Gigabytes/second
of continuous network data can cause issues related to Big Data for Intrusion
Detection even when inspecting deep data packets. Traditional storage methods
like Relational databases inherently have the capacity to effectively gamut
against the nature of Big data challenges caused by detection of intrusion. A
common solution used to handle such problems is an open sourced tool called
Hadoop. It is a storage platform which is distributed and is generously open-
sourced. It has the capability to run on most of the hardware and has been
exploited to better entertain the Big Data storage requirements of gigantic
Volume and agility along with potentially very versatile data structures that
are natively heterogeneous. Hadoop is collective name for several other
technologies such as Pig, MapReduce, Hive, HDFS, etc.. 4  proposes the use of
these technologies to overcome the hurdles in big data and states that the 3Vs
are not sufficient to handle these issues and hence introduces the 3Cs concept
– Cardinality, Continuity, and Complexity.

The process of mining this Big data and analyzing it is called Big Data
analytics. The fast-paced emendations in aspects of processing, storing and
analyzing of Big Data in recent years includes :

  * •

The rapid decrease in storage expense and CPU power.

  * •

Cost-effectiveness and Flexibility of Cloud computing and Datacenters in
accordance with storage and elastic computation.

  * •

Development of revolutionary and new frameworks such as Hadoop.

The ultimate objective of Big Data analytics for IDS is to ensure promising,
market-ready real-time intelligence. Even though it has a compelling future,
there are several problems (such as data provenance, privacy) that must be
diminished in order to unleash the actual potential of it.

##  3 Neural Networks

A Neural Network is a paradigm related to the processing of information whose
thought process is motivated by the way the nervous systems of several
biological parts, such as the brain, analyze and segregate information. The
important aspect of this approach is the novel assembly of the system that
processes information. It is built with numerous neurons which are highly
connected with each other, working in unison to understand the training
dataset and solve problems specific to the set. These, like people, learn by
example. Learning in biological environment systems involves tuning of these
links that between the neurons of multiple layers. This is true of artificial
neural networks as well.

##  4 Deep Learning

It is a natural extension to the existing field of ML exercised with models
that are motivated by the assembly and function of brain cells. One of the key
aspects of Neural networks is scalability which signifies that the result’s
superiority is directly proportional to the amount of data that we use but it
also implies the need for significantly more computational time. In addition
to the advantage of easy scaling, another often stated plus point is their
intuitive ability to eliminate the burden of large-scale feature engineering.
Especially, DL shines in scenarios which require analogue input-outputs which
enables it to work with RGB pixel data from images, text data from documents
or audio data from music files rather than limiting the user to use quantities
in a tabular format.

##  5 Neural Networks and Deep learning techniques for Network based IDS
(NIDS)

The aim of an IDS is to detect abnormalities in the traffic and mitigate a
possible intrusion. This must be done in an efficient way considering the
ever-increasing variants of malware and types of attacks in the field of
cybersecurity. The use of signature-based and traditional if-else algorithm
based detection layers have been drawn ineffective which calls for the
inclusion of AI in the field of NIDS to predict an inbound attack that is
likely to cause damage or theft of digital assets.

Many researchers have been actively working for years now to experiment with
different types of techniques and datasets to establish a Network-based IDS
that has superhuman accuracies and capabilities. The use of deep learning and
artificial neural networks have proven effective in such applications due to
the improvement in the computational power of the commercially viable
processors. To overcome the principle challenges of a traditional intrusion
detection systems, 5  has proposed and implemented an analysis method that
combines the abilities of kernel PCA and LSTM-RNN. To obtain the desired
result, features like pre-processing of data, extraction of features,
detection of attacks have been incorporated into the layer proposed by it.
They have used the NSL-KDD dataset for benchmarking and has outperformed the
capabilities shown by SVM, neural network, and Bayesian methods.

The experiments done by 6  with LSTM-RNNs on the KDD99 dataset has shown an
accuracy of 93.82% due to their inherent ability to correlate consecutive
records of connections by peeking back in time. An avant-garde DL technique
for NIDs has been discussed by 7 , in which they have adduced an NDAE (Non-
symmetric Deep Auto-Encoder) for learning in unsupervised conditions. The
outcome of it when benchmarked against both KDD99 and NSL-KDD datasets has
shown a 5% improvement in accuracy and 98.81% reduction in training time. 8
has done a comparative study of deep learning like vanilla DNN, self-taught
learning (STL) approach, RNN based LSTM on the KDD99 dataset and then compared
with a baseline shallow algorithm that uses multinomial logistic regression to
evaluate if deep learning models perform better in this scenario. An important
observation that STL model could be used in a scenario where the data is
unclean is made by this experimentation.

As a whole, when various methods of deep learning and neural network methods
are compared with each other and studied, one can attain a state of
understanding that different scenarios require different AI models. Even in
NIDS, most of the models can be made to have high accuracy when proper
hyperparameter tuning is done. One of the major drawback reported by all the
researchers is the availability of latest ID dataset. All the research that is
currently done is based on either original or modified old datasets which
don’t consider the more advanced threats that the modern day NIDS is required
to filter.

11  takes a rather different approach to the whole intrusion detection system.
The model learns the user behaviour and when it the new behaviour doesn’t
match it, the alarm is triggered indicating a possible security breach. In
this, a backpropagation neural network has been used and the results turned
out with a 96% accuracy in detection of events that are unusual, with a false-
alarm rate of 7%. On the other hand, 12  takes into account a variety of
neural network structures to figure out the appropriate model for the current
scenario. Fortunately, an early barring corroboration method is also
administered during the training to substantially escalate the capability of
the network to generalize, which lends an accuracy rate of 91% and 87% with
two hidden neuron layers and one hidden neuron layers respectively.

The use of Genetic Network Programming is undertaken by 13  and has been
combined with the fuzzy set theory for better results. KDD99 and 98 datasets
have been used for benchmarking which lends success rates that are high
compared with other ML techniques and data mining which is crisp with GNP. A
common problem faced by all the existing approaches are false-alarm rates,
however small it may be, they can cause a lot of unwanted distractions to the
system administrators and an ideal IDS would have zero false-alarm rates. In
the path towards the ideal IDS, 14  has tried to enhance the system’s
performance by using keyword selection and neural networks. 15  have applied
the technique of deletion of features one at a time for performing experiments
on NNs and SVMs to estimate the hierarchy of priority of DARPA ID dataset’s
input features. Hence, doing so, they have identified the important features
for IDS and as a result, drastically reduced the training time.

Taking into account the reality that, terabytes of data pour through the IDS
if we tried to analyze the entire traffic flow, 16  has developed an anomaly
detector that considers the only certain portion of attack space that is
deemed vital for the detecting things such as protocols, that are
forecastable. Within the limitations of the experimental setup, the
conclusions show that this model truly responds to the anomalies, not to
signatures of known attacks.

17  has taken a notable approach to IDS. Convolutional neural networks have a
structure that favours computer vision and has provided significant results in
that field. This significance is being transported to cybersecurity by using
CNN-RNN in IDS layers and one of the main reason for choosing such an approach
is the model’s ability to extract high-level features and represent the
consolidated form of low-level features. Both 18  and 19  have used KDD99 as
their benchmarking datasets and the former tries to run a comparison between
shallow and deep neural networks in the scenario of NIDS and approves that
many of the inherent characteristics of deep networks provide an overall
advantage for it compared to shallow networks while the later experiments the
usage of stacked-RNNs. The stacked-RNN models show reduced false-alarm rate
due to its nature and potential to learn complex temporal behaviours quickly.

##  6 Neural Networks and Deep learning techniques for Host based IDS (HIDS)

A host-based intrusion detection system (HIDS) is a scheme that supervises and
logs activities of the computer on which it is installed to alert the admin in
case the protocols are breached. Host-based IDS can be conceived as an agent
that oversees and analyzes whether anything or anyone (both internal and
external), has bypassed the security policy of the system. NIDS is usually
placed at the demilitarized zone of the network. Its purpose is to check the
incoming data and eliminate threats and then pass the data forward to the
proprietor if the existence of potentially malicious packets is non-existent
with the transmission of data. It is more focused on the singular local node,
unlike NIDS. 9  has made comprehensive experimentation by using DNNs and SVMs
(Support Vector Machines) on the KDD-DARPA benchmarking dataset for making an
Host-based IDS to overcome the drawbacks of the traditional hard-coded logic
and algorithms. The accuracy has been concluded as greater than 99% with a
training time of 17.77 sec. But here too the pitfall of inability to train
with new attack patterns have been cautioned.

An approach where Radial basis functions neural networks as containers are
being used by 10  for host-based IDS which proves robust compared to the soft
computing methods with very low training time and high accuracy in prediction.
Therefore the answer to the question about superiority is very important. The
ultimate solution is using both NIDS and HIDS in conjunction with each other
and AI integrated into them. In order for the host (and network) to remain
secure, it is essential that we should select the right IDS scheme that is
appropriate for the provided circumstances.

##  7 Conclusion

As the importance of IDS in the day-to-day digital world is more obvious, the
necessity for a more accurate model with no false alarm rate has become
apparent. To attain an algorithm that is more robust to the ever-increasing
variety of malware and threats, researchers have taken an array of paths and
have succeeded in achieving exponentially high accuracy rates while
benchmarking with the existing publicly available dataset. It is clear that,
with proper hyperparameter tuning, most of the model existing in the market
can achieve superhuman ability in identifying inbound threats, which leads us
to the conjunction of most of the research papers that have been made on
artificial intelligence based IDS. Due to the basic principle of deep learning
which dictates that more data equals more accuracy, we need to train the
models with a dataset that has signatures and features of more recent advanced
cyber warfare attacks, in order to prepare the research stage IDS into a
robust market-ready product.

###### Acknowledgements.

This paper would have not been possible without the continued guidance of Dr.
KP. Soman of Centre for Computational Engineering and Networking (CEN), Amrita
Vishwa Vidyapeetham, India and Dr. Prabharan Poornachandran of Centre for
Cyber Security Systems and Networks, Amrita School of Engineering, Amrita
Vishwa Vidyapeetham, Amritapuri, India. Also, We thank the anonymous reviewers
for their insightful comments and suggestions.

## References

  * (1) Hodo, E., Bellekens, X., Hamilton, A., Tachtatzis, C. and Atkinson, R., 2017. Shallow and deep networks intrusion detection system: A taxonomy and survey. arXiv preprint arXiv:1701.02145.
  * (2) Manzoor, M.A. and Morgan, Y., 2017. Network intrusion detection system using apache storm. Probe, 4107, p.4166.
  * (3) Nassar, M., al Bouna, B., and Malluhi, Q., 2013, June. Secure outsourcing of network flow data analysis. In Big Data (BigData Congress), 2013 IEEE International Congress on (pp. 431-432). IEEE.
  * (4) Suthaharan, S., 2014. Big data classification: Problems and challenges in network intrusion prediction with machine learning. ACM SIGMETRICS Performance Evaluation Review, 41(4), pp.70-73.
  * (5) Meng, F., Fu, Y. and Lou, F., 2018, March. A network threat analysis method combined with kernel PCA and LSTM-RNN. In Advanced Computational Intelligence (ICACI), 2018 Tenth International Conference on (pp. 508-513). IEEE.
  * (6) Staudemeyer, R.C., 2015. Applying long short-term memory recurrent neural networks to intrusion detection. South African Computer Journal, 56(1), pp.136-154.
  * (7) Shone, N., Ngoc, T.N., Phai, V.D. and Shi, Q., 2018. A deep learning approach to network intrusion detection. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(1), pp.41-50.
  * (8) Lee, B., Amaresh, S., Green, C. and Engels, D., 2018. Comparative Study of Deep Learning Models for Network Intrusion Detection. SMU Data Science Review, 1(1), p.8.
  * (9) Mukkamala, S., Janoski, G. and Sung, A., 2002. Intrusion detection using neural networks and support vector machines. In Neural Networks, 2002. IJCNN’02. Proceedings of the 2002 International Joint Conference on (Vol. 2, pp. 1702-1707). IEEE.
  * (10) Ahmed, U. and Masood, A., 2009, October. Host based intrusion detection using RBF neural networks. In Emerging Technologies, 2009. ICET 2009. International Conference on (pp. 48-51). IEEE.
  * (11) Ryan, J., Lin, M.J. and Miikkulainen, R., 1998. Intrusion detection with neural networks. In Advances in neural information processing systems (pp. 943-949).
  * (12) Moradi, M. and Zulkernine, M., 2004, November. A neural network based system for intrusion detection and classification of attacks. In Proceedings of the IEEE International Conference on Advances in Intelligent Systems-Theory and Applications (pp. 15-18).
  * (13) Mabu, S., Chen, C., Lu, N., Shimada, K. and Hirasawa, K., 2011. An intrusion-detection model based on fuzzy class-association-rule mining using genetic network programming. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(1), pp.130-139.
  * (14) Lippmann, R.P. and Cunningham, R.K., 2000. Improving intrusion detection performance using keyword selection and neural networks. Computer networks, 34(4), pp.597-603.
  * (15) Sung, A.H. and Mukkamala, S., 2003, January. Identifying important features for intrusion detection using support vector machines and neural networks. In Applications and the Internet, 2003. Proceedings. 2003 Symposium on (pp. 209-216). IEEE.
  * (16) Lee, S.C. and Heinbuch, D.V., 2001. Training a neural-network based intrusion detector to recognize novel attacks. IEEE Transactions on systems, man, and Cybernetics-Part A: Systems and Humans, 31(4), pp.294-299.
  * (17) Vinayakumar, R., Soman, K.P. and Poornachandran, P., 2017, September. Applying convolutional neural network for network intrusion detection. In Advances in Computing, Communications and Informatics (ICACCI), 2017 International Conference on (pp. 1222-1228). IEEE.
  * (18) Rahul, V.K., Vinayakumar, R., Soman, K.P. and Poornachandran, P., 2018, July. Evaluating Shallow and Deep Neural Networks for Network Intrusion Detection Systems in Cyber Security. In 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT) (pp. 1-6). IEEE.
  * (19) Vinayakumar, R., Soman, K.P. and Poornachandran, P., 2017. Evaluation of Recurrent Neural Network and its Variants for Intrusion Detection System (IDS). International Journal of Information System Modeling and Design (IJISMD), 8(3), pp.43-63.