# Integrating Pre-trained Language Model into Neural Machine Translation

Soon-Jae Hwang  Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Chang-Sung Jeong  Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Neural Machine Translation (NMT) has become a significant technology in
natural language processing through extensive research and development.
However, the deficiency of high-quality bilingual language pair data still
poses a major challenge to improving NMT performance. Recent studies have been
exploring the use of contextual information from pre-trained language model
(PLM) to address this problem. Yet, the issue of incompatibility between PLM
and NMT model remains unresolved. This study proposes PLM-integrated NMT
(PiNMT) model to overcome the identified problems. PiNMT model consists of
three critical components, PLM Multi Layer Converter, Embedding Fusion, and
Cosine Alignment, each playing a vital role in providing effective PLM
information to NMT. Furthermore, two training strategies, Separate Learning
Rates and Dual Step Training, are also introduced in this paper. By
implementing the proposed PiNMT model and training strategy, we achieve state-
of-the-art performance on the IWSLT’14 En$\leftrightarrow$De dataset. This
study’s outcomes are noteworthy as they demonstrate a novel approach for
efficiently integrating PLM with NMT to overcome incompatibility and enhance
performance.

###### Index Terms:

Neural Machine Translation, Pre-trained Language Model, Catastrophic
Forgetting, Incompatibility, Fine-tuning, Distillation

##  I Introduction

Neural Machine Translation (NMT) has emerged as a prominent research topic in
artificial intelligence and natural language processing over recent years.
Particularly, Transformer model [1] utilizing the Attention mechanism has
played a decisive role in substantially enhancing the performance of NMT.
However, there remain several challenges in training NMT model. One primary
challenge is the requirement of vast amounts of high-quality bilingual pair
language data. Collecting and curating such data entail significant costs and
time. Following previous studies [2, 3], the absence of high-quality bilingual
pair language data complicates the training of NMT model and leads to
performance deterioration.

Against this backdrop, Pre-trained Language Models (PLMs) such as ELMo [4],
GPT [5], BERT [6], XLNet [7], BART [8], and T5 [9] acquire rich contextual
information from readily available large-scale monolingual data. Leveraging
this information, they undergo fine-tuning for downstream tasks and have
achieved impressive results on key natural language processing benchmarks like
GLUE [10] and SUPERGLUE [11].

Given the evident superior effects of fine-tuning PLMs for downstream tasks,
we have delved into existing research to understand the potential integration
of PLM into NMT model. Some explored methods include: initializing model
parameters with PLM checkpoint instead of random initialization followed by
fine-tuning [12, 13, 14, 15, 16]; indirectly employing PLM output in NMT
through distillation [17, 18]; and directly utilizing PLM output as input for
NMT model [14, 19, 18, 20].

However, incorporating PLM into NMT is not straightforward. Fine-tuning
resulted in relatively lower performance when using the output of a frozen PLM
as NMT input [19]. The reason is the occurrence of Catastrophic Forgetting
[21] during the process of transferring pre-existing knowledge from PLM to
NMT. Opting not to fine-tune and leaving PLM frozen also led to decreased
performance [14]. This drop is due to the incompatibility arising from
differences in the training task, model structure, and domain of train data
between PLM and NMT. For instance, while PLMs like BERT [6] operate with an
encoder structure restoring masked monolingual language data, NMT models
employ an encoder-decoder structure, translating source language data to
target language data. In conclusion, a new strategy is needed to overcome the
issues identified above.

This paper presents a novel PLM-integrated NMT (PiNMT) Model as a solution to
the previously identified challenges, effectively merging PLM and NMT. The
PiNMT model is composed of three primary components: PLM Multi Layer Converter
that effectively transforms the deep and rich multi-layer contextual
information from PLM into information suitable for NMT; Embedding Fusion that
addresses the complex fine-tuning issues of PLM; and Cosine Alignment, which
prevents potential information loss during the information transfer process
between the two models. Additionally, to enhance the efficiency and accuracy
of model training, we introduce two strategies: Separate Learning Rates,
applying different learning rates considering the complexity and scale between
PLM and NMT model; Dual Step Training, which further amplifies model
performance through the use of bidirectional data. The code implementation is
publicly accessible at the following repository111Available at:
https://github.com/vhch/PiNMT.

Through these strategically designed approaches, our model exhibits a
remarkable improvement on the IWSLT’14 En$\leftrightarrow$De dataset,
showcasing a 5.16 BLEU score increase compared to the basic model. Notably,
this result surpasses the previously highest-performing model on the same
dataset by an additional 1.55 BLEU score, thereby solidifying its superior
performance.

##  II Related Work

###  II-A Pretrained Language Model

In the field of NLP, various PLMs have been proposed to exploit large-scale
monolingual data across different languages and domains. Mikolov et al. [22]
introduced two architectures, CBOW and Skip-Gram, effectively learning word
vectors that reflect the context among words in a sentence. Although these
methods efficiently learned context-reflecting vectors, they were unable to
capture context-dependent meanings for polysemous words. To address this,
Peters et al. [4] introduced ELMo, utilizing Bi-LSTM to generate dynamic word
embeddings according to the given context, proving effective in various NLP
tasks. Radford et al. [5] proposed GPT, based on Transformer decoder, which
learned contextual information during sentence generation, showing outstanding
results in text generation. Devlin et al. [6] introduced BERT, based on
Transformer encoder, considering bidirectional context, and achieved
remarkable performance in a range of NLP tasks including question answering,
named entity recognition, etc. Building on these impressive performances, our
research aims to utilize BERT, which considers bidirectional contextual
information, to convey information to NMT.

###  II-B Integrating PLM into NMT

Numerous studies have been conducted on integrating PLM into NMT through
various approaches. Ramachandran et al. [12] proposed initializing parts of
NMT model with PLM and subsequently fine-tuning it. Ding et al. [23] suggested
leveraging PLM embeddings in NMT. The pre-trained embeddings were kept static
while being combined with additional embeddings, and only these new embeddings
were trained, highlighting the significance of these supplementary embeddings.
Yang et al. [17] introduced Asymptotic Distillation for transferring knowledge
from PLM to NMT. A dynamic switch is employed to utilize the information from
PLM in NMT dynamically. Additionally, the importance of differentiating
learning rates between PLM and NMT during fine-tuning is conveyed through a
proposed rate-scheduled learning. Zhu et al. [19] revealed in their
preliminary exploration that using PLM output as an input to NMT proved to be
more effective than initializing NMT with PLM parameters followed by fine-
tuning. They also proposed BERT-fuse approach, which integrates an additional
attention layer that interacts with PLM output in both the encoder and
decoder. Weng et al. [18] incorporated a Dynamic Fusion Mechanism that
considers information from all PLM layers in NMT Encoder. They also proposed a
knowledge distillation paradigm for decoder, emphasizing the importance of
utilizing multiple layers from PLM. Xu et al. [20] presented a methodology
combining stochastic layer selection with bidirectional pre-training to
effectively utilize multi-layers of PLM, underscoring the significance of
bidirectional pre-training. Weng et al. [24] replaced NMT encoder with PLM and
proposed a Layer-wise Coordination Structure to adjust the learning between
PLM and NMT decoders. Subsequently, they introduced a segmented multi-task
learning method for fine-tuning the pre-trained parameters, highlighting the
need to reduce incompatibility between PLM and NMT.

##  III Background

###  III-A Neural Machine Translation

The core principle of Neural Machine Translation involves learning the process
of converting a given parallel sentence pair {x, y} from the source sequence x
to the target sequence y. This transformation is facilitated through the
Transformer model [1]. The overall structure of Transformer model is as
follows:

Input Encoding: The input sequences x, y first pass through an embedding
layer, transforming them into continuous vector representations. Subsequently,
position encoding, containing location information, is added to form the final
input representation.

$E_{x}=\text{Emb}(x)+\text{Pos}(x)$ (1) $E_{y}=\text{Emb}(y)+\text{Pos}(y)$
(2)

Encoder: Multiple encoder layers process $E_{x}$. The i-th encoder layer
$H_{E}^{i}$ comprises layer normalization (LN) [25], multi-head attention
mechanism (MHA), and a feed-forward network (FFN) [1]. Each layer uses the
output of the previous layer as input. $S_{E}^{i}$ represents self-attention
result of encoder layer.

$S_{E}^{i}=\text{LN}(H_{E}^{i-1}+\text{MHA}(H_{E}^{i-1},H_{E}^{i-1},H_{E}^{i-1}))$
(3) $H_{E}^{i}=\text{LN}(S_{E}^{i}+\text{FFN}(S_{E}^{i}))$ (4)

Here, $H_{E}^{0}=E_{x}$.

Decoder: Multiple decoder layers process $E_{y}$. The i-th decoder layer
$H_{D}^{i}$ consists of LN, MHA, and FFN networks, and also includes attention
relationship with the final output of encoder $H_{E}^{N}$. $S_{D}^{i}$
indicates self-attention result of decoder layer, which is used to calculate
attention with the final output of encoder $H_{E}^{N}$, obtaining $C^{i}$.

$S_{D}^{i}=\text{LN}(H_{D}^{i-1}+\text{MHA}(H_{D}^{i-1},H_{D}^{i-1},H_{D}^{i-1}))$
(5) $C^{i}=\text{LN}(S_{D}^{i}+\text{MHA}(S_{D}^{i},H_{E}^{N},H_{E}^{N}))\\\ $
(6) $H_{D}^{i}=\text{LN}(C^{i}+\text{FFN}(C^{i}))$ (7)

Here, $H_{D}^{0}=E_{y}$.

Output: The final output of decoder is transformed into the probability
distribution of the next token through a linear layer and softmax function.

$\text{Output}=\text{Softmax}(\text{Linear}(H_{D}^{N}))$ (8)

Here, $H_{D}^{N}$ represents the output of the last layer of decoder.

Loss Function: The training objective of NMT is to minimize the difference
between the actual target and the model’s prediction. The model’s Output
represents the probability distribution for each token of the target sequence.
If the one-hot encoding of the actual target token is $y_{true}$, then the
cross-entropy loss is defined as follows:

$L_{CE}=-\sum y_{\text{true}}\log(\text{Output})$ (9)

This loss function measures how close the model’s predictions are to the
actual target and updates the model’s parameters to minimize this loss during
training.

###  III-B Pretrained Langue Model

In recent years, a variety of Pre-trained Language Models (PLMs) such as ELMo
[4], GPT [5], BERT [6], XLNet [7], BART [8], and T5 [9], capable of leveraging
large-scale monolingual data, have been proposed. There are mainly two methods
for training PLMs. The first is the auto-regressive approach [5], where the
model operates by predicting the k-th token $z_{k}$ based on the given context
$z_{<k}$ (i.e., the sequence before the k-th token). This can be represented
mathematically as $PLM(z_{k}|z_{<k};\theta)$. The second method is the masked
language modeling approach introduced by BERT [6]. In this method, random
tokens are masked, and these masked tokens are predicted using the surrounding
context information. Mathematically, this can be expressed as
$PLM(z_{m}|z_{-m};\theta)$.

##  IV Approach

In this study, we introduce a novel model called PLM-integrated NMT (PiNMT),
which integrates PLM into NMT. PiNMT is composed of three components: PLM
Multi Layer Converter, Embedding Fusion, and Cosine Alignment. Fig. 1 provides
a detailed representation of PiNMT architecture, and our approach is designed
to address the issues previously raised. Additionally, we describe two
training strategies necessary for overcoming these issues: Separate Learning
Rates and Dual Step Training.

Figure 1: The architecture of the PiNMT model

###  IV-A PLM Multi Layer Converter (PMLC)

PLM are composed of multiple layers, and each layer captures a variety of
contextual information [4, 26]. Previous research lacked a deep exploration of
utilizing the multi-layered nature of PLM [2, 20]. We introduce a Converter
technique to transform the multi-layer of PLM into a suitable source
embeddings for NMT model. Additionally, we introduce a Dimensional Compression
method to apply the high-dimensional information of PLM output to NMT model
with parameter constraints, allowing NMT model to leverage PLM’s information
more effectively.

Converter
Converter transforms the multi-layer output of PLM into a source embeddings
suitable for NMT model.

####  IV-A1 Vanilla

As seen in Fig. 2(a), only the final layer of PLM is used for output. This
method is based on the theory that the information extracted from the model’s
final layer is the richest and has the highest contextual understanding [27].
The last layer of PLM typically captures complex characteristics of the input
data and has the capacity to understand sophisticated linguistic features and
nuances.

####  IV-A2 Residual

Inspired by existing research [28], we introduce the concept of shortcut
connections to PLM, utilizing their multi layer architecture. The central idea
is to combine the outputs of all layers before the last layer with the output
of the final layer. This method simply adds the values of existing layers
without additional parameters or complex operations, thus being
computationally efficient. The equation is as follows, where $H_{PLM}^{i}$
represents the output of the i-th layer of PLM.

$\hat{H}=\sum_{i=1}^{M}H_{PLM}^{i}$ (10)

A key difference between our study and ResNet [28] lies in the implementation
and scope of shortcut connections. ResNet [28] modified the foundational model
by introducing shortcut connections to its intermediate layers. In contrast,
our proposed method maximizes the use of information from all layers without
affecting the intermediate structure of the model. This approach minimizes the
risk of pre-trained information degradation while allowing for more
comprehensive utilization of information.

####  IV-A3 Concat Linear

We propose a novel approach to resolve incompatibility issues when integrating
PLM with NMT model. Our method involves transforming the multi layers of PLM
into the input for an NMT model using learnable parameters for additional
training. This process effectively converts the multi layers of PLM, enhancing
compatibility between the two models. Consequently, it efficiently integrates
the robust language understanding capabilities of PLM into NMT model.
Specifically, we concatenate the outputs of each layer of PLM and then use a
Linear Layer to reduce the dimensions and produce the final output.

$H^{\prime}=[H_{PLM}^{1};H_{PLM}^{2};\dots;H_{PLM}^{M}]$ (11)
$\hat{H}=WH^{\prime}+b$ (12)

This method is similar to the existing Linear Combination approach [29] but
with several notable differences. The traditional approach combines each
layer’s output after passing through a linear layer without a bias term. In
contrast, our method first concatenates each layer’s output and then passes it
through a linear layer that includes a bias term. Our proposal introduces a
low-dimensional bias parameter to the model. The introduction of this bias
parameter enhances the convergence speed during the learning process and
significantly contributes to the overall performance improvement of the model.

####  IV-A4 Hierarchical

As observed in the study by Vaswani et al. [1], structuring layers in a deep
and complex manner can capture information more effectively. From this
perspective, instead of using a simple linear layer, we design a deeper, more
intricate Converter. Our objective is to propose a structure that merges nodes
hierarchically and deeply, an idea inspired by the research on Hierarchical
Aggregation [29].

The core concept introduces an aggregation (AGG) node $\hat{H}^{i}$. This node
combines information from either two or three layers using AGG function,
depending on specific conditions. AGG function concatenates multiple input
layers and forwards them to a Feed Forward Network (FFN). FFN’s outcome
connects back to the original inputs through a shortcut connection, and the
final result is normalized through Layer Normalization (LN).

$\hat{H}^{i}=\begin{cases}\text{AGG}(H^{2i-1},H^{2i})&\text{if }i=1\\\
\text{AGG}(H^{2i-1},H^{2i},\hat{H}^{i-1})&\text{if }i>1\end{cases}$ (13)

$\text{AGG}(x,y,z)=\text{LN}(\text{FFN}([x;y;z])+x+y+z)$ (14)

While earlier studies [29] employed a structure that fed aggregation nodes
back into the original backbone. In our approach, we modify this to prevent
aggregation node from being re-supplied to the backbone. This adjustment stems
from the risk of compromising the pre-trained information in PLM.

(a) Vanilla

(b) Residual

(c) Concat Linear

(d) Hierarchical

Figure 2: Converter

Dimensional Compression
The output from Converter encompasses deep, high-dimensional information
acquired from large datasets. This high-dimensional data, although rich in
meaning, is problematic due to its excessive dimensionality, especially when
incorporated into an NMT model with parameter constraints.

To address this, we suggest compressing the output dimensions of the Converter
via a Linear layer. This enables NMT model to effectively leverage the
information extracted from PLM. Mathematically, this can be represented as:

$H_{PLM}^{\prime}=WH_{PLM}+b$ (15)

Such a compressed output is then feed into NMT model, allowing for the
efficient use of high-dimensional information while optimizing parameter
efficiency.

###  IV-B Embedding Fusion

Embedding Fusion is an approach designed to overcome the limitations of fine-
tuning PLM. Most PLMs are expansive, making direct fine-tuning a challenging
task. Prior studies have frozen the parameters of PLM and integrated it with
an NMT model. An Extra Source Embeddings was added to this combined model,
which was then fine-tuned to enhance performance [23]. This method essentially
emulates the effects of directly fine-tuning PLM. It alleviates the
incompatibility issues between PLM and NMT model, while preserving the pre-
trained information. However, research on the effective utilization of Extra
Source Embeddings remains scant. Hence, this study proposes an optimized
method to harness the potential of Extra Source Embeddings.

####  IV-B1 Addition

To maximize the combination of PLM output and Extra Source Embeddings, we
apply a simple yet effective element-wise sum technique. Specifically, PLM
output $H_{PLM}$ and Extra Source Embeddings $E_{x}$ are summed to generate a
new embedding $E_{x}^{\prime}$. This method preserves features from both
sources and effectively combines their information. Formally, this can be
represented as:

$E_{x}^{\prime}=H_{PLM}+E_{x}$ (16)

####  IV-B2 Multiplication

We employ an element-wise multiplication technique to more vividly model
interactions between the two embeddings. This emphasizes the interdependency
and relevance of each feature, resulting in an embedding that closely
intertwines the characteristics of the two original embeddings.
Mathematically, it is depicted as:

$E_{x}^{\prime}=H_{PLM}\odot E_{x}$ (17)

####  IV-B3 Weighted Sum

Mere combination might not adequately reflect the relative importance between
two embeddings. To address this, we introduce a learnable weight to balance
between the two embeddings. Specifically, the weight $\gamma$ dynamically
adjusts the importance of the two embeddings, striving for an optimal
combination. This is mathematically captured as:

$E_{x}^{\prime}=\gamma H_{PLM}+(1-\gamma)E_{x}$ (18)

####  IV-B4 Projection

In the projection approach, each embedding undergoes a linear layer
transformation before combining. This ensures both embeddings map onto the
same feature space, facilitating efficient information amalgamation and
adjustment. This can be mathematically represented as:

$E_{x}^{\prime}=(W_{1}H_{PLM}+b_{1})+(W_{2}E_{x}+b_{2})$ (19)

####  IV-B5 Concatenation

The embeddings are concatenated. By directly merging features obtained from
various sources through concatenation, the model can utilize the information
from both embeddings. However, as the combined embedding might differ in
dimensionality from the original space, a Linear Layer is employed for
adjustments. The formula for this is:

$E_{x}^{\prime}=Linear([H_{PLM};E_{x}])$ (20)

####  IV-B6 Dynamic Switch

Based on Dynamic switch [17], this method features the introduction of a
context gate, crucial for the optimal integration of the two embeddings. The
context gate, rooted in sigmoid neural network layer, determines the
importance of each element within the input vectors received from PLM and
Extra Source Embeddings. It’s given by the equation:

$g=\sigma(WH_{\text{PLM}}+UE_{x}+b)$ (21)

Using the computed value of $g$, the two embeddings are dynamically combined.
The value $g$ adjusts the significance of each embedding, ensuring balanced
information integration. This is performed through the equation:

$h=g\odot h_{\text{PLM}}+(1-g)\odot E_{x}$ (22)

While the previous research [17] applied Dynamic Switch to each individual
encoder layer, this study focuses solely on Extra Source Embeddings.

###  IV-C Cosine Alignment

In previous studies, methods have been proposed for the effective transfer of
knowledge from large models to smaller ones through Distillation [30, 17, 18].
Among these, the methodology presented by Yang et al. [17] for distilling
information from PLM to an NMT minimizes the mean-squared-error loss between
PLM output and the outputs of NMT encoder or decoder, thereby transferring
PLM’s knowledge. However, subsequent experimental results indicate that
distilling from PLM to an NMT model in low-resource data scenarios either
results in suboptimal performance enhancement or even degradation. A primary
reason for this phenomenon appears to be the incompatibility between PLM and
NMT model.

To address this, Cosine Alignment is proposed. This approach adds cosine
similarity between the output of PLM and the last layer of NMT model’s decoder
to the existing loss function. Since the outputs of PLM and NMT Decoder do not
match in sequence length, the average value of each sequence is used.

$\displaystyle H_{PLM}=\\{h_{PLM}^{1},\dots,h_{PLM}^{I}\\}$ (23)
$\displaystyle H_{D}=\\{h_{D}^{1},\dots,h_{D}^{L}\\}$ (24) $\displaystyle
h_{PLM_{avg}}=\frac{1}{I}\sum_{i=1}^{I}h_{PLM}^{i}$ (25) $\displaystyle
h_{D_{avg}}=\frac{1}{L}\sum_{l=1}^{L}h_{D}^{l}$ (26)
$\displaystyle\begin{aligned} L_{similarity}&=\text{cosine
similarity}(h_{PLM_{avg}},h_{D_{avg}})\\\ &=\frac{h_{PLM_{avg}}\cdot
h_{D_{avg}}}{||h_{PLM_{avg}}||\times||h_{D_{avg}}||}\end{aligned}$ (27)

Here, $H_{PLM}$ represents the output of PLM with a sequence length of $I$,
and $H_{D}$ represents the last layer of decoder’s output with a sequence
length of $L$. The proposed $L_{similarity}$ is combined with the traditional
cross-entropy loss $L_{CE}$ to define the final loss function:

$L=L_{CE}+\alpha L_{similarity}$ (28)

In this context, $\alpha$ serves as a hyper-parameter that adjusts the weights
between the two losses.

###  IV-D Separate Learning Rates

PLMs are often large and intricate, and during fine-tuning, there’s a risk of
losing pre-trained information. Conversely, not fine-tuning PLM can cause
incompatibility issues between PLM and NMT model. To mitigate these
challenges, research has been proposed to set varying learning rates for
different layers [31, 27, 17]. Previous studies have demonstrated the efficacy
of this approach.

In this study, we incorporate the training strategy suggested by Yang et al.
[17] to implement Separate Learning Rates in our PiNMT model. We adjust the
learning rate for PLM to be relatively lower compared to that of NMT model.
Mathematically, this can be represented as follows:

$\eta^{PLM}=\rho\times\eta^{NMT}$ (29)

Where $\eta^{PLM}$ denotes the learning rate of PLM, $\eta^{NMT}$ denotes the
learning rate of NMT model, and $\rho$ indicates the relative coefficient
between these learning rates.

###  IV-E Dual Step Training

Many NMT models are trained solely on unidirectional data, which makes it
challenging to harness bidirectional linguistic features effectively. However,
recent studies [32, 20] have reported that implementing bidirectional training
can substantially enhance NMT performance.

In this study, we refer to the pre-existing training method [20] and apply
Dual Step Training to PiNMT model. The core idea behind this approach is to
invert the direction of unidirectional data, thereby augmenting it to create
bidirectional data (e.g., from En $\to$ De, we derive En + De $\to$ De + En).

Utilizing this newly formulated bidirectional data, we conduct a pre-training
phase for NMT model. This pre-training facilitates the model in learning
bidirectional linguistic attributes, thereby enhancing its generalization
capabilities. Subsequently, we fine-tune the pre-trained NMT model with the
original unidirectional data to optimize the model’s performance for specific
translation directions.

##  V Dataset and Baseline Settings

###  V-A Dataset

To validate the efficacy of our proposed methodology, we evaluate it using the
IWSLT’14 dataset [33] for the English$\leftrightarrow$German
(En$\leftrightarrow$De) language pair. The IWSLT’14
English$\leftrightarrow$German dataset comprises a total of 160K parallel
bilingual language pairs, allowing for a quantitative grasp of the model’s
performance. The distribution ratios of the training, validation, and testing
data are detailed in Table I.

TABLE I: Data Distribution IWSLT’14 (En$\leftrightarrow$De) | Count
---|---
train | 160239
valid | 7283
test | 6750

###  V-B Evaluation

For evaluation metrics, we adopt the commonly used tokenized BLEU Score [34].
Without the use of Dual Step Training, we set the beam search width to 4 and
the length penalty to 0.6. When employing Dual Step Training, the beam search
width is increased to 5, and the length penalty is set at 1.0.

###  V-C Settings

####  V-C1 PLM

In our study, we choose BiBERT [20] as our PLM. The original BERT model [6] is
pre-trained for a single language. However, BiBERT is concurrently trained on
both English and German. Built upon the RoBERTa architecture [35], BiBERT
model consists of 12 layers, has a model dimension of 768, and includes 12
attention heads. The training data for BiBERT combined and shuffled 145GB of
German text and 146GB of English text from OSCAR [36]. For the text
tokenization process, 67GB of randomly sampled English and German texts from
the training dataset were used. Using WordPiece tokenizer [37], a total of 52K
vocabulary was constructed.

####  V-C2 NMT

For NMT model implementation, we utilize fairseq framework [38]. As the base
model, we choose Transformer model [1] with transformer_iwslt_de_en settings.
This model comprises 6 encoder-decoder layers, has a model dimension of 512,
and includes 4 attention heads. Without employing Dimensional Compression, we
set the model’s dimension to match PLM output, which is 768. However, when
applying Dimensional Compression, it is set to 512. Various parameters is used
during the training phase to optimize the model’s performance. We apply a
label smoothing rate of 0.1 to the cross-entropy loss. The maximum tokens per
batch are set at 2048, with an update frequency of 16. For learning rate
scheduling, we opt for the inverse_sqrt method. The Beta values for the Adam
optimizer are set at (0.9, 0.98), and the initial learning rate is established
at 4e-4.

The vocabulary construction for NMT follow BiBERT [20] implementation method.
The encoder use a vocabulary size of 52k, matching PLM. The decoder’s
vocabulary is built based on the IWSLT’ 14 data. Without using Dual Step
Training, a vocabulary size of 8K is created from the target language data.
Conversely, when leveraging Dual Step Training, we construct a 12k-sized
English-German joint vocabulary.

####  V-C3 PiNMT

When applying Dimension Compression, both Concat Linear and Linear Combination
methods are compressed using the existing linear layer as they already contain
a linear layer. In experiments not using PMLC, PLM is utilized with Vanilla as
the base model. For conveying information in Cosine Alignment to encoder, we
do not use length averages and instead use the original distillation method.
Conversely, when conveying information to decoder in distillation, we use
length averages, similar to Cosine Alignment. The hyperparameter $\alpha$ is
set to 500.

##  VI Results and Analysis

In this section, we review proposed PiNMT model and two training strategies
using the IWSLT’14 En$\leftrightarrow$De dataset.

###  VI-A How does Dimensional Compression affect performance?

Significant findings can be observed in Table II. Compared to using
Transformer that solely trains on NMT model with dimensional compression,
there is a larger performance enhancement when performing dimensional
compression in Vanilla, which utilizes PLM as an input to NMT model. This
emphasizes the importance of dimensional compression in the interaction
between PLM’s information and NMT model. Furthermore, since PLM possesses rich
contextual information and high-dimensional features, it suggests that
appropriate compression in the model’s dimension is required to effectively
convey this information to NMT model, which has a limited number of
parameters.

TABLE II: Dimensional Compression Effects on De→En Models | BLEU
---|---
Transformer ($d_{model}$=768) | 33.99
Transformer ($d_{model}$=512) | 34.12
Vanilla ($d_{model}$=768) | 37.64
Vanilla ($d_{model}$=512) | 38.16

###  VI-B Performance Comparison of PMLC

We aim to compare the performance of various PMLC methods with Vanilla serving
as the baseline model. Firstly, we introduce strategies proposed in previous
studies. Linear Combination [29] linearly combines the outputs of all layers
without any bias term. Next, ELMo [2] combines the outputs of each layer with
learnable scalar weights to generate a new embedding. Additionally, Stochastic
Layer Selection [20] involves randomly selecting and utilizing various layers
of PLM during the training process.

According to the results presenting in Table III, all models that harness the
multi-layer capabilities of PLM outperform basic Vanilla approach. Notably,
models equipped with learnable parameters display more significant
improvements than their counterparts. This enhancement can be attributed to
the learnable parameters’ effectiveness in addressing the model’s
incompatibility issues. By using vector-based methods, which deploy more
parameters than scalar approaches, the model achieves exceptional performance.
Likewise, Hierarchical approach, with its more profound layer structure,
facilitates intricate learning, marking the highest performance among all the
discussed strategies. In our subsequent experiments, we delve deeper into
analyzing the PMLC, helping us ascertain the most effective strategy.

TABLE III: Performance Comparison of PMLC on De→En Models | BLEU | Existing Models | BLEU
---|---|---|---
Vanilla ($d_{model}$=512) | 38.16 | Linear Combination | 38.77
Residual | 38.67 | ELMo | 38.66
Concat Linear | 38.78 | Stochastic Layer Selection | 38.39
Hierarchical | 38.96 |  |

###  VI-C Performance Comparison of Embedding Fusion

Embedding Fusion methods are evaluated using Vanilla as the base. Upon
examining the results in Table IV, we observe performance enhancements in all
methods, with the exception of Multiplication approach, compared to Vanilla.
Multiplication emphasizes the interaction between the two embeddings. However,
its relatively lower performance suggests that the Extra Source Embeddings
provides novel information specialized for NMT, which has a comparatively
lower correlation with PLM.

The performance improvement noted in all techniques, excluding Multiplication,
indicates that the additional learning of Extra Source Embeddings can serve as
a solution to incompatibility while preserving the pre-trained information of
PLM.

The methods show higher performance in the order of Addition, Weighted Sum,
and others, which had fewer parameters. This is due to the insufficient
quantity of data required for training parameters that model the interaction
between the two embeddings. Consequently, it is evident that choosing an
effective model structure based on the amount of data is crucial. In
subsequent experiments, Addition method is employed as Embedding Fusion.

TABLE IV: Performance Comparison of Embedding Fusion on De→En Models | BLEU | Models | BLEU
---|---|---|---
Vanilla ($d_{model}$=768) | 37.64 | Projection | 37.97
Addition | 38.48 | Concatenation | 38.00
Multiplication | 35.70 | Dynamic switch | 37.99
Weighted Sum | 38.07 |  |

###  VI-D Distillation vs. Cosine Alignment

In Table V, the results on the left indicate that, generally, both
Distillation and Cosine Alignment methods enhance performance in Transformer.
However, applying Distillation to NMT decoder results in a performance
decline. Moreover, the performance improvements compared to Vanilla model are
not substantial for either method.

The results on the right side of Table V show a performance deterioration in
both the encoder and decoder when Distillation technique is applied to Vanilla
model. Cosine Alignment, on the other hand, improve performance but only in
the decoder. Both methods lead to performance degradation in the encoder,
primarily because the encoder, already processing PLM’s output, experiences an
information collision.

Upon closer examination of the reasons for the performance decrease with
Distillation in the decoder for both Transformer and Vanilla models, and why
it increases with Cosine Alignment. It reveals that Distillation conveys both
magnitude and direction of PLM’s output vectors to NMT model. In contrast,
Cosine Alignment only conveys the vector’s directional information. This
characteristic alleviates compatibility issues between PLM and NMT model,
allowing only the necessary information to be transmitted efficiently.

TABLE V: Distillation vs Cosine Alignment on De→En Transformer ($d_{model}=768$) | Vanilla ($d_{model}=768$)
---|---
Models | BLEU | Models | BLEU
Baseline | 33.99 | Baseline | 37.64
Distillation Enc | 35.36 | Distillation Enc | 37.21
Distillation Dec | 33.29 | Distillation Dec | 36.39
Cosine Alignment Enc | 35.33 | Cosine Alignment Enc | 36.95
Cosine Alignment Dec | 35.13 | Cosine Alignment Dec | 38.26

###  VI-E Is Separate Learning Rates Strategy Effective?

We conduct experiments using Vanilla model with a dimension of 712 as the
base. Upon reviewing the results in Table VI, it is evident that the magnitude
of the $\rho$ value significantly impacts performance. If the $\rho$ value is
too large, there’s a risk of damaging the contextual information from PLM,
potentially leading to a decline in performance. Conversely, if the $\rho$
value is too small, incompatibility issues may arise between PLM and NMT
model, resulting in potential performance degradation. Therefore, setting an
appropriate $\rho$ value is of paramount importance.

Rate-scheduled learning method [17] demonstrated exemplary performance on
extensive resources in past research but fails to replicate the same effect on
the low-resource IWSLT’14. This suggests that different datasets, with their
unique characteristics and sizes, may require distinct learning rate
strategies.

In the experiments, a $\rho$ value of 0.01 yields the most optimal results.
Thus, this value will be employed in subsequent experiments.

TABLE VI: Evaluation of Learning Rate Strategies on De→En Models | BLEU | Models | BLEU
---|---|---|---
$\rho=0$ | 37.64 | $\rho=0.01$ | 38.65
$\rho=0.0001$ | 38.06 | $\rho=0.05$ | 35.66
$\rho=0.0005$ | 38.46 | $\rho=0.1$ | 33.88
$\rho=0.001$ | 38.57 | $\rho=1$ | 11.81
$\rho=0.005$ | 38.59 | Rate-scheduled learning | 11.82

###  VI-F What is an optimal PMLC for Combination?

We establish our baseline by using Dimensional Compression to set the model’s
dimension at 512. Performance is analyzed by combining PMLC with Embedding
Fusion (EF), Cosine Alignment (CA), and Separate Learning Rates (SLR). Our
primary experiments focus on three PMLC approaches: Hierarchical, Linear
Combination [29], and Concat Linear.

Based on the results in Table VII, when applying Embedding Fusion and Cosine
Alignment, Hierarchical approach witnesses a decline in performance. This
suggests that the intricate structure of Hierarchical method can lead to
excessive complexity when integrating additional techniques, making
optimization challenging.

On the other hand, both Linear Combination and Concat Linear methods have a
relatively straightforward structure. This simplicity allows for more
potential improvements in performance when implementing additional techniques.
Notably, Concat Linear method consistently exhibit superior performance across
various combinations. This indicates the inherent flexibility of Concat Linear
approach, effectively integrating diverse forms of data and techniques.

However, Linear Combination without bias term, despite some enhancements, is
comparatively limited. This can be attributed to the bias term providing an
additional degree of freedom to the model, enabling it to better capture
specific data structures or patterns. Without the bias term, the model might
not have the extra information it needs to detect subtle patterns, which could
limit its performance gains.

In summary, this research demonstrates that comparatively simpler and more
flexible models are better adapted for integrating and combining diverse
techniques. It emphasizes the importance of balancing complexity and
flexibility when considering technique integration. As a result, we have
chosen Concat Linear approach for Converter.

TABLE VII: Evaluation of PMLC with Various Methods on De→En Models | Linear Combination | Concat Linear | Hierarchical
---|---|---|---
Baseline | 38.77 | 38.78 | 38.96
\+ EF | 38.91 | 39.07 | 38.68
\+ CA | 38.87 | 39.06 | 38.22
\+ SLR | 38.97 | 39.12 | 39.17
\+ ALL | 38.92 | 39.71 | 37.97

###  VI-G Is Dual Step Training Strategy Effective in PiNMT?

Using a Transformer Model with a dimension of 512 as the base, we analyze the
results of a study applying a Dual Step Training to PiNMT combined with
Separate Learning Rates. Observing Table VIII, it is evident that even the
sole application of Bidirectional Pre-training enhances the model’s
performance. This implies that the model benefits from recognizing the
bidirectional characteristics of translation. Additionally, performance
improvements are observable with Unidirectional Fine-tuning, indicating that
deep learning across extensive data alone is insufficient. It emphasizes the
need for learning tailored to specific domains.

TABLE VIII: PiNMT performance with Dual Step Training Models | En$\rightarrow$De | De$\rightarrow$En
---|---|---
Transformer (vocab size=12K) | 28.19 | 34.13
PiNMT with Separate Learning Rates | 31.48 | 40.03
Bidirectional Pre-training | 32.09 | 40.12
Unidirectional Fine-tuning | 32.20 | 40.43

###  VI-H Compared with Previous Work

Table IX compares our research with various studies based on the IWSLT’14
En$\leftrightarrow$De dataset. BERT-Fuse [19] introduced a new attention layer
to augment PLM interactions. UniDrop [39] consolidated multiple dropout
strategies. R-Drop [40] employed a regularization technique utilizing dropout,
BIBERT [20] harnessed PLM multi layers and engaged in bidirectional pre-
training, and Bi-SimCut [41] enhanced performance by seamlessly integrating
data augmentation and bidirectional pre-training. Compared to Transformer, our
method showcases an ascent of 5.16 in the BLEU score, and it exceeds the prior
peak performance set by Bi-SimCut by an additional 1.55 BLEU score. These
results provide compelling evidence for the effective resolution of the
identified challenges.

TABLE IX: Comparison with Existing Models Models | En$\rightarrow$De | De$\rightarrow$En | Average
---|---|---|---
Transformer (vocab size=12K) | 28.19 | 34.13 | 31.16
BERT-Fuse | 30.45 | 36.11 | 33.28
UniDrop | 29.33 | 36.41 | 32.87
R-Drop | 30.72 | 37.25 | 33.99
BIBERT | 30.45 | 38.61 | 34.53
Bi-SimCut | 31.16 | 38.37 | 34.77
Our Model | 32.20 | 40.43 | 36.32

##  VII Conclusion

We focused on the incompatibility issues arising when integrating PLM into
NMT. While PLMs were designed to understand and generate text within a single
language, NMT models were tasked with translating between different languages.
The inherent differences between these tasks led to incompatibility issues. To
address these, we proposed PiNMT model, incorporating key components like
PMLC, Embedding Fusion, and Cosine Alignment.

We designed PiNMT model to leverage the rich contextual insights from PLM, all
the while overcoming the challenges of their integration with NMT. In
addition, to make model training more effective, we incorporated strategies
like Separate Learning Rates and Dual Step Training. By adopting these
methodologies, we achieved SOTA performance on the IWSLT’14
En$\leftrightarrow$De dataset.

As with any research, our methodology can be further refined. Further tests
considering diverse languages and scales, as well as extended research, are
necessary. In conclusion, this study served as a foundational step in
strengthening the linkage between PLM and NMT, laying a critical groundwork
for future advancements in translation models.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [2] S. Edunov, A. Baevski, and M. Auli, “Pre-trained language model representations for language generation,” pp. 4052–4059. [Online]. Available: https://github.com/pytorch/fairseq/tree/
  * [3] P. Koehn and R. Knowles, “Six challenges for neural machine translation,” _arXiv preprint arXiv:1706.03872_ , 2017.
  * [4] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” 2 2018. [Online]. Available: http://arxiv.org/abs/1802.05365
  * [5] A. R. Openai, K. N. Openai, T. S. Openai, and I. S. Openai, “Improving language understanding by generative pre-training.” [Online]. Available: https://gluebenchmark.com/leaderboard
  * [6] J. Devlin, M.-W. Chang, K. Lee, K. T. Google, and A. I. Language, “Bert: Pre-training of deep bidirectional transformers for language understanding,” pp. 4171–4186. [Online]. Available: https://github.com/tensorflow/tensor2tensor
  * [7] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” 6 2019. [Online]. Available: http://arxiv.org/abs/1906.08237
  * [8] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” 10 2019. [Online]. Available: http://arxiv.org/abs/1910.13461
  * [9] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 10 2019. [Online]. Available: http://arxiv.org/abs/1910.10683
  * [10] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” _arXiv preprint arXiv:1804.07461_ , 2018.
  * [11] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 4938–4947.
  * [12] P. Ramachandran, P. J. Liu, and Q. V. Le, “Unsupervised pretraining for sequence to sequence learning,” 11 2016. [Online]. Available: http://arxiv.org/abs/1611.02683
  * [13] G. Lample and A. Conneau, “Cross-lingual language model pretraining,” 1 2019. [Online]. Available: http://arxiv.org/abs/1901.07291
  * [14] S. Clinchant, K. W. Jung, and V. Nikoulina, “On the use of bert for neural machine translation,” _arXiv preprint arXiv:1909.12744_ , 2019.
  * [15] S. Rothe, S. Narayan, and A. Severyn, “Leveraging pre-trained checkpoints for sequence generation tasks,” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 264–280, 2020.
  * [16] S. Ma, J. Yang, H. Huang, Z. Chi, L. Dong, D. Zhang, H. H. Awadalla, A. Muzio, A. Eriguchi, S. Singhal, X. Song, A. Menezes, and F. Wei, “Xlm-t: Scaling up multilingual machine translation with pretrained cross-lingual transformer encoders,” 12 2020. [Online]. Available: http://arxiv.org/abs/2012.15547
  * [17] J. Yang, M. Wang, H. Zhou, C. Zhao, Y. Yu, W. Zhang, and L. Li, “Towards making the most of bert in neural machine translation,” 8 2019. [Online]. Available: http://arxiv.org/abs/1908.05672
  * [18] R. Weng, H. Yu, S. Huang, S. Cheng, and W. Luo, “Acquiring knowledge from pre-trained model to neural machine translation,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 05, 2020, pp. 9266–9273.
  * [19] J. Zhu, Y. Xia, L. Wu, D. He, T. Qin, W. Zhou, H. Li, and T.-Y. Liu, “Incorporating bert into neural machine translation.” [Online]. Available: https://github.com/bert-nmt/bert-nmt.
  * [20] H. Xu, B. V. Durme, and K. Murray, “Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation,” 9 2021. [Online]. Available: http://arxiv.org/abs/2109.04588
  * [21] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” _arXiv preprint arXiv:1312.6211_ , 2013.
  * [22] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” 1 2013. [Online]. Available: http://arxiv.org/abs/1301.3781
  * [23] S. Ding and K. Duh, “How do source-side monolingual word embeddings impact neural machine translation?” 6 2018. [Online]. Available: http://arxiv.org/abs/1806.01515
  * [24] R. Weng, H. Yu, W. Luo, and M. Zhang, “Deep fusing pre-trained models into neural machine translation,” 2022. [Online]. Available: www.aaai.org
  * [25] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [26] G. Jawahar, B. Sagot, and D. Seddah, “What does bert learn about the structure of language?” in _ACL 2019-57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * [27] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” 5 2019. [Online]. Available: http://arxiv.org/abs/1905.05583
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 12 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [29] Z.-Y. Dou, Z. Tu, X. Wang, S. Shi, and T. Zhang, “Exploiting deep representations for neural machine translation,” 10 2018. [Online]. Available: http://arxiv.org/abs/1810.10181
  * [30] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 3 2015. [Online]. Available: http://arxiv.org/abs/1503.02531
  * [31] J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” 1 2018. [Online]. Available: http://arxiv.org/abs/1801.06146
  * [32] L. Ding, D. Wu, and D. Tao, “Improving neural machine translation by bidirectional training,” pp. 3278–3284. [Online]. Available: https://iwslt.org/2021/low-resource
  * [33] M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico, “Report on the 11th iwslt evaluation campaign, iwslt 2014.” [Online]. Available: http://translations.ted.org/wiki.
  * [34] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation.”
  * [35] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 7 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
  * [36] P. J. O. Suárez, L. Romary, and B. Sagot, “A monolingual approach to contextualized word embeddings for mid-resource languages,” pp. 1703–1714. [Online]. Available: https://oscar-corpus.com
  * [37] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey _et al._ , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _arXiv preprint arXiv:1609.08144_ , 2016.
  * [38] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” _arXiv preprint arXiv:1904.01038_ , 2019.
  * [39] Z. Wu, L. Wu, Q. Meng, Y. Xia, S. Xie, T. Qin, X. Dai, and T.-Y. Liu, “Unidrop: A simple yet effective technique to improve transformer without extra cost,” _arXiv preprint arXiv:2104.04946_ , 2021.
  * [40] L. Wu, J. Li, Y. Wang, Q. Meng, T. Qin, W. Chen, M. Zhang, T.-Y. Liu _et al._ , “R-drop: Regularized dropout for neural networks,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 10 890–10 905, 2021.
  * [41] P. Gao, Z. He, H. Wu, and H. Wang, “Bi-simcut: A simple strategy for boosting neural machine translation,” _arXiv preprint arXiv:2206.02368_ , 2022.


# Behavioural Reports of Multi-Stage Malware

Marcus Carpenter, Chunbo Luo
Department of Computer Science
University of Exeter
Exeter, UK
Email<EMAIL_ADDRESS>
###### Abstract

The extensive damage caused by malware requires anti-malware systems to be
constantly improved to prevent new threats. The current trend in malware
detection is to employ machine learning models to aid in the classification
process. We propose a new dataset with the objective of improving current
anti-malware systems. The focus of this dataset is to improve host based
intrusion detection systems by providing API call sequences for thousands of
malware samples executed in Windows 10 virtual machines. A tutorial on how to
create and expand this dataset is provided along with a benchmark
demonstrating how to use this dataset to classify malware. The data contains
long sequences of API calls for each sample, and in order to create models
that can be deployed in resource constrained devices, three feature selection
methods were tested. The principal innovation, however, lies in the multi-
label classification system in which one sequence of APIs can be tagged with
multiple labels describing its malicious behaviours.

##  I Introduction

There are billions of malware attacks worldwide every year [1]. Most malicious
programs are created by cyber-criminals in an attempt to generate profit for
organised crime groups. Malware exploit known and unknown vulnerabilities in
cyber-systems to gain access into restricted areas of a computer network. Once
access is gained, they can steal data, disrupt services, and damage both
physical and virtual assets. Anti-malware systems have existed for decades [2]
and are continuously updated to keep their intrusion detection on par with
modern attacks. Currently the main trend in the field of intrusion detection
is to implement machine learning models that segregate malicious anomalies
from benign behaviours [3].

Machine learning models require datasets that represent the task to be solved,
in this case, detecting malware. The proposed dataset provides the cyber-
security community with two main contributions. A textual representation of
API calls, and a multi-label classification system for the provided API
sequences. This dataset will allow researchers to create and improve host
based intrusion detection systems (HIDS). The proposed HIDS monitors the host
for API calls and then tag processes with labels that define its behaviour.
However, this paper only provides the dataset and a model benchmark as a proof
of concept.

This dataset only provides API calls because even though network features are
a powerful tool in detecting malware [4], they inherently involve other hosts.
Malicious behaviour can be monitored on the local device because the malware
must execute locally on the targeted device. Static features such as packers
and imports [5] were avoided because it is not always possible to analyse a
malware file before it executes on a computer. For example, a malicious
process that spreads over the network or a fileless malware would still
generate API calls, but may leave no trace after the attack. Hardware usage
features do not seem to be representative of a malware attack because it is
hard to understand how, for example, CPU usage relates to a malware attack
even though they have been successfully used to detect them in previous work
[6].

The API calls were generated by malware and benign samples running in Windows
10 virtual machines. Windows 10 possesses about 70% of the market share out of
the available Windows operating systems meaning that it still represents a
modern computer [7]. Other works using older operating systems may not provide
an accurate representation of a modern malware attack. However, it is
important to note that the proposed methods and models are not dependant on
operating system. The models can be deployed and trained on any device that
generates API calls and the multi-label classification system can be applied
to any type of malware data, not just API sequences.

The dataset provides API sequences that vary in size from one API call to over
seven hundred thousand, and while one API call may not provide much
information that aids in classification, overly long sequences contain more
words than a novel. It is had to train models on long sequences because of
high GPU RAM requirements and slow training times. This constraint is
exacerbated if deploying these models on end user devices with limited
hardware capabilities such as budget personal computers and IoT devices.

Models need to be updated regularly via retraining to maintain suitable
detection performance. This means that the HIDS must contain sustainable
models for hardware constrained devices. Therefore, the length of the API
sequences was constricted to an artificial limit of 512 API calls. This number
was chosen based on BERT, Google’s natural language processing neural network
[8]. Three feature selection methods where used in order to select the most
adequate sub-sequence of APIs, these are explained in Section V.

The main contribution brought by this dataset is the labelling system. To the
best of our knowledge all known works so far have focused on mutually
exclusive labels. Binary classification can only detect the presence of an
intrusion and does not go into detail on what the threat is, making it harder
for the analyst to mitigate the attack. Multi-class classification identifies
a threat by name and assumes that the analyst will have knowledge on how to
mitigate the attack from the given label. The proposed multi-label
classification identifies multiple malware types (if present) within the API
sequence. This allows the analyst to further understand the malware attack
without extensive knowledge of malware taxonomy.

Labelling behaviours and features allows models to detect new malware that
does not fit into the current taxonomy because behaviours are common among
malware of different classes and families. This means that the multi-label
classification system allows new malicious behaviour to be detected even if
there is no current specific label for it. In other words, the multi-label
classification system is akin to tagging an image with labels such as white,
poodle, and toy, instead of classifying it as dog, or poodle.

Section III provides a tutorial on how to built and setup a malware analysis
laboratory. Section IV provides a tutorial on how to create or expand the
dataset. The expandability of the dataset is possible due to its modularity.
The data is divided into batches each containing a theme, in this case there
is one batch for the benign files and two for malware samples. However, all
data is compiled into one when training the models. The dataset is described
in detail in Section V. The reason for this modular structure comes from the
growing trend of research into concept drift further discussed in Section
VIII.

Six different models were trained on each view of the dataset totalling
eighteen models in the benchmark. The benchmark results show that the first
512 API calls allow the models to reach higher classification accuracy, while
other feature selection algorithms make the data harder to classify. Out of
all six model architectures, the GRU performed best on the first feature
selection method. However, on the harder slices of the data the bidirectional
GRU was the best model arguably making it the superior model overall. The
results are further discussed in Section VII.

The code for the experiments can be found at:

https://github.com/marcusCarpenter97/Malware-data

##  II Related works

There has been extensive research on detecting malware using dynamic analysis
and API calls to train machine learning models. However, many of these works
use features that are undesirable for a HIDS such as network data or use
features that may not be necessary for accurate detection of malware. There is
also the case for the multi-label classification system proposed by this
dataset. Most works focus on binary malware detection and some make use of
multi-class classification, but not many provide an opportunity to tag a
sample with multiple labels.

There are many datasets that provide a good benchmark for machine learning
models. Datasets such as the CTU-13 [9], CICISD2017 [10], and NSL-KDD [11] are
good benchmarks for detecting malware attacks on network data. However,
network intrusions are out of scope for this work because the objective is to
create a host based detection system and not a network based system. While
both, host and network, data can complement each other, they ultimately have
different use cases. Network detection focuses on malicious behaviours
detectable on IP traffic while the proposed data focuses on detecting
malicious behaviours on API calls. It is possible to argue that a HIDS can
serve as a backup for network intrusion detection systems were a malware to
bypass it, or as a tool to mitigate non-networked transmissions of malware
such as USB drives.

Datasets that provide API calls are scarce. Microsoft’s Big 2015 dataset [12]
contains a large amount of malware samples represented as opcodes which allows
for natural language processing of the data and the implementation of a HIDS.
However, opcodes are extracted from disassembled binary files and it may not
always be possible to acquire a malicious file in a live environment, on the
other hand monitoring suspicious processes and extracting API calls is a
feasible solution for a HIDS. The ADAF-LD dataset [13] provides API calls for
malware running on a Linux environment. The drawbacks of this dataset are that
the operating system does not represent a modern environment, and the labels
do not support a multi-label classification system.

Aside from the popular datasets mentioned above, many researchers produce
their own data or are given data by a private contractor. These datasets tend
to be more inline with the proposed dataset, although they are not readily
available for the public making them unsuitable for a benchmark. Two types of
datasets were reviewed, binary labels and multi-class labels.

Amer et al. [14] used a Markov Chain for the binary classification of Windows
API calls. Word embeddings for each API sequence were clustered into benign
and malicious groups with the objective of predicting whether the sequences
are malicious or not. The transition probabilities between the two states
where calculated using Markov Chains.

Jindal et al. [15] classified Cuckoo Sandbox analysis reports as belonging to
a malicious or benign file. The JSON reports generated by Cuckoo were
processed to fit the fixed standard of the neural network. The model itself
used a combination of CNN, LSTM and Attention. Their proposed innovation is
that no feature selection is required and the model can choose the best
features from the report by itself.

Liras et al. [5] trained four machine learning binary classifiers, none of
them neural networks, on multiple features extracted from dynamic and static
analysis of malware. Manual and statistical feature selection was used to
extract features that best discriminate each class. The manual method is based
on heuristics and knowledge of the domain, and the statistical method involves
a small comparison of algorithms that rank features selected by the manual
method.

Rhode et al. [6] analysed malware with Cuckoo Sandbox, CAPE Sandbox’s
predecessor, and used a bidirectional GRU to classify the features extracted
from the analysis as either malicious or benign. The features selected by the
authors were based on the usage of hardware resources. For example, CPU and
RAM usage and network packets transmitted. They claim that training models on
API calls is hard because APIs are categorical data and unseen APIs cannot be
interpolated unless the vocabulary is comprehensive. However, the relation
between CPU usage and malicious behaviour is left unexplained.

Imtiaz et al. [16] trained a neural network to classify android malware. This
is one of the few attempts at tagging malware with multiple labels. However,
they did not use a multi-label classification system, instead they proposed a
two step classification method. First, the sample is classified as either
benign or malware and if the file is malicious, it is further labelled into a
malware category and a malware family in parallel. Their results show that the
models perform better on static features meaning that dynamic analysis
provides a harder dataset.

Downing et al. [17] used an autoencoder model to recreate the binary structure
of benign files, and when given a malicious file the model failed to reproduce
some sections. These errors were clustered into groups and labelled as a
malicious function of the malware, essentially tagging one sample with
multiple malicious functionalities. This is different than the proposed multi-
label system because it requires the help of an unsupervised clustering
algorithm rather than our method that uses a multi-label supervised end-to-end
deep learning model.

A few conclusions can be made from this brief literature review. First, binary
classification is often preferred even if labelling malware samples with
multiple labels at once has been shown to be a powerful tool, likely because
binary classification is less challenging. Second, word embeddings have been
used to process API calls, although many researchers often combine the APIs
with other features rather than use them on their own. And thirdly, feature
selection seems to be mostly based on heuristics and domain knowledge although
some have attempted to use end-to-end deep learning for feature selection.

##  III System Design

Dynamic malware analysis of thousands of samples requires dedicated software
running on a powerful workstation. This section explains the choices made when
selecting software and hardware components for building the malware analysis
laboratory. The software is chosen first as it is the main component of the
system while the hardware specifications are defined simply to accommodate the
choices in software.

###  III-A Choice in software

There are four major software components that make up the malware laboratory.
The sandbox, the virtual machine, the label maker, and the custom scripts.

The sandbox used to analyse the samples is called CAPE Sandbox [18]. CAPE is a
project forked from Cuckoo Sandbox and was chosen instead of its predecessor
for two reasons. First, Cuckoo is no longer maintained while CAPE is in active
development. Second, CAPE has pre-made scripts that help with installation.

The virtual machine (VM) creates a secure environment in which the sandbox can
run and analyse the malware. Despite KVM being recommended, VirtualBox was
used as the virtualisation software because it is easier to setup. Windows 10
was chosen as the target operating system for the VM because it still holds
the majority of the market share [7].

VirusTotal was used as the label maker. This is a famous and trusted software
in the cyber-security community that takes in a file and labels it according
to results from over 70 commercial antivirus products. Even though VirusTotal
is trusted by the community, Roh et al. state that an automated generation of
labels by computer programs results in the weakest type of machine learning
label [19]. However, the alternative of using human experts to label thousands
of files would have been too costly in terms of both time and money.

The last software component are the custom scripts for moving and processing
the raw data into a usable dataset. Bash scripts and Python were used. These
algorithms are presented in the future sections.

###  III-B Choice in hardware

As mentioned above, hardware requirements exist only to accommodate the
software, therefore a different software setup will have different hardware
demands.

The sandbox was setup to run 10 VMs in parallel in order to speed up the
analysis. Each VM was given 2 cores and 2 GB of RAM, therefore an Intel Core
i9-10980XE CPU combined with 64 GB of RAM were used. Storage is also an
important component of the system because VMs, thousands of files and their
respective reports, plus the final dataset occupy hundreds of GBs, so a 4TB
SDD was used. The GPU, while not relevant for the creation of the dataset
itself, was used to train the benchmark models. A Nvidia Titan RTX was used as
the GPU.

##  IV Malware analysis pipeline

The creation of the proposed dataset can be broken down into four steps of a
pipeline: sample collection, labelling malware with VirusTotal, analysing
samples with CAPE Sandbox, and processing the collected data. The analysis and
the labelling can be completed in parallel to save time. Each step is
described below and its accompanied by a diagram detailing the whole pipeline
in Figure 1.

###  IV-A Sample collection

Samples were downloaded from the VirusShare repository
111https://virusshare.com/. The data from VirusShare comes in batches of tens
or hundreds of gigabytes in size which are downloaded via torrent. However,
not all samples in a batch are desirable. Only the files that contained a
portable executable (PE) header were selected for this dataset. These are
Windows executable files (exe or dll).

To remove undesirable samples the Python module ‘pefile’
222https://pypi.org/project/pefile/ was used in a custom script. If the PE
header is not detected in a file by the third party module, then the file is
deleted. This rule was applied to all files downloaded from VirusShare.
However, collecting samples is more than just acquiring malware.

Benign files are also part of the dataset albeit they only compose a small
section of it. This task was more complicated than acquiring malware samples.
It is of course possible to use system files from the Windows operating
system, and some were used, but manually downloading benign apps is a tedious
chore without a web crawler. And because the program must run during the
analysis, these benign apps must either already be installed in the system or
the installer itself must be analysed.

###  IV-B Labelling malware with VirusTotal

VirusTotal provides a web service that receives files, analyses them using
their custom setup, and produces a downloadable report. Once an account is
registered (with an academic API), the API can be accessed through various
means, all of which can be found in their official GitHub repository
333https://github.com/VirusTotal/vt-py.

Only the malicious files were submitted to VirusTotal because benign files
already have a label by default, although this could generate interesting
results in terms of false positives and the trust placed in companies to
provide non-malicious software.

Custom Python scripts were written to handle the submission of files and the
retrieval of reports from VirusTotal. The reports provide a multi-label
classification for each file based on over 70 commercial antivirus products.
This means that one malware sample can have one or more labels defining its
class. The label structure will be further discussed in Section V.

###  IV-C Analysing samples with CAPE Sandbox

CAPE Sandbox has multiple components and requires careful configuration for
all of them to work. This section briefly explains how everything is setup and
then how the analysis process works. Once setup, the analysis can occur in
parallel to the VirusTotal labelling stage.

CAPE can be installed by running the helper script provided in the official
GitHub page [18]. Once installed there are two parts to consider when
configuring CAPE Sandbox, configuration files and virtual machines.

There are two configuration files that need to be edited before CAPE can work
properly, ‘cuckoo.conf’ and ‘machinery.conf’, both detailed in the
documentation. In the ‘cuckoo.conf’ file the ‘machinery’ field was edited to
use VirtualBox because KVM is the default choice. The only other field that
needs changing is the IP address under the ResultServer section which should
take the IP address of the hypervisor’s network interface.

The ‘machinery.conf’ file takes the name of the hypervisor of choice, in this
case ‘virtualbox.conf’. The file structure is comprehensive but only contains
one entry. If using multiple VMs, simply copy and paste the configuration
under a different name and IP address within the same file. The VMs must be
setup once CAPE is configured. The VM’s name must match whatever was inserted
in the ‘machinery.conf’ file. Basic software is required to analyse certain
files for example if analysing .docx, MS Word must be installed.

The agent is a script that allows the VM to communicate with the sandbox
during the analysis of a file. The agent must be executed with administrator
privileges inside the VM. It is also important to deactivate the firewall and
any Windows 10 feature that blocks the execution of potentially malicious
software. This includes Windows Defender, SmartScreen, and User Account
Control, etc.

The IP of the VM must be fixed for the sandbox to work, this can be achieved
via Windows networking configuration and the virtualisation software of choice
must be set to host-only networking. The VM must be able to ping the host and
vice-versa. A snapshot of the fully configured VM running the agent script is
advised, as CAPE can be configured to directly start the VM from this state.

Once the VM and CAPE are fully configured and working the analysis can begin.
A custom Bash script was written to submit samples for analysis. The script
uses CAPE’s submission utility to send samples to each of the ten VMs in an
alternating manner. This guarantees that each VM will execute in parallel.

###  IV-D Processing collected data

After the CAPE and VirusTotal analyses are complete, all result files must be
processed into a usable dataset. There are two algorithms that were used to
process the two parts of the data, one to extract API calls from CAPE reports,
and another to transform the VirusTotal reports into labels for machine
learning models.

The reports generated by CAPE Sandbox are JSON files and the API calls are
stored under the following set of keys: `behavior`, `processes`, `calls` and
then there are the `timestamp` and `api` keys. Each malware can spawn one or
more processes, and each process generates a sequence of timestamped APIs. The
APIs along with their timestamps are extracted from the JSON report into one
sequence and sorted by time.

Sub-processes were not considered separately because models can, or ideally
should, detect relationships between non-consecutive APIs. Moreover, as the
API calls are sorted by time the malicious events can still be modelled in the
order in which they occurred in the VM.

PE samplesCAPEVTCOMPILEDataset Figure 1: System overview of the malware
analysis laboratory. The files to be analysed, in this case Windows PE files,
are submitted to CAPE Sandbox and VirusTotal (VT) by separate Python scripts.
After the analysis is complete, one script extracts and saves the CAPE reports
and another performs the same action for the VT reports. From there the
compile script processes all results by extracting the relevant information
such as API calls and labels and compiles all this data into the multiple
batches seen in the final dataset.

Similar to the CAPE reports, VirusTotal reports are also stored in JSON format
and the labels used are under the following set of keys: `data`, `attributes`,
`popular_threat_classification`, `popular_threat_category`. This contains
textual labels describing potential malware classes for a sample. These
textual labels are converted into a binary array containing fifteen digits
based on a lookup table generated from all unique textual labels in the
dataset. Each position in the binary array represents a class in the data,
including benign and generic malware that cannot be further classified. Each
set bit in the binary array indicates that the API sequence belongs to a
specific class in the data.

##  V The data

There are three points to explain about the proposed dataset. First, this
section explains the file structure of the dataset and the data structure in
which the dataset is formatted in. The second point is a discussion on how
feature selection was applied onto the API sequences to reduce hardware
resource usage and optimise training times. And the third and final point is
on the label structure and the label distribution for the training and testing
sets.

The data has a physical and logical structure. The physical structure details
how the files are organised, and the logical structure relates to the internal
data structure of the file once it is loaded into memory. The data is composed
of JSON files, each file representing a batch of samples. A batch is a logical
partition of data that contains four fields, name, year, APIs, and labels. The
name is an arbitrary label given to the batch, in this particular dataset most
batch names relate to VirusShare names. The year is the year in which the
samples were collected, also extracted form VirusShare.

The APIs are stored as a list of strings, each string containing one function
call name. Some samples generate multiple processes during their execution,
these multiple API branches were merged into one sequence according to their
timestamp. Therefore all API calls are represented chronologically but
information about child and parent processes is lost. The data provides very
long API sequences that make training models very challenging. The length of
the API sequences varies greatly from one system call to over seven hundred
thousand, the latter is seven times longer than the word count of the average
novel.

Training neural networks on long textual sequences is only feasible in small
models but not scalable for two main reasons, one the GPU RAM is a limiting
factor, and two the training times become increasingly longer. To address the
variable lengths of the API sequences, only 512 API calls per sample were
processed by the model, this number is based on Google’s BERT model [8]. Using
only 512 API calls allowed for models to be trained faster and with a
reasonable GPU RAM usage. If only 512 API calls are to be processed by the
model, then the selected APIs must contain the greatest amount of information
that can be used to classify the sample. For this reason three feature
selection experiments were conducted.

Each feature selection method created a different view of the same data. All
views have the same train and test label distribution. The first view only
uses the first 512 API calls from the sequence, this is called ‘First’. The
second view uses the last 512 API calls, this is called ‘Last’. The third
takes a random slice of 512 consecutive APIs from the sequence, this is called
‘Random’. In each of the three views, if any API sequence is shorter than 512
API calls, then the sequence is used as a whole and padded later on.

The labels are stored as binary arrays to accommodate the multi-labelling
classification system. Each label is one row of 15 columns, and each column
represents one possible class the sample can be labelled as. The values in
each cell can only be 0 or 1, where 0 represents a lack of membership for the
class related to the column and 1 represents membership. A sample must belong
to at least one class, and in theory can belong to all classes at once.
However, in practice there are some mutually exclusive labels. For example,
benign labels are not marked as malicious, or malware without any further
classification can only be labelled as generic malware. In the future work,
Section VIII, the possibility of labelling benign files with malicious classes
is discussed.

To make experiments reproducible the Python 3.9 module ‘random’ used the seed
888 and the `train_test_split` method from sklearn 1.0.2 used the random state
parameter at 888. Table I shows the result of the train and test split along
with the distribution of samples for each available label. The main
observation to make is that the classes in the data are heavily imbalanced and
some might not have enough samples for a machine learning model to learn from.
There are 8087 samples in the dataset, although label count is greater because
of the multi-label system.

TABLE I: Distribution of labels in dataset Name | Train | Test | Total
---|---|---|---
benign | 66 | 35 | 101
malware | 905 | 426 | 1331
trojan | 4369 | 2167 | 6536
banker | 1089 | 521 | 1610
pua | 544 | 278 | 822
downloader | 452 | 266 | 718
adware | 642 | 332 | 974
dropper | 236 | 108 | 344
spyware | 122 | 65 | 187
virus | 208 | 118 | 326
miner | 69 | 33 | 102
ransomware | 177 | 87 | 264
worm | 155 | 82 | 237
hacktool | 87 | 36 | 123
fakeav | 4 | 6 | 10

##  VI The models

This section explains the six model architectures used to produce the
benchmark. Each model architecture was trained on all three views created by
the feature selection making a total of eighteen models. The six architectures
were the Simple Embedding Model (SEM), Multilayer Perceptron (MLP), Long-Short
Term Memory (LSTM), Gated Recurrent Unit (GRU), and the bidirectional versions
of the LSTM and GRU.

All six models used the Adam algorithm at $0.001$ learning rate to optimise
the binary cross entropy loss function. Fifteen epochs were used to train the
models, although early stopping was employed to prevent unnecessary
iterations.

The most basic model is the Simple Embedding Model (SEM). First, each sequence
of APIs is converted into a string and then vectorized. Vectorization is the
process of converting text into a sequence of integers, where each unique
token is mapped onto a unique integer. The vectorization layer is only trained
on the training corpus, and not the testing data, as a means to prevent data
leakage.

The second step of the SEM is to train an embedding layer on the vectorized
input. This helps the model learn how to best represent each API call in the
sequence and the relations between them by assigning each system call a set of
floating point features. The experiments in this paper used an embedding
dimension of 16.

The third and final step of the SEM is the output layer which converts the
learnt embeddings into a multi-label binary array. The output is produced by a
fully connected layer with the same number of nodes as classes in the label.
The activation function is the Sigmoid. The SEM is a very simple model, text
vectorization, embedding, and output, that serves as a foundation for the
other five models.

Both the SEM and the MLP require a Global Average Pooling layer between the
embedding layer and the fully connected layer to process the multi-dimensional
data. All models, except the SEM, use a fully connected layer with 64 nodes
and a ReLU activation before the output layer. The recurrent models build upon
the MLP by replacing the Global Average Pooling layer with one of the
following layers: LSTM, GRU, and the bidirectional version of each. All
recurrent layers also have 64 nodes.

TABLE II: Comparison of accuracy and F1-score Models | First | Last | Random
---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1
SEM | 0.443 | 0.682 | 0.407 | 0.655 | 0.414 | 0.655
MLP | 0.510 | 0.711 | 0.467 | 0.690 | 0.426 | 0.662
LSTM | 0.416 | 0.656 | 0.448 | 0.674 | 0.408 | 0.654
GRU | 0.531 | 0.722 | 0.479 | 0.687 | 0.452 | 0.673
BiLSTM | 0.487 | 0.684 | 0.480 | 0.695 | 0.441 | 0.665
BiGRU | 0.525 | 0.707 | 0.483 | 0.698 | 0.467 | 0.685

##  VII Benchmark results

The benchmark contains two types of results based on six error metrics which
are defined in sub-section VII-A. The first set of results provides an overall
model benchmark for all experiments, this is detailed in sub-section VII-B.
The second set of results explains how the model performed for each malware
class in the data, this is discussed in sub-section VII-C.

Each table separates the error metrics by feature selection method. The
‘First’ only used the first 512 APIs from the entire sequence. The ‘Last’ only
used the last 512 APIs from the sequence. And the ‘Random’ took a random slice
of 512 consecutive APIs from the whole sequence.

###  VII-A Error metrics

Two famous machine learning Python packages were used to calculate the error
values for the models. Scikit learn and Tensorflow. And six metrics were used
to calculate the errors. The accuracy, F1-score, precision, recall, and the
binary cross entropy which was used as the loss function for optimising the
models. There are only five metrics in the list because Scikit learn and
Tensorflow provide different accuracy measurements.

The Scikit learn accuracy only provides a score if the predicted label matches
the ground truth in its entirety. On the other hand, the binary accuracy
provided by Tensorflow considers each individual element in the binary array
when scoring the output of the model. This means that the Scikit accuracy is
more punitive than Tensorflow’s.

Scikit learn computes the F1-score as:

$F1=2\times(precision\times recall)\div(precision+recall)$

This metric was used in two ways. First by averaging the F1 over all samples
using the `average=samples` parameter to provide a global score for each
model, results can be found in Table II. Second was by calculating the F1
scores for each class in the data with `average=None`, these results can be
found in Tables IV, V, and VI. Both methods used the parameter
`zero_division=1`.

Table III provides the results from Tensorflow’s built in evaluate method.
This differs from the other tables where the predictions and labels were used
by Scikit learn to calculate error values. The loss is the binary cross
entropy and the other metrics include the binary accuracy, explained above,
precision and recall. The loss is the only metric that should be minimised,
all other values are meant to be maximised. In other words the higher the
better.

###  VII-B Model benchmark

The model benchmark is split between two tables. Table II details the results
calculated by Scikit learn by using the labels and the model’s predictions.
And, Table III contains the results generated by Tensorflow’s evaluate
function.

Table II shows that the GRU was the best performing model on the ‘First’ data
slice, and the BiGRU achieved best performance on the ‘Last’ and ‘Random’
slices. The ‘First’ data slice is easier to classify than the ‘Last’ and
‘Random’, this finding is based on the error values observed on the tables.
This phenomenon shows that the malware’s behaviour is more distinguishable in
the first 512 API calls than in the rest of the sequence, likely due to it
containing the launch of the attack.

The GRU outperforms non-sequential models because its recurrence is ideal for
the sequential nature of the API data, and in this case they outperformed the
LSTM model even though Chung et al. found no difference between them other
than reduced computational cost [20]. The bidirectional GRU outperforms the
GRU because of its extra backwards layer that allows it to find meaningful
connections between the API calls that have occurred and future calls adding
context to the current sequence.

Table III corroborates previous results further proving that the GRU is the
best performing model in the ‘First’ data slice, and the BiGRU is the best
model overall. Scikit Learn’s accuracy score illustrates that the models are
randomly guessing the labels because of the scores around 0.5. However,
Tensorflow’s accuracy reaches over 0.9 because they match each digit in the
binary array separately inflating the accuracy by counting all the predicted
0s. This is only possible because of the sparsity of the labels.

The precision in Table III is of about 0.8 meaning that the models do not over
predict the labels, in other words they do not produce too many false
positives when detecting malware. However, this is the only metric in which
the BiGRU does not outperform the other models. The recall, however, is around
0.6, slightly above random guessing meaning that there are many samples which
the models cannot properly tag as malware, likely due to there not being many
samples in each class for the model to learn from. Overall the models achieve
an F1 score between 0.6 and 0.7 (Table II) making the benchmark good while
also allowing for further improvement.

###  VII-C Performance on malware classes

TABLE III: Comparison of binary cross entropy (loss) and three other error
metrics on each data slice of the test set

Models | Loss | Binary accuracy | Precision | Recall
---|---|---|---|---
First | Last | Random | First | Last | Random | First | Last | Random | First | Last | Random
SEM | 0.177 | 0.184 | 0.19 | 0.94 | 0.937 | 0.937 | 0.845 | 0.824 | 0.832 | 0.581 | 0.575 | 0.565
MLP | 0.159 | 0.172 | 0.183 | 0.946 | 0.943 | 0.938 | 0.847 | 0.847 | 0.84 | 0.649 | 0.611 | 0.569
LSTM | 0.181 | 0.18 | 0.198 | 0.938 | 0.941 | 0.937 | 0.823 | 0.85 | 0.821 | 0.585 | 0.594 | 0.571
GRU | 0.148 | 0.171 | 0.178 | 0.951 | 0.942 | 0.942 | 0.87 | 0.836 | 0.86 | 0.671 | 0.616 | 0.586
BiLSTM | 0.165 | 0.171 | 0.189 | 0.945 | 0.943 | 0.938 | 0.858 | 0.843 | 0.827 | 0.623 | 0.621 | 0.581
BiGRU | 0.149 | 0.168 | 0.17 | 0.949 | 0.944 | 0.942 | 0.856 | 0.849 | 0.844 | 0.664 | 0.626 | 0.61

Tables IV, V and VI detail the model’s performance on the individual classes
in each of the data slices. The results show that the ‘First’ data slice
produces better trained models than the ‘Last’ and the ‘Random’ data slices,
in that order. Better trained models are defined as having higher scores in
their classification metrics.

From the tables mentioned above, it is possible to make four conclusions on
how the models detect each class in the data. Even though the classes are not
mutually exclusive, these conclusions are made on each individual class.

Benign, Dropper, Spyware, Miner, Hacktool, and Fakeav, are the hardest classes
to detect. Only the BiGRU on the First data slice detected benign samples with
a low F1 score of 0.21. It is possible that this is a statistical outlier and
the model did not learn how to classify the benign samples. The other classes
were not detected by any of the models in any of the data slices. This is
likely because there are not enough training samples in each of these
categories with the biggest containing 236 samples.

Pua and Downloader have 544 and 452 samples respectively and were detected
more often by the models in the ‘First’ slice while not detected in the ‘Last’
and ‘Random’ slices. The Adware class contains 642 samples in the train set, a
quantity big enough to train the models consistently at a reasonable F1 score
on the ‘First’ slice. This performance decreased on the ‘Last’ slice and
became more unstable on the ‘Random’ slice.

Unlike other classes, the Ransomware, Worm, and Virus, are detected by many
models in the ‘First’ slice with good F1 scores even though they have a small
number of samples. The largest class size is 208. Models performing better on
classes with less samples implies that the malware types are easier to
classify due to some feature in the API sequences that compensates for the
lack of data.

The most populous classes are the Trojan and the Banker malware, followed by
the generic malware class. A label reserved for when there are no other labels
available but the file is known to be malicious. The size of these classes
makes it easier for the neural networks to learn how to classify them with
good F1 scores. However, the generic malware class scores low on the F1 score
in all three data slices even though it contains 905 samples in the training
set.

The difficulty in classifying the generic malware class comes from the fact
that these API sequences may actually belong to other classes but labels were
missing. The variance in the data creates a challenge for the neural networks
to reach adequate performance.

##  VIII Conclusion

The proposed dataset aims to address the lack of research on classifying
malware based on dynamic analysis. The data introduces malware attacks
represented as a textual sequence of API calls. In addition, the main
innovation is the multi-label classification system that allows one malware
attack to be tagged with multiple malware classes. This new classification
system was benchmarked using six neural network models and three feature
selection methods. Moreover, a tutorial on how to create a malware analysis
laboratory and how to compile the dataset was given.

Five potential future work directions include: addressing the class imbalance
of the dataset by either improving the models or expanding the dataset by
collecting more samples; some benign software can be labelled as malicious
either as a false positive or as a potentially malicious behaviour, these
samples could be added to the data in an attempt to determine how much trust
can be placed on software development companies; modifying the labelling
system to include the probabilities of a sample belonging to a class rather
than a binary membership value, this would rely on a consensus of how many
antivirus label a sample as a certain class; sub-classing malware into
families, this was avoided because it would generate an overwhelming amount of
labels that may not be significant; studying concept drift and creating models
that can accurately classify malware across multiple time periods.

TABLE IV: F1 score on each class of first data slice of the test set

Model | benign | malware | trojan | banker | pua | downloader | adware | dropper | spyware | virus | miner | ransomware | worm | hacktool | fakeav
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SEM | 0 | 0.399 | 0.909 | 0.824 | 0 | 0 | 0.011 | 0 | 0 | 0 | 0 | 0.242 | 0 | 0 | 0
MLP | 0 | 0.385 | 0.908 | 0.874 | 0.069 | 0 | 0.543 | 0 | 0 | 0.613 | 0 | 0.562 | 0.728 | 0 | 0
LSTM | 0 | 0.004 | 0.897 | 0.83 | 0 | 0 | 0 | 0 | 0 | 0.526 | 0 | 0 | 0.656 | 0 | 0
GRU | 0 | 0.462 | 0.912 | 0.9 | 0.227 | 0.132 | 0.691 | 0 | 0 | 0.587 | 0 | 0.549 | 0.696 | 0 | 0
BiLSTM | 0 | 0.401 | 0.9 | 0.859 | 0.021 | 0 | 0.593 | 0 | 0 | 0.537 | 0 | 0.585 | 0.691 | 0 | 0
BiGRU | 0.21 | 0.378 | 0.906 | 0.899 | 0.238 | 0.195 | 0.678 | 0 | 0 | 0.61 | 0 | 0.592 | 0.748 | 0 | 0

TABLE V: F1 score on each class of last data slice of the test set

Model | benign | malware | trojan | banker | pua | downloader | adware | dropper | spyware | virus | miner | ransomware | worm | hacktool | fakeav
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SEM | 0 | 0.022 | 0.895 | 0.864 | 0 | 0 | 0 | 0 | 0 | 0.287 | 0 | 0 | 0.07 | 0 | 0
MLP | 0 | 0.401 | 0.907 | 0.87 | 0 | 0 | 0.117 | 0 | 0 | 0.581 | 0 | 0 | 0.745 | 0 | 0
LSTM | 0 | 0.307 | 0.901 | 0.859 | 0 | 0 | 0.116 | 0 | 0 | 0.545 | 0 | 0 | 0.689 | 0 | 0
GRU | 0 | 0.384 | 0.901 | 0.877 | 0 | 0 | 0.388 | 0 | 0 | 0.548 | 0 | 0.022 | 0.693 | 0 | 0
BiLSTM | 0 | 0.352 | 0.907 | 0.882 | 0 | 0 | 0.337 | 0 | 0 | 0.554 | 0 | 0 | 0.703 | 0 | 0
BiGRU | 0 | 0.351 | 0.909 | 0.883 | 0 | 0.064 | 0.364 | 0 | 0 | 0.563 | 0 | 0.085 | 0.698 | 0 | 0

TABLE VI: F1 score on each class of random data slice of the test set

Model | benign | malware | trojan | banker | pua | downloader | adware | dropper | spyware | virus | miner | ransomware | worm | hacktool | fakeav
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SEM | 0 | 0.2 | 0.897 | 0.829 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
MLP | 0 | 0.305 | 0.9 | 0.832 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
LSTM | 0 | 0 | 0.896 | 0.856 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
GRU | 0 | 0.388 | 0.905 | 0.853 | 0 | 0 | 0.303 | 0 | 0 | 0.065 | 0 | 0 | 0 | 0 | 0
BiLSTM | 0 | 0.302 | 0.9 | 0.814 | 0 | 0 | 0.275 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
BiGRU | 0 | 0.381 | 0.905 | 0.88 | 0.06 | 0 | 0.394 | 0 | 0 | 0.419 | 0 | 0 | 0.442 | 0 | 0

## References

  * [1] Statista, “Annual number of malware attacks worldwide from 2015 to first half 2022,” _Statista_ , Accessed on 07/12/2022. [Online]. Available: https://www.statista.com/statistics/873097/malware-attacks-per-year-worldwide/
  * [2] D. Denning, “An intrusion-detection model,” _IEEE Transactions on Software Engineering_ , vol. SE-13, no. 2, pp. 222–232, feb 1987.
  * [3] J. Singh and J. Singh, “A survey on machine learning-based malware detection in executable files,” _Journal of Systems Architecture_ , vol. 112, p. 101861, jan 2021.
  * [4] M. Wozniak, J. Silka, M. Wieczorek, and M. Alrashoud, “Recurrent neural network model for IoT and networking malware threat detection,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 8, pp. 5583–5594, aug 2021.
  * [5] L. F. M. Liras, A. R. de Soto, and M. A. Prada, “Feature analysis for data-driven APT-related malware discrimination,” _Computers & Security_, p. 102202, jan 2021.
  * [6] M. Rhode, P. Burnapb, and K. Jones, “Early-stage malware prediction using recurrent neural networks,” _Computers & Security_, vol. 77, 2018.
  * [7] Statcounter, “Desktop windows version market share worldwide,” _GloabalStats_ , Accessed on 07/12/2022. [Online]. Available: https://gs.statcounter.com/os-version-market-share/windows/desktop/worldwide
  * [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint_ , Oct. 2018.
  * [9] S. García, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” _Computers & Security_, vol. 45, pp. 100–123, sep 2014.
  * [10] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in _Proceedings of the 4th International Conference on Information Systems Security and Privacy_. SCITEPRESS - Science and Technology Publications, 2018.
  * [11] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the KDD CUP 99 data set,” in _2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications_. IEEE, jul 2009.
  * [12] R. Ronen, M. Radu, C. Feuerstein, E. Yom-Tov, and M. Ahmadi, “Microsoft malware classification challenge,” _arXiv preprint_ , Feb. 2018.
  * [13] G. Creech and J. Hu, “A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns,” _IEEE Transactions on Computers_ , vol. 63, no. 4, pp. 807–819, apr 2014\.
  * [14] E. Amer and I. Zelinka, “A dynamic windows malware detection and prediction method based on contextual understanding of API call sequence,” _Computers & Security_, vol. 92, p. 101760, may 2020.
  * [15] C. Jindal, C. Salls, H. Aghakhani, K. Long, C. Kruegel, and G. Vigna, “Neurlux,” in _Proceedings of the 35th Annual Computer Security Applications Conference_. ACM, dec 2019\.
  * [16] S. I. Imtiaz, S. ur Rehman, A. R. Javed, Z. Jalil, X. Liu, and W. S. Alnumay, “DeepAMD: Detection and identification of android malware using high-efficient deep artificial neural network,” _Future Generation Computer Systems_ , vol. 115, pp. 844–856, feb 2021.
  * [17] E. Downing, Y. Mirsky, K. Park, and W. Lee, “Deepreflect: Discovering malicious functionality through binary reconstruction,” in _30th USENIX Security Symposium (USENIX Security 21)_. USENIX Association, Aug. 2021, pp. 3469–3486. [Online]. Available: https://www.usenix.org/conference/usenixsecurity21/presentation/downing
  * [18] K. O’Reilly, “Cape sandbox,” _GitHub_ , 2022. [Online]. Available: https://github.com/kevoreilly/CAPEv2
  * [19] Y. Roh, G. Heo, and S. E. Whang, “A survey on data collection for machine learning: A big data - AI integration perspective,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 33, no. 4, pp. 1328–1347, apr 2021.
  * [20] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _arXiv preprint_ , 2014\.