# An Ensemble of Pre-trained Transformer Models For Imbalanced Multiclass
Malware Classification

Ferhat Demirkıran  Aykut Çayır  Uğur Ünal  Hasan Dağ

###### Abstract

Classification of malware families is crucial for a comprehensive
understanding of how they can infect devices, computers, or systems. Hence,
malware identification enables security researchers and incident responders to
take precautions against malware and accelerate mitigation. API call sequences
made by malware are widely utilized features by machine and deep learning
models for malware classification as these sequences represent the behavior of
malware. However, traditional machine and deep learning models remain
incapable of capturing sequence relationships among API calls. Unlike
traditional machine and deep learning models, the transformer-based models
process the sequences in whole and learn relationships among API calls due to
multi-head attention mechanisms and positional embeddings. Our experiments
demonstrate that the Transformer model with one transformer block layer
surpass the performance of the widely used base architecture, LSTM. Moreover,
BERT or CANINE, the pre-trained transformer models, outperforms in classifying
highly imbalanced malware families according to evaluation metrics: F1-score
and AUC score. Furthermore, our proposed bagging-based random transformer
forest (RTF) model, an ensemble of BERT or CANINE, reaches the state-of-the-
art evaluation scores on the three out of four datasets, specifically it
captures a state-of-the-art F1-score of 0.6149 on one of the commonly used
benchmark dataset.

###### keywords:

Transformer , Tokenization-free , API Calls , Imbalanced , Multiclass , BERT ,
CANINE , Ensemble , Malware Classification

††journal: Computers & Security

[inst1]organization=Cyber Security Gradaute Program,addressline=Kadir Has
Unviersity, city=Istanbul, country=Turkey

[inst3]organization=Huawei R&D Center,city=Istanbul, country=Turkey

[inst2]organization=Management Information System,addressline=Kadir Has
University, city=Istanbul, country=Turkey

##  1 Introduction

In recent times, with our dependence on information technologies, the Internet
has been widely used by people of all ages. Those who want to quickly meet
their daily needs such as online banking, online shopping, health, and
transportation-related transactions cause an enormous increase in internet
usage as well. This exponential growth of the usage of Internet plays a
significant role in making life easier. On the other hand, this situation
poses a severe threat as cyber attacks increase drastically in parallel with
the growth of the Internet. Among these cyber attacks, malicious software
(malware) is the primary weapon for attackers to conduct their malicious
activities against a victim’s machine such as computer, smartphone, or
computer networks in order to disrupt system’s functions and gain unauthorized
access [1, 2].

Cybercriminals use several ways to spread malware, such as phishing e-mails
with malicious links and attachments, text messages, and malicious
advertisements etc. According to the state of e-mail security report released
by Mimecast in 2021, 61% of organizations were exposed to e-mail-based
ransomware in 2020, with an increase of 10% compared to the previous year [3].
The average amount spent to recover from a ransomware attack, when factors
such as downtime, device, human, and network costs are included, is about
$1.85 million as reported by Sophos [4]. According to another cyber threat
report published by SonicWall, 5.6 billion malware attacks were carried out in
2020 [5].

In lieu all of these findings, one can safely claim that excessive malware,
without considering the identification/classification methods, affects many
victims destructively. Since the numbers of malicious software and the damages
they cause to the institutions are increasing every day, it is crucial to map
malware behavior that can be provided by malware family identification so that
security researchers and incident responders can speed up the recognition and
mitigation processes.

There are two main approaches used the most to detect malware. One of them is
the signature-based malware detection method. The signatures, sequences of
bytes, created using static, dynamic, or hybrid methods are uniquely located
in the database. Whether a given file is malware or not is determined by
looking at the unique signature of this file from a predefined database [6].
Although signature-based methods are the most generally utilized procedure in
antivirus programming, since the only predefined list of known malware
variants are kept, they are not able to catch previously unidentified malware
[7].

The other main malware detection approach is behavior-based method, which
examines the behavior and characteristics of a given file and then decides
whether the related file is malware, and if it is a malware, then the approach
also defines the malware family the file belongs to. Although the effort and
time spent and the storage complexity are much more, the unknown attacks can
be detected and classified by using behavior-based methodologies better
contrary to the signature-based methods [8].

In the report released by SonicWall, among detected malware, 268,362 of them
have never been seen before in 2020, with a rise of 74% from the preceding
year [5]. Considering the increasing number of unseen malware over the years,
performing a behavior-based approach is more reasonable. This report indicates
the significance of developing more innovative and effective malware defense
mechanisms to detect and classify unknown malware.

The effectiveness of the malware defense mechanism is directly associated with
the right choice of behavioral features exploited from malware. Several
features can be extracted from malware due to its diverse nature. Obtaining
adequate features is time-consuming for a model. This situation can make
learning difficult for a model if some of the features used are non-
distinctive [9]. In our study, both static and dynamic API call sequences are
leveraged to classify malware families since these sequences represent
behavioral patterns for each sample. Considering API call sequences, machine
learning becomes the primary choice to capture sequence relationships between
the sequence elements for malware classification.

Different machine learning algorithms have been used in the literature for
malware detection and classification so far [10, 7]. Considering the sequence,
traditional machine learning models may not be sufficient as the relations
among API calls must be taken into account to successfully predict the malware
families of unseen API call sequences. The current deep learning based models,
mainly pre-trained transformer models outperform traditional machine learning
based approaches for sequential text classification [11, 12].

In this paper, we have answered the following research questions respectively:

> _RQ.1: What are the suitable classification metrics for imbalanced datasets
> in multiclass malware classification?_

> _RQ.2: What are the appropriate base models for multiclass malware
> classification based on API call sequences?_

> _RQ.3: What are the effects of pre-processing on API call sequences to the
> model results?_

> _RQ.4: What are the effects of tokenizer-based (word piece) pre-trained
> transformer model (e.g. BERT) and tokenizer-free transformer model (e.g.
> CANINE) to our model results?_

> _RQ.5: What is the effect of ensemble of pre-trained transformer models,
> BERT and CANINE, which is based on bagging for imbalanced multiclass malware
> classification?_

Our main contributions through this study, in the light of the answers to our
research questions, can be summarized as follows:

  * 1.

Noticing inconsistent evaluation results due to a logical error in the code of
a published article [13].

  * 2.

To the best of our knowledge, we have used the pre-trained CANINE transformer
model for the first time in the field of malware in this study.

  * 3.

Again, to the best of our knowledge, a bagging-based ensemble of pre-trained
transformer models has been used for the first time in malware classification.

  * 4.

Our proposed model Random Transformer Forest (RTF), has surpassed the state-
of-the-art results obtained in the malware classification.

  * 5.

We have achieved a state-of-the-art result on one of the well-known API call
dataset in the literature [14] with our proposed RTF model.

This paper is structured as follows: Section 2 presents the related work. The
description of the datasets, base models, pre-trained models, and our proposed
model are presented in Section 3. The test results are discussed and compared
with the related studies in Section 4, and lastly, the conclusion and future
work are given in Section 5.

##  2 Related Work

Cybercriminals leverage malware to exploit any device or system to steal
sensitive data and hence cause enormous problems for victims. Analyzing and
classifying incoming malware helps us define the problem and understand how to
recover from the damage as quickly as possible.

There are two techniques most commonly used in malware analysis, static
analysis and dynamic analysis. Static analysis is a process of malware
analysis that analyzes the given malware without running it. Unlike static
analysis, a given malware file is executed in an isolated environment to avoid
harming the computer system in dynamic analysis.

Malware developers may implement various techniques to evade detection
mechanisms such as code obfuscation, dynamic code loading, polymorphism, and
metamorphism. For instance, the MD5 hash based detection method can be easily
bypassed by malware authors with the methods mentioned above. As these methods
cause the binary of the file to change, they also cause a change in the hash
of the file. While the hash of the malicious file is changed and the file is
defined as benign, the behavior of the file, thus its effect, remains
unchanged [7].

Since dynamic analysis requires the execution of a given sample to be
monitored and observed in an isolated environment, malware even written with
code obfuscation techniques hardly eludes dynamic analysis contrary to static
analysis. This identified situation provides dynamic analysis to be more
robust than static analysis [15].

Performing dynamic analysis requires more time than static analysis and
organizations are dealing with millions of attacks carried out in a day. These
shortcomings provide an excellent opportunity for machine learning to
collaborate with dynamic and static analysis since machine learning can handle
large volumes of data [16].

In the malware detection and classification process, understanding malware
behavior is one of the substantial parts of detecting and classifying malware.
Dynamic API calls are obtained by tracing the sequences of calls by way of
calling operating system services such as creating a file and allocation of
virtual memory by malware samples. On the other hand, static API calls are
extracted from portable executable (PE) format of the executable files. Static
API calls are unordered as the sequences of calls are not traced unlike
Dynamic API calls [17]. In general, since API call sequences generate specific
behavioral patterns and hence represent malware families, they can be
considered as one of the most distinguished features among malware families
[18, 19].

Related studies about base models for malware analysis using API calls and
transformer-based models on sequence problems will be examined respectively in
the rest of this section.

###  2.1 Base Models for Malware Analysis using API Calls

In [20], the authors used DNA sequence alignment algorithms, Multiple Sequence
Alignment (MSA), and Longest Common Subsequence (LCS) to extract the most
critical API call sequence patterns among different malware families and
generate a signature-based malware detection mechanism to determine whether a
program is a malware or not based on these extracted patterns. The API call
sequences determined by the MSA and LCS algorithms can be misleading for the
model if sequences get more extended than a preset API call sequence length.

In [21], the authors proposed a model using text mining and topic modeling for
feature extraction and selection processes based on API call sequences.
Machine Learning based Group Method of Data Handling (GMDH) method,
traditional machine learning models, Random Forest (RF), Decision Tree (DT),
Support Vector Machine (SVM), and Multilayer Perceptron (MLP) are compared on
two different datasets. Although DT and SVM models outperformed the results,
and they suggest DT for malware detection expert system, the size of the
datasets are inadequate to rely on the models.

In [22], the authors integrated Convolutional Neural Network (CNN) and
Recurrent Neural Network (RNN) layer into one neural network architecture.
With this model, they have achieved the accuracy score of 89.4%, the precision
score of 85.6%, and the recall score of 89.4% to classify malware families.
Newly generated subsequences of original API call sequences are given to the
model as an input. For each API call sequence, if the same API call is
repeated more than two times in a row, only two consecutive identical API
calls are included in the resulting sequence. Since their corpus contains only
60 different API calls, they did not set any boundaries. Otherwise, they may
have to set a predetermined length to avoid tracking loops and make the model
less complex.

In [23], two stages of Deep Neural Networks are applied for the malware
detection process. The proposed model CNN is used to classify feature images
extracted with Long Short-Term Memory (LSTM) model using API calls. Although
the authors achieved an Area Under Curve (AUC) score of 96%, since the size of
the dataset is relatively small, the score may be misleading.

In [24], the authors leveraged N-gram and Term Frequency–Inverse Document
Frequency (TF–IDF) for feature extraction and selection, respectively. The
proposed LSTM model is used for binary classification, benign or malware,
using API call sequences. The authors reached a 92% accuracy score on unknown
test API call sequences.

In [25], the authors trained two different LSTM networks on system call
sequences for both malware and benign Android applications, respectively. The
new sequence has been classified by comparing two similarity scores obtained
from two different LSTM networks. The LSTM model has been compared with two
n-gram models, MALINE and BMSCS, based on accuracy, precision, recall, and
False Positive Rate (FPR). They have shown that the LSTM model outperformed
MALINE and BMSCS models.

In [26], the authors tried several models from LSTM to traditional machine
learning models, RF, DT, SVM, and K-Nearest Neighbour (KNN), on a dataset
containing 7,107 samples of API call sequences generated by them. In
multiclass classification, they have achieved the highest F1-score of 47%
using the single-layer LSTM model compared to the two-tier LSTM and common
machine learning models.

In [27], the authors generated new API call sequences by applying data pre-
processing steps to benchmark dataset [14]. If any unique API call repeated
more than once in the sequence, they kept only one and removed the
continuously same API. Similarly, they removed continuously same sub sequences
when the length of sub sequences are two or three. They have proposed two
different models, LSTM and Gated Recurrent Unit (GRU) models, which are based
on RNN architecture. They have significantly increased their precision,
recall, and F1-score after data pre-processing.

In [28], similar to feature pre-processing method applied in previous work
[22], the authors prepared new API call sequences and kept only 100 non-
consecutive sequences to avoid repeating API calls loop. They achieved similar
AUC score and F1-score results compared to LSTM with their proposed model,
which is based on Deep Graph Convolutional Neural Networks (DGCNNs).

LSTM model is widely used as the underlying architecture for malware detection
and classification based on API calls, as seen in the above mentioned studies.

###  2.2 Transformer Based Models

This section is structured as follows: Section 2.2.1 introduces the pre-
trained transformer models. Section 2.2.2 presents the related work relevant
to the transformer based models on sequence problems.

####  2.2.1 Pre-trained Transformer Models

With the increasing use of deep learning approaches, the number of model
parameters and hence the need for a much larger dataset to train these model
parameters have increased. Since constructing a large labeled dataset is time-
consuming and requires extremely expensive annotation costs, contrary to
constructing a large unlabeled dataset, pre-trained models have gained
importance. Models that are pre-trained on the huge unlabeled text data learn
good universal representations, which are used to fine-tune the model on the
downstream tasks [29]. On the other hand, the pre-trained transformer models
have reached state-of-the-art results as these models are capable of capturing
dependencies over a wide range of scales unlike convolutional and recurrent
networks [30, 31]. Pre-trained techniques leveraged to capture these
dependencies are elaborated in the sections 3.5.1 and 3.5.2.

####  2.2.2 Transformer Based Models on Sequence Problems

In [32], the authors used several deep learning models from mostly used
traditional deep learning methods such as CNN, RNN, LSTM, GRU, and BiLSTM with
GLOVE and fastText embedding to pre-trained transformer models for multiclass
text classification. For their experiment, they utilized the highly imbalanced
RCV1-v2 dataset, which contains 800,000 news stories. They have shown that
transformer models outperformed traditional deep learning models based on
F1-score for multiclass text classification.

In [33], the authors used the BERT model for the cyber-bullying detection
task. The proposed model has been tested on three different datasets taken
from Twitter, Wikipedia, and FormSpring. Compared to common machine learning
models, SVM and Logistic Regression (LR), and deep-learning based models, CNN,
RNN + LSTM, and Bidirectional LSTM (BiLSTM), they have achieved higher
F1-scores.

In [34], the authors generated word embeddings for each opcode of malware
samples by using Word2Vec and BERT. They classified malware with different
classifiers such as LR, SVM, and MLP to see the effect of different word
embeddings. They have achieved higher results using BERT for word embeddings
with the same set of input parameters and the same set of classifiers based on
classification accuracy among five unique malware families distributed almost
balanced.

In [35], they have proposed transformer-based architecture for detection and
classification of malware using opcode sequences of windows executable files.
The proposed transformer model has achieved better results compared to
Gradient Boosting Method (GBM) and BiLSTM based on accuracy, precision,
recall, and F1-score evaluation metrics.

In [36], the authors proposed a pre-trained transformer model, Malbert, which
is pre-trained on 15,000 malware and 15,000 benign samples [20] first to learn
the relationships among API calls. This pre-trained transformer model and
existing pre-trained transformer model, _Bert-base-uncased_ are fine-tuned on
two different datasets for the malware detection process. Pre-trained
transformer models have achieved higher results compared to LSTM model and
traditional machine learning models based on mostly used evaluation metrics,
such as accuracy, precision, recall, and F1-score.

In [37], the authors proposed a model, called CyberBert which uses
bidirectional transformer architecture for two different tasks, session-based
recommendation, and malware classification based on API calls. Compared to the
LSTM model and transformer-encoder, a unidirectional model, they have achieved
higher F1-scores for binary and multiclass classification with CyberBert.

In [38], the authors leveraged pre-trained BERT transformer model for malware
detection, malware, and benign, on Android operating system API calls called
by the application. The set of experiments made by the study shows BERT model
obtained state-of-the-art results compared to the LSTM model on sequence
classification.

Recent surveys [29, 12] with the studies mentioned above clearly show that
current transformer-based models, mainly pre-trained transformer models fine-
tuned on downstream tasks, outperformed traditional machine and deep learning
models on sequence classification.

##  3 Methodology

In the methodology section, firstly, the datasets used in experiments are
introduced. Secondly, the most suitable evaluation metrics for highly
imbalanced datasets are specified. Then, base model structures, the effect of
the pre-processing method, pre-trained transformer models, CANINE and BERT,
and the proposed RTF model architectures are explained.

###  3.1 Datasets

To verify how effectively a model classifies malware, it is necessary to test
the model on different malware datasets. Since malware constantly evolves,
working on an up-to-date malware dataset is required to assess the
effectiveness of proposed models. Comparing several models on a single dataset
containing outdated malware samples and highlighting one model might not be
reliable. Thus, four different datasets containing API call sequences of
malware samples and their corresponding malware families are utilized to
evaluate the models used in this study. Of these four datasets used in this
study, Catak and Oliveira datasets are created with dynamic API calls and
VirusSample, whereas VirusShare datasets are constructed with static API
calls.

####  3.1.1 Dynamic API Call Datasets

#####  3.1.1.1 Catak Dataset

This study obtained sequences of Windows Operating System API calls within the
Cuckoo Sandbox isolated environment for each malware file. Malware family
labels were determined using unique hash codes of each malware on the Virus
Total website. In total, 7,107 samples, which contain hash codes of malware,
Windows operating system API call sequences, and their malware family classes,
were created [14].

#####  3.1.1.2 Oliveira Dataset

42,797 malware and 1,079 benign API call sequences were obtained via Cuckoo
Sandbox for dynamic malware analysis. Instead of using whole API call
sequences, the first 100 non-consecutive API call sequences were extracted
from the parent processes to reduce complexity and detect the malicious
pattern as quickly as possible. The generated dataset containing hashcodes,
label (malware or benign), and 100 non-consecutive API calls for each sample
has been used for binary malware classification [28].

Since we are working on a multiclass classification problem, malware families
of 42,797 malware samples are determined through virus total. Out of 42,797
malware samples, 2,081 were labeled as "unknown" by virus total. Several
malware families hold a small number of malware samples, less than 100. These
malware samples are removed since they could be misleading for the models.
Thus, the dataset in question has been turned into a multiclass classification
case. The compiled dataset used in our study consists of 40,566 malware
samples with their API call sequences and malware families.

####  3.1.2 Static API Call Datasets

#####  3.1.2.1 VirusShare

Unique hash codes represent malware samples obtained from Virus Share. Each
unique hash code in text files is passed to Virus Total to learn their
corresponding malware families. Python module named PEfile is leveraged to
extract API calls from each malware sample. Lastly, malware families having
less than 100 samples are removed. Thus, 13,849 malware samples with their
corresponding API call sequences and malware families are obtained [39].

#####  3.1.2.2 VirusSample

Malware samples taken by Virus Sample are kept with their unique hash code
text files. Corresponding malware families and API calls are obtained from
Virus Total site and PEfile module, respectively, as in Virus Share. Finally,
malware families having less than 100 are removed from the dataset. Therefore,
9,732 malware samples with their corresponding API call sequences and malware
families are obtained [39]. Since the malware samples in this dataset consist
of the most up-to-date data based on API calls, we also find an opportunity to
test our models on recent malware samples. Table 1 shows the total samples of
malware families for each dataset.

Table 1: Distribution of malware families. Malware Family | Oliveira | VirusShare | Catak | VirusSample
---|---|---|---|---
Trojan | 31,979 | 8,919 | 1,001 | 6,153
Virus | 102 | 2,490 | 1,001 | 2,367
Adware | 5,444 | 908 | 379 | 222
Backdoor | 135 | 510 | 1,001 | 447
Downloader | 1,948 | 218 | 1,001 | N/A
Worms | N/A | 524 | 1,001 | 441
Agent | 220 | 165 | N/A | 102
Ransomware | 404 | 115 | N/A | N/A
Dropper | 118 | N/A | 891 | N/A
Riskware | 216 | N/A | N/A | N/A
Spyware | N/A | N/A | 832 | N/A
Total | 40,566 | 13,849 | 7,107 | 9,732

###  3.2 RQ.1-) What are the suitable classification metrics for imbalanced
datasets in multiclass malware classification?

The degree of imbalance may vary within different domains. One of these
domains is malware, as specific malware families are used chiefly in
particular periods for cyber attacks.

According to the report released by Malwarebytes: The total number of Trojan
detected by Malwarebytes is almost 26 times higher than the total number of
Worm in 2018. The total number of Riskware detected by Riskware tools in 2019
was 6,632,817, with a decrease of 35% compared to the previous year. In
another chart containing the number of detection of malware families by
months, it is seen that the number of Trojan attacks increased dramatically at
the beginning of 2019, with the spread of the Emotet, one of the advanced
Trojan campaign in that period [40].

These situations demonstrate that there could be significant differences in
the distribution of malware families according to years or even months.
Therefore, when the collected malware is classified according to their
families, the distribution will vary according to the malware type prevailing
at the time of collection and hence lead to imbalance.

For these reasons, almost all of the datasets belonging to malware have an
imbalanced class distribution in the literature. Thus, we are required to
leverage the most suitable metrics to evaluate our classification performance
on imbalanced datasets.

The datasets leveraged in our study have highly imbalanced class distribution
as expected and shown in Table 1.

Evaluation metrics are one of the crucial steps to assess model performance.
An incorrectly chosen evaluation metric can make a poor performance algorithm
seems effective. The metrics used to evaluate a model performance may vary for
balanced and imbalanced datasets. For instance, using accuracy metric for a
balanced dataset may provide an objective evaluation, yet may not be the right
choice for an imbalanced dataset as it has a bias against the majority class.
Taking malware classification for example. Assume there are six different
classes in the dataset, and 95% of the samples belong to Trojan. In this case,
a dummy model that predicts all samples in the test data as Trojan will
achieve an accuracy score of 95% even though it does not predict any other
classes correctly. It may not always be correct to use the most widely
preferred evaluation metric without examining the distribution of classes in
the dataset for the reasons mentioned above.

Recently, researchers have started to use Matthew’s Correlation Coefficient
(MCC) to evaluate model performance on imbalanced datasets. Although this
metric was previously used mostly in biomedical research, it has now been used
in many areas, including malware classifications [41, 42] yet according to the
experimental results to investigate the behavior of MCC metric, MCC is not
suitable for directly applying on imbalanced datasets [43]. Empirical research
conducted on 54 imbalanced datasets demonstrates that the AUC score is more
discriminating than MCC [44]. Most frequently used metrics to assess model
performance for multiclass classification tasks have been shown to be
inadequate on imbalanced datasets such as Precision, Recall, MCC, Confusion
Entropy (CEN), Classes Average Accuracy (AvAcc), and Class Balanced Accuracy
(CBA) [45].

Although the choice of right metrics is still an open issue, following the
searches to find the most suitable metrics used for multiclass classification
on imbalanced datasets, we have used AUC, which is a summary of probability
curve, Receiver Operating Characteristic (ROC), based on FPR and True Positive
Rate (TPR), as an evaluation metric. On the other hand, F1-score has been used
to be comparable with studies conducted on one of the well-known datasets in
the literature [26].

The equation (1) and (2) define the recall and precision metrics respectively.
The equation (3) defines F1-score in terms of precision and recall. Also, the
equation (3) contains the explicit form of the formula in terms of True
Positive (TP), False Negative (FN), and False Positive (FP).

$Recall=\frac{TP}{TP+FN}$ (1) $Precision=\frac{TP}{TP+FP}$ (2)
$F_{1}score=\frac{2*Precision*Recall}{Precision+Recall}=\frac{2*TP}{2*TP+FP+FN}$
(3)

###  3.3 Base Models

LSTM and One-layer Transformer Block Based Transformer architecture are
leveraged as base models for malware classification based on _RQ.2_. For the
rest of the paper, One-layer Transformer Block Based Transformer architecture
is referred to as _Transformer_ model.

####  3.3.1 LSTM Based Malware Classification

Recurrent Neural Networks (RNNs) have a structure that uses recurrent
relation, which is a situation of performing the same step, processing current
output depends on the previously computed hidden state, for each element of a
sequence repeatedly. In these structures, information is retained through the
previous hidden state, and hence processing continues in time steps. The
recursion between sequence elements hinders parallelization during the
training phase and consequently causes a longer runtime for training.

LSTM can learn relatively long-term dependencies compared to other RNNs
because it provides deeper processing of hidden states through specific units.
This situation causes an increase in the number of parameters used for
training. Besides, since LSTM has a recursive structure like other RNNs
inherently and hence can not be trained in parallel, the training period may
take relatively longer compared to other RNNs [46].

Since our purpose is to classify malware families, fully connected layer
output that captured the information from the LSTM networks, is given to the
softmax layer for multiclass classification.

The standard LSTM network is preferred as one of the base models for
comparison since it has been used widely as a base network and performed
successfully for several malware classification problems using API call
sequences [47].

####  3.3.2 Transformer Model

Transformer model is a recently used network architecture that designed to
overcome the deficiencies of sequence-to-sequence neural network approaches
such as LSTM and RNN for the sequence modeling and transduction problems in
2017 [48].

Figure 1: Transformer model architecture.

Since transformer-based architectures avoid recursion, they can overcome the
parallelization problem that both the LSTM and other RNNs suffer from. In
traditional sequence-to-sequence architectures the information coming from the
previously hidden state is processed recursively to capture dependencies. On
the contrary, since the transformer models refrain from recurrence and
convolution, positional encodings are used to preserve the order of the
sequence and provide position-related information of the tokens in the
sequence. Transformer models leverage the attention mechanism to caption and
preserve long-term dependencies for the sequences processed as a whole.
Positional information is retained with attention layers instead of the
recurrent and convolutional layers in transformer model [48]. Figure 1 shows
the Transformer model architecture utilized in this study.

###  3.4 Pre-processing Method on Datasets

Each API call sequence is pre-processed, similar to the steps taken part in
[27]. Pre-processing part consists of 3 main steps.

In the first step, for any API call in a sequence that repeated more than one
time in a row, the continuously same API calls are removed from the sequence.
This pre-processing step generates a new sequence that does not consist of the
consecutive same API call. In the second and third steps respectively,
repetitive binary and triple sub-sequences are removed from the new sequence
created by the first step. The following Figure 2 shows the pre-processing
step outputs respectively based on randomly given sequences.

Figure 2: The outputs of pre-processing steps.

These pre-processing steps have not been applied to the Oliveira dataset as
this dataset has already given pre-processed and limited to the 100 non-
consecutive API calls only. On the other hand, although the pre-processing
steps have been applied to the VirusSample and VirusShare datasets as well,
only one sample out of 9,732 samples for VirusSample dataset and only two
samples out of 13,849 samples for VirusShare dataset are affected. Of these
affected samples, only two or three API calls are affected. The results of the
pre-processing steps applied on VirusSample and VirusShare datasets show us
that static API calls do not include noisy and redundant API calls as expected
contrary to dynamic API calls. Thus, we have continued with the original API
call sequences of VirusSample and VirusShare datasets. After performing pre-
processing steps on Catak dataset, only 11 API call sequences remained
constant. The effect of the pre-processing steps on the Catak dataset can
easily be seen by the Figure 3 shown below.

Figure 3: The effect of pre-processing on Catak dataset.

After pre-processing steps, the number of samples where the length of the API
call sequences is less than 200 has increased more than twice, and the number
of samples where the length of the API call sequences is more than 200 has
decreased more than twice. These changes clearly show the effect of pre-
processing steps on the Catak dataset. Finally, after performing all the pre-
processing steps to the Catak set only, the effect of the pre-processing is
examined with model performances.

###  3.5 CANINE and BERT

Large-scale pre-trained models have recently become very popular in the field
of artificial intelligence. Due to the model previously trained on large-scale
data, captured information can be used for specific tasks that utilize the
pre-trained model by fine-tuning [49]. This study utilizes two different pre-
trained models architectures, BERT and CANINE.

####  3.5.1 BERT

BERT, Bidirectional Encoder Representations from Transformers, is a language
model that uses transformer architecture which is pre-trained on Wikipedia and
Book Corpus of unlabelled text [50].

BERT preserves the semantic content thanks to the masked language modeling
(MLM) and next sentence prediction (NSP) unsupervised tasks, which enables to
generate deep bidirectional representations while pre-training. BERT uses
Word-piece tokenization to create a token vocabulary that consists of learned
representations of the words. Figure 4 shows BERT architecture in details.

####  3.5.2 CANINE

CANINE, Character Architecture with No tokenization In Neural Encoders, is a
tokenizer-free pre-trained encoder model that is designed to overcome the
shortcomings of the tokenization process such as word-piece and sentence-piece
tokenization [51]. For example, a pre-trained model that uses specific
tokenization may not be convenient for specialized domains. In [52], it is
shown that word-piece tokenization based pre-training strategy is not well-
suited compared to character-piece when fine-tuned on medical data.

Similar to BERT [50], CANINE is pre-trained on the MLM and NSP tasks as well.
Unlike commonly used pre-trained models, CANINE uses neural encoders that
encode the sequence of characters or optionally sub-words are used as a soft
inductive bias without doing explicit tokenization on input data. The CANINE
structure is shown in Figure 5.

In general, the BERT model is widely used by many studies for malware
classification, as shown in the Related Work. We have assumed that the
tokenization-free strategy used by the CANINE model might be well-suited for
API calls since an API call such as ’ldrloaddll’ may not be appropriate for
word-tokenization. Therefore, we have included the CANINE model in our study.

Thus, BERT, CANINE-C (Pre-trained with autoregressive character loss), and
CANINE-S (Pre-trained with subword loss) pre-trained transformer models are
leveraged regarding the _RQ.4_.

Figure 4: BERT architecture. Figure 5: CANINE architecture.

###  3.6 Proposed Model: Random Transformer Forest (RTF)

There are several important studies that show the success of using different
ensemble types of pre-trained transformer models such as stacking and majority
voting of heterogeneous pre-trained transformer models on varying downstream
tasks. [53, 54, 55]. Unlike these type of ensemble models, Random Transformer
Forest (RTF) is a bagging-based ensemble model inspired from the Random Forest
(RF) machine learning model [56]. Similar to RF, using an ensemble of pre-
trained transformer models is assumed to increase classification performance
on highly imbalanced malware datasets rather than using a single pre-trained
transformer model [57, 58].

The training phase requires creating _N_ different training subsets by using
the bootstrap sampling method from the original training set. The malware
class distribution coming from the original training set must be preserved in
the resampling step due to the highly imbalanced class distribution. After the
resampling step, each training subset is used to fine-tune the pre-trained
transformer model. Each pre-trained transformer model has the same structure.
Therefore base estimators are homogeneous.

In the testing phase, each fine-tuned transformer model takes a given malware
API call sequence, and the class probabilities of each fine-tuned transformer
model are aggregated by taking the average. For the majority voting, the final
prediction is accepted as the malware family, which takes the highest
probability coming from the aggregation part. Figure 6 shows the structure of
RTF model.

Figure 6: The proposed RTF model architecture.

##  4 Experiment and Results

Experimental setup and conducted experiments will be clarified respectively
regarding the predetermined Research Questions except _RQ.1_ which is
elaborated in _Methodology_ part.

###  4.1 Experimental Setup

We have utilized the google-cloud colab pro+ for our experiments. We have
worked on Tesla T4 GPU with 51 GB available RAM provided by the colab
platform. The Keras framework [59] is used for base model comparison. The
PyTorch framework [60] is leveraged with pre-trained models taken from
HuggingFace [61] for pre-trained transformer models and RTF model. All jupyter
notebook files that contain source codes regarding the research questions,
from base model comparison to RTF model, can be found in the Github repository
111https://github.com/Ferhat94/Random-Transformer-Forest.

###  4.2 RQ.2-) What are the appropriate base models for multiclass malware
classification based on API call sequences?

In this part of the study, Transformer and LSTM model are leveraged as base
architectures for comparison on four datasets.

All datasets used to evaluate the base model performances are divided into
three parts, training, validation, and testing. 20% of the original datasets
are allocated for testing. The splitting data process is performed in a
stratified way as to preserve class distribution is one of the crucial steps
on highly imbalanced datasets. Considering the imbalanced distribution of the
classes, we have leveraged the class weight approach to give different weights
to both the majority and minority classes so that class weights are taken into
account by training algorithms.

Stratified 10 Fold strategy is used on training data for each dataset, and 10%
of training data is used for validation for each iteration. Thus, we guarantee
that each fold has the same distribution of malware families and ensure that
every sample from the dataset has the chance of appearing in both training and
validation data. Standard deviation and mean of 10 validation results are
calculated for each evaluation metric used in our study, and training runtime
for robust interpretation.

We have provided a dummy classifier as a simple baseline since class
distribution of each dataset is imbalanced. We have used the "most frequent"
strategy for our dummy classifiers. Base model comparison results on each
dataset is shown in the tables 3, 2, 4 and 5.

Table 2: Base model comparison results for Catak [14] dataset. Base Model | F1-score | AUC score | Training Time (s)
---|---|---|---
Validation Mean Scores
LSTM | 0.4873 $\pm$ 0.0126 | 0.7887 $\pm$ 0.0128 | 68.62 $\pm$ 11.44
Transformer | 0.5689 $\pm$ 0.0578 | 0.8676 $\pm$ 0.0329 | 11.74 $\pm$ 3.43
Test Scores
LSTM | 0.4638 | 0.7885 |
Transformer | 0.5042 | 0.8246 |
Dummy | 0.0308 | 0.5000 |
Table 3: Base model comparison results for Oliveira [28] dataset. Base Model | F1-score | AUC score | Training Time (s)
---|---|---|---
Validation Mean Scores
LSTM | 0.5570 $\pm$ 0.0182 | 0.8853 $\pm$ 0.0142 | 111.99 $\pm$ 20.11
Transformer | 0.5792 $\pm$ 0.0379 | 0.9280 $\pm$ 0.0142 | 33.48 $\pm$ 7.91
Test Scores
LSTM | 0.5637 | 0.8844 |
Transformer | 0.5650 | 0.8855 |
Dummy | 0.0980 | 0.5000 |
Table 4: Base model comparison results for VirusSample [39] dataset. Base Model | F1-score | AUC score | Training Time (s)
---|---|---|---
Validation Mean Scores
LSTM | 0.7690 $\pm$ 0.0419 | 0.9656 $\pm$ 0.0110 | 21.80 $\pm$ 4.04
Transformer | 0.8070 $\pm$ 0.0323 | 0.9885 $\pm$ 0.0055 | 13.26 $\pm$ 3.38
Test Scores
LSTM | 0.7531 | 0.9701 |
Transformer | 0.7548 | 0.9680 |
Dummy | 0.1291 | 0.5000 |
Table 5: Base model comparison results for VirusShare [39] dataset. Base Model | F1-score | AUC score | Training Time (s)
---|---|---|---
Validation Mean Scores
LSTM | 0.7121 $\pm$ 0.0231 | 0.9274 $\pm$ 0.0130 | 96.79 $\pm$ 8.38
Transformer | 0.7641 $\pm$ 0.0297 | 0.9700 $\pm$ 0.0182 | 31.76 $\pm$ 9.80
Test Scores
LSTM | 0.7071 | 0.9298 |
Transformer | 0.7125 | 0.9350 |
Dummy | 0.0980 | 0.5000 |

Considering the two base models, LSTM and Transformer model, the standard
deviation of the mean validation scores of evaluation metrics for the
Transformer model is higher. Even in this case, we get higher results on
unseen test data.

Evaluation results demonstrate that the Transformer model is more reasonable
to continue with compared to the LSTM model in terms of evaluation results and
training time.

###  4.3 RQ.3-) What are the effects of pre-processing on API call sequences
to the model results?

Data pre-processing steps mentioned in Pre-processing Method on datasets part
are applied to the Catak dataset [14] only as pre-processing has no effect on
VirusSample and VirusShare datasets given in the study [39] and Oliveira
dataset [28] has already been pre-processed and limited with 100 non-
consecutive API calls.

Original API call sequences and pre-processed API call sequences on Catak
dataset have been compared with the LSTM and Transformer model.

Table 6 shows the comparison results of Original API call sequences and pre-
processed API call sequences.

Table 6: Comparison of the original and pre-processed Catak datasets. Base Model | F1-score | AUC score | Training Time (s)
---|---|---|---
On Original API Call Sequences
LSTM | 0.4638 | 0.7885 | 68.62 $\pm$ 11.44
Transformer | 0.5042 | 0.8246 | 11.74 $\pm$ 3.43
On Pre-processed API Call Sequences
LSTM | 0.5020 | 0.8156 | 71.58 $\pm$ 12.27
Transformer | 0.5106 | 0.8372 | 18.27 $\pm$ 8.74

Evaluation results demonstrate that pre-processing step outperforms for Catak
dataset. Both the AUC score and F1-score have increased after the pre-
processing steps for both LSTM and Transformer model. Although we expect
training runtime to decrease after pre-processing steps, training runtime on
the original Catak dataset is lower as the monitored evaluation metric,
validation AUC, has stopped improving during the learning phase. Therefore,
both LSTM and Transformer models have started to learn after pre-processing
steps on the Catak dataset. New sequences generated after pre-processing steps
are leveraged for the following experiments, pre-trained models and RTF model
for the Catak dataset.

###  4.4 RQ.4-) What are the effects of tokenizer-based (word piece) pre-
trained transformer model (e.g. BERT) and tokenizer-free transformer model
(e.g. CANINE) to our model results?

In this part, Due to the large number of parameters used in pre-trained
models, 20% of the training data is allocated for validation instead of the
stratified k fold strategy. The CANINE model uses two different pre-training
strategies: sub-word loss (Canine-s) and character loss (Canine-c). In our
studies both pre-training strategies are leveraged to see which pre-training
strategy performs better for each dataset. Only the best Canine model,
Canine-c or Canine-s, is included to our final results from tables 8, 9, 10
and 11 for each individual dataset.

The comparison of the pre-trained models, BERT and CANINE, is shown at the end
of the experiments with the RTF model results to see the comparison clearly.

###  4.5 RQ.5-) What is the effect of ensemble of pre-trained transformer
models, BERT and CANINE, which is based on bagging for imbalanced multiclass
malware classification?

In the RTF model _N_ different training subsets, thus _N_ different base
estimators are utilized to fine-tune the _N_ different pre-trained BERT or
CANINE model. We have tried several combinations of _N_ and pre-trained
transformer models, BERT and CANINE, for each dataset. As a result of several
trials, the combinations that provided the best scores are accepted as our RTF
score. Table 7 shows the best combination for each dataset and model
comparison results on each dataset are shown in tables 8, 9, 10 and 11.

Table 7: Best parameters for RTF model. Dataset | Number of Base Estimators (N) | Pre-trained Model
---|---|---
Catak [14] | 6 | BERT
Oliveira [28] | 2 | BERT
VirusSample [39] | 10 | CANINE-S
VirusShare [39] | 5 | CANINE-S
Table 8: All model comparison on Catak [14] dataset. Model | F1-score | AUC score
---|---|---
Transformer | 0.5106 | 0.8372
CANINE-S | 0.5633 | 0.8339
BERT | 0.5919 | 0.8735
RTF | 0.6149 | 0.8818
Table 9: All model comparison on Oliveira [28] dataset. Model | F1-score | AUC score
---|---|---
Transformer | 0.5650 | 0.8855
CANINE-S | 0.4725 | 0.8636
BERT | 0.4839 | 0.8321
RTF | 0.4061 | 0.8714
Table 10: All model comparison on VirusSample [39] dataset. Model | F1-score | AUC score
---|---|---
Transformer | 0.7548 | 0.9680
CANINE-C | 0.7893 | 0.9570
BERT | 0.7759 | 0.9690
RTF | 0.8059 | 0.9773
Table 11: All model comparison on VirusShare [39] dataset. Model | F1-score | AUC score
---|---|---
Transformer | 0.7125 | 0.9350
CANINE-S | 0.7064 | 0.9286
BERT | 0.7145 | 0.9364
RTF | 0.7275 | 0.9513

According to the results, at least one of the pre-trained transformer models,
CANINE or BERT, surpassed Transformer model, and the RTF model obtained the
highest scores on three out of four datasets.

The performance of our proposed model RTF on VirusShare, VirusSample, and
Oliveira dataset for each imbalanced class is shown in Figure 7. The
performance of the RTF on Catak dataset for each imbalanced class is shown in
a separate figure as it is compared to the original authors’ confusion matrix
(CM) in Figure 8(b) for the Catak dataset.

(a) VirusSample CM

(b) VirusShare CM

(c) Oliveria CM

Figure 7: Performance results of RTF on each dataset for imbalanced classes

Results show us that our proposed model is more accomplished in identifying
minority classes on VirusSample and VirusShare dataset contrary to Oliveira
dataset. Considering false negative predictions on these three dataset for
each class, the model is prone to predict some minority classes as majority
class which is Trojan in these cases.

In our study, our proposed architecture (RTF) is compared with base models
(LSTM and Transformer) and pre-trained transformer models (BERT and CANINE )
for each given dataset (Catak, Oliveira VirusSample, and VirusShare). Besides,
since VirusShare and VirusSample datasets are newly published and Oliveira
dataset is transformed to a multiclass problem by us, we have compared our
approach with other approaches in the literature for the Catak dataset.

In [13], they have proposed a model for multiclass classification on Catak
Dataset. In this study, all the samples in the Catak dataset are shuffled, and
80% of the dataset is allocated for the train set. Then, all the samples in
the Catak dataset are shuffled _again_ and 20% of the dataset is allocated for
the test set. The logical error made here is whole samples are shuffled twice.
This situation causes the test part to have some samples which exactly fall
into the train part. Thus, the model might test what it learned from the
train. We have performed a test on the code shared with us by the authors [13]
from the GitHub link 222https://github.com/MattScho/MalwareClassificationCNN.
We have performed a test to divide the whole dataset into the training and
testing dataset with the exact code script performed by authors for their
study. Finally, we realized that 1,117 of 1,371 samples allocated for the test
set intersect with the train set. For these reasons, evaluation scores
obtained by this article are not taken into account to compare our results.
The logical error has been reported to the article authors.

To the best of our knowledge highest F1-score obtained on the Catak dataset
for multiclass classification is 0.57 [27] compared to the baseline score of
0.47 obtained by the Catak dataset creators [26]. Although the F1-score
reported by [26] is 0.47, the calculated F1-score from the given CM in [26] is
0.41 as in Figure 8(a). The 20% of original Catak dataset is allocated as
unseen test data for RTF experiments like in [26]. In [26], the authors showed
their CM of LSTM model results on unseen test dataset. Their CM is referred to
as source CM as this is the first study performed on Catak dataset. Since only
this study contains CM, we have compared their CM with our proposed RTF model
CM on unseen test data.

Among the experimental studies conducted on the Catak dataset for multiclass
classification [27, 37, 26] our proposed RTF model has surpassed and reached
the state-of-the-art F1-score of 0.6149 as shown in Figure 8(b).

(a) SOURCE CM

(b) RTF CM

Figure 8: Comparison of confusion matrix on Catak dataset.

In our study, model performances are assessed according to F1-score and AUC
score for pre-trained models, BERT, and CANINE, and RTF model thus far.
However, besides F1-score and AUC score, training time is one of the factors
to evaluate model efficiency. Therefore, training runtime results for BERT,
CANINE, and RTF are given in Table 12.

Table 12: Training time (min.) comparison of BERT, CANINE, and RTF. Model | Oliveira | VirusShare | Catak | VirusSample
---|---|---|---|---
BERT | 141.02 | 26.25 | 28.63 | 18.73
CANINE | 58.27 | 11.12 | 12.03 | 7.77
RTF | 145.19 | 11.38 | 27.10 | 8.03

As seen in the Table 12, training runtime for pre-trained transformer models
and our proposed model has increased dramatically compared to LSTM and
Transformer model (tables 3, 2, 4 and 5). The reason of high training runtime
is BERT, and CANINE models are pre-trained on huge text data.

Considering the security aspects of our study, higher training time does not
hurt reaction time as these models will be trained first by AV scanners.
Therefore, the inference time spent by AV scanners to process unseen data and
make a prediction is the essential concern for security researchers to take an
action as soon as possible. Inference time for each model is given in Table
13.

Table 13: Inference time (s) comparison of each model. Model | Oliveira | VirusShare | Catak | VirusSample
---|---|---|---|---
LSTM | 0.3130 | 0.1110 | 0.3710 | 0.1070
Transformer | 0.0944 | 0.0931 | 0.0945 | 0.0924
BERT | 4.5137 | 2.7555 | 4.9658 | 1.4297
CANINE | 4.2778 | 2.6898 | 4.4324 | 1.4224
RTF | 4.5316 | 2.6856 | 4.9524 | 1.4183

Inference time given in Table 13, shows us the processing and prediction time
together for single unseen observation.

###  4.6 Limitations

As described in Table 1, even the four datasets used in our study consist of
different malware families, we have total of 11 unique malware families. We
think having a total of 11 different malware families is inadequate to
understand malicious behavioural characteristics of a malware sample
comprehensively. According to [62], malware taxonomy is divided into four
different categories as stealing, evasion, disruption, and modification as
these categories represent main malware behaviours. Therefore, among the
defined behavior types such as Trojan, Virus, and Ransomware, several types
need to be expanded and refined.

##  5 Conclusion and Future Work

In this study, we have leveraged several deep learning models for highly
imbalanced multiclass malware classification based on API calls, which are
inherently sequence problems. We have assessed the performance of our models
with AUC score and F1-score evaluation metrics as the four datasets used in
this study are imbalanced.

Our evaluation results demonstrate that the Transformer model with one
transformer block layer has achieved slightly better results than the LSTM
model. Moreover, the pre-trained transformer models, BERT or CANINE,
outperformed one transformer block layer Transformer architecture. On the
other hand, Transformer and LSTM models are noticeably faster than pre-trained
models, BERT, and CANINE, in both training and inference times. However,
considering the fact that training time does not directly affect the response
time, and there are differences in the inference time on the basis of seconds,
RTF model is more reasonable to continue with.

The CANINE model has been used for the first time in the field of malware
classification in this study. We have reached a state-of-the-art results on
the static API calls datasets, VirusShare and VirusSample, with a bagging-
based ensemble of the CANINE model. Therefore, we have demonstrated the
success of the CANINE model with the strength of RTF.

We have achieved a state-of-the-art F1-score of 0.6149 on the Catak dataset
with the power of bagging-based ensemble of BERT model and pre-processing
since pre-processing steps have enabled us to increase our results
significantly on the Catak dataset which has been built with dynamic API call
sequences. In addition, the F1-score of 0.6149 obtained on the well-known
benchmark Catak dataset has proved the success of our proposed RTF Model. In
general, our proposed RTF Model has obtained state-of-the-art results on three
out of four datasets.

This study can be extended by integrating our proposed ensemble model with the
AUC maximization paradigm [63]. We believe we may increase our results in this
way.

## References

  * [1] J. Jang-Jaccard, S. Nepal, A survey of emerging threats in cybersecurity, Journal of Computer and System Sciences 80 (5) (2014) 973–993.
  * [2] Ö. A. Aslan, R. Samet, A comprehensive review on malware detection approaches, IEEE Access 8 (2020) 6249–6271.
  * [3] Mimecast, The state of email security, Tech. rep., Mimecast (2021).
  * [4] Sophos, The state of ransomware 2021, Tech. rep., Sophos (2021).
  * [5] SonicWall, Cyber threat report, Tech. rep., SonicWall (2021).
  * [6] P. Shijo, A. Salim, Integrated static and dynamic analysis for malware detection, Procedia Computer Science 46 (2015) 804–811.
  * [7] D. Ucci, L. Aniello, R. Baldoni, Survey of machine learning techniques for malware analysis, Computers & Security 81 (2019) 123–147.
  * [8] D. Gibert, C. Mateu, J. Planes, The rise of machine learning for detection and classification of malware: Research developments, trends and challenges, Journal of Network and Computer Applications 153 (2020) 102526.
  * [9] C. Jindal, C. Salls, H. Aghakhani, K. Long, C. Kruegel, G. Vigna, Neurlux: Dynamic malware analysis without feature engineering, in: Proceedings of the 35th Annual Computer Security Applications Conference, 2019, pp. 444–455.
  * [10] R. Komatwar, M. Kokare, A survey on malware detection and classification, Journal of Applied Security Research 16 (3) (2021) 390–420.
  * [11] Q. Li, H. Peng, J. Li, C. Xia, R. Yang, L. Sun, P. S. Yu, L. He, A survey on text classification: From shallow to deep learning, arXiv preprint arXiv:2008.00364 (2020).
  * [12] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, J. Gao, Deep learning–based text classification: A comprehensive review, ACM Computing Surveys (CSUR) 54 (3) (2021) 1–40.
  * [13] M. Schofield, Comparison of malware classification methods using convolutional neural network based on api call stream, International Journal of Network Security & Its Applications (IJNSA) Vol 13 (2021).
  * [14] F. O. Catak, A. F. Yazı, A benchmark api call dataset for windows pe malware classification, arXiv preprint arXiv:1905.01999 (2019).
  * [15] O. Or-Meir, N. Nissim, Y. Elovici, L. Rokach, Dynamic malware analysis in the modern era—a state of the art survey, ACM Computing Surveys (CSUR) 52 (5) (2019) 1–48.
  * [16] J. B. Fraley, J. Cannady, The promise of machine learning in cybersecurity, in: SoutheastCon 2017, IEEE, 2017, pp. 1–6.
  * [17] W. Han, J. Xue, Y. Wang, L. Huang, Z. Kong, L. Mao, Maldae: Detecting and explaining malware based on correlation and fusion of static and dynamic characteristics, computers & security 83 (2019) 208–233.
  * [18] Y. Ding, X. Xia, S. Chen, Y. Li, A malware detection method based on family behavior graph, Computers & Security 73 (2018) 73–86.
  * [19] A. Fujino, J. Murakami, T. Mori, Discovering similar malware samples using api call topics, in: 2015 12th annual IEEE consumer communications and networking conference (CCNC), IEEE, 2015, pp. 140–147.
  * [20] Y. Ki, E. Kim, H. K. Kim, A novel approach to detect malware based on api call sequence analysis, International Journal of Distributed Sensor Networks 11 (6) (2015) 659101.
  * [21] G. G. Sundarkumar, V. Ravi, I. Nwogu, V. Govindaraju, Malware detection via api calls, topic models and machine learning, in: 2015 IEEE International Conference on Automation Science and Engineering (CASE), IEEE, 2015, pp. 1212–1217.
  * [22] B. Kolosnjaji, A. Zarras, G. Webster, C. Eckert, Deep learning for classification of malware system call sequences, in: Australasian joint conference on artificial intelligence, Springer, 2016, pp. 137–149.
  * [23] S. Tobiyama, Y. Yamaguchi, H. Shimada, T. Ikuse, T. Yagi, Malware detection with deep neural network using process behavior, in: 2016 IEEE 40th annual computer software and applications conference (COMPSAC), Vol. 2, IEEE, 2016, pp. 577–582.
  * [24] J. Mathew, M. A. Kumara, Api call based malware detection approach using recurrent neural network—lstm, in: International Conference on Intelligent Systems Design and Applications, Springer, 2018, pp. 87–99.
  * [25] X. Xiao, S. Zhang, F. Mercaldo, G. Hu, A. K. Sangaiah, Android malware detection based on system call sequences and lstm, Multimedia Tools and Applications 78 (4) (2019) 3979–3999.
  * [26] F. O. Catak, A. F. Yazı, O. Elezaj, J. Ahmed, Deep learning based sequential model for malware analysis using windows exe api calls, PeerJ Computer Science 6 (2020) e285.
  * [27] C. Li, J. Zheng, Api call-based malware classification using recurrent neural networks, Journal of Cyber Security and Mobility (2021) 617–640.
  * [28] A. Oliveira, R. Sassi, Behavioral malware detection using deep graph convolutional neural networks, TechRxiv (2019).
  * [29] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, X. Huang, Pre-trained models for natural language processing: A survey, Science China Technological Sciences (2020) 1–26.
  * [30] P. Ganesh, Y. Chen, X. Lou, M. A. Khan, Y. Yang, H. Sajjad, P. Nakov, D. Chen, M. Winslett, Compressing large-scale transformer-based models: A case study on bert, Transactions of the Association for Computational Linguistics 9 (2021) 1061–1080.
  * [31] T. Lin, Y. Wang, X. Liu, X. Qiu, A survey of transformers, arXiv preprint arXiv:2106.04554 (2021).
  * [32] N. E. Erciyes, A. K. Görür, Deep learning methods with pre-trained word embeddings and pre-trained transformers for extreme multi-label text classification, in: 2021 6th International Conference on Computer Science and Engineering (UBMK), IEEE, 2021, pp. 50–55.
  * [33] S. Paul, S. Saha, Cyberbert: Bert for cyberbullying identification, Multimedia Systems (2020) 1–8.
  * [34] J. L. Alvares, Malware classification with bert, Master’s thesis, San José State University (2021).
  * [35] F. Nassar, N. Hubballi, Malware detection and classification using transformer-based learning, Ph.D. thesis, Discipline of Computer Science and Engineering, IIT Indore (2021).
  * [36] Z. Xu, X. Fang, G. Yang, Malbert: A novel pre-training method for malware detection, Computers & Security 111 (2021) 102458.
  * [37] S. McDonnell, O. Nada, M. R. Abid, E. Amjadian, Cyberbert: A deep dynamic-state session-based recommender system for cyber threat recognition, in: 2021 IEEE Aerospace Conference (50100), IEEE, 2021, pp. 1–12.
  * [38] R. Oak, M. Du, D. Yan, H. Takawale, I. Amit, Malware detection on highly imbalanced data through sequence modeling, in: Proceedings of the 12th ACM Workshop on artificial intelligence and security, 2019, pp. 37–48.
  * [39] B. Düzgün, A. Çayır, F. Demirkıran, C. N. Kayha, B. Gençaydın, H. Dağ, New datasets for dynamic malware classification, arXiv preprint arXiv:2111.15205 (2021).
  * [40] malwarebytes Labs, 2020 state of malware report, Tech. rep., malwarebytes (2020).
  * [41] M. Kim, D. Kim, C. Hwang, S. Cho, S. Han, M. Park, Machine-learning-based android malware family classification using built-in and custom permissions, Applied Sciences 11 (21) (2021) 10244.
  * [42] A. N. Jahromi, S. Hashemi, A. Dehghantanha, K.-K. R. Choo, H. Karimipour, D. E. Newton, R. M. Parizi, An improved two-hidden-layer extreme learning machine for malware hunting, Computers & Security 89 (2020) 101655.
  * [43] Q. Zhu, On the performance of matthews correlation coefficient (mcc) for imbalanced dataset, Pattern Recognition Letters 136 (2020) 71–80.
  * [44] C. Halimu, A. Kasem, S. S. Newaz, Empirical comparison of area under roc curve (auc) and mathew correlation coefficient (mcc) for evaluating machine learning algorithms on imbalanced datasets for binary classification, in: Proceedings of the 3rd international conference on machine learning and soft computing, 2019, pp. 1–6.
  * [45] P. Branco, L. Torgo, R. P. Ribeiro, Relevance-based evaluation metrics for multi-class imbalanced domains, in: Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer, 2017, pp. 698–710.
  * [46] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (8) (1997) 1735–1780.
  * [47] D. S. Berman, A. L. Buczak, J. S. Chavis, C. L. Corbett, A survey of deep learning methods for cyber security, Information 10 (4) (2019) 122.
  * [48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [49] X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, L. Zhang, W. Han, M. Huang, et al., Pre-trained models: Past, present and future, AI Open (2021).
  * [50] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  * [51] J. H. Clark, D. Garrette, I. Turc, J. Wieting, Canine: Pre-training an efficient tokenization-free encoder for language representation, arXiv preprint arXiv:2103.06874 (2021).
  * [52] H. E. Boukkouri, O. Ferret, T. Lavergne, H. Noji, P. Zweigenbaum, J. Tsujii, Characterbert: Reconciling elmo and bert for word-level open-vocabulary representations from characters, arXiv preprint arXiv:2010.10392 (2020).
  * [53] M. Marcinczuk, Punctuation restoration with ensemble of neural network classifier and pre-trained transformers, Proceedings ofthePolEval2021Workshop (2021) 47.
  * [54] G. Morio, T. Morishita, H. Ozaki, T. Miyoshi, Hitachi at semeval-2020 task 11: An empirical study of pre-trained transformer family for propaganda detection, in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, 2020, pp. 1739–1748.
  * [55] S. Malla, P. Alphonse, Covid-19 outbreak: An ensemble pre-trained deep learning model for detecting informative tweets, Applied Soft Computing 107 (2021) 107495\.
  * [56] L. Breiman, Random forests, Machine learning 45 (1) (2001) 5–32.
  * [57] A. Çayır, U. Ünal, H. Dağ, Random capsnet forest model for imbalanced malware type classification task, Computers & Security 102 (2021) 102133\.
  * [58] S. Kobayashi, J. von Oswald, B. Grewe, On the reversed bias-variance tradeoff in deep ensembles, in: ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning, 2021.
  * [59] F. Chollet, et al., Keras: The python deep learning library, Astrophysics Source Code Library (2018) ascl–1806.
  * [60] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch (2017).
  * [61] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Huggingface’s transformers: State-of-the-art natural language processing, arXiv preprint arXiv:1910.03771 (2019).
  * [62] A. R. A. Grégio, V. M. Afonso, D. S. F. Filho, P. L. d. Geus, M. Jino, Toward a taxonomy of malware behaviors, The Computer Journal 58 (10) (2015) 2758–2777.
  * [63] Z. Yuan, Y. Yan, M. Sonka, T. Yang, Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3040–3049.