malware binary features are mapped to continuous space before feeding to
generator network of GAPGAN which generates adversarial perturbations to be
appended at the end of original malware binaries. Dynamic thresholding
preserves generated subtle perturbations while mapping back to discrete space
from continuous space. The balance on the attention of the generator across
payloads and adversarial samples is maintained using an automatic weight
tuning strategy. Authors are able to achieve 100% evasion rate against MalConv
[263] with the addition of only 2.5% perturbations data. Concurrently trained
generator and discriminator keeps improving each other and transferability
property of adversarial attack enables it to bypass detector. Variable input
and output size gives great flexibility to GAPGAN model in contrast to prior
research works.

####  7.1.5 Recurrent Neural Network Based Attack

Recent works have focused on use of Recurrent Neural Network (RNN) for malware
detection and classification [21, 285, 286, 287]. Sequential malware API is
used by RNN to predict whether the program is malware or benign. Papernot et
al. [288] introduced adversarial sequence for RNN processing sequential data.
The authors demonstrated the transferability property of adversarial examples
generated from feed-forward neural networks against recurrent neural networks.
Table VIII summarizes comparison among RNN, explainable ML and malware
visualization based adversarial attacks. Hu et al. [240] proposed RNN based
adversarial attack for RNN malware detector. The approximation of victim RNN
model is done by training substitute RNN and generative RNN outputs sequential
adversarial examples. Some irrelevant API sequence is generated and inserted
in vulnerabilities of original sequence. API sequences, represented as a one-
hot vector, are the input for generator network which generates adversarial
API sequence. The generative part of RNN generates small API sequence pieces
after each API which gets inserted after the API. Small sequence generation is
done by sequence decoder where the hidden states are initialized with zero. A
benign sequence and the Gumbel-Softmax [289] output is used to train the
substitute network to fit the victim RNN based detector. Due to the use of
bidirectional RNN equipped with an attention mechanism, substitute RNN can
grasp the complex sequential pattern. Both forward and backward connections
are present inside the bidirectional connection to represent the individual
unidirectional connection. The attention mechanism helps by spreading the
focus on different parts of the sequence. The output label provided by victim
RNN for training data is used as training labels for substitute RNN. Gumbel-
Softmax [289] exchanges gradient information across the generative RNN and the
substitute RNN while also smoothing API symbols. Generative RNN function tries
to minimize the probability of malicious prediction $p_{s}$ on $S_{Gumbel}$
which is given in Equation 17.

$L_{G}=log(p_{s})-\gamma\mathbb{E}_{t=1\sim T,\tau=1\sim L}\pi^{M}_{t\tau}$
(17)

where $\gamma$ is the regularization coefficient used for restricting the
number of inserted APIs by maximing null API’s expectation probability, M is
the index of the null API, $\mathbb{E}$ denotes the expected value, $p_{s}$ is
predicted malicious probability, $\pi$ is any parameter and $\tau$ is learned
parameter. By separately tuning hyper-parameters of generative and substitute
RNN, proposed architecture was successful in evading LSTM and BiLSTM based RNN
malware detectors.

####  7.1.6 Explainable Machine Learning based attack

TABLE VIII: RNN, explainable ML and visualization based adversarial.

Paper/Year |  Key-Motivation |  Target Model |  Algorithms Used |  Approach
---|---|---|---|---
Hu et al. 2017 [240] |  Attack against RNN processing sequential data |  LSTM and BiLSTM based detectors |  Bidirectional RNN with attention mechanism |  • Substitute RNN approximates victim RNN • Generative RNN gives sequential adversarial example • Irrelevant API sequence generated and inserted in vulnerabilities of original sequence
Rosenberg et al. 2020 [290] |  Use of explainable machine learning for adversarial generation |  GBDT Classifer |  Integrated Gradient, LRP, DeepLIFT, SHAP |  • Unearthing most impactful features using explainability algorithm • Manual selection of easily modifiable features • Feature by feature modification without harming functionality and interdependent features
Liu et al. 2019 [291] |  Adversarial malware against visualization based detection |  CNN, SVM and RF based malware detectors |  ATMPA framework using GoogLeNet, FGSM and C&W |  • Data transformation to convert code segments into grayscale images • Pre-training module to find function of malware detectors • Optimized FGSM and C&W attack is used to generate actual AE
Khormali et al. 2019 [292] |  Targetted and Untargetted misclassification on windows and IoT malware dataset |  Convolutional Neural Network |  FGSM, C&W, DeepFool, MIM and PGD |  • Adversarial generation using different algorithms • Conversion of adversarial dimension, same as of original image • Appending pixels at the end or injecting
Benkraouda et al. 2021 [293] |  Attack against visualization based detection with ability to evade pre-processing filtering without losing functionality |  Convolutional Neural Network |  Modified version of CW attack[256], Euclidean distance |  • Mask generator to flag the locations for perturbation • Modified version of CW attack to generate optimal perturbation • NOP generator to replace the perturbation from CW attack by semantic NOPs • AE optimizer to choose optimal viable NOPs

Key-Motivation: The major motive behind the published work, Target Model:
Target defense for adversarial attack, Algorithms Used: Algorithm used for
crafting adversarial example, Approach: Key procedures to carry out
adversarial attack

Malware detection is one of the most relevant domain for adversarial crafting
as attackers are continuously attempting to evade detection networks. However,
one of the biggest challenge of machine learning is the lack of explainability
or reasoning behind such intelligent decisions. Recent researches have been
able to bypass malware detectors using concept of explainable machine
learning. Explainability approach involves finding the significance of each
features and then conducting feature specific modification based on their
importance.

Rosenberg et al. [290] proposed explainable ML approach to generate
adversaries against multi-feature type malware classifiers. Authors not only
performed feature addition like existing approaches but also modified feature-
by-feature. Adversarial attackers first evaluate for the most effective list
of features, and the features that are easy to modify are selected.
Transferability of explainability allows the proposed attack achieve a very
high impact on target classifier even in black-box attack. This approach
assumes that the malware classifier and the substitute model possess similar
feature importance, leading to modification in feature to impact the target
malware classifier. Four different explainability algorithms on white-box
[294, 295, 296] and black-box [297] are evaluated to make comparisons between
substitute model and victim model. Brief introduction of each explainability
algorithms is:

  * •

Integrated gradient [294] satisfies the completeness property by computing the
average of the gradient on varying input while moving in a linear path.

  * •

Layer-wise Relevance Propagation (LRP) [295] works on backward pass by
starting from the output layer. The relevance of each target neuron is given
corresponding to the output of the neuron.

  * •

DeepLIFT [296] works in similar fashion to that of LRP but in backward order.

  * •

SHAP (SHapley Additive exPlanation) [297] attributes the classifier output to
the totality of individual feature effects. SHAP can work without any
knowledge about the architecture of the network to explain.

The proposed end-to-end PE adversarial examples performs feature modification
without harming the malware functionality as well as interdependent features.
Using naive and engineered features of EMBER dataset, explainable ML approach
is successful in bypassing GBDT classifier. Rosenberg et al.’s work presents
explainability as a duel edged sword that can be used by adversaries to make
more explainable models as well as to carry out more robust adversarial
attacks.

####  7.1.7 Malware Visualization based attack

Figure 27: An adversarial generation against malware visualization detection.

Machine learning-based visualization detection has been popular due to its
ability to prevent zero-day attacks and make detection without extracting pre-
selected features [298, 299, 300]. These approaches convert binary code into
image data and visualizes the features of sample, improving the detection
speed for malicious program. Visualization based techniques are similar to
adversarial generation in image domain where pixel perturbations are
introduced as shown in Figure 27. Liu et al. [291] introduced Adversarial
Texture Malware Perturbation Attack (ATMPA) against visualization based
malware detection using rectifier in neural-network hidden layers. The
framework allows an attacker to probe with the malware image while visualizing
and also hiding them from malware detectors. The ATMPA framework is made up of
three functional modules: data transformation, adversarial pre-training and
malware detection. Code segments are converted into grayscale images during
data transformation module. In the adversarial pre-training module, an
attacker uses a machine learning approach to train an adversarial example
generation model which produces a noise signal $\delta$. The training process
is divided into two modules: one to capture the normal malware behaviors and
other to generate AEs. ATMPA framework uses GoogLeNet [75] for generating
adversarial examples in the pre-training process where malware samples are
represented using binary indicator vector without any structural properties or
inter-dependencies. Adversarial example training uses a feedforward neural
network having a rectifier as an activation function.

For generation of AEs, optimized FGSM and C&W attack is used. Using FGSM
approach, perturbation with distortion $\epsilon$ = 0.35 was able to convert
malware with 88.5% confidence into benign files with 99.8% confidence. ATMPA
method also used $L_{p}$-based C&W attack to generate adversarial, including
$l_{0},l_{2}$ and $l_{\infty}$ attack. Kaggle Microsoft Malware Classification
Challenge232323https://www.kaggle.com/c/malware-classification/overview
dataset is used against malware detection model built on commonly used CNN,
SVM and RF-based detectors. Inception V3 architecture followed by FGSM and C&W
attack methods generated pseudo-benign samples, successfully deceiving the
detector. Authors also carried out transferability attack comparision on CNN,
SVM and wavelet-combined algorithms with transfer rate as high as 88.7%.

COPYCAT approach proposed by Khormali et al. [292] produced both targeted and
untargetted misclassification on Windows and IoT malware dataset. Author used
two approaches: AE padding and sample injection to produce adversarial malware
for visualization based detector. For padding method, COPYCAT generated
adversarial $x^{{}^{\prime}}$ using five different attack methods namely: FGSM
[74], C&W [256], DeepFool [257], Momentum Iterative Method (MIM) [301] and
Projection Gradient Descent (PGD) [302]. The generated adversarial needs to be
converted to the dimensions same as that of original image before appending at
the end of image. The binary samples from the targeted class are injected into
an unreachable section of the target sample. The approach of this work is
inspired by Cha et al. [303], where binaries of an application having
different architecture are padded to alter the behavior based on the
underlying system.

In order to provide adversarial attack that can evade visualization based
detection system in presence of pre-processing filtering, Benkraouda et
al.[293] proposed a binary rewriting based attack on malware files. A mask
generator creates the space in the instruction boundary to insert the
perturbations. Once the perturbation mask is created, the modified version of
CW attack [256] is used to generated adversarial example in image space. The
modified version is in the sense that, the perturbation mask is imposed while
carrying out attack to restrict the positions of perturbations. NOP generator
will replace the perturbation introduced by CW attack with the corresponding
binaries, that preserve the malware functionality. And finally AE optimizer
will use Euclidean distance metric to choose semantic NOPs that are closed to
sequences in the allowed perturbation space. The approach is highly
successfully in generating adversarial examples but are constrained by
perturbation generation time as time is found to increase drastically with
increase in size of malware.

###  7.2 Android Malware Adversarial

TABLE IX: An Android adversarial attacks.

Paper/Year |  Target/Dataset |  Approach |  Modification |  Limitations
---|---|---|---|---
Grosse et al. 2017 [304] |  Feed forward neural network based detector / DREBIN |  • Binary feature vector extraction using static evaluation • Jacobian matrix of neural network for adversarial generation • Direction for generated perturbation is given by gradient of the given function with respect to the input • Selection of perturbation with maximal positive gradient towards target class |  • Feature addition to AndroidManifest.xml • Changing features leading to only one line of code |  • Constrained on maximum feature space perturbation • Feature modifications confined inside AndroidManifest.xml
Yang et al. 2017 [305] |  KNN, DT, SVM, RF/ DREBIN, VirusShare, Genome |  • Malware Recomposition Variation conducting semantic analysis • Feature mutation analysis and phylogenetic analysis to perform automatic program transplantation • Malware evolution attack focusing on mimicking and automating the evolution of malware • Conufsion attack making features less differentiable to malware detection |  • Resource, temporal, locale and dependency features used • Mutation following feature pattern of existing malware |  • Significant alteration of semantic leading to higher failure rate of app
Rosenberg et al. 2018 [205] |  RNN variaant and Feed forward neural networks/ VirusTotal |  • Mimicry attacks against surrogate model • Surrogate model by querying black-box detectors with synthetic inputs selected by Jacobian based heuristics in prioritizing directions • Closest API call in direction indicated by Jacobian are selected |  • No-op attack by adding API call with valid parameters • Functionality verification using sandbox after modification |  • Detectable Residual artifacts during app transformation
Liu et al. 2019 [306] |  Neural network, logistic regression, DT and RF based detectors/DREBIN |  • Random forest to filter most significant features • Disturbance randomly generated and disturbance size calculated using genetic algorithm • Mutation using fitness function till fit and evading individual is produced |  • Restricted permission modification on AndroidManifest file • Functionality changing modifications are deemed unfit |  • Increased constraint on perturbation • Random perturbation affecting convergence
Shahpasand et al. 2019 [206] |  SVM, Neural network, RF and LR / DREBIN |  • GAN architecture with threshold on generated distortion • Different loss function to generate benign like adversarial and to produce high mis-classification |  • Perturbation addition limited by threshold distortion amount |  • Highly unstable learning of GAN architecture
Li et al. 2020 [307] |  AdaBoost, CNN, SVM / Tencent Myapp, AndroZoo, VirusShare and Contagio |  • Bi-objective GAN with two discriminator and one generator • First discriminator to distinguish malware and benign sample • Second discriminator to distinguish original and adversarial sample |  • Iterative perturbation addition till evasion • Perturbation evading both malware and adversarial detection |  • Very limited feature vectors ( Permission, action and API calls) are considered
Pierazzi et al. 2020 [207] |  Linear SVM, Sec-SVM/DREBIN |  • Formalization of adversarial evasion attacks in the problem feature space including transformations, semantics, robustness and plausibility • Automated software transplantation to extract benign slices from donor • Side effect features to find projections that maps perturbation to feasible problem-space regions • Gradient based strategy based on greedy algorithm to choose perturbation |  • Perturbations appended at the end • Restricted addition of permissions • Cyclomatic Complexity to take heuristic approach maintaining existing homogeneity |  • Heuristic based approaches are time and resource consuming
Bostani et al. 2021 [308] |  DREBIN [309], Sec-SVM [310], MaMaDroid [311] / AndroZoo [312] |  • Automated Software Transplantation Technique to prepare action set which includes gadgets extracted from benign Android apps • n-gram-based similarity method to identify benign APKs, closely similar to malware files • Applying extracted gadgets from benign samples into malicious files • Iterative and incremental manipulation |  • Random Search(RS) for moving malware sample in problem space applying sequence of transformation in action set • New contents injected inside an IF statement |  • In Random Search (RS) algorithm, actions from action space are random • Increase in adversarial size, increasing chances of adversarial detection

Target/Dataset: Target defense for adversarial attack/Dataset used, Approach:
Key procedures to carry out adversarial attack, Modification: Changes on file
to craft the adversarial perturbation, Limitations: Shortcomings of proposed
approach

Android has over 2.8 billion active users and owns 75% market share in mobile
phone industry [313]. The wide usage of Android platform has attracted
security threats in numerous forms and adversarial evasion attack is one of
them. Table IX provides brief comparison among different adversarial attacks
crafted against Android files. Grosse et al. [314] [304] generated adversarial
examples for state-of-art Android malware detection trained on DREBIN dataset
[309] with more than 63% accuracy. Authors migrated the method proposed by
Papernot et al. [251] to handle binary features of Android malware while
preserving the malicious functionality. Binary features are derived by
statically evaluating code based on system call and usage of specific
hardware. These derived features are also known as binary indicator vectors.
The final goal of adversarial attack is to find perturbation/noise $\delta$
such that the prediction results of $y^{{}^{\prime}}$ of $F(X+\delta)$ is
different from the original result, i.e, $y^{{}^{\prime}}\neq y$.
$y^{{}^{\prime}}$ gives target class while crafting adversarial. Authors
adopted Jacobian matrix of neural network $F$ for adversarial generation. To
get adversarial, the gradient of function $F$ with respect to $X$ is
calculated to get the direction of perturbation such that output of
classification function will change. Perturbation $\delta$ with highest
positive gradient in direction of target is selected and is kept small enough
to prevent negative change due to intermediary alterations of gradient.
Functionality is preserved in this approach by changing features resulting in
addition of only single line of code. Research also confine the modifications
to manifest features related to AndroidManifest.xml file contained within
Android application. With permissions, intents and activities being the most
frequently modified features, authors successfully evaded DREBIN
classifier[309] preserving the semantics of malware.

To overcome the white box attack issues, Rosenberg et al. [205] implemented
GADGET framework to convert malware binary to an adversarial binary without
access to malware source code. Proposed end-to-end black-box method is
extended to bypass the multi-feature based malware classifiers relying on the
transferability in RNN variants. For target RNN detector, malicious API call
sequence is the adversarial example to be generated. Authors perform mimicry
attacks where malicious code mimics the system calls of benign code to evade
[315]. Adversaries train surrogate model having same decision boundaries as
that of detector and then execute white-box attack on surrogate model. To
build the surrogate model, black-box detector is queried with synthetic input
values chosen by a Jacobian based heuristics in the prioritizing directions
where model output varies. API calls which are nearest to the direction given
by Jacobian are inserted to generate adversarial sequence. The label assigned
to the input value by a black-box model gives the sign of the Jacobian matrix
dimension. Jacobian matrix of the surrogate model is used for evaluation and
after each iteration, synthetic example is added to each existing sample.
However, finite set of legitimate API embeddings may not be enough for
adversarial insertion, causing insertion of most impactful API call in
direction indicated by Jacobian. Adversarial examples that are able to fool
surrogate model has high likelihood of fooling original model as well [193].
Adversarial generation showed same success against the substitute and blackbox
model with short API sequences, making adversarial generation faster.
Framework also uses Cuckoo Sandbox to verify the malicious functionality of
generated adversarial malware. GADGET framework wraps malware binary with
proxy code and increases the risk even higher providing malware-as-a-service.

Adversarial attack on malware domain has not considered to manipulate the
feature vector to see impact of mutation due to strict functionality
preserving requirements of malware. Malware Recomposition Variation (MRV)
based approach proposed by Yang et al. [305] performed an analysis of malware
file semantically and construct a new variant of malware. Mutation strategies
synthesized by conducting semantic-feature mutation analysis and phylogenetic
analysis are used to perform automatic program transplantation [316]. Changing
the traditional belief over mutation, authors followed feature patterns of
existing malware to preserve the functionality. The proposed framework
performs inter-component, inter-app, and inter-method transplantation. More
comprehensive attack is performed on both the manifest file as well as dex
code. Use of RTLD features allows substitute model to approximate targeted
detector and also helps in separation of essential features and contextual
features.

  * •

Resource features: These features provide the security-sensitive resources
that are impacted by the malware. Resource features are extracted by forming
call graphs and pinpointing call-graph nodes of those security-sensitive
methods.

  * •

Temporal features: These features provide the environmental context at the
trigger point of the malicious property and the context is inferred from the
attributes at entry points.

  * •

Locale features: These features describe the location of programs where
malicious activities are observed and can be either of the Android components
of concurrency constructs.

  * •

Dependency features: These features are provided by constructing an inter-
procedure control-flow graph, indicating the control dependencies when
malicious activities are invoked.

The goal of malware evolution attack is to imitate and automate the evolution
of malware using phylogenetic evolutionary tree [317]. A phylogenetic
evolutionary tree shows the inferred relations between different samples based
on the similarities and differences of feature representation. A pairwise
distance between samples is fed to phylogenetic tree generation algorithm,
Unweighted Pair Group Method Average (UPGMA), to generate phylogenetic tree
which reflects the structure present in a similarity matrix. Another approach
proposed by authors, confusion attack, tries to complement malware evolution
attack against robust malware detectors [318]. The feature values modified by
confusion attacks can be shared by both the malware sample and benign apps.
This approach is more complete due to flexibility of mutation by different
number of means but introduces challenges on keeping functionality intact due
to higher volume of modification [207].

Several adversarial generation approaches have been conducted by making minor
perturbations on existing attacks. Liu et al. [306] proposed a Testing
framework for Learning-based Android Malware Detection systems (TLAMD).
Framework uses genetic algorithm to perform black-box attack against Android
malware detection system. Android files are modified by adding the request
permission code to AndroidManifest.xml file which was originally proposed by
Grosse et al. [304]. The restriction was imposed on the types and magnitude of
permissions that can be added to AndroidManifest file. A random population is
generated giving the characteristics of permission to add and followed by
calculating the disturbance size for the sample malware. Using the evaluated
perturbation size, adversarial is generated and tested against the detection
model. Based on the result of detection either new disturbance size is
calculated using genetic algorithms or perturbation is successfully added on
Android application. Another important aspect of genetic algorithm is to model
the fitness function which is this framework has been defined in Equation 18.

$S(\delta)=\mathrm{min}~{}w_{1}.F(X+\delta)+w_{2}.num(\delta)$ (18)

where $w_{1}$ and $w_{2}$ are the two weights, $\delta$ is the added
disturbance, $num(\delta)$ is number of permission features added and
$F(X+\delta)$ gives the probability of malicious sample being detected as
malware. Fitness function searches for optimal solution to perform mutation
leading to a new fit individual able to evade detection. Random forest
approach is used to filter out insignificant features during feature
extraction. Disturbance generated by genetic algorithm are able to bypass
malware detectors trained on neural networks, logistic regression, decision
trees and random forest.

Shahpasand et al. [206] implemented GAN to generate adversarial by keeping
threshold on the distortion values of generated samples. The generated optimum
perturbation $\delta$ is added to existing malware to produce adversarial.
Like every other GAN architecture, generator can learn the distribution of
benign samples, generating perturbations which are able to bypass learning
based detector. The discriminator implicitly enhances the perturbation by
escalating the loss of generator while the adversarial samples are
identifiable with benign files. Loss function for GAN is similar to that of
Goodfellow et al.’s [182] work given in Equation 19.

$\displaystyle L_{GAN}=\;_{x\sim P_{Benign}}logD(x)\;+\;_{x\sim
P_{Malware},z\sim P_{z}(Z)}log(1-$ (19) $\displaystyle
D(x+G(z)));\;s.t.|G(z)|<c$

where $D(x)$ gives probability of sample $x$ coming from benign software
distribution, $L_{GAN}$ helps maximizing resemblance of adversarial malware
with benign sample while limiting distortion less than c. Loss function of
adversarial malware $(L_{adv.Mal})$ is given as:

$L_{adv.Mal}=_{x,z}l_{f}(x+G(z),0)$ (20)

where benign class is targeted for an adversarial sample $x+G(z)$ against
classifier $f$ trained on $l_{f}$ loss function. Finally, model loss is
defined as:

$L=\alpha L_{GAN}+(1-\alpha)L_{adv.Mal}$ (21)

In above equation, first loss forces the GAN to generate adversarial similar
to benign samples while the second loss inclines adversarial samples to have
higher miss-classification rate.

The goal of adversarial generation has been to bypass malware detector without
losing functionality. However, due to growth in adversarial malware in recent
time, defenders are employing firewalls to stop adversarial sample. Li et al.
[307] extended the work of MalGAN [282] to make it robust against detection
system equipped with firewall. Despite its high evasion rate against malware
detectors, MalGAN is found to be less effective against detection systems
using firewall. Bi-objective GAN with two discriminator having different
objectives are used. One of the discriminator helps distinguish between
malware and benign whereas another discriminator helps to find out whether the
samples are adversarial or normal ones. Due to this feature, adversarial
generated by generator can successfully bypass through the firewall as well as
the malware detection. Authors used permissions, actions and application
programming interface calls as a features to generate adversarial. In every
round of training, gradient descent is used to update the parameters of
discriminators and generator, represented by $\theta_{d_{1}}$,
$\theta_{d_{2}}$ and $\theta_{g}$ respectively. $G$, $D_{1}$ and $D_{2}$ are
considered functions implemented by generator, discriminator 1 and
discriminator 2. First discriminator, used to separate benign and malicious
class is trained to update $\theta_{d_{1}}$, by minimizing its loss

$L_{d1}=\mathbb{E}_{x\in N_{B}}log(1-D_{1}(x))+\mathbb{E}_{x\in
N_{M}}log(D_{1}(x))$ (22)

where $N_{B}$ and $N_{M}$ denote the distributions of benign and malicious
samples. Second discriminator used to separate between normal and adversarial
samples is trained by updating $\theta_{d_{2}}$ by minimizing its loss

$L_{d2}=\mathbb{E}_{x\in N}log(1-D_{2}(x))+\mathbb{E}_{x\in M}log(D_{2}(x))$
(23)

where $N$ is the distribution of normal samples and M is the distribution of
generated samples detected as malicious. Generator is updated twice each
round. After updating $\theta_{d_{1}}$, $\theta_{g}$ is updated by minimizing

$L_{g}^{1}=\mathbb{E}_{m\in M,z\in P}log(1-D_{1}(G(m,z)))$ (24)

where $M$ is distribution of malicious samples fed to generator and $P$ is the
distribution of noises fed to generator. After $\theta_{d_{2}}$ is updated,
$\theta_{g}$ is updated by minimizing

$L_{g}^{2}=\mathbb{E}_{m\in M,z\in P}log(1-D_{2}(G(m,z)))$ (25)

TABLE X: Adversarial attacks on PDF malware.

Paper/Year |  Target |  Approach |  Modification |  Limitations
---|---|---|---|---
Maiorca et al. 2013 [319] |  PJScan, Malware Slayer and PDFRate |  • Reverse mimicry attack by manipulating binary files to make it malicious • Malicious embedded EXE payload insertion • Malicious PDF file insertion inside a benign one • Encapsulating malicious JavaScript code |  • Malicious EXE payload as a new version after trailer • Unrestrained embedded PDF structure insertion • JavaScript code without reference to any other object to minimize structure variation |  • Less control on malicious goal
Biggio et al. 2014 [320, 321] |  SVM and neural network based detectors |  • Gradient based optimization inspired by Golland’s discriminative directions technique • Additional panalizing term to reshape objective function, biasing gradient descent towards region of negative class concentration |  • Insertion of objects creating new PDF files |  • Feature mapping issues • Non-differential discriminating functions can not be evaded
Srndic et al. 2014 [322] |  PDFrate employed on Random Forest |  • Taking advantage of discrepancy between operation of PDF reader and PDFrate • Mimicry attack to mimic 30 different benign files • GD-KDE attack to defeat classifier with differentiable decision function |  • Insertion of dummy contents, ignored by PDF readers but affect detector • Trailer section moved away from cross reference table for file injection space |  • Feature mappings are assumed to be perfect which is unrealistic
Carmony et al. 2016 [323] |  PDFrate and PJScan |  • Reference JavaScript extractor by directly tapping into a Adobe reader at locations identified by dynamic binary analysis • Parser confusion attack combined with reverse mimicry attack |  • Obfuscation based on output of reference extractor |  • Useful only for JavaScript based detector • Dependent on versions of Adobe Reader
Xu et al. 2016 [247] |  PDFrate and Hidost |  • Stochastic manipulations using genetic algorithm to generate population • Iterative population generation till evasion • Successful mutation traces reused for initialization efficiency • Fitness score based on maliciousness detected by oracle |  • Inserting new, removing and modifying existing contents • Oracle confirming the maliciousness of file |  • Stochastic approaches are resource intensive • No exact way to choose best fitness function

Target: Target defense for adversarial attack, Approach: Key procedures to
carry out adversarial attack, Modification: Changes on file to craft the
adversarial perturbation, Limitations: Shortcomings of proposed approach

Pierazzi et al. [207] formalized the adversarial ML evasion attacks in the
problem space and proposed a problem space attack on Android malware. The
proposed approach formalizes the set of restriction in transformations
available, semantics preserved, robustness of preprocessing approaches, and
veracity. Research work is focused on evasion attacks at test time by
modifying the objects in real input space corresponding to feature vector.
With a goal of overcoming inverse feature-mapping problem from previous
researches, author presents the idea of side-effect features. Side effect
feature defines and proves the necessary as well as sufficient conditions
behind the problem space attacks. An attack on a feature space is projected
towards a feasibility region satisfying the problem space constraints to
obtain the side effect features. To formally demonstrate the side-effect
features, an object $x\in X$ is initialized within a feasible region. A
gradient-based attack takes an object $x$ in feature space to $x+\delta^{*}$,
with $\delta^{*}$ being the perturbation. The addition of perturbation
misclassifies malware files as benign with high confidence. However, the new
point with perturbed feature on feature vector may not be inside the
feasibility region. Side effect features in perturbation help to map
$x+\delta^{*}$ to the region of feasible problem-space.

Since side effect features contribute towards preserving validity of malware
due to impact of original gradient based perturbation, side effect features
alone can have both positive or negative influence on the classification
score. Authors use automated software transplantation [316] to extract byte-
codes from benign donor applications to inject into a malicious host, also
known as organ harvesting. Insertion into a host is carried out between
statements in non-system package to preserve the functionality and Cyclomatic
Complexity is used to take a heuristic approach maintaining existing
homogeneity and preventing violation of plausibility. Prior research works
relied heavily on addition of permissions to the Android Manifest which is
considered dangerous in Android documentation [324]. Authors bind the
modifications to inject single permission to the host app. Gradient based
strategy using greedy algorithm proposed in this approach overcomes previous
limitations of preserving semantics and pre-processing robustness.

To overcome the challenges of limited access to target classifiers while
circumventing black-box Android malware detectors, Bostani et al. [308]
proposed a novel iterative and incremental manipulation strategy. The attack
is carried out in two-step: preparation and manipulation. In preparation
phase, automated software transplantation is employed to prepare action sets
from Android apps. The n-gram-based similarity method is used to identify
benign apps that closely matches to malware files. Insertion of extracted
gadgets of closely matching benign files force malware sample towards the
blind spots of the classifier. In the manipulation stage, perturbation on
malware samples are applied incrementally, choosing from the collected action
set. The search method randomly chooses suitable transformation and applies
them to malware samples. This approach shows a high success rate in query
efficient approach but increases the size of adversarial perturbation which in
turn increases the risk of perturbation being easily detected.

###  7.3 PDF Malware Adversarial

Along with widespread applications and adoption, PDF documents have been one
of the most exploited avenues for adversarial malware attacks. Initially,
JavaScript based and structural properties detection were prominent for
recognising malware in PDF. But freedom to distribute chunks of Javascript
code and assemble together at run-time and high degree expressiveness in
JavaScript language led to failure of Javascript based detection. Despite
significant growth in PDF malware detection from JavaScript using deep
learning techniques, the challenges posed by adversarial examples still exist.
Early evasion attempts on PDF documents were crafted by Smutz et al. [325] and
Šrndić et al. [326] using heuristic approaches. The authors proposed approach
to build more robust PDF malware detection techniques, showcased the
adversarial ability to mislead linear classification algorithm successfully.

Flexible logical structure of PDF has allowed to craft adversarial by
carefully analyzing its structure. Maiorca et al. [319] demonstrated evasion
technique called reverse mimicry attack against popular state-of-art malware
detectors [327, 328, 325]. Traditionally, malicious PDF files are believed to
be structurally different from benign PDF files. Taking advantage of this
structural difference, most of malware detectors were able to discriminate PDF
files with very high accuracy. However, malware files which can imitate the
benign file structure or vice-versa can easily fool the detector. Reverse
mimicry attacks can make benign files malicious with minimal changes in their
structure. Malicious payloads poison the samples, initially classified as
benign. Three kinds of malicious payloads introduced to benign files take the
sample across the decision boundary of malware detector. First one is EXE
payload with malicious embedding, which is introduced using Social Engineering
Toolkit242424https://www.secmaniac.com/ as a new version after its trailer. On
the addition of a new root object, the new trailer will point to a new object.
In this payload, authors embedded malicious PDF files inside another benign
PDF files using embedded function of PeePDF[329] tool. The embedded PDF file
automatically opens without user interaction, allowing malicious PDF to be
embed inside a benign one without any restriction on embedding file. PDF file
injection enabled an attacker to have fine-grained control of structural
features in the carrier file. A final kind of payload insertion is carried out
by encapsulating a malicious JavaScript code without reference to other
object. It helps to minimize variation in the benign-file structure by adding
only one object to the tree. Such attacks based on structure are even capable
of evading detectors using non-structural features. Table X provides overview
of adversarial attacks carried out on PDF files.

Optimization based evasion attack against PDF malware detection was introduced
by Biggio et al. [320, 321]. The attack was carried out using a gradient based
optimization procedure inspired by Golland’s discriminative directions
technique [330] to evade linear as well as non-linear classifiers. The
proposed work was able to carry out complete knowledge and constrained
knowledge attacks on non-linear models like Support Vector Machine(SVM) and
neural networks. Work relied on easiness to insert new objects than to remove
an embedded object to prevent from corrupting the PDF’s file structure. This
approach used a gradient descent procedure with special consideration to avoid
getting stuck on local optima. To increase the probability of successful
evasion, an attacker needs to reach attack points that are legitimate, and to
reach this, the additional penalizer term is introduced using a density
estimator. The extra component helps imitate features of known legitimate
samples, reshaping the objective function by biasing the gradient descent
towards the negative class concentration region. This optimization-based
approach dates before the realization of adversarial examples against deep
learning architectures [331].

Figure 28: A PDF file modification approach.

Srndic et al. [322] further enhanced optimization based attack against
deployed system PDFrate [325] using mimicry attack, and Gradient Descent and
Kennel Density Estimation (GD-KDE) attack. The attack takes advantage of
discrepancy between functioning of PDF readers and PDFrate in terms of
interpretation of semantic gaps as explained in [332]. The dummy contents to
insert should be ignored by PDF readers but affect the feature computation in
PDFrate. PDFrate evaluates sets of regular expressions from raw bytes, reading
from beginning of PDF files while PDF readers parse PDF files using PDF format
authorized by ISO 32000-1. PDF reader looks at the end of PDF for cross-
reference table and goes to locate the object directly. Among 135 features of
PDF rate, MIMICUS252525https://github.com/srndic/mimicus, 35 features are
modified while incrementing values for 33 features with preserved
functionality. Trailer section of PDF files were moved arbitrarily far away
from cross-reference table, generating an empty space for file injection
without affecting functionality of PDF document. A string pattern that is
separated by whitespace is injected into the gap between CRT and the trailer
of targeted PDF files as demonstrated in Figure 28. To match with specific
PDFrate regular expression, patterns are also crafted. Two attack algorithms
are implemented as explained:

  * •

Mimicry attack independent of underlying classifier, mimics a benign file by
changing the modifiable features of malicious file. To increase effectiveness
of the approach, each malware is trained to mimic 30 different benign files.

  * •

Gradient Descent and Kernel Density Estimation (GD-KDE) attack defeats
classifier with a known and differentiable decision function [320]. GD-KDE
algorithm follows the gradients of the classifier’s decision function and the
estimated density function of benign samples.

Figure 29: A parser confusion attack.

PDF detection techniques are mostly reliant on PDF parser to extract features
for classification [333, 334, 335]. These parser are unable to extract all
JavaScript of PDF file. Carmony et al. [323] created a reference JavaScript
extractor that measured the difference between the parser and Adobe Reader by
tapping Adobe reader on locations given by binary analysis. Manual analysis
refines the few candidate tap points provided by dynamic binary analysis.
JavaScript extraction tap points are a function from which Adobe Reader
extracts and executes JavaScript code from PDF documents. The memory accessed
by Adobe Readers when reading PDF files using automatically executable
JavaScript is analyzed to determine the raw JavaScript extraction tapping
points. Proposed PDF parser confusion attack apply obfuscation on malicious
PDF sample by analyzing the weaknesses of extractors in approach as shown in
Figure 29. Reference extractor enables several new obfuscation in comparison
to existing extractors and combination of these obfuscation were able to
bypass all JavaScript extractor based detector. Metadata based detection
systems require parser confusion attacks to be combined with reverse mimicry
attack as core content of sample is not changed by confusion attacks.

In order to preserve maliciousness, most of research works take conservative
approach by only inserting new contents and refraining from modification or
removal of existing contents. Xu et al. [247] proposed a black-box generic
method to evade the classifier as shown in Figure 30. As in figure, first the
population is initialized by performing random modifications on malicious
file. Then, each member of populations are passed through target classifier to
measure maliciousness and through oracle to confirm the functionality. If no
any samples are able to evade target classifier with functionality intact,
subset of initialized population are chosen for next generation based on
fitness score which indicates the progress towards evasive sample. Now, the
population generation is repeated and this process is continued till the
evasive sample is found or threshold iterations is met. Efficiency of search
is enhanced by collecting traces of used mutation operation and reuse the
effective operations. These effective traces are used for population
initialization to generate variants for other malware. The author uses genetic
programming (GP) to bring off stochastic modifications in an iterative manner
till evasion. The oracle output and results of prediction from the target
classifier need to be fed to the fitness function. Non malicious samples
determined by oracle are assigned with low fitness score while malicious
samples are provided with high score. Tools used as an oracle for
maliciousness verification of adversarial samples, is an open computational
challenge along with the selection of appropriate fitness function. The
generic method is capable of automatically finding evasive variants
irrespective of detection algorithms.

Figure 30: A PDF adversarial generation based on genetic algorithm [247].

###  7.4 Hardware Based Malware Adversarial

TABLE XI: Summary of hardware based malware adversarial.

Paper/Year |  Target Model |  Data Collection |  Feature Vectors |  Approach
---|---|---|---|---
Khasawneh et al. 2017 [336] |  Logistic Regression and neural network based detectors |  • Running malware and cleanware files on a virtual machine operating on Windows 7 • Dynamic traces collected using Pin instrumentation tool • Dynamic traces providing run time behaviour of the programs |  • Instructions Feature • Memory address patterns • Architectural Events |  • Reverse engineering to create surrogate model of HMD • Dynamically instruction insertion into malware execution through Dynamic Control Flow Graph • Weighted injection strategy with insertion instruction selection proportional to negative weight
Dinakarrao et al. 2019 [337] |  Logistic Regression and neural network based detectors |  • Captured using Hardware Performance Counters (HPC) • Perf tool available under Linux used |  • Low-level micro-architectural events • LLC load misses, branch instructions, branch misses and executed instructions |  • Reverse engineering of Black-box HMD • HPC patterns perturbation mechanism determined using FGSM • Perturbation calculated using neural network • Adversarial generators running as separate thread to avoid interference with original source code
Nozawa et al. 2021 [338] |  Neural network architecture |  • Structural features analysis |  • Gate level netlist |  • Hardware circuits represented in graph structure and converted to feature space • During design step or after logic synthesis • Trojan-net concealment degree to prevent from detection • Modification evaluating value to limit the extent of modification

Target Model: Defense model under adversarial attack, Data Collection: Feature
value collection process, Feature Vectors: Types of features considered,
Approach: Process of crafting adversarial

Figure 31: An adversarial generation workflow against HMD.

Hardware malware detectors use low-level information of features collected
from hardware performance monitoring units available in CPUs. Hardware malware
detectors are prone to reverse engineering [339], allowing mimicry attack
[340] to reverse-engineered models. Adversarial against such detectors are
carried out by generating perturbations in form of low-level hardware
features, following the architecture shown in Figure 31. These adversarial
generation approaches differ only on type of features used in comparison to
previously discussed works. Table XI provides brief comparison of adversarial
attacks against hardware malware detectors. Khasawneh et al. [336, 341]
demonstrated evasion of Hardware Malware Detectors(HMD) after being reverse
engineered, using low overhead evasion strategies. Data collected by running
malware and cleanware programs on a virtual machine operating on Windows 7 are
used to train a reverse-engineered model. Execution of malware requires
disabling of Windows security services and firewall. Data required for
training are dynamic traces while executing the program and are collected by
using Pin instrumentation tool [342] by executing 500 system call or 15
millions of committed instructions. These dynamic traces are profiling of a
run time behaviour of the programs. This dataset is comprised of three types
of feature vectors:

  * •

Instructions feature giving the frequency of instructions.

  * •

Memory address patterns giving distribution of memory references.

  * •

Architectural events giving the occurrence of architectural events.

Reverse engineering allows to methodically create model similar to HMD, given
the attacker has ability to query the target detector. Target detectors are
considered to be based on logistic regression and neural networks to
generalize most of the classification algorithms. Reverse engineering is
carried out in all three types of feature vectors using both algorithms.
Attackers have no information about size of instruction window, the specific
features used by detector or the classification algorithm, detectors are
trained on but have similar detectors to test their hypothesis. Authors[336]
constructed a Dynamic Control Flow Graph (DCFG) of the malware to insert
instructions into the executing malware dynamically. Injection of instruction
feature increases the weight of the corresponding feature while memory feature
injection alters the histogram of memory reference frequencies. Instructions
are inserted using two approaches, block level and function level. Khasawneh
et al. picked the instructions with negative weights to move the malware away
from the decision boundary. Heuristic approach was taken to identify the
candidate instructions for insertion. Weighted injection strategy where
probability of selecting particular instruction is proportional to negative
weight allowed to bypass HMD with around 10% dynamic overhead.

Dinakarrao et al. [337] also proposed an adversarial attack on low-level
micro-architectural events captured through Hardware Performance Counters
(HPC). Victim’s defense system (HMD) being black-box needs to be reverse
engineered to mimic the behaviour. Number of HPC patterns required to bypass
HMD is unknown which leads to need of adversarial sample predictor. The HPC
patterns perturbing mechanism are implemented using a lower-complexity
gradient approach, Fast Gradient Sign Method (FGSM).The adversarial
perturbations needed to misclassify HPC trace is calculated using gradient
based cost function of neural network. With $\theta$ being hyperparameters of
neural network, $x$ being input HPC trace to the model and $y$ as output, cost
function $L(\theta,x,y)$ is defined as:

$x^{adv}=x+\epsilon~{}sign(\Delta_{x}L(\theta,x,y))$ (26)

where $\epsilon$ is a scaling constant ranging between 0.0 to 1.0 and used to
limit the perturbation to a very small value. The LLC load misses and branch
misses are the most significant micro-architectural events of malicious
application [343]. Linear models are built to find dependency of array
sizes(n) and elements flushed(k) in route to determine LLC load misses
generated. The adversarial perturbation generator is executed separately with
a sample application to misclassify. The significance of running in a
separated thread is to stop adversarial HPC generators from interfering with
the application’s source code.

Micro-architectural events are useful when the malware program executes on a
hardware device causing some changes in the performance of hardware. In
addition to malware program execution, threat of injecting a malicious circuit
during fabrication of hardware devices has grown [344]. Malicious circuits,
also known as hardware Trojans, can be inserted into circuits producing
logically equivalent results. Hardware circuits like processors include
contribution of number of vendors before reaching to final product, and hence,
carries high risk of adversarial circuit insertion anywhere between designing
stage to manufacturing stage. Modifications in manufacturing stage are more
tedious in comparison to design stage as few changes in hardware description
language (HDL) are enough to embed hardware Trojans to the circuit. A trigger
circuit allows the payload circuit to trigger malicious behavior such as
information leakage and degrading performance after satisfying the trigger
condition. A gate level netlist (the lists of nets and circuit elements) is
used for malware detection by analyzing its structural features [345]. Several
hardware Trojan detection have been recently adopted towards machine learning
approaches including neural networks [346, 347, 348, 349].

Nozawa et al. [338] proposed an architecture to develop the adversarial
hardware Trojan using Trojan-net Concealment Degree (TCD) and Modification
Evaluating Value (MEV). Feature mapping issues like in all other adversarial
attacks in Windows, Android and PDF are prevalent in hardware Trojan as well.
Hardware circuits are represented in graphs structure and modifications in
feature space does not guarantee the transfer back of modification to graph
structure. Two stages in designing period are known for adversarial attack.
First one being RTL (Register-transfer level) description design step and
second one after logic synthesis. Authors take the assumption of Trojan
detector using neural network architecture and the availability of raw output
values from the detector to train adversarial model. Goal of adversarial is to
maximize the loss function which is given as cross entropy(H) in Equation 27:

$H=-\sum_{i=1}^{K}p_{i}(x(e))log\;q_{i}(x(e))$ (27)

where K is the number of units in output layer, $p_{i}(x(e))$ is the function
to return label of $x(e)$ and $q_{i}(x(e))$ is function that returns the
prediction result from classifier.

Summing up the loss function for each net and calculating the average gives
Trojan-net concealment degree (TCD):

$TCD=-\frac{1}{|E_{t}|}\sum_{e_{t}\epsilon
E_{t}}(\sum_{i=1}^{K}p_{i}(x(e_{t}))log\;q_{i}(x(e_{t}))$ (28)

Larger TCD indicates the bigger difference in values of the prediction and
answer, enabling to achieve the concealment of Trojan nets. However, to
monitor the amount of modification authors used modification evaluating value
(MEV).

$\displaystyle MEV=-TCD+\sum_{j=1}^{N}\lambda_{j}m_{j}$ (29)
$\displaystyle=\\-\frac{1}{|E_{t}|}\sum_{e_{t}\epsilon
E_{t}}(\sum_{i=1}^{K}p_{i}(x(e_{t}))log\;q_{i}(x(e_{t}))+\sum_{j=1}^{N}\lambda_{j}m_{j}$

where $m_{j}(1\leq j\leq N)$ is one of the $N$ kinds of evaluation indicators
and $\lambda_{j}(1\leq j\leq N)$ being the corresponding coefficient. The
success of research work in evading Trojan nets from Trojan detectors reveal
threat of adversarial at a whole new level.

###  7.5 Linux Malware Adversarial

Distributed edge computing has increased use of IoT devices. With large number
of devices using Linux systems, robust malware detection is paramount. Both
deep learning networks and Control Flow Graph (CFG) based malware detectors in
IoT devices are found to be vulnerable against adversarial samples [350]. In
off-the-shelf adversarial attack, authors examined different well known
adversarial algorithms based on feature extraction. Generic adversarial
algorithms are successful in adversarial generation with high evasion rate but
limited on applicability of practical changes to feature space. In response to
these challenges, adversarial based on control flow graph has been
proposed[350]. Programs are structurally analyzed using vertices and edges
with help of CFG. Graph embedding and augmentation (GEA) approach combines
original graph with target graph, producing mis-classification while
preserving the functionality of original program. GNU compiler collection
command compiles in a way that only functionality related to original sample
is executed. Linux based malware binaries easily evade IoT malware detection
from different graph algorithmic constructs. In our search of literature, we
found very limited works carried out as adversarial malware attacks in Linux
domain.

##  8 Challenges and Future Directions

Following the introduction of adversarial attacks against deep learning by
Szegedy et al. [71], machine learning research community have been concerned
about its impact in different application domains. The research on adversarial
attacks and its countermeasures is gaining momentum. To contribute towards the
literature, we conducted a comprehensive research on various adversarial
evasion attacks carried out against malware detection domain. Although, our
survey highlights several successful adversarial attacks crafted against anti-
malware engines, novel attacks are still evolving. In this section, we will
discuss potential research open challenges and future direction as the
adversarial approaches in malware analysis domain become more prevalent. Our
intention is in no way to overlook or understate the contributions of existing
adversarial attack researchers in malware domain.

###  8.1 Realistic (Practical) Attacks

Most of the existing adversarial attacks discussed in the survey are carried
out using white-box approaches. In white-box approach, attackers are assumed
to have full access to target model providing all required internal
information to attackers. White-box approach is considered by most an
unrealistic scenario in itself as it is unlikely that any ML based anti-
malware engine will reveal information such as algorithms used, gradients of
the model and hyper-parameters used to fine tune the model. Getting this
information about a target model provides ‘superpower’ to attackers as they
can camouflage the data in any way they want. In the future, research is
expected to be more inclined towards complete black-box attacks. Few of the
existing black-box attacks also depend on the performance of models provided
in numeric form. Obtaining numerical performance is also not a realistic
approach for attackers. So, we believe that further work should seek for
completely black-box approaches for carrying out adversarial attacks.

In addition, the attacks discussed previously are primarily focused on static
malware detection. The main reason behind it is the limited research carried
out to test the robustness of dynamic detection. The modern industrial malware
detection engines merge both static and dynamic detection techniques. Further,
the attacker rarely gets the privilege to work with data at rest. There has
been very few successful attempts to craft adversarial examples against data
in motion [351, 352]. The malware domain can have data which is moving at a
very high pace and may require to perform attack on data in motion. Data
stored in a storage device or in transit may have enough time to let attackers
generate adversarial examples for them while data moving at very high speed
might allow smallest of time frames for attacker. Adversarial attacks are not
always swift enough to work with data moving across network channels. So, more
adversarial attacks are to be experimented for systems deployed with both
static and dynamic detection as well as against data at motion.

###  8.2 Perturbation Insertion Space

Smart perturbation insertion plays a key role in determining the success of
adversarial attacks. Initial adversarial evasion attacks on malware began by
placing perturbations at the end of the malware file [265]. Most of the
existing attack approaches are concentrated on additive adversarial
perturbation. Demetrio et al. [99] later discovered that perturbations
embedded at header sections of file resulted in effective adversarial attacks
compared to perturbations appended at the end. However, header perturbation
invited the risk of perturbation being detected and also increased the chances
of breaking the malware file functionality. Suciu et al. [100] further
investigated the possibility of inserting perturbations in slack regions of
file which are left behind by the compilers. Experiments demonstrated that
perturbations that are inserted in slack region are more effective than
perturbations at other locations. These experiments provide inconclusive
information about suitable insertion space for perturbation. Hence, further
research is needed to determine optimal locations for perturbation that are
more effective as well as undetected.

###  8.3 Enhancing Efficiency

Adversarial efficiency can be defined in terms of different parameters. The
first and obvious efficiency criteria is the length of the payload to be
generated as perturbation. Generating random noise is an inefficient approach
while gradient based algorithms are developed to make an efficient adversarial
attacks. The significance of the inserted payload determines the efficiency of
perturbation. One way to insert efficient features is to first decipher the
importance of each feature in the decision making of the machine learning
model. Highly influential features can be modified to reach the adversarial
goal with minimal perturbation volume. Despite the gradient helping attackers
to generate perturbation in right direction, efficiency may be limited due to
uncountable iterations to reach the adversarial goal. Applying small
perturbations iteratively results in high quality adversarial evasion.
However, these approaches will require immense amount of time, making it
impossible for real time operation. To challenge this limitation, approaches
like Fast Gradient Sign Method are proposed, which produce perturbations at
very high pace but are less effective and have a high chance of being
detected. Hence, research is needed to ensure that efficiency is looked both
in terms of quantity and quality of noise generated to produce adversarial
evasion. In addition, trade off between performance and computational
complexity should be analysed to evaluate the worth of performing adversarial
attacks [353].

###  8.4 Mapping Space Challenge

Mapping between problem space and feature space is performed by an embedding
layer present in between them. The features in problem space can be of any
form like n-grams, API names or other non-numeric parameters which can not be
directly processed by machine learning models. This causes the problem space
vectors to be converted into feature space which are some form of numeric
values. The embedding layer however is an approximation mapping table between
features in problem space and feature space. One of the biggest challenge of
adversarial attacks is to map features in problem space and feature space
precisely. Machine learning models require malware features to be converted to
feature space from problem space so that adversarial examples can be crafted
on them. However, there is no exact mapping between these spaces which results
in approximate mapping, leading to slightly altered feature space than
original problem space. After adversarial examples are crafted on malware
files, mapping features back to problem space also lose few crafted
perturbations due to lack of absolute mapping. Therefore, the challenge for
defining adversarial space and efficiently searching elements approaching the
best replacement has always been there in adversarial domain. Further research
is needed to identify and map features in embedding space.

###  8.5 Automated Attacks

All of the discussed adversarial attacks require manual intervention at a few
steps of the attack procedure. Human intervention makes the process time-
consuming and impractical in many cases. In white-box attacks, the loss
function of deep neural network can be used to determine most influential
features and the corresponding features can be automatically modified [93].
Current literature relies on human efforts for feature extraction, mapping to
adversarial generation and functionality verification. Minimizing human effort
while moving towards automated adversarial generation could be the interesting
arena to work on the future [354]. Novel research is needed to fully automate
the adversarial attack ecosystem.

###  8.6 Explainable Adversarial

Adversarial vulnerabilities have been considered blind spots of machine
learning models but current research work fails to assert concrete reasoning
behind these blind spots. Having no consensus behind such reasoning leaves
explaining the existence of adversarial example an open research domain.
Goodfellow et al [74] first attributed vulnerability to the linear behavior of
model in high dimensional space. However, there have been research that
contradicts the accountability of adversarial behavior solely to linearity of
model as highly non-linear models are also evaded successfully [355].
Explaining the adversarial phenomenon both in terms of models’ functionality
and features’ contribution can pave a path for more robust adversarial
attacks. Features can be assigned appropriate weights based on their
contribution to alleviate the adversarial effect in the model. With the
current state of the literature, explainable adversarial is still at an
immature stage and requires concrete efforts from the community.

###  8.7 Transferable Attacks

Transferability refers to generalization property of the attack methods. A
machine learning model with transferability property, trained on one
particular dataset can generalize well for another different dataset as well.
Transferability is a common property for evasion attack and is extensively
exploited by black-box attacks. Untargeted attacks are found more transferable
than targeted ones due to their generality [354]. Transferability can also
take three different forms such as, same architecture with different data,
different architecture with same application and different architecture with
different data [81]. Although some studies have already been carried out on
transferability, there is no any universally accepted postulation. The ability
to use same data, model or algorithm to attack all available targets should be
one of the goals of future research on adversarial attacks. Thus, attacker
having transferability in their models should be able to attack defensive
system irrespective of input and context in new targets.

###  8.8 Attacking Adversarial Defense

The influx of research on adversarial domain during last few years demonstrate
the extent and importance of work in performing adversarial attack. The
profound activity has not been limited to attack side but considering the
threat posed to entire machine learning family, researchers have been equally
active on defensive side as well. The cyber war between adversarial attackers
and defenders are marching on an extremely high pace to overtake each other.
Performing adversarial attacks are turning out to be harder than ever as many
systems are designed robustly with adversarial defense in mind. Defensive
approaches like adversarial training [356], defensive distillation [357] are
proposed to stop adversarial attacks. Some recent techniques are hiding the
gradients of target model [358], which if carried out successfully can
completely nullify the threat of gradient based adversarial attacks. Hence,
future adversarial attacks are required not only to bypass the machine
learning detection but also overcome adversarial defenses. At the same time,
novel defense mechanisms and approaches must be designed to make our models
resilient against growing adversarial attack ecosystem.

###  8.9 Functionality Verification

Adversarial attacks on image domain carries a longer history than in malware
domain. Image classifiers are attacked by modifying the image pixels to create
adversarial. While modifying images, the pixels can be abundantly disrupted
till it impacts the human perception. However, adversarial attacks is
completely a different story in malware domain. The modifications carried out
in a malware file should not alter the functionality of malware. The contents
in executable file could be very sensitive and modification of a single byte
can completely change the functionality of malware or even break the file,
making it unexecutable. Most of the adversarial attacks have constrained
themselves in perturbation type, volume and insertion techniques, just to
preserve the functionality of executable. Malware functionality should not be
compromised at the cost of any other adversarial constraints. Despite such
gravity, most of the adversarial attacks are still not able to preserve the
functionality of a modified files. Moreover, limited mechanisms exist to
verify the functionality of malware after perturbing the file. One of the
available approach is to run the malware file in an isolated environments like
Cuckoo Sandbox [359]. But running every individual malware in a sandbox is
inefficient and unrealistic. Therefore, further research should be directed to
develop tools that can automatically and efficiently verify the functionality
of malware post perturbations.

###  8.10 Attacking Federated Learning

Federated learning [360] is a hot topic as it allows individual nodes in a
system to train a shared prediction model by confining all the data on
individual devices [361]. This has gained a great popularity recently as it
reduces the computational cost of centralized machine while preserving data
privacy of each individual nodes. However, federated learning comes with
greater risk of adversarial attacks as central system has no control over the
training data. Learning system can easily be poisoned while training, leaving
the backdoors in trained model [362]. The training model with backdoors can be
easily evaded using targeted or untargeted attacks. As federated learning is
coming into limelight recently, the adversarial risks are yet to be properly
quantified before embracing the technology that could be the future.

###  8.11 Benign Files Attack

Adversarial attacks are performed in malware files by inserting some non-
malicious contents which do not tamper with any functionality other than
classification decision. Modifying malware files slowly has been a mainstream
approach for adversarial. However, no limited or no existing research has
studied the possibility of inserting malicious contents to a benign files.
This approach works in a reverse way than the established adversarial
approaches. Inserting and hiding malicious payload at different locations of
file without affecting the classification decision is also a future research
topics in adversarial and requires attention.

###  8.12 Targeting Unexplored Algorithms

Most of the machine learning algorithms have already been victimized by
adversarial attackers, including sophisticated deep neural network
architectures. However, there are some deep neural networks that haven’t yet
been compromised by adversarial attackers such as Generative Adversarial
Networks (GANs), Deep Reinforcement learning (DRL) and Variational Auto-
Encoders (VAEs) [93]. These algorithms are itself in development stage, which
has capped the adversarial attempt to them till date. Differentiable neural
computer [363] are only attacked once [364]. These new sets of algorithms are
yet to be explored by adversarial attackers.

###  8.13 Standardizing Testbed and Metrices

Adversarial attacks discussed in the survey are carried out in lab
environments, taking numerous assumptions which may be unpragmatic for real
world challenges. Most of the works have assumed unlimited access to machine
learning model, favourable dataset and weak classifiers to bolster their
results. The current literature lacks standardized dataset and detection
mechanisms to measure the exact performance of adversarial attack. As vast
number of research works are performed on different dataset and target model,
it is not possible to compare the performance of attacks. Hence, the attack
testbed should be standarized to bring the assessment uniformity across the
research community and uplift the attack standard out from the lab
environment.

The issue is not limited to test environment but also with evaluation
metrices. More than often, the performance of adversarial attacker is
reflected in terms of evasion accuracy inherited from machine learning models.
However, accuracy only provides small fraction of attacker’s performance in
adversarial domain. To provide the overall quality of attacks, metrics such as
transferability, universality and imperceptability need to be studied [353].
The metrics should be descriptive, fair, and complete to evaluate the quality
of attacks performed across different environments. Some metrics should also
be developed to measure the degree of functionality preservation while
manipulating the files. Incorporating attack’s meaning preservation capability
as a quality metrics have shown the benefit in recent works [365]. Some
distance metrics can be used to determine the differences between original
file and adversarial modification. Metrics can also be designed to determine
the sensitivity of file structure, helping attacker to determine the level of
cautiousness required while modification. These complete and fair metrices
will not only help to understand and compare the adversarial quality but also
to enhance the performance of attacks.

###  8.14 Adversarial Defense

The growth in adversarial attacks and novel approaches will also require
developing advanced defense mechanisms. Although, our survey is focused on
adversarial evasion attacks, we believe it is important to briefly highlight
future defense directions to present a comprehensive review paper. Among
several defense techniques proposed, defensive distillation [366] and
adversarial training [367, 368] are found to be the most effective. While
talking about the effectiveness of existing works, we cannot undermine the
challenges faced by them. Collection of adversarial samples in large amount to
perform adversarial training is a tedious task as neural networks require
massive volume of adversarial data [369]. An adversarial generation approach
was proposed by Goodfellow et al. [182], however is still very far away from
being efficient and accurate enough to perform the robust adversarial
training. In addition, many defensive approaches that have been tested in an
image domain [370, 371] are yet to be introduced for a defense in malware
adversarial domains. Recent research using robust machine learning
architectures like Generative Adversarial Networks (GANs) [372] for defending
against adversarial attacks require more exploration to thwart or detect
sophisticated evasion attacks. Overall, the future research works on
adversarial malware should be directed to build more robust, efficient,
generalized and reliable defense mechanisms that can protect malware detection
models against the adversarial attacks.

##  9 Conclusion

Machine learning and AI solutions are increasingly playing an important role
in cyber security domain. However, these data driven systems can be easily
manipulated, misled and evaded which can have serious implications. Recent
surge and research in adversarial attacks highlight the vulnerability of ML
models making them ineffective against even minor perturbations. In this
paper, we provide a comprehensive survey of recent work that focuses on
adversarial evasion attacks in malware analysis domain. We have summarized the
state-of-art adversarial attacks carried out against anti-malware engines in
different file domains. The survey demonstrates the flaw of machine learning
architectures against minute perturbations in form of adversarial attacks. We
taxonomize the adversarial evasion world of malware based on attack domain and
approach taken to realize adversarial evasion attacks. Survey briefly
discusses approaches taken by researchers, comparing them with other
concomitant works. We conclude the survey highlighting current challenges,
open issues and future research directions in adversarial malware analysis.
This work will provide a definitive guide to researchers and community, to
understand the current scenarios of adversarial malware evasion attacks and
prompting unexplored research territories in this highly dynamic and evolving
domain.

## References

  * [1] Total data volume worldwide 2010-2025.  https://www.statista.com/statistics/871513/worldwide-data-created/.
  * [2] Daniel W. Otter, Julian R. Medina, and Jugal K. Kalita.  A Survey of the Usages of Deep Learning for Natural Language Processing.  IEEE Transactions on Neural Networks and Learning Systems, 32(2):604–624, 2021.
  * [3] Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria.  Recent Trends in Deep Learning Based Natural Language Processing [review article].  IEEE Computational Intelligence Magazine, 13(3):55–75, 2018.
  * [4] Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer.  AllenNLP: A Deep Semantic Natural Language Processing Platform.  arXiv preprint arXiv:1803.07640, 2018.
  * [5] D Jude Hemanth and V Vieira Estrela.  Deep Learning for Image Processing Applications, volume 31.  IOS Press, 2017.
  * [6] Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib.  Deep Learning for Medical Image Processing: Overview, Challenges and the Future.  Classification in BioApps, pages 323–350, 2018.
  * [7] Luis Perez and Jason Wang.  The Effectiveness of Data Augmentation in Image Classification using Deep Learning.  arXiv preprint arXiv:1712.04621, 2017.
  * [8] Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma.  PCANet: A Simple Deep Learning Baseline for Image Classification?  IEEE Transactions on Image Processing, 24(12):5017–5032, 2015.
  * [9] Mohammed Al-Qizwini, Iman Barjasteh, Hothaifa Al-Qassab, and Hayder Radha.  Deep learning algorithm for autonomous driving using GoogLeNet.  In IEEE Intelligent Vehicles Symposium (IV), pages 89–96, 2017\.
  * [10] Hironobu Fujiyoshi, Tsubasa Hirakawa, and Takayoshi Yamashita.  Deep learning-based image recognition for autonomous driving.  IATSS research, 43(4):244–252, 2019.
  * [11] Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani.  Deep Reinforcement Learning Framework for Autonomous Driving.  Electronic Imaging, 2017(19):70–76, 2017.
  * [12] Brody Huval, Tao Wang, Sameep Tandon, Jeff Kiske, Will Song, Joel Pazhayampallil, Mykhaylo Andriluka, Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue, et al.  An Empirical Evaluation of Deep Learning on Highway Driving.  arXiv preprint arXiv:1504.01716, 2015.
  * [13] Adam H Marblestone, Greg Wayne, and Konrad P Kording.  Toward an Integration of Deep Learning and Neuroscience.  Frontiers in computational neuroscience, 10:94, 2016.
  * [14] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al.  A Deep Learning Framework for Neuroscience.  Nature neuroscience, 22(11):1761–1770, 2019.
  * [15] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen A Baccus, and Surya Ganguli.  From Deep Learning to Mechanistic Understanding in Neuroscience: The Structure of Retinal Prediction.  In Proceedings of the International Conference on Neural Information Processing Systems, 2019.
  * [16] Mackenzie Weygandt Mathis and Alexander Mathis.  Deep learning tools for the measurement of animal behavior in neuroscience.  Current Opinion in Neurobiology, 60:1–11, 2020.
  * [17] Mahmoud Abdelsalam, Ram Krishnan, Yufei Huang, and Ravi Sandhu.  Malware Detection in Cloud Infrastructures Using Convolutional Neural Networks.  In IEEE International Conference on Cloud Computing, pages 162–169, 2018.
  * [18] Justin Sahs and Latifur Khan.  A machine learning approach to android malware detection.  In 2012 European Intelligence and Security Informatics Conference, pages 141–147. IEEE, 2012.
  * [19] Andrew McDole, Maanak Gupta, Mahmoud Abdelsalam, Sudip Mittal, and Mamoun Alazab.  Deep learning techniques for behavioral malware analysis in cloud IaaS.  In Malware Analysis using Artificial Intelligence and Deep Learning, pages 269–285. Springer, 2021.
  * [20] Andrew McDole, Mahmoud Abdelsalam, Maanak Gupta, and Sudip Mittal.  Analyzing CNN based behavioural malware detection techniques on cloud IaaS.  In International Conference on Cloud Computing, pages 64–79. Springer, 2020.
  * [21] Jeffrey C Kimmel, Andrew D Mcdole, Mahmoud Abdelsalam, Maanak Gupta, and Ravi Sandhu.  Recurrent Neural Networks Based Online Behavioural Malware Detection Techniques for Cloud Infrastructure.  IEEE Access, 9:68066–68080, 2021.
  * [22] Jeffrey C Kimmell, Mahmoud Abdelsalam, and Maanak Gupta.  Analyzing Machine Learning Approaches for Online Malware Detection in Cloud.  In IEEE Smartcomp, 2021.
  * [23] Thiago S Guzella and Walmir M Caminhas.  A review of machine learning approaches to Spam filtering.  Expert Systems with Applications, 36(7):10206–10222, 2009.
  * [24] Konstantin Tretyakov.  Machine Learning Techniques in Spam Filtering.  In Data Mining Problem-oriented Seminar, MTAT, volume 3, pages 60–79. Citeseer, 2004.
  * [25] Chih-Chin Lai.  An empirical study of three machine learning methods for spam filtering.  Knowledge-Based Systems, 20(3):249–254, 2007.
  * [26] Emmanuel Gbenga Dada, Joseph Stephen Bassi, Haruna Chiroma, Adebayo Olusola Adetunmbi, Opeyemi Emmanuel Ajibuwa, et al.  Machine Learning for Email Spam Filtering: Review, Approaches and Open Research Problems.  Heliyon, 5(6):e01802, 2019.
  * [27] John O Awoyemi, Adebayo O Adetunmbi, and Samuel A Oluwadare.  Credit card fraud detection using Machine Learning Techniques: A Comparative Analysis.  In IEEE International Conference on Computing Networking and Informatics, 2017.
  * [28] Dejan Varmedja, Mirjana Karanovic, Srdjan Sladojevic, Marko Arsenovic, and Andras Anderla.  Credit Card Fraud Detection-Machine Learning methods.  In International Symposium INFOTEH-JAHORINA (INFOTEH), pages 1–5. IEEE, 2019.
  * [29] Aderemi O Adewumi and Andronicus A Akinyelu.  A Survey of Machine-Learning and Nature-Inspired Based Credit Card Fraud Detection Techniques.  International Journal of System Assurance Engineering and Management, 8(2):937–953, 2017.
  * [30] Abhimanyu Roy, Jingyi Sun, Robert Mahoney, Loreto Alonzi, Stephen Adams, and Peter Beling.  Deep Learning Detecting Fraud in Credit Card Transactions.  In IEEE Systems and Information Engineering Design Symposium, pages 129–134, 2018.
  * [31] Dinggang Shen, Guorong Wu, and Heung-Il Suk.  Deep Learning in Medical Image Analysis.  Annual Review of Biomedical Engineering, 19:221–248, 2017.
  * [32] Miles N Wernick, Yongyi Yang, Jovan G Brankov, Grigori Yourganov, and Stephen C Strother.  Machine Learning in Medical Imaging.  IEEE Signal Processing Magazine, 27(4):25–38, 2010.
  * [33] Igor Kononenko.  Machine Learning for Medical Diagnosis: History, State of the Art and Perspective.  Artificial Intelligence in Medicine, 23(1):89–109, 2001.
  * [34] Bradley J Erickson, Panagiotis Korfiatis, Zeynettin Akkus, and Timothy L Kline.  Machine Learning for Medical Imaging.  Radiographics, 37(2):505–515, 2017.
  * [35] Ashwin Dhakal, Cole McKay, John J Tanner, and Jianlin Cheng.  Artificial intelligence in the prediction of protein-ligand interactions: recent advances and future directions.  Briefings in Bioinformatics, page bbab476.
  * [36] Evan Martin and Tao Xie.  Inferring access-control policy properties via machine learning.  In IEEE International Workshop on Policies for Distributed Systems and Networks, 2006.
  * [37] Maanak Gupta, Farhan Patwa, and Ravi Sandhu.  POSTER: Access control model for the Hadoop ecosystem.  In Proceedings of the ACM on Symposium on Access Control Models and Technologies, 2017.
  * [38] Leila Karimi and James Joshi.  An Unsupervised Learning Based Approach for Mining Attribute Based Access Control Policies.  In IEEE International Conference on Big Data (Big Data), 2018.
  * [39] Maanak Gupta and Ravi Sandhu.  Towards Activity-Centric Access Control for Smart Collaborative Ecosystems.  In Proceedings of the ACM Symposium on Access Control Models and Technologies, 2021.
  * [40] Zahra Bazrafshan, Hashem Hashemi, Seyed Mehdi Hazrati Fard, and Ali Hamzeh.  A Survey on Heuristic Malware Detection Techniques.  In IEEE Conference on Information and Knowledge Technology, pages 113–120, 2013.
  * [41] Min Zheng, Mingshen Sun, and John CS Lui.  DroidAnalytics: A Signature Based Analytic System to Collect, Extract, Analyze and Associate Android Malware.  In IEEE Conference on Trust, Security and Privacy in Computing and Communications, 2013.
  * [42] Deepak Venugopal and Guoning Hu.  Efficient signature based malware detection on mobile devices.  Mobile Information Systems, 4(1):33–49, 2008.
  * [43] Abhaya Kumar Sahoo, Kshira Sagar Sahoo, and Mayank Tiwary.  Signature based malware detection for unstructured data in Hadoop.  In International Conference on Advances in Electronics Computers and Communications, pages 1–6. IEEE, 2014.
  * [44] Muhamed Fauzi Bin Abbas and Thambipillai Srikanthan.  Low-Complexity Signature-Based Malware Detection for IoT Devices.  In International Conference on Applications and Techniques in Information Security, pages 181–189. Springer, 2017.
  * [45] Jin-Young Kim, Seok-Jun Bu, and Sung-Bae Cho.  Zero-day Malware Detection using Transferred Generative Adversarial Networks Based on Deep Autoencoders.  Information Sciences, 460:83–102, 2018.
  * [46] Gavin Phillips, Gavin Phillips (987 Articles Published) Gavin is the Junior Editor for Windows, and Technology Explained.  These 4 Antivirus Tools Are Using AI to Protect Your System.  https://www.makeuseof.com/tag/artificial-intelligence-antivirus-tools/, Aug 2018.
  * [47] Konrad Rieck, Philipp Trinius, Carsten Willems, and Thorsten Holz.  Automatic analysis of malware behavior using machine learning.  Journal of Computer Security, 19(4):639–668, 2011.
  * [48] State of IoT 2021: Number of connected IoT devices growing 9% to 12.3 billion globally, cellular IoT now surpassing 2 billion, Sep 2021.
  * [49] Sara Morrison.  How a major oil pipeline got held for ransom.  https://www.vox.com/recode/22428774/ransomeware-pipeline-colonial-darkside-gas-prices, May 2021.
  * [50] Electronics Retailer MediaMarkt Hit by Ransomware Demand for $50M Bitcoin Payment: Report.  https://www.yahoo.com/lifestyle/electronics-retailer-mediamarkt-hit-ransomware-151038565.html.
  * [51] Lawrence Abrams.  Computer giant Acer hit by $50 million ransomware attack.  https://www.bleepingcomputer.com/news/security/computer-giant-acer-hit-by-50-million-ransomware-attack/, Mar 2021.
  * [52] MHR Khouzani, Saswati Sarkar, and Eitan Altman.  Maximum Damage Malware Attack in Mobile Wireless Networks.  IEEE/ACM Transactions on Networking, 20(5):1347–1360, 2012.
  * [53] Hao Hu, Steven Myers, Vittoria Colizza, and Alessandro Vespignani.  Wifi networks and malware epidemiology.  Proceedings of the National Academy of Sciences, 106(5):1318–1323, 2009.
  * [54] Michael Schwarz, Samuel Weiser, and Daniel Gruss.  Practical enclave malware with Intel SGX.  In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pages 177–196. Springer, 2019.
  * [55] Surbhi Gupta, Abhishek Singhal, and Akanksha Kapoor.  A Literature Survey on Social Engineering Attacks: Phishing Attack.  In IEEE International conference on computing, communication and automation (ICCCA), pages 537–540, 2016.
  * [56] Jason Hong.  The State of Phishing Attacks.  Communications of the ACM, 55(1):74–81, 2012.
  * [57] Tzipora Halevi, Nasir Memon, and Oded Nov.  Spear-Phishing in the Wild: A Real-World Study of Personality, Phishing Self-Efficacy and Vulnerability to Spear-Phishing Attacks.  Phishing Self-Efficacy and Vulnerability to Spear-Phishing Attacks, 2015.
  * [58] Ulrike Meyer and Susanne Wetzel.  A man-in-the-middle attack on UMTS.  In Proceedings of the 3rd ACM workshop on Wireless security, pages 90–97, 2004.
  * [59] Franco Callegati, Walter Cerroni, and Marco Ramilli.  Man-in-the-Middle Attack to the HTTPS Protocol.  IEEE Security & Privacy, 7(1):78–81, 2009.
  * [60] Simon Eberz, Martin Strohmeier, Matthias Wilhelm, and Ivan Martinovic.  A Practical Man-In-The-Middle Attack on Signal-Based Key Generation Protocols.  In European Symposium on Research in Computer Security, pages 235–252. Springer, 2012.
  * [61] Jelena Mirkovic, Sven Dietrich, David Dittrich, and Peter Reiher.  Internet Denial of Service: Attack and Defense Mechanisms (Radia Perlman Computer Networking and Security Book Series).  12 2004.
  * [62] Christoph L Schuba, Ivan V Krsul, Markus G Kuhn, Eugene H Spafford, Aurobindo Sundaram, and Diego Zamboni.  Analysis of a denial of service attack on TCP.  In Proceedings IEEE Symposium on Security and Privacy, pages 208–223, 1997.
  * [63] Tauseef Jamal, Zeeshan Haider, Shariq Aziz Butt, and Assim Chohan.  Denial of Service Attack in Cooperative Networks.  arXiv preprint arXiv:1810.11070, 2018.
  * [64] William G Halfond, Jeremy Viegas, Alessandro Orso, et al.  A Classification of SQL-Injection Attacks and Countermeasures.  In Proceedings of the IEEE international symposium on secure software engineering, volume 1, pages 13–15. IEEE, 2006.
  * [65] Adam Kieyzun, Philip J Guo, Karthick Jayaraman, and Michael D Ernst.  Automatic creation of SQL injection and cross-site scripting attacks.  In 2009 IEEE 31st international conference on software engineering, pages 199–209. IEEE, 2009.
  * [66] Chris Anley.  Advanced SQL injection in SQL server applications.  2002\.
  * [67] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein.  Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks.  arXiv preprint arXiv:1804.00792, 2018.
  * [68] Xuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, and Cho-Jui Hsieh.  A Unified Framework for Data Poisoning Attack to Graph-Based Semi-supervised Learning.  arXiv preprint arXiv:1910.14147, 2019.
  * [69] Di Cao, Shan Chang, Zhijian Lin, Guohua Liu, and Donghong Sun.  Understanding Distributed Poisoning Attack in Federated Learning.  In 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), pages 233–239. IEEE, 2019.
  * [70] Juncheng Shen, Xiaolei Zhu, and De Ma.  TensorClog: An Imperceptible Poisoning Attack on Deep Neural Network Applications.  IEEE Access, 7:41498–41506, 2019.
  * [71] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [72] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio.  Adversarial Examples in the Physical World.  ArXiv, abs/1607.02533, 2017.
  * [73] Ibrahim Yilmaz.  Practical Fast Gradient Sign Attack Against Mammographic Image Classifier.  arXiv preprint arXiv:2001.09610, 2020.
  * [74] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and Harnessing Adversarial Examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [75] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going Deeper with Convolutions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.
  * [76] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A Large-scale Hierarchical Image Database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009.
  * [77] Tom Huddleston Jr.  These Chinese hackers tricked Tesla’s Autopilot into suddenly switching lanes.  https://www.cnbc.com/2019/04/03/chinese-hackers-tricked-teslas-autopilot-into-switching-lanes.html, Apr 2019.
  * [78] Marco Barreno, Blaine Nelson, Anthony Joseph, and J. Tygar.  The security of machine learning.  Machine Learning, 81:121–148, 11 2010.
  * [79] Joseph Gardiner and Shishir Nagaraja.  On the Security of Machine Learning in Malware c&c Detection: A Survey.  ACM Computing Surveys (CSUR), 49(3):1–39, 2016.
  * [80] Atul Kumar and Sameep Mehta.  A survey on resilient machine learning.  arXiv preprint arXiv:1707.03184, 2017.
  * [81] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li.  Adversarial examples: Attacks and defenses for deep learning.  IEEE transactions on neural networks and learning systems, 2019\.
  * [82] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay.  Adversarial attacks and defences: A survey.  arXiv preprint arXiv:1810.00069, 2018.
  * [83] Naveed Akhtar and Ajmal Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  IEEE Access, 6:14410–14430, 2018.
  * [84] Vasisht Duddu.  A Survey of Adversarial Machine Learning in Cyber Warfare.  Defence Science Journal, 68:356–366, 2018.
  * [85] Guofu Li, Pengjia Zhu, Jin Li, Zhemin Yang, Ning Cao, and Zhiyi Chen.  Security matters: A survey on adversarial machine learning.  arXiv preprint arXiv:1810.07339, 2018.
  * [86] Qiang Liu, Pan Li, Wentao Zhao, Wei Cai, Shui Yu, and Victor C. M. Leung.  A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View.  IEEE Access, 6:12103–12117, 2018.
  * [87] Battista Biggio and Fabio Roli.  Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning.  Pattern Recognition, 84:317–331, Dec 2018.
  * [88] Lu Sun, Mingtian Tan, and Zhe Zhou.  A survey of practical adversarial example attacks.  Cybersecurity, 1(1):1–9, 2018.
  * [89] Nikolaos Pitropakis, Emmanouil A. Panaousis, Thanassis Giannetsos, Eleftherios Anastasiadis, and George Loukas.  A taxonomy and survey of attacks against machine learning.  Comput. Sci. Rev., 34, 2019.
  * [90] Xianmin Wang, Jing Li, Xiaohui Kuang, Yu an Tan, and Jin Li.  The security of machine learning in an adversarial setting: A survey.  J. Parallel Distributed Comput., 130:12–23, 2019.
  * [91] Shilin Qiu, Qihe Liu, Shijie Zhou, and Chunjiang Wu.  Review of Artificial Intelligence Adversarial Attack and Defense Technologies.  Applied Sciences, 2019.
  * [92] Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain.  Adversarial attacks and defenses in images, graphs and text: A review.  International Journal of Automation and Computing, 17(2):151–178, 2020.
  * [93] Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li.  Adversarial attacks on deep-learning models in natural language processing: A survey.  ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1–41, 2020.
  * [94] N. Martins, J. M. Cruz, T. Cruz, and P. Henriques Abreu.  Adversarial Machine Learning Applied to Intrusion and Malware Scenarios: A Systematic Review.  IEEE Access, 2020.
  * [95] Ilja Moisejevs.  Adversarial Attacks and Defenses in Malware Classification: A Survey.  2019\.
  * [96] Olakunle Ibitoye, Rana Abou-Khamis, Ashraf Matrawy, and M Omair Shafiq.  The Threat of Adversarial Attacks on Machine Learning in Network Security–A Survey.  arXiv preprint arXiv:1911.02621, 2019.
  * [97] Hyrum S Anderson, Anant Kharkar, Bobby Filar, and Phil Roth.  Evading Machine Learning Malware Detection.  Black Hat, pages 1–6, 2017.
  * [98] Bojan Kolosnjaji, Ambra Demontis, Battista Biggio, Davide Maiorca, Giorgio Giacinto, Claudia Eckert, and Fabio Roli.  Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables.  In 2018 26th European SIgnal Processing COnference (EUSIPCO), pages 533–537. IEEE, 2018.
  * [99] Luca Demetrio, Battista Biggio, Giovanni Lagorio, Fabio Roli, and Alessandro Armando.  Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries.  arXiv preprint arXiv:1901.03583, 2019.
  * [100] Octavian Suciu, Scott E Coull, and Jeffrey Johns.  Exploring Adversarial Examples in Malware Detection.  In 2019 IEEE Security and Privacy Workshops (SPW), pages 8–14. IEEE, 2019.
  * [101] Deqiang Li, Qianmu Li, Yanfang Ye, and Shouhuai Xu.  Sok: Arms Race in Adversarial Malware Detection.  arXiv preprint arXiv:2005.11671, 2020.
  * [102] Judith S Hurwitz, Alan Nugent, Fern Halper, and Marcia Kaufman.  Big Data for Dummies.  John Wiley & Sons, 2013.
  * [103] Seref Sagiroglu and Duygu Sinanc.  Big Data: A Review.  In IEEE International Conference on Collaboration Technologies and Systems (CTS), pages 42–47, 2013.
  * [104] Gerard George, Martine R Haas, and Alex Pentland.  Big Data and Management.  Academy of Management Journal, 57(2):321–326, 2014.
  * [105] Maanak Gupta, Farhan Patwa, and Ravi Sandhu.  Object-tagged RBAC model for the Hadoop ecosystem.  In IFIP Annual Conference on Data and Applications Security and Privacy. Springer, 2017.
  * [106] Maanak Gupta, Farhan Patwa, James Benson, and Ravi Sandhu.  Multi-layer authorization framework for a representative Hadoop ecosystem deployment.  In Proceedings of the ACM on Symposium on Access Control Models and Technologies, 2017.
  * [107] Maanak Gupta, Farhan Patwa, and Ravi Sandhu.  An attribute-based access control model for secure big data processing in Hadoop ecosystem.  In Proceedings of the Third ACM Workshop on Attribute-Based Access Control, 2018.
  * [108] J Bulao.  How Much Data Is Created Every Day in 2021?[You’ll be shocked!].  TechJury.[Online]. Available at https://techjury. net/blog/how-much-data-is-createdevery-day/.[Accessed: 20-Mar-2021], 2021.
  * [109] Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025.  https://www.statista.com/statistics/871513/worldwide-data-created/.
  * [110] Sean Fleming.  This is how many websites exist globally.  https://www.weforum.org/agenda/2019/09/chart-of-the-day-how-many-websites-are-there/, 2019\.
  * [111] Kit Smith.  60 Incredible and Interesting Twitter Stats and Statistics.  https://www.brandwatch.com/blog/twitter-stats-and-statistics/, 2020\.
  * [112] Thomas M. Mitchell.  Machine Learning.  McGraw-Hill, Inc., USA, 1 edition, 1997.
  * [113] Pat Langley.  Elements of Machine Learning.  Morgan Kaufmann, 1996.
  * [114] Sotiris B Kotsiantis, I Zaharakis, P Pintelas, et al.  Supervised Machine Learning: A Review of Classification Techniques.  Emerging Artificial Intelligence Applications in Computer Engineering, 160(1):3–24, 2007.
  * [115] Raphael Bost, Raluca Ada Popa, Stephen Tu, and Shafi Goldwasser.  Machine Learning Classification over Encrypted Data.  Cryptology ePrint Archive, 2014.
  * [116] Marco Pennacchiotti and Ana-Maria Popescu.  A machine learning approach to Twitter user classification.  In Fifth International AAAI Conference on Weblogs and Social Media, 2011.
  * [117] Shen Rong and Zhang Bao-wen.  The research of regression model in machine learning field.  In MATEC Web of Conferences, volume 176, page 01033. EDP Sciences, 2018.
  * [118] Andrii Babii, Eric Ghysels, and Jonas Striaukas.  Machine Learning Time Series Regressions With an Application to Nowcasting.  Journal of Business & Economic Statistics, pages 1–23, 2021.
  * [119] Ewout W Steyerberg, Tjeerd van der Ploeg, and Ben Van Calster.  Risk prediction with machine learning and regression methods.  Biometrical Journal, 56(4):601–606, 2014.
  * [120] Anthony McGregor, Mark Hall, Perry Lorier, and James Brunskill.  Flow Clustering Using Machine Learning Techniques.  In International Workshop on Passive and Active Network Measurement, pages 205–214. Springer, 2004.
  * [121] LV Bijuraj.  Clustering and its Applications.  In Proceedings of National Conference on New Horizons in IT-NCNHIT, volume 1, pages 169–172, 2013.
  * [122] DT Pham and AA Afify.  Clustering techniques and their applications in engineering.  Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 221(11):1445–1459, 2007.
  * [123] G Thippa Reddy, M Praveen Kumar Reddy, Kuruva Lakshmanna, Rajesh Kaluri, Dharmendra Singh Rajput, Gautam Srivastava, and Thar Baker.  Analysis of Dimensionality Reduction Techniques on Big data.  IEEE Access, 8:54776–54788, 2020.
  * [124] Dunja Mladenić.  Feature Selection for Dimensionality Reduction.  In International Statistical and Optimization Perspectives Workshop” Subspace, Latent Structure and Feature Selection”. Springer, 2005.
  * [125] Matthew Richardson, Amit Prakash, and Eric Brill.  Beyond PageRank: Machine Learning for Static Ranking.  In Proceedings of the International Conference on World Wide Web, 2006.
  * [126] David Cossock and Tong Zhang.  Subset Ranking Using Regression.  In International Conference on Computational Learning Theory, pages 605–619. Springer, 2006.
  * [127] Shivani Agarwal, Deepak Dugar, and Shiladitya Sengupta.  Ranking Chemical Structures for Drug Discovery: A New Machine Learning Approach.  Journal of Chemical Information and Modeling, 50(5):716–731, 2010\.
  * [128] Shai Shalev-Shwartz and Shai Ben-David.  Understanding Machine Learning: From Theory to Algorithms.  Cambridge University Press, 2014.
  * [129] Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  Overview of Supervised Learning.  In The Elements of Statistical Learning, pages 9–41. Springer, 2009\.
  * [130] Michael Schuster.  On supervised learning from sequential data with applications for speech recognition.  Nara Institute of Science and Technology (PhD dissertation), 1999\.
  * [131] Horace B Barlow.  Unsupervised Learning.  Neural computation, 1(3):295–311, 1989.
  * [132] Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  Unsupervised Learning.  In The Elements of Statistical Learning, pages 485–585. Springer, 2009.
  * [133] Xiaojin Zhu and Andrew B Goldberg.  Introduction to Semi-Supervised Learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1):1–130, 2009.
  * [134] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling.  Semi-Supervised Learning with Deep Generative Models.  In Advances in Neural Information Processing Systems, pages 3581–3589, 2014.
  * [135] Marco A Wiering and Martijn Van Otterlo.  Reinforcement Learning.  Adaptation, Learning, and Optimization, 12(3), 2012.
  * [136] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing Atari with Deep Reinforcement Learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [137] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun.  Learning Hierarchical Features for Scene Labeling.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1915–1929, 2012.
  * [138] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [139] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
  * [140] Li Deng, Geoffrey Hinton, and Brian Kingsbury.  New Types of Deep Neural Network Learning for Speech Recognition and Related Applications: An Overview.  In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8599–8603, 2013.
  * [141] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.  Deep Speech 2: End-to-End Speech Recognition in English and Mandarin.  In International Conference on Machine Learning, pages 173–182. PMLR, 2016.
  * [142] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep Speech: Scaling up end-to-end speech recognition.  arXiv preprint arXiv:1412.5567, 2014.
  * [143] Adam H Marblestone, Greg Wayne, and Konrad P Kording.  Toward an Integration of Deep Learning and Neuroscience.  Frontiers in Computational Neuroscience, 10:94, 2016.
  * [144] Zhenlong Yuan, Yongqiang Lu, Zhaoguo Wang, and Yibo Xue.  Droid-Sec: Deep Learning in Android Malware Detection.  In Proceedings of the 2014 ACM Conference on SIGCOMM, pages 371–372, 2014.
  * [145] Zhenlong Yuan, Yongqiang Lu, and Yibo Xue.  DroidDetector: Android Malware Characterization and Detection Using Deep Learning.  Tsinghua Science and Technology, 21(1):114–123, 2016.
  * [146] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep Learning.  Nature, 521(7553):436–444, 2015.
  * [147] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al.  Recent Advances in Convolutional Neural Networks.  Pattern Recognition, 77:354–377, 2018.
  * [148] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel.  Handwritten Digit Recognition with a Back-Propagation Network.  Advances in Neural Information Processing Systems, 2, 1989.
  * [149] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-Based Learning Applied to Document Recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [150] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  Advances in Neural Information Processing Systems, 25:1097–1105, 2012.
  * [151] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [152] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning.  In Thirty-first AAAI Conference on Artificial Intelligence, 2017\.
  * [153] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated Residual Transformations for Deep Neural Networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1492–1500, 2017.
  * [154] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely Connected Convolutional Networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700–4708, 2017.
  * [155] Csaba Szepesvári.  Algorithms for Reinforcement Learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1):1–103, 2010.
  * [156] Andrew Y Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang.  Autonomous Inverted Helicopter Flight via Reinforcement Learning.  In Experimental Robotics IX, pages 363–372. Springer, 2006.
  * [157] Nate Kohl and Peter Stone.  Policy Gradient Reinforcement Learning for Fast Quadrupedal Locomotion.  In IEEE International Conference on Robotics and Automation, volume 3, pages 2619–2624, 2004.
  * [158] S. Singh, D. Litman, M. Kearns, and M. Walker.  Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System.  Journal of Artificial Intelligence Research, 16:105–133, Feb 2002\.
  * [159] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.  Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates.  In IEEE International Conference on Robotics and Automation, pages 3389–3396. IEEE, 2017.
  * [160] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous Control with Deep Reinforcement Learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [161] Lei Tai, Giuseppe Paolo, and Ming Liu.  Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 31–36. IEEE, 2017.
  * [162] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller.  Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards.  arXiv preprint arXiv:1707.08817, 2017.
  * [163] Guillaume Lample and Devendra Singh Chaplot.  Playing FPS Games with Deep Reinforcement Learning.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * [164] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through Deep Reinforcement Learning.  Nature, 518(7540):529–533, 2015.
  * [165] Johannes Heinrich and David Silver.  Deep Reinforcement Learning from Self-Play in Imperfect-Information Games.  arXiv preprint arXiv:1603.01121, 2016.
  * [166] Zachary C Lipton, John Berkowitz, and Charles Elkan.  A Critical Review of Recurrent Neural Networks for Sequence Learning.  arXiv preprint arXiv:1506.00019, 2015.
  * [167] Haşim Sak, Andrew Senior, and Françoise Beaufays.  Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition.  arXiv preprint arXiv:1402.1128, 2014.
  * [168] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning Representations by Back-Propagating Errors.  Nature, 323(6088):533–536, 1986.
  * [169] Sepp Hochreiter and Jürgen Schmidhuber.  Long Short-Term Memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [170] Stefan Kombrink, Tomáš Mikolov, Martin Karafiát, and Lukáš Burget.  Recurrent Neural Network Based Language Modeling in Meeting Recognition.  In Twelfth Annual Conference of the International Speech Communication Association, 2011.
  * [171] Jen-Tzung Chien and Yuan-Chu Ku.  Bayesian Recurrent Neural Network for Language Modeling.  IEEE Transactions on Neural Networks and Learning Systems, 27(2):361–374, 2015.
  * [172] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney.  LSTM Neural Networks for Language Modeling.  In Annual Conference of the International Speech Communication Association, 2012.
  * [173] D Pawade, A Sakhapara, Mansi Jain, Neha Jain, and Krushi Gada.  Story Scrambler-Automatic Text Generation Using Word Level RNN-LSTM.  International Journal of Information Technology and Computer Science (IJITCS), 10(6):44–53, 2018.
  * [174] Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu.  Neural Text Generation: Past, Present and Beyond.  arXiv preprint arXiv:1803.07133, 2018.
  * [175] Lei Sha, Lili Mou, Tianyu Liu, Pascal Poupart, Sujian Li, Baobao Chang, and Zhifang Sui.  Order-Planning Neural Text Generation From Structured Data.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [176] Yajie Miao, Mohammad Gowayyed, and Florian Metze.  EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding.  In IEEE Workshop on Automatic Speech Recognition and Understanding, pages 167–174, 2015.
  * [177] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton.  Speech Recognition With Deep Recurrent Neural Networks.  In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6645–6649. IEEE, 2013.
  * [178] Jinyu Li, Rui Zhao, Hu Hu, and Yifan Gong.  Improving RNN Transducer Modeling for End-to-End Speech Recognition.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 114–121. IEEE, 2019.
  * [179] Ramesh Nallapati, Bing Xiang, and Bowen Zhou.  Sequence-to-sequence RNNs for text summarization. CoRR abs/1602.06023 (2016).  preprint arXiv:1602.06023, 2016.
  * [180] Shuming Ma, Xu Sun, Jingjing Xu, Houfeng Wang, Wenjie Li, and Qi Su.  Improving Semantic Relevance for Sequence-to-Sequence Learning of Chinese Social Media Text Summarization.  arXiv preprint arXiv:1706.02459, 2017.
  * [181] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al.  Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond.  preprint arXiv:1602.06023, 2016.
  * [182] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Nets.  Advances in Neural Information Processing Systems, 27, 2014.
  * [183] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath.  Generative adversarial networks: An overview.  IEEE Signal Processing Magazine, 35(1):53–65, 2018.
  * [184] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral Normalization for Generative Adversarial Networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [185] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [186] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved Techniques for Training GANs.  Advances in Neural Information Processing Systems, 29:2234–2242, 2016.
  * [187] Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos.  To Learn Image Super-Resolution, Use a GAN To Learn How To Do Image Degradation First.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 185–200, 2018.
  * [188] Pourya Shamsolmoali, Masoumeh Zareapoor, Ruili Wang, Deepak Kumar Jain, and Jie Yang.  G-GANISR: Gradual Generative Adversarial Network for Image Super Resolution.  Neurocomputing, 366:140–153, 2019.
  * [189] Weixiang Hong, Zhenzhen Wang, Ming Yang, and Junsong Yuan.  Conditional Generative Adversarial Network for Structured Domain Adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1335–1344, 2018.
  * [190] Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa.  Mastering Sketching: Adversarial Augmentation for Structured Prediction.  ACM Transactions on Graphics (TOG), 37(1):1–13, 2018.
  * [191] Junbo Zhao, Michael Mathieu, and Yann LeCun.  Energy-Based Generative Adversarial Network.  arXiv preprint arXiv:1609.03126, 2016.
  * [192] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine.  A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models.  arXiv preprint arXiv:1611.03852, 2016.
  * [193] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami.  Practical Black-Box Attacks Against Machine Learning.  In ACM Asia Conference on Computer and Communications Security, pages 506–519, 2017.
  * [194] Yuanzhang Li, Yaxiao Wang, Ye Wang, Lishan Ke, and Yu-an Tan.  A Feature-Vector Generative Adversarial Network for Evading PDF Malware Classifiers.  Information Sciences, 523:38–48, 2020.
  * [195] Jin-Young Kim, Seok-Jun Bu, and Sung-Bae Cho.  Malware Detection using Deep Transferred Generative Adversarial Networks.  In International Conference on Neural Information Processing, pages 556–564. Springer, 2017.
  * [196] Kevin J Liang, Chunyuan Li, Guoyin Wang, and Lawrence Carin.  Generative Adversarial Network Training is a Continual Learning Problem.  arXiv preprint arXiv:1811.11083, 2018.
  * [197] Martin Arjovsky and Léon Bottou.  Towards Principled Methods for Training Generative Adversarial Networks.  arXiv preprint arXiv:1701.04862, 2017.
  * [198] Zhaoqing Pan, Weijie Yu, Xiaokai Yi, Asifullah Khan, Feng Yuan, and Yuhui Zheng.  Recent Progress on Generative Adversarial Networks (GANs): A Survey.  IEEE Access, 7:36322–36333, 2019.
  * [199] Animesh Karnewar, Oliver Wang, and Raghu Sesha Iyengar.  MSG-GAN: Multi-Scale Gradient GAN for Stable Image Synthesis.  ArXiv, abs/1903.06048, 2019.
  * [200] Kexugit.  Inside Windows: Win32 Portable Executable File Format in Detail.  https://docs.microsoft.com/en-us/archive/msdn-magazine/2002/february/inside-windows-win32-portable-executable-file-format-in-detail.
  * [201] Portable Executable File Format.  https://blog.kowalczyk.info/articles/pefileformat.html.
  * [202] https://ivanlef0u.fr/repo/madchat/vxdevl/papers/winsys/pefile/pefile.html.
  * [203] Karl-Bridge-Microsoft.  PE Format - Win32 apps.  https://docs.microsoft.com/en-us/windows/win32/debug/pe-format.
  * [204] Heng Li, ShiYao Zhou, Wei Yuan, Jiahuan Li, and Henry Leung.  Adversarial-Example Attacks Toward Android Malware Detection System.  IEEE Systems Journal, 14(1):653–656, 2019.
  * [205] Ishai Rosenberg, Asaf Shabtai, Lior Rokach, and Yuval Elovici.  Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers.  In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 490–510. Springer, 2018.
  * [206] Maryam Shahpasand, Len Hamey, Dinusha Vatsalan, and Minhui Xue.  Adversarial Attacks on Mobile Malware Detection.  In 2019 IEEE 1st International Workshop on Artificial Intelligence for Mobile (AI4Mobile), pages 17–20. IEEE, 2019.
  * [207] Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro.  Intriguing Properties of Adversarial ML Attacks in the Problem Space.  In 2020 IEEE Symposium on Security and Privacy (SP), pages 1332–1349. IEEE, 2020.
  * [208] App Manifest Overview : Android Developers.  https://developer.android.com/guide/topics/manifest/manifest-intro.
  * [209] Ajin Asokan.  APK File Contents - In-Depth Explanation.  https://ajinasokan.com/posts/apk-file-postmortem/, Sep 2016.
  * [210] Wojtek Kaliciński.  SmallerAPK, Part 1: Anatomy of an APK.  https://medium.com/androiddevelopers/smallerapk-part-1-anatomy-of-an-apk-da83c25e7003, May 2016.
  * [211] PDF File Structure - Four Main Parts: The Header, Body, Cross-Reference Table and Trailer.  http://www.simpopdf.com/resource/pdf-file-structure.html.
  * [212] Cybercrimemag.  Cybercrime To Cost The World 10.5 Trillion Annually By 2025\.  https://cybersecurityventures.com/hackerpocalypse-cybercrime-report-2016/#:~:text=A2017reportfromCybersecurity,figureroseto\$11.5billion., Apr 2021.
  * [213] Alireza Souri and Rahil Hosseini.  A state-of-the-art survey of malware detection approaches using data mining techniques.  Human-centric Computing and Information Sciences, 8(1):1–22, 2018\.
  * [214] Hao Sun, Xiaofeng Wang, Rajkumar Buyya, and Jinshu Su.  CloudEyes: Cloud-based Malware Detection with Reversible Sketch for Resource-constrained Internet of Things (IoT) Devices.  Software: Practice and Experience, 47(3):421–441, 2017.
  * [215] Ping Wang and Yu-Shih Wang.  Malware behavioural detection and vaccine development by using a support vector model classifier.  Journal of Computer and System Sciences, 81(6), 2015.
  * [216] James B Fraley and Marco Figueroa.  Polymorphic Malware Detection Using Topological Feature Extraction with Data Mining.  In SoutheastCon 2016, pages 1–7. IEEE, 2016.
  * [217] Yong Tang, Bin Xiao, and Xicheng Lu.  Signature Tree Generation for Polymorphic Worms.  IEEE Transactions on Computers, 60(4):565–579, 2010.
  * [218] J Zico Kolter and Marcus A Maloof.  Learning to Detect and Classify Malicious Executables in the Wild.  Journal of Machine Learning Research, 7(12), 2006.
  * [219] Boyun Zhang, Jianping Yin, Jingbo Hao, Dingxing Zhang, and Shulin Wang.  Malicious Codes Detection Based on Ensemble Learning.  In International Conference on Autonomic and Trusted Computing, pages 468–477. Springer, 2007.
  * [220] Daniel Bilar.  Opcodes as Predictor for Malware.  International Journal of Electronic Security and Digital Forensics, 1(2), 2007.
  * [221] Weijie Han, Jingfeng Xue, Yong Wang, Lu Huang, Zixiao Kong, and Limin Mao.  MalDAE: Detecting and Explaining Malware Based on Correlation and Fusion of Static and Dynamic Characteristics.  Computers & Security, 83:208–233, 2019.
  * [222] William Fleshman, Edward Raff, Richard Zak, Mark McLean, and Charles Nicholas.  Static Malware Detection & Subterfuge: Quantifying the Robustness of Machine Learning and Current Anti-virus.  In 2018 13th International Conference on Malicious and Unwanted Software (MALWARE), pages 1–10. IEEE, 2018.
  * [223] Andreas Moser, Christopher Kruegel, and Engin Kirda.  Limits of Static Analysis for Malware Detection.  In Annual Computer Security Applications Conference, pages 421–430. IEEE, 2007.
  * [224] Brad Miller, Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Rekha Bachwani, Riyaz Faizullabhoy, Ling Huang, Vaishaal Shankar, Tony Wu, George Yiu, et al.  Reviewer Integration and Performance Measurement for Malware Detection.  In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pages 122–141. Springer, 2016.
  * [225] Blake Anderson, Daniel Quist, Joshua Neil, Curtis Storlie, and Terran Lane.  Graph-based Malware Detection Using Dynamic Analysis.  Journal in Computer Virology, 7(4):247–258, 2011.
  * [226] Wen-Chieh Wu and Shih-Hao Hung.  DroidDolphin: a Dynamic Android Malware Detection Framework Using Big Data and Machine Learning.  In Proceedings of the 2014 Conference on Research in Adaptive and Convergent Systems, pages 247–252, 2014.
  * [227] Pengbin Feng, Jianfeng Ma, Cong Sun, Xinpeng Xu, and Yuwan Ma.  A Novel Dynamic Android Malware Detection System With Ensemble Learning.  IEEE Access, 6:30996–31011, 2018.
  * [228] Carsten Willems, Thorsten Holz, and Felix Freiling.  Toward Automated Dynamic Malware Analysis Using CWSandbox.  IEEE Security & Privacy, 5(2):32–39, 2007.
  * [229] Muhammad Ijaz, Muhammad Hanif Durad, and Maliha Ismail.  Static and Dynamic Malware Analysis Using Machine Learning.  In 2019 16th International Bhurban Conference on Applied Sciences and Technology (IBCAST), pages 687–691. IEEE, 2019.
  * [230] Hengli Zhao, Ming Xu, Ning Zheng, Jingjing Yao, and Qiang Ho.  Malicious Executables Classification Based on Behavioral Factor Analysis.  In 2010 International Conference on e-Education, e-Business, e-Management and e-Learning, pages 502–506. IEEE, 2010.
  * [231] Ronghua Tian, Rafiqul Islam, Lynn Batten, and Steve Versteeg.  Differentiating Malware from Cleanware Using Behavioural Analysis.  In 2010 5th International Conference on Malicious and Unwanted Software, pages 23–30. IEEE, 2010.
  * [232] Martina Lindorfer, Clemens Kolbitsch, and Paolo Milani Comparetti.  Detecting Environment-Sensitive Malware.  In International Workshop on Recent Advances in Intrusion Detection, pages 338–357. Springer, 2011.
  * [233] Zacharias Tzermias, Giorgos Sykiotakis, Michalis Polychronakis, and Evangelos P Markatos.  Combining Static and Dynamic Analysis for the Detection of Malicious Documents.  In Proceedings of the Fourth European Workshop on System Security, pages 1–6, 2011.
  * [234] PV Shijo and AJPCS Salim.  Integrated Static and Dynamic Analysis for Malware Detection.  Procedia Computer Sci., 46, 2015.
  * [235] Michael Spreitzenbarth, Thomas Schreck, Florian Echtler, Daniel Arp, and Johannes Hoffmann.  Mobile-Sandbox: Combining Static and Dynamic Analysis with Machine-Learning Techniques.  International Journal of Information Security, 14(2):141–153, 2015\.
  * [236] Sanjeev Das, Yang Liu, Wei Zhang, and Mahintham Chandramohan.  Semantics-Based Online Malware Detection: Towards Efficient Real-Time Protection Against Malware.  IEEE Transactions on Information Forensics and Security, 11(2):289–302, 2015.
  * [237] Meltem Ozsoy, Caleb Donovick, Iakov Gorelik, Nael Abu-Ghazaleh, and Dmitry Ponomarev.  Malware-Aware Processors: A Framework for Efficient Online Malware Detection.  In 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), pages 651–661. IEEE, 2015.
  * [238] John Demme, Matthew Maycock, Jared Schmitz, Adrian Tang, Adam Waksman, Simha Sethumadhavan, and Salvatore Stolfo.  On the Feasibility of Online Malware Detection with Performance Counters.  ACM SIGARCH Computer Architecture News, 2013.
  * [239] Mahmoud Abdelsalam, Ram Krishnan, and Ravi Sandhu.  Online Malware Detection in Cloud Auto-scaling Systems Using Shallow Convolutional Neural Networks.  In IFIP Annual Conference on Data and Applications Security and Privacy. Springer, 2019.
  * [240] Weiwei Hu and Ying Tan.  Black-Box Attacks against RNN based Malware Detection Algorithms.  In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  * [241] BS Vivek, Konda Reddy Mopuri, and R Venkatesh Babu.  Gray-box Adversarial Training.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 203–218, 2018.
  * [242] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay.  Adversarial Attacks and Defences: A Survey.  arXiv preprint arXiv:1810.00069, 2018.
  * [243] Battista Biggio, Giorgio Fumera, and Fabio Roli.  Security Evaluation of Pattern Classifiers under Attack.  IEEE Transactions on Knowledge and Data Engineering, 26(4):984–996, Apr 2014.
  * [244] Yizhen Wang and Kamalika Chaudhuri.  Data Poisoning Attacks against Online Learning.  arXiv preprint arXiv:1808.08994, 2018.
  * [245] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning.  arXiv preprint arXiv:1712.05526, 2017.
  * [246] Pang Wei Koh, Jacob Steinhardt, and Percy Liang.  Stronger Data Poisoning Attacks Break Data Sanitization Defenses.  arXiv preprint arXiv:1811.00741, 2018.
  * [247] Weilin Xu, Yanjun Qi, and David Evans.  Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers.  In Annual Network and Distributed System Security Symposium, 2016\.
  * [248] Junkun Yuan, Shaofang Zhou, Lanfen Lin, Feng Wang, and Jia Cui.  Black-Box Adversarial Attacks Against Deep Learning Based Malware Binaries Detection with GAN.  In ECAI 2020, pages 2536–2542. IOS Press, 2020.
  * [249] Nuno Martins, José Magalhães Cruz, Tiago Cruz, and Pedro Henriques Abreu.  Adversarial Machine Learning Applied to Intrusion and Malware Scenarios: A Systematic Review.  IEEE Access, 8, 2020.
  * [250] Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout Networks.  In International Conference on Machine Learning, pages 1319–1327. PMLR, 2013.
  * [251] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami.  The Limitations of Deep Learning in Adversarial Settings.  In IEEE European Symposium on Security and Privacy, pages 372–387, 2016.
  * [252] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [253] Jean M Steppe and Kenneth W Bauer Jr.  Feature Saliency Measures.  Computers & Mathematics with Applications, 33(8):109–126, 1997\.
  * [254] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning Deep Features for Discriminative Localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921–2929, 2016.