# Constrained Adversarial Learning and its applicability to Automated Software
Testing: a systematic review

João Vitorino<EMAIL_ADDRESS>Tiago Dias<EMAIL_ADDRESS>Tiago Fonseca
<EMAIL_ADDRESS>Eva Maia<EMAIL_ADDRESS>Isabel Praça<EMAIL_ADDRESS>School of Engineering, Polytechnic of Porto (ISEP/IPP), 4249-015, Porto,
Portugal  Research Group on Intelligent Engineering and Computing for Advanced
Innovation and Development (GECAD), 4249-015, Porto, Portugal

###### Abstract

Every novel technology adds hidden vulnerabilities ready to be exploited by a
growing number of cyber-attacks. Automated software testing can be a promising
solution to quickly analyze thousands of lines of code by generating and
slightly modifying function-specific testing data to encounter a multitude of
vulnerabilities and attack vectors. This process draws similarities to the
constrained adversarial examples generated by adversarial learning methods, so
there could be significant benefits to the integration of these methods in
automated testing tools. Therefore, this systematic review is focused on the
current state-of-the-art of constrained data generation methods applied for
adversarial learning and software testing, aiming to guide researchers and
developers to enhance testing tools with adversarial learning methods and
improve the resilience and robustness of their digital systems. The found
constrained data generation applications for adversarial machine learning were
systematized, and the advantages and limitations of approaches specific for
software testing were thoroughly analyzed, identifying research gaps and
opportunities to improve testing tools with adversarial attack methods.

###### keywords:

software development , constrained data generation , adversarial attacks ,
machine learning

††journal: Information and Software Technology

##  1 Introduction

From decision-making processes to electronic commerce and a variety of online
services, the dependency of modern organizations on digital systems and
computer programs is not slowing down soon [1]. However, no computer code is
flawless, so the integration of different technologies and software components
can add hidden vulnerabilities ready to be exploited in novel cyber-attacks.
Not even systems that rely on artificial intelligence are secure, due to the
susceptibility of machine learning models to attacks with adversarial
examples. As organizations rush the deployment of computer applications
without thoroughly testing them, they are increasing the risk of an attacker
with malicious intents disrupting their critical business processes and
impacting their business continuity [2].

To safeguard organizations and their personnel from the damage caused by an
attack, it is imperative to integrate security best practices in software
development workflows and ensure that all computer applications are adequately
tested before being deployed. Nonetheless, despite growing concerns for
software testing, it is still mostly a manual process where developers design
the tests that each distinct part of the code must be validated against. This
process is time-consuming, expensive, and most of the times incomplete,
because humans cannot formulate tests for the entire attack surface of a
digital system nor tackle all possible attack vectors [3].

Automated software testing is a very promising research field because an
automated and methodical approach can address the attack vectors that would
otherwise be left unnoticed. By quickly analyzing thousands of lines of code,
both time and monetary costs can be reduced. Nonetheless, to perform
trustworthy tests and ensure that a program is functioning correctly, the
generated testing data must address the different constraints of the tested
functions and software components. So, in short, an automated testing tool
slightly modifies the parameters of a function to check if it deviates from
the expected behaviour [4]. In another pertinent research field, adversarial
machine learning, a similarity can be noticed: an adversarial attack method
creates slight modifications in data features to deceive a model, causing
misclassifications and unexpected behaviors [5]. Therefore, both automated
testing tools and adversarial learning methods serve similar purposes and
there could be significant benefits to their integration.

Due to the lack of publications addressing the use of adversarial learning
methods to improve software testing tools, with data perturbations generated
according to the specific constraints of the tested functions and software
components, this work investigates the recent scientific developments and
technological advances of these fields. The main research question of this
systematic review was: ”How can automated software testing tools be improved
with constrained adversarial machine learning methods?”. The main question was
divided into two narrower sub-questions:

RQ1:

How can adversarial learning methods be leveraged for constrained data
generation?

RQ2:

How can automated software testing tools benefit from constrained data
generation?

By investigating the current approaches in a systematic manner, this work can
guide researchers and developers to enhance automated testing tools with
adversarial learning methods and improve the resilience and robustness of the
System Under Test (SUT). This paper is organized into multiple sections.
Section 2 describes the employed research methodology. Section 3 and Section 4
present and discuss the found publications for RQ1 and RQ2. Finally, Section 5
presents the concluding remarks and future research topics.

##  2 Research Methodology

This section describes the methodical review process employed for both RQ1 and
RQ2. To achieve a transparent, replicable, and complete systematic review, the
PRISMA methodology [6] was followed. Search terms were created to be used in
reputable bibliographic databases, and several inclusion and exclusion
criteria were defined. After an initial screening phase with the titles and
abstracts of the found publications, their full texts were assessed for
eligibility, and only then they were included in the review.

###  2.1 Search Terms

Search terms were chosen after a careful initial analysis of the literature.
Since constrained data generation is a relatively unexplored research topic
without standardized keywords, the search had to cover broader terms to
prevent narrowing it down too much and obtain relevant publications for the
scope of the review. Therefore, some word variations were also considered to
widen the search to more common concepts like `number generators` and
`conditional generators`, as well as approaches that address `constraints` and
`restrictions`, which are word variations represented by ”*”.

Considering that RQ1 and RQ2 share a common scope, constrained data
generation, only their target field was different in the search queries. For
RQ1, the query included the `adversarial` term to focus on approaches related
to adversarial machine learning and adversarial perturbations (see Table
LABEL:tab:queryrq1). For RQ2, the target field was `software testing` to
analyze the publications specific to this field (see Table
LABEL:tab:queryrq2). The terms of each scope were aggregated with AND
operators.

Table 1: Search query for RQ1. Scope |  Keywords
---|---
Constrained |  constrained* OR conditional* OR restricted*
Field |  ”adversarial”
Data |  data OR sample OR example OR number
Generation |  generation OR generator
Table 2: Search query for RQ2. Scope |  Keywords
---|---
Constrained |  constrained* OR conditional* OR restricted*
Field |  ”software testing”
Data |  data OR sample OR example OR number
Generation |  generation OR generator

###  2.2 Search Sources

The primary search source was Science Direct [7], which is a large
bibliographic database of scientific journals and conference proceedings
provided by the internationally recognized publisher Elsevier. Due to their
acknowledged relevance for scientific literature of software engineering,
computing, and information technology, the search also included the digital
libraries of the Association for Computing Machinery (ACM) [8], the Institute
of Electrical and Electronics Engineers (IEEE) [9], and the Multidisciplinary
Digital Publishing Institute (MDPI) [10].

###  2.3 Inclusion and Exclusion Criteria

The review was performed in the beginning of 2023 and was focused on peer-
reviewed conference papers and journal articles introducing or applying
constrained generation approaches, in the English language. Considering that
both adversarial learning and software testing are active areas of research
and had relevant technological advances in recent years, the selected time
frame was 2017 to 2022, the latest 6 whole years of scientific developments.
To limit the findings to publications stored in these acknowledged databases
and in the intended time frame, backward snowballing was not performed.

The widened search queries led to more relevant publications within this time
frame, but also to many publications that were not aligned with the purpose of
this review. To screen the found publications and assess their eligibility,
several exclusion criteria were defined to remove general surveys and reviews
that did not specify concrete approaches, publications that mentioned
constraints or conditions but not during their data generation processes, and
publications that did not detail how the utilized constraints were applied nor
how they affected the data generation. Table 1 provides an overview of the
defined inclusion and exclusion criteria for both RQ1 and RQ2.

Table 3: Inclusion and exclusion criteria. Inclusion Criteria |  Exclusion Criteria
---|---
IC1 Peer-reviewed journal article or conference paper IC2 Published from 2017 to 2022 IC3 Available in the English language IC4 Introduced or applied an approach for constrained data generation |  EC1 Duplicated publication EC2 Survey or review EC3 Constraints not in generation EC4 Constraints not detailed EC5 Full text not available

##  3 Adversarial Machine Learning

This section presents and discusses the findings of RQ1, systematizing the
constrained data generation applications for adversarial machine learning. A
total of 1757 records were initially obtained by applying the query to the
contents of the publications stored in the selected databases. After the
screening and eligibility assessment phases, 94 publications were included in
the review. This process is detailed in Figure 1.

Figure 1: PRISMA search process for RQ1.

The included publications were systematized to identify the current
applications of constrained data generation and the methods they are based on.
Since some addressed multiple tasks, a total of 98 applications were
identified across 11 sectors and industries, although a few were more general
and not specific to a single industry. The number of applications grew from 2
in 2017 to 34 in 2022, with the cybersecurity and healthcare sectors being the
most prominent and having more diverse applications, followed by the aerospace
and energy sectors. Table LABEL:tab:adv summarizes the reviewed applications
and the methods they were based on. These base methods were either explicitly
built upon or their concepts were implicitly used in new implementations.

Table 4: Applications of constrained data generation. Sector / Industry |  Applications |  Base Methods
---|---|---
Aerospace |  Synthetic-Aperture Radar (SAR) image reconstruction, and hyperspectral classification [11, 12, 13, 14] |  CGAN, CVAE
Satellite mapping of surface water, road surface area, reflectance, and radiance [15, 16, 17, 18] |  CGAN
Precipitation estimation, aircraft detection, and remote sensing image segmentation [19, 20, 21] |  CGAN
Object detection in unmanned drone inspection with Unmanned Aerial Vehicles (UAVs) [22, 23] |  CGAN
Automotive |  Urban traffic trajectory and parking occupancy estimation [24, 25] |  CGAN, CVAE
Vehicle transmission gear and Inertial Measurement Unit (IMU) signal analysis [26, 27] |  CGAN
Ultrasonic signal sensors for autonomous driving systems [28] |  CGAN
Vehicular and transportation network simulation [29, 30] |  CGAN
Chemical |  Chemical production process fault detection and diagnosis [31, 32] |  CGAN, WGAN
Soft sensors for multiple chemical compound analysis [33] |  CGAN, WGAN
Extraction of environmental features from aerosol optical depth sensors [34] |  CGAN
Cybersecurity |  Network intrusion and anomaly detection, and cyber-attack classification [35, 36, 37, 38, 39] |  CGAN, evolutionary algorithms
Illegal webpage and malicious domain name detection [36, 40, 41] |  CGAN, evolutionary algorithms
Software testing and malware detection in code blocks [42, 43, 44] |  CGAN, fuzzy algorithms
Fake user detection and behavior generation for recommendation systems [45, 46] |  CGAN
Speech recognition and denoising, and voice spoofing detection [47, 48, 49, 50] |  CGAN
Data masking, obfuscation, anonymization, and compression [51, 52, 53] |  CGAN
Economy |  Monte Carlo simulation design for economic studies [54] |  CGAN, WGAN, econometric algorithms
Energy |  Power distribution and economic load dispatch simulation for electrical grids [30, 55, 56] |  CGAN, graph algorithms
Anomalous energy consumption and electricity theft detection [57, 58] |  CVAE, WGAN
Charging behavior and electric load forecasting [59, 60, 61, 62] |  CGAN
Geological |  Seismic data denoising and interpolation [63] |  CGAN
Virtual landscape and terrain authoring [64, 65] |  CGAN
Healthcare |  Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) analysis [66, 67, 68, 69, 70] |  CGAN, WGAN
Electrocardiogram (ECG) and Electroencephalogram (EEG) signal analysis [71, 72, 73] |  CGAN, WGAN
Electrodermal Activity (EDA) generation for stress detection [74] |  CGAN
Mammographic density segmentation and breast cancer diagnosis [75, 76, 77] |  CGAN
Lung and skin lesions detection [78] |  CGAN
Microscopic blood smear test [79] |  CGAN
Vocal cord and voice disorder detection [80] |  CGAN
Connectomes segmentation for reconstruction of neural circuits [81] |  CGAN
Mechanical |  Rotating machinery and rolling bearing fault diagnosis [82, 83, 84, 85, 86, 87] |  CGAN, CVAE, WGAN
Industrial equipment failure and conveyor belt damage detection [88, 89, 90] |  CGAN, CVAE, WGAN
Wind turbine gearbox fault diagnosis [91, 92, 93] |  CGAN, CVAE
Music |  Variable-length music and audio generation [94, 95] |  CGAN, CVAE
Cross-modal musical performance generation [96] |  CGAN
Water |  Water flow simulation for water supply and distribution systems [30, 56] |  CGAN, graph algorithms
General |  Motion, stretches, and compressions simulation [97, 98] |  CGAN, WGAN, skinning algorithms
Picture generation according to age and facial attributes [99, 100, 101] |  CGAN
Wireless indoor positioning and tracking [102] |  CGAN
Handwritten text and digit recognition [103, 104] |  CGAN

The increasing use of constraints across various sectors and industries
highlights that better data can be generated when the specificities of the
task at hand are considered. Nonetheless, since most publications handle image
classification datasets, they employ methods with simple constraints to
generate new synthetic images according to the classes of a dataset. Most
approaches are based on improved versions of the Generative Adversarial
Network (GAN) [105] and the Variational Autoencoder (VAE) [106], such as
Conditional GAN (CGAN) [107], Wasserstein GAN (WGAN) [108], and Conditional
VAE (CVAE) [109].

Even though these improved versions of deep learning algorithms spread across
the literature because they can learn to generate different data for different
classes, the samples of each class are freely generated without addressing any
complex constraint. For more complex tasks in sectors that require other data
types like tabular data and time series, some approaches are starting to rely
on methods that support more rigorous configurations and more complex
constraints. For instance, some researchers employ evolutionary computation
with swarm intelligence and custom-built Genetic Algorithms (GAs) [35], and
others apply fuzzy logic in their algorithms to deal with the uncertainty of
the information [42].

Despite the wide range of applications of constrained data generation, the few
approaches that could potentially be transferable to automated software
testing tools were mainly developed for cybersecurity applications related to
communication networks and cyber-physical applications in energy and water
networks. Due to the low-quality data that conventional methods would generate
in these complex domains, some researchers encapsulated the data generation
processes in mechanisms that enforce task-specific constraints.

In [56], the authors intended to simulate electrical grids and water supply
and distribution systems. Each action or event in one part of these cyber-
physical systems would affect the entire energy and water networks, so the
presence of a given value at a given feature would restrict the values that
other features could have. Linear equality constraints were defined to fulfill
the physical capacity requirements of each network and multiple simulations
were performed to validate them. This optimization approach could be useful to
model the relationships between different nodes of a network and different
software components, but the constraints would need to be carefully redesigned
for each minor change in the tested functions or to be transferred to
different SUT architectures.

To provide a structure that could be adapted to changes, in [30], energy and
water networks were addressed through graph theory. The graph structure
provided a more rigorous representation of the electric and water flow physics
between different nodes in a network, enabling the simulation of cyclic trends
and acyclic congestions, as well as their effects in other nodes. Nodes and
their connections could be added or removed to adapt the structure to
different flow networks, although it was tailored to the specific physical
constraints of those domains. Additionally, it is also relevant to highlight
the mechanism introduced in [98] to perform physically accurate simulations of
stretches and compressions of objects in three dimensional spaces. Despite not
being as rigorous as flow graphs and only being tested with triangulated cloth
meshes of human body poses and joint rotations, this mechanism could be
transferable to the perturbation of images and objects provided to the SUT and
test its complex code structure.

Regarding biomedical images, in [78], the authors addressed the insufficient
label granularity of most datasets. Since multiple sub-types of a disease can
be aggregated into a single broader label, the data samples of a class can
exhibit entirely different data distributions and feature correlations. Even
if a CGAN is used to distinguish between different classes of a dataset, the
generated data may mix the characteristics of multiple sub-types and therefore
may not correspond to an actual disease. Conditioning vectors were used to
restrict the modification of relevant disease features according to the values
exhibited by other similar samples that were presumed to be of the same sub-
type. Even though this mechanism is still based on a CGAN and would need to be
redesigned for other data types, the concept of configuring class-specific
value constraints could be useful to improve data quality and generate
relevant parameter combinations in a reduced amount of time.

These sub-classes were further explored in [38], where the authors established
that an adversarial example must be valid within its domain structure, which
corresponds to the input of the software component being tested, and be
coherent with the characteristics and purposes of each class. Constrained data
perturbations were generated by analysing the value intervals of individual
features and the value combinations of groups of features. Despite being
developed for network intrusion detection, the method created constrained
adversarial examples that fulfilled complex constraints. This method could be
useful to software testing because the tested functions could be aggregated
into classes according to their functionalities and function-specific value
perturbations could be performed, quickly generating more adequate parameter
combinations to achieve a high code coverage.

Overall, the reviewed adversarial machine learning approaches present valuable
insights for automated software testing applications, although they are
tailored to the specificities of cyber-physical systems. These mechanisms
could be redesigned to consider the value constraints of the parameters of
each tested function and generate high-quality data to test different software
components. Therefore, it is also pertinent to explore the recent advances in
automated software testing tools and analyze the ways that constrained data
generation could be used to improve them.

##  4 Automated Software Testing

This section presents and discusses the findings of RQ2, thoroughly analyzing
the advantages and limitations of approaches specific for software testing. A
total of 325 records were identified in the selected databases. After the
screening of their titles and abstracts, and the assessment of their full
texts for eligibility, 17 publications were included in the review. This
process is detailed in Figure 2.

Figure 2: PRISMA search process for RQ2.

Software testing approaches can be divided into a three-fold: (i) white-box,
(ii) black-box, and (iii) grey-box. The first is mostly executed from a
developer point of view, where all the domain constraints and system’s inner
workings are known and leveraged. The black-box approach is characterized for
its lack of prior knowledge [110]. In a black-box testing setting, the tester
has no information regarding the specificities of the SUT [110]. Lastly, grey-
box testing combines both approaches. However, the implementation of the SUT
and its domain is only partially known [111].

Software testing automation can be extremely challenging, depending on the
approach and levels of testing considered. The key aspect to achieve
automation relies on the construction of the test cases, which are a
reflection of the SUT’s functions parameter values and their execution
sequence [112]. When considering the generation of input values to test the
system, one must deal with the difficulties related with finding the right
input values along with their combination and the right execution sequence,
since the prior execution of system functionality may alter its state, which
leads to different system execution paths [112]. Depending on the information
available to the automated testing tool these can have enormous search spaces,
which is not ideal when considering the possible execution time [113].
Depending on the software and the levels of testing which the automated tool
is trying to achieve, a white-box automated testing tool with access to more
domain information might perform better and more efficient testing of the
SUT’s domain. Nonetheless, testing a system from a user perspective using
black-box techniques may sometimes raise different errors that are not usually
expected nor visually understandable in the source code [114].

Considering the different domain constraints that different systems have, the
conducted systematic review has shown that although random generation of data
is capable of covering most test paths of a system, it is highly unpractical.
There are many dependencies in each method and so the combination of
parameters becomes extremely complex. Moreover, the literature shows that in a
black-box context, the input space for certain data types is continuous, and
therefore different techniques such as providing boundary values[115, 116,
117], leveraging system feedback [115, 118, 119] and applying search
techniques can reduce complexity [120]. In a grey-box setting, the authors
have also attempted to reduce the input space by resorting to specifications
which reflect source code knowledge to constraint the input [121] and
techniques for quick exploration of the input space [122]. In a white-box
setting, automated testing can be interpreted as a search problem, with most
methods focusing on metaheuristics and their optimization to find more diverse
test cases [123, 124, 125, 126, 127, 128]. One publication has attempted to
use GANs for software testing [43], but like many white-box approaches, it
only focuses on numerical input generation, which may be insufficient to fully
test modern software and digital systems.

The gathered publications were analyzed to identify the test scope, the main
input generation methods, the covered data types and the utilized data
sources. This information is summarized in Table LABEL:tab:soft. The keyword
”All” refers to numerical, textual, byte, boolean, and composite data types.

Table 5: Characteristics of constrained testing methods. Test Scope |  Method |  Data Types |  Data Sources
---|---|---|---
Black-box |  Custom Data Generators [115] |  All |  OpenAPI specifications and public knowledge bases
Input Distance [120] |  Numerical |  Navigational model
Metamorphic and Search-based Testing [116] |  All |  Input/Output data and API configuration
Custom Mutators [117] |  All |  Faulty real user input
Test By Committee [118] |  All |  Custom specifications
Reinforcement Learning [119] |  Numerical |  Random data
Grey-box |  Pairwise hybrid Artificial Bee Colony [121] |  Numerical |  Constraint specifications
Locally Spreading, Points Algorithm and Adaptive Random Test [122] |  Numerical |  Defect-free confidence interval
White-box |  Genetic Algorithm and Reinforcement Learning [123] |  Numerical |  Control flow graph
Genetic Algorithm [124, 125, 126] |  Numerical |  Random data
Particle Swarm Optimization [127] |  Numerical |  Control flow graph
D-Algorithm [129] |  Numerical |  Control flow graph and define-use chains
Multi-objective Optimization Algorithm [130] |  Numerical |  Random data
Coevolutionary Genetic Algorithm [128] |  Numerical |  Random data
Generative Adversarial Networks [43] |  Numerical |  Gcov file

Regarding the methods for input generation from a black-box testing
perspective, since there is no knowledge regarding the SUT, authors have found
ways of constraining the input generation using other available data. For
instance, Martin-Lopez et al. [115] decided to test a Representational State
Transfer (REST) Application Programming Interface (API). The authors were able
to produce constrained input via three different methods: (i) OpenAPI
Specifications (OAS), which establishes the baseline information to interact
with an API, and may include request examples, (ii) custom data generators
which produce constrained input, and (iii) public knowledge bases. Similarly,
in [117], the authors propose using the data collected from a public API and
mutate faulty input to produce valuable test cases. Even though both
approaches are capable of diminishing the input space and producing realistic
test data, they rely on publicly available data, which might not be sufficient
to cover the entirety of the SUT test paths.

Recent works have also considered the use of model-based approaches to solve
the search problem by producing a model of the SUT capable of generating
constrained input. For instance, Martin-Lopez [116] resorts to three different
methods for testing web APIs: (i) search-based methods, (ii) mutating API JSON
input and output, and (iii) computing metamorphic testing. These methods
naturally preserve the quality of the initial data, since they don’t tend to
diverge from the initial input. The author’s goal is to be able to have
artificial intelligence ”bots” capable of learning the workings of the SUT and
generating constrained test input. Walkinshaw et al. in [118] propose a method
that combines Learning-based Testing (LBT) and Query By Committee (QBC). QBC
is used in this context to circumvent LBT’s dependence between model inference
algorithm and the test-generation algorithm. This mechanism allows LBT to
select inputs based on the combined uncertainty of the models, inferring a
behavioural model of the SUT and selecting the test that it is least certain
about. Additionally, Kim et al. [119] reinterpreted this search problem as a
reinforcement learning one. Despite achieving 100% branch coverage during
testing, for unforeseen arbitrary functions it only achieved 60.06% branch
coverage. Moreover, the algorithm only considers numeric input which depending
on the SUT may render this approach unpractical for software testing. These
approaches are as good as the amount of knowledge one has of the SUT, but as
modern software and digital systems grow in size and complexity, so do the
domain constraints and the number of branches that should be included in the
configurations.

In [120], the authors produce a navigational model of a web application by
crawling it to discover possible test cases. The presented method’s first test
is generated randomly and is added to a set of executed tests. The subsequent
tests and concrete input vectors are selected depending on the distance
between each candidate and the current set of executed test cases, where only
the farthest case is computed. Their goal is to generate a set of test cases
that diversify the coverage of the navigational graph. Even though they
present a method that does not rely on exhaustive testing of the API that
requires in-browser executions, they still rely on random input which dictates
how long the system will take to find the next valuable input data to cover
different test paths, since it is calculated by the distance formula. From a
grey-box perspective, Alazzawi et al. [121] also tackle the same problem using
a Pairwise hybrid Artificial Bee Colony algorithm, which takes random input
and a configuration file with the constraints to find the most diversified
test cases. However, depending on the manual configuration, the test cases can
lack quality and diversity.

In a grey-box setting, Huang et al. [122] tackle the broad input space and
constraint it by resorting to the combination of a Locally Spreading Points
algorithm along with a code defect-free confidence interval. The algorithm
receives multiple generated test cases and evolves them to improve the minimum
distance between points, achieving a better spread set of test cases. The
approach improves effectiveness when used as an add-on to Adaptive Random
Testing (ART). However, the method requires manual labeling of the test cases,
regarding the confidence interval.

Even though black-box and grey-box methods can be quite useful to test a
system regardless of its characteristics, the findings show that most research
works attempt to automate software testing from a white-box perspective
interpreting input generation as a search-based problem and resorting to
metaheuristics to perform the search. As such, to constraint the test cases,
the authors in [123, 124, 125, 126] resort to the GA. Esnaashari et al. [123]
uses the GA to solve the search-based along with reinforcement learning as a
local search step within the GA. Their work is capable of achieving results
faster, but it does not produce better coverage than other methods. In
particular, [124] and [125] focus on improving the genetic operators. [125]
improves: (i) selection by resorting to symbolic execution technique to select
test cases with more useful heuristic information, (ii) crossover by analyzing
the coverage of the combination of test cases, and (iii) mutation by enforcing
the mutation of values with specific constraints. On the other hand, Avdeenko
et al. [124] focused on improving the fitness function of the algorithm, and
were able to achieve 100% code coverage after multiple generations. Despite
the quality of their work, Zhu et al. [125] only consider the generation of
numeric input in an extremely simple code sample. Avdeenko et al. [124] do not
provide much detail regarding the generated input, but the tool was tested in
a more complex code structure. Yao et al. [126] tackled the problem that is
testing software with randomness and uses GA along with a mathematical model
capable of generating data according to a certain criterion, by observing the
impact of random behavior on the software when given random input numbers and
modifying them. In the white-box testing context, GAs are very popular.
However, most approaches rely on generating random input and evolve it by
testing iteratively the new input, using metrics such as the code coverage as
fitness. Depending on the size of the software that is being tested, this
approach may be just as unfeasible as the random approach.

Other authors have also approached the problem without necessarily resorting
to the most traditional metaheuristic methods. For instance, Jaiswal et al.
[127] decided to approach the problem using particle swarm optimization
algorithm along with an improved fitness function that uses Control Flow Graph
(CFG) to generate constrained input. This technique led to authors to achieve
100% code coverage in only two iterations. However, the method only considers
numeric input and the code analysed lacks complexity. Therefore, further tests
should be made to understand if this is indeed a viable solution for
constrained input generation. Zhang et al. [129] propose a version of the
D-algorithm for software capable of constraining the input generation process
to directed local search. Their method requires a CFG of the code, as well as
define-use chains, which represent where a variable is defined in the node and
when it is used in the CFG. Feng et al. [130] developed their own multi-
objective optimization algorithm for path coverage-oriented test data
generation. They start by randomly generating data and use it to discover the
easy-to-cover paths. Then they perform mutations on the input data generated
to discover the more complex paths. Dang et al. [128] attempt to automate and
reduce the search domain in the context of mutation testing, which is a fault-
oriented software testing technique. Their work consists of a method for
generating test data applying a dimensionality reduction on the search domain
based on a mutant stubbornness. The authors approach this problem as an
optimization one and decide to use a coevolutionary GA to perform the task of
generating test data to kill mutants.

Guo et al. [43] took one step further and decided to use WGAN + Gradient
Penalty (WGAN-GP) to produce constrained test data input. The authors focus on
performing unit and integration testing. The WGAN-GP is trained on the
execution path information to learn the behaviour of the SUT. The trained GAN
is then capable of producing and selecting new test data to increase branch
coverage. The GAN is trained with the execution path of test input using the
Gcov tool. Gcov is a source code coverage analysis tool capable of counting
the number of times each statement is a program is executed. The algorithm
then produces new data that is used along with Gcov to generate execution path
information, which is then fed into the GAN again. This way, the authors are
capable of maximizing branch coverage. Compared to random testing, their work
was capable of improving test coverage of 5 out of 7 functions in the unit
testing.

Overall, the white-box testing approaches mostly resort to heuristics and
perform iterative testing of the generated input against the SUT to check if
the coverage increases. Exhaustive evaluation of all possibilities should be
avoided, for it is very expensive when considering the complexity of modern
systems. From a grey-box and black-box perspective, the approaches usually
lack knowledge regarding the SUT and its constraints, which can lead to more
time-consuming and resource-intensive testing. Automated testing tools could
leverage adversarial methods like GANs capable of constraining the input more
efficiently and reducing the search space. By testing specific parameter
combinations tailored to the characteristics of each function, constrained
adversarial data generation approaches could present significant improvements
to these tools. Therefore, it is pertinent to perform further research and
develop new methods and tools to improve the quality of the data utilized in
automated software testing tools.

##  5 Conclusions

This work reviewed the recent scientific advances of constrained data
generation in a systematic manner. The state-of-the-art methods and tools for
constrained data generation in the adversarial machine learning and automated
software testing fields were analyzed, identifying research gaps and
opportunities to transfer knowledge from adversarial attacks to enhance
testing tools and improve the resilience and robustness of digital systems.

Regarding the current testing tools, there are several challenges that hinder
full automation. Recent works attempt to constraint the input that is
generated by analyzing the SUT empirically or resorting to specifications,
although most perform exhaustive search by mutating parameters in order to
find different branches of the SUT that are yet to be tested, which is
unpractical. This is the key aspect where constrained adversarial learning
could be valuable. Even though most approaches were developed for different
applications, they can potentially be redesigned and adapted to generate high-
quality data according to the constraints of the SUT. Adversarial examples
could be used to further improve testing tools with different parameter
combinations specifically crafted to deceive the tested functions and software
components, reducing the time and the computational resources required to
achieve a high test coverage in large and complex systems.

In the future, the use of adversarial machine learning methods and other
artificial intelligence techniques to enhance software development and testing
should be further explored. For instance, to perform white-box unit testing,
natural language processing could be leveraged to analyze the code and extract
its constraints to quickly adapt the adversarial attack methods for the
relevant test cases and improve computer code coverage in a more efficient
way. From a black-box perspective, it may be harder to have a fully accurate
code coverage analysis, but fuzzy logic could be combined with adversarial
reinforcement learning as an API behaviour learning technique to generate
faulty test input according to the characteristics of different APIs,
providing more adaptive software testing tools.

#### Author Contributions

Conceptualization, J.V. and I.P.; methodology, J.V., T.D. and T.F.;
validation, E.M. and I.P.; investigation, J.V., T.D. and T.F.; writing, J.V.,
T.D. and T.F.; supervision, I.P.; project administration, I.P.; funding
acquisition, I.P. All authors have read and agreed to the published version of
the manuscript.

#### Funding

The present work was partially supported by the Norte Portugal Regional
Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership
Agreement, through the European Regional Development Fund (ERDF), within
project ”Cybers SeC IP” (NORTE-01-0145-FEDER-000044). This work has also
received funding from UIDB/00760/2020.

#### Conflicts of Interest

The authors declare no conflict of interest.

## References

  * [1] European Commission, Directorate-General for Communications Networks Content and Technology, A path to the digital decade, Publications Office of the European Union, 2021.  doi:10.2759/027045.
  * [2] European Union Agency for Cybersecurity, I. Lella, E. Tsekmezoglou, R. S. Naydenov, C. Ciobanu, A. Malatras, M. Theocharidou, ENISA Threat Landscape 2022, Tech. rep. (2022).  doi:10.2824/764318.
  * [3] European Union Agency for Cybersecurity, N. Christoforatos, I. Lella, E. Rekleitis, C. V. Heurck, A. Zacharis, Cyber Europe 2022: After Action Report, Tech. rep. (2022).  doi:10.2824/397622.
  * [4] G. J. Myers, C. Sandler, T. Badgett, The art of software testing (2012) 240doi:10.1002/9781119202486.
  * [5] J. Vitorino, I. Praça, E. Maia, Towards Adversarial Realism and Robust Learning for IoT Intrusion Detection and Classification, Annals of Telecommunications (3 2023).  doi:10.1007/s12243-023-00953-y.
  * [6] D. Moher, et al., Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement, Systematic Reviews 4 (1) (2015) 1.  doi:10.1186/2046-4053-4-1.
  * [7] Elsevier ScienceDirect (1 2023).
URL https://www.sciencedirect.com/search

  * [8] Association for Computing Machinery Digital Library (1 2023).
URL https://dl.acm.org/search/advanced

  * [9] Institute of Electrical and Electronics Engineers Xplore (1 2023).
URL https://ieeexplore.ieee.org/search/advanced

  * [10] Multidisciplinary Digital Publishing Institute (1 2023).
URL https://www.mdpi.com/search

  * [11] M. Fuentes Reyes, S. Auer, N. Merkle, C. Henry, M. Schmitt, SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks—Optimization, Opportunities and Limits, Remote Sensing 11 (17) (2019).  doi:10.3390/rs11172067.
  * [12] X. Wang, K. Tan, Q. Du, Y. Chen, P. Du, CVA2E: A Conditional Variational Autoencoder With an Adversarial Training Process for Hyperspectral Imagery Classification, IEEE Transactions on Geoscience and Remote Sensing 58 (8) (2020) 5676–5692.  doi:10.1109/TGRS.2020.2968304.
  * [13] Z. Chen, L. Tong, B. Qian, J. Yu, C. Xiao, Self-Attention-Based Conditional Variational Auto-Encoder Generative Adversarial Networks for Hyperspectral Classification, Remote Sensing 13 (16) (2021).  doi:10.3390/rs13163316.
  * [14] C. J. Costa, S. Tiwari, K. Bhagat, A. Verlekar, K. M. C. Kumar, S. Aswale, Three-Dimensional Reconstruction of Satellite images using Generative Adversarial Networks, in: 2021 Int. Conf. on Technological Advancements and Innovations (ICTAI), 2021, pp. 121–126.  doi:10.1109/ICTAI53825.2021.9673457.
  * [15] Y. Kim, S. Hong, Deep Learning-Generated Nighttime Reflectance and Daytime Radiance of the Midwave Infrared Band of a Geostationary Satellite, Remote Sensing 11 (22) (2019).  doi:10.3390/rs11222713.
  * [16] W. Han, L. Wang, R. Feng, L. Gao, X. Chen, Z. Deng, J. Chen, P. Liu, Sample generation based on a supervised Wasserstein Generative Adversarial Network for high-resolution remote-sensing scene classification, Information Sciences 539 (2020) 177–194.  doi:10.1016/j.ins.2020.06.018.
  * [17] H. Mizuochi, Y. Iijima, H. Nagano, A. Kotani, T. Hiyama, Dynamic Mapping of Subarctic Surface Water by Fusion of Microwave and Optical Satellite Data Using Conditional Adversarial Networks, Remote Sensing 13 (2) (2021).  doi:10.3390/rs13020175.
  * [18] C.-I. Cira, M. Kada, M.-Á. Manso-Callejo, R. Alcarria, B. Bordel Sanchez, Improving Road Surface Area Extraction via Semantic Segmentation with Conditional Generative Learning for Deep Inpainting Operations, ISPRS International Journal of Geo-Information 11 (1) (2022).  doi:10.3390/ijgi11010043.
  * [19] Y. Zhang, H. Sun, J. Zuo, H. Wang, G. Xu, X. Sun, Aircraft Type Recognition in Remote Sensing Images Based on Feature Learning with Conditional Generative Adversarial Networks, Remote Sensing 10 (7) (2018).  doi:10.3390/rs10071123.
  * [20] N. Hayatbini, B. Kong, K.-l. Hsu, P. Nguyen, S. Sorooshian, G. Stephens, C. Fowlkes, R. Nemani, S. Ganguly, Conditional Generative Adversarial Networks (cGANs) for Near Real-Time Precipitation Estimation from Multispectral GOES-16 Satellite Imageries—PERSIANN-cGAN, Remote Sensing 11 (19) (2019).  doi:10.3390/rs11192193.
  * [21] X. Pan, J. Zhao, J. Xu, Conditional Generative Adversarial Network-Based Training Sample Set Improvement Model for the Semantic Segmentation of High-Resolution Remote Sensing Images, IEEE Transactions on Geoscience and Remote Sensing 59 (9) (2021) 7854–7870.  doi:10.1109/TGRS.2020.3033816.
  * [22] W. Chen, Y. Li, Z. Zhao, InsulatorGAN: A Transmission Line Insulator Detection Model Using Multi-Granularity Conditional Generative Adversarial Nets for UAV Inspection, Remote Sensing 13 (19) (2021).  doi:10.3390/rs13193971.
  * [23] H. S. Munawar, F. Ullah, A. Heravi, M. J. Thaheem, A. Maqsoom, Inspecting Buildings Using Drones and Computer Vision: A Machine Learning Approach to Detect Cracks and Damages, Drones 6 (1) (2022).  doi:10.3390/drones6010005.
  * [24] J. Zhang, M. Zhu, L. Peng, Customized Parking Data Generation based on Multi-conditional GAN, in: 2020 IEEE 23rd Int. Conf. on Intelligent Transportation Systems (ITSC), 2020, pp. 1–6.  doi:10.1109/ITSC45102.2020.9294436.
  * [25] H. Cheng, W. Liao, M. Y. Yang, B. Rosenhahn, M. Sester, AMENet: Attentive Maps Encoder Network for trajectory prediction, ISPRS Journal of Photogrammetry and Remote Sensing 172 (2021) 253–266.  doi:10.1016/j.isprsjprs.2020.12.004.
  * [26] A. Jaafer, G. Nilsson, G. Como, Data Augmentation of IMU Signals and Evaluation via a Semi-Supervised Classification of Driving Behavior, in: 2020 IEEE 23rd Int. Conf. on Intelligent Transportation Systems (ITSC), 2020, pp. 1–6.  doi:10.1109/ITSC45102.2020.9294496.
  * [27] J. Li, B. Zhao, K. Wu, Z. Dong, X. Zhang, Z. Zheng, A Representation Generation Approach of Transmission Gear Based on Conditional Generative Adversarial Network, Actuators 10 (5) (2021).  doi:10.3390/act10050086.
  * [28] M. Pöpperli, R. Gulagundi, S. Yogamani, S. Milz, Realistic Ultrasonic Environment Simulation Using Conditional Generative Adversarial Networks, in: 2019 IEEE Intelligent Vehicles Symposium (IV), 2019, pp. 2278–2283.  doi:10.1109/IVS.2019.8814091.
  * [29] F. Falahatraftar, S. Pierre, S. Chamberland, A Conditional Generative Adversarial Network Based Approach for Network Slicing in Heterogeneous Vehicular Networks, Telecom 2 (1) (2021) 141–154.  doi:10.3390/telecom2010009.
  * [30] F. Kocayusufoglu, A. Silva, A. K. Singh, FlowGEN: A Generative Model for Flow Graphs, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, Association for Computing Machinery, 2022, pp. 813–823.  doi:10.1145/3534678.3539406.
  * [31] Q.-X. Zhu, K.-R. Hou, Z.-S. Chen, Y. Xu, Y.-L. He, Research and Application of Virtual Sample Generation Method Based on Conditional Generative Adversarial Network, in: 2021 China Automation Congress (CAC), 2021, pp. 351–355.  doi:10.1109/CAC53003.2021.9728144.
  * [32] R. Qin, J. Zhao, High-Efficiency Generative Adversarial Network Model for Chemical Process Fault Diagnosis, IFAC-PapersOnLine 55 (7) (2022) 732–737.  doi:10.1016/j.ifacol.2022.07.531.
  * [33] Y.-L. He, X.-Y. Li, J.-H. Ma, S. Lu, Q.-X. Zhu, A novel virtual sample generation method based on a modified conditional Wasserstein GAN to address the small sample size problem in soft sensing, Journal of Process Control 113 (2022) 18–28.  doi:10.1016/j.jprocont.2022.03.008.
  * [34] L. Zhang, P. Liu, L. Wang, J. Liu, B. Song, Y. Zhang, G. He, H. Zhang, Improved 1-km-Resolution Hourly Estimates of Aerosol Optical Depth Using Conditional Generative Adversarial Networks, Remote Sensing 13 (19) (2021).  doi:10.3390/rs13193834.
  * [35] E. Alhajjar, P. Maxwell, N. Bastian, Adversarial machine learning in Network Intrusion Detection Systems, Expert Systems with Applications 186 (2021) 115782\.  doi:10.1016/j.eswa.2021.115782.
  * [36] A. Chernikova, A. Oprea, FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments, ACM Trans. Priv. Secur. 25 (4) (7 2022).  doi:10.1145/3544746.
  * [37] A. S. Dina, A. B. Siddique, D. Manivannan, Effect of Balancing Data Using Synthetic Data on the Performance of Machine Learning Classifiers for Intrusion Detection in Computer Networks, IEEE Access 10 (2022) 96731–96747.  doi:10.1109/ACCESS.2022.3205337.
  * [38] J. Vitorino, N. Oliveira, I. Praça, Adaptative Perturbation Patterns: Realistic Adversarial Learning for Robust Intrusion Detection, Future Internet 14 (4) (3 2022).  doi:10.3390/fi14040108.
  * [39] J. Wang, X. Yan, L. Liu, L. Li, Y. Yu, CTTGAN: Traffic Data Synthesizing Scheme Based on Conditional GAN, Sensors 22 (14) (2022).  doi:10.3390/s22145243.
  * [40] M. Wan, H. Yao, X. Yan, Generation of malicious webpage samples based on GAN, in: 2020 IEEE 19th Int. Conf. on Trust, Security and Privacy in Computing and Communications (TrustCom), 2020, pp. 864–869.  doi:10.1109/TrustCom50675.2020.00116.
  * [41] Q. Liu, G. Yu, Y. Wang, Z. Yi, A Novel DGA Domain Adversarial Sample Generation Method By Geometric Perturbation, in: 2021 3rd Int. Conf. on Advanced Information Science and System (AISS 2021), AISS 2021, Association for Computing Machinery, 2021.  doi:10.1145/3503047.3503080.
  * [42] X. Gao, R. K. Saha, M. R. Prasad, A. Roychoudhury, Fuzz Testing Based Data Augmentation to Improve Robustness of Deep Neural Networks, in: Proceedings of the ACM/IEEE 42nd Int. Conf. on Software Engineering, ICSE ’20, Association for Computing Machinery, 2020, pp. 1147–1158.  doi:10.1145/3377811.3380415.
  * [43] X. Guo, H. Okamura, T. Dohi, Automated Software Test Data Generation With Generative Adversarial Networks, IEEE Access 10 (2022) 20690–20700.  doi:10.1109/ACCESS.2022.3153347.
  * [44] S. Kasarapu, S. Shukla, R. Hassan, A. Sasan, H. Homayoun, S. M. PD, CAD-FSL: Code-Aware Data Generation Based Few-Shot Learning for Efficient Malware Detection, in: Proceedings of the Great Lakes Symposium on VLSI 2022, GLSVLSI ’22, Association for Computing Machinery, 2022, pp. 507–512.  doi:10.1145/3526241.3530825.
  * [45] P. Chonwiharnphan, P. Thienprapasith, E. Chuangsuwanich, Generating Realistic Users Using Generative Adversarial Network With Recommendation-Based Embedding, IEEE Access 8 (2020) 41384–41393.  doi:10.1109/ACCESS.2020.2976491.
  * [46] A. Esmaili, S. Farzi, Effective synthetic data generation for fake user detection, in: 2021 26th International Computer Conference, Computer Society of Iran (CSICC), 2021, pp. 1–5.  doi:10.1109/CSICC52343.2021.9420570.
  * [47] Y. Qian, H. Hu, T. Tan, Data augmentation using generative adversarial networks for robust speech recognition, Speech Communication 114 (2019) 1–9.  doi:10.1016/j.specom.2019.08.006.
  * [48] L. Chen, Y. Liu, W. Xiao, Y. Wang, H. Xie, SpeakerGAN: Speaker identification with conditional generative adversarial network, Neurocomputing 418 (2020) 211–220.  doi:10.1016/j.neucom.2020.08.040.
  * [49] S. R. Ram, V. K. M, B. Subramanian, N. Bacanin, M. Zivkovic, I. Strumberger, Speech enhancement through improvised conditional generative adversarial networks, Microprocessors and Microsystems 79 (2020) 103281.  doi:10.1016/j.micpro.2020.103281.
  * [50] Y.-Y. Ding, H.-J. Lin, L.-J. Liu, Z.-H. Ling, Y. Hu, Robustness of Speech Spoofing Detectors Against Adversarial Post-Processing of Voice Conversion, IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021) 3415–3426.  doi:10.1109/TASLP.2021.3124420.
  * [51] J. Yoon, L. N. Drumright, M. van der Schaar, Anonymization Through Data Synthesis Using Generative Adversarial Networks (ADS-GAN), IEEE Journal of Biomedical and Health Informatics 24 (8) (2020) 2378–2388.  doi:10.1109/JBHI.2020.2980262.
  * [52] Z. Liu, L. Meng, Y. Tan, J. Zhang, H. Zhang, Image compression based on octave convolution and semantic segmentation, Knowledge-Based Systems 228 (2021) 107254\.  doi:10.1016/j.knosys.2021.107254.
  * [53] A. S. Khwaja, A. Anpalagan, B. Venkatesh, Smart Meter Data Masking Using Conditional Generative Adversarial Networks, Electric Power Systems Research 209 (2022) 108033.  doi:10.1016/j.epsr.2022.108033.
  * [54] S. Athey, G. W. Imbens, J. Metzger, E. Munro, Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations, Journal of Econometrics (2021).  doi:10.1016/j.jeconom.2020.09.013.
  * [55] D. Fang, X. Guan, B. Hu, Y. Peng, M. Chen, K. Hwang, Deep Reinforcement Learning for Scenario-Based Robust Economic Dispatch Strategy in Internet of Energy, IEEE Internet of Things Journal 8 (12) (2021) 9654–9663.  doi:10.1109/JIOT.2020.3040294.
  * [56] J. Li, Y. Yang, J. S. Sun, K. Tomsovic, H. Qi, ConAML: Constrained Adversarial Machine Learning for Cyber-Physical Systems, in: Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS ’21, Association for Computing Machinery, 2021, pp. 52–66.  doi:10.1145/3433210.3437513.
  * [57] X. Gong, B. Tang, R. Zhu, W. Liao, L. Song, Data Augmentation for Electricity Theft Detection Using Conditional Variational Auto-Encoder, Energies 13 (17) (2020).  doi:10.3390/en13174291.
  * [58] H. Liu, Z. Li, Y. Li, Noise Reduction Power Stealing Detection Model Based on Self-Balanced Data Set, Energies 13 (7) (2020).  doi:10.3390/en13071763.
  * [59] J. Moon, S. Jung, S. Park, E. Hwang, Conditional Tabular GAN-Based Two-Stage Data Generation Scheme for Short-Term Load Forecasting, IEEE Access 8 (2020) 205327–205339.  doi:10.1109/ACCESS.2020.3037063.
  * [60] P. Qin, X. Wang, Z. Qiao, X. Li, Q. Hu, W. Shu, GAN-based Residential Load Data Generation Model Considering Users’ Privacy, in: 2022 7th Asia Conference on Power and Electrical Engineering (ACPEE), 2022, pp. 838–843.  doi:10.1109/ACPEE53904.2022.9783676.
  * [61] Y. Zhang, X. Deng, Y. Zhang, Y. Zhang, Generation of sub-item load profiles for public buildings based on the conditional generative adversarial network and moving average method, Energy and Buildings 268 (2022) 112185.  doi:10.1016/j.enbuild.2022.112185.
  * [62] S. Zhang, Z. Ji, J. Zhang, Y. Bao, W. Wang, Multi-dimensional Data Generation Method of Electric Vehicle Charging Behaviors Based on Improved Generative Adversarial Network, in: 2022 IEEE/IAS Industrial and Commercial Power System Asia, 2022, pp. 1827–1832.  doi:10.1109/ICPSAsia55496.2022.9949855.
  * [63] D. Chang, W. Yang, X. Yong, G. Zhang, W. Wang, H. Li, Y. Wang, Seismic Data Interpolation Using Dual-Domain Conditional Generative Adversarial Networks, IEEE Geoscience and Remote Sensing Letters 18 (10) (2021) 1856–1860.  doi:10.1109/LGRS.2020.3008478.
  * [64] É. Guérin, J. Digne, É. Galin, A. Peytavie, C. Wolf, B. Benes, B. Martinez, Interactive Example-Based Terrain Authoring with Conditional Generative Adversarial Networks, ACM Trans. Graph. 36 (6) (11 2017).  doi:10.1145/3130800.3130804.
  * [65] J. Zhang, C. Li, P. Zhou, C. Wang, G. He, H. Qin, Authoring multi-style terrain with global-to-local control, Graphical Models 119 (2022) 101122.  doi:10.1016/j.gmod.2021.101122.
  * [66] G. Silva, I. Domingues, H. Duarte, J. A. M. Santos, Automatic Generation of Lymphoma Post-Treatment PETs using Conditional-GANs, in: 2019 Digital Image Computing: Techniques and Applications (DICTA), 2019, pp. 1–6.  doi:10.1109/DICTA47822.2019.8945835.
  * [67] J. F. Teixeira, M. Dias, E. Batista, J. Costa, L. F. Teixeira, H. P. Oliveira, Adversarial Data Augmentation on Breast MRI Segmentation, Applied Sciences 11 (10) (2021).  doi:10.3390/app11104554.
  * [68] S. Amirrajab, Y. Al Khalil, C. Lorenz, J. Weese, J. Pluim, M. Breeuwer, Label-informed cardiac magnetic resonance image synthesis through conditional generative adversarial networks, Computerized Medical Imaging and Graphics 101 (2022) 102123.  doi:10.1016/j.compmedimag.2022.102123.
  * [69] M. Liu, W. Zou, W. Wang, C.-B. Jin, J. Chen, C. Piao, Multi-Conditional Constraint Generative Adversarial Network-Based MR Imaging from CT Scan Data, Sensors 22 (11) (2022).  doi:10.3390/s22114043.
  * [70] N. Qiang, Q. Dong, H. Liang, J. Li, S. Zhang, C. Zhang, B. Ge, Y. Sun, J. Gao, T. Liu, H. Yue, S. Zhao, Learning brain representation using recurrent Wasserstein generative adversarial net, Computer Methods and Programs in Biomedicine 223 (2022) 106979.  doi:10.1016/j.cmpb.2022.106979.
  * [71] J. L. Hagad, T. Kimura, K.-i. Fukui, M. Numao, Learning Subject-Generalized Topographical EEG Embeddings Using Deep Variational Autoencoders and Domain-Adversarial Regularization, Sensors 21 (5) (2021).  doi:10.3390/s21051792.
  * [72] X. Zhou, X. Zhu, K. Nakamura, M. Noro, Electrocardiogram Quality Assessment with a Generalized Deep Learning Model Assisted by Conditional Generative Adversarial Networks, Life 11 (10) (2021).  doi:10.3390/life11101013.
  * [73] S. E. Karabulut, M. M. Khorasani, A. Pantanowitz, Neurocartographer: CC-WGAN Based SSVEP Data Generation to Produce a Model toward Symmetrical Behaviour to the Human Brain, Symmetry 14 (8) (2022).  doi:10.3390/sym14081600.
  * [74] M. Ehrhart, B. Resch, C. Havas, D. Niederseer, A Conditional GAN for Generating Time Series Data for Stress Detection in Wearable Physiological Sensor Data, Sensors 22 (16) (2022).  doi:10.3390/s22165969.
  * [75] N. Saffari, H. A. Rashwan, M. Abdel-Nasser, V. Kumar Singh, M. Arenas, E. Mangina, B. Herrera, D. Puig, Fully Automated Breast Density Segmentation and Classification Using Deep Learning, Diagnostics 10 (11) (2020).  doi:10.3390/diagnostics10110988.
  * [76] A. Gür, Deep Feature Synthesis for Accurate Breast Cancer Prediction, in: 2022 Medical Technologies Congress (TIPTEKNO), 2022, pp. 1–4.  doi:10.1109/TIPTEKNO56568.2022.9960237.
  * [77] F. Zhang, Y. Zhang, X. Zhu, X. Chen, H. Du, X. Zhang, PregGAN: A prognosis prediction model for breast cancer based on conditional generative adversarial networks, Computer Methods and Programs in Biomedicine 224 (2022) 107026.  doi:10.1016/j.cmpb.2022.107026.
  * [78] M. Havaei, X. Mao, Y. Wang, Q. Lao, Conditional generation of medical images via disentangled adversarial inference, Medical Image Analysis 72 (2021) 102106\.  doi:10.1016/j.media.2021.102106.
  * [79] O. Bailo, D. Ham, Y. M. Shin, Red Blood Cell Image Generation for Data Augmentation Using Conditional Generative Adversarial Networks, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, pp. 1039–1048.  doi:10.1109/CVPRW.2019.00136.
  * [80] K. T. Chui, M. D. Lytras, P. Vasant, Combined Generative Adversarial Network and Fuzzy C-Means Clustering for Multi-Class Voice Disorder Detection with an Imbalanced Dataset, Applied Sciences 10 (13) (2020).  doi:10.3390/app10134571.
  * [81] K. Chen, D. Zhu, J. Lu, Y. Luo, An Adversarial and Densely Dilated Network for Connectomes Segmentation, Symmetry 10 (10) (2018).  doi:10.3390/sym10100467.
  * [82] Y. Yu, B. Tang, R. Lin, S. Han, T. Tang, M. Chen, CWGAN: Conditional Wasserstein Generative Adversarial Nets for Fault Data Generation, in: 2019 IEEE Int. Conf. on Robotics and Biomimetics (ROBIO), 2019, pp. 2713–2718.  doi:10.1109/ROBIO49542.2019.8961501.
  * [83] S. Dixit, N. K. Verma, A. K. Ghosh, Intelligent Fault Diagnosis of Rotary Machines: Conditional Auxiliary Classifier GAN Coupled With Meta Learning Using Limited Data, IEEE Transactions on Instrumentation and Measurement 70 (2021) 1–11.  doi:10.1109/TIM.2021.3082264.
  * [84] M. Ahang, M. Jalayer, A. Shojaeinasab, O. Ogunfowora, T. Charter, H. Najjaran, Synthesizing Rolling Bearing Fault Samples in New Conditions: A Framework Based on a Modified CGAN, Sensors 22 (14) (2022).  doi:10.3390/s22145413.
  * [85] Y. Li, W. Zou, L. Jiang, Fault diagnosis of rotating machinery based on combination of Wasserstein generative adversarial networks and long short term memory fully convolutional network, Measurement 191 (2022) 110826.  doi:10.1016/j.measurement.2022.110826.
  * [86] Y. Liu, H. Jiang, Y. Wang, Z. Wu, S. Liu, A conditional variational autoencoding generative adversarial networks with self-modulation for rolling bearing fault diagnosis, Measurement 192 (2022) 110888.  doi:10.1016/j.measurement.2022.110888.
  * [87] Y. Peng, Y. Wang, Y. Shao, A novel bearing imbalance Fault-diagnosis method based on a Wasserstein conditional generative adversarial network, Measurement 192 (2022) 110924.  doi:10.1016/j.measurement.2022.110924.
  * [88] X. Wang, H. Liu, Data supplement for a soft sensor using a new generative model based on a variational autoencoder and Wasserstein GAN, Journal of Process Control 85 (2020) 91–99.  doi:10.1016/j.jprocont.2019.11.004.
  * [89] C.-H. Chen, C.-K. Tsung, S.-S. Yu, Designing a Hybrid Equipment-Failure Diagnosis Mechanism under Mixed-Type Data with Limited Failure Samples, Applied Sciences 12 (18) (2022).  doi:10.3390/app12189286.
  * [90] X. Guo, X. Liu, G. Królczyk, M. Sulowicz, A. Glowacz, P. Gardoni, Z. Li, Damage Detection for Conveyor Belt Surface Based on Conditional Cycle Generative Adversarial Network, Sensors 22 (9) (2022).  doi:10.3390/s22093485.
  * [91] Y.-r. Wang, G.-d. Sun, Q. Jin, Imbalanced sample fault diagnosis of rotating machinery using conditional variational auto-encoder generative adversarial network, Applied Soft Computing 92 (2020) 106333.  doi:10.1016/j.asoc.2020.106333.
  * [92] X. Liu, H. Ma, Y. Liu, A Novel Transfer Learning Method Based on Conditional Variational Generative Adversarial Networks for Fault Diagnosis of Wind Turbine Gearboxes under Variable Working Conditions, Sustainability 14 (9) (2022).  doi:10.3390/su14095441.
  * [93] L. Zhang, H. Zhang, G. Cai, The Multiclass Fault Diagnosis of Wind Turbine Bearing Based on Multisource Signal Fusion and Deep Learning Generative Model, IEEE Transactions on Instrumentation and Measurement 71 (2022) 1–12.  doi:10.1109/TIM.2022.3178483.
  * [94] K. N. Haque, R. Rana, B. W. Schuller, High-Fidelity Audio Generation and Representation Learning With Guided Adversarial Autoencoder, IEEE Access 8 (2020) 223509–223528.  doi:10.1109/ACCESS.2020.3040797.
  * [95] S. Li, Y. Sung, INCO-GAN: Variable-Length Music Generation Method Based on Inception Model-Based Conditional GAN, Mathematics 9 (4) (2021).  doi:10.3390/math9040387.
  * [96] L. Chen, S. Srivastava, Z. Duan, C. Xu, Deep Cross-Modal Audio-Visual Generation, in: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Thematic Workshops ’17, Association for Computing Machinery, 2017, pp. 349–357.  doi:10.1145/3126686.3126723.
  * [97] E. Barsoum, J. Kender, Z. Liu, HP-GAN: Probabilistic 3D Human Motion Prediction via GAN, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 1499–149909.  doi:10.1109/CVPRW.2018.00191.
  * [98] Z. Geng, D. Johnson, R. Fedkiw, Coercing machine learning to output physically accurate results, Journal of Computational Physics 406 (2020) 109099.  doi:10.1016/j.jcp.2019.109099.
  * [99] Y. Li, T. Zhang, L. Duan, C. Xu, A Unified Generative Adversarial Framework for Image Generation and Person Re-Identification, in: Proceedings of the 26th ACM Int. Conf. on Multimedia, MM ’18, Association for Computing Machinery, 2018, pp. 163–172.  doi:10.1145/3240508.3240573.
  * [100] A. Marzouk, P. Barros, M. Eppe, S. Wermter, The Conditional Boundary Equilibrium Generative Adversarial Network and its Application to Facial Attributes, in: 2019 International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–7.  doi:10.1109/IJCNN.2019.8852164.
  * [101] X. Chen, Y. Sun, X. Shu, Q. Li, Attention-aware conditional generative adversarial networks for facial age synthesis, Neurocomputing 451 (2021) 167–180.  doi:10.1016/j.neucom.2021.04.068.
  * [102] A. Belmonte-Hernández, G. Hernández-Peñaloza, D. Martín Gutiérrez, F. Álvarez, Recurrent Model for Wireless Indoor Tracking and Positioning Recovering Using Generative Networks, IEEE Sensors Journal 20 (6) (2020) 3356–3365.  doi:10.1109/JSEN.2019.2958201.
  * [103] B. Jahić, N. Guelfi, B. Ries, Software Engineering for Dataset Augmentation using Generative Adversarial Networks, in: 2019 IEEE 10th Int. Conf. on Software Engineering and Service Science (ICSESS), 2019, pp. 59–66.  doi:10.1109/ICSESS47205.2019.9040806.
  * [104] L. Kang, P. Riba, M. Rusiñol, A. Fornés, M. Villegas, Content and Style Aware Generation of Text-Line Images for Handwriting Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (12) (2022) 8846–8860.  doi:10.1109/TPAMI.2021.3122572.
  * [105] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Nets, in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems, Vol. 27, Curran Associates, Inc., 2014\.
  * [106] D. P. Kingma, M. Welling, Auto-Encoding Variational Bayes, in: 2nd Int. Conf. on Learning Representations, ICLR 2014 - Conference Track Proceedings, 2014, pp. 1–14.  doi:10.48550/ARXIV.1312.6114.
  * [107] M. Mirza, S. Osindero, Conditional Generative Adversarial Nets, arXiv (2014).  doi:10.48550/ARXIV.1411.1784.
  * [108] M. Arjovsky, S. Chintala, L. Bottou, Wasserstein Generative Adversarial Networks, in: D. Precup, Y. W. Teh (Eds.), Proceedings of the 34th Int. Conf. on Machine Learning, Vol. 70 of Proceedings of Machine Learning Research, PMLR, 2017, pp. 214–223.
  * [109] K. Sohn, H. Lee, X. Yan, Learning Structured Output Representation using Deep Conditional Generative Models, in: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 28, Curran Associates, Inc., 2015.
  * [110] A. Martin-Lopez, A. Arcuri, S. Segura, A. Ruiz-Cortés, Black-box and white-box test case generation for restful apis: Enemies or allies?, 2021.  doi:10.1109/ISSRE52982.2021.00034.
  * [111] L. Rajamanickam, N. A. B. M. Saat, S. N. B. Daud, Software testing: The generation tools, International Journal of Advanced Trends in Computer Science and Engineering 8 (2019) 231–234.  doi:10.30534/ijatcse/2019/20822019.
  * [112] Y. Zheng, Y. Liu, X. Xie, Y. Liu, L. Ma, J. Hao, Y. Liu, Automatic web testing using curiosity-driven reinforcement learning, IEEE Press, 2021, p. 423–435.  doi:10.1109/ICSE43902.2021.00048.
URL https://doi.org/10.1109/ICSE43902.2021.00048

  * [113] M. E. Khan, F. Khan, A comparative study of white box, black box and grey box testing techniques, International Journal of Advanced Computer Science and Applications 3 (2012) 12–15.
  * [114] N. Anwar, S. Kar, Review paper on various software testing techniques, Global Journal of Computer Science and Technology 19 (C2) (2019) 43–49.
  * [115] A. Martin-Lopez, S. Segura, A. Ruiz-Cortés, RESTest: Automated Black-Box Testing of RESTful Web APIs, in: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2021, Association for Computing Machinery, 2021, pp. 682–685.  doi:10.1145/3460319.3469082.
  * [116] A. Martin-Lopez, AI-Driven Web API Testing, in: Proceedings of the ACM/IEEE 42nd Int. Conf. on Software Engineering: Companion Proceedings, ICSE ’20, Association for Computing Machinery, 2020, pp. 202–205.  doi:10.1145/3377812.3381388.
  * [117] S. O. Haraldsson, J. R. Woodward, A. I. E. Brownlee, The Use of Automatic Test Data Generation for Genetic Improvement in a Live System, in: Proceedings of the 10th International Workshop on Search-Based Software Testing, SBST ’17, IEEE Press, 2017, pp. 28–31.
  * [118] N. Walkinshaw, G. Fraser, Uncertainty-Driven Black-Box Test Data Generation, in: 2017 IEEE Int. Conf. on Software Testing, Verification and Validation (ICST), IEEE, 2017, pp. 253–263.  doi:10.1109/ICST.2017.30.
  * [119] J. Kim, M. Kwon, S. Yoo, Generating Test Input with Deep Reinforcement Learning, in: Proceedings of the 11th International Workshop on Search-Based Software Testing, SBST ’18, Association for Computing Machinery, 2018, pp. 51–58.  doi:10.1145/3194718.3194720.
  * [120] M. Biagiola, A. Stocco, F. Ricca, P. Tonella, Diversity-Based Web Test Generation, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, Association for Computing Machinery, 2019, pp. 142–153.  doi:10.1145/3338906.3338970.
  * [121] A. K. Alazzawi, H. M. Rais, S. Basri, Y. A. Alsariera, PhABC: A Hybrid Artificial Bee Colony Strategy for Pairwise test suite Generation with Constraints Support, in: 2019 IEEE Student Conference on Research and Development (SCOReD), 2019, pp. 106–111.  doi:10.1109/SCORED.2019.8896324.
  * [122] X. Huang, L. Huang, S. Zhang, L. Zhou, M. Wu, M. Chen, Improving Random Test Sets Using a Locally Spreading Approach, in: 2017 IEEE Int. Conf. on Software Quality, Reliability and Security (QRS), IEEE, 2017, pp. 32–41.  doi:10.1109/QRS.2017.13.
  * [123] M. Esnaashari, A. H. Damia, Automation of software test data generation using genetic algorithm and reinforcement learning, Expert Systems with Applications 183 (2021) 115446.  doi:10.1016/j.eswa.2021.115446.
  * [124] T. V. Avdeenko, K. E. Serdyukov, Z. B. Tsydenov, Formulation and research of new fitness function in the genetic algorithm for maximum code coverage, Procedia Computer Science 186 (2021) 713–720.  doi:10.1016/j.procs.2021.04.194.
  * [125] Z. Zhu, L. Jiao, Improving Search-Based Software Testing by Constraint-Based Genetic Operators, in: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19, Association for Computing Machinery, 2019, pp. 1435–1442.  doi:10.1145/3321707.3321720.
  * [126] X. Yao, D. Gong, B. Li, X. Dang, G. Zhang, Testing Method for Software With Randomness Using Genetic Algorithm, IEEE Access 8 (2020) 61999–62010.  doi:10.1109/ACCESS.2020.2983762.
  * [127] U. Jaiswal, A. Prajapati, Optimized Test Case Generation for Basis Path Testing Using Improved Fitness Function with PSO, in: 2021 Thirteenth Int. Conf. on Contemporary Computing (IC3-2021), IC3 ’21, Association for Computing Machinery, 2021, pp. 475–483.  doi:10.1145/3474124.3474197.
  * [128] X. Dang, X. Yao, D. Gong, T. Tian, Efficiently Generating Test Data to Kill Stubborn Mutants by Dynamically Reducing the Search Domain, IEEE Transactions on Reliability 69 (1) (2020) 334–348.  doi:10.1109/TR.2019.2922684.
  * [129] J. Zhang, S. K. Gupta, W. G. J. Halfond, A New Method for Software Test Data Generation Inspired by D-algorithm, in: 2019 IEEE 37th VLSI Test Symposium (VTS), IEEE, 2019, pp. 1–6.  doi:10.1109/VTS.2019.8758641.
  * [130] X. Feng, R. Ding, B. Chai, T. Huo, Multi-Objective Heuristic Information Optimization Algorithm for Path Coverage-Oriented Test Data Generation, in: 2021 3rd Int. Conf. on Artificial Intelligence and Advanced Manufacture, Association for Computing Machinery, 2021, pp. 650–654.  doi:10.1145/3495018.3495135.