#  iBiR: Bug Report driven Fault Injection

Ahmed Khanfir, Anil Koyuncu, Mike Papadakis, Maxime Cordy, Tegawendé F.
Bissyandé,
Jacques Klein and Yves Le Traon  SnT, University of Luxembourg, Luxembourg
<EMAIL_ADDRESS>
###### Abstract

Much research on software engineering and software testing relies on
experimental studies based on fault injection. Fault injection, however, is
not often relevant to emulate real-world software faults since it “blindly”
injects large numbers of faults. It remains indeed challenging to inject few
but realistic faults that target a particular functionality in a program. In
this work, we introduce iBiR, a fault injection tool that addresses this
challenge by exploring change patterns associated to user-reported faults. To
inject realistic faults, we create mutants by re-targeting a bug report driven
automated program repair system, i.e., reversing its code transformation
templates. iBiR is further appealing in practice since it requires deep
knowledge of neither of the code nor the tests, but just of the program’s
relevant bug reports. Thus, our approach focuses the fault injection on the
feature targeted by the bug report. We assess iBiR by considering the
Defects4J dataset. Experimental results show that our approach outperforms the
fault injection performed by traditional mutation testing in terms of semantic
similarity with the original bug, when applied at either system or class
levels of granularity, and provides better, statistically significant,
estimations of test effectiveness (fault detection). Additionally, when
injecting 100 faults, iBiR injects faults that couple with the real ones in
36% of the cases, while mutants from mutation testing inject less than  1%.
Overall, iBiR targets real functionality and injects realistic and diverse
faults.

##  I Introduction

A key challenge of fault injection techniques (such as mutation analysis) is
to emulate the effects of real faults. This property of representativeness of
the injected faults is of particular importance since fault injection
techniques are widely used by researchers when evaluating and comparing bug
finding, testing and debugging techniques, e.g., test generation, bug fixing,
fault localisation, etc, [1]. This means that there is a high risk of
mistakenly asserting test effectiveness in case the injected faults are non-
representative.

Typically, fault injection techniques introduce faults by making syntactic
changes in the target programs’ code using a set of simple syntactic
transformations [2, 3, 4], usually called mutation operators. These
transformations have been defined based on the language syntax [5] and are
“blindly” mutating the entire codebase of the projects, injecting large
numbers of mutants, with the hope to inject some realistic faults. This means
that there is a limited control on the fault types and the locations where to
inject faults. In other words, the appropriate “what” and “where” to inject
faults in order to make representative fault injection has been largely
ignored by existing research.

Fault injection techniques may also draw on recent research that mines fault
patterns [6, 7] and demonstrate some form of realism w.r.t. real faults. These
results are encouraging because they indicate that the injected faults may
carry over the realism of the patterns. This may remove a potential validity
threat, but at the same time, it is limited as it does not provide any control
on the locations and target functionality, thus impacting fault
representativeness [3, 8, 9].

This is an important limitation especially for large real-world systems
because of the following two reasons: a) injecting faults everywhere escalates
the application cost due to the large number of mutants introducedand b) the
results could be misleading since a tiny ratio of the injected faults are
coupled to the real ones [9] and the injected set of faults do not represent
the likelihood of faults appearing in the field [3]. Therefore,
representativeness of the injected faults in terms of fault types and
locations is of outmost importance w.r.t. both application cost and accuracy
of the method.

To bypass these issues, one could use real faults (mined from the projects’
repositories) or directly apply the testing approach to a set of programs and
manually identify potential faults. While such a solution brings realism into
the evaluations, it is often limited to few fault instances (of limited
diversity), requires expensive manual effort in identifying the faults and
fails to offer the experimental control required by many evaluation scenarios.

We advance in this research direction by bringing realism in the fault
injection via leveraging information from bug reports. Bug reports often
include sufficient information for debugging techniques in order to localize
[10], debug [11] and repair faults [12] that happened in the field. Therefore,
together with specially crafted defect patterns (mined through systematic
examination of real faults) such information can guide fault injection to
target critical functionality, mimic real faulty behaviour and make realistic
fault injection. Perhaps more importantly, the use of bug reports removes the
need for knowledge of the targeted system or code.

Our method starts from the target project and a bug report written in natural
language. It then applies Information Retrieval (IR)-based fault localization
[10] in order to identify the relevant places where to inject faults. It then
injects recurrent fault instances (fault patterns) that were manually crafted
using a systematic analysis of frequent bug fixes, prioritized according to
their position and type. This way our method performs fault injection, using
realistic fault patterns, by targeting the features described by the bug
reports.

We implemented our approach in a system called iBiR and evaluated its ability
to imitate  157 real faults. In particular we evaluated a) the semantic
similarity of real and injected faults, b) the coupling111Injected faults
couple with the real ones when injected faults are detected only by test cases
that detect the real faults. This implies that the injected faults provide
good indications on whether tests are capable of detecting the coupled faults.
relation between injected and real faults, and c) the ability of the injected
faults to indicate test effectiveness (fault detection) when tested with
different test suites. Our results show that iBiR manages to imitate the
targeted faults, with a median semantic similarity value of  0.58, which is
significantly higher than the  0.0 achieved by using traditional mutation
testing, when injecting the same number of faults.

Interestingly, we found that iBiR injects faults that couple with the real
ones in  36% of the targeted cases. This is achieved by injecting 100 faults
per target (real) fault and it is approximately  50 times higher than the
coupled mutants produced by mutation testing. Fault coupling is one of the
most important testing properties [13, 14], here indicating that one can use
the injected faults instead of the real ones.

Another key finding of our study is that the injected faults provide much
better indication on test effectiveness (fault detection) than mutation
testing as their detection ratios discriminate between actual failing and
passing test suites, while mutant detection rates cannot. This implies that
the use of iBiR yields more accurate results than the use of traditional
mutation testing.

Overall, our primary contributions are:

  * •

We introduce the notion of bug report-driven fault injection. Bug reports can
be used to inject realistic faults.

  * •

We introduce a set of mutation operators based on frequently used patch
patterns that are reverted to inject realistic faults.

  * •

We present iBiR, an automatic fault injection method, which is driven by bug
reports to emulate real faults.

  * •

We provide empirical evidence demonstrating that iBiR outperforms the current
state of practice in mutation testing w.r.t. fault representativeness and
coupling.

##  II Background

###  II-A Fault Localization

Fault localization is the activity of identifying the suspected fault
locations, which will be transformed to generate patches. Several automated
fault localization techniques have been proposed [15], such as slice-based
[16], spectrum-based [17], statistics-based [18], mutation-based [11] and etc.

Fault localisation techniques based on Information Retrieval (IR) [19, 20, 21,
22] exploit textual bug reports to identify code chunks relevant to the bug,
without relying on test cases. IR-based fault localisation tools extract
tokens from the bug report to formulate a query to be matched with the
collection of documents formed by the source code files [23, 24, 25, 26, 27,
10]. Then, they rank the documents based on their relevance to the query, such
that source files ranked higher are more likely to contain the fault.
Recently, automated program repair methods have been designed on top of IR-
based fault localization [12]. They achieve comparable performance to methods
using spectrum-based fault localization, yet without relying on the assumption
that test cases are available.

We leverage IR-based fault localization to achieve a different goal: instead
of localising the reported bug, we aim at _injecting faults_ at code locations
that implement a functionality similar to the fault targeted by the bug report
description.

###  II-B Mutation Testing

Mutation testing is a popular fault-based testing technique [1]. It operates
by inserting artificial faults into a program under test, thereby creating
many different versions (named _mutants_) of the program. The artificial
faults are injected through syntactic changes to all program locations in the
original program, based on predefined rules named _mutation operators_. Such
operators can, for instance, invert relational operators (e.g., replacing
$\geq$ with $<$).

Mutants can be used to indicate the strengths of test suites, based on their
ability to distinguish the mutants from the original program. If there exists
a test case distinguishing the original program from a particular mutant, then
the mutant is said to be killed. Then, we term a mutant to be “coupled” with
respect to a particular fault if the test cases that kill it are a subset of
the test cases that can also detect that fault (make the program fail by
exerting the fault).

Previous research has shown that the choice of mutation operators and location
can affect the fault-revealing ability of the produced mutants [28, 29]. Thus,
it is important to select appropriate mutation testing strategies.
Nevertheless, previous research has shown that random mutant sampling achieves
comparable results with the mutation testing state of the art [30, 8], making
the random mutant sampling a natural baseline to compare with.

Another issue involved in mutation testing campaigns is the application cost
of the method. The problem stems from the vast number of faults that are
injected, which need to be executed with a large number of test suites,
thereby escalating the computational demands of the method [1]. Unfortunately,
the mutant execution problem becomes intractable when test execution is
expensive or the test suites involve system level tests, thereby often
limiting mutation testing application to unit level. This is a major problem
when performing fault tolerance [3], or other large-scale testing campaigns.
Luckily, recent studies have shown that only a tiny number of the injected
faults are useful [9, 31, 30], suggesting that a handful number of injected
faults should be sufficient to perform testing. Though, it remains an open
question on how to identify them.

We fill this gap, by using bug report-driven fault injection. In essence we
leverage IR-based fault localization techniques to identify the locations
where fault injection should happen, i.e., locations relevant to the targeted
functionality described in the bug report, and apply frequent fault patterns
to produce mutants that behave similar to real faults.

###  II-C Fix Patterns

In automated program repair [32], a common way to generate patches is to apply
fix patterns [33] (also named fix templates [34] or program transformation
schemes [35]) in suspicious program locations (detected by fault
localization). Patterns used in the literature [33, 36, 37, 34, 35, 38, 39,
40, 41] have been defined manually or automatically (mined from bug fix
datasets).

Instead of fix patterns, we use _fault patterns_ that are fix patterns
inverted. Since fix patterns were designed using recurrent faults their
related fault patterns introduce them. This helps injecting faults that are
similar to those described in the bug reports. iBiR inverts and uses the
patterns implemented by _TBar_ [42] as we detail in the following Section.

Figure 1: The iBiR fault injection workflow.

##  III Approach

We propose iBiR, the first fault injection approach that utilizes information
extracted from bug reports to emulate real faults. A high level view of the
way iBiR works is shown in Figure 1. Our approach takes as input (1) the
source code of the program of interest and (2) a resolved bug report of that
program, written in natural language. The objective is to inject artificial
faults in the program (one by one, creating multiple faulty versions of the
program) that imitate the original bug. To do so, iBiR proceeds in three
steps.

First step: iBiR identifies relevant locations to inject the faults. It
applies IR-based fault localization to determine, from the bug report, the
code locations (statements) that are likely to be relevant to the target
fault. These locations are ranked according to their likelihood to be the
feature described by the bug report, hence are relevant to inject faults.

Second step: iBiR applies fault patterns on the identified code locations. We
build our patterns by inverting fix patterns used in automated program repair
approaches [42]. Our intuition is that, since fix patterns are used to fix
bugs, inverted patterns may introduce a fault similar to the original bug. For
each location, we apply only patterns that are syntactically compatible with
the code location. This step yields a set of faults to inject, i.e., pairs
composed of a location and a pattern.

Third step: our method ranks the location-pattern pairs wrt. the location
likelihood and priority order of the patterns. Then iBiR takes each pair (in
order) and applies the pattern to the location, injecting a fault in the
program. We repeat the process until the desired number of injected faults has
been produced or until all location-pattern pairs have been considered.

###  III-A Bug Report driven Fault Localization

IR-based fault localization (IRFL) [43, 44] leverages potential similarity
between the terms used in a bug report and the program source code to identify
relevant buggy code locations. It typically starts by extracting tokens from a
given bug report to formulate a query to be matched in a search space of
documents formed by the collections of source code files and indexed through
tokens extracted from source code [10, 26, 45, 24, 25, 23]. IRFL approaches
then rank the documents based on a probability of relevance. Top-ranked files
are likely to contain the buggy code.

We follow the same principle to identify promising locations where to inject
realistic faults. We rely on the information contained in the bug report to
localize the code location with the highest similarity score. Most IRFL
techniques have focused on file-level localization, which is too coarse-
grained for our purpose of injecting fault. Thus, we rather use a statement-
level IRFL approach that has been successfully applied to support program
repair [12].

It is to be noted that, contrary to program repair, we do not aim to identify
the exact bug location. We are rather interested in locations that allow
injecting realistic faults (similar to the bug). This means that IRFL may
pinpoint multiple locations of interest for fault injection even if those were
not buggy code locations.

###  III-B Fault patterns

TABLE I: iBIr fault injection patterns. Pattern context category | Bug injection pattern | example input | example output
---|---|---|---
Insert Statement |  | Insert a method call,
---
before or after the localised statement.
someMethod(expression); |  | someMethod(expression);
---
method(expression);
|  | Insert a return statement,
---
before or after the localised statement.
statement; |  | statement;
---
return VALUE;
| Wrap a statement with a try-catch. | statement; |  |  try{
---
statement;
} catch (Exception e){ … }
|  | Insert an if checker: wrap a
---
statement with an if block.
statement; |  |  if (conditional_exp) {
---
statement; }
Mutate Class Instance Creation |  | Replace an instance creation call by
---
a cast of the super.clone() method call.
… new T(); | … (T) super.clone();
Mutate Conditional Expression | Remove a conditional expression. | condExp1 && condExp2 | condExp1
| Insert a conditional expression. | condExp1 | condExp1 && condExp2
| Change the conditional operator. | condExp1 && condExp2 | condExp1 $||$ condExp2
Mutate Data Type | Change the declaration type of a variable. | T1 var …; | T2 var …;
| Change the casting type of an expression. | … (T1) expression …; | … (T2) expression …;
Mutate float or double Division | Remove a float or a double cast | … dividend / (float) divisor …; | … dividend / divisor …;
| from the divisor. | … intVarExp / 10d …; | … intVarExp / 10 …;
| Remove a float or a double cast | … (float) dividend / divisor …; | … dividend / divisor …;
| from the dividend. | … 1.0 / var …; | … 1 / var …;
| Replace float or double multiplication | … (1.0 / divisor) * dividend … | … dividend / divisor …;
| by an int division. | … 0.5 * intVarExp …; | … intVarExp / 2 …;
Mutate Literal Expression |  | Change boolean, number or string
---
literals in a statement by another literal
or expression of the same type.
| … string_literal1 …
---
… int_literal …
| … string_literal2 …
---
… int_expression …
Mutate Method Invocation | Replace a method call by another one. | … method1(args) … | … method(args) …
| Replace a method call argument by another one. | … method(arg1, arg2) … | … method(arg1, arg3) …
| Remove a method call argument. | … method(arg1, arg2) … | … method(arg1) …
| Add an argument to a method call | … method(arg1) … | … method(arg1, arg2) …
Mutate Return Statement | Replace a return experession by an other one. | return expr1; | return exp2;
Mutate Variable |  | Replace a variable by another variable
---
or an expression of the same type.
| … var1 …
---
… var1 …
| … var2 …
---
… exp …
Move Statement | Move a statement to another position. |  | statement;
---
…
| …
---
statement;
Remove Statement | Remove a statement. |  | statement;
---
…
…
| Remove a method. |  method(args){ statement; } | …
Mutate Operators | Replace an Arithmetic operator. | … a + b … | … a - b …
| Replace an Assignment operator. | … c += b … | … c -= b …
| Replace a Relational operator. | … a <b … | … a >b …
| Replace a Conditional operator. | … a && b … | … a $||$ b …
| Replace a Bitwise or a Bit Shift operator. | … a & b … | … a $|$ b …
| Replace an Unary operator. | a++ | a--
| Change arethmetic operations order. | a + b * c | c + b * a

We start from the fix patterns developed in TBar [42], a state of the art
pattern-based program repair tool. Any pattern is described by a context,
i.e., an AST node type to which the pattern applies, and a recipe, a
syntactical modification to be performed. For each pattern, we define a
related fault injection pattern that represents the inverse of that pattern.
For instance, inverting the fix pattern that consists of adding an arbitrary
statement yields a _remove statement_ fault pattern. Interestingly, some fix
patterns are symmetric in the sense that their inverse pattern is also a fix
pattern, e.g., inverting a Boolean connector. These patterns can thus be used
for both bug fixing and fault injection. Table I enumerates the resulting set
of fault injection patterns used by our approach.

Given a location (code statement) to inject a fault into, we identify the
patterns that can be applied to the statement. To do so, our method starts
from the AST node of the statement and visits it exhaustively, in a breadth-
first manner. Each time it meets an AST node that matches the context of a
fault pattern, it memorizes the node and the pattern for later application.
Then the method continues until it has visited all AST nodes under the
statement node. This way, we enumerate all possible applications of all fault
patterns onto the location.

Since more than one pattern may apply to a given location, we prioritize them
by leveraging heuristic priority rules previously defined in automated program
repair methods (these were inferred from real-world bug occurrences [42]).
This means that every fault injection pattern gets the priority order of its
inverse fix pattern.

###  III-C Fault injection

The last step consists of applying, one by one, the fault patterns to inject
faults at the program locations identified by IRFL. Locations of higher
ranking are considered first. Within a location, pattern applications are
ordered based on the pattern priority. By applying a pattern to a
corresponding AST node of the location, we inject a fault within the program
before recompiling it. If the program does not compile, we discard the fault
and restart with the next one. We continue the process until it reaches the
desired number of (compilable) injected faults or all locations and patterns
have been considered.

##  IV Research Questions

Our approach aims at injecting faults that imitate real ones by leveraging the
information included in bug reports. Therefore, a natural question to ask is
how well iBiR’s faults imitate the targeted (real) ones. Thus, we ask:

RQ1

_(Imitating bugs):_ Are the iBiR faults capable of emulating, in terms of
semantic similarity, the targeted (real) ones?

To answer this question, we check whether any of the injected faults imitate
well the targeted ones. Following the recommendations from the mutation
testing literature [9] we approximate the program behaviour through the
project test suites and compare the behaviour similarity of the test cases
w.r.t. their pass and failing status using the Ochiai similarity coefficient.
This is a typical way of computing the semantic similarity of mutants and
faults in mutation-based fault localization [11, 46].

We then turn our attention to the similarity of the injected fault sets and
contrast them with mutants such as those used by modern mutation testing tools
[14]. Hence we ask:

RQ2

_(Comparison with mutation testing):_ How does iBiR compare with mutation
testing, in terms of semantic similarity?

We answer this question by injecting mutants using the standard operators
employed by mutation testing tools [14] and measuring their semantic
similarity with the targeted faults. To make a fair comparison, we inject the
same number of faults per target. For iBiR we selected the top-ranked mutants
while for mutation testing we randomly sampled mutants across the entire
project codebase. Random mutant sampling forms our baseline since it performs
comparably to the alternative mutant selection methods [30, 8]. Also, since we
are interested in the relative differences between the injected fault sets, we
repeat our experiments multiple times using the same number of faults
(mutants).

Our approach identifies the locations where bugs should be injected through an
IR-based fault localization method. This may give significant advantages when
applied at the project level, but these may not carry on individual classes.
Such class level granularity level may be well suited for some test evaluation
tasks, such as automatic test generation [47]. To account for this, we
performed mutation testing (using the traditional mutation operators) at the
targeted classes (classes where the faults were fixed). To make a fair
comparison we also restricted iBiR to the same classes and compared the same
number of mutants. This leads us to the following question:

RQ3

_(Comparison at the target class):_ How does iBiR compare with mutation
testing, in terms of semantic similarity, when restricted to particular
classes?

We answer this question by injecting faults in only the target classes using
the iBiR bug patterns and the traditional mutation operators. Then we compare
the two approaches the same way as we did in RQ1 and RQ2.

Up to this point, the answers to the posed questions provide evidence that
using our approach yields mutants that are semantically similar to the
targeted bugs. Although, this is important and demonstrates the potential of
our approach, it does not necessarily mean that the injected faults are
strongly coupled with the real ones222Mutants are coupled with real faults if
they are killed only by test cases that also reveal the real faults. Mutant
and fault coupling is an important property for mutants that significantly
helps testing [48]. Therefore, we seek to investigate:

RQ4

_(Mutant and fault coupling):_ How does iBiR compare with mutation testing
with respect to mutant and fault coupling?

To answer this question we check whether the faults that we inject are
detected only by the failing tests, i.e., only by the tests that also reveal
the target fault. Compared to similarity metrics, this coupling relation is
stricter and stronger.

After answering the above questions we turn our attention to the actual use of
mutants in test effectiveness evaluations. Therefore, we are interested in
checking the correlations between the failure rates of the sets of the
injected faults we introduce and the real ones. To this end, we ask:

RQ5

_(Failure estimates):_ Are the injected faults leading to failure estimates
that are representative of the real ones? How do these estimates compare with
mutation testing?

The difference of RQ5 from the other RQs is that in RQ5, a set of injected
faults is evaluated while, in the previous RQs only isolated mutant instances.

##  V Experimental Setup

###  V-A Dataset & Benchmark

To evaluate iBiR we needed a set of benchmark programs, faults and bug
reports. We decided to use Defects4J [49] since it is a benchmark that
includes real-world bugs and it is quite popular in software engineering
literature.

####  V-A1 Linking the bugs with their related reports

To identify which bug report describes a given bug in the Defects4J, we
followed the same process as in the study of Koyuncu et al. [12].
Unfortunately, it was not possible to link the bug reports with the defects
for the Joda-Time, JFreeChart and Closure because their repositories and issue
tracking systems have been migrated into GitHub without any mapping of the bug
report identifiers. This means that in these projects the bug identifiers that
were used in the commit are meaningless. We therefore decided to ignore these
projects in an attempt to make our evaluation data as clean as possible.

For the Lang and Math projects, we used the bug linking strategies that are
implemented in the Jira issue tracking software and used the approach of
Fischer et al. [50] and Thomas et al. [51] to map the sought bugs with the
corresponding reports. Precisely, we crawled the relevant bug reports and
checked their links. We selected bug reports that were tagged as “BUG” and
marked as “RESOLVED” or “FIXED” and have a “CLOSED” status. Then we searched
the commit logs to identify related identifiers (IDs) that link the commits
with the corresponding bug.

Our resulting bug dataset included the 171 faults of Defect4J related to the
Lang and Math projects. We discarded  10 defects because they had a bug report
with undesired status in the bug tracking system, or there were issues with
the buggy program versions such as missing files from the repository at the
reporting time. We also discarded another  4 defects because iBiR generated
less than 5 mutants in total. This leaves us with 157 faults.

###  V-B Experimental Procedure

To compare the fault injection techniques we need to set a common basis for
comparison. We set this basis as the number of injected faults since it forms
a standard cost metric [52] that puts the studied methods under the same cost
level. We used sets of 5, 10, 30, and 100 injected faults since our aim is to
equip researchers with few representative faults, per targeted fault, in order
to reach reasonable execution demands.

To measure how well the injected faults imitate the real ones (answer RQ1, RQ2
and RQ3) we use a semantic similarity metric (Ochiai coefficient) between the
test failures on the injected and real (targeted) faults. This coefficient
quantifies the similarity level of the program behaviours exercised by the
test suites and is often used in mutation testing literature [9]. The metric
takes values in the range [0, 1] with 0 indicating complete difference and 1
exact match. We treated the injected faults that were not detected by any of
the test suites as equivalent mutants [53, 54]. This choice does not affect
our results since we approximate the program behaviours through the projects
test suites, i.e., they are never killed.

To measure whether the injected faults couple with the existing ones (answer
RQ4), we followed the process suggested by Just et al. [48] and identified
whether there were any injected faults that were killed by at least one
failing test (test that detects the real fault) and not by any passing test
(test that does not detect the real fault). In RQ5 we randomly sampled 50 test
suites, subsets of the accompanied test suites, that included between  10% to
30% test cases of the original test suite and recorded the ratios of the
injected faults that are detected when injecting 5, 10, 30 and 100 faults. We
also recorded binary variables indicating whether or not each test suite
detects the targeted fault. This process simulates cases where test suites of
different strengths are compared. Based on these data, we computed two
statistical correlation coefficients, the Kendall and Pearson.

To further validate whether the two approaches provide sufficient indicators
on the effectiveness of the test suites, we check whether the detection ratios
of the injected faults are statistically higher when test suites detect the
targeted faults than when they do not.

To reduce the influence of stochastic effects we used the Wilcoxon test with a
significance level of 0.05. This helped deciding whether the differences we
observe can be characterised as statistically significant. Statistical
significance does not imply sizable differences and thus, we also used the
Vargha Delaney effect size $\hat{\text{A}}_{12}$ [55]. In essence, the
$\hat{\text{A}}_{12}$ values quantify the level of the differences. For
instance, a value $\hat{\text{A}}_{12}=0.5$ can be interpreted as a tendency
of equal value of the two samples. $\hat{\text{A}}_{12}>0.5$ suggest that the
first set has higher values, while $\hat{\text{A}}_{12}<0.5$ suggest the
opposite.

###  V-C Implementation

To perform our experiments we set the following parameters in our framework:
First, we limit the IR fault localization on the 20 top ranked suspicious
files, per bug report. We then searched them for the exact statements where to
inject faults. We also ensured that the IR engine is not trained with bug
reports that we aim to localize. Second, for the mutation testing, denoted as
“Mutation” in our experiments, we used randomly sampled mutants from those
produced by typical mutation operators, coming from mutation testing
literature. In particular we implemented the muJava intra-method mutation
operators [56], which are the most frequently used [14]. Third to reduce the
noise from stillborn mutants, i.e., mutants that do not compile, we discarded
without taking into any consideration, i.e., prior to our experiment, every
mutant that did not compile or its execution with the test suite exceeded a
timeout of  5 minutes. Fourth, when answering the RQ3, we found out that there
were many cases where iBiR injected less than 100 faults. To perform a fair
comparison, we discarded these cases (for both approaches). This means that we
always report results where both studied approaches manage to inject the same
number of faults.

##  VI Results

###  VI-A RQ1: Semantic similarity between injected and real faults

Figure 2: Distribution of semantic similarities of 100 injected faults per
targeted (real) fault.  Figure 3: Semantic similarity per targeted (real)
fault, top values. iBiR injects faults with higher similarity coefficients
than mutation testing.

To check whether the injected faults imitate well the targeted ones, we
measured their behaviour (semantic) similarity w.r.t. the project test suites
(please refer to Section  V for details). Figure  2 shows the distribution of
the similarity coefficient values that were recorded in our study. As can be
seen, iBiR injects hundreds of faults that are similar to real ones, whereas
mutation (denoted as  Mutation in Figure  2) did not manage to generate any.
At the same time, as typically happens in mutation testing [9], a large number
of injected faults have low similarity. This is evident in our data, where
mutations have 0 similarity.

To investigate whether iBiR successfully injects any fault that is similar
(semantically) to the targeted ones, we collected the best similarity
coefficients, per targeted fault, when injecting 5, 10, 30 and 100 faults.
Figure  3 shows the distribution of these results. For more than half of the
targeted faults, iBiR yields a best similarity value higher than 0.5, when
injecting 100 faults, indicating that iBiR’s faults imitate relatively well
the targeted ones. We also observe that in many faults the best similarity
values are above 0 by injecting just 10 faults. This is important since it
indicates that iBiR successfully identifies relevant locations for fault
injection.

To establish a baseline and better understand the value of iBiR, we need to
contrast iBiR’s performance with that of mutation testing when injecting the
same number of faults. Mutation testing forms the current SoA of fault
injection and thus a related baseline. As can be seen from Figure  3, the
similarity values of mutation testing are significantly lower than those of
iBiR. In the following subsection we further compare iBiR with mutation
testing.

###  VI-B RQ2: iBiR Vs Mutation Testing

Figure 4: Semantic similarity of all injected faults. iBiR injects faults with
higher similarity coefficients than mutation testing.

Figure  4 shows the distribution of the semantic similarities, between real
and injected faults, when injecting 5, 10, 30 and 100 faults. As can be seen
from the boxplots, the trend is that a large portion of faults injected by
iBiR imitates the targeted ones, (at least much better than mutation testing).
Interestingly, in mutation testing, only outliers have their similarity above
0. In particular, mutation testing injected faults with similarity values
higher than 0 in  3, 8, 19, 40 of the targeted faults (when injecting 5, 10,
30, 100 faults), while iBiR injected in  75, 88, 101, 123 of the targeted
faults, respectively.

To validate this finding, we performed a statistical test (Wilcoxon paired
test) on the data of both figures  3 and  4 to check for significant
differences. Our results showed that the differences are significant,
indicating the low probability of this effect to be happening by chance. The
size of the difference is also big, with iBiR yielding $\hat{\text{A}}_{12}$
values between  0.73 and  0.84 indicating that iBiR injects faults with higher
semantic similarity to real ones in the great majority of the cases. Due to
the many cases with 0 similarity values, the average similarity values of
iBiR’s faults is  0.166, while for mutation it is  0.002, indicating the
superiority of iBiR.

###  VI-C RQ3: iBiR Vs Mutation Testing at particular classes

Figure 5: Semantic similarity of injected faults at particular classes. iBiR
injects faults with higher similarity coefficients than mutation testing.

To check the performance of iBiR at the class level of granularity we repeated
our analysis by discarding, from our priority lists, every mutant that is not
located on the targeted classes, i.e., classes where the targeted faults have
been fixed. Figure  5 shows the distribution of the semantic similarities when
injecting 5, 10, 30 and 100 faults at a particular class. As expected,
mutation testing scores are higher than those presented before, but still
mutation testing falls behind.

To validate this finding, we performed a statistical test and found that the
differences are significant. The size of the difference is 0.6, meaning that
iBiR score  60% times higher than mutation testing. The average similarity
values of the iBiR faults is  0.240, while for mutation is  0.114, indicating
that iBiR is better.

###  VI-D RQ4: Fault Coupling

Figure 6: Percentage of injected faults that are coupled to the real ones.

The coupling between the injected and the real faults forms a fundamental
assumption of the fault-based testing approaches [49]. An injected fault is
coupled to a real one when a test case that reveals the injected fault also
reveals the real fault [49]. This implies that revealing these coupled
injected faults results in revealing potential real ones. We therefore, check
this property in the faults we inject and contrast it with the baseline
mutation testing approach.

Figure  6 shows the percentage of targeted faults where there is at least one
injected fault that is coupled to a real one. This is shown for the scenarios
where 5, 10, 30 and 100 faults, per target, are injected. As we can see from
these data, iBiR injects coupled faults for approximately  16% of the target
faults when it aims at injecting 5 faults. This percentage increases to  36%
when the number of injected faults is increased to 100.

Perhaps surprisingly, mutation testing did not perform well (it injected
coupled faults for  less than 1% of the targeted, when injecting 100 faults
per target). These results differ from those reported by previous research
[48, 9], because a) previous research only injected faults at the faulty
classes and not the entire project and b) previous research injected all
possible mutant instances and not 100 as we do.

TABLE II: Vargha and Deianey $\hat{\text{A}}_{12}$ (iBiR VS Mutation) of Kendall and Pearson correlation coefficients.  Number of injected faults | 5 | 10 | 30 | 100
---|---|---|---|---
Kendall | 0.720 | 0.756 | 0.725 | 0.756
Pearson | 0.726 | 0.737 | 0.744 | 0.788

###  VI-E RQ5: Fault detection estimates

The results presented so far provide evidence that some of the injected faults
imitate well the targeted ones. Though, the question of whether the injections
provide representative results of real faults remains, especially since we
observe a large number of faults with low similarity value. Therefore, we
check the correlations between the failure rates of the sets of injected
faults and the real faults when executed with different test suites, (please
refer to section  V for details).

Figures  7 and  8 show the distribution of the correlation coefficients, when
injecting different numbers of faults. Interestingly, the results on both
figures show a trend in favour of iBiR. This difference is statistically
significant, shown by a Wilcoxon test, with an effect size of approximately
0.72. Table II records the effect size values, $\hat{\text{A}}_{12}$, for the
examined strategies. In essence, these effect sizes mean that iBiR outperforms
the mutant injection in 72% of the cases, suggesting that iBiR could be a much
better choice than mutation testing, especially in cases of large test suites
with expensive test executions.

Figure 7:  Kendall correlation coefficients of test suites (samples from the
original project test suite). The two related variables are a) the percentage
of injected faults that was detected by the sampled test suites and b) whether
the targeted fault was detected or not by the same test suites. Figure 8:
Pearson correlation coefficients of test suites (samples from the original
project test suite). The two related variables are a) the percentage of
injected faults that was detected by the sampled test suites and b) whether
the targeted fault was detected or not by the same test suites.

To further validate whether iBiR’s faults provide good indicators (estimates)
of test effectiveness (fault detection) we split our test suites between those
that detect the targeted faults and those that do not. We then tested whether
detection ratios of the injected faults in the test suite group that detects
the real faults are significantly (statistically) higher than those in the
group that does not detect it. In case this happens, we can conclude that test
suites capable of detecting a higher number of injected faults have similarly
higher chances to detect the real ones. This is important when comparing test
generation techniques, where the aim is to identify the most effective (at
detecting faults) technique.

Figure 9 records the number of faults where the test suites detecting the
(real) targeted fault also detect a statistically higher number of injected
faults than those test suites that do not detect it. As can be seen by these
results, iBiR has a big difference from mutation, i.e., it distinguishes
between passing and failing test suites in  80 faults, while Mutation in  21
faults. Since statistical significance does not imply practical significance,
we also measured the Vargha and Delaney $\hat{\text{A}}_{12}$ effect size
values on the same data, recorded in Figure 10. Of course it does not make
sense to contrast insignificant cases, so we only performed that on the
results where iBiR has statistically significant difference. Interestingly the
results demonstrate big differences (in approximately 80% of the cases) in
favour of our approach.

Figure 9: Number of faults where injected faults provided good indications of
fault detection. Particularly, number of cases with test suites detecting the
real fault have statistically significant difference, in terms of ratios of
injected faults detected, from those that do not detect the real fault. Figure
10: Vargha and Deianey values for iBiR. $\hat{\text{A}}_{12}$ values computed
on the detection ratios of injected faults of the test suites that detect and
do not detect the (real) faults.

##  VII Threats to Validity

The question of whether our findings generalise, forms a typical threat to
validity of empirical studies. To reduce this threat, we used real-world
projects, developer test suites, real faults and their associated bug reports,
from an established and independently built benchmark. Still though, we have
to acknowledge that these may not be representative of projects from other
domains or industrial systems.

Other threats may also arise from the way we handled the injected faults and
mutants that were not killed by any test case. We believe that this validation
process is sufficient since the test suites are relatively strong and somehow
form the current state of practice, i.e., developers tend to use this
particular level of testing. Though, in case the approach is putted into
practice things might be different. We also applied our analysis on the fixed
program version provided by Defects4J. This was important in order to show
that we actually inject the actual targeted faults. Though, our results might
not hold on the cases that the code has drastically changed since the time of
the bug report. We believe that this threat is not of actual importance as we
are concerned with fault injection at interesting program locations, which
should be pinpointed by the fault localization technique we use. Still future
research should shed some light on how useful these locations and faults are.

Finally, our evaluation metrics may induce some additional threats. Our
comparison basis measurement, i.e., number of injected faults, approximates
the execution cost of the techniques and their chances to provide misleading
guidance [9], while the fault couplings and semantic similarity metrics
approximate the effectiveness of the approaches. These are intuitive metrics,
used by previous research [30, 8] and aim at providing a common ground for
comparison.

##  VIII Related Work

Software fault injection [57] has been widely studied since 1970s. Injected
faults have been used for the purpose of testing [1], debugging [11, 58],
assessing fault tolerance [3], risk analysis [59, 60] and dependability
evaluation [61].

Despite the many years of research, the majority of previous research is
focused on the fault types. In mutation testing research, mutation operators
(fault types) are usually designed based on the grammar of the targeted
language [5, 1], which are then refined through empirical analysis, aiming at
reducing the redundancy between the injected faults [52, 62]. The most
prominent mutant selection approach is that of Offutt et al. [52], which
proposed a set of 5 mutation operators. This set has been incorporated in most
of the modern mutation testing tools [14] and is the one that we use in our
baseline.

Recently, Brown et al. [7] aimed at inferring fault patterns from bug fixes.
Their results showed that a large number of mutation operators could be
inferred. Along the same lines Tufano et al. [6] developed a neural machine
translation tool that learns to mutate through bug fixes. A key assumptions of
these methods are a) the availability of a comprehensive number of clean bug
fixing commits, and b) the absence of fault couplings [63], which are often
not met and can often be reduced to what simple mutations do. For instance,
the study of Brown et al. found that with few exceptions, almost all mutation
operators designed based on the C language grammar appeared in the inferred
operator set. Perhaps more importantly, the studies of Natella et al. [3] and
Chekam et al. [8] found that the pair of mutant location and type are what
makes mutants powerful and not the type itself. Nevertheless, iBiR goal is
complementary to the above studies as it aims at injecting faults that mimic
specifically targeted faults, those described in bug reports. This way, one
can inject the most important and severe faults experienced.

Some studies attempt to identify the program locations where to inject faults.
Sun et al. [64] suggested injecting faults in diverse places within different
program execution paths. Gong et al. [65] used graph analysis to inject faults
in different and diverse locations of the program spectra. Mirshokraie et al.
[66] employed complexity metrics together with actual program executions to
inject faults at places with good observability. These strategies, aim at
reducing the number of injected faults and not to mimic any real fault as our
approach. Moreover, their results should be resembled by the random mutant
sampling baseline that we use.

Random mutant sampling forms a natural cost-reduction method proposed since
the early days of mutation testing [2]. Despite that, most of the mutant
selection methods fail to perform better than it. Recently, Kurtz et al. [30]
and Chekam et al. [8] demonstrated that selective mutation and random mutant
sampling perform similarly. From this, it should be clear that despite the
advances in selective mutation, the simple random sampling is one of the most
effective fault injection techniques. This is the reason why we adopt it as a
baseline in our experiments.

Natella et al. [3] used complexity metrics as machine learning features and
applied them on a set of examples in order to identify (predict) which
injected faults have the potential to emulate well the behaviour of real ones.
Chekam et al. [8] also used machine learning, with many static mutant-related
features to select and rank mutants that are likely fault revealing (have high
chance to couple with a fault). These studies assume the availability of a
historical faults and do not aim at injecting specific faults as done by iBiR.

The relationship between injected and real faults has also received some
attention [1]. The studies of Papadakis et al. [9], Just et al. [48], Andrews
et al. [53] investigated whether mutant kills and fault detection ratios
follow similar trends. The results show the existence of a correlation and,
thus, that mutants can be used in controlled experiments as alternatives to
real faults. In the context of testing, i.e., using mutants to guide testing,
injected faults can help identifying corner cases and reveal existing faults.
The studies of Frankl et al. [67], Li et al. [68] and Chekam et al. [69]
demonstrated that guidance from mutants leads to significantly higher fault
revelation than that of other test techniques (test criteria).

##  IX Conclusion

We presented iBiR; a bug-report driven fault injection tool. iBiR (1) equips
researchers with faults (to inject) targeting the critical functionality of
the target systems, (2) mimics real faulty behaviour and (3) makes relevant
fault injection.

iBiR’s use case is simple; given a program and some carefully selected bug
reports, it injects faults emulating the related bugs, i.e., iBiR generates
few faults per target bug report. This allows constructing realistic fault
pools to be used for test or fault tolerance assessment.

This means that iBiR’s faults can be used as substitutes of real faults, in
controlled studies. In a sense, iBiR can bring the missing realism into fault
injection and therefore support empirical research and controlled experiments.
This is important since a large number of empirical studies rely on
artificially-injected faults [70], the validity of which is always in
question.

While the use case of iBiR is in research studies, the use of the tool can
have applications in a wide range of software engineering tasks. It can, for
instance, be used for asserting that future software releases do not introduce
the same (or similar) kind of faults. Such a situation occurs in large
software projects [71], where iBiR could help by checking for some of the most
severe faults experienced.

Another potential application of iBiR is fault tolerance assessment, by
injecting faults similar to previously experienced ones and analysing the
system responses and overall dependability.

Finally, testers could use iBiR for testing all system areas that could lead
to similar symptoms than the ones observed and resolved. This will bring
significant benefits when testing software clones [72] and similar
functionality implementations. We hope that we will address these points in
the near future.

## References

  * [1] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. L. Traon, and M. Harman, “Chapter six - mutation testing advances: An analysis and survey,” _Advances in Computers_ , vol. 112, pp. 275–378, 2019. [Online]. Available: https://doi.org/10.1016/bs.adcom.2018.03.015
  * [2] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” _IEEE Computer_ , vol. 11, no. 4, pp. 34–41, 1978. [Online]. Available: https://doi.org/10.1109/C-M.1978.218136
  * [3] R. Natella, D. Cotroneo, J. Durães, and H. Madeira, “On fault representativeness of software fault injection,” _IEEE Trans. Software Eng._ , vol. 39, no. 1, pp. 80–96, 2013. [Online]. Available: https://doi.org/10.1109/TSE.2011.124
  * [4] A. Lanzaro, R. Natella, S. Winter, D. Cotroneo, and N. Suri, “Error models for the representative injection of software defects,” in _Software Engineering & Management 2015, Multikonferenz der GI-Fachbereiche Softwaretechnik (SWT) und Wirtschaftsinformatik (WI), FA WI-MAW, 17. März - 20. März 2015_, ser. LNI, vol. P-239. GI, 2015, pp. 118–119.
  * [5] P. Ammann and J. Offutt, _Introduction to Software Testing_. Cambridge University Press, 2008. [Online]. Available: https://doi.org/10.1017/CBO9780511809163
  * [6] M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “Learning how to mutate source code from bug-fixes,” in _2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019_. IEEE, 2019, pp. 301–312. [Online]. Available: https://doi.org/10.1109/ICSME.2019.00046
  * [7] D. B. Brown, M. Vaughn, B. Liblit, and T. W. Reps, “The care and feeding of wild-caught mutants,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017_. ACM, 2017, pp. 511–522. [Online]. Available: https://doi.org/10.1145/3106237.3106280
  * [8] T. T. Chekam, M. Papadakis, T. F. Bissyandé, Y. L. Traon, and K. Sen, “Selecting fault revealing mutants,” _Empirical Software Engineering_ , vol. 25, no. 1, pp. 434–487, 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09778-7
  * [9] M. Papadakis, D. Shin, S. Yoo, and D. Bae, “Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults,” in _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018_ , 2018, pp. 537–548. [Online]. Available: https://doi.org/10.1145/3180155.3180183
  * [10] J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,” in _Proceedings of the 2012 International Conference on Software Engineering (ICSE)_ , 2012, pp. 14–24.
  * [11] M. Papadakis and Y. Le Traon, “Metallaxis-fl: mutation-based fault localization,” _Software Testing, Verification and Reliability_ , vol. 25, no. 5-7, pp. 605–628, 2015.
  * [12] A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, M. Monperrus, J. Klein, and Y. L. Traon, “iFixR: Bug report driven program repair,” in _Proceedings of the 13th Joint Meeting on Foundations of Software Engineering (FSE)_ , 2019.
  * [13] M. Papadakis, T. T. Chekam, and Y. L. Traon, “Mutant quality indicators,” in _2018 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops, Västerås, Sweden, April 9-13, 2018_. IEEE Computer Society, 2018, pp. 32–39. [Online]. Available: http://doi.ieeecomputersociety.org/10.1109/ICSTW.2018.00025
  * [14] M. Kintis, M. Papadakis, A. Papadopoulos, E. Valvis, N. Malevris, and Y. L. Traon, “How effective are mutation testing tools? an empirical analysis of java mutation testing tools with manual analysis and real faults,” _Empir. Softw. Eng._ , vol. 23, no. 4, pp. 2426–2463, 2018. [Online]. Available: https://doi.org/10.1007/s10664-017-9582-5
  * [15] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” _IEEE Transactions on Software Engineering_ , vol. 42, no. 8, pp. 707–740, 2016.
  * [16] W. E. Wong, V. Debroy, and B. Choi, “A family of code coverage-based heuristics for effective fault localization,” _Journal of Systems and Software_ , vol. 83, no. 2, pp. 188–208, 2010.
  * [17] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “Spectrum-based multiple fault localization,” in _Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2009, pp. 88–99.
  * [18] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” in _Proceedings of the 26th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)_ , 2005, pp. 15–26.
  * [19] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman, “Indexing by latent semantic analysis,” _Journal of the American Society for Information Science_ , vol. 41, no. 6, pp. 391–407, Sep. 1990.
  * [20] W. B. Frakes and R. Baeza-Yates, _Information Retrieval: Data Structures and Algorithms_ , 1st ed. Prentice Hall, Jun. 1992.
  * [21] C. D. Manning and H. Schütze, _Foundations of Statistical Natural Language Processing_ , 1st ed. Cambridge, Mass: The MIT Press, Jun. 1999.
  * [22] G. Salton and M. J. McGill, _Introduction to Modern Information Retrieval_. New York, NY, USA: McGraw-Hill, Inc., 1986.
  * [23] S. K. Lukins, N. A. Kraft, and L. H. Etzkorn, “Bug localization using latent Dirichlet allocation,” _Information and Software Technology_ , vol. 52, no. 9, pp. 972–990, 2010.
  * [24] R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” in _Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2013, pp. 345–355.
  * [25] S. Wang and D. Lo, “Version History, Similar Report, and Structure: Putting Them Together for Improved Bug Localization,” in _Proceedings of the 22nd International Conference on Program Comprehension (ICPC)_ , 2014, pp. 53–63.
  * [26] M. Wen, R. Wu, and S.-C. Cheung, “Locus: Locating bugs from software changes,” in _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2016, pp. 262–273.
  * [27] K. C. Youm, J. Ahn, J. Kim, and E. Lee, “Bug Localization Based on Code Change Histories and Bug Reports,” in _Proceedings of the 2015 Asia-Pacific Software Engineering Conference (ICSE)_ , 2015, pp. 190–197.
  * [28] J. Andrews, L. Briand, Y. Labiche, and A. Namin, “Using mutation analysis for assessing and comparing testing coverage criteria,” _Software Engineering, IEEE Transactions on_ , vol. 32, no. 8, pp. 608–624, 2006.
  * [29] T. Laurent, M. Papadakis, M. Kintis, C. Henard, Y. L. Traon, and A. Ventresque, “Assessing and improving the mutation testing practice of pit,” in _2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)_ , March 2017, pp. 430–435.
  * [30] B. Kurtz, P. Ammann, J. Offutt, M. E. Delamaro, M. Kurtz, and N. Gökçe, “Analyzing the validity of selective mutation with dominator mutants,” in _Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016_ , 2016, pp. 571–582. [Online]. Available: https://doi.org/10.1145/2950290.2950322
  * [31] P. Ammann, M. E. Delamaro, and J. Offutt, “Establishing theoretical minimal sets of mutants,” in _2014 IEEE Seventh International Conference on Software Testing, Verification and Validation_. IEEE, 2014.
  * [32] C. L. Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” _Commun. ACM_ , vol. 62, no. 12, pp. 56–65, 2019. [Online]. Available: https://doi.org/10.1145/3318162
  * [33] D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” in _Proceedings of the 35th ICSE_. IEEE, 2013, pp. 802–811.
  * [34] K. Liu, D. Kim, T. F. Bissyandé, S. Yoo, and Y. Le Traon, “Mining fix patterns for findbugs violations,” _IEEE Transactions on Software Engineering_ , 2018.
  * [35] J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Towards practical program repair with on-demand candidate generation,” in _Proceedings of the 40th International Conference on Software Engineering (ICSE)_ , 2018, pp. 12–23.
  * [36] R. K. Saha, Y. Lyu, H. Yoshida, and M. R. Prasad, “Elixir: Effective object-oriented program repair,” in _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2017, pp. 648–659.
  * [37] T. Durieux, B. Cornu, L. Seinturier, and M. Monperrus, “Dynamic patch generation for null pointer exceptions using metaprogramming,” in _Proceedings of the 24th SANER_. IEEE, 2017, pp. 349–358.
  * [38] A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “Fixminer: Mining relevant fix patterns for automated program repair,” _Empirical Software Engineering_ , October 2019.
  * [39] M. Martinez and M. Monperrus, “Ultra-large repair search space with automatically mined templates: The cardumen mode of astor,” in _Proceedings of the 10th SSBSE_. Springer, 2018, pp. 65–86.
  * [40] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Avatar: Fixing semantic bugs with fix patterns of static analysis violations,” in _Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2019, pp. 1–12.
  * [41] K. Liu, D. Kim, T. F. Bissyandé, S. Yoo, and Y. Le Traon, “Mining fix patterns for findbugs violations,” _TSE_ , 2018.
  * [42] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “TBar: Revisiting template-based automated program repair,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_ , 2019, pp. 31–42.
  * [43] C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” in _Proceedings of the 20th ISSTA_. ACM, 2011, pp. 199–209.
  * [44] Q. Wang, C. Parnin, and A. Orso, “Evaluating the usefulness of ir-based fault localization techniques,” in _Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA)_ , 2015, pp. 1–11.
  * [45] C.-P. Wong, Y. Xiong, H. Zhang, D. Hao, L. Zhang, and H. Mei, “Boosting Bug-Report-Oriented Fault Localization with Segmentation and Stack-Trace Analysis,” in _Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2014, pp. 181–190.
  * [46] S. Moon, Y. Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating faulty programs for fault localization,” in _Seventh IEEE International Conference on Software Testing, Verification and Validation, ICST 2014_. IEEE Computer Society, 2014, pp. 153–162. [Online]. Available: https://doi.org/10.1109/ICST.2014.28
  * [47] G. Fraser and A. Arcuri, “Whole test suite generation,” _IEEE Trans. Software Eng._ , vol. 39, no. 2, pp. 276–291, 2013. [Online]. Available: https://doi.org/10.1109/TSE.2012.14
  * [48] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?” in _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014_ , 2014, pp. 654–665. [Online]. Available: https://doi.org/10.1145/2635868.2635929
  * [49] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” in _Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA)_ , 2014, pp. 437–440.
  * [50] M. Fischer, M. Pinzger, and H. C. Gall, “Populating a release history database from version control and bug tracking systems,” in _19th International Conference on Software Maintenance (ICSM 2003), The Architecture of Existing Systems, 2003_. IEEE Computer Society, 2003, p. 23. [Online]. Available: https://doi.org/10.1109/ICSM.2003.1235403
  * [51] S. W. Thomas, M. Nagappan, D. Blostein, and A. E. Hassan, “The impact of classifier configuration and classifier combination on bug localization,” _IEEE Trans. Software Eng._ , vol. 39, no. 10, pp. 1427–1443, 2013. [Online]. Available: https://doi.org/10.1109/TSE.2013.27
  * [52] A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf, “An experimental determination of sufficient mutant operators,” _ACM Trans. Softw. Eng. Methodol._ , vol. 5, no. 2, pp. 99–118, 1996. [Online]. Available: https://doi.org/10.1145/227607.227610
  * [53] J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin, “Using mutation analysis for assessing and comparing testing coverage criteria,” _IEEE Trans. Software Eng._ , vol. 32, no. 8, pp. 608–624, 2006. [Online]. Available: https://doi.org/10.1109/TSE.2006.83
  * [54] M. Papadakis, M. E. Delamaro, and Y. L. Traon, “Mitigating the effects of equivalent mutants with mutant classification strategies,” _Sci. Comput. Program._ , vol. 95, pp. 298–319, 2014. [Online]. Available: https://doi.org/10.1016/j.scico.2014.05.012
  * [55] A. Vargha and H. D. Delaney, “A critique and improvement of the cl common language effect size statistics of mcgraw and wong,” _Journal of Educational and Behavioral Statistics_ , vol. 25, no. 2, pp. 101–132, 2000.
  * [56] Y. Ma, J. Offutt, and Y. R. Kwon, “Mujava: an automated class mutation system,” _Softw. Test. Verification Reliab._ , vol. 15, no. 2, pp. 97–133, 2005. [Online]. Available: https://doi.org/10.1002/stvr.308
  * [57] J. M. Voas and G. McGraw, _Software Fault Injection: Inoculating Programs against Errors_. USA: John Wiley & Sons, Inc., 1997.
  * [58] Y. Lou, A. Ghanbari, X. Li, L. Zhang, H. Zhang, D. Hao, and L. Zhang, “Can automated program repair refine fault localization? a unified debugging approach,” in _ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020_. ACM, 2020, pp. 75–87. [Online]. Available: https://doi.org/10.1145/3395363.3397351
  * [59] J. Christmansson and R. Chillarege, “Generation of error set that emulates software faults based on field data,” in _Digest of Papers: FTCS-26, The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing, 1996_. IEEE Computer Society, 1996, pp. 304–313. [Online]. Available: https://doi.org/10.1109/FTCS.1996.534615
  * [60] J. M. Voas, F. Charron, G. McGraw, K. W. Miller, and M. Friedman, “Predicting how badly ”good” software can behave,” _IEEE Softw._ , vol. 14, no. 4, pp. 73–83, 1997. [Online]. Available: https://doi.org/10.1109/52.595959
  * [61] J. Arlat, A. Costes, Y. Crouzet, J. Laprie, and D. Powell, “Fault injection and dependability evaluation of fault-tolerant systems,” _IEEE Trans. Computers_ , vol. 42, no. 8, pp. 913–923, 1993. [Online]. Available: https://doi.org/10.1109/12.238482
  * [62] M. Marcozzi, S. Bardin, N. Kosmatov, M. Papadakis, V. Prevosto, and L. Correnson, “Time to clean your test objectives,” in _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, pp. 456–467. [Online]. Available: https://doi.org/10.1145/3180155.3180191
  * [63] A. J. Offutt, “Investigations of the software testing coupling effect,” _ACM Trans. Softw. Eng. Methodol._ , vol. 1, no. 1, pp. 5–20, 1992. [Online]. Available: https://doi.org/10.1145/125489.125473
  * [64] C. Sun, F. Xue, H. Liu, and X. Zhang, “A path-aware approach to mutant reduction in mutation testing,” _Information & Software Technology_, vol. 81, pp. 65–81, 2017. [Online]. Available: https://doi.org/10.1016/j.infsof.2016.02.006
  * [65] D. Gong, G. Zhang, X. Yao, and F. Meng, “Mutant reduction based on dominance relation for weak mutation testing,” _Information & Software Technology_, vol. 81, pp. 82–96, 2017. [Online]. Available: https://doi.org/10.1016/j.infsof.2016.05.001
  * [66] S. Mirshokraie, A. Mesbah, and K. Pattabiraman, “Guided mutation testing for javascript web applications,” _IEEE Trans. Software Eng._ , vol. 41, no. 5, pp. 429–444, 2015. [Online]. Available: https://doi.org/10.1109/TSE.2014.2371458
  * [67] P. G. Frankl, S. N. Weiss, and C. Hu, “All-uses vs mutation testing: An experimental comparison of effectiveness,” _J. Syst. Softw._ , vol. 38, no. 3, pp. 235–253, 1997. [Online]. Available: https://doi.org/10.1016/S0164-1212(96)00154-9
  * [68] N. Li, U. Praphamontripong, and J. Offutt, “An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage,” in _Second International Conference on Software Testing Verification and Validation, ICST, 2009, Workshops Proceedings_. IEEE Computer Society, 2009, pp. 220–229. [Online]. Available: https://doi.org/10.1109/ICSTW.2009.30
  * [69] T. T. Chekam, M. Papadakis, Y. L. Traon, and M. Harman, “An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption,” in _Proceedings of the 39th International Conference on Software Engineering, ICSE 2017_ , 2017, pp. 597–608. [Online]. Available: https://doi.org/10.1109/ICSE.2017.61
  * [70] M. Papadakis, C. Henard, M. Harman, Y. Jia, and Y. L. Traon, “Threats to the validity of mutation-based test assessment,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016_ , 2016, pp. 354–365. [Online]. Available: https://doi.org/10.1145/2931037.2931040
  * [71] N. Palix, G. Thomas, S. Saha, C. Calvès, J. Lawall, and G. Muller, “Faults in linux: Ten years later,” in _Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems_ , 2011, pp. 305–318.
  * [72] M. Mondal, M. S. Rahman, R. K. Saha, C. K. Roy, J. Krinke, and K. A. Schneider, “An empirical study of the impacts of clones in software maintenance,” in _The 19th IEEE International Conference on Program Comprehension, ICPC 2011_. IEEE Computer Society, 2011, pp. 242–245. [Online]. Available: https://doi.org/10.1109/ICPC.2011.14