# How Does Regression Test Selection Affect Program Repair?
An Extensive Study on 2 Million Patches

Yiling Lou  Department of Computer SciencePeking UniversityBeijing, China
<EMAIL_ADDRESS>,  Samuel Benton  Department of Computer Science The
University of Texas at DallasTX, USA<EMAIL_ADDRESS>,  Dan Hao  ,
Lu Zhang  Department of Computer SciencePeking UniversityBeijing, China
haodan<EMAIL_ADDRESS>and  Lingming Zhang  Department of Computer
ScienceUniversity of Illinois at Urbana-ChampaignIL, USA<EMAIL_ADDRESS>
###### Abstract.

In the past decade, Automated Program Repair (APR) has been extensively
studied in academia and even drawn wide attention from industry. However, APR
techniques can be extremely time consuming since (1) a large number of patches
can be generated for a given bug, and (2) each patch needs to be executed on
the original tests to ensure its correctness. In the literature, various
techniques (e.g., based on learning, mining, and constraint solving) have been
proposed/studied to reduce the number of patches. However, there is limited
study on the impact of test selection for each patch (e.g., only the tests
affected by the patch need to be executed as the other tests would keep the
same outcomes and can be skipped), and few APR systems actually apply test
selection. Therefore, this paper conducts the first extensive study to
investigate the impact of Regression Test Selection (RTS) on APR. More
specifically, we implemented widely-used RTS techniques at different levels
(i.e., class/method/statement levels) for 12 state-of-the-art APR systems with
over 2M patches. Our study reveals various practical guidelines for future
APR, including: (1) the number of patches widely used for measuring APR
efficiency can incur skewed conclusions, and the use of inconsistent RTS
configurations can further skew the conclusion; (2) all studied RTS techniques
can substantially improve APR efficiency and should be considered in future
APR work; (3) method- and statement-level RTS outperform class-level RTS
substantially, and should be preferred; (4) RTS techniques can substantially
outperform state-of-the-art test prioritization techniques for APR, and
combining them can further improve APR efficiency; and (5) traditional
regression test prioritization widely studied in regression testing performs
even better than APR-specific test prioritization when combined with most RTS
techniques. Furthermore, we also present the detailed impact of different
patch categories and patch validation strategies on our findings.

††copyright: none

##  1\. Introduction

Software bugs can trigger unexpected program behaviors and result in fatal
consequences. To reduce manual debugging efforts, Automated Program Repair
(APR) (Xiong et al., 2017; Wen et al., 2018; Ghanbari et al., 2019; Jiang et
al., 2018; Yuan and Banzhaf, 2020; Koyuncu et al., 2020) has been proposed to
automatically generate patches for buggy programs with minimal human
intervention. Modern APR techniques often follow a generate-and-validate
procedure and leverage test suites as the partial specification for the
desired program behavior. In particular, a test-based APR system repeatedly
generates patches and validates them against the whole test suite until a
patch that can pass all tests is found. To date, a large number of APR
techniques have been proposed, effectively fixing a considerable number of
real bugs and improving software quality/productivity (Xiong et al., 2017; Wen
et al., 2018; Ghanbari et al., 2019; Jiang et al., 2018).

Although receiving wide attention from both academia and industry, APR
techniques can be extremely time consuming (Hua et al., 2018; Goues et al.,
2013; Long and Rinard, 2016; Weimer et al., 2013; Ghanbari et al., 2019; Liu
et al., 2020): (1) for a given bug, there are a vast number of patches
generated; (2) for each generated patch, it often takes non-trivial time to
execute the original tests for correctness validation. To reduce overheads in
the repair procedure, researchers have proposed various cost reduction
approaches. For example, numerous APR techniques have been proposed to reduce
the number of generated patches, including constraint-based (Xiong et al.,
2017; Martinez and Monperrus, 2018; Durieux and Monperrus, 2016), heuristic-
based (Yuan and Banzhaf, 2020; Jiang et al., 2018; Wen et al., 2020),
template-based (Liu et al., 2019b; Koyuncu et al., 2020; Liu et al., 2019a),
and learning-based techniques (Lutellier et al., 2020; Chen et al., 2019; Li
et al., 2020). Orthogonal to these optimizations, some recent APR work also
leveraged Regression Test Selection (RTS) to reduce the number of test
executions within each patch validation. RTS (Legunsen et al., 2016; Orso et
al., 2004; Gligoric et al., 2015; Rothermel and Harrold, 1997, 1993) has been
extensively studied in regression testing: during software evolution, only the
tests affected by code changes are selected/executed on the new revision,
since the other tests should have the same outcomes on the original and new
revisions. Researchers have shown that RTS can significantly reduce regression
testing costs (Gligoric et al., 2015; Zhang, 2018) and has been widely
incorporated into build systems of open-source projects (apa, 2020a, b, c). In
fact, APR shares similar procedure with regression testing that the buggy
program is repeatedly modified (by patches) and each generated patch needs to
be exhaustively validated by the existing test suite. Therefore, RTS can be
naturally applied to reduce APR costs.

Unfortunately, after systematically revisiting the APR literature, we find
that such an important optimization receives little attention: most existing
APR systems do not apply any RTS, while the few systems using RTS adopt RTS at
different granularities. For example, PraPR (Ghanbari et al., 2019) adopts
statement-level RTS while CapGen (Wen et al., 2018) uses class-level RTS in
its implementation. Therefore, it remains unknown how important it is to adopt
RTS in APR systems and how different RTS strategies affect APR efficiency. To
bridge such a knowledge gap, we perform the first extensive study to
investigate the impact of RTS on APR efficiency. More specifically, we
investigate the cost reduction achieved by widely-used RTS techniques at
different levels (i.e., class/method/statement levels). Furthermore, we also
study the joint impact of RTS and various test prioritization techniques (Li
et al., 2007; Lou et al., 2019; Qi et al., 2013) (which reorder test
executions and can also potentially speed up patch validation) on APR
efficiency. Our study is conducted on 12 state-of-the-art APR systems with
over 2M patches generated for the widely-used Defects4J (Just et al., 2014)
benchmark.

Our study reveals various practical guidelines for the APR community. (1) The
number of patches widely used for measuring APR efficiency can incur skewed
conclusions, and the use of inconsistent RTS configurations can further skew
the conclusion. (2) RTS can indeed reduce a significant portion of test
executions on all studied APR systems and should be considered in future APR
work. (3) The performance varies among different RTS strategies. In
particular, statement- and method-level RTS can significantly outperform
class-level RTS, and are recommended for future APR work. (4) RTS techniques
can substantially outperform state-of-the-art test prioritization techniques
for APR, and combining them can further improve APR efficiency. (5)
Traditional regression test prioritization (Li et al., 2007; Lou et al., 2019)
widely studied in regression testing performs even better than APR-specific
test prioritization (Qi et al., 2013) when they are combined with most RTS
techniques. Lastly, we also present the detailed impact of different patch
categories and patch validation strategies on our findings.

To summarize, as the first extensive study investigating the impact of RTS on
APR, this paper makes the following contributions:

  * •

Literature review. Revisiting the literature and implementations of existing
APR systems, highlighting RTS as an important optimization mostly neglected
and inconsistently-configured by existing work.

  * •

Extensive study. Performing the first extensive study on the impact of RTS
strategies on APR efficiency, exploring 3 representative RTS techniques for 12
state-of-the-art APR systems on a well-established benchmark (Defects4J) with
395 real bugs, involving over 2M patches in total.

  * •

Practical guidelines. Demonstrating the importance of RTS for APR and
revealing various practical guidelines regarding the adoption of RTS for the
APR community and future APR work.

##  2\. Background and Motivation

###  2.1. Regression Test Selection

Regression Test Selection (RTS) (Legunsen et al., 2016; Orso et al., 2004;
Gligoric et al., 2015; Rothermel and Harrold, 1997, 1993) is proposed to
accelerate regression testing by executing only a subset of regression tests.
Its basic intuition is that the tests not affected by code changes would have
the same results on the original and modified revisions. A typical RTS
strategy involves analyzing two dimensions of information: (1) changed code
elements between the original and modified program revisions, and (2) test
dependencies on the original revision. RTS then selects the tests whose
dependencies overlap with the changed code elements.

Based on the granularities of changed code elements and test dependencies, RTS
techniques can be categorized as class-level (Gligoric et al., 2015; Legunsen
et al., 2016), method-level (Ren et al., 2004), and statement-level RTS
(Harrold et al., 2001; Orso et al., 2004; Rothermel and Harrold, 1997). For
example, Gligoric et al. (Gligoric et al., 2015) proposed a class-level RTS
which was shown as more efficient than method-level RTS. Zhang (Zhang, 2018)
proposed the first RTS strategy of hybrid granularities and found that the
combination of class-level and method-level RTS achieved significant
improvement but there was no clear benefit when further incorporated with
statement-level RTS. In addition, according to how test dependencies are
analyzed, RTS techniques can be categorized as dynamic (Gligoric et al., 2015;
Harrold et al., 2001; Orso et al., 2004) and static RTS (Legunsen et al.,
2016; Kung et al., 1995). For example, Legunsen et al. (Legunsen et al., 2016)
compared the performance and safety of static regression test selection
approaches in modern software evolution, and their results suggested that
class-level static RTS was comparable to dynamic class-level RTS but method-
level static RTS had a poor performance. A RTS technique is regarded as safe
if it selects all tests that may be affected by changed code elements, because
missing any of such tests may fail to detect some regression bugs. To date,
RTS has been shown to substantially reduce the end-to-end regression testing
costs (Gligoric et al., 2015; Zhang, 2018) and has been widely incorporated
into build systems of open-source projects (apa, 2020a, b, c). While existing
work has extensively studied different RTS approaches in the regression
testing scenario, in this work, we perform the first extensive study of RTS in
the APR scenario.

Figure 1. Workflow of regression test selection

###  2.2. Automated Program Repair

Automated Program Repair (APR) (Martinez and Monperrus, 2018; Durieux and
Monperrus, 2016; Yuan and Banzhaf, 2020; Jiang et al., 2018; Liu et al.,
2019b; Koyuncu et al., 2020; Smith et al., 2015; Yi et al., 2018) has been
proposed to automatically fix program bugs with minimal human intervention.
Given a buggy program, APR generates patches and then validates them to check
their correctness. A test suite often serves as a partial specification to
check whether generated patches satisfy the desired behavior. Figure 2
overviews the workflow of test-based automated program repair. The input
includes a buggy program and its test suite (with at least one failed test). A
typical APR system consists of three phases. (1) Fault localization: before
the repair process, off-the-shelf fault localization techniques are leveraged
to diagnose suspicious code elements (e.g., statements). (2) Patch generation:
the APR system then applies repair operations on suspicious locations
following the suspiciousness ranking list. Each modified program version is
denoted as a candidate patch. (3) Patch validation: lastly, each candidate
patch is validated by the test suite until a patch that can pass all tests is
found, i.e., plausible patch. Existing APR systems often terminate once
finding one plausible patch or reaching the time budget.

Figure 2. Workflow of test-based automated program repair

Existing work suggests that patch validation can be very time consuming (Lou
et al., 2020; Hua et al., 2018), because (1) there are a large number of
patches generated and (2) each patch validation requires non-trivial time to
execute the original tests. To date, various APR systems have been proposed to
reduce the number of generated patches, and they can be categorized into the
following groups according to the way of patch generation. (1) Heuristic-based
APR (Yuan and Banzhaf, 2020; Jiang et al., 2018), which iteratively explores
the search space of program modifications via certain heuristics. For example,
GenProg (Yuan and Banzhaf, 2020) utilizes genetic programming to search donor
code for patch generation. (2) Constraint-based APR (Xiong et al., 2017;
Martinez and Monperrus, 2018; Durieux and Monperrus, 2016), which generates
patches for conditional expressions by transforming them into constraint
solving tasks. For example, Nopol (Xuan et al., 2017) utilizes SMT solvers to
synthesize simple condition code with arithmetic and first-order-logic
operators. (3) Template-based APR (Liu et al., 2019b; Koyuncu et al., 2020;
Liu et al., 2019a), which designs predefined patterns to guide patch
generation. For example, KPar (Liu et al., 2019a) manually summarizes patterns
based on human-written patches, and TBar (Liu et al., 2019c) constructs a
large pattern pool by integrating all patterns from existing template-based
APR systems. (4) Learning-based APR (Lutellier et al., 2020; Li et al., 2020;
Chen et al., 2019), which adopts machine/deep learning techniques to generate
patches based on existing code corpus. For example, CocoNut (Lutellier et al.,
2020) automatically generates patches by combining convolutional neural
networks (Dauphin et al., 2017) and neural machine translation (Bahdanau et
al., 2015).

Table 1. Revisiting RTS strategies on APR systems

APR Time RTS APR Time RTS PAR (Kim et al., 2013) 2013 No RTS CapGen (Wen et
al., 2018) 2018 Class level jGenProg (Martinez and Monperrus, 2016) 2016 No
RTS ARJA (Yuan and Banzhaf, 2020) 2018 Stmt level jKali (Martinez and
Monperrus, 2016) 2016 No RTS GenProg-A (Yuan and Banzhaf, 2020) 2018 Stmt
level jMutRepair (Martinez and Monperrus, 2016) 2016 No RTS Kali-A (Yuan and
Banzhaf, 2020) 2018 Stmt level DynaMoth (Durieux and Monperrus, 2016) 2016 No
RTS RSRepair-A (Yuan and Banzhaf, 2020) 2018 Stmt level xPAR (Le et al., 2016)
2016 No RTS SequenceR (Chen et al., 2019) 2019 No RTS HDRepair (Le et al.,
2016) 2016 No RTS kPAR (Liu et al., 2019a) 2019 No RTS NPEFix (Durieux et al.,
2017) 2017 No RTS DeepRepair (White et al., 2019) 2019 No RTS ACS (Xiong et
al., 2017) 2017 No RTS PraPR (Ghanbari et al., 2019) 2019 Stmt level Genesis
(Long et al., 2017) 2017 No RTS Hercules (Saha et al., 2019) 2019 No RTS
jFix/S3 (Le et al., 2017) 2017 No RTS GenPat (Jiang et al., 2019) 2019 No RTS
EXLIR (Saha et al., 2017) 2017 No RTS AVATAR (Liu et al., 2019b) 2019 No RTS
JAID (Chen et al., 2017) 2017 No RTS TBar (Liu et al., 2019c) 2019 No RTS
ssFix (Xin and Reiss, 2017) 2017 No RTS Nopol (Xuan et al., 2017) 2019 No RTS
SimFix (Jiang et al., 2018) 2018 No RTS ConFix (Kim and Kim, 2019) 2019 No RTS
Cardumen (Martinez and Monperrus, 2018) 2018 No RTS FixMiner (Koyuncu et al.,
2020) 2020 No RTS SketchFix (Hua et al., 2018) 2018 Stmt level CocoNut
(Lutellier et al., 2020) 2020 No RTS LSRepair (Liu et al., 2018) 2018 No RTS
DLFix (Li et al., 2020) 2020 No RTS SOFix (Liu and Zhong, 2018) 2018 No RTS -
- -

###  2.3. RTS in Patch Validation

In addition to the large number of generated patches, each patch also needs to
be executed against the original tests, which can be rather costly. In fact,
the generate-and-validate procedure in APR is very similar to the regression
testing scenario that patches are modifications of the original buggy program
and each of them needs to be validated by the existing test suite. Hence, it
is intuitive to adopt RTS in patch validation, which can improve the repair
efficiency by executing only the tests affected by the patch.

We first systematically revisit RTS strategies adopted in existing APR
systems. In particular, we consider APR systems targeting at JAVA programs due
to its popularity in the APR community, and use two sources of information to
collect representative APR systems (i.e., program-repair.org and the living
review of APR (Monperrus, 2020)). For each APR system, we check which RTS
strategy is used both in its paper and its implementation (if the source code
is released). Table 1 summarizes the results of our literature review.

Based on the table, we have the following findings. (1) Most existing APR
systems, including the latest systems proposed in 2020, have not adopted any
RTS strategy to optimize the efficiency of patch validation. In other words,
the whole test suite is repeatedly executed against each generated patch,
including the tests that are not affected by the patch at all. (2) The few APR
systems which adopt RTS are all based on dynamic RTS. This makes sense as
prior regression testing studies show that static RTS can be imprecise and
unsafe (e.g., due to Java Reflections) (Legunsen et al., 2016; Shi et al.,
2019). (3) For the few APR systems using RTS, they adopt RTS at different
granularities (highlighted by different colors in the table). For example,
PraPR performs dynamic statement-level RTS, while CapGen uses class-level RTS
in implementation.

Motivation: RTS is neglected by most APR systems and different RTS strategies
are adopted by those few APR systems using RTS. Therefore, it remains unknown
how necessary it is to adopt RTS in APR and how different RTS strategies
affect APR efficiency. To bridge this knowledge gap, we perform the first
study on the impact of RTS strategies on APR efficiency.

##  3\. Study Design

###  3.1. Preliminaries

We first formally define key conceptions used in this paper.

####  3.1.1. Patch Validation Matrix

Given a buggy program $\mathcal{P}_{b}$ and its test suite $\mathcal{T}$ (with
at least one failed test), an APR system generates a list of candidate patches
$\mathbb{P}{}$ after the patch generation phase.

######  Definition 3.1.

Patch validation matrix $\mathbb{M}$ defines the execution results of all
tests on all patches. In particular, each cell $\mathbb{M}[\mathcal{P},t]$
represents the execution result of test $t\in\mathcal{T}$ on patch
$\mathcal{P}\in\mathbb{P}$, which can have the following values: (1) ❍ , if
$t$ has not been selected by RTS and thus skipped for execution, (2) ✗, if $t$
fails on the patch $\mathcal{P}$, (3) ✓, if $t$ passes on the patch
$\mathcal{P}$, and (4) —, if $t$ is selected by RTS but has not been executed
(i.e., its execution result remains unknown).

The early-exit mechanism is a common efficiency optimization widely enabled in
existing APR systems (Jiang et al., 2018; Ghanbari et al., 2019), which stops
validation for the current patch once it fails on any test. In this scenario,
there are many tests whose results remain unknown, resulting in a validation
matrix with “—” cells. We denote such a matrix as a partial patch validation
matrix $\mathbb{M}_{p}$. In addition, in the other scenario where the
execution results of all tests are required (i.e., unified debugging (Benton
et al., [n.d.]; Lou et al., 2020)), the early-exit mechanism is often
disabled. In this scenario, all the selected tests would be executed even when
there are some tests that already failed during patch validation, leaving no
“—” cell in the matrix $\mathbb{M}$. We denote such a matrix as a full patch
validation matrix $\mathbb{M}_{f}$. The following are examples for full and
partial matrices when there is no RTS adopted in patch validation. For
example, patch validation for $\mathcal{P}_{1}$ stops by the failure of
$t_{1}$, and thus the execution results of $t_{2}$ and $t_{3}$ on
$\mathcal{P}_{1}$ remain unknown in $\mathbb{M}_{p}$.

(1)
$\displaystyle\scriptsize\mathbb{M}_{f}=\begin{bmatrix}&t_{1}&t_{2}&t_{3}\\\
\mathcal{P}_{1}&$\char 55$&$\char 51$&$\char 51$\\\ \mathcal{P}_{2}&$\char
55$&$\char 55$&$\char 51$\\\ \mathcal{P}_{3}&$\char 51$&$\char 51$&$\char
51$\\\ \end{bmatrix}$
$\displaystyle\scriptsize\mathbb{M}_{p}=\begin{bmatrix}&t_{1}&t_{2}&t_{3}\\\
\mathcal{P}_{1}&$\char 55$&$\texttwelveudash$&$\texttwelveudash$\\\
\mathcal{P}_{2}&$\char 55$&$\texttwelveudash$&$\texttwelveudash$\\\
\mathcal{P}_{3}&$\char 51$&$\char 51$&$\char 51$\\\ \end{bmatrix}$

####  3.1.2. Studied RTS Strategies

Based on the literature review, we focus on dynamic RTS strategies that
utilize test coverage in buggy programs as test dependencies, since static RTS
can be imprecise and unsafe (Legunsen et al., 2016; Shi et al., 2019). Given a
candidate patch $\mathcal{P}$, $\mathcal{P}_{\Delta}$ denotes the set of code
elements modified by $\mathcal{P}$, while $\mathbb{C}[\mathcal{P}_{b},t]$
denotes the set of code elements in $\mathcal{P}_{b}$ that are covered by the
test $t$.

######  Definition 3.2.

For a given patch $\mathcal{P}$ and the whole test suite $\mathcal{T}$, a
dynamic RTS strategy for APR selects a subset of tests $\mathcal{T^{\prime}}$
for execution. In particular, for precise and safe RTS, each test in
$\mathcal{T^{\prime}}$ should cover at least one modified code element, i.e.,
$\forall{}t{}\in\mathcal{T^{\prime}}{},\mathcal{P}_{\Delta}{}\cap\mathbb{C}[\mathcal{P}_{b},t]\neq{}\emptyset{}$,
while each test not in $\mathcal{T^{\prime}}$ should not cover any modified
element, i.e.,
$\forall{}t{}\notin\mathcal{T^{\prime}}{},\mathcal{P}_{\Delta}{}\cap{}\mathbb{C}[\mathcal{P}_{b},t]=\emptyset{}$

Note that the above definition is simplified for the ease of understanding.
Our actual implementation for ensuring RTS precision and safety is actually
more complicated and handles all types of Java code changes (e.g., method-
overriding hierarchy changes) following prior safe RTS work (Zhang, 2018;
Gligoric et al., 2015). According to the granularity of modified code
elements, we study three RTS strategies: class-level ($RTS_{class}$), method-
level ($RTS_{method}$), and statement-level RTS ($RTS_{stmt}$). In addition,
we regard no RTS adoption as the baseline test selection strategy, denoted as
$RTS_{no}$. In the example, if $t_{1}$ and $t_{3}$ cover the statements
modified by $\mathcal{P}_{1}$, and $t_{1}$ and $t_{2}$ cover the statements
modified by $\mathcal{P}_{2}$ and $\mathcal{P}_{3}$, the partial and full
validation matrices associated with $RTS_{stmt}$ can be represented as
Equation 2. For example, since $t_{3}$ does not cover any statement modified
by $\mathcal{P}_{3}$, $RTS_{stmt}$ skips $t_{3}$ in the validation for
$\mathcal{P}_{3}$. However, if $t_{3}$ happens to cover other statements in
the same class as the modified statements (i.e., $t_{3}$ covers the classes
modified by $\mathcal{P}_{3}$), $RTS_{class}$ would select $t_{3}$ when
validating $\mathcal{P}_{3}$. In this way, RTS at coarser granularities tend
to select more tests.

(2)
$\displaystyle\vspace{1mm}\scriptsize\mathbb{M}_{f}=\begin{bmatrix}&t_{1}&t_{2}&t_{3}\\\
\mathcal{P}_{1}&$\char 55$&$\char 109 $&$\char 51$\\\ \mathcal{P}_{2}&$\char
55$&$\char 55$&$\char 109 $\\\ \mathcal{P}_{3}&$\char 51$&$\char 51$&$\char
109 $\\\ \end{bmatrix}$
$\displaystyle\scriptsize\mathbb{M}_{p}=\begin{bmatrix}&t_{1}&t_{2}&t_{3}\\\
\mathcal{P}_{1}&$\char 55$&$\char 109 $&$\texttwelveudash$\\\
\mathcal{P}_{2}&$\char 55$&$\texttwelveudash$&$\char 109 $\\\
\mathcal{P}_{3}&$\char 51$&$\char 51$&$\char 109 $\\\ \end{bmatrix}$

####  3.1.3. Efficiency Measurement

In this study, we mainly measure the repair efficiency by the number of test
executions. Here we do not use the time cost as the measurement, because it
tends to introduce bias as suggested by previous work (Liu et al., 2020): (1)
it is dependent on many factors (e.g., specific implementations and test
execution engines) unrelated to APR approaches and (2) it is often unstable.
Previous work also investigated the efficiency of APR by the number of patch
candidates validated by APR systems. However, simply counting the number of
patches can be very imprecise, because it treats each patch as equally costly
and measures repair efficiency in an oversimplified way. For example, given
$\mathbb{M}_{p}{}$ in Equation 1, both $\mathcal{P}_{2}$ and $\mathcal{P}_{3}$
would be regarded as one program execution, even if $\mathcal{P}_{3}$ actually
executes more tests than $\mathcal{P}_{2}$. Therefore, we further define the
following metrics based on the number of test executions for precise
efficiency measurement.

######  Definition 3.3.

Given a patch validation matrix $\mathbb{M}{}$, its accumulated number of test
executions $NT(\mathbb{M})$, sums up the number of test executions on all the
validated patches, i.e., $NT(\mathbb{M})=\sum{}{}{\mathbb{M}[\mathcal{P},t]}$,
if $\ \mathbb{M}[\mathcal{P},t]\neq{}$❍ $\ \land\
\mathbb{M}[\mathcal{P},t]\neq{}$ —.

######  Definition 3.4.

Given the patch validation matrix $\mathbb{M}{}$ without RTS, and the matrix
$\mathbb{M}^{RTS}{}$ generated by certain RTS strategy, reduction of test
executions, measures the efficiency improvement gained by the RTS strategy
compared to no RTS, i.e., $Reduction$
$=\frac{NT(\mathbb{M})-NT(\mathbb{M}^{RTS})}{NT(\mathbb{M})}$

In the example, without any RTS (in Equation 1), $NT(\mathbb{M}_{f})$ is 9 and
$NT(\mathbb{M}_{p})$ is 5. Meanwhile, with $RTS_{stmt}$ (in Equation 2),
$NT(\mathbb{M}{}_{f}^{RTS})$ is 6 and $NT(\mathbb{M}{}_{p}^{RTS})$ is 4.
Therefore, $RTS_{stmt}$ achieves 33.33%/20.00% $Reduction{}$ in the
full/partial matrix respectively. Using the number of patches is definitely
unaware of such efficiency difference between the matrices with and without
RTS. For example, if only considering the number of patches in
$\mathbb{M}{}_{p}^{RTS}$ and $\mathbb{M}{}_{p}$, there is no reduction
achieved by RTS at all. Therefore, it is important to measure repair
efficiency by the number of test executions.

###  3.2. Research Questions

  * •

RQ1: Revisiting APR efficiency. We revisit the efficiency of APR systems by
making the first attempt to (1) measure repair efficiency by the number of
test executions, and (2) compare the efficiency of different APR systems with
consistent RTS configurations to eliminate the bias from RTS strategies.

  * •

RQ2: Overall impact of RTS strategies. We investigate the efficiency
improvement achieved by different RTS strategies based on their reduction of
test executions.

  * •

RQ3: Impact on different patches. We further study the impact of RTS
strategies on patches of different characteristics to find out what kinds of
patches are more susceptible to RTS strategies. In particular, we categorize
patches as follows.

    * –

RQ3a: Impact on patches of different fixing capabilities. We categorize
patches according to their fixing capabilities, i.e., the patches that can
pass more tests are considered to have stronger fixing capabilities. We then
study the impact of RTS strategies on these patches with different fixing
capabilities.

    * –

RQ3b: Impact on patches of different fixing scopes. We categorize patches
according to their scopes of patched code, i.e., the number of code elements
modified by the patches. We then study the impact of RTS strategies on such
patches.

  * •

RQ4: Impact with the full patch validation matrix. In the first three RQs, we
investigate APR efficiency with the default _partial_ patch validation matrix
(with the early-exit mechanism), which is widely adopted in all APR systems
for the sake of efficiency. In this RQ, we further study the impact of RTS
strategies with the _full_ validation matrix (without the early-exit
mechanism). It is the common setting when execution results of all tests are
required, e.g., in unified debugging (Lou et al., 2020; Benton et al.,
[n.d.]).

  * •

RQ5: Combining test selection with prioritization. In addition to test
selection strategies, when the early-exit mechanism is enabled, the execution
order of tests jointly decides the number of test executions. Therefore, in
this RQ, we further combine the studied RTS strategies with state-of-the-art
test prioritization techniques, and investigate their impact on APR
efficiency.

###  3.3. Benchmark

We perform our study on the benchmark Defects4J V.1.2.0 (Just et al., 2014),
which is the version used most widely by existing APR work (Xiong et al.,
2017; Liu et al., 2019c; Ghanbari et al., 2019). Defects4J V.1.2.0 includes
395 real-world bugs from six real-world software systems. Table 2 presents
detailed information.

Table 2. Benchmark information

ID Name #Bug #Test LoC Lang Apache commons-lang 65 2,245 22K Math Apache
commons-math 106 3,602 85K Time Joda-Time 27 4,130 28K Chart JFreeChart 26
2,205 96K Closure Google Closure compiler 133 7,927 90K Mockito Mockito
framework 38 1,366 23K Defects4J (V1.2.0) 395 21,475 344K

###  3.4. Studied APR Systems

Following the recent studies on APR (Liu et al., 2020; Benton et al., [n.d.]),
our study selects APR systems for JAVA program according to the following
criteria. (1) The source code of the APR system is publicly available because
we need to modify patch validation settings. (2) The APR systems can be
successfully applied to Defects4J subjects. (3) The APR systems require no
extra input data besides program source code and the available test suite. In
this way, we successfully collect 16 APR systems fulfilling the requirements
above, and all of them are also used as the studied APR systems in previous
work (Liu et al., 2020; Benton et al., [n.d.]). However, these APR systems use
test cases at different granularities: the Astor family (Martinez and
Monperrus, 2016) (including jMutRepair, jGenProg, jKali) and Cardumen
(Martinez and Monperrus, 2018) regard each test class as a test case, while
the remaining systems all regard each test method as a test case. Since RTS
results at different test-case granularities are incomparable, we use the more
popular test-method granularity with 12 APR systems in our study. Table 3
presents the 12 APR systems studied in this work, including heuristic-based,
constraint-based, and template-based ones. In total, the studied APR systems
generate 2,532,915 patches for all the Defects4J subjects studied, while
_each_ patch can involve up to thousands of test executions.

Table 3. Studied APR systems

Tool Category Tools Constraint-based ACS(Xiong et al., 2017), Dynamoth(Durieux
and Monperrus, 2016) Heuristic-based Arja(Yuan and Banzhaf, 2020),
GenProg-A(Yuan and Banzhaf, 2020), Kali-A(Yuan and Banzhaf, 2020),
RSRepair-A(Yuan and Banzhaf, 2020), SimFix(Jiang et al., 2018) Template-based
AVATAR(Liu et al., 2019b), FixMiner(Koyuncu et al., 2020), kPar(Liu et al.,
2019a), PraPR(Ghanbari et al., 2019), TBar (Liu et al., 2019c)

###  3.5. Implementation Details

To implement studied RTS strategies, following state-of-the-art Ekstazi
(Gligoric et al., 2015) and HyRTS (Zhang, 2018), we leverage on-the-fly
bytecode instrumentation with ASM (ASM, 2020) and Java Agent (jav, 2020) for
lightweight test dependency collection at the class/method/statement level on
the buggy programs. For all studied APR systems, we modify their source code
to collect patch/test execution results with different RTS strategies and
validation settings. In all RQs, we schedule all originally-failed tests prior
to originally-passed tests, which is also the common practice of all recent
APR systems because the former are more likely to fail again on patches. From
RQ1 to RQ4, within the originally-failed tests or the originally-passed tests,
we prioritize them by their default lexicographical order to eliminate the
impact of different test execution orders.

###  3.6. Threat to Validity

External validity. Our study focuses on APR systems for JAVA programs, which
might threaten generality of our findings. We mitigate this threat by
performing our study on a large spectrum of APR systems belonging to different
categories and a widely-used benchmark with hundreds of real bugs (Ghanbari et
al., 2019; Wen et al., 2018; Jiang et al., 2018).

Internal validity. To mitigate the threat in faulty implementations, the first
two authors carefully reviewed our RTS implementations, as well as the scripts
for running experiments and parsing experimental results. In addition, the
source code of all the studied APR systems was directly obtained from their
authors or original open-source repositories, and we further ensured that each
APR system performs consistently before and after our modifications.

Construct validity. The construct validity may be threatened by the metrics
used in our study. Considering the instability of time (Liu et al., 2020), we
use both the number of test executions and the number of patches to measure
APR efficiency.

##  4\. Results analysis

Table 4. Number of test executions among APR systems

Subject RTS Strategy PraPR SimFix AVATAR kPar TBar FixMiner Dynamoth ACS Arja
Kali-A GenProg-A RSRepair-A Lang #Patch 381.27 62.55 4.63 4.79 4.76 4.56 0.16
0.05 522.97 26.92 628.85 289.23 #Test ($RTS_{no}$) 2,117.61 424.75 321.41
163.19 437.52 233.15 1,256.86 608.00 68,448.64 799.43 86,910.25 10,945.44
#Test ($RTS_{class}$) 447.31 85.96 19.48 18.30 24.63 18.88 25.71 17.00
1,571.25 59.54 2,746.19 620.69 #Test ($RTS_{method}$) 414.15 82.38 12.11 11.89
12.63 11.88 2.86 1.00 1,196.83 51.54 1,376.81 556.39 #Test ($RTS_{stmt}$)
412.59 82.21 11.44 11.59 11.96 11.50 2.00 1.00 1,176.53 50.63 1,369.64 534.58
Math #Patch 1,483.12 30.87 5.84 5.12 5.01 4.76 0.23 0.05 804.09 10.11 858.19
411.58 #Test ($RTS_{no}$) 11,879.77 1,185.13 168.78 237.05 640.65 239.38
2,042.74 789.33 104,804.90 705.64 73,174.10 9,661.55 #Test ($RTS_{class}$)
1,711.16 158.06 12.92 46.66 34.66 9.47 50.79 6.33 4,828.21 22.00 5,231.08
828.93 #Test ($RTS_{method}$) 1,573.21 56.41 8.20 7.25 8.18 7.03 32.53 6.33
2,105.64 15.97 1,610.38 498.28 #Test ($RTS_{stmt}$) 1,557.41 55.59 7.81 6.86
7.25 6.23 27.00 1.67 1,382.11 14.10 1,419.85 463.63 Time #Patch 1,466.54 22.81
2.50 2.00 2.23 1.08 0.23 0.04 335.92 13.23 380.62 107.46 #Test ($RTS_{no}$)
7,782.19 768.55 1,065.00 4.73 359.09 490.00 1,303.67 1.00 34,310.82 724.00
141,456.00 2,652.50 #Test ($RTS_{class}$) 2,513.88 184.27 232.55 4.73 80.82
107.38 537.33 1.00 7,337.91 325.70 29,401.40 635.00 #Test ($RTS_{method}$)
1,848.62 58.64 9.73 4.73 6.55 5.25 8.67 1.00 3,822.82 55.30 10,596.50 413.90
#Test ($RTS_{stmt}$) 1,808.12 56.64 9.73 4.73 6.55 5.25 8.67 1.00 1,781.91
38.50 1,903.60 291.50 Chart #Patch 784.88 62.96 5.00 4.56 4.40 4.48 0.40 -
627.76 42.36 729.72 333.16 #Test ($RTS_{no}$) 5,517.96 218.61 272.84 107.61
367.74 102.12 1,640.80 - 202,236.87 2,035.27 104,090.08 9,552.46 #Test
($RTS_{class}$) 1,053.80 73.61 8.26 8.00 8.79 7.12 77.30 - 27,627.48 133.45
5,694.38 758.38 #Test ($RTS_{method}$) 868.24 72.65 6.89 6.89 6.37 6.59 16.70
- 6,853.48 61.45 2,645.12 450.67 #Test ($RTS_{stmt}$) 863.96 72.65 6.89 6.89
6.37 6.59 16.30 - 2,356.57 56.82 1,746.42 386.71 Closure #Patch 14,725.00 4.30
2.96 2.61 2.94 0.42 - - - - - - #Test ($RTS_{no}$) 86,651.19 4,239.84 342.78
378.73 478.09 2.89 - - - - - - #Test ($RTS_{class}$) 47,374.48 2,329.55 140.81
154.94 197.02 2.89 - - - - - - #Test ($RTS_{method}$) 28,476.71 449.64 6.17
9.90 6.67 2.89 - - - - - - #Test ($RTS_{stmt}$) 21,825.07 166.45 4.49 3.92
4.40 2.89 - - - - - - Mockito #Patch 2,307.92 - 1.67 2.00 1.81 1.67 - - - - -
- #Test ($RTS_{no}$) 3,753.19 - 267.75 41.27 200.27 4.62 - - - - - - #Test
($RTS_{class}$) 2,857.81 - 97.56 26.00 146.47 4.62 - - - - - - #Test
($RTS_{method}$) 2,555.61 - 7.62 5.80 48.00 4.62 - - - - - - #Test
($RTS_{stmt}$) 2,547.64 - 7.38 5.80 7.33 4.62 - - - - - -

###  4.1. RQ1: Revisiting APR Efficiency

Table 4 presents the number of test executions with different RTS strategies.
We also present the number of validated patches (#Patch) in the table. Note
that cells are empty when there is no valid patch generated. Based on the
table, we have the following observations.

First, measuring APR efficiency by a more precise metric (i.e., the number of
test executions) can draw totally different conclusions from the number of
validated patches advocated by prior work (Liu et al., 2020). For example, for
each buggy version of Time, SimFix (i.e., 22.81 patches) validates many more
patches than AVATAR (i.e., 2.50 patches), showing that SimFix is less
efficient than AVATAR in terms of the number of validated patches. Such a
conclusion is consistent with the finding in previous work (Liu et al., 2020),
which also measures APR efficiency by the number of validated patches.
However, the conclusion can be opposite if these systems are compared by the
number of test executions. For example, with class-level RTS, SimFix executes
184.27 tests and AVATAR executes 232.55 tests per buggy version of Time,
indicating that SimFix is more efficient than AVATAR. The reason is that
although AVATAR generates less patches than SimFix, each patch of AVATAR
executes many more tests than SimFix. Such inconsistencies between the number
of test executions and patches are prevalent in the table. In summary, it is
imprecise to measure APR efficiency by simply counting the number of patches,
since different patches can execute totally different number of tests. We thus
suggest future work on APR efficiency to adopt the number of test executions
since it is a more precise metric.

Second, comparing APR efficiency with different RTS strategies can also
deliver opposite conclusions. For example, on subject Time, AVATAR executes
232.55 and 9.73 tests with $RTS_{class}$ and $RTS_{method}$, while SimFix
executes 184.27 and 58.64 tests with $RTS_{class}$ and $RTS_{method}$. When
both systems are configured with method-level RTS, AVATAR is considered as
more efficient than SimFix; however, when they are compared under class-level
RTS, AVATAR is actually less efficient than SimFix. Moreover, if APR systems
are compared under different RTS strategies respectively, the conclusion may
further be skewed. Therefore, it is important to adopt the same RTS strategy
when comparing the repair efficiency among different APR systems. In addition,
we strongly recommend that future APR work should explicitly describe the RTS
strategy adopted in their experimental procedure, so that their follow-up work
can mitigate the threats in RTS configuration.

Finding 1: the number of patches widely used for measuring APR efficiency can
incur skewed conclusions, and the use of inconsistent RTS configurations can
further skew the conclusion. Future APR work should explicitly describe their
adopted RTS strategies and the efficiency comparison among multiple APR
systems should guarantee a consistent RTS configuration.

Table 5. Reduction of test executions by different RTS strategies

Subject RTS Strategy PraPR SimFix AVATAR kPar TBar FixMiner Dynamoth ACS Arja
Kali-A GenProg-A RSRepair-A Lang $RTS_{class}$ 33.33% 18.16% 17.92% 10.57%
21.48% 14.80% 69.99% 32.44% 36.67% 22.16% 26.11% 30.12% $RTS_{method}$ 34.51%
18.36% 18.35% 10.95% 22.05% 15.22% 71.31% 33.32% 38.16% 22.50% 26.68% 31.17%
$RTS_{stmt}$ 34.55% 18.37% 18.40% 10.99% 22.09% 15.26% 71.36% 33.32% 38.17%
22.52% 26.70% 31.20% Math $RTS_{class}$ 48.91% 29.72% 5.46% 5.36% 19.91% 8.63%
92.17% 33.12% 38.66% 16.44% 28.77% 31.73% $RTS_{method}$ 50.41% 33.03% 6.27%
6.30% 21.08% 8.99% 93.06% 33.12% 40.16% 16.79% 30.16% 32.44% $RTS_{stmt}$
50.50% 33.06% 6.28% 6.31% 21.11% 9.05% 93.33% 33.32% 40.42% 16.85% 30.35%
32.68% Time $RTS_{class}$ 31.25% 14.68% 7.14% 0.00% 7.15% 9.83% 19.62% 0.00%
19.59% 17.90% 7.97% 16.26% $RTS_{method}$ 40.12% 17.87% 9.06% 0.00% 9.06%
12.45% 33.15% 0.00% 21.41% 28.27% 9.31% 17.38% $RTS_{stmt}$ 40.31% 17.92%
9.06% 0.00% 9.06% 12.45% 33.15% 0.00% 22.60% 29.01% 9.93% 18.01% Chart
$RTS_{class}$ 58.50% 8.36% 15.68% 5.49% 20.88% 5.85% 85.96% - 53.39% 40.87%
34.93% 39.33% $RTS_{method}$ 62.45% 8.41% 15.76% 5.55% 21.02% 5.88% 89.10% -
59.10% 43.78% 38.81% 42.02% $RTS_{stmt}$ 62.52% 8.41% 15.76% 5.55% 21.02%
5.88% 89.12% - 59.67% 44.10% 39.22% 42.45% Closure $RTS_{class}$ 31.73% 25.15%
3.22% 3.79% 3.90% 0.00% - - - - - - $RTS_{method}$ 46.18% 51.50% 5.28% 6.56%
7.38% 0.00% - - - - - - $RTS_{stmt}$ 51.80% 55.28% 5.30% 6.65% 7.42% 0.00% - -
- - - - Mockito $RTS_{class}$ 18.03% - 21.10% 2.77% 5.49% 0.00% - - - - - -
$RTS_{method}$ 21.17% - 30.69% 6.43% 14.82% 0.00% - - - - - - $RTS_{stmt}$
21.30% - 30.72% 6.43% 19.67% 0.00% - - - - - - Average $RTS_{class}$ 36.96%
19.21% 11.75% 4.66% 13.13% 6.52% 66.93% 21.85% 37.08% 24.34% 24.45% 29.36%
$RTS_{method}$ 42.47% 25.84% 14.23% 5.96% 15.90% 7.09% 71.66% 22.15% 39.71%
27.83% 26.24% 30.75% $RTS_{stmt}$ 43.50% 26.61% 14.25% 5.99% 16.73% 7.11%
71.74% 22.21% 40.22% 28.12% 26.55% 31.09%

###  4.2. RQ2: Overall Impact of RTS Strategies

Table 5 presents the reduction of test executions achieved by each RTS
strategy compared to no test selection. Based on the table, for all the
studied APR systems on almost all the subjects, adopting RTS can remarkably
reduce the number of test executions in patch validation. For example, on the
subject Math, Dynamoth skips 93.33% of test executions if it adopts
$RTS_{stmt}$. Even for RTS at the coarsest level (i.e., $RTS_{class}$), the
average reduction ranges from 4.66% to 66.93% among different APR systems.
Note that at some cases there is no difference among RTS strategies, e.g., ACS
on Time, because all generated patches fail at the first executed test (i.e.,
the first originally-failed test still fails on these patches). We would
further analyze the individual impact on different patches in Section 4.3. In
summary, our results suggest that RTS can significantly improve the efficiency
of patch validation and future APR work should no longer overlook such an
important optimization.

Finding 2: for all studied APR systems, adopting regression test selection can
remarkably improve APR efficiency and should be considered in future APR work.

Table 6. Wilcoxon signed-rank test between RTS strategies based on reduction
of test executions

Subject RTS Strategy PraPR SimFix AVATAR kPar TBar FixMiner Dynamoth ACS Arja
Kali-A GenProg-A RSRepair-A Lang $RTS_{class}$ v.s $RTS_{method}$ ✔ ✔ ❍ ❍ ❍ ❍
❍ ❍ ✔ ✔ ✔ ✔ $RTS_{method}$ v.s $RTS_{stmt}$ ✔ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ✔ Math
$RTS_{class}$ v.s $RTS_{method}$ ✔ ✔ ❍ ❍ ✔ ✔ ✔ ❍ ✔ ✔ ✔ ✔ $RTS_{method}$ v.s
$RTS_{stmt}$ ✔ ✔ ❍ ❍ ❍ ❍ ✔ ❍ ✔ ✔ ✔ ✔ Time $RTS_{class}$ v.s $RTS_{method}$ ✔ ❍
❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ $RTS_{method}$ v.s $RTS_{stmt}$ ✔ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍
Chart $RTS_{class}$ v.s $RTS_{method}$ ✔ ❍ ❍ ❍ ❍ ❍ ✔ - ✔ ✔ ✔ ✔ $RTS_{method}$
v.s $RTS_{stmt}$ ❍ ❍ ❍ ❍ ❍ ❍ ❍ - ✔ ❍ ❍ ❍ Closure $RTS_{class}$ v.s
$RTS_{method}$ ✔ ✔ ❍ ❍ ✔ ❍ - - - - - - $RTS_{method}$ v.s $RTS_{stmt}$ ✔ ✔ ❍ ❍
✔ ❍ - - - - - - Mockito $RTS_{class}$ v.s $RTS_{method}$ ✔ - ❍ ❍ ❍ ❍ - - - - -
- $RTS_{method}$ v.s $RTS_{stmt}$ ❍ - ❍ ❍ ❍ ❍ - - - - - -

Overall, we can observe a common trend that $RTS_{stmt}$ always executes the
least tests while $RTS_{class}$ exhibits the most number of test executions.
This is not surprising because class-level coverage is the coarsest selection
criterion, based on which more tests would inherently be selected. To further
investigate the difference between RTS strategies, we perform the Wilcoxon
Signed-Rank Test (Wilcoxon, 1992) at the significance level of 0.05 to compare
the reduction achieved by different selection strategies in Table 6. We use ❍
and ✔ to denote “no statistical difference” and “significant difference
(p-value¡0.05)”, respectively. From the table, we can observe that RTS
strategies cause significant difference on the reduction of test executions.
In particular, both $RTS_{method}$ and $RTS_{stmt}$ significantly outperform
$RTS_{class}$. Meanwhile, interestingly, the difference between $RTS_{method}$
and $RTS_{stmt}$ is often subtle. For example, as suggested by Table 5 and
Table 6, the average difference between $RTS_{method}$ and $RTS_{stmt}$ is
less than 1% at most cases and is mostly not significant according to the
statistical analysis. In fact, this observation is consistent with prior work
in traditional regression testing (Zhang, 2018), which shows that method-level
RTS outperforms class-level RTS but is often not significantly worse than
statement-level RTS. The reason is that modern system design principles
recommend writing simple method bodies for the ease of maintenance, making the
majority of method body changes directly affect all tests covering the methods
(i.e., $RTS_{stmt}$ is close to $RTS_{method}$). In summary, method- and
statement-level RTS are more recommended for APR.

Finding 3: different RTS strategies exhibit different reduction performance.
$RTS_{stmt}$ and $RTS_{method}$ perform similarly, but both of them
significantly outperform $RTS_{class}$.

###  4.3. RQ3: Impact on Different Patches

We now investigate the impact of RTS strategies on the patches of different
characteristics. Based on the intuition that a patch involving more test
executions might be more sensitive to RTS strategies, we categorize patches by
their fixing capabilities or fixing code scopes. Section 4.3.1 and Section
4.3.2 present the impact on these patch categorizations respectively.

Table 7. Reduction on $\mathbb{P}_{P2F}$ and $\mathbb{P}_{\checkmark}$

APR $\mathbb{P}_{P2F}$ $\mathbb{P}_{\checkmark}$ $RTS_{class}$ $RTS_{method}$
$RTS_{stmt}$ $RTS_{class}$ $RTS_{method}$ $RTS_{stmt}$ PraPR 73.24% 85.74%
87.95% 87.33% 97.18% 97.88% SimFix 82.88% 99.11% 99.15% 84.50% 96.77% 98.22%
AVATAR 81.16% 98.95% 99.17% 82.27% 99.36% 99.53% kPar 66.33% 98.07% 98.58%
83.47% 99.31% 99.80% TBar 68.34% 98.63% 99.13% 76.07% 95.39% 99.55% FixMiner
85.02% 96.90% 99.14% 93.28% 99.71% 99.80% Dynamoth 94.75% 96.71% 97.22% 87.50%
99.19% 99.27% ACS - - - 98.34% 99.66% 99.95% Arja 90.09% 95.56% 97.28% 93.58%
99.15% 99.44% Kali 88.76% 97.97% 99.30% 84.30% 99.00% 99.59% GenProg 88.85%
96.84% 99.05% 94.28% 99.33% 99.57% RSRepair 92.83% 97.94% 99.42% 95.79% 99.11%
99.58%

####  4.3.1. Impact on patches of different fixing capabilities

Based on test execution results, we can categorize patches into groups of
different fixing capabilities. (1) $\mathbb{P}_{F2F}$: a patch cannot fix all
originally-failed tests and thus its validation gets aborted by some
originally-failed test; (2) $\mathbb{P}_{P2F}$: a patch can fix all
originally-failed tests but fails on some originally-passed test, and thus its
validation gets aborted by some originally-passed test; (3)
$\mathbb{P}_{\checkmark}$: a patch that can pass all originally-failed and
originally-passed tests (i.e., plausible patch). Obviously, $\mathbb{P}_{F2F}$
has the weakest fixing capability and its validation halts extremely early,
and thus adopting RTS would not make any difference on its number of test
executions. Therefore, we present the impact of RTS strategies on the other
two patch groups, i.e., $\mathbb{P}_{P2F}$ and $\mathbb{P}_{\checkmark}$ in
Table 7. In particular, each cell shows the reduction ratio of test executions
compared to no test selection. Note that since ACS generates only
$\mathbb{P}_{F2F}$ and $\mathbb{P}_{\checkmark}$, its corresponding cells in
$\mathbb{P}_{P2F}$ are empty. Based on the table, we can observe that,
compared to the overall reduction on all patches (i.e., the reduction in Table
5), the impact of test selection strategies on $\mathbb{P}_{\checkmark}$ and
$\mathbb{P}_{P2F}$ is even more remarkable. For example, the reduction ratio
ranges from 66.33% to 99.42% on $\mathbb{P}_{P2F}$, and ranges from 76.07% to
99.95% on $\mathbb{P}_{\checkmark}$. In addition, compared to all patches, the
difference between RTS strategies is also enlarged on
$\mathbb{P}_{\checkmark}$ and $\mathbb{P}_{P2F}$. In summary, RTS can achieve
larger reductions for patches with stronger fixing capabilities.

Finding 4: the impact of RTS strategies is even more prominent on
$\mathbb{P}_{\checkmark}$ and $\mathbb{P}_{P2F}$, where an extremely large
portion of tests (e.g., up to 99.95%) get reduced.

Figure 3. Ratio of patches of different fixing capabilities

Considering the drastic impact of RTS on $\mathbb{P}_{\checkmark}$ and
$\mathbb{P}_{P2F}$, we proceed to check whether an APR system is more
susceptible to test selection if it exhibits a higher ratio of
$\mathbb{P}_{\checkmark}$ or $\mathbb{P}_{P2F}$. To this end, we present the
stacked bar plot for the ratio of patch categories of different fixing
capabilities in Figure 3, from which we can make the following observations.

First, it is noteworthy that almost all the existing APR systems exhibit a
very high ratio (e.g., around 90%) of $\mathbb{P}_{F2F}$, indicating that
$\mathbb{P}_{P2F}$ and $\mathbb{P}_{\checkmark}$ are sparse in the space of
compilable patches. Liu et al. (Liu et al., 2020) empirically compared the
number of un-compilable patches and inplausible patches generated by different
APR systems, to measure their costs wasted in patch validation towards
generating a valid patch. Their work showed that the state-of-the-art APR
techniques can avoid generating un-compilable patches. However in our work, we
make the first attempt to investigate patches of different fixing capabilities
(e.g., how many tests can pass on each patch). Our results suggest that for
most existing APR systems, the majority of their inplausible patches have a
low capability of fixing originally-failed tests and get immediately
terminated in the very beginning stage of validation. We also find that the
two constraint-based APR systems (e.g., Dynamoth and ACS) generate a lower
ratio of $\mathbb{P}_{F2F}$. The potential reason may be that constraint-based
APR systems target at conditional expressions, which are more likely to impact
the test execution outcomes, i.e., making the originally-failed test pass. In
addition, ACS designs ranking strategies on fixing ingredients for condition
synthesis, which might bring the plausible patch forward, end up the whole
validation process earlier, and thus reduce the ratio of $\mathbb{P}_{F2F}$.

Second, there is no apparent correlation between the ratio of
$\mathbb{P}_{P2F}$ (or $\mathbb{P}_{\checkmark}$) and the impact degree of RTS
strategies. For example, although PraPR exhibits a quite low ratio of
$\mathbb{P}_{P2F}$ and $\mathbb{P}_{\checkmark}$ (e.g., 0.29% and 0.93%), it
reduces a larger portion of test executions than other APR systems. One
straightforward reason is that the number of originally-failed tests is
significantly less than the originally-passed tests. For example, each buggy
version in Closure has 2.63 failed tests and 7,180.89 passed tests on average.
The number of skipped originally-passed tests is often significantly larger
than the number of originally-failed tests, and thus the impact of RTS on
originally-passed tests can still dominate the overall efficiency. It also
explains why even with such a low ratio of $\mathbb{P}_{P2F}$ and
$\mathbb{P}_{\checkmark}$ we can still observe a remarkable impact of RTS on
overall efficiency.

Finding 5: the majority of compilable patches generated by existing APR
systems immediately fail on the originally-failed tests. Even with such a low
ratio of $\mathbb{P}_{P2F}$ and $\mathbb{P}_{\checkmark}$, the APR efficiency
is still sensitive to RTS strategies, since most bugs exhibit substantially
more originally-passed tests than originally-failed tests.

Table 8. Reduction on single/multiple edit patches

Subject $RTS_{class}$ $RTS_{method}$ $RTS_{stmt}$ $\mathbb{P}_{MC}$
$\mathbb{P}_{SC}$ $\mathbb{P}_{MM}$ $\mathbb{P}_{SM}$ $\mathbb{P}_{MS}$
$\mathbb{P}_{SS}$ Lang 21.83% 27.36% 21.83% 27.86% 15.25% 25.75% Math 26.18%
28.82% 17.67% 30.35% 20.45% 29.75% Time 9.85% 13.26% 20.78% 16.49% 7.53%
14.77% Chart 32.92% 30.99% 27.95% 35.14% 19.27% 33.64% Closure - 11.30% -
19.48% 55.06% 17.34% Mockito - 9.48% - 14.62% - 15.62%

####  4.3.2. Impact on patches of different fixing scopes

Based on the scope of fixed code, patches can be categorized as single-edit or
multiple-edit patches. In particular, according to the granularity of edited
code elements, we categorize patches as: (1) patches editing single class
(denoted as $\mathbb{P}_{SC}$) v.s. patches editing multiple classes (denoted
as $\mathbb{P}_{MC}$), (2) patches editing single method (denoted as
$\mathbb{P}_{SM}$) v.s. patches editing multiple methods (denoted as
$\mathbb{P}_{MM}$), and (3) patches editing single statement (denoted as
$\mathbb{P}_{SS}$) v.s. patches editing multiple statements (denoted as
$\mathbb{P}_{MS}$).

Figure 4 is a stacked bar plot for the ratio of patch categories of different
fixing code scopes. The figure shows that the majority of generated patches
are single-edit patches, especially when edited code elements are at coarser
granularities (i.e., class or method).

Table 8 presents the reduction achieved by RTS at the corresponding
granularity on single-edit and multiple-edit patches. For example, the cell in
the column “$RTS_{class}$” and the row “Lang” means that on Lang $RTS_{class}$
can reduce 21.83% and 27.36% of test executions on patches that edit multiple
classes ($\mathbb{P}_{MC}$) and on patches that edit single class
($\mathbb{P}_{SC}$), respectively. Based on the table, at most cases, RTS can
reduce more test executions on single-edit patches than multiple-edit patches.
The reason is that multiple-edit patches often involve more code elements and
thus are covered by more tests, resulting in a lower ratio of reduction. Note
there are several counter cases, e.g., on Closure the reduction on multiple-
statement patches is much larger than on single-statement patches (55.06% v.s.
17.34%). We find that at these cases a considerable ratio of generated
multiple-edit patches have stronger fixing capabilities (i.e., belong to
$\mathbb{P}_{P2F}$ or $\mathbb{P}_{\checkmark}$) and thus exhibit a higher
reduction.

Finding 6: most patches generated by existing APR systems are single-edit
patches. RTS can reduce more test executions on single-edit patches than on
multiple-edit patches, since usually fewer tests can cover the modified code
elements of single-edit patches. Meanwhile, for some special cases, multiple-
edit patches can have high probabilities in producing high-quality patches
passing more tests, leaving more room for RTS.

Figure 4. Ratio of patches of different fixing code scopes

###  4.4. RQ4: Impact with the Full Matrix

In previous research questions, we investigate the impact of test selection
with the partial validation matrix ($\mathbb{M}_{p}{}$), i.e., the early-exit
mechanism is enabled and the validation for each patch would be terminated
after any test failure. In this RQ, we further study the impact of RTS with
the full matrix ($\mathbb{M}_{f}{}$), i.e., the validation for each patch
would continue even when there are failed tests. A full validation matrix is
common in the scenario that execution results of all tests are required, such
as unified debugging (Lou et al., 2019; Benton et al., [n.d.]).

Table 9 shows the accumulated number of test executions $NT(\mathbb{M}_{f})$
on each APR system. For space limits, we present the average number of all
subjects and buggy versions. It is notable that the full patch validation
matrix collection is extremely resource consuming, e.g., from thousands to
millions of tests are executed when there is no RTS. We can observe a
prominent reduction when RTS strategies at finer granularities are integrated
in APR systems. For example, on average, $RTS_{stmt}$ even reduces millions of
test executions for PraPR. In summary, our results show that RTS should
definitely be applied in the full-matrix scenario.

Finding 7: the full validation matrix is extremely resource-consuming, for
which RTS has a more remarkable impact by reducing even up to millions of test
executions on average.

Table 9. Number of test executions with full matrices

APR $RTS_{no}$ $RTS_{class}$ $RTS_{method}$ $RTS_{stmt}$ PraPR 20,018,923.45
12,495,932.50 9,958,460.34 9,600,813.31 SimFix 70,588.42 11,408.54 1,483.51
1,039.05 AVATAR 11,065.76 2,821.98 1,219.36 591.52 kPar 9,903.78 2,435.97
1,088.50 509.45 TBar 10,256.57 2,856.96 1,237.31 522.12 FixMiner 6,441.49
740.07 144.00 32.86 Dynamoth 596.65 114.32 17.75 16.86 ACS 93.46 5.12 1.44
0.71 Arja 1,435,025.26 177,282.34 32,882.52 23,861.27 Kali 54,751.70 3,999.24
915.59 755.44 GenProg 1,585,441.03 199,088.14 33,824.91 23,840.29 RSRepair
668,634.75 48,785.24 7,650.52 5,131.89

We further compare the reduction achieved by RTS between full and partial
matrices in Figure 5. First, the reduction curves of the full matrix are far
beyond the partial matrix. For example, with the full matrix, all the RTS
strategies can help all APR systems save more than 70% of test executions. In
particular, $RTS_{stmt}$ and $RTS_{method}$ can save more than 95% of test
executions at most cases. Second, the difference between partial and full
matrices varies among different APR systems. For example, Dynamoth achieves a
slightly larger reduction while AVATAR achieves a much larger reduction after
they switch from partial matrices into full matrices. Furthermore, consistent
with the finding for partial matrices, method- and statement-level RTS perform
similarly, and both of them substantially outperform class-level RTS.
Interestingly, their superiority over class-level RTS is even enlarged with
full matrices. In fact, the early-exit mechanism in the partial matrix
prevents more tests from executing, which mitigates the impact from test
selection strategies. In other words, the reduction achieved in the full-
matrix scenario represents the upper bound of the impact.

Finding 8: RTS can remarkably reduce more test executions with full matrices
than with partial matrices. Method- and statement-level RTS still perform
similarly, but their superiority over class-level RTS is further enlarged with
full matrices.

Figure 5. Reduction with full/partial matrices

###  4.5. RQ5: Test Selection+Prioritization

Table 10. Reduction of combining selection and prioritization

APR $RTS_{no}$ (%) $RTS_{class}$ (%) $RTS_{method}$ (%) $RTS_{stmt}$ (%)
$TP_{trp}$ $TP_{tot}$ $TP_{add}$ $TP_{base}$ $TP_{trp}$ $TP_{tot}$ $TP_{add}$
$TP_{base}$ $TP_{trp}$ $TP_{tot}$ $TP_{add}$ $TP_{base}$ $TP_{trp}$ $TP_{tot}$
$TP_{add}$ PraPR 14.53 5.08 15.21 36.96 41.44 35.44 40.38 42.47 44.55 41.13
44.95 43.50 44.82 42.91 44.35 SimFix 2.18 2.24 3.30 19.21 19.80 19.80 20.26
25.84 25.85 25.85 26.02 26.61 26.62 26.62 26.62 AVATAR 0.98 1.44 4.31 11.75
11.28 11.48 10.47 14.23 14.24 14.24 14.31 14.25 14.26 14.25 14.30 kPar -1.75
-0.83 0.66 4.66 3.81 3.93 4.21 5.96 5.98 5.99 4.43 5.99 6.00 6.01 5.90 TBar
-1.08 -1.49 2.79 13.13 11.96 11.66 13.86 15.90 15.90 15.89 15.44 16.73 16.73
16.72 16.14 FixMiner -1.44 -0.62 0.08 6.52 6.54 6.51 6.07 7.09 7.10 7.10 6.64
7.11 7.11 7.11 8.53 Dynamoth 2.07 1.49 5.44 66.93 67.01 66.95 66.89 71.66
71.71 71.67 71.65 71.74 71.79 71.74 71.77 ACS 0.00 0.00 1.74 21.85 21.85 21.85
21.88 22.15 22.15 22.15 22.16 22.21 22.21 22.21 22.21 Arja 20.58 5.33 13.30
37.08 39.33 37.71 39.71 39.71 40.72 40.01 40.92 40.22 40.79 40.48 40.62 Kali
6.80 6.90 10.29 24.34 25.81 25.29 26.16 27.83 28.10 27.87 28.98 28.12 28.17
28.13 28.14 GenProg 14.03 7.85 11.51 24.45 25.79 24.57 25.87 26.24 26.62 26.11
27.14 26.55 26.66 26.49 26.56 RSRepair 14.31 9.00 10.95 29.36 30.47 29.71
31.08 30.75 31.04 30.79 32.02 31.09 31.13 31.06 31.11

So far we have studied the impact of RTS strategies on APR efficiency
controlled in a default test execution order (i.e., the lexicographic order),
and our results suggest that test selection can largely reduce the number of
test executions and improve APR efficiency. In fact, when the early-exit
mechanism is enabled (the default setting for all modern APR systems), the
test execution order can also affect the final number of test executions since
it decides when the first failure-triggering test would be executed.
Therefore, in this RQ, we further combine studied RTS strategies with
different test prioritization strategies and study their joint impact on APR
efficiency. For each patch, we apply RTS to decide the tests for execution and
apply test prioritization to decide their execution order. We still follow the
common practice of all recent APR systems that all the originally-failed tests
are executed before all the originally-passed tests, because the former are
more likely to fail again on patches. Within the originally-failed tests or
the originally-passed tests, we further apply the following test
prioritization strategies to schedule their execution order.

  * •

Baseline prioritization. Tests are scheduled by their lexicographic order,
which is also the default setting in previous RQs. We denote this
prioritization strategy as $TP_{base}$.

  * •

Patch-history-based prioritization. Qi et al. proposed TrpAutoRepair (Qi et
al., 2013), a prioritization strategy based on the patch execution history.
TrpAutoRepair considers the tests that have failed on more validated patches
as more likely to fail on the current patch and assigns them a higher
execution priority. We consider TrpAutoRepair since it is the state-of-the-art
test prioritization strategy specifically designed for APR, and we make the
first attempt to study the performance of its combination with RTS. We denote
this prioritization strategy as $TP_{trp}$.

  * •

Regression test prioritization. Test prioritization techniques have also been
widely studied in traditional regression testing to reorder regression tests
for early detection of regression bugs (Li et al., 2007; Lou et al., 2019;
Wang et al., 2017). Since each patch can be treated as a new revision in
regression testing, we can also naturally apply regression test prioritization
techniques for APR. Note that, surprisingly, we are the first to study
regression test prioritization for APR. In particular, we consider two most
widely studied approaches, i.e., the _total_ and _additional_ regression test
prioritization techniques based on statement coverage (Rothermel et al.,
1999). They share the similar intuition that tests covering more statements or
more yet-uncovered statements are more likely to detect regression bugs. We
denote them as $TP_{tot}$ and $TP_{add}$, respectively.

In Table 10, we present the reductions achieved by the combination of
different test selection and prioritization strategies when compared to
$RTS_{no}$ with $TP_{base}$. The highest reduction within each RTS strategy is
highlighted. Based on the table, we have the following observations. First,
combining test selection with test prioritization can further improve APR
efficiency. However, compared to test selection, test prioritization brings
much lower reduction ratios. For example, when PraPR adopts no RTS, the best
prioritization (i.e., $TP_{add}$) can achieve only 15.21% reduction, but the
worst RTS alone (i.e., $RTS_{class}$ with $TP_{base}$) can achieve 36.96%
reduction. This indicates that test selection strategies play a more essential
role in efficiency optimization. Second, the best prioritization strategy
varies when combined with different RTS strategies. For most APR systems,
surprisingly, $TP_{add}$ reduces the most test executions when combined with
$RTS_{no}$, $RTS_{class}$, or $RTS_{method}$. To the best of our knowledge,
this is the first study demonstrating that in the APR scenario, the
traditional additional prioritization technique even outperforms state-of-the-
art APR-specific prioritization technique at the most cases. One potential
reason may be that the generated patches share little commonality and it is
challenging for $TP_{trp}$ to infer the results of un-executed patches from
historical patch executions. In contrast, $TP_{add}$ executes tests with
diverse statement coverage as early as possible and thus can advance buggy
patch detection. Furthermore, it is notable that $TP_{add}$ becomes less
effective than $TP_{trp}$ when combined with $RTS_{stmt}$. Intuitively, the
tests selected by coarser-grained coverage criteria (e.g., class-level RTS)
tend to exhibit a more diverse distribution of statement coverage than the
tests selected by the statement-level RTS. Therefore, when combined with
$RTS_{stmt}$, $TP_{trp}$ has a stronger capability in distinguishing tests
based on their historical execution results. Third, we can observe that
$TP_{add}$ outperforms $TP_{tot}$ in terms of the repair efficiency, which is
consistent with the previous findings in traditional regression testing (Lu et
al., 2016; Rothermel et al., 1999).

Finding 9: this study demonstrates for the first time that test selection
outperforms test prioritization for APR, and they can be combined to further
improve APR efficiency. Also, surprisingly, traditional regression test
prioritization $TP_{add}$ outperforms state-of-the-art APR-specific test
prioritization $TP_{trp}$ for most cases.

##  5\. Related work

Since we have discussed the background about RTS and APR in Section 2, in this
section, we mainly focus on the closely related work for improving APR
efficiency. APR is resource consuming due to the large number of generated
patches and the non-trivial costs in validating each patch. In addition to the
large body of APR approaches that aim to reduce the number of generated
patches (as mentioned in Section 2.2), researchers have proposed various
approaches to reduce the test executions for each patch. For example, Qi et
al. (Qi et al., 2013) proposed a patch-history-based test prioritization
strategy, TrpAutoRepair, to schedule test execution order, so that the
failure-triggering tests could be executed earlier. Later, Mehne et al. (Mehne
et al., 2018) proposed to reduce the number of test executions for each patch
based on statement coverage (i.e., statement-level RTS). Similarly, as
suggested by Table 1, several existing APR systems also leveraged RTS to
accelerate patch validation, e.g., the ARJA family (Yuan and Banzhaf, 2020)
adopted statement-level RTS and CapGen (Wen et al., 2018) adopted class-level
RTS. However, there is limited study on the effectiveness of RTS techniques,
and our work conducts the first study to extensively investigate the impact of
representative RTS techniques on a wide range of state-of-the-art APR systems.

Besides reducing the number of test executions, researchers have also looked
into speeding up the time for each test/patch execution during APR. For
example, JAID (Chen et al., 2017) and SketchFix (Hua et al., 2018) transformed
the buggy programs into meta-programs or sketches to accelerate patch
validation. Ghanbari et al.(Ghanbari et al., 2019) proposed a bytecode-level
APR approach which requires no patch compilation and system reloading. More
recently, Chen et al. (Chen and Zhang, 2020) leveraged on-the-fly patch
validation to save patch loading and execution time to speed up all existing
source-code-level APR techniques. Such techniques are orthogonal to test
execution reduction as studied in this work, and can be combined with test
execution reduction to further reduce APR cost.

In addition to cost reduction techniques, researchers have also paid attention
to the efficiency measurement in APR systems. Most of existing work assessed
repair efficiency based on the CPU time spent during repair process, which has
been suggested as unstable and biased in previous work (Liu et al., 2020).
Besides time costs, Ghanbari et al. (Ghanbari et al., 2019) reported the
number of patches generated by the APR system PraPR. Liu et al. (Liu et al.,
2020) further performed an extensive study to measure the efficiency of
existing 16 APR systems based on the number of generated patches. Compared to
previous work, our work makes the first attempt to study the APR efficiency
based on the number of test executions. In addition, we also address potential
threats from the inconsistent RTS strategies in existing APR efficiency work.
Our results reveal that the number of patches widely used for measuring APR
efficiency can incur skewed conclusions, and inconsistent RTS configurations
can further skew the conclusion.

##  6\. Conclusion

In this work, we first pointed out an important test execution optimization
(RTS) largely neglected and inconsistently configured by existing APR systems.
We then performed the first extensive study on the impact of 3 representative
RTS strategies on 12 state-of-the-art APR systems with over 2M patches. Our
study has provided various practical guidelines for future APR work,
including: (1) the number of patches widely used for measuring APR efficiency
and the use of inconsistent RTS configurations can both incur skewed
conclusions; (2) all studied RTS techniques can substantially improve APR
efficiency and should be considered in future APR work; (3) method- and
statement-level RTS substantially outperform class-level RTS, and are more
recommended for APR; (4) RTS techniques can substantially outperform state-of-
the-art test prioritization techniques for APR, and they can be combined to
further improve APR efficiency; and (5) traditional regression test
prioritization widely studied in regression testing performs even better than
APR-specific test prioritization when combined with most RTS techniques.
Furthermore, we have also presented the detailed impact of different patch
categories and patch validation strategies on our findings.

## References

  * (1)
  * apa (2020a) 2020a.  Apache Camel.  http://camel.apache.org/.
  * apa (2020b) 2020b.  Apache Commons Math.  https://commons.apache.org/proper/commons-math/.
  * apa (2020c) 2020c.  Apache CXF.  https://cxf.apache.org/.
  * ASM (2020) 2020\.  Asm java bytecode manipulation and analysis framework.  http://asm.ow2.org.
  * jav (2020) 2020\.  JavaAgent.  https://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package-summary.html.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Neural Machine Translation by Jointly Learning to Align and Translate. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1409.0473
  * Benton et al. ([n.d.]) Samuel Benton, Xia Li, Yiling Lou, and Lingming Zhang. [n.d.].  On the Effectiveness of Unified Debugging: An Extensive Study on 16 Program Repair Systems.  ([n. d.]).
  * Chen et al. (2017) Liushan Chen, Yu Pei, and Carlo A. Furia. 2017.  Contract-based program repair without the contracts. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017_ , Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (Eds.). IEEE Computer Society, 637–647.  https://doi.org/10.1109/ASE.2017.8115674
  * Chen and Zhang (2020) Lingchao Chen and Lingming Zhang. 2020.  Fast and Precise On-the-fly Patch Validation for All.  _arXiv preprint arXiv:2007.11449_ (2020).
  * Chen et al. (2019) Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019.  Sequencer: Sequence-to-sequence learning for end-to-end program repair.  _IEEE Transactions on Software Engineering_ (2019).
  * Dauphin et al. (2017) Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017.  Language Modeling with Gated Convolutional Networks. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ _(Proceedings of Machine Learning Research, Vol. 70)_ , Doina Precup and Yee Whye Teh (Eds.). PMLR, 933–941.  http://proceedings.mlr.press/v70/dauphin17a.html
  * Durieux et al. (2017) Thomas Durieux, Benoit Cornu, Lionel Seinturier, and Martin Monperrus. 2017.  Dynamic patch generation for null pointer exceptions using metaprogramming. In _IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER 2017, Klagenfurt, Austria, February 20-24, 2017_ , Martin Pinzger, Gabriele Bavota, and Andrian Marcus (Eds.). IEEE Computer Society, 349–358.  https://doi.org/10.1109/SANER.2017.7884635
  * Durieux and Monperrus (2016) Thomas Durieux and Martin Monperrus. 2016.  DynaMoth: dynamic code synthesis for automatic program repair. In _Proceedings of the 11th International Workshop on Automation of Software Test, AST@ICSE 2016, Austin, Texas, USA, May 14-15, 2016_ , Christof J. Budnik, Gordon Fraser, and Francesca Lonetti (Eds.). ACM, 85–91.  https://doi.org/10.1145/2896921.2896931
  * Ghanbari et al. (2019) Ali Ghanbari, Samuel Benton, and Lingming Zhang. 2019\.  Practical program repair via bytecode mutation. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019_ , Dongmei Zhang and Anders Møller (Eds.). ACM, 19–30.  https://doi.org/10.1145/3293882.3330559
  * Gligoric et al. (2015) Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015\.  Practical regression test selection with dynamic file dependencies. In _Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, Baltimore, MD, USA, July 12-17, 2015_ , Michal Young and Tao Xie (Eds.). ACM, 211–222.  https://doi.org/10.1145/2771783.2771784
  * Goues et al. (2013) Claire Le Goues, Stephanie Forrest, and Westley Weimer. 2013\.  Current challenges in automatic software repair.  _Softw. Qual. J._ 21, 3 (2013), 421–443.  https://doi.org/10.1007/s11219-013-9208-0
  * Harrold et al. (2001) Mary Jean Harrold, James A. Jones, Tongyu Li, Donglin Liang, Alessandro Orso, Maikel Pennings, Saurabh Sinha, S. Alexander Spoon, and Ashish Gujarathi. 2001.  Regression Test Selection for Java Software. In _Proceedings of the 2001 ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications, OOPSLA 2001, Tampa, Florida, USA, October 14-18, 2001_ , Linda M. Northrop and John M. Vlissides (Eds.). ACM, 312–326.  https://doi.org/10.1145/504282.504305
  * Hua et al. (2018) Jinru Hua, Mengshi Zhang, Kaiyuan Wang, and Sarfraz Khurshid. 2018\.  Towards practical program repair with on-demand candidate generation. In _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 12–23.  https://doi.org/10.1145/3180155.3180245
  * Jiang et al. (2019) Jiajun Jiang, Luyao Ren, Yingfei Xiong, and Lingming Zhang. 2019\.  Inferring Program Transformations From Singular Examples via Big Code. In _34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019_. IEEE, 255–266.  https://doi.org/10.1109/ASE.2019.00033
  * Jiang et al. (2018) Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018.  Shaping program repair space with existing patches and similar code. In _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 2018_ , Frank Tip and Eric Bodden (Eds.). ACM, 298–309.  https://doi.org/10.1145/3213846.3213871
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst. 2014\.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In _International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014_ , Corina S. Pasareanu and Darko Marinov (Eds.). ACM, 437–440.  https://doi.org/10.1145/2610384.2628055
  * Kim et al. (2013) Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013\.  Automatic patch generation learned from human-written patches. In _35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013_ , David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, 802–811.  https://doi.org/10.1109/ICSE.2013.6606626
  * Kim and Kim (2019) Jindae Kim and Sunghun Kim. 2019.  Automatic patch generation with context-based change application.  _Empir. Softw. Eng._ 24, 6 (2019), 4071–4106.  https://doi.org/10.1007/s10664-019-09742-5
  * Koyuncu et al. (2020) Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020\.  FixMiner: Mining relevant fix patterns for automated program repair.  _Empir. Softw. Eng._ 25, 3 (2020), 1980–2024.  https://doi.org/10.1007/s10664-019-09780-z
  * Kung et al. (1995) David Chenho Kung, Jerry Gao, Pei Hsia, Jeremy Lin, and Yasufumi Toyoshima. 1995.  Class Firewall, Test Order, and Regression Testing of Object-Oriented Programs.  _J. Object Oriented Program._ 8, 2 (1995), 51–65.
  * Le et al. (2017) Xuan-Bach D. Le, Duc-Hiep Chu, David Lo, Claire Le Goues, and Willem Visser. 2017\.  S3: syntax- and semantic-guided repair synthesis via programming by examples. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017_ , Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 593–604.  https://doi.org/10.1145/3106237.3106309
  * Le et al. (2016) Xuan-Bach D. Le, David Lo, and Claire Le Goues. 2016\.  History Driven Program Repair. In _IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016 - Volume 1_. IEEE Computer Society, 213–224.  https://doi.org/10.1109/SANER.2016.76
  * Legunsen et al. (2016) Owolabi Legunsen, Farah Hariri, August Shi, Yafeng Lu, Lingming Zhang, and Darko Marinov. 2016\.  An extensive study of static regression test selection in modern software evolution. In _Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016_ , Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (Eds.). ACM, 583–594.  https://doi.org/10.1145/2950290.2950361
  * Li et al. (2020) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020.  DLFix: context-based code transformation learning for automated program repair. In _ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June \- 19 July, 2020_ , Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 602–614.  https://doi.org/10.1145/3377811.3380345
  * Li et al. (2007) Zheng Li, Mark Harman, and Robert M. Hierons. 2007.  Search Algorithms for Regression Test Case Prioritization.  _IEEE Trans. Software Eng._ 33, 4 (2007), 225–237.  https://doi.org/10.1109/TSE.2007.38
  * Liu et al. (2019a) Kui Liu, Anil Koyuncu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, and Yves Le Traon. 2019a.  You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems. In _12th IEEE Conference on Software Testing, Validation and Verification, ICST 2019, Xi’an, China, April 22-27, 2019_. IEEE, 102–113.  https://doi.org/10.1109/ICST.2019.00020
  * Liu et al. (2019b) Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. 2019b.  AVATAR: Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations. In _26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019_ , Xinyu Wang, David Lo, and Emad Shihab (Eds.). IEEE, 456–467.  https://doi.org/10.1109/SANER.2019.8667970
  * Liu et al. (2019c) Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. 2019c.  TBar: revisiting template-based automated program repair. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019_ , Dongmei Zhang and Anders Møller (Eds.). ACM, 31–42.  https://doi.org/10.1145/3293882.3330577
  * Liu et al. (2018) Kui Liu, Anil Koyuncu, Kisub Kim, Dongsun Kim, and Tegawendé F. Bissyandé. 2018\.  LSRepair: Live Search of Fix Ingredients for Automated Program Repair. In _25th Asia-Pacific Software Engineering Conference, APSEC 2018, Nara, Japan, December 4-7, 2018_. IEEE, 658–662.  https://doi.org/10.1109/APSEC.2018.00085
  * Liu et al. (2020) Kui Liu, Shangwen Wang, Anil Koyuncu, Kisub Kim, Tegawendé F. Bissyandé, Dongsun Kim, Peng Wu, Jacques Klein, Xiaoguang Mao, and Yves Le Traon. 2020.  On the efficiency of test suite based program repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs. In _ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020_ , Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 615–627.  https://doi.org/10.1145/3377811.3380338
  * Liu and Zhong (2018) Xuliang Liu and Hao Zhong. 2018.  Mining stackoverflow for program repair. In _25th International Conference on Software Analysis, Evolution and Reengineering, SANER 2018, Campobasso, Italy, March 20-23, 2018_ , Rocco Oliveto, Massimiliano Di Penta, and David C. Shepherd (Eds.). IEEE Computer Society, 118–129.  https://doi.org/10.1109/SANER.2018.8330202
  * Long et al. (2017) Fan Long, Peter Amidon, and Martin Rinard. 2017.  Automatic inference of code transforms for patch generation. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017_ , Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 727–739.  https://doi.org/10.1145/3106237.3106253
  * Long and Rinard (2016) Fan Long and Martin C. Rinard. 2016.  An analysis of the search spaces for generate and validate patch generation systems. In _Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016_ , Laura K. Dillon, Willem Visser, and Laurie A. Williams (Eds.). ACM, 702–713.  https://doi.org/10.1145/2884781.2884872
  * Lou et al. (2019) Yiling Lou, Junjie Chen, Lingming Zhang, and Dan Hao. 2019\.  Chapter One - A Survey on Regression Test-Case Prioritization.  _Adv. Comput._ 113 (2019), 1–46.  https://doi.org/10.1016/bs.adcom.2018.10.001
  * Lou et al. (2020) Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020.  Can automated program repair refine fault localization? a unified debugging approach. In _ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020_ , Sarfraz Khurshid and Corina S. Pasareanu (Eds.). ACM, 75–87.  https://doi.org/10.1145/3395363.3397351
  * Lu et al. (2016) Yafeng Lu, Yiling Lou, Shiyang Cheng, Lingming Zhang, Dan Hao, Yangfan Zhou, and Lu Zhang. 2016.  How does regression test prioritization perform in real-world software evolution?. In _Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016_ , Laura K. Dillon, Willem Visser, and Laurie A. Williams (Eds.). ACM, 535–546.  https://doi.org/10.1145/2884781.2884874
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  CoCoNuT: combining context-aware neural translation models using ensemble for program repair. In _ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020_ , Sarfraz Khurshid and Corina S. Pasareanu (Eds.). ACM, 101–114.  https://doi.org/10.1145/3395363.3397369
  * Martinez and Monperrus (2016) Matias Martinez and Martin Monperrus. 2016.  ASTOR: a program repair library for Java (demo). In _Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016_ , Andreas Zeller and Abhik Roychoudhury (Eds.). ACM, 441–444.  https://doi.org/10.1145/2931037.2948705
  * Martinez and Monperrus (2018) Matias Martinez and Martin Monperrus. 2018.  Ultra-Large Repair Search Space with Automatically Mined Templates: The Cardumen Mode of Astor. In _Search-Based Software Engineering - 10th International Symposium, SSBSE 2018, Montpellier, France, September 8-9, 2018, Proceedings_ _(Lecture Notes in Computer Science, Vol. 11036)_ , Thelma Elita Colanzi and Phil McMinn (Eds.). Springer, 65–86.  https://doi.org/10.1007/978-3-319-99241-9_3
  * Mehne et al. (2018) Ben Mehne, Hiroaki Yoshida, Mukul R. Prasad, Koushik Sen, Divya Gopinath, and Sarfraz Khurshid. 2018\.  Accelerating Search-Based Program Repair. In _11th IEEE International Conference on Software Testing, Verification and Validation, ICST 2018, Västerås, Sweden, April 9-13, 2018_. IEEE Computer Society, 227–238.  https://doi.org/10.1109/ICST.2018.00031
  * Monperrus (2020) Martin Monperrus. 2020\.  The living review on automated program repair.  (2020).
  * Orso et al. (2004) Alessandro Orso, Nanjuan Shi, and Mary Jean Harrold. 2004\.  Scaling regression testing to large software systems. In _Proceedings of the 12th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2004, Newport Beach, CA, USA, October 31 - November 6, 2004_ , Richard N. Taylor and Matthew B. Dwyer (Eds.). ACM, 241–251.  https://doi.org/10.1145/1029894.1029928
  * Qi et al. (2013) Yuhua Qi, Xiaoguang Mao, and Yan Lei. 2013.  Efficient Automated Program Repair through Fault-Recorded Testing Prioritization. In _2013 IEEE International Conference on Software Maintenance, Eindhoven, The Netherlands, September 22-28, 2013_. IEEE Computer Society, 180–189.  https://doi.org/10.1109/ICSM.2013.29
  * Ren et al. (2004) Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia C. Chesley. 2004.  Chianti: a tool for change impact analysis of java programs. In _Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2004, October 24-28, 2004, Vancouver, BC, Canada_ , John M. Vlissides and Douglas C. Schmidt (Eds.). ACM, 432–448.  https://doi.org/10.1145/1028976.1029012
  * Rothermel and Harrold (1993) Gregg Rothermel and Mary Jean Harrold. 1993.  A Safe, Efficient Algorithm for Regression Test Selection. In _Proceedings of the Conference on Software Maintenance, ICSM 1993, Montréal, Quebec, Canada, September 1993_ , David N. Card (Ed.). IEEE Computer Society, 358–367.  https://doi.org/10.1109/ICSM.1993.366926
  * Rothermel and Harrold (1997) Gregg Rothermel and Mary Jean Harrold. 1997.  A Safe, Efficient Regression Test Selection Technique.  _ACM Trans. Softw. Eng. Methodol._ 6, 2 (1997), 173–210.  https://doi.org/10.1145/248233.248262
  * Rothermel et al. (1999) Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 1999.  Test Case Prioritization: An Empirical Study. In _1999 International Conference on Software Maintenance, ICSM 1999, Oxford, England, UK, August 30 - September 3, 1999_. IEEE Computer Society, 179–188.  https://doi.org/10.1109/ICSM.1999.792604
  * Saha et al. (2017) Ripon K. Saha, Yingjun Lyu, Hiroaki Yoshida, and Mukul R. Prasad. 2017.  ELIXIR: effective object oriented program repair. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017_ , Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (Eds.). IEEE Computer Society, 648–659.  https://doi.org/10.1109/ASE.2017.8115675
  * Saha et al. (2019) Seemanta Saha, Ripon K. Saha, and Mukul R. Prasad. 2019\.  Harnessing evolution for multi-hunk program repair. In _Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019_ , Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (Eds.). IEEE / ACM, 13–24.  https://doi.org/10.1109/ICSE.2019.00020
  * Shi et al. (2019) August Shi, Milica Hadzi-Tanovic, Lingming Zhang, Darko Marinov, and Owolabi Legunsen. 2019\.  Reflection-aware static regression test selection.  _Proc. ACM Program. Lang._ 3, OOPSLA (2019), 187:1–187:29.  https://doi.org/10.1145/3360613
  * Smith et al. (2015) Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. 2015.  Is the cure worse than the disease? overfitting in automated program repair. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015_ , Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (Eds.). ACM, 532–543.  https://doi.org/10.1145/2786805.2786825
  * Wang et al. (2017) Song Wang, Jaechang Nam, and Lin Tan. 2017.  QTEP: quality-aware test case prioritization. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017_ , Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 523–534.  https://doi.org/10.1145/3106237.3106258
  * Weimer et al. (2013) Westley Weimer, Zachary P. Fry, and Stephanie Forrest. 2013\.  Leveraging program equivalence for adaptive program repair: Models and first results. In _2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11-15, 2013_ , Ewen Denney, Tevfik Bultan, and Andreas Zeller (Eds.). IEEE, 356–366.  https://doi.org/10.1109/ASE.2013.6693094
  * Wen et al. (2018) Ming Wen, Junjie Chen, Rongxin Wu, Dan Hao, and Shing-Chi Cheung. 2018.  Context-aware patch generation for better automated program repair. In _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 1–11.  https://doi.org/10.1145/3180155.3180233
  * Wen et al. (2020) Ming Wen, Yepang Liu, and Shing-Chi Cheung. 2020.  Boosting automated program repair with bug-inducing commits. In _ICSE-NIER 2020: 42nd International Conference on Software Engineering, New Ideas and Emerging Results, Seoul, South Korea, 27 June - 19 July, 2020_ , Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 77–80.  https://doi.org/10.1145/3377816.3381743
  * White et al. (2019) Martin White, Michele Tufano, Matias Martinez, Martin Monperrus, and Denys Poshyvanyk. 2019\.  Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities. In _26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019_ , Xinyu Wang, David Lo, and Emad Shihab (Eds.). IEEE, 479–490.  https://doi.org/10.1109/SANER.2019.8668043
  * Wilcoxon (1992) Frank Wilcoxon. 1992\.  Individual comparisons by ranking methods.  In _Breakthroughs in statistics_. Springer, 196–202.
  * Xin and Reiss (2017) Qi Xin and Steven P. Reiss. 2017.  Leveraging syntax-related code for automated program repair. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017_ , Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (Eds.). IEEE Computer Society, 660–670.  https://doi.org/10.1109/ASE.2017.8115676
  * Xiong et al. (2017) Yingfei Xiong, Jie Wang, Runfa Yan, Jiachen Zhang, Shi Han, Gang Huang, and Lu Zhang. 2017.  Precise condition synthesis for program repair. In _Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017_ , Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard (Eds.). IEEE / ACM, 416–426.  https://doi.org/10.1109/ICSE.2017.45
  * Xuan et al. (2017) Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian R. Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2017.  Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs.  _IEEE Trans. Software Eng._ 43, 1 (2017), 34–55.  https://doi.org/10.1109/TSE.2016.2560811
  * Yi et al. (2018) Jooyong Yi, Shin Hwei Tan, Sergey Mechtaev, Marcel Böhme, and Abhik Roychoudhury. 2018\.  A correlation study between automated program repair and test-suite metrics. In _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 24.  https://doi.org/10.1145/3180155.3182517
  * Yuan and Banzhaf (2020) Yuan Yuan and Wolfgang Banzhaf. 2020.  ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming.  _IEEE Trans. Software Eng._ 46, 10 (2020), 1040–1067.  https://doi.org/10.1109/TSE.2018.2874648
  * Zhang (2018) Lingming Zhang. 2018\.  Hybrid regression test selection. In _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 199–209.  https://doi.org/10.1145/3180155.3180198