# Generating Exceptional Behavior Tests with
Reasoning Augmented Large Language Models

Jiyang Zhang  The University of Texas at Austin, USA
<EMAIL_ADDRESS>Yu Liu  The University of Texas at Austin, USA
<EMAIL_ADDRESS>Pengyu Nie  Junyi Jessy Li  University of Waterloo,
Canada
<EMAIL_ADDRESS>The University of Texas at Austin, USA
<EMAIL_ADDRESS>Milos Gligoric  The University of Texas at Austin,
USA
<EMAIL_ADDRESS>
###### Abstract

Many popular programming languages, including C#, Java, and Python, support
exceptions. Exceptions are thrown during program execution if an unwanted
event happens, e.g., a method is invoked with an illegal argument value.
Software developers write exceptional behavior tests (EBTs) to check that
their code detects unwanted events and throws appropriate exceptions. Prior
research studies have shown the importance of EBTs, but those studies also
highlighted that developers put most of their efforts on “happy paths”, e.g.,
paths without unwanted events. To help developers fill the gap, we present the
first framework, dubbed exLóng, that automatically generates EBTs. exLóng is a
large language model instruction-tuned from CodeLlama and embeds reasoning
about traces that lead to throw statements, conditional expressions that guard
throw statements, and non-exceptional behavior tests that execute similar
traces. We compare exLóng with the state-of-the-art models for test generation
(CAT-LM) and one of the strongest foundation models (GPT3.5), as well as with
analysis-based tools for test generation (Randoop and EvoSuite). Our results
show that exLóng outperforms existing models and tools. Furthermore, we
contributed several pull requests to open-source projects and 23 EBTs
generated by exLóng were already accepted.

##  I Introduction

⬇

1class SearchCommandParser extends CommandParser {

2 static final char CHR_SPACE = ’ ’;

3 static final char CHR_CR = ’\r’;

4

5 public SearchTerm searchTerm(ImapRequestLineReader request) throws
ProtocolException {

6 ...

7 char next;

8 while ((next = request.nextChar()) != ’\n’ &&

9 next != CHR_CR) {

10 next = request.consumeAll(CHR_SPACE);

11 if (isAtomSpecial(next)) {

12 if (next == ’(’) {

13 ...

14 } else if (next == ’)’) {

15 ...

16 } else {

17 throw new IllegalStateException("Unsupported atom special char <" + next +
">");

18 }

19 }

20 ...

21 }

22 return handleOperators();

23 }

24}

target throw statement

((a)) Method under test: $\mathtt{searchTerm}$.

⬇

1public class SearchCommandParserTest {

2 private SearchTerm parse(String line)

3 throws ProtocolException {

4 final byte[] bytes = (line.endsWith("\n")?line:(line + ’\n’)).getBytes();

5 ByteArrayInputStream ins = new ByteArrayInputStream(bytes);

6 return new SearchCommandParser().searchTerm(new ImapRequestLineReader(ins,
null));

7 }

8

9 @Test(expected = IllegalStateException.class)

10 public void testUnsupportedAtomSpecialChar()

11 throws ProtocolException {

12 parse("*");

13 }

14}

method under test exceptional behavior test

((b)) Exceptional behavior test for the highlighted statement above.

Figure 1: An EBT (‘testUnsupportedAtomSpecialChar’) from greenmail-mail-
test/greenmail and the target throw statement.

Many popular programming languages, including C#, Java, and Python, support
exceptions [19, 15, 54]. Exceptions are thrown during program execution if an
unwanted event happens, e.g., a method is invoked with an illegal argument
value. To throw an exception, a developer writes a _throw statement_ in their
code. These throw statements are commonly guarded with conditional statements
(e.g., $\mathtt{if}$), as exceptions should be thrown only under exceptional
circumstances. Figure 1(a) shows a code snippet, in Java, that throws an
$\mathtt{IllegalStateException}$ (line 17) when the $\mathtt{next}$ character,
parsed from argument $\mathtt{request}$, is identified as a special atom (line
11) but is neither an opening (line 12) nor a closing parenthesis (line 14).

Software developers write _exceptional behavior tests_ (EBTs) to check that
their code properly detects unwanted events and throws desired exceptions.
Figure 1(b) shows an example EBT. An EBT, similar to a _non-exceptional
behavior test_ (non-EBT), first performs necessary setup of the system under
test, e.g., creates objects (lines 4-5), then invokes a method under test
(line 6), and finally checks the expected behavior (line 9). For an EBT, the
expected behavior is that an exception was thrown and the type of the
exception matches the one specified by a developer.

Prior research has studied EBTs in practice [8, 2, 14, 26, 25] and observed
that most projects already have some EBTs, but that the number of EBTs is not
as high as the number of non-EBTs. Simply put, developers focus on “happy
paths” and have limited time to test exceptional behaviors. Furthermore,
through interviews and surveys [8, 26], prior studies confirmed the importance
of EBTs and developers’ desire to improve the testing of exceptional
behaviors.

Sadly, tool support for automatically generating EBTs is limited. Most
existing analysis-based test generation tools (e.g., Randoop [32, 38] and
EvoSuite [12]) and learning-based test generation tools (e.g., CAT-LM [36] and
TeCo [29]) have no special settings for targeting EBTs and are primarily
evaluated on non-EBTs. Random test generation tools can be guided by
reinforcement learning to target exceptional behaviors [1], but the generation
works only on the entire code base, and not for a specific throw statement
that a developer might select. Additionally, tests produced by analysis-based
tools lack readability [7, 33, 6].

We present the first framework, dubbed exLóng, an _instruction-tuned_ large
language model (LLM) that automatically generates EBTs. LLMs are shown to be
effective in code generation, including test generation [52, 36, 29, 23, 59].
Such strong prior provides a good foundation yet is not enough. EBTs
contribute to only a very small percentage in existing code bases, i.e., they
are not well-represented in LLM training data. The reasoning trajectory that
trigger an EBT is _uncommon_ and not captured by the Causal Language Modeling
pretraining objective [5, 35], and thus LLMs trained on typical (non-EBT) code
does not perform well on the task of generating EBTs.

Using CodeLlama [39] as its foundation, exLóng is instruction fine-tuned [56,
58, 41] with a novel task instruction and fine-tuning data, designed
specifically to embed the reasoning about a context that includes: (a) traces
that lead to target throw statements, (b) guard expressions (i.e., conditional
expressions that guard those throw statements), and (c) non-EBTs that execute
similar traces. This context is used as the input to generate an EBT that
triggers the target throw statement. The EBT that we already showed in Figure
1(b) was generated by exLóng.

We assess the power of exLóng using two use cases. In the first use case,
which we call _developer-oriented use case_ , a developer selects a method
under test and a target throw statement, as well as a destination test file.
exLóng takes these inputs and automatically generates an EBT that executes the
target throw statement.

In this use case, we compare exLóng with the state-of-the-art models for test
generation (CAT-LM [36]) and one of the strongest foundation models (GPT3.5
[5, 31]). We use a number of standard metrics (BLEU [34], CodeBLEU [37], edit
similarity [60, 45] and exact match), as well as metrics specific to code,
including percentage of compilable tests, executable tests, and executable
tests that cover the target throw statement. Our results show that exLóng
generates 68.6% and 10.9% more executable EBTs than CAT-LM and GPT3.5,
respectively.

In the second use case, which we call _machine-oriented use case_ , a
developer uses exLóng to automatically generate EBTs for the entire code base
with the goal to cover all existing throw statements (with one EBT per
statement). exLóng takes the entire code base as input, finds throw statements
that are in public methods already covered by at least one non-EBT and
generates one EBT for each of the throw statements. This use case is similar
to the traditional test generation setup targeted by analysis-based generation
tools.

In this use case, we compare exLóng with popular analysis-based test
generation tools: Randoop [32, 38] and EvoSuite [12]. Although tools
complement each other (i.e., each tool can generate EBTs for some target throw
statements that other tools cannot), our findings show that exLóng outperforms
Randoop and EvoSuite.

Finally, we selected a subset of EBTs generated by exLóng and created pull
requests for several open-source projects. By the time of this writing, 23
tests generated by exLóng have already been accepted by developers of those
projects.

The key contributions of this paper include:

  * •

Task. We define a novel task for LLMs: generating exceptional behavior tests
(EBTs).

  * •

Model. We designed and implemented exLóng, an instruction-tuned LLM built on
CodeLlama, which reasons about traces to methods that contain throw
statements, guard expressions, and non-EBTs that cover similar traces.

  * •

Use cases. We recognized two use cases for exLóng: developer- and machine-
oriented use cases.

  * •

Evaluation. We assess the power of exLóng in both use cases. In developer-
oriented use case, we compare exLóng with existing models for code and test
generation. In machine-oriented use case, we compare exLóng with analysis-
based testing tools: Randoop and EvoSuite. We find that exLóng outperforms
existing state-of-the-art models and tools.

  * •

Dataset. We developed a novel dataset for the presented task. We plan to make
this dataset publicly available upon acceptance of this paper.

##  II Task

Figure 2: Overview of exLóng. Two use cases for exLóng: (1) developer-oriented
use case and (2) machine-oriented use case. In the developer-oriented use
case, a developer specify the method under test, a target throw statement and
a destination test file and ask exLóng to generate an EBT that cover the
target throw statement. In the machine-oriented use case, a developer gives an
entire repository to exLóng.

At a high level, exLóng is designed to help software developers write EBTs
that cover the throw statements within the given repository. We propose two
use cases for exLóng: developer-oriented use case (Section II-A) and machine-
oriented use case (Section II-B). Note that in this work we do _not_ consider
generating EBTs that cover throw statements in the dependency libraries of the
given repository, e.g. $\mathtt{ArithmeticException}$ thrown from the
$\mathtt{java.lang.Math}$, as we assume that those throw statements are
already covered by EBTs available in dependency libraries; developers can
configure exLóng to include such throw statements if necessary.

###  II-A Developer-oriented use case

In the developer-oriented use case (box 1 in Figure 2), a developer will
specify the _method under test_ (MUT), a _target throw statement_ , and a
_destination test file_ , then ask exLóng to generate an EBT that invokes MUT
and triggers the target throw statement. When the developer specifies a target
throw statement that is not within the MUT, exLóng will first use a static
program analysis technique to find all possible throw statements reachable
through a sequence of method calls starting from the MUT, and then prompt the
developer to select the target statement from the list.

###  II-B Machine-oriented use case

In the machine-oriented use case (box 2 in Figure 2), a developer gives an
entire repository to exLóng. exLóng will first find public methods and throw
statements in those methods. Furthermore, exLóng will heuristically select a
destination test file. Then, exLóng will create (public method, throw
statement, destination test file) triples. Finally, it will generate one EBT
to test each triple, without requiring any developer intervention. exLóng uses
static+dynamic program analyses to obtain necessary context such as traces
leading to throw statements and guard expressions.

##  III exLóng

Figure 2 shows the workflow of exLóng. Given an MUT ($\mathtt{m_{ut}}$), a
target throw statement ($\mathtt{s}$), and a destination test file
($\mathtt{d}$), exLóng reasons about the following context, collected using
static and dynamic program analyses ( 3):

  * •

stack trace ($\mathtt{r}$): the sequence of method invocations that start from
the MUT and lead to the target throw statement.

  * •

guard expression ($\mathtt{g}$): the logical formula representing the
constraints on the symbolic variables that must be true to follow that
particular trace.

  * •

relevant non-EBTs ($\hat{\mathtt{T}}_{\mathtt{neb}}$): a subset of non-EBTs
whose snippets and coding styles may help with generating EBTs for the given
inputs.

The above constitutes the prompt
$\mathtt{p}=(\mathtt{m_{ut}},\mathtt{s},\mathtt{d},\mathtt{r},\mathtt{g},\hat{\mathtt{T}}_{\mathtt{neb}})$
that encompasses the task inputs and the task context ( 4). During training,
exLóng is instruction-tuned over a foundation LLM to produce EBT
($\mathtt{t}_{\mathtt{eb}}$) conditioned on the input $\mathtt{p}$. During
inference, exLóng generates the EBT ( 6) given the prompt.

While the format of the LLM’s input ($\mathtt{p}$) and output
($\mathtt{t}_{\mathtt{eb}}$) is static, the steps for preparing the task
inputs and reasoning about the task context differ for training and inference,
and differ slightly between the developer- and machine-oriented use cases. We
describe the methodology for training exLóng in Section III-A and inference in
Section III-B.

###  III-A Training

Algorithm 1 Collecting training corpus.

1:input: $\mathtt{T}_{\mathtt{eb}}$, $\mathtt{T}_{\mathtt{neb}}$ \- existing
EBTs and non-EBTs

2: $\mathtt{SUT}$ \- the system under test

3:output: $\mathcal{C}$ \- the training corpus of
{$(\mathtt{m_{ut}},\mathtt{s},\mathtt{d},\mathtt{r},\mathtt{g},\hat{\mathtt{T}}_{\mathtt{neb}},\mathtt{t}_{\mathtt{eb}})$}

4:procedure CollectTrainingCorpus($\mathtt{T}_{\mathtt{eb}}$,
$\mathtt{T}_{\mathtt{neb}}$, $\mathtt{SUT}$)

5: $\mathcal{C}$ $\leftarrow$ $\emptyset$

6: for $\mathtt{t}_{\mathtt{eb}}$ $\in$ $\mathtt{T}_{\mathtt{eb}}$ do

7: $\mathtt{d}$ $\leftarrow$ GetFile($\mathtt{t}_{\mathtt{eb}}$)

8: $\mathtt{r}$ $\leftarrow$
Execute(InstrumentPrintException($\mathtt{t}_{\mathtt{eb}}$))

9: $\triangleright$ instrumenting and executing EBT to get stack trace

10: $\mathtt{r}$ $\leftarrow$ ExcludeTestAndUtilMethods($\mathtt{r}$,
$\mathtt{d}$)

11: $\mathtt{m_{ut}}$ $\leftarrow$ $\mathtt{r}$[0].method $\triangleright$ MUT
comes first in stack trace

12: $\mathtt{s}$ $\leftarrow$ GetSourceCode($\mathtt{r}$[-1])

13: $\triangleright$ last stack trace item points to target throw statement

14: $\mathtt{g}$ $\leftarrow$ ComputeGuardExp($\mathtt{r}$)

15: $\hat{\mathtt{T}}_{\mathtt{neb}}$ $\leftarrow$ $\emptyset$
$\triangleright$ initialize set of relevant non-EBTs

16: $\mathcal{C}$ $\leftarrow$ $\mathcal{C}$ $\cup$ {($\mathtt{m_{ut}}$,
$\mathtt{s}$, $\mathtt{d}$, $\mathtt{r}$, $\mathtt{g}$,
$\hat{\mathtt{T}}_{\mathtt{neb}}$, $\mathtt{t}_{\mathtt{eb}}$)}

17: $\mathtt{SUT}$ $\leftarrow$ InstrumentPrintMethod($\mathtt{SUT}$)

18: $\triangleright$ instrumentation for getting methods covered by non-EBTs

19: for $\mathtt{t}_{\mathtt{neb}}$ $\in$ $\mathtt{T}_{\mathtt{neb}}$ do

20: $\mathtt{m_{ut}}^{\prime}$ $\leftarrow$
Execute($\mathtt{t}_{\mathtt{neb}}$)[0]

21: $\triangleright$ get MUT directly invoked by non-EBT

22: for $\mathtt{c}$ $\in$ $\mathcal{C}$ do

23: if $\mathtt{c}$.$\mathtt{m_{ut}}$ == $\mathtt{m_{ut}}^{\prime}\lor$
$\mathtt{c}$.$\mathtt{d}$ == GetFile($\mathtt{t}_{\mathtt{neb}}$) then

24:
$\mathtt{c}.\hat{\mathtt{T}}_{\mathtt{neb}}\leftarrow\mathtt{c}.\hat{\mathtt{T}}_{\mathtt{neb}}\cup\\{\mathtt{t}_{\mathtt{neb}}\\}$

25: return $\mathcal{C}$

To instruction-tune exLóng’s foundation LLM, we collect a corpus
($\mathcal{C}$) of instruction data
$\mathtt{c}=(\mathtt{p},\mathtt{t}_{\mathtt{eb}})=(\mathtt{m_{ut}},\mathtt{s},\mathtt{d},\mathtt{r},\mathtt{g},\hat{\mathtt{T}}_{\mathtt{neb}},\mathtt{t}_{\mathtt{eb}})$
from a set of repositories with developer-written EBTs and non-EBTs. Algorithm
1 shows the process of collecting the training corpus for exLóng.

####  III-A1 Identifying EBTs and non-EBTs

For each repository in the training set, we first parse the source code in the
repository to identify test methods and categorize them into EBTs and non-
EBTs. A test method is categorized as an EBT if it conforms to one of the four
patterns that are widely used by developers [26]: Test(expected), assertThrow,
expectedRule, tryFailCatch. All the other test methods are categorized as non-
EBTs. The set of EBTs and non-EBTs are used as the inputs to the training
corpus collection algorithm (line 1 in Algorithm 1).

####  III-A2 Executing EBT and collecting stack trace

Each EBT will be expanded to one instruction data ($\mathtt{c}$) in the
training corpus ($\mathcal{C}$). Naturally, the file that contains the EBT is
the destination test file (line 7 in Algorithm 1). To avoid data leakage
problems, we remove all test methods in the test file ($\mathtt{d}$) and only
keep the test class structure and utility methods.

For training, stack trace is the sequence of method invocations from EBT (non
inclusive) that lead to the target throw statement under test (inclusive).
Line 8 in Algorithm 1 shows how to collect the stack trace: first, instrument
the EBT by adding “$\mathtt{print(exception.getCause())}$” to the code
location after the exception is thrown and caught by the EBT; then, execute
the instrumented EBT to get the printed stack trace. To avoid duplicate
information, we exclude EBT itself and any utility methods in the destination
test file from the stack trace (line 10 in Algorithm 1).

The first method invoked in the stack trace is the MUT by definition (line 11
in Algorithm 1). The last method invocation and line number in the stack trace
point to the target throw statement (line 12 in Algorithm 1).

####  III-A3 Computing the guard expression

Algorithm 2 Collect AST nodes along the stack trace.

1:inputs: $\mathtt{r}$ \- the stack trace for a target throw statement

2:outputs: $\mathtt{N}$ \- the collected nodes

3:procedure CollectNodes($\mathtt{r}$)

4: $\mathtt{N}$ $\leftarrow\emptyset$

5: for ($\mathtt{m}$, $\mathtt{lineno}$) $\in$ Reversed($\mathtt{r}$) do

6: $\mathtt{current}$ $\leftarrow$ GetSourceCode($\mathtt{m}$,
$\mathtt{lineno}$)

7: $\mathtt{parent}\leftarrow\mathtt{current}$

8: $\mathtt{N}$ $\leftarrow$ $\mathtt{N}$ $\cup$ {$\mathtt{current}$}

9: while $\mathtt{current}\neq\mathtt{m}$ do

10: if $\mathtt{parent}$ is ForStmt then

11: $\mathtt{N}$ $\leftarrow$ $\mathtt{N}$ $\cup$
{$\mathtt{parent}$.CompareExpression}

12: if $\mathtt{parent}$ is IfStmt then

13: if $\mathtt{parent}$.ThenStmt == $\mathtt{current}$ then

14: $\mathtt{N}$ $\leftarrow$ $\mathtt{N}$ $\cup$
{$\mathtt{parent}$.ConditionExpression}

15: if $\mathtt{parent}$.ElseStmt == $\mathtt{current}$ then

16: $\mathtt{N}$ $\leftarrow$ $\mathtt{N}$ $\cup$ {$\lnot$
$\mathtt{parent}$.ConditionExpression}

17: $\triangleright$ Other cases (while, switch, block and assignment
statements) are in supplementary material.

18: $\mathtt{current}=\mathtt{parent}$

19: $\mathtt{parent}=\mathtt{parent}$.getParent()

20: return $\mathtt{N}$

Algorithm 3 Compute guard expression based on stack trace.

1:inputs: $\mathtt{r}$ \- the stack trace for a target throw statement

2:outputs: $\mathtt{g}$ \- the guard expression for the target throw statement

3:procedure ComputeGuardExp($\mathtt{r}$)

4: $\mathtt{N}$ $\leftarrow$ CollectNodes($\mathtt{r}$)

5: $\triangleright$ collect all condition (if, while, etc.) and assignment
(assign, method call) nodes along the stack trace

6: $\mathcal{E}\leftarrow\emptyset$ $\triangleright$ the set of conditions in
the guard expression

7: for $\mathtt{n}$ $\in$ $\mathtt{N}$ do

8: if $\mathtt{n}$ is ConditionalExpr then

9: $\mathcal{E}$ $\leftarrow$ $\mathcal{E}$ $\cup$ {$\mathtt{n}$}

10: if $\mathtt{n}$ is AssignStmt then

11: $\mathcal{E}$ $\leftarrow$ Merge($\mathcal{E}$, {$\mathtt{n}$.lhs
$\mapsto$ $\mathtt{n}$.rhs})

12: if $\mathtt{n}$ is MethodDeclaration then

13: $\mathtt{n}^{\prime}\leftarrow\mathtt{n}$ $\triangleright$ process with
next node (method call)

14: if $\mathtt{n}$ is MethodCallExpr then

15: $\mathtt{argmap}$ $\leftarrow\emptyset$

16: for $\mathtt{arg},\mathtt{argname}\in$ $\mathtt{n}$.getArgs(),
$\mathtt{n}^{\prime}$.getParams() do

17: $\mathtt{argmap}$ $\leftarrow$ $\mathtt{argmap}$
$\cup\\{\mathtt{argname}\mapsto\mathtt{arg}\\}$

18: $\mathcal{E}$ $\leftarrow$ Merge($\mathcal{E}$, $\mathtt{argmap}$)

19: return $\bigwedge_{\begin{subarray}{c}\end{subarray}}\mathcal{E}$

20:

21:procedure Merge($\mathcal{E}$, $\mathtt{argmap}$)

22: $\mathcal{E}$’ $\leftarrow\emptyset$

23: for $\mathtt{e}$ $\in$ $\mathcal{E}$ do

24: for $\mathtt{name}\mapsto\mathtt{expr}\in\mathtt{argmap}$ do

25: if $\mathtt{e}$.contains($\mathtt{name}$) then

26: $\mathtt{e}$ $\leftarrow$ $\mathtt{e}$.replace($\mathtt{name}$,
$\mathtt{expr}$)

27: $\mathcal{E}$’ $\leftarrow$ $\mathcal{E}$’ $\cup$ {$\mathtt{e}$}

28: return $\mathcal{E}$’

Stack trace provides the sequence of method invocations that lead to the
target throw statement, but knowing only the names of the methods is
insufficient for generating EBTs. To aid the reasoning about the setup of the
system under test, which lead to exceptional behaviors, we propose _guard
expression_ : a logical formula representing the constraints on the symbolic
variables that must be true to follow the particular code trace. Specifically,
we use conjunctions of expressions extracted from the invoked methods in the
stack trace to form the guard expression. For instance, the guard expression
for the throw statement highlighted in Figure 1(a) is present in the Figure 2
(box 3).

The first step in the computation is to collect the list of guard-related AST
nodes along the stack trace starting from the target throw statement to the
MUT, as described in Algorithm 2. We traverse each method in the stack trace
in reversed order (line 5 in Algorithm 2). Inside each method, we start from
the AST node specified by the line number in the stack trace. We always
include this node to the list of collected nodes (line 8 in Algorithm 2)
because the variable names in that statement may be used by the next step for
replacing method call arguments. Starting from the statement, we traverse the
AST by maintaining the pointer $\mathtt{current}$ that constantly moves from
the child AST node to its parent node. AST nodes entailing condition
expressions within the ‘for’ loop (lines 10 to 11 in Algorithm 2), ‘if’
statement (lines 12 to 16 in Algorithm 2), ‘while’ loop and ‘switch’ statement
will be added to the set $\mathtt{N}$. We also collect the assignment
statements, method call expressions, and method declarations. They are omitted
from Algorithm 2 due to space constraints.

The second step of computing guard expressions is to process the collected
nodes by propagating the symbolic variables through the stack trace, as
described in Algorithm 3. The nodes are visited in the order being collected,
and conditional expressions are directly added to the guard expression (line 9
in Algorithm 3). Assignment statements, method declaration, and method call
expressions will trigger a Merge operation (line 21 in Algorithm 3). The goal
of the Merge operation is to review the current guard expression and replace
the symbolic variables (e.g., variables appearing in the target throw
statement) with their corresponding values (e.g., constant values or MUT’s
arguments). Therefore, the guard expression reflects the MUT’s arguments and
public fields that are usable by EBT, rather than local variables that EBT may
not have access to. For assignment statement (line 11 in Algorithm 3), we
replace the left-hand side variable in $\mathcal{E}$ with the right hand side
expression. For method declaration and method call expression (lines 15 to 18
in Algorithm 3; these two nodes always appear in pairs in the collected
nodes), we replace the method declaration’s argument names in $\mathcal{E}$
with the actual arguments in the method call expression.

####  III-A4 Connecting EBTs to relevant non-EBTs

We add the _relevant_ non-EBTs to the prompt (lines 17 to 24 in Algorithm 1),
to encourage the LLM to reason about the procedures to set up the object under
test and the condition under which the exception will be triggered grounding
the existing non-EBTs. Additionally, we believe the non-EBTs in the same
repository will promote the consistency between the generated code and the
existing code in terms of the both format and coding conventions. Given an
MUT, we use two approaches to retrieve the relevant non-EBTs: non-EBTs that
directly invoke the same MUT and non-EBTs that are already present in the
destination test file (line 23 in Algorithm 1). If there is no relevant non-
EBT retrieved, this part of the prompt is left empty.

####  III-A5 Instruction fine-tuning

We use CodeLlama-Instruct-7B [39], an open-source foundation model designed
for code generation and instruction following, as the foundation model of
exLóng. CodeLlama is pretrained on auto-regressive and infilling objectives,
enabling tasks like code completion and document generation. We instruction
fine-tune CodeLlama on our collected training corpus of instruction data
$\mathcal{C}=\\{(\mathtt{p},\mathtt{t}_{\mathtt{eb}})\\}$, with the novel
instructions shown in Figure 2 ( 4). Given the instruction that includes the
collected context, the model is expected to produce the EBT:
$\mathtt{LLM}(\mathtt{p})=\mathtt{t}_{\mathtt{eb}}$.

We instruction fine-tune the CodeLlama model using the parameter-efficient
Low-Rank Adaptation (LoRA) technique [21]. Rather than updating the entire set
of parameters in the LLM, LoRA injects trainable low-rank matrices into each
layer of the model. This approach dramatically reduces the number of trainable
parameters and the amount of required computational resources.

###  III-B Inference

Algorithm 4 Prepare the pool stack traces from non-EBTs and assemble the
prompt for evaluating exLóng.

1:inputs: $\mathtt{T}_{\mathtt{neb}}$ \- existing non-EBTs

2: $\mathtt{SUT}$ \- the system under test

3:outputs: $\mathcal{Q}$ \- stack traces to reach target throw statements

4:procedure CollectStackTraceSet($\mathtt{T}_{\mathtt{neb}}$, $\mathtt{SUT}$)

5: $\mathtt{SUT}$ $\leftarrow$ InstrumentPrintTrace($\mathtt{SUT}$)

6: $\mathcal{Q}$ $\leftarrow\emptyset$

7: for $\mathtt{t}_{\mathtt{neb}}$ $\in$ $\mathtt{T}_{\mathtt{neb}}$ do

8: for $\mathtt{r}$ $\in$ Execute($\mathtt{t}_{\mathtt{neb}}$) do

9: $\mathtt{r}$ $\leftarrow$ ExcludeTestAndUtilMethods($\mathtt{r}$,
GetFile($\mathtt{t}_{\mathtt{neb}}$))

10: for $\mathtt{s}$ $\in$ GetThrowStmts($\mathtt{r}$) do

11: $\mathcal{Q}$ $\leftarrow$ $\mathcal{Q}$ $\cup$ ($\mathtt{r}$,
$\mathtt{t}_{\mathtt{neb}}$, $\mathtt{s}$)

12: return $\mathcal{Q}$

13:

14:global var: $\mathcal{Q}$ =
CollectStackTraceSet($\mathtt{T}_{\mathtt{neb}}$, $\mathtt{SUT}$)

15:inputs: $\mathtt{m_{ut}}$, $\mathtt{s}$, $\mathtt{d}$ \- task inputs
selected by developers or inferred $\mathtt{T}_{\mathtt{neb}}$ \- existing
non-EBTs

16:outputs: $\mathtt{p}$ \- the prompt to give to the LLM

17:procedure AssemblePrompt($\mathtt{m_{ut}}$, $\mathtt{s}$, $\mathtt{d}$)

18: $\mathtt{R}\leftarrow\emptyset$

19: $\hat{\mathtt{T}}_{\mathtt{neb}}$ $\leftarrow\emptyset$

20: for $\mathtt{q}$ $\in$ $\mathcal{Q}$ do

21: if $\mathtt{m_{ut}}$ $\in$ $\mathtt{q}$.$\mathtt{r}$ $\land$
$\mathtt{q}$.$\mathtt{s}$ == $\mathtt{s}$ then

22: $\mathtt{R}\leftarrow\mathtt{R}\ \cup$ {$\mathtt{q}$.$\mathtt{r}$}

23: if $\mathtt{m_{ut}}$ == $\mathtt{q}$.$\mathtt{r}$[0].method then

24: $\hat{\mathtt{T}}_{\mathtt{neb}}$ $\leftarrow$
$\hat{\mathtt{T}}_{\mathtt{neb}}$ $\cup$
{$\mathtt{q}$.$\mathtt{t}_{\mathtt{neb}}$}

25: if $\mathtt{R}\neq\emptyset$ then

26: $\mathtt{r}$ $\leftarrow$ RandomSelect($\mathtt{R}$)

27: $\mathtt{g}$ = ExtractGuardExp($\mathtt{r}$)

28: $\hat{\mathtt{T}}_{\mathtt{neb}}$ $\leftarrow$
$\hat{\mathtt{T}}_{\mathtt{neb}}$ $\cup$
$\\{\mathtt{t}_{\mathtt{neb}}|\text{GetFile}(\mathtt{t}_{\mathtt{neb}})\text{==}d\\}$

29: return ($\mathtt{m_{ut}}$, $\mathtt{s}$, $\mathtt{d}$, $\mathtt{r}$,
$\mathtt{g}$, $\hat{\mathtt{T}}_{\mathtt{neb}}$)

The inference workflow of exLóng is different from its training workflow in
that we cannot rely on executing EBTs to collect the context (e.g., stack
trace), as our goal is to generate those EBTs. Instead, exLóng reasons about
the context based on task inputs (MUT, target throw statement, and destination
test file) and leveraging existing non-EBTs for the system under test.
Algorithm 4 describes the key steps in preparing the inference prompt of
exLóng.

####  III-B1 Collecting non-EBTs’ stack traces to reach potential target throw
statements

We first prepare a set of stack traces, from the execution of non-EBTs, that
can reach potential target throw statements in the repository (lines 4 to 12
in Algorithm 4). This only needs to be done once per repository. Specifically,
we first instrument all methods that have throw statements to log the current
stack trace (using $\mathtt{Thread.currentThread().getStackTrace();}$) upon
invoking those methods (line 5 in Algorithm 4). Then, we execute all non-EBTs
and collect the logged stack traces (lines 7 to 11 in Algorithm 4). The
execution of each non-EBT may generate multiple stack traces, as it may cover
multiple methods with throw statements.

####  III-B2 Selecting task inputs

Next, we select the task inputs (MUT, target throw statement, and destination
test file), which can be specified by the developer or inferred by heuristics
depending on the use case that exLóng is targeting:

  * •

Developer-oriented use case: developer specify the MUT and target throw
statement to generate the EBT for, and the destination test file where the EBT
should be placed.

  * •

Machine-oriented use case: given a repository, exLóng locates all throw
statements and generates one EBT for each of them. For a target throw
statement, the MUT is the method containing the throw statement, and the
destination test file is selected based on (a) file name matching, and (b)
test coverage analysis. Specifically, similar to prior work [36], given a code
file named FNM, we search for test file named FNMTest or TestFNM. If there is
no result based on file name matching, we run the existing non-EBTs to find
any existing test class that cover the MUT or the class of MUT. If there is
again no result based on test coverage analysis, exLóng will not generate an
EBT for this target throw statement (and move to the next one).

The selected task inputs are then used to assemble the prompt for exLóng (line
15 in Algorithm 4).

####  III-B3 Assembling the prompt

We first need to find stack traces from the set of non-EBTs’ stack traces that
match the given MUT and target throw statement (line 21 in Algorithm 4). If
multiple matching stack traces are found, we randomly select one (line 26 in
Algorithm 4). Given the stack trace, we use the same algorithm in Section
III-A3 to compute the corresponding guard expression (line 27 in Algorithm 4).
The relevant non-EBTs are selected using the similar criteria as Section
III-A4: have the same MUT (line 24 in Algorithm 4) or be in the same
destination test file (line 28 in Algorithm 4). However, if no matching stack
trace is found, exLóng will not generate an EBT for the given inputs.

##  IV Dataset

In this section, we describe details on collecting the dataset (Section IV-A),
as well as the statistics of our dataset used for training and evaluation
(Section IV-B).

TABLE I: Statistics of the collected dataset. #MUTs is the number of unique
method under test; #Exception Types is the number of unique exception types
tested by the EBTs.

| #Projects | #Tests | #EBTs | #MUTs |  #Exception Types
---|---|---|---|---|---
All | 562 | 111,230 | 12,574 | 6,250 | 821
Train | 501 | 100,030 | 11,182 | 5,508 | 725
Valid | 29 | 5,298 | 550 | 279 | 66
Eval | 32 | 5,902 | 842 | - | -

TABLE II: Statistics of the evaluation dataset for developer-oriented use case. #MUT is the number of unique method under test; #Exception Types is the number of unique exception types tested by the EBTs; #Throw Statements is the number of unique throw statements covered by the EBTs. | #EBTs | #MUT |  #Exception Types |  #Throw Statements
---|---|---|---|---
Developer-Oriented | 434 | 267 | 41 | 278
TABLE III: Statistics of the evaluation dataset for machine-oriented use case. #Throw Statements is the number of target throw statements we extracted from the repository according to Section III-B2. #Exception Types is the number of unique exception types thrown by the target throw statements.  |  #Throw Statements |  #Exception Types
---|---|---
Machine-Oriented | 649 | 81

###  IV-A Dataset Collection

Following prior work [29], we collect data from Java projects from
CodeSearchNet [22], which are available on GitHub and satisfy the following:
(1) use the Maven build system; (2) compile successfully; (3) do not have test
failures; (4) have at least one EBT that follows one of the four patterns [26]
(Section III-A1), and (5) have a license that permits the use of its data.
Requirements 1-4 simplify the automation steps and ensure that we can run
existing tests to collect dynamic data (e.g., stack traces), as well as run
EBTs that we generate.

###  IV-B Dataset Statistics

The statistics for the collected dataset are presented in Table I. In total,
we collected 111,230 tests from 562 projects, where 12,574 of these tests are
EBTs. Collected EBTs cover a range of 821 unique exception types (e.g.,
$\mathtt{RuntimeException}$, $\mathtt{IllegalArgumentException}$).

The dataset is randomly split by projects into training (Train), validation
(Valid), and evaluation (Eval) sets, where the training set is used to
instruction fine-tune exLóng, the validation set is used for early-stopping
the training process and guiding our design decision of exLóng, and the
evaluation set is used for evaluating the performance of exLóng and baselines.

Table II presents the statistics of the evaluation data for developer-oriented
use case. Note that this is a subset of the last row from Table I. Under
developer-oriented use case, we benchmark exLóng on the subset of 434 data
points for which we are able to extract stack traces.

Table III presents the statistics of the evaluation data for machine-oriented
use case. Note that this is a subset of the last row from Table I. For
machine-oriented use case, we evaluate on 649 data points as we filter the
data for which we were not able to locate the destination test file with our
designed heuristics (Section III-B2).

##  V Evaluation Design

We assess the performance of exLóng by answering the following research
questions:

RQ1: How does exLóng perform under the developer-oriented use case compared
with the state-of-the-art models?

RQ2: How much do stack traces and guard expressions help exLóng in generating
EBTs?

RQ3: How much does the selection of non-EBTs help exLóng in generating EBTs?

RQ4: How does the size of the underlying LLM model impact our findings?

RQ5: How does exLóng perform under the machine-oriented use case compared with
analysis-based test generation tools?

We next describe metrics used to compare models and tools (Section V-A) and
then describe the baselines used in our comparison (Section V-B) and hardware
used for experiments (Section V-C). We answer all research questions in
Section VI.

###  V-A Evaluation Metrics

####  V-A1 Developer-Oriented Use Case

For developer-oriented use case, we compare (using data shown in Table II) the
generated EBTs against the developer-written EBTs by benchmarking on
similarity-based and functional-correctness metrics.

Following prior work on learning-based test generation [52, 36, 29], we use
the following similarity-based metrics to compare generated EBTs and ground-
truth (i.e., developer-written ones):

Exact-match accuracy (xMatch): the percentage of the predictions that are
exactly the same as the ground-truth.

BLEU [34]: the number of n-grams in the prediction that also appear in the
ground-truth.

CodeBLEU [37]: adapted version of BLEU score for code. In addition to n-grams
overlapping, it also computes the overlap of AST nodes, nodes in the data-flow
graph between the prediction and ground-truth.

Edit similarity [60, 45]: calculates 1-Levenshtein distance which is the
minimum number of character-level edits (insertions, deletions, or
substitutions) required to change the prediction into the ground-truth.

The similarity metrics only capture the surface-level similarity between the
prediction against an existing EBT. However, such surface metrics do not
adequately capture the functional validity of the generated EBT (e.g., whether
the code can be compiled or executed), especially since the developer-written
EBTs may not be the only correct implementation to cover a specific target
throw statement. Thus, we additionally include the following functional-
correctness metrics:

Compilable%: percentage of the generated EBTs that can be compiled.

Matched-E%: percentage of EBTs that check the specified exception type.
Namely, whether the exception class following ‘$\mathtt{@Test(expected=}$’ is
the same as user specified one.

Runnable%: percentage of EBTs that check the specified exception type, and can
be compiled and executed without any error.

ThrowCov%: out of all developer-specified target throw statements (Table II),
the percentage of target throw statements with successfully generated EBTs,
i.e., compilable, runnable, and checking the specified exception type.

####  V-A2 Machine-Oriented Use Case

For machine-oriented use case, we benchmark tools ability to cover the throw
statements within a given repository:

ThrowCov%: out of all target throw statements selected in repositories (Table
III), the percentage of the target throw statements with successfully
generated EBTs, i.e., compilable, runnable, and checking the correct exception
type.

###  V-B Baselines

####  V-B1 Learning-based tools

We compare exLóng with one of the strongest foundation models and one LLM that
is specifically pretrained to generate tests.

GPT3.5: We instruct GPT3.5 [5, 31] to write EBTs by first providing one random
example from the training data. Namely, one prompt and the corresponding
ground-truth EBT. The prompt we use to query GPT3.5 includes the MUT, the
target exception type to test, the method containing the target throw
statement, one relevant non-EBT, and the destination test file. We sample a
single EBT from the output.

CAT-LM: CAT-LM [36] is a GPT-style LLM that is pretrained on Java and Python
repositories. It is pretrained with a novel objective that considers the
mapping between source code and the corresponding test files. CAT-LM has shown
strong performance on several test generation tasks. To be consistent with its
pretraining objective, we prompt CAT-LM with the MUT followed by the
destination test file, one randomly-selected relevant non-EBT, and the test
annotation (‘$\mathtt{@Test(expected=}$’), encouraging the model to complete
the EBT. Just like in other cases, we sample a single EBT.

####  V-B2 Automatic test generation tools

In machine-oriented use case, we compare exLóng with two widely-used analysis-
based test generation tools.

Randoop: Randoop [32, 38] is a random test generation tool that creates tests
by randomly generating inputs and recording the sequences of method calls. We
run Randoop with a time limit of 100 seconds per class to generate unit tests
for each project (per the Randoop user manual [50]), we set $\mathtt{seed}$ to
42, $\mathtt{usethreads}$ to true, and other options to their default values.

EvoSuite: EvoSuite [12] is a search based test generation tool that randomly
generates inputs and employs a genetic algorithm to evolve these inputs,
aiming to maximize code coverage. We run EvoSuite for 120 seconds per class
(as suggested in a recent SBST competition [43]). We also set the
$\mathtt{seed}$ to 42. Unlike Randoop, which generates tests for the entire
project, to generate more EBTs for the target throw statements within the time
limit, we generate tests on a subset of classes when running EvoSuite.
Starting from classes that contain throw statements, we use jdeps [49] to
retrieve all classes that transitively depend on these initial classes,
thereby creating a targeted subset for evaluation.

###  V-C Hardware

We fine-tune exLóng on 4 Nvidia A100 GPUs, 2 AMD Milan 7413 @ 2.65 GHz. We run
Randoop and EvoSuite on a machine with Intel Core i7-11700K @ 3.60GHz (8
cores, 16 threads) CPU, 64 GB RAM, Ubuntu 20.04, Java 8, and Maven 3.8.6. All
random seeds are set to 42.

##  VI Results

In this section, we present the evaluation results and answer each research
question.

TABLE IV: Results on developer-oriented use case with ground-truth EBT’s name in the prompt. Models | BLEU | CodeBLEU | EditSim | XMatch | Compilable% | Matched-E% | Runnable% | ThrowCov%
---|---|---|---|---|---|---|---|---
GPT3.5-few-shot | 56.61 | 64.28 | 82.30 | 14.98 | 75.12 | 100.00 | 61.29 | 48.39
CAT-LM | 53.96 | 60.41 | 80.00 | 10.14 | 71.20 | 100.00 | 40.32 | 32.26
exLóng | 64.17 | 68.60 | 85.78 | 19.82 | 82.49 | 100.00 | 67.97 | 58.53
exLóng-sample | 69.60 | 73.40 | 88.77 | 18.89 | 87.10 | 100.00 | 75.12 | 66.82
TABLE V: Results on developer-oriented use case without ground-truth EBT’s name in the prompt. Models | BLEU | CodeBLEU | EditSim | XMatch | Compilable% | Matched-E% | Runnable% | ThrowCov%
---|---|---|---|---|---|---|---|---
GPT3.5-few-shot | 38.67 | 49.33 | 69.32 | 0.00 | 82.83 | 100.00 | 57.58 | 32.32
CAT-LM | 38.00 | 49.11 | 69.55 | 1.38 | 70.05 | 100.00 | 25.35 | 17.05
exLóng | 47.84 | 55.89 | 80.29 | 2.07 | 82.95 | 100.00 | 69.59 | 58.99
exLóng-sample | 51.58 | 59.76 | 83.23 | 1.84 | 89.63 | 100.00 | 79.49 | 68.66

###  VI-A RQ1: Developer-Oriented Use Case

To answer RQ1, we compare the EBTs generated by exLóng with developer-written
tests. The results of exLóng and baselines are shown in tables IV and V. Table
IV presents the results when we inform LLMs the method name of the target EBT
while in Table V we do not. (We describe the last row in these tables in a
later subsection.) Note that we add the same random-selected relevant non-EBT
to the prompt for all the models.

exLóng outperforms all the baselines on both similarity-based metrics (left
side in tables) and functional-correctness metrics (right side in tables).
exLóng achieves higher performance than baselines for both generating
executable EBTs (Runnable%) and EBTs that cover the target throw statements
(ThrowCov%). In Table IV, we can see that exLóng outperforms GPT3.5 by 10.9%
and 20.9% on Runnable% and ThrowCov%, respectively. Similarly, we can see that
exLóng outperforms CAT-LM by 68.6% and 57.53% on Runnable% and ThrowCov%,
respectively. This highlights the benefit of the stack traces and guard
expressions extracted via program analysis, and exLóng’s capability of
reasoning about them.

Comparing functional-correctness metrics in Table IV and Table V, performance
of both GPT3.5 and CAT-LM declines significantly when the EBT method name is
omitted. This result aligns with expectations, as the method name frequently
implies the conditions under which the exception is supposed to be thrown,
e.g., “$\mathtt{should\\_fail\\_if\\_time\\_provider\\_is\\_null}$”. In
contrast, exLóng demonstrates robustness regarding the inclusion or exclusion
of the test method name, maintaining consistent performance on functional-
correctness metrics. This further emphasizes the reasoning ability of exLóng
on the stack traces and guard expressions.

###  VI-B RQ2: Ablation Study of Stack Traces and Conditions

To evaluate the contribution of the components in exLóng, we perform an
ablation study. In Table VI, we show the results while including EBT’s name in
the prompt. We find that ablating each component deteriorates performance
especially across functional correctness metrics. Removing stack traces
slightly hurts the performance of exLóng which is expected, because a guard
expression is, in a way, the summary of a stack trace.

We observe a small drop in Compile% but a rather larger drop in ThrowCov% if
removing both stack trace and guard expression from the context. This
underlines the importance of the condition information present by these two
components on which exLóng reasons about when generating EBTs that cover the
target throw statements. Relevant non-EBTs mostly contribute to the Compile%
because the relevant non-EBTs covers the same MUT which gives model a starting
point to construct the EBTs.

###  VI-C RQ3: Selection of non-EBTs

In addition to randomly choosing one relevant non-EBT to add to the prompt for
exLóng, we try prompting exLóng with no more than 5 different relevant non-
EBTs (one prompt for each non-EBT) and reporting the best performance, dubbed
exLóng-sample in tables IV and V. We see that that performance differences can
be substantial.

TABLE VI: Ablations on different context of exLóng. Models | BLEU | CodeBLEU | EditSim | XMatch | Compilable% | Matched-E% | Runnable% | ThrowCov%
---|---|---|---|---|---|---|---|---
exLóng | 64.17 | 68.60 | 85.78 | 19.82 | 82.49 | 100.00 | 67.97 | 58.53
No stack trace | 64.38 | 69.16 | 86.12 | 18.20 | 82.49 | 100.00 | 66.82 | 56.91
No stack trace, guard expression | 63.61 | 68.08 | 85.64 | 18.43 | 80.41 | 100.00 | 60.60 | 49.31
No stack trace, guard expression, non-EBT | 57.17 | 61.79 | 82.30 | 13.59 | 62.21 | 100.00 | 47.93 | 37.79
TABLE VII: Comparison between using different non-EBTs to prompt and using one non-EBT but sampling multiple times. Note that we add the target EBT’s name to the prompt and we only report results on examples that have more than one candidate non-EBT. Models | BLEU | CodeBLEU | EditSim | XMatch | Compilable% | Matched-E% | Runnable% | ThrowCov%
---|---|---|---|---|---|---|---|---
exLóng | 62.81 | 67.29 | 85.87 | 15.02 | 85.77 | 100.00 | 66.40 | 55.34
exLóng sample w. one non-EBT | 63.83 | 69.13 | 86.86 | 14.62 | 87.75 | 100.00 | 73.12 | 62.06
exLóng sample w. different non-EBT | 72.26 | 75.60 | 90.87 | 13.83 | 94.07 | 100.00 | 81.42 | 71.54
TABLE VIII: Results for exLóng-base 13B and exLóng 13B models; exLóng-base 13B does not have access to non-EBTs, stack trace and guard expression. Models | BLEU | CodeBLEU | EditSim | XMatch | Compilable% | Matched-E% | Runnable% | ThrowCov%
---|---|---|---|---|---|---|---|---
exLóng-base 13B | 57.89 | 63.40 | 82.52 | 13.59 | 65.90 | 100.00 | 53.46 | 43.09
exLóng 13B | 64.44 | 68.77 | 85.93 | 19.82 | 80.65 | 99.43 | 69.59 | 62.21

To study how the diversity of non-EBTs used in the prompt affect the exLóng
performance, our ablation study on the number of different relevant non-EBTs
is shown in Table VII. We present results of 1) exLóng only sampling one EBT
for each target throw statement; 2) exLóng with the same non-EBT as 1) but
sample more than one EBT for each target throw statement; 3) exLóng sampling
one time for at most five different non-EBTs (exLóng-sample). For a fair
comparison, we keep the number of sampled EBTs for 2) and 3) same and we only
report metrics on a subset of data on which we extracted more than one
relevant non-EBT.

We find that increasing the number of sampled EBTs improves the performance
when compared to only sampling one EBT, which can be attributed to more
randomness when the sampling size is small. However, sampling with a diverse
non-EBTs further boost the performance across most of the metrics under the
same sample size. This motivates a new research question on how to choose
better non-EBTs to the prompt for EBT generation, which we leave for future
work.

###  VI-D RQ4: Transferring Findings to Different LLM Size

To explore if our design and contributions are transferable to a model of a
different size, we show the results for a variant of exLóng that is built on
CodeLlama-13B. We introduce one ablation model called exLóng-base where the
stack trace, guard expression, and non-EBT are excluded from model’s input.
Table VIII shows the performance of exLóng-13B-base and exLóng-13B, and we
find that the benefits brought by our proposed context are consistent across
different model sizes, i.e., 7B and 13B. This shows that combining program
analysis technique with LLMs is promising and necessary to further boost the
LLMs when generating EBTs regardless of the number of parameters in a model.

###  VI-E RQ5: Machine-Oriented Use Case

TABLE IX: Throw Statements Coverage Rate for exLóng, Randoop and EvoSuite. Tools | ThrowCov%
---|---
Subset Projects | All Projects
exLóng | 29.72 | 28.81
EvoSuite | 20.37 | 20.95
Randoop | 21.87 | 18.03

((a)) Covered throw statements on a subset of 27 projects.

((b)) Covered throw statements on all 30 projects.

Figure 3: Venn diagram that shows target throw statements coverage by exLóng,
Randoop, and EvoSuite.

In Table IX, we present the throw statement coverage rate of exLóng-sample and
two analysis-based test generation tools. EvoSuite could generate tests for
all projects, while Randoop could not generate tests for three projects. We
inspected the issues and found that: (1) Randoop crashed on OpenNMS/newts
because this project kept throwing runtime exceptions (all related to
com.codahale.metrics.ScheduledReporter); (2) Randoop crashed on
pinterest/secor because this project requires the configuration of Kafka; (3)
Randoop could not load a class and crashed on OpenHFT/Chronicle-Map.

We report results on the subset of 27 projects where all the tools can be run
successfully (Subset Projects) and results on all 30 projects (All Projects).
Among the given target throw statements, exLóng achieves higher throw
statement coverage rate than analysis-based tools. Figure 3 illustrates the
overlap and difference among the sets of target throw statements covered by
exLóng, EvoSuite, and Randoop. All three tools cover different sets of throw
statements. exLóng covers the most target throw statements that other two
cannot. In this experiment, for exLóng, we sample only a single EBT per given
prompt, and we expect that our results would be even better if with a larger
sample size.

##  VII Case Study

We submitted a subset of EBTs generated by exLóng to open-source repositories.
We already submitted pull requests (PRs) to 9 repositories on GitHub, and 23
generated EBTs have been accepted by developers. In one instance, a developer
responded and merged our PR only 30 minutes after we create the PR. This was
encouraging, and future tool development should integrate exLóng into an IDE,
such that exLóng continuously provide EBTs for code that a developer is
editing. We plan to submit PRs to other projects in the coming weeks.

##  VIII Limitations

We discuss several limitations and potential future work.

Programming language. In this work, we focused on supporting the Java
programming language, which is among the most popular languages nowadays. We
expect no substantial differences in our approach for similar programming
languages, e.g., C#. Future work could evaluate and tune our model for
dynamically typed languages, e.g., Python.

Project boundaries. In our evaluation, we generate EBTs for throw statements
within a single project, and we ignore throw statements that are in libraries
used by the project. We could not come up with a use case that targets throw
statements in libraries, so we left it out of our work. If we were to target
such a case, we would need to collect context (throw statement and conditions)
from those libraries. One could take several directions, e.g., finding code of
those libraries, building a model on bytecode level, or decompiling code and
then extracting the context.

Not every MUT has a destination test file. We leave the problem on finding or
generating destination test file for any given method under test as future
work.

LLMs. We built exLóng around CodeLlama [39], a recent open-source model. We
believe that building on CodeLlama provides reproducibility guarantees that
will help us and others to build on this work. Our contributions are the task,
definition of a context for the task, tools for extracting the context,
instruction fine-tuned model, and extensive evaluation. We expect that
building exLóng on LLMs with similar architecture as CodeLlama would lead to
similar results.

##  IX Related Work

There has been significant work on test generation [36, 32, 12, 11, 55, 44].
We cover related work on (1) LLM-based test generation, (2) generating tests
for exceptional behavior, and (3) other test generation techniques.

LLM-based test generation. Transformer models have been used to generate tests
[55, 10, 42, 52, 36, 29, 28, 23, 59] and test oracles [57, 9, 53]. CAT-LM [36]
is a 2.7B model that is pretrained on a large dataset of Java and Python
projects. It outperforms existing test generation tools StarCoder [24] and
CodeGen 16B [30] in terms of the number of valid tests and test completion
tool TeCo [29, 28]. So we compared our work with CAT-LM in this paper.

Conditions are useful for guiding the generation of tests and finding bugs [4,
3, 40]. SymPrompt [40] introduced path constraint prompting to guide LLMs to
generate high-coverage tests without additional training. Their path
constraints were generated by parsing the abstract syntax tree of the target
method while our guard expressions are derived by searching backwards from the
throw statement to the top of target method that contains this throw
statement.

Existing test cases (including the setup and teardown methods) serve as a
useful context to guide the generation [36, 52, 29]. Haji et al. [10]
empirically studies the effectiveness of generating tests using GitHub Copilot
and discovers that using existing test cases as context can increase the
passing rate of generated tests by 37.73%. Our work uses existing non-EBTs as
context and collects stack traces from those tests to guide the generation of
EBTs.

Generating tests for exceptional behavior. Exception handling [62, 61, 26, 63]
is an important aspect of software development. There are several techniques
[51, 14, 2, 1] to generate EBTs. However, they either generate EBTs from
specifications or use random-based or search-based strategies to generate
EBTs. Our work is the first to use LLMs to generate EBTs. Also, they generate
tests for the whole program, while our exLóng allows users to specify which
throw statements to cover.

Guo et al. [16] introduces boundary coverage distance (BCD) to evaluate the
quality of test inputs, which can be used to guide the random generation of
test inputs by minimizing BCD. Goffi [14] proposed throw statement coverage to
measure the effectiveness of test inputs in triggering exceptions. We also use
throw statement coverage in our evaluation.

Other test generation techniques. Other techniques incorporate random-based
[32, 38], search-based [12, 18] and constraint-based [11, 20, 13] strategies
to automatically generate tests. Tests can be derived from multiple sources,
including the code under test, error messages [17], and specifications [27]
like comments [48, 46, 47].

Randoop [32, 38] generates tests by randomly generating inputs and saving the
sequences of method calls. EvoSuite [12] is a search based test generation
tool that randomly generates inputs and uses a genetic algorithm to evolve the
inputs to maximize code coverage. During the test generation process, both
Randoop and EvoSuite create tests that cover normal as well as exceptional
behaviors. However, since they generate inputs randomly, they do not guarantee
to generate tests with high coverage or meaningful and readable inputs.
Moreover, there is no assurance that these inputs will successfully trigger
certain exceptional behaviors. EvoSuiteFIT [1] adapted EvoSuite’s search
algorithm with reinforcement learning to generate exceptional tests.
Unfortunately, we cannot directly use EvoSuiteFIT (has error “Invalid or
corrupt jarfile”).

##  X Conclusion

We presented the first work on generating tests for exceptional behavior
(EBTs) using large language models. We introduced exLóng that builds on top of
CodeLlama to embed reasoning about traces that lead to throw statements,
conditional expressions along those traces, and non-exceptional tests that
cover similar traces. We evaluated exLóng in two use cases: developer-oriented
use case (i.e., generate EBT for a given method under test and a target throw
statement) and machine-oriented use case (i.e., automatically generate tests
for all throw statements available in a repository). Our results show that
exLóng outperforms existing test generation models and analysis-based test
generation tools. We contributed a number of tests generated by exLóng to
open-source projects, and 23 EBTs are already accepted. We believe that exLóng
targets an important task, has good performance, and helps developers increase
code quality assurance by automatically providing high quality EBTs.

## Acknowledgments

We thank Nader Al Awar, Yu Liu, Jayanth Srinivasa, Aditya Thimmaiah, Zijian Yi
and Zhiqiang Zang for their comments and feedback. We acknowledge the Texas
Advanced Computing Center (TACC) at The University of Texas at Austin and
Digital Research Alliance of Canada (the Alliance) for providing HPC resources
that have contributed to the research results reported within this paper. This
work is partially supported by the US National Science Foundation under Grant
Nos. CCF-2107291, CCF-2217696 and CCF-2313027. This work was in part supported
by Cisco Research. Any opinions, findings and conclusions, or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views of Cisco Research.

## References

  * [1] H. Almulla and G. Gay, “Learning how to search: Generating exception-triggering tests through adaptive fitness function selection,” in _International Conference on Software Testing, Verification, and Validation_ , 2020, pp. 63–73.
  * [2] R. D. Bernardo, R. Sales Jr., F. Castor, R. Coelho, N. Cacho, and S. Soares, “Agile testing of exceptional behavior,” in _Brazilian Symposium on Software Engineering_ , 2011, pp. 204–213.
  * [3] A. Blasi, A. Gorla, M. D. Ernst, and M. Pezzè, “Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints,” in _Automated Software Engineering_ , 2022, pp. 1–11.
  * [4] I. Bouzenia and M. Pradel, “When to say what: Learning to find condition-message inconsistencies,” in _International Conference on Software Engineering_ , 2023, pp. 868–880.
  * [5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _arXiv preprint arXiv:2005.14165_ , 2020.
  * [6] E. Daka, “Improving readability in automatic unit test generation,” Ph.D. dissertation, University of Sheffield, 2018.
  * [7] E. Daka, J. M. Rojas, and G. Fraser, “Generating unit tests with descriptive names or: Would you name your children thing1 and thing2?” in _International Symposium on Software Testing and Analysis_ , 2017, pp. 57–67.
  * [8] F. Dalton, M. Ribeiro, G. Pinto, L. Fernandes, R. Gheyi, and B. Fonseca, “Is exceptional behavior testing an exception? An empirical assessment using Java automated tests,” in _International Conference on Evaluation and Assessment in Software Engineering_ , 2020, pp. 170–179.
  * [9] E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, “TOGA: A neural method for test oracle generation,” in _International Conference on Software Engineering_ , 2022, pp. 2130–2141.
  * [10] K. El Haji, C. Brandt, and A. Zaidman, “Using github copilot for test generation in python: An empirical study,” _International Workshop on Automation of Software Test_ , 2024.
  * [11] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, “The daikon system for dynamic detection of likely invariants,” _Science of computer programming_ , vol. 69, no. 1-3, pp. 35–45, 2007.
  * [12] G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in _International Symposium on the Foundations of Software Engineering_ , 2011, pp. 416–419.
  * [13] P. Godefroid, “Test generation using symbolic execution,” in _Foundations of Software Technology and Theoretical Computer Science_ , 2012\.
  * [14] A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè, “Automatic generation of oracles for exceptional behaviors,” in _International Symposium on Software Testing and Analysis_ , 2016, pp. 213–224.
  * [15] J. Gosling, _The Java language specification_. Addison-Wesley Professional, 2000.
  * [16] X. Guo, H. Okamura, and T. Dohi, “Optimal test case generation for boundary value analysis,” _Software Quality Journal_ , pp. 1–24, 2024.
  * [17] X. Han, T. Yu, and D. Lo, “Perflearner: Learning from bug reports to understand and generate performance test frames,” in _Automated Software Engineering_ , 2018, pp. 17–28.
  * [18] M. Harman and P. McMinn, “A theoretical and empirical study of search-based testing: Local, global, and hybrid search,” _Transactions on Software Engineering_ , vol. 36, no. 2, pp. 226–247, 2009.
  * [19] A. Hejlsberg, S. Wiltamuth, and P. Golde, _C# language specification_. Addison-Wesley Longman Publishing Co., Inc., 2003.
  * [20] J. Holmes, I. Ahmed, C. Brindescu, R. Gopinath, H. Zhang, and A. Groce, “Using relative lines of code to guide automated test generation for Python,” _Transactions on Software Engineering and Methodology_ , vol. 29, no. 4, pp. 1–38, 2020.
  * [21] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2022.
  * [22] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” _arXiv preprint arXiv:1909.09436_ , 2019.
  * [23] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in _International Conference on Software Engineering_ , 2023, pp. 919–931.
  * [24] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim _et al._ , “Starcoder: May the source be with you!” _arXiv preprint arXiv:2305.06161_ , 2023.
  * [25] L. P. Lima, L. S. Rocha, C. I. M. Bezerra, and M. Paixao, “Assessing exception handling testing practices in open-source libraries,” _Empirical Software Engineering_ , vol. 26, no. 5, 2021.
  * [26] D. Marcilio and C. A. Furia, “How Java programmers test exceptional behavior,” in _International Working Conference on Mining Software Repositories_ , 2021, pp. 207–218.
  * [27] M. Motwani and Y. Brun, “Automatically generating precise oracles from structured natural language specifications,” in _International Conference on Software Engineering_ , 2019, pp. 188–199.
  * [28] P. Nie, “Machine learning for executable code in software testing and verification,” Ph.D. dissertation, The University of Texas at Austin, 2023.
  * [29] P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Gligoric, “Learning deep semantics for test completion,” in _International Conference on Software Engineering_ , 2023, pp. 2111–2123.
  * [30] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “CodeGen: An open large language model for code with multi-turn program synthesis,” _arXiv preprint arXiv:2203.13474_ , vol. 30, 2022.
  * [31] OpenAI, “Gpt-3.5-turbo,” https://platform.openai.com/docs/models/gpt-3-5-turbo, 2024.
  * [32] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in _International Conference on Software Engineering_ , 2007, pp. 75–84.
  * [33] A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn, “Test smells 20 years later: detectability, validity, and reliability,” _Empirical Software Engineering_ , vol. 27, no. 7, p. 170, 2022.
  * [34] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in _Annual Meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [35] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [36] N. Rao, K. Jain, U. Alon, C. L. Goues, and V. J. Hellendoorn, “CAT-LM training language models on aligned code and tests,” in _Automated Software Engineering_ , 2023, pp. 409–420.
  * [37] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “CodeBLEU: A method for automatic evaluation of code synthesis,” _arXiv preprint arXiv:2009.10297_ , 2020.
  * [38] B. Robinson, M. D. Ernst, J. H. Perkins, V. Augustine, and N. Li, “Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs,” in _Automated Software Engineering_ , 2011, pp. 23–32.
  * [39] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin _et al._ , “Code Llama: Open foundation models for code,” _arXiv preprint arXiv:2308.12950_ , 2023.
  * [40] G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage guided test generation in regression setting using llm,” _arXiv preprint arXiv:2402.00097_ , 2024.
  * [41] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja _et al._ , “Multitask prompted training enables zero-shot task generalization,” _arXiv preprint arXiv:2110.08207_ , 2021.
  * [42] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” _Transactions on Software Engineering_ , 2023.
  * [43] S. Schweikl, G. Fraser, and A. Arcuri, “EvoSuite at the sbst 2022 tool competition,” in _SBST_ , 2022, pp. 33–34.
  * [44] M. L. Siddiq, J. C. Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V. C. Lopes, “Using large language models to generate junit tests: An empirical study,” _arXiv_ , 2024.
  * [45] A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in _International Symposium on the Foundations of Software Engineering_ , 2020, pp. 1433–1443.
  * [46] L. Tan, D. Yuan, G. Krishna, and Y. Zhou, “/* icomment: Bugs or bad comments?*,” in _Symposium on Operating Systems Principles_ , 2007, pp. 145–158.
  * [47] L. Tan, Y. Zhou, and Y. Padioleau, “acomment: Mining annotations from comments and code to detect interrupt related concurrency bugs,” in _International Conference on Software Engineering_ , 2011, pp. 11–20.
  * [48] S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens, “@tComment: Testing Javadoc comments to detect comment-code inconsistencies,” in _International Conference on Software Testing, Verification, and Validation_ , 2012, pp. 260–269.
  * [49] J. Team, “Jdeps manual,” https://docs.oracle.com/en/java/javase/11/tools/jdeps.html#GUID-A543FEBE-908A-49BF-996C-39499367ADB4, 2024\.
  * [50] R. Team, “Randoop manual,” https://randoop.github.io/randoop/manual/, 2024\.
  * [51] N. Tracey, J. Clark, K. Mander, and J. McDermid, “Automated test-data generation for exception conditions,” _Software: Practice and Experience_ , vol. 30, no. 1, pp. 61–79, 2000.
  * [52] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” _arXiv preprint arXiv:2009.05617_ , 2020.
  * [53] M. Tufano, D. Drain, A. Svyatkovskiy, and N. Sundaresan, “Generating accurate assert statements for unit test cases using pretrained transformers,” in _International Workshop on Automation of Software Test_ , 2022, pp. 54–64.
  * [54] G. VanRossum and F. L. Drake, _The python language reference_. Python Software Foundation Amsterdam, The Netherlands, 2010, vol. 561.
  * [55] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” _Transactions on Software Engineering_ , 2024.
  * [56] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-Instruct: Aligning language models with self-generated instructions,” in _Annual Meeting of the Association for Computational Linguistics_ , 2023.
  * [57] C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” in _International Conference on Software Engineering_ , 2020, pp. 1398–1409.
  * [58] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [59] Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” _arXiv preprint arXiv:2305.04207_ , 2023.
  * [60] L. Yujian and L. Bo, “A normalized levenshtein distance metric,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 29, no. 6, pp. 1091–1095, 2007.
  * [61] H. Zhang, J. Luo, M. Hu, J. Yan, J. Zhang, and Z. Qiu, “Detecting exception handling bugs in C++ programs,” in _International Conference on Software Engineering_ , 2023, pp. 1084–1095.
  * [62] J. Zhang, X. Wang, H. Zhang, H. Sun, Y. Pu, and X. Liu, “Learning to handle exceptions,” in _Automated Software Engineering_ , 2021, pp. 29–41.
  * [63] H. Zhong, “Which exception shall we throw?” in _Automated Software Engineering_ , 2022, pp. 1–12.