How Effective Are Neural Networks for Fixing Security Vulnerabilities
Security vulnerability repair is a difficult task that is in dire need of
automation. Two groups of techniques have shown promise: (1) large code
language models (LLMs) that have been pre-trained on source code for tasks such
as code completion, and (2) automated program repair (APR) techniques that use
deep learning (DL) models to automatically fix software bugs.
  This paper is the first to study and compare Java vulnerability repair
capabilities of LLMs and DL-based APR models. The contributions include that we
(1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder),
four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java
vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations
to address the training and test data overlapping threat to Codex, (3) create a
new Java vulnerability repair benchmark VJBench, and its transformed version
VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed
vulnerabilities in VJBench-trans.
  Our findings include that (1) existing LLMs and APR models fix very few Java
vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities.
(2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing
capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix
many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing
cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes
8.3 transformed vulnerabilities, outperforming all the other LLMs and APR
models on transformed vulnerabilities. The results call for innovations to
enhance automated Java vulnerability repair such as creating larger
vulnerability repair training data, tuning LLMs with such data, and applying
code simplification transformation to facilitate vulnerability repair.
Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, Sameena Shah
# How Effective Are Neural Networks for Fixing Security Vulnerabilities

###### Abstract.

Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs.

This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DI-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans, to better evaluate LLMs and APR techniques, and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans.

Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. Many of the generated patches are unexplable patches. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

Automated Program Repair, Large Language Model, Vulnerability, AI and Software Engineering +
Footnote †: This work is done when Hung Viet Pham and Thibaud Lutellier were at University of Waterloo.
fix was released more than one month later. As a result, there is a need for faster vulnerability-fixing solutions.

Most vulnerability benchmarks and vulnerability repair solutions focus either on C/C++ (Kumar et al., 2017; Kumar et al., 2018; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019) or binaries (Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019). There is a lack of solutions and benchmarks for Java, despite it being a widely-used programming language (the third most popular language in the open-source community (Kumar et al., 2019)) with _many seven vulnerabilities_.

Java has been used to implement important servers, including web servers and services (e.g., Tomcat, Spring, CFX, Log4), which are especially vulnerable to attackers. Consequently, many of the most critical vulnerabilities are in Java software. For example, Google assessed that the Log4Shell vulnerability in the Log4] package affected 17,000 Maven projects (Maven, 2017), and Microsoft even reported that nation-state attackers exploited the vulnerability (Kumar et al., 2019).

Benchmarks and solutions for other programming languages often do not work or work poorly for fixing Java vulnerabilities. For example, the most common vulnerabilities in C/C++ are buffer overflows (Kumar et al., 2019; Kumar et al., 2019). Java, as a type-safe language, is designed to avoid buffer overflow. Thus, most C/C++ techniques focusing on buffer overflow vulnerabilities are irrelevant to Java. We need new benchmarks and techniques for fixing Java security vulnerabilities.

Instead of building a technique to fix Java vulnerabilities automatically, we study and compare the space and feasibility of applying two types of techniques--learning-based automated program repair and LLMs--to fix Java security vulnerabilities automatically. First, learning-based program repair has gained popularity (Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019). These encoder-decoder approaches learn from a large number of pairs of bugs and their fixes (in open-source projects) to fix unseen Java software bugs automatically. _It would be interesting to study how effective such learning-based program repair models are in fixing a subset of software bugs, i.e., software vulnerabilities_.

Secondly, LLMs have recently been applied to source code (Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019) and are pre-trained models that have been trained on a tremendous amount of source code (e.g., the entirety of GitHub). Different from APR models, pre-trained LLMs learn from large corpus of source code (instead of pairs of bugs and their fixes) for various tasks such as identifier tagging and code completion. Despite learning to perform tasks different from repairing, recent study (Kumar et al., 2019; Kumar et al., 2019) shows that pre-trained LLMs have competitive capabilities of fixing general Java bugs (Kumar et al., 2019; Kumar et al., 2019). _It would be interesting to study how effective such LLMs are for a different task, i.e., fixing software vulnerabilities, when they do not see how bugs are fixed._

Thirdly, it would be interesting to compare deep learning (DL)-based APR techniques' and LLMs' capabilities of fixing Java vulnerabilities. DL-based APR techniques and LLMs represent two angles of applying models for a different task. Applying DL-based APR techniques to fix vulnerabilities is using models learned from a general dataset for a specific subset of the dataset (software vulnerability is a type of software bugs). Applying LLMs to fix vulnerabilities is using models learned from a different format of dataset (sequences of code) for another format (pairs of buggy and fixed code). Since LLMs do not require pairs of bugs and their fixes, LLMs are typically built from data that is orders of magnitude larger than the training data used to train APR models. _Would more data win or data-format matching win?_

Lastly, pre-trained LLMs are often fine-tuned to adapt to different downstream tasks (Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019; Kumar et al., 2019). A recent study (Kumar et al., 2019) shows that fine-tuning improves LLMs' fixing capabilities by at least 31%. However, given the lack of Java vulnerability data, it is unrealistic to fine-tune LLMs for fixing Java vulnerabilities. Thus, _it would be interesting to study how effective LLMs fine-tuned with general APR data are in fixing software vulnerabilities_. And when compared with DL-based APR techniques, _would more data plus fine-tuning win or data-format matching win?_

### Our Approach

We conduct the first study to evaluate and compare APR techniques' and LLMs' abilities of fixing Java vulnerabilities. We evaluate five LLMs (Codex (Codex, 2017), CodeT5 (Kumar et al., 2019), CodeGen (Kumar et al., 2019), PLIBART (Maven, 2017) and InCder (Kumar et al., 2019)), four LLMs that are fined-tuned with general APR data, and four APR techniques (CURE (Kumar et al., 2019), Recoder (Kumar et al., 2019), RewardRepair (Kumar et al., 2019), and KNOD (Kumar et al., 2019)) on two Java vulnerability benchmarks (Vul4J and a new VJBench that we create). There are two main challenges.

First, there are few benchmarks available for evaluating Java vulnerability repair tools. While Vul4J (Kumar et al., 2019) contains 79 reproducible Java vulnerabilities, they belong to only 25 CWEs, i.e., types of vulnerabilities. In addition, 60% of the CWEs in the dataset (15 types of vulnerabilities) are covered by only a single reproducible vulnerability.

To address this challenge, we develop new benchmarks. We analyze the entire National Vulnerability Database (NVD) (Bartos et al., 2017) to identify reproducible real-world Java vulnerabilities that are suitable for vulnerability repair evaluation, and use these to create our VJBench benchmark. These vulnerabilities cover an additional twelve CWE types not included by the Vul4J dataset and add more vulnerabilities to four CWE types with which Vul4J has only one vulnerability associated. The new benchmark can facilitate the evaluation of future Java vulnerability repair techniques.

The second challenge arises from the fact that Codex was trained on a substantial code corpus collected from GitHub (Kumar et al., 2019) and the training dataset is unreleased. Since the projects in Vul4J and VJBench are public repositories on GitHub, one cannot be certain that the vulnerabilities in Vul4J and VJBench are not in Codex's training data. This is a major known threat to the validity of evaluation (Kumar et al., 2019; Kumar et al., 2019). While dataset HumanEval (Kumar et al., 2019) is not in Codex's training data, it is for Python code completion and does not contain Java vulnerabilities. Creating new real-world benchmarks is not only expensive (Kumar et al., 2019; Kumar et al., 2019), but might also be impracticable if LLMs have been trained on all public datasets.

Our best-effort solution to mitigate this challenge is to transform the vulnerability code in existing benchmarks. We use two types of code transformation: identifier renaming and code structure change. These transformations generate new equivalent programs that still retain the vulnerabilities but are not included in any open-source dataset that Codex and other LLMs may have seen. As a result, we create VJBench-trans, a benchmark of transformed vulnerabilities, by applying two transformation strategies on vulnerabilities from Vul4J and VJBench.
### Contributions

Our paper makes the following contributions:

* We conduct the first study that evaluates the fixing capabilities of five LLMs, four fine-tuned LLMs, and four APR techniques on real-world Java vulnerabilities from two benchmarks Vul4J and our new VJBench. Our findings include:
* Existing LLMs and APR techniques fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%) vulnerabilities on average, exhibiting the best fixing capability. (Section 6.1)
* Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. Fine-tuned InCoder fixes 9 vulnerabilities, exhibiting competitive fixing capability to Codex's. (Section 6.1)
* Codex has the highest compilation rate of 79.7%. Other LLMs (fine-tuned or not) and APR techniques have low compilation rates (the lowest being 6.4% with CodeT5 and the rest between 24.5% to 65.2%), showing a lack of syntax domain knowledge. (Section 6.1)
* LLMs and APR models, except Codex, only fix vulnerabilities that require simple changes, such as a single deletion or variable/method replacement. (Section 6.2)
* Our new VJBench reveals that LLMs and APR models fail to fix many CWE types including CWE-172 Encoding error, CWE-325 Missing cryptographic step, CWE-444 HTTP request smuggling, CWE-668 Exposure of resource to wrong sphere, and CWE-1295 Debug messages revealing unnecessary information. (Section 6.2)
* We create two Java vulnerability benchmarks for automated program repair: (1) _VJBench_, which contains 42 reproducible real-world Java vulnerabilities that cover twelve new CWE types, and (2) _VJBench-trans_, which contains 150 transformed Java vulnerabilities.
* We use code transformations to mitigate the threat that LLMs and black-box Codex may have seen the evaluated benchmarks.
* We evaluate LLMs and APR techniques' fixing capabilities on transformed vulnerabilities (VJBench-trans).
* Code transformations make LLMs and APR techniques fix fewer number of vulnerabilities. Some models such as Codex and fine-tuned PLBART are more robust to code transformations. On the other hand, some transformations make the vulnerabilities easier to fix. (Section 6.3)
* We provide implications and suggestions for future directions (Section 6).

## 2. New Benchmark of Java Vulnerabilities

A Java APR benchmark must contain reproducible Java vulnerabilities with test cases exposing the vulnerabilities. While there is an abundance of such benchmarks for Java bugs, including Defects4J (Velick et al., 2018), QuixBugs (QuixBugs, 2018), Bugs.jar (Searns et al., 2018), and Bears (Bears, 2018), the only Java vulnerability benchmark for APR is Vul4J (Searns et al., 2018). Vul4J contains 79 vulnerabilities from 51 projects covering 25 CWE types. Despite a valuable first step, Vul4J offers limited coverage of CWE categories as explained in Introduction. In addition, only 35 of these vulnerabilities are applicable for evaluating state-of-the-art learning-based APR systems (Zhao et al., 2018; Wang et al., 2018; Wang et al., 2018) since these APR models only fix single-hunk bugs. Specifically, 39 of the 79 vulnerabilities are single-hunk. We can only reproduce 35 of the 39 vulnerabilities, as two bugs fail to compile, and two bugs are not reproducible with the Docker container provided by the Vul4J authors.

To extend this benchmark, we collect Java vulnerabilities following prior work (Velick et al., 2018): i) The vulnerability should only be related to Java source code, ii) The fixing commit should contain at least one test case that passes on \(V_{f\text{Jx}}\) but fails on \(V_{bug}\), iii) The fixing patch should only include changes that fix the vulnerability and should not introduce unrelated changes such as features or refactoring, and iv) the vulnerability is not already in Vul4J.

We download all available vulnerability data in JSON format on May 13, 2022 from NVD. We parse this data and obtain a list of 7,116 GitHub projects by collecting the reference URLs of these vulnerabilities. We exclude projects which have less than 50% of their code in Java, resulting in 400 Java projects containing 933 unique vulnerabilities. We then try to identify the fixing commits for each of the 933 vulnerabilities by manually checking the reference links provided in the vulnerability report or by searching the vulnerability ID in the GitHub repository if no link is provided. We find vulnerability-fixing commits for 698 vulnerabilities. Then we manually filter out 185 vulnerabilities whose fixing commits contain non-Java changes and 314 vulnerabilities that do not have test cases in their fixing commits. We now have 199 vulnerabilities, each with test cases and a corresponding Java-only fixing commit. We then successfully reproduce 42 Java vulnerabilities that are not included in Vul4J, using building tools such as Maven or Gradle.

We end up with a dataset of **42 new reproducible real-world Java vulnerabilities** from thirty open-source projects. In detail, our dataset consists of _27 multi-hunk vulnerabilities_ from twenty-two projects and _15 single-hunk vulnerabilities_ from eleven projects. As Figure 1 shows, these 42 vulnerabilities covers a total of 23 CWE types. Furthermore, our dataset introduces **12 new CWE types** (denoted by " in Figure 1) not included in Vul4J and supplements four CWE types (CWE-78, CWE-200, CWE-310, CWE-863) for which Vul4J only has one example.

Table 1 describes the 15 new single-hunk vulnerabilities of twelve CWE types in our _VJBench_ benchmark. There are six new unique CWE types of vulnerabilities not present in Vul4J. As a result, there are 15 vulnerabilities from VJBench and 35 vulnerabilities from Vul4J, a total of **50 vulnerabilities** that we use in our study.

Figure 1. CWE Type Distribution of VJBench (* denotes the new CWE types not included in Vul4J).
## 3. Large Language Models and APR Techniques

### Large Language Models

We select five LLMs, i.e., Codex, PLBART, CodeT5, CodeGen and InCoder, because they are (1) state-of-the-art, (2) capable of performing code generation tasks without any modifications to the models or additional components (e.g., CodeBERT (Zhu et al., 2019) GraphCodeBERT (Zhu et al., 2019) are excluded), and (3) trained with enough source code so that they can understand code to some extent (e.g., we exclude T5 (Zhu et al., 2019), GPT-2 (Zhu et al., 2019), GPT-Neo (Zhu et al., 2019) and GPT-J (Zhu et al., 2019), whose training data is over 90% text). In this work, we study the LLMs in two settings: as is and fine-tuned with general APR data.

#### 3.1.1. Large Language Models As Is

In this section, we introduce the details of the studied LLMs and how to use them for fixing vulnerabilities. Table 3 provides the model sizes and their training data information.

**Codex (Zhu et al., 2019):** Codex is a GPT-3-based (Zhu et al., 2019; Zhu et al., 2019) language model with 12B parameters trained on both natural language and source code. We use the davinci-002 model (as of July 2022), which is supposed to be the most accurate Codex model (Boward et al., 2018). We focus on Codex's insertion mode as it provided the best results in our preliminary study among the three main modes: completion, insertion, and edit.

**CodeT5 (Zhu et al., 2019):** CodeT5 is an encoder-decoder transformer model (Zhu et al., 2019) pre-trained with an identifier-aware denoising objective and with bimodal dual generation tasks. It is trained on a corpus of 5.2 million code functions and 8.3 million natural language sentences from open-source repositories in six programming languages including Java. In this work, we use the largest CodeT5 model released, which has 770M parameters.

**CodeGen (Zhu et al., 2019):** CodeGen models are a series of autoregressive decoder-only transformers trained for conversational program synthesis. Their training data consists of 354.7B natural language tokens from THEPLE dataset and 150.8B programming language tokens extracted from a subset of the Google BigQuery database. In this work, we apply the CodeGen model which contains 6B parameters (the larger model with 16B parameters is not used due to the limitation of our machine).

**PLBART (Zhu et al., 2019):** PLBART uses an encoder-decoder transformer architecture with an additional normalization layer on the encoder and decoder. It's pre-trained on functions extracted from Java and Python GitHub repositories via denoising autoencoding. Two PLBART models of different sizes are available, and we use the larger model containing 400M parameters.

**InCoder (Zhu et al., 2019):** InCoder models follow XGLM (Zhu et al., 2019)'s decoder-only architecture and are pre-trained on the masked span prediction task. Its pre-training data comes from open-sourced projects on GitHub and GitLab, and StackOverflow posts. There are two InCoder models of different sizes released, and we use the larger one which contains 6B parameters.

**Input Formats:** Table 2 illustrates the input format we used for each model. For Codex, we adopt an input format similar to the one used in prior work (Zhu et al., 2019). The prompt includes the commented buggy code with hint words "BUG." and "FIXED." to signify the location of the bug and to guide Codex towards generating a fixed version of the code. If the number of input tokens exceeds the maximum number for a model, we truncate the code and input the code around the buggy lines. Since it is unclear how the commented buggy line prompts will affect the models' fixing capabilities, we experiment with the input with and without commented buggy lines for each model. Figure 2 shows an example of the input and expected output of Codex with buggy lines commented by /*BUG.. FIXED */.

#### 3.1.2. Fine-tuned Large Language Models

We also study the fixing capabilities of fine-tuned LLMs, since fine-tuning is a common technique to adapt a pre-trained LLM to a specific downstream task, such as code summarization or code translation (Zhu et al., 2019; Zhu et al., 2019; Zhu et al., 2019).

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**CWE** & **Description** & **Vulnerability IDs** \\ \hline
20 & Improper Input Validation & Pulsar-1 \\
22 & Improper limitation of path name & Halo-1 \\  & to a restricted directory & \\
74 & Improper Neutralization of Elements & Ratpack-1 \\  & in Output (injection) & \\  & Cross-site Scripting & Json-santizer-1 \\
172* Encoding error & Flow-1 & Flow-1 \\
200 & Exposure of sensitive information & Jenkins-1, Jenkins-2, Jenkins-3 \\
325* Missing cryptographic step & Jenkins-1 & BC-Java-1 \\
347* & Improper Verification of & \\  & Cryptographic Signature & \\
444* HTTP request swapping & Netty-1, Netty-2 \\
611 & Improper restriction of XML external entity reference & Quartz-1, Retrofii-1 \\
648* Exposure of resource to wrong sphere & Flow-2 \\
1295* Debug messages revealing & Flow-2 \\  & unnecessary information & \\ _unk_ & no specific CWE category & Jinjava-1 \\ \hline \hline \end{tabular}
\end{table}
Table 1. List of the 15 new single-hunk vulnerabilities categorized by their corresponding CWE. The vulnerability IDs compose of the project name and the bug index. * denotes the six new CWE types that our benchmark adds compared to Vul4J. Jenkins-1 and Flow-2 both belong to two CWE categories.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}} \hline \hline
**Model** & **Input Format** \\ \hline Codex & Comment buggy lines (BL) with hint ”BUG.” and ”FIXED.” Prefix prompt: Beginning of the buggy function to BL comment & \\  & Suffix prompt: Line after BL comment to end of the buggy function & \\ \hline CodeT5 & Mask buggy lines with \textless{extra\_id\_0}- and input the buggy function & \\ \hline CodeGen & Input beginning of the buggy method to line before buggy lines & \\ \hline PLBART & Mask buggy lines with \textless{mask\_5} and input the buggy function & \\ \hline InCoder & Mask buggy lines with \textless{mask\_5} and input the buggy function & \\ \hline Tuned LLMs & Comment buggy lines and input the buggy function & \\ \hline \hline \end{tabular}
\end{table}
Table 2. Input Formats of Large Language Models

Figure 2. An example input to Codex and its expected output
However, due to the lack of vulnerabilities as fine-tuning data, we use the LLMs fine-tuned with general APR data, shared by existing work (Xiong et al., 2018). Prior work (Xiong et al., 2018) fine-tuned LLMs with a training dataset containing 143,666 instances collected from open-source GitHub Java projects (Xiong et al., 2018). Each data instance is a pair of buggy code and fixed code. In detail, (Xiong et al., 2018) used the Adam optimizer with a learning rate of \(1e^{-5}\), set batch size to one and fine-tuned for one epoch. The fine-tuned LLMs are supposed to be adjusted to vulnerability fixing task to some extent due to the similarity between vulnerability fixing and general bug fixing. We perform a search and confirm that none of the vulnerabilities we study in this work is present in the APR training data used to fine-tune the LLMs.

We cannot fine-tune Codex, since it does not offer any fine-tuning API and there is also no fine-tuned Codex available. The last row of Table 2 describes the input format for using fine-tuned LLMs, where the buggy lines are given as commented lines, and the entire function is input into the fine-tuned LLMs to generate the patched lines (Xiong et al., 2018).

### APR Techniques

We select four state-of-the-art learning-based APR techniques trained for Java bugs. These APR techniques need to be open-sourced so that we can run them on our new vulnerability benchmarks.

**CURE**(Xiong et al., 2018) applies a small language model (pre-trained with 4.04M code instances) to the CoCoNuT's (Xiong et al., 2018) encoder-decoder architecture to learn code syntax and propose a new code-aware strategy to remove invalid identifiers and increase the compilation rate during inference. CURE is trained with 2.72M APR instances.

**Decoder**(Xiong et al., 2018) uses an tree-based deep learning network that is trained on 82.87K APR training instances. It focuses on generating edits to modify buggy ASTs to form the patched ASTs.

**RewardRepair**(Xiong et al., 2018) includes compilation in the calculation of the model's loss function to increase the number of compilable (and correct) patches. This is different from CURE as the loss function increases the number of compilable patches during training. Overall, RewardRepair is trained with 3.51M APR training instances.

**KNOD**(Xiong et al., 2018) proposes a novel three-stage tree decoder to generate the patched ASTs, and also uses domain-knowledge distillation to modify the loss function to let the models learn code syntax and semantics. KNOD is trained with 576K APR training instances, and is the state-of-the-art DL-based APR techniques.

## 4. Code Transformation

To address the challenge of training-testing data overlap, we need to create vulnerabilities and their fixes that have not been seen by existing LLMs or APR techniques. We generate unseen vulnerabilities by transforming existing vulnerabilities to their semantically equivalent forms. None of the APR models and LLMs, including Codex, have seen these transformed buggy code and the corresponding fixes in their training set. We apply two categories of transformations to Vul4J and VJBench, which are described below:

**(1) Identifier Renaming:** To prevent LLMs and APR models from simply memorizing the exact correct patches associated with identifier names, we rename identifiers in the buggy code and the corresponding fixed code. All variables, functions, and classes defined in the project are renamed using synonyms for the original identifier names according to Java specifications. We use synonyms to keep the word meaning of the original identifiers. We do not rename identifiers from external libraries or default Java class libraries, since one often cannot modify external libraries. Figure 3 shows an example of identifier renaming for Halo-1.

We first use the tool src2abs (Cordes et al., 2016) to extract all variable, function, and class names in the buggy function, and filter out those identifiers from Java or third-party libraries. We tokenize each identifier based on camel case or snake case conventions, then use NLTK WordNet (Chen et al., 2017) to generate synonyms for each word. After that, we reassemble these synonyms to form a complete identifier. We manually review and adjust the synonyms to ensure they fit the

Figure 4. Function chaining for VUL4J-30

Figure 5. Function-argument passing for Halo-1.

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline  & **Codex** & **CodeT5** & **CodeGen** & **PLBART** & **InCoder** \\ \hline \#Parameters & \multicolumn{2}{c}{12B} & 770M & \multicolumn{2}{c}{6B} & 400M & \multicolumn{1}{c}{6B} \\ \hline Training Data & NL & 45.0TB & - & 1.1TB & 79.0GB & 57.0GB \\ Raw Size & PL & 159.0GB & - & 436.3GB & 576.0GB & 159.0GB \\ \hline Training Data & NL & 499.0B & - & 354.7B & 6.7B & - \\ \#Tokens & PL & 100.0B & - & 150.8B & 64.4B & - \\ \hline Training Data & NL & - & 5.2M & - & 47.0M & - \\ \#instances & PL & - & 8.3M & - & 680.0M & - \\ \hline \hline \end{tabular}
\end{table}
Table 3. Model size (number of parameters) and training data size of the five LLMs we apply and report in this work
code context. Since some APR techniques need to extract identifiers from the whole project, we rename the identifiers used in the buggy function across the entire project.

**(2) Code Structure Change:** We define six transformation rules to change code structures.

* **If-condition flipping:** negates an if-condition and swaps the code blocks in the if and else branches.
* **Loop transformation:** converts a for loop to a while loop and vice versa.
* **Conditional-statement transformation:** turns a ternary expression (var = cond? exprTrue: exprFalse;) into an if-else statement (if (cond) (var = exprTrue;) else (var = exprFalse;)), and transform a switch statement into multiple if and elseif statements, and vice versa.
* **Function chaining:** merges multiple function invocations into one call chain, or conversely splits a function call chain into separate function invocations. Figure 4 shows an example where value.getClass().equals(...); is split into Class value_class = value.getClass(); and value_class.equals(...);.
* **Function-argument passing:** If a locally defined variable or object is only used as a function argument, we replace the function argument with its definition statement, or we extract the function call that is passed as a function argument into a separate variable/object definition. Figure 5 shows an example where the argument parentPath.normalize() is extracted and declared as a local object normalizedParentPath.
* **Code-order change:** alters the order of statements if changing the order does not affect the execution results. For example, funcA(); int n=%; can be transformed into int n=%; funcA(); as invoking funcA() and declaring int n do not affect each other.

For code structure change, we manually transform the buggy function. For each buggy function, we apply all applicable transformations at once. We further confirm the equivalence of the transformed bug by reproducing them using the same test set and applying semantically equivalent patches to pass the tests.

**A new benchmark (VJBench-trans):** In summary, to create bugs and patches that LLMs have not seen in their training set, we apply three sets of transformations (identifier renaming only, code structure change only, and both at the same time) to VJBench and Vul4J, and create _VJBench-trans_ that contains 3x50 = 150 transformed Java vulnerabilities. We search in GitHub and Google the transformed code, and find no public code that is the same as the transformed buggy function.

**Recover patches for evaluation:** The transformed code is still realistic and human-readable. However, for the ease of evaluating the correctness of plausible patches, we maintain a dictionary that stores the mapping between the renamed identifiers and their original names. For each vulnerability, we also write a patched program for its code structure transformed version, providing a reference for future dataset users.

## 5. Experiment Setup

Figure 6 provides an overview of our study. First, we build a new dataset of vulnerabilities, VJBench, that contains 42 new vulnerabilities. We use this new dataset and the original dataset (Vul4J) to benchmark the vulnerability-fixing capabilities of DL-based APR techniques, LLMs and fine-tuned LLMs. Each language model generates 10 patches for each bug through inference. For each APR model, we use its default beam search size and validate its top 10 patches. The generated patches are then validated using test cases and manual verification of all the patches that pass the test cases. Then, we apply code transformations on Vul4J and VJBench to generate VJBench-trans. Finally, we evaluate the impact of code transformations on the vulnerability-repair capabilities of all the LLMs, fine-tuned LLMs and APR techniques.

### Dataset

In this work, we focus on fixing single-hunk Java vulnerabilities as state-of-the-art DL-based APR models are designed to fix single-hunk bugs. We filter and obtain 35 single-hunk bugs from Vul4J dataset. Along with the 15 single-hunk vulnerabilities from VJBench, we have a total of 50 Java vulnerabilities. We use the perfect fault localization for these Java vulnerabilities, that is, we use the code lines that are modified in the developers' patches as the buggy lines.

### Large Language Model Setups

We evaluate each LLM with two input setups: (1) the buggy lines are commented as part of the input and (2) without the buggy lines. We observe that InCoder fixes more vulnerabilites when the input contains buggy line comments, while the other LLMs perform better without buggy lines. We then report the best-performing setup for each model in the rest of this paper. For fine-tuned LLMs, we follow the input format with buggy line comments used in (Zhu et al., 2018) which is described in Table 2.

We configure each model to generate 10 patches for each vulnerability. For CodeT5, CodeGen, PLBART and InCoder, we set their beam search size to 10. For Codex, we set its parameter \(n\), the number of candidates to generate, to 10. Considering the inherent randomness of the sampling method adopted by Codex, we run it twenty-five times for each vulnerability to obtain the average results. We run twenty-five times to control the margin of error small (\(\leq\)0.3) at 95% confidence level. We set the sampling temperature of Codex to 0.6, which is shown to have the best performance when sampling ten candidates in prior work (Cordex et al., 2017). We set the max number

Figure 6. Overview of our study
of newly generated tokens to 400 for Codex due to its request rate limit, and to 512 for all other LLMs.

### Patch Validation

Codex insertion mode generates code to be inserted between the prefix prompt and the suffix prompt. Since we use the code before and including the buggy line comment as its prefix prompt and the code after the buggy line comment as its suffix prompt, we replace the original buggy code with the code that Codex generates. Similarly, CodC5 generates code to replace the masked label in its input. PLBART generates the entire patched function that replaces the whole buggy function. CodeGen and InCoder are completion models that generate code to complete the given prefix prompt. We take the first complete function CodeGen and InCoder generate to replace the original buggy function. For all the fine-tuned LLMs, the fine-tuned CodeT5, CodeGen, PLBART and InCoder directly generate the patched code to replace the buggy code.

For each LLM and APR techniques, we first validate the top-10 patches they generate using the test cases from the project. Following prior work (Zhu et al., 2017; Zhu et al., 2017; Zhu et al., 2018; Zhu et al., 2019), _plausible patches_ are patches that pass all test cases, while _correct patches_ are semantically equivalent to developer patches, and _over-fitted patches_ are patches that pass all test cases but are incorrect. We manually inspect each plausible patch to identify if it is a correct patch.

## 6. Results and Findings

We evaluate the vulnerability fixing capabilities of five LLMs, four fine-tuned LLMs and four DL-based APR techniques on two real-world Java vulnerability benchmarks.

### RQ1: Vulnerability Fixing Capabilities

We run Codex twenty-five times and report the average number of fixed vulnerabilities with the margin of error, because Codex's patch generation is non-deterministic. For other LLMs, we only run them once since their patch generation is deterministic (Section 5).

Table 4 shows the fixing capabilities, i.e., the number of vulnerabilities that each approach fixes correctly, of five LLMs, four fine-tuned LLMs and four APR models. We consider the top ten patches since a recent study shows that almost all developers are only willing to examine ten patches at most (Zhu et al., 2017). Results in Table 4 are reported as X/Y, where X is the number of vulnerabilities correctly fixed by each technique and Y is the number of vulnerabilities that are plausibly fixed. A vulnerability is plausibly fixed by a model if the model generates a plausible patch (definition in Section 5.3).

#### 6.1.1. LLMs vs. APR Techniques

We first compare using LLMs as is with APR techniques. Here, _LLMs as is_ refers to that we apply Codex and LLMs under zero-shot learning and without fine-tuning. Our results show that Codex exhibits the best fixing capability. Out of a total of 50 vulnerabilities in VuldJ and VJBench, Codex fixes an average of 10.2 vulnerabilities with a margin of error of 0.3 (at 95% confidence). InCoder demonstrates the second best capability, fixing 5 vulnerabilities. The other LLMs and DL-based APR techniques only fix very few vulnerabilities. Overall, LLMs and APR techniques show very limited vulnerability fixing capabilities.

Our finding of Codex performing the best on fixing Java vulnerabilities is consistent with Codex's superior performance in repairing general bugs (Zhu et al., 2017) and in other domains (Brandt et al., 2016; Zhu et al., 2018; Zhu et al., 2019; Zhu et al., 2019), possibly due to its significantly larger model size and training data size as indicated in Table 3. Our result is also consistent with recent work (Zhu et al., 2019) in showing that LLMs without fine-tuning have competitive fixing capabilities - InCoder fix three more vulnerabilities than the best APR technique (RewardRepair). However, while (Zhu et al., 2019) shows that CodGen, PLBART and InCoder as is can fix 18%-23% general bug of Java APR benchmarks, our result shows that they can fix only 4%(250)-10%(5/50) vulnerabilities of VuldJ and VJBench. In real-world, only about 1-7% of bugs are vulnerabilities, resulting in few data for models to learn from. This means that, for neural networks, fixing vulnerabilities is more difficult than general bugs and requires more domain-specific knowledge.

**Finding 1**: Existing large language models and APR techniques fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%) vulnerabilities on average, exhibiting the best fixing capability.

#### 6.1.2. LLMs Fine-tuned with APR Data

We applied LLMs finetuned with general APR data by (Zhu et al., 2019) on the vulnerability benchmarks. We cannot fine-tune Codex as OpenAI does not provide a public API for fine-tuning. Table 4 shows that all the finet-tuned LLMs fix more vulnerabilities than their original models. In detail, fine-tuned InCoder fixes 9 vulnerabilities, 4 more than its original model. The second best models is fine-tuned CodeGen, which fixes 8 vulnerabilities, 6 more than its original model. Fine-tuned CodeT5 and fine-tuned PLBART each fixes 3 and 2 more vulnerabilities.

Overall, fine-tuning with general APR data can improve the fixing capabilities of LLMs for vulnerabilities. First, fine-tuning could adapt LLMs to APR tasks better, making LLMs be aware of generating patches instead of open-ending code or text. Second, though vulnerabilities have special characteristics (root causes) compared to general bugs, some vulnerabilities still share similar repair patterns with general bugs, such as replacing a function argument with another variable, which can be well learned during fine-tuning. Given the scarcity of real-world vulnerability data, our results implicate that fine-tuning LLMs with general APR data can be beneficial.

**Finding 2**: Fine-tuning with general APR data improves all four LLMs' vulnerability-fixing capabilities. Fine-tuned InCoder fixes 9 vulnerabilities, exhibiting competitive fixing capability compared to Codex's.

We also evaluate the compilation rates (i.e., portions of generated patches that compile) to study the quality of the patches. Uncompilable patches cannot be correct patches. Codex, the best model overall, has a compilation rate of 79.7%, which is significantly higher than that of the best fine-tuned LLM, fine-tuned InCoder (55.2%) and the best APR model, Recoder (57.6%). Fine-tuning notably improves CodeT5 and CodeGen's compilation rates, from 6.4% to 46.8% and from 35.8% to 47.2% respectively. On the other hand, the compilation rate of fine-tuned PLBART is 45.2%, slightly lower than the original PLBART's compilation rate of 47.8%. Despite the higher 65.2% compilation rate of InCoder compared to its fine-tuned model, it generates 82.0% duplicate patches, whereas the fine-tuned InCoder generates patches with more diverse modifications that result in more correct fixes. Overall, compared with compilation rates of repairing general bugs (Zhu et al., 2019), these compilation rates of fixing
vulnerability are lower. PLBART, CodeGen and InCoder without fine-tuning when repairing general bugs show an average of 65%-73% compilation rate (Wang et al., 2018), outperforming both of their original and fine-tuned models when repairing vulnerabilities.

Figure 6(a) shows an example of uncomompliable patches of Vul4J-12: The function signature declares t to be final, thus t's value is not allowed to be changed. However, Codex fails to capture this constraint, even though the function signature is only two lines above the buggy line. As a result, it generates code t-- to decrease t's value which makes the patch uncompilable. Similarly, RewardR ignores the fact that v and vt are both of type int, and invokes the invalid function equals on them. Figure 6(b) shows another example of uncomplable patch for Vul4J-1: parseArray is a method defined in another class in the project that accepts two or three arguments only. All the four fine-tuned LLMs generate the same uncompilable patches where they pass null as the fourth argument, because they do not have the information that parseArray does not accept four arguments.

These results suggest that LLMs' abilities to learn code syntax could be improved. Recent work (Wang et al., 2018; Wang et al., 2018) are steps in the right direction to add domain knowledge to models to help them learn code syntax and semantics. Another direction is prompt engineering, such as providing method signatures or type information in the prompt to specify the constraints. This would enable LLMs to utilize syntax information from across the entire project, rather than being limited to the code within the buggy function.

### RQ2: What kinds of vulnerabilities do LLMs and learning-based APR techniques fix?

Table 5 shows the vulnerabilities that are correctly fixed by the LLMs, fine-tuned LLMs, and APR techniques. In total, 16 vulnerabilities (belonging to ten CWE categories as shown in column _CWE_ with their description in column _Description_) from both benchmarks are fixed by at least one of the models. The IDs of these vulnerabilities are listed under column _Vul. ID_. Some vulnerabilities belong to no specific CWE category and are listed as _unk_.

Vul4J-47 is a vulnerability that only Codex can fix. Figure 7(a) shows the developer patch for Vul4J-47 of type CWE-611 (Improper Restriction of XML External Entity Reference). The correct fix requires inserting a statement xmlIn.setProperty(XMLInputFactory.SUPPORT_DTD,Boolean.FALSE) to disable the support of Document Type Definition (DTD), because DTD processing can be used to perform XML External Entity (XXE) attacks. The original buggy code only disables the support for external entities by setting the IS_SUPPORTIME_EXTERNAL_ENTITIES property to false, which is not enough to prevent the attack. Figure 7(b) shows an incorrect patch generated by fine-tuned CodeGen, which merely replaces the Boolean.FALSE with Boolean.TRUE. In general, except Codex, other LLMs and fine-tuned LLMs only fix vulnerabilities that require simple modifications such as deleting statements or replacing variable/method names.

On the other hand, Codex fixes 15 out of the 16 vulnerabilities (the union of all bugs, for which Codex generates at least one correct patch in twenty-five runs). The one vulnerability fixed by other LLMs but not Codex is Vul4J-39 of type CWE-200 (Exposure of Sensitive Information to an Unauthorized Actor). This vulnerability can be fixed by simply deleting the entire buggy code. However, for Vul4J-39, Codex generates patches by applying different modifications to the buggy code, rather than deleting it.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**LLMs**} & \multicolumn{4}{c}{**Fine-tuned LLMs**} & \multicolumn{4}{c}{**APR models**} \\ \cline{2-13}  & & Codex & CodexT & CodexGen & PLBART & InCoder & CodexT & CodexGen & PLBART & InCoder & CURE & Recoder & RewardR & KNDD \\ \hline Vllench (15) & **4.0**/ 4.0 & 0/0 & 1/2 & 2/3 & 2/2 & 3/4 & 3/4 & 2/3 & 3/4 & 0/1 & 1/2 & 2/3 & 0/0 \\ Vul4J (35) & **6.2**/ 10.9 & 2/2 & 1/6 & 0/4 & 3/4 & 2/7 & 5/8 & 2/6 & 6/9 & 1/4 & 0/4 & 0/2 & 1/1 \\ \hline
**Total (50)** & **10.2**/ 14.9 & 2/2 & 2/8 & 2/7 & 5/6 & 5/11 & 8/12 & 4/9 & 9/13 & 1/5 & 1/6 & 2/5 & 1/1 \\ \hline Compilation Rate (vt) & 79.7 & 6.4 & 35.8 & 47.8 & 65.2 & 46.8 & 47.2 & 45.2 & 55.2 & 24.5 & 57.6 & 37.7 & 37.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4. Comparison of LLMs and APR models on fixing Java vulnerabilities. For x/y in a cell, x denotes the number of correctly-fixed bugs, and y is plausibly-fixed bugs (with at least one patch that passes the test cases). RewardR is RewardRepair.

Figure 8. Java vulnerability Vul4J-47 and its patches

Figure 7. Vul4J-12’s and VulJ-1’s developer patch and uncompilable patch
Ratpack-1, Vul4J-12, Vul4J-39 and Jenkins-2 are four vulnerabilities fixed by the most number (6-7 out of 13) of models. Ratpack-1 (Figure 9) when initializing DefaultttttpHeaders, sets the constructor argument to false, which disables the validation for user-supplied header values. The correct patch is simply removing false or changing it to true to enable the validation. The fix for Vul4J-12 (Figure 6(a)) is to change the keyword while to if, and the fix for both Jenkins-2 and Vul4J-39 is to simply delete of an if statement that exposes sensitive information to unauthorized actors. The simplicity of these patches are evident from the number of models that can fix them.
### RQ3: Fixing Capabilities on Transformed Vulnerabilities

To mitigate the training-testing data overlapping threat, we apply code transformations to the benchmarks to study the generalization abilities of Codex and LLMs on unseen data (Section 4). Table 6 shows the number of vulnerabilities that LLMs as is, fine-tuned LLMs, and APR techniques can fix in four settings: (1) _No transformation_--the original vulnerability dataset, (2) _Rename only_--only identifier renaming is applied, (3) _Code structure change only_--only code structure change_ is applied, and (4) _Rename + code structure change_--both transformations are applied.

Overall, code transformations make LLMs (fine-tuned or not) and APR techniques fix fewer vulnerabilities. For example, fine-tuned InCoder fixes nine vulnerabilities in Vul4J and VJBench (no transformation), but only fixes five fully transformed vulnerabilities (Rename + Code structure change). The impact of transformation is smaller on some models, e.g., Codex and fine-tuned CodeT5, demonstrating these models' robustness against code transformations and generalized learning capabilities. This result, to some extent, addresses the threat of Codex's non-public training data and reveals Codex's strong learning and vulnerability-fixing capability. Many models only fix two or fewer vulnerabilities without transformations, thus the impact of transformations cannot be big for these models. However, we see a general trend across almost all models that these code transformations make models fix fewer number of vulnerabilities.

Figure 11(a) shows an example, Halo-1, whose correct fix is to call normalize() on pathToCheck to remove any redundant elements in the file path. This bug can be correctly fixed by Codex, fine-tuned CodeGen, and fine-tuned InCoder. Yet, after applying both transformations, only Codex can fix it (Figure 11(b)).

For fine-tuned LLMs, different transformations have different effects but each transformation significantly affects at least one LLM. For example, although identifier renaming has small effect on CodeT5 and CodeGen, it decreases the number of vulnerabilities that InCoder fixes by four. The result shows that our code transformation effectively tests the generalization ability of LLMs on unseen data.

One interesting observation is that some models fix transformed vulnerabilities that they cannot fix in the original dataset. This is a reasonable phenomenon because our transformation may convert a code snippet into a simpler form for the models to fix. For example, Vul4J-30 is a bug that none of the models fixes in its original form, but its transformed version is fixed by all four fine-tuned LLMs when code structure transformation is applied. Figure 13 shows that the fix of Vul4J-30 is to call trim() on String.valueOf(value). The original vulnerability is hard to fix as String.valueOf(value) is a part of a complex if-condition. Yet, after code transformation, String.valueOf(value) stands out as a single statement, which is easier for LLMs to repair. This phenomenon suggests that equivalent code transformation could be a promising direction to simplify the vulnerable code and enhance the effectiveness of fixing vulnerabilities.

## 7. Threats to Validity

Java vulnerabilities are diverse. It is hard for benchmarks to represent all of them. Thus, our findings might not generalize to all Java

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{LLMs} & \multicolumn{4}{c}{Fine-tuned LLMs} & \multicolumn{4}{c}{APR Techniques} \\ \cline{2-13}  & \multicolumn{1}{c}{Codex} & CodF5 & CodeGen & PLRM & InCoder & \multicolumn{1}{c}{CodF5} & CodeGen & PLRM & InCoder & \multicolumn{1}{c}{CURE} & Recoder & \multicolumn{1}{c}{Reward} & \multicolumn{1}{c}{KNDO} \\ \hline No transformation & **10.2 \(\pm\)0.3** & 2 & 2 & 5 & 5 & 8 & 4 & 9 & 1 & 1 & 2 & 1 \\ \multicolumn{13}{c}{Rename only} & **8.1 \(\pm\)0.3** & 0 & 1 & 0 & 2 & 4 & 6 & 1 & 5 & 0 & 1 & 1 & 1 \\ Code structure change only & **9.9 \(\pm\)0.3** & 0 & 2 & 2 & 1 & 4 & 6 & 4 & 5 & 0 & 1 & 1 & 2 \\ \multicolumn{13}{c}{Rename + code structure change} & **8.3 \(\pm\)0.4** & 0 & 1 & 1 & 1 & 3 & 4 & 3 & 5 & 0 & 1 & 1 & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 6. Impact of code transformation on LLMs’ and APR models’ vulnerability repair capabilities. For Codex, x \(\pm\) y: x denotes the average number of correctly fixed bug, and y denotes the margin of error (95% confidence).

Figure 12. Halo-1 before and after transformation

Figure 13. Vul4J-30 before and after code structure change
vulnerabilities. We address this threat by expanding the existing Java vulnerability benchmark with a new dataset of vulnerabilities.

We rely on developers' patches to assess whether a vulnerability is fixed. Developers may make a mistake in fixing vulnerabilities. Therefore, our ground truth might be incorrect. We mitigate this threat by only looking at vulnerabilities that are publicly disclosed in the NVD dataset that are reproducible and include test cases indicating that the fixed version is no more exploitable.

Another threat is that Codex (and other LLMs) may have been trained on the vulnerability patches in Vul4J and VJBench dataset. To mitigate this problem, we apply code transformations to create semantically equivalent vulnerabilities that are not included in their training dataset. Then we apply Codex to repair these transformed programs to prove that Codex is indeed able to repair new vulnerabilities that it has not seen.

## 8. Related Work

### DL-based Vulnerability Fixing Techniques

Much work uses DL to fix vulnerabilities. Encoder-decoder approaches have been proposed for repairing C vulnerabilities: (Kumar et al., 2017) fine-tuned a CodeT5 model with C vulnerability repair data; (Kumar et al., 2017) trained a transformer model on a large bug fixing dataset and then tuned on a small vulnerability fixing dataset, but they use sequence accuracy as the evaluation metric rather than practical APR settings. Previous work (Kumar et al., 2017) applied both CodeBERT and GraphCodeBert to fix vulnerabilities, but they only evaluated on a _synthetic_ vulnerability database, the Juliet 1.1 C/C++ test suite (Kumar et al., 2017), which is a benchmark for evaluating static analyzers only. As a result, the vulnerabilities in the dataset are isolated and simplified to fit within a few lines and are not representative of code vulnerabilities in the production. Our work is different since we use a dataset of _real-world_ vulnerabilities for our evaluation, making our results closer to what researchers and developers can expect of the quality of LLM vulnerability repair in real-world production code.

Prior work (Zhao et al., 2018) applied LLMs with zero-shot learning to repair seven hand-crafted C/Python vulnerabilities and 12 real-world C vulnerabilities. They explored the effectiveness of different prompt templates and used the static analysis tool CodeQL or C sanitizers to detect the vulnerabilities to incorporate the obtained error messages into the input prompts. Our work differs from (Zhao et al., 2018) in several main aspects. First, we study not only LLMs but also DL-based APR tools and LLMs fine-tuned with general APR data. Second, we evaluate our approach on a larger dataset of 50 real-world Java vulnerabilities. Third, we apply code transformations to mitigate the data leakage problem and suggest a new direction of using transformations to simplify the repair for some vulnerabilities. Most vulnerabilities in Vul4J and VJBench cannot be detected by state-of-the-art Java security analysis tools, so we cannot incorporate error messages in the input prompts as (Zhao et al., 2018) did.

### Vulnerability Benchmarks

Previous work proposed benchmarks and datasets to help evaluate vulnerability fixing approaches. Maestro (Maestro, 2018) propose a platform for benchmarking tools on Java and C++ vulnerabilities. As Maestro does not support running LLMs and APR models, we directly use the same Java vulnerability dataset, Vul4J (Maestro, 2018), with our new dataset VJBench. Other benchmarks and datasets of real-world vulnerabilities have been proposed (Kumar et al., 2017; Li et al., 2018; Li et al., 2018; Li et al., 2018). However, these datasets only contain code snippets from the fixing commits and do not have test cases. Therefore, such datasets can only support code matching when evaluating the correctness of patches, and cannot be used in automated program repair in practice.

### LLMs for Repair and Other Tasks

Researchers use LLMs to improve many software engineering tasks such as automated program repair (Zhao et al., 2018; Li et al., 2018; Li et al., 2018), auto-complete suggestions (Li et al., 2018), and pair-programming (Zhao et al., 2018). Much work also discusses the implication of LLMs for software developers (Li et al., 2018; Li et al., 2018; Li et al., 2018) and current limitations of LLMs (Li et al., 2018; Li et al., 2018; Li et al., 2018). Our work explores a different application domain of LLMs, with its own challenges (vulnerabilities are notoriously difficult to fix (Li et al., 2018)) that have not been well explored yet.

## 9. Conclusion

This work is the first to investigate LLMs' and DL-based APR models' capacity at repairing vulnerabilities in Java. We evaluate five LLMs, four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks including a new one that we create. We use code transformations to address the training and testing data overlapping threat of LLMs and create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans. We find that existing LLMs and APR models fix very few Java vulnerabilities, and call for new research innovations to improve automated Java vulnerability repair such as creating larger vulnerability repair training datasets, fine-tuning LLMs with such data, exploring few-shot learning, and leveraging simplifying transformations to improve program repair.

**Replication package**: Our benchmark and artifacts are available at (Li et al., 2018).

## Acknowledgement

We thank the reviewers for their insightful comments and suggestions. This work was funded in part by NSF 1901242, NSF 2006688, J.P. Morgan AI Faculty Research Awards, and Meta/Facebook Research Awards. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
