# A Novel Approach for Automated Program Repair using Round-Trip Translation
with Large Language Models

Fernando Vallecillos Ruiz<EMAIL_ADDRESS>0000-0001-7213-3732 Simula
Research LaboratoryOsloNorway ,  Anastasiia Grishina  0000-0003-3139-0200
<EMAIL_ADDRESS>Simula Research LaboratoryOsloNorway ,  Max Hort
0000-0001-8684-5909<EMAIL_ADDRESS>Simula Research LaboratoryOsloNorway and
Leon Moonen  0000-0002-1761-6771<EMAIL_ADDRESS>Simula Research
LaboratoryOsloNorway BI Norwegian Business SchoolOsloNorway

(2024)

###### Abstract.

Research shows that grammatical mistakes in a sentence can be corrected by
translating it to another language and back using neural machine translation
with language models. We investigate whether this correction capability of
Large Language Models (LLMs) extends to Automated Program Repair (APR).
Current generative models for APR are pre-trained on source code and fine-
tuned for repair. This paper proposes bypassing the fine-tuning step and using
Round-Trip Translation (RTT): translation of code from one programming
language to another programming or natural language, and back. We hypothesize
that RTT with LLMs restores the most commonly seen patterns in code during
pre-training, i.e., performs a _regression toward the mean_ , which removes
bugs as they are a form of noise w.r.t. the more frequent, _natural_ , bug-
free code in the training data. To test this hypothesis, we employ eight
recent LLMs pre-trained on code, including the latest GPT versions, and four
common program repair benchmarks in Java. We find that RTT with English as an
intermediate language repaired 101 of 164 bugs with GPT-4 on the HumanEval-
Java dataset. Moreover, 46 of these are unique bugs that are not repaired by
other LLMs fine-tuned for APR. Our findings highlight the viability of round-
trip translation with LLMs as a technique for automated program repair and its
potential for research in software engineering.

automated program repair, large language models, machine translation

††copyright: cc††journalyear: 2024††ccs: Software and its engineering
Correctness††ccs: Software and its engineering Automatic programming††ccs:
Software and its engineering Software testing and debugging

##  1\. Introduction

As software becomes ubiquitous and more people engage in software engineering
(SE) tasks, the need to ensure its reliability and integrity increases.
Automated program repair (APR) aims to fix errors in source code with minimal
human involvement, thus reducing code maintenance needs and releasing
resources for creative code writing. With the advent of language models
trained on source code, learning-based methods that use generative and
translation models to fix bugs have started to compete with traditional
heuristic and constraint-based approaches for APR (Monperrus, 2018; Le Goues
et al., 2019).

Despite all progress, it remains a challenge for both learning-based and other
repair methods to correctly address all bugs in APR benchmarks. A particular
challenging class of bugs are those that require the model to understand
complex contexts and logic. To overcome this situation, alternative techniques
for APR must be investigated.

The starting point for this paper is the observation that grammatical errors
in natural language can be fixed by translating sentences to another
intermediate language and then back, a process known as _round-trip
translation_ (RTT) (Hermet and Désilets, 2009). Inspired by how RTT can
correct errors in natural language, we set out to investigate to what extent
it can fix bugs in code in a similar fashion.

We propose a novel RTT pipeline for APR using state-of-the-art large language
models (LLMs) for code translation, summarization, and generation. The
pipeline uses either a programming language (PL) or a natural language (NL) as
intermediate representation. Moreover, the LLMs are used in zero-shot fashion:
Unlike other neural APR methods, RTT does not fine-tune models on the bug
repair task, but applies them _off-the-shelf_ as provided by the model
authors.

Our hypothesis is that RTT is capable of fixing bugs as a result of the
_regression toward the mean_ performed by generative language models. Studies
show that frequent code patterns in large code corpora are bug-free (Ray et
al., 2016). Thus, as a result of training LLMs on these corpora, RTT will
regress toward the same mean of bug-free code.

To empirically investigate this hypothesis, we conduct the first comprehensive
study on RTT with LLMs for APR. Our experiments use eight LLMs, including the
latest GPT versions, and four APR benchmarks. The models vary in size from
140M to ca.1.7T parameters and in objectives from code and docstring infilling
to code summarization, translation and generation. The benchmarks contain code
with different context size and bug complexity, ranging from student
assignments to real-world projects.

Contributions.  The main contributions of this work are: _(i)_ we propose a
novel approach for automated program repair using round-trip translation with
large language models; _(ii)_ we thoroughly test RTT with eight language
models, four APR benchmarks with various context sizes and bug types, and 10
different seeds for open-source models; _(iii)_ we investigate the performance
of RTT with two intermediate representations: another PL and English language;
_(iv)_ we explore the trade-off between LLM size and repair performance; _(v)_
we show that RTT repairs 101 of 164 bugs in the HumanEval-Java benchmark and
repairs 46 bugs that were not fixed by other methods, even those fine-tuned on
APR tasks; _(vi)_ we release the code for RTT and results obtained to ensure
replication and verification of our work.111 Replication package on Zenodo:
https://doi.org/10.5281/zenodo.10500593.

Figure 1. High-level overview of the RTT process with concrete examples taken
from our empirical evaluation. The red highlight on the left indicates the
buggy line, the green highlight on the right is the repaired line.

##  2\. Background and Related Work

###  2.1. Neural Machine Translation

Traditional methods for translating text from one language into another are
increasingly replaced by Neural Machine Translation (NMT), where neural
networks are used to predict a sequence of translated words (Stahlberg, 2020;
Yang et al., 2020). Sequence-to-sequence models enable NMT to automatically
learn complex mappings between different languages, efficiently capturing
context and offering more accurate translations compared to their
predecessors.

To generate translated sentences, NMT methods use the whole source sentence
and the initial part, or _prefix_ , of the target sentence. Assuming
$x=\left\\{x_{1},...,x_{n}\right\\}$ is the source sentence split into $n$
tokens, $y=\left\\{y_{1},...,y_{m}\right\\}$ is the target sentence and
$y_{<i}=\left\\{y_{1},...,y_{i-1}|\ i\leq m\right\\}$ is the beginning of the
target sequence generated up to token $i-1$, we can formulate the generation
as a conditional probability $P(y|x)$:

(1) $P(y|x)=\prod_{i=1}^{m}P(y_{i}|x,y_{<i}).$

###  2.2. Software Naturalness

_Software naturalness_ is the observation that source code exhibits patterns
and follows conventions that are statistically similar to other forms of human
expression (Hindle et al., 2012; Allamanis et al., 2018). This means that NLP
techniques such as NMT can be applied to source code. Neural Program
Translation (NPT) applies NMT to understand the underlying logic and semantics
of the source code and generates functionally equivalent programs in the
target language (Roziere et al., 2020).

###  2.3. Language Models for Automated Program Repair

Ray et al. (Ray et al., 2016) observed bugs to be deviations that manifest as
unnatural _noise_ which increases entropy in otherwise predictable and
repetitive natural code. This observation has been used to address various
tasks, such as APR (Tufano et al., 2019), vulnerability identification
(Buratti et al., 2020), and patch ranking (Kang and Yoo, 2022; Kolak et al.,
2022).

Interpreting APR as a translation from buggy to fixed sequences further
stimulated the use of language models (Chen et al., 2021a; Jiang et al.,
2021). Early models used RNN and LSTM architectures that cannot handle long-
range dependencies and scale poorly (Zhang et al., 2023). Transformers
(Vaswani et al., 2017) addressed these challenges and are now the prevalent
choice. Transformers for APR and other SE tasks have evolved by incorporating
new representations (Guo et al., 2021), new loss functions (Jain et al.,
2021), and increasing their size (Xia et al., 2023). This enables them to
understand more complex syntactical structures (Niu et al., 2022; Zeng et al.,
2022), and make human-competitive repairs (Fan et al., 2023). These techniques
differ from the RTT approach proposed in this paper in that they are fine-
tuned on, or prompted to perform, the APR task, whereas the proposed RTT
approach uses LLMs without any fine-tuning or prompting for APR. Furthermore,
recent work has leveraged the use of zero-shot LLMs to successfully perform
_cloze-style_ APR on numerous benchmarks (Xia and Zhang, 2022; Xia et al.,
2023; Jiang et al., 2023). These studies masked buggy lines in a function and
used off-the-shelf LLMs to predict the masked lines given surrounding code
tokens as context.

###  2.4. Round-Trip Translation

RTT involves translating a text from its original language to an intermediate
language and then translating it back to the original language. Our use of RTT
was inspired by the practical observation that, for our secondary languages,
we would check or correct errors using RTT through publicly available NMT
tools. The value of this practice was confirmed in a study by Hermet and
Désilets (Hermet and Désilets, 2009). Other uses of RTT in NMT include
improving translation results (Mehta et al., 2020), and testing the accuracy
of a translation model (Zhuo et al., 2023). In the context of APR research,
RTT has previously been used for data augmentation (Ahmad et al., 2023; Silva
et al., 2023).

##  3\. Repair through Round-Trip Translation

We propose a novel method for APR that is based on round-trip translation
using state-of-the-art LLMs. A high-level overview of the approach is
presented in Figure 1, together with concrete examples that are taken from our
experiments. The method uses two LLMs to translate code from one programming
language to another programming or natural language, and back to the first
programming language.

###  3.1. Motivation to Use Round-Trip Translation for Program Repair

We hypothesize that RTT is capable of repairing bugs as a result of the
_regression toward the mean_ or homogenization performed by generative
language models. The reasoning is as follows: The LLMs employed in RTT are
trained on vast real-world code corpora. Such models can treat code as natural
language due to the naturalness hypothesis (Hindle et al., 2012). They
generate or summarize code by iteratively selecting the sequence of the most
probable tokens, or the most probable sub-sequences of tokens using beam
search (Sutskever et al., 2014). The probability is estimated by the language
model based on its weights, and adapted during the model training to return
the most frequently occurring tokens in similar contexts. Ray et al. (Ray et
al., 2016) have shown that frequent code patterns in large real-word code
corpora are bug-free. Thus, as a result of training LLMs on these corpora,
they have a tendency to generate code that is also bug-free. This process in
which LLMs return the most probable tokens during generation can be viewed as
regression toward the mean, where noisy samples are replaced by samples closer
to the mean. Therefore, each translation step in RTT homogenizes the source
fragment toward a less noisy variant that is closer to the expected most
probable code, a patch candidate. In the context of code, bugs have been shown
to act as a form of noise that is less natural than the mean (Ray et al.,
2016), so they should be reduced or eliminated over the course of round-trip
translation.

###  3.2. Formulation of Round-Trip Translation

Formally, our approach can be described as follows. Let
$x=\left\\{x_{1},...,x_{n}\right\\}$ be a buggy code snippet split into $n$
tokens and $\tilde{x}=\left\\{\tilde{x}_{1},...,\tilde{x}_{m}\right\\}$ its
round-trip translated version with $m$ tokens, i.e., a candidate patch. We use
LLMs as neural machine translation models: $LLM_{A\rightarrow B}(\cdot)$ from
language $A$ to $B$, where $A\neq B$, and $LLM_{B\rightarrow A}(\cdot)$. The
round-trip translation of a code snippet $x$ is a two-legged translation. The
first leg, _forward translation_ , produces a sequence in language $B$, and
the second one, _backward translation_ , generates code in language $A$ from
the sequence in language $B$. The whole process can be expressed as:

(2) $\tilde{x}=LLM_{B\rightarrow A}(LLM_{A\rightarrow B}(x)).$

The total probability of the candidate patch generated by RTT, can be
expressed with an intermediate representation $r$ of $x$ as follows:

(3) $P\left(\tilde{x}\right)=P\left(\tilde{x}|r\right)\cdot P(r).$

Probabilities $P(r)$ and $P\left(\tilde{x}|r\right)$ can be approximated with
available LLMs according to Eq. 1 as follows:

(4) $\displaystyle P(r)\approx\prod_{i=1}^{k}P_{LLM_{A\rightarrow
B}}(r_{i}|x,r_{j<i}),$ (5) $\displaystyle
P\left(\tilde{x}|r\right)\approx\prod_{i=1}^{m}P_{LLM_{B\rightarrow
A}}(\tilde{x}_{i}|r,\tilde{x}_{j<i}).$

Therefore, we use two legs of translation to approximate the candidate patch
$\tilde{x}$ in Eq. 3. In this work, we use different intermediate languages
with the goal of encouraging a diverse range of representations, namely
natural language (English) and programming languages.

We also formalize the notion used to investigate if RTT can indeed repair
bugs. We denote a benchmark with $N$ buggy code snippets as
$\\{x^{i}\\}_{i=1}^{N}$, and let $Plausible(x)\rightarrow\left\\{0;1\right\\}$
be a function that returns $1$ if code snippet $x$ passes all test cases
(Zhang et al., 2023). Then, to evaluate if RTT can indeed repair bugs, we
check if a collection of snippets after round-trip translation has a higher
overall plausibility than the original collection, expressed by the following
equation:

(6) $\sum_{i=1}^{N}Plausible(\tilde{x}^{i})>\sum_{i=1}^{N}Plausible(x^{i}).$

The practical implementation and evaluation of RTT is discussed in more detail
in Section 4.3.

##  4\. Experiment Design

Our evaluation of RTT is guided by three research questions: RQ1: How well
does RTT perform with a programming language as intermediate representation?
RQ2: How well does RTT perform with a natural language (in particular,
English) as an intermediate? RQ3: What qualitative trends can be observed in
the patches generated by RTT?

To address these research questions, we use eight LLMs and four APR benchmarks
discussed in detail below. The selection of these models and benchmarks was
guided by ensuring a diverse and thorough evaluation of RTT for APR.

###  4.1. Models

We use eight distinct transformer-based language models for our evaluation.
Their sizes, architectures, and characteristics of training datasets are shown
in Table 1. We select the models based on two main requirements: _(i)_ they
are trained on large code corpora and perform well on code-related tasks;
_(ii)_ they can perform both legs of a round-trip translation, through an NL
or another PL. None of the model variants we use were originally trained or
fine-tuned for code repair, and we use them _as-provided_ , without additional
fine-tuning or training. Although the models’ original goal was not code
repair, we consider the _outputs_ of the backward (second) translation leg as
_candidate patches_ in our experiments. Observe that one can choose to use
different models in each leg of the translation, removing the need for our
second requirement. However, in the context of this paper, we focus on using
the same model in each leg.

Model |  Size | Architecture |  Data Source
---|---|---|---
PLBART |  base (140M) |  BART (encoder-decoder) |  StackOverflow BigQuery
CodeT5 |  base (220M) |  T5 (encoder-decoder) |  CodeSearchNet BigQuery
TransCoder |  $\sim$440M |  T5 (encoder-decoder) |  Google BigQuery
SantaCoder | 1.1B |  GPT-2 (decoder) |  The Stack (v1.1)
InCoder |  1.3B 6.7B |  MoE (decoder) |  StackOverflow GitHub/GitLab
StarCoderBase | 15.5B |  GPT-2 (decoder) |  The Stack (v1.2)
GPT-3.5 | 175B |  GPT-3 (decoder) | Public Data
GPT-4 |  (estim.) $\sim$1.7T |  GPT-4 (decoder) | Public Data
Table 1. Overview of language models used for RTT.

PLBART is an LLM for code-related tasks pre-trained on Java, Python, and
natural languages (Ahmad et al., 2021). It is released in two sizes, base and
large, and 53 versions fine-tuned on different PLs and tasks. Of those, we use
the _base_ models fine-tuned on code-to-code translation between Java and C#
and the _base_ models fine-tuned on code summarization (Java $\rightarrow$ NL)
and code generation (NL $\rightarrow$ Java).

CodeT5 extends T5 capabilities (Raffel et al., 2020) to code-related tasks
(Wang et al., 2021). The model comes in three sizes (small, base, large) and
versions fine-tuned on multiple tasks and programming languages. We use the
_base_ size models fine-tuned on the same types of tasks as PLBART.

TransCoder is an LLM designed for translation between C++, Python, and Java
(Roziere et al., 2020). The model was trained using a denoising auto-encoding
objective where source sequences in one PL contain noise, and the goal is to
generate target sequences in another PL without noise. This makes it a
promising candidate for an RTT pipeline where we aim to reduce noise or bugs
in two steps, from the original buggy example to the translated intermediate
representation and the final candidate patch.

SantaCoder is a model primarily for Java, JavaScript, and Python code
generation via infilling (Allal et al., 2023). The model was designed to
address ethical concerns about the use of LLMs for code. Despite its smaller
size, it performs comparably to larger models on code generation and
infilling. We use SantaCoder for RTT with NL because the model learned to
operate with docstrings during pre-training.

InCoder is an LLM able to perform code generation and editing via infilling
across 28 languages (Fried et al., 2023). The model achieves high performance
on code generation and infilling due to the use of bidirectional context and
the casual masking objective used during pre-training. The model is released
in two sizes trained on the same amount of data. We use InCoder in a similar
fashion to SantaCoder.

StarCoder is a generative LLM trained with infilling objective on more than 80
programming languages, Jupyter Notebooks, and Git communications (Li et al.,
2023). Two versions are released: StarCoderBase and StarCoder (fine-tuned for
Python), of which we use StarCoderBase in the infilling mode for experiments
with NL as intermediate.

GPT-3.5 (Brown et al., 2020) and GPT-4 (OpenAI, 2023) are the two latest
iterations of the Generative Pre-Training Transformer series, the black-box
models released via the OpenAI API.222 See
https://platform.openai.com/docs/guides/gpt. Specifically, we use the
gpt-3.5-turbo and gpt-4 models. They are not specifically designed for code-
related tasks, but perform well due to their diverse training dataset. Because
RTT with PLs as intermediate shows worse results than with NL as intermediate,
we chose to use the GPT models only with NL as intermediate to limit the
costs.

###  4.2. Benchmarks

We have chosen four diverse APR benchmarks, following Jiang et al. (Jiang et
al., 2023). This choice enables direct comparison between the RTT approach and
results of previous work on the NMT-style APR, in which buggy code is directly
translated to patches. All the benchmarks are in Java and contain buggy and
fixed code, as well as tests to check the test pass rate of the candidate
patches generated by RTT. Note that we use the concepts _problem_ , _bug_ ,
and _code example_ interchangeably, because we use buggy code examples with
single-hunk bugs only.

Defects4J v1.2 is a collection of 395 (4 depreciated) reproducible bugs from
open-source Java projects (Just et al., 2014). We followed Jiang et al. (Jiang
et al., 2023) and selected the 130 single-hunk bugs. The bugs have a wide
range of complexity and are related to multiple domains.

Defects4J v2.0 is the latest stable version of Defects4J. This version
provides an even more comprehensive and diverse collection of defects to test
APR techniques. It includes 438 additional bugs from nine open-source Java
projects from which we selected the 89 single-hunk ones.

QuixBugs contains 40 common student-level algorithmic programs, such as
bitcount, and spans 17 types of single-hunk bugs (Lin et al., 2017).

HumanEval-Java (Jiang et al., 2023) is a synthetic program repair dataset of
164 programs and unit tests translated to Java from the original version of
HumanEval (Chen et al., 2021b) in Python. The injected single-hunk bugs range
from simple incorrect operator usages to more complex logical bugs. One
advantage of HumanEval-Java is that it was not available during the training
of the models used in this study, which eliminates the data leakage risk.

###  4.3. Implementation of Round-Trip Translation

The RTT pipeline comprises four main steps: preprocessing of input buggy code,
generation of translations, postprocessing of RTT-generated outputs, and their
validation. We refer to _input_ source code as _buggy code_ , _buggy examples_
or simply _bugs_ , while _outputs_ of the RTT pipeline correspond to
_candidate patches_ in APR terminology. The first three steps are shown in
Figure 1. We add the fourth evaluation step in the current section. The result
is a versatile and parallelizable pipeline able to generate and validate RTT
patches with diverse models and test against different benchmarks. Our
pipeline extends the framework of Jiang et al. (Jiang et al., 2023).

Step 1: Preprocessing and Prompting.  We follow the common practice and
extract solely the buggy function as is also done by Jiang et al. (Jiang et
al., 2023). To conform with language models requirements, we add prefixes,
suffixes, masks and/or general style changes, such as removing newline
characters, before tokenization. We insert a Javadoc header that serves as a
prompt for the infilling models (SantaCoder, StarCoderBase, InCoder). Section
4.5 contains the exact prompts used for the models.

Step 2: Round-trip Translation.  In the round-trip translation step, we
generate two translations for each buggy example using the same type of LLM
for both RTT legs: from preprocessed buggy code to an intermediate language
and from the intermediate language back to the original language. We generate
5 different translations per leg in the round-trip translation using LLMs with
non-zero temperature to ensure the diversity of the intermediate
representations and final candidate patches. Therefore, for each buggy example
in a benchmark, we obtain 5 translations in the intermediate language and 25
final candidate patches, i.e., 5 from each intermediate translation.
Temperature and other model-specific hyperparameters are described in Section
4.4.

Step 3: Postprocessing.  We also perform minor postprocessing of the RTT-
generated candidate patches to ensure that function signatures are as expected
by the test suites. Therefore, we extract code if both code and text are
generated and remove extra tokens. This process also increases the readability
of patches. Section 4.6 contains more details on postprocessing with an NL and
PLs as intermediate translations.

Step 4: Evaluation of RTT Results.  The final step evaluates the postprocessed
RTT results against the test suites provided by the benchmarks. We calculate
additional metrics for each candidate patch to evaluate the performance of the
models and measure the effectiveness of RTT for APR.

###  4.4. Hyperparameters for Language Models

We use the recommendation of the authors of the models when choosing the
hyperparameters, unless specified otherwise in the current section. All the
final hyperparameter values are reported in Table 2. In detail, we set the
number of beams to 10 and the temperature to 1 for PLBART,333
https://github.com/wasiahmad/PLBART CodeT5,444
https://github.com/salesforce/CodeT5 and TransCoder.555
https://github.com/facebookresearch/TransCoder For SantaCoder,666
https://huggingface.co/bigcode/santacoder StarCoder,777
https://huggingface.co/bigcode/starcoderbase and InCoder,888
https://huggingface.co/facebook/incoder-1B,999
https://huggingface.co/facebook/incoder-6B we follow the hyperparameter setup
reported in their public demos with a number of beams of 1 and Top-P nucleus
sampling of 0.95. To account for the increased number of generated outputs (5
on each translation step), we modify the recommended temperature to 0.3 for
the first leg (code-to-text) and 0.4 for the second leg (text-to-code).

For the GPT models, we follow the advice where code-generating tasks, dealing
with structured code, should have a lower temperature than natural language.
We decide to use top-p close to 1 to increase diversity but follow OpenAI
guidelines to not drastically modify temperature and top-p simultaneously.
Moreover, we experimented with different other hyperparameters. For example,
we tried different tags in the Javadoc header and values of
_repetition_penalty_ , _length_penalty_ , _no_repeat_ngram_size_. Albeit we
did not perform a systematic study, the results from other prompts and the use
of these hyperparameters did not show any promising improvement in our use
case. Therefore, we have used the parameters as reported in Table 2 and the
prompts described in Section 4.5.

Model |  Number of beams |  Temperature first leg |  Temperature second leg |  Top-p
---|---|---|---|---
PLBART | 10 | 1 | 1 | -
CodeT5 | 10 | 1 | 1 | -
TransCoder | 10 | 1 | 1 | -
SantaCoder | 1 | 0.3 | 0.4 | 0.95
InCoder | 1 | 0.3 | 0.4 | 0.95
StarCoderBase | 1 | 0.3 | 0.4 | 0.95
GPT-3.5 | 1 | 0.3 | 0.2 | 0.95
GPT-4 | 1 | 0.3 | 0.2 | 0.95
Table 2. Hyperparameters used in RTT.

###  4.5. Prompt Choice

The prompts for the instruction and cloze-style models differ due to the
presence of the system message in the GPT models and the special infilling
tokens for the cloze-style models.

####  4.5.1. GPT models

We do not use conversation memory in-built in GPT-3.5 and 4 and run only one
forward and one backward translation step. The system message is as follows:
You are an expert programmer in all programming languages.
The user prompt for PL $\rightarrow$ NL summarization (forward translation)
has been chosen in the following way:
Create a Javadoc for the Java function delimited by triple backquotes. Do not
return generate the method again, return only the Javadoc. Java function:
```{buggy code}```.

For NL $\rightarrow$ PL code generation (backward translation), the user
prompt is as follows:
Given the signature of a Java function and its Javadoc delimited by triple
backquotes, generate the body of the function. Do not generate any additional
methods nor repeat the Javadoc nor give any explanations. Return only the
completed function without any comments.```{NL description as a comment and
function signature}```.

####  4.5.2. Open-source Models

For the rest of the models, we follow the default settings with the exception
of the following modifications. We ban the word TODO when generating code with
StarCoder and SantaCoder, since a considerable amount of generated candidate
patches were left blank after the use of the word. In addition, the tag
@description is inserted in the Javadoc header to prompt the models to
generate natural language summaries after the description tag.

The resulting prompt for the PL $\rightarrow$ NL step (forward translation)
has the following form:
/* @description <INFILL>
*/
{buggy code}.

We prepend the NL summary with the header $<|file\ ext=.java|>$ when using
InCoder for the code generation (second leg) to improve the overall results of
the model by giving context. The prompt for NL $\rightarrow$ PL code
generation (backward translation) is, therefore:
<| file ext=.java |>
/* @description {NL description}
*/
{function signature}.

The prompts are chosen code or NL summary generation are straightforward.
Investigation into other prompting strategies may positively affect the
results.

###  4.6. Ensuring Testability of Candidate Patches

When generating a candidate patch with RTT, we take two approaches according
to the intermediate language used, PL or NL. When using a PL, we overwrite the
scope and name of the generated candidate patch with the appropriate ones from
the buggy code. This ensures that the candidate patch is tested regardless of
small errors such as an upper-cased name.

When using NL, we provide the function signature known from the original buggy
example alongside the NL description generated in the first RTT leg (forward
translation step). We choose this procedure, because we observe that the
models do not generate such detailed descriptions in NL that can direct the
NL$\rightarrow$ PL models to recreate the name and types of input arguments
and return values consistently.

Finally, we set up the RTT pipeline so that we skip a buggy example if its
original code or the generated translation does not fit in the context window
of the model and mark such cases in the final results. However, out of the
four benchmarks and eight models from our evaluation setup, it happens only
for the _Jsoup 15_ bug and the StarCoderBase model that we are forced to skip
a buggy example due to a lack of computing resources.

###  4.7. Evaluation Metrics

We compute a total of seven common APR metrics for each candidate patch
generated by the RTT pipeline to evaluate the performance of RTT with
different models and assess the effectiveness of RTT for APR (Zhang et al.,
2023). We report the following metrics to _Weights & Biases_,101010
https://wandb.ai an online tool to analyze the models and their results:

  * •

_Compilability_ $\in\left\\{0,1\right\\}$: ability of the candidate patch to
be compiled successfully.

  * •

_Plausibility_ $\in\left\\{0,1\right\\}$: ability of the candidate patch to
pass all test cases of the corresponding benchmark.

  * •

_Test pass rate_ $\in\left[0,100\right]$: percentage of tests passed by the
candidate patch.

  * •

_Exact Match_ $\in\left\\{0,1\right\\}$: binary metric to check if the
candidate patch exactly matches the target solution.

  * •

_BiLingual Evaluation Understudy (BLEU)_ $\in\left[0,1\right]$ (Papineni et
al., 2002): evaluates by comparing the n-grams against target solution.

  * •

_CodeBLEU_ $\in\left[0,1\right]$ (Ren et al., 2020): extension of BLEU
designed for source code. It includes abstract syntax trees and code semantics
in the calculation of the score.

  * •

_CrystalBLEU_ $\in\left[0,1\right]$ (Eghbali and Pradel, 2023): extension of
BLEU designed for source code. It takes into account common n-grams due to
syntactic verbosity and coding conventions in the calculation of the score.

###  4.8. Addressing Model Stochasticity

To account for randomness in LLMs with non-zero temperatures, we run each
experiment that uses open source models with 10 different seeds and refer to
these as _10 runs_. This helps mitigate the impact of randomness and presents
a more accurate representation of RTT capabilities. We perform only one run
for each of the experiments with GPT-3.5 and GPT-4, because these models do
not allow setting a seed.111111 This has the added benefit of limiting overall
experiment costs, especially for GPT-4. The approximate cost for the single
run using GPT-3.5 was $\sim$10USD, while for GPT-4 it was $\sim$140USD.

###  4.9. Hardware

We run the patch generation on 3 NVIDIA V100 GPUs or 2 NVIDIA A100 GPUs,
depending on model needs. We run the test suites for patch validation on a
32-Core AMD EPYC 7601 CPU with 2TB RAM.

Model |  Defects4J v1.2 (130 bugs) |  Defects4J v2.0 (89 bugs) |  QuixBugs (40 bugs) |  Human Eval-Java (164 bugs)
---|---|---|---|---
PLBART (C#) | 1.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0
CodeT5 (C#) | 2.0 ± 0.0 | 1.3 ± 0.4 | 0.0 ± 0.0 | 1.0 ± 0.0
TransCoder (C++) | 1.8 ± 0.4 | 3.0 ± 0.0 | 0.0 ± 0.0 | 7.0 ± 0.0
TransCoder (Python) | 3.0 ± 0.0 | 2.0 ± 0.0 | 0.1 ± 0.3 | 8.0 ± 0.0
Table 3. Average number of plausible patches ± standard deviation over 10
runs, generated with a PL as intermediate. The best results are highlighted in
bold.

##  5\. Results and Discussion

###  5.1. Round-Trip Translation through PL

We first investigate the capabilities of LLMs to fix bugs via RTT using
another programming language as the intermediate. For this purpose, we use the
LLMs which are able to translate Java code into another PL: PLBART (Java
$\leftrightarrow$ C#), CodeT5 (Java $\leftrightarrow$ C#), and TransCoder
(Java $\leftrightarrow$ C++, Java $\leftrightarrow$ Python). Table 3
summarises the bug fixing performance of RTT along with the intermediate
language used in RTT. We set plausibility to 1 if at least one of the 25
generated candidate patches passes all the tests for a given buggy code
sample, sum up plausibility over buggy code examples in a dataset, and then
take an average over 10 runs with different seeds. We refer to the number of
buggy samples with plausible patches for a dataset as _plausibility rate._ For
RTT through PL, we observe that the average plausibility rate is low, with at
most eight bugs repaired on average for the HumanEval-Java dataset with 164
buggy code examples and at most three code examples repaired on average for
the remaining three datasets. PLBART only fixes a single bug across the four
datasets with RTT. CodeT5 provides at most two plausible patches for any of
the datasets.

The best performance with RTT through PL is achieved by TransCoder with Python
as intermediate PL on three out of four datasets. Moreover, TransCoder is the
only model that provided a plausible patch for QuixBugs. We observe that
larger models fix more buggy examples than smaller ones, which is aligned with
the general tendency of larger models to perform better on downstream tasks
(Wei et al., 2022).

The models tend to repeat the candidate patches for a given buggy example over
the 10 runs regardless of the non-zero temperature and different random seeds.
This trend, in addition to the low plausibility rates and standard deviation
obtained, can indicate a potential rigidity in the conceptual mapping between
languages, which may limit the model to literal translation, preventing
efficient use of context to filter out noise, or bugs. In other words, code-
to-code NMT models with similar target and source languages keep the same
tokens and logical bugs. This is supported by the fact that TransCoder with
Python as intermediate performs the best on three out of four datasets. Python
and Java are less alike than C# or C++ and Java, which motivates bigger
changes when translating.

RTT through PL. The use of PL as an intermediate language in our approach,
while yielding a very low number of plausible patches, has shed light on a few
key points: _(a)_ the intermediate translation should differ enough from the
buggy code; _(b)_ larger models produce better RTT results on APR through PL.

###  5.2. Round-Trip Translation through NL

We continue our experiments with RTT that uses a natural language (English) as
intermediate representation. We report the number of buggy code examples with
plausible patches in Table 4. We include average and standard deviation of the
plausibility rate over 10 runs with different seeds, as well as the exact
values observed in the union of all runs (Any Run) and their intersection
(Every Run). Note that the models in Table 4 are ordered by size.

| Model |  Defects4J v1.2 (130 bugs) |  Defects4J v2.0 (89 bugs) |  QuixBugs (40 bugs) |  Human Eval-Java (164 bugs)
---|---|---|---|---|---
Avg ± STD | PLBART | 2.0 ± 0.0 | 4.3 ± 0.5 | 3.0 ± 0.0 | 4.0 ± 0.0
CodeT5 | 2.0 ± 0.0 | 2.0 ± 0.0 | 1.0 ± 0.0 | 5.0 ± 0.0
SantaCoder | 9.0 ± 1.4 | 6.9 ± 2.2 | 16.1 ± 0.9 | 37.0 ± 1.7
InCoder (1.3B) | 4.6 ± 0.7 | 4.7 ± 1.2 | 5.8 ± 1.2 | 20.4 ± 1.8
InCoder (6.7B) | 7.0 ± 0.8 | 6.7 ± 1.1 | 10.5 ± 1.0 | 31.7 ± 1.3
StarCoderBase | 10.5 ± 1.5 | 7.2 ± 1.2 | 21.6 ± 0.9 | 42.8 ± 7.9
Any Run | PLBART | 2 | 5 | 3 | 4
CodeT5 | 2 | 2 | 1 | 5
SantaCoder | 14 | 13 | 23 | 50
InCoder (1.3B) | 6 | 10 | 8 | 34
InCoder (6.7B) | 16 | 10 | 16 | 43
StarCoderBase | 19 | 10 | 26 | 61
GPT-3.5 | 14 | 3 | 25 | 60
GPT-4 | 17 | 7 | 27 | 101
Every Run | PLBART | 2 | 4 | 3 | 4
CodeT5 | 2 | 2 | 1 | 5
SantaCoder | 3 | 4 | 8 | 26
InCoder (1.3B) | 4 | 2 | 2 | 12
InCoder (6.7B) | 2 | 1 | 7 | 20
StarCoderBase | 6 | 3 | 18 | 18
Table 4. Number of plausible patches over 10 runs, with NL as intermediate.
GPT-3.5 and GPT-4 are run once, and reported in the last two rows of “Any
Run”. The best results in each group are shown in bold.

(a) Defects4J

(b) QuixBugs

(c) HumanEval-Java

Figure 2.  Number of unique bugs fixed in various datasets by RTT through NL
with a fixed language model.

A clear correlation is observed between the model size and the average
plausibility rate (Pearson’s $r=0.78$). The only outlier is SantaCoder, which
also performs comparably to larger models on other tasks in related work
(Allal et al., 2023). The growth of the average plausibility rate from
SantaCoder (1.1B) to StarCoderBase (15.5B) is less pronounced. This result can
be affected by a larger proportion of Java code within SantaCoder training
data compared to StarCoderBase training set. In addition, the majority of
models with more than 1B parameters fix at least one bug with RTT through NL
that is not repaired by RTT through NL with other models. Note that for a
fixed model, bug repair performance differs based on the dataset. Thus, the
proportion of the buggy input examples with plausible patches is lower for
more complex datasets (Defects4J variants) and larger for simpler tasks (in
QuixBugs and HumanEval-Java).

The low standard deviation of the plausibility rate indicates that most models
tend to repair a similar number of bugs in every run. However, the number of
fixed bugs on _Any Run_ is two to three times higher on average than the
number of repaired bugs in _Every Run_. For example, StarCoderBase fixes 19
Defects4J v1.2 buggy input examples in the aggregation of 10 runs and only 6
same bugs in every run. This confirms RTT is capable of repairing diverse
bugs.

The aggregated metrics over 10 runs bring additional perspectives, such as a
comparison of unique bugs fixed by RTT and those in related work. Although the
number of repaired bugs tends to vary between the models, Figure 2 indicates
that in our approach, most models tend to solve at least one unique problem.
Therefore, the kind of problems solved by RTT are not only size-dependant, but
also model-dependant. Table 5 compares the results of our RTT approach with
those obtained in previous work using the exact same models, but either fine-
tuned for APR (Jiang et al., 2023), or using APR specific prompting on GPT-3.5
(Sobania et al., 2023). The majority of models used in RTT repair at least one
buggy example not repaired by the same models fine-tuned for NMT-type of APR
methods.

We compare the top-performing model in RTT, GPT-4, against the 10 LLMs studied
in the work of Jiang et al. (2023), fine-tuned and non-finetuned for the APR
task, in Figure 3. Not only is GPT-4 able to generate more plausible patches
than any of the tested models in (Jiang et al., 2023), but 30 out of the 101
bugs are only fixed by RTT through NL and not repaired by any of the tested
models without RTT. This comparison highlights that RTT generates plausible
patches for the bugs that common APR approaches have not fixed and emphasizes
the added value of RTT in the APR landscape.

Studies show that generating more candidate patches can result in higher
repair performance on the datasets used in our evaluation (Xia and Zhang,
2023; Xia et al., 2023). The authors generate 200 patches per model, compared
to 25 patches in our case, which may be the reason why they fix all bugs in
QuixBugs with GPT-3.5 (Xia and Zhang, 2023) or get 26 and 29 plausible patches
with InCoder 1.3B and 6.7B, respectively (Xia et al., 2023). However, we could
not find enough details or replication packages for those studies and do not
include them in Table 5. The latter work also shows that plausible patches
have, on average, lower entropy than non-plausible ones. In our experiments,
we obtain an almost uniform distribution of the number of plausible patches
depending on their position, i.e., how far on the list of 25 candidate patches
the plausible ones occur. This suggests that results can be improved by
generating more candidate patches with RTT, at higher resource usage.

Model |  Defects4J v1.2 (130 bugs) |  Defects4J v2.0 (89 bugs) |  QuixBugs (40 bugs) |  Human Eval-Java (164 bugs)
---|---|---|---|---
PLBART | 33 / 2 / 0 | 24 / 5 / 3 | 15 / 3 / 1 | 36 / 4 / 0
CodeT5 | 33 / 2 / 1 | 25 / 2 / 1 | 17 / 1 / 1 | 54 / 5 / 0
InCoder (1.3B) | 43 / 6 / 3 | 38 / 10 / 2 | 20 / 8 / 4 | 64 / 34 / 14
InCoder (6.7B) | 56 / 16 / 5 | 38 / 10 / 3 | 24 / 16 / 4 | 70 / 43 / 16
GPT-3.5 |  - |  - | 19 / 25 / 12 |  -

Table 5. Number of unique problems with plausible patches, shown as P / O / N,
with P in previous work, O in our work using RTT through NL (Any Run), and N
only in our work, not in previous.

Figure 3. GPT-4 comparison on HumanEval-Java.

RTT through NL. We observe that _(a)_ the leading models for bug fixing with
RTT are SantaCoder, StarCoderBase, GPT-3.5 and GPT-4, thus confirming the
trend that larger models obtain better results; _(b)_ although standard
deviation of plausibility rate is low, the number of bugs repaired in the
union of runs is 2-3 times higher on average than the rate for every run;
_(c)_ RTT through NL is able to repair bugs not repaired by the same models
fine-tuned on the APR task.

###  5.3. Qualitative Analysis of Generated Candidate Patches

Through a close inspection of candidate patches generated, we are able to gain
more insights into the quality of RTT-generated patches. We investigate
compilability, test pass rates, CodeBLEU and other characteristics of patch
candidates.

####  5.3.1. Compilability

We explore the average ratio of compilable patches out of all candidate
patches generated over 10 runs and present the results in Figures 4(a) and
4(b) for RTT through PL and NL, correspondingly. In general, RTT through PL
generates less compilable patches than RTT through NL. TransCoder helps RTT
through PL generate a high number of compilable patches, which we associate
with its rigorous denoising pre-training objective.

For RTT through NL, the trend of obtaining better compilability ratios with
larger models holds. The previously discussed pattern, where SantaCoder
obtains better results than InCoder and slightly worse than StarCoderBase,
holds, too. It is worth mentioning that GPT-3.5 yields high compilability rate
for the Defects4J datasets, which can be appointed to possible data leakage
during training. Overall, compilability on the datasets with challenging
contexts (Defects4J’s) is lower than on simpler tasks. Although compilability
rates vary across the models and datasets, the majority (80-96%) of candidate
patches generated by RTT have low test pass rate (0-10%).

(a) with PL as intermediate language

(b) with NL as intermediate language.

Figure 4. Percentage of compilable candidate patches generated in 10 runs,
where applicable, and at 25 attempts for each buggy example.

The trends observed for plausibility rates are also present for the ratio of
compilable candidate patches. Similarly to plausibility rates, compilability
percentage is higher for experiments with NL than for RTT through PL. On
average, the percentage of compilable patches out of all generated candidate
patches ranges from 7.3% to 72.7% per dataset for RTT through PL and from
11.1% to 88.2% for RTT through NL. Moreover, RTT generates a higher proportion
of compilable candidate patches on average for datasets with simpler tasks
(QuixBugs, HumanEval-Java) than for datasets with more complex contexts and
bugs (Defects4J variants). SantaCoder and StarCoderBase are the leading models
in terms of the average compilability for the RTT through NL, in addition to
the GPT variants. However, the best average results are obtained with
TransCoder (Java $\leftrightarrow$ C++) in the RTT through PL, unlike for
plausibility rate which was the best for TransCoder (Java $\leftrightarrow$
Python).

####  5.3.2. Test Pass Rates

To further explore the properties of RTT-generated candidate patches, we
analyze the test pass rate of the non-plausible patches. We aim to explore
what proportion of non-plausible patches miss smaller and larger ratios of
benchmark tests and are close to plausible solutions.

The test pass rate results are presented in Figures 5(a) and 5(b) for RTT
through PL and NL, respectively. We take a union of all non-plausible
candidate patches over four benchmarks and all runs for each model and report
the percentage of candidate patches in these unions that fall into test pass
rate ranges from 0–10%, 10–20% and so on, including the beginning and
excluding the end of the intervals. The vast majority of such a union of RTT-
generated candidate patches pass 0 to 10% of test cases both for PL and NL as
intermediate. This result indicates that non-plausible patches require more
fixing updates to repair the bugs in the chosen benchmarks. One avenue for
future work is to experiment with more iterations in the RTT and update the
model prompts or descriptions with bug summaries or results of not passing
test cases, similarly to Liventsev et al. (Liventsev et al., 2023).

(a) with PL as intermediate.

(b) with NL as intermediate.

Figure 5. Percentage of candidate patches in the different test passed rate
ranges. We explore what ratio of candidate patches generated by a specific
models for any of the datasets pass from A% (incl.) to B% (excl.) tests and
report percentage over all generated candidate patches. For example, 96% of
candidate patches generated with PLBART with NL as intermediate pass between
0% (incl.) and 10% (excl.) of tests.

####  5.3.3. Characteristics of RTT-generated candidate patches

To reiterate, we generate 5 intermediate translations in the first RTT leg,
for example, in English language, and 5 final translations from each of the
intermediates. We denote the first 5 intermediate translations as A, B, C, D,
and E. We enumerate backward translations obtained from each of the five
forward translations as A1–A5, B1–B5, …, E1–E5. The ratio of compilable
patches out of a union over all runs and datasets of all generated patches
with a fixed model is presented in Figure 6 for RTT through PL.

With RTT through PL, the number of compilable patches is decreasing from the
first to the last candidate patch generated from a fixed intermediate
translation. In other words, RTT-generated candidate patches at positions A1,
B1, …, E1 compile more frequently than candidate patches at positions A5, B5,
…, E5. The percentage of compilable patches out of all generated patches at
positions between A1 and A5, B1 and B5, …, E1 and E5 decreases for PLBART
(C#), CodeT5 (C#), and TransCoder (C++) but does not follow any trend for
TransCoder (Python). The number of plausible patches obtained with RTT through
PL is below eight on average, as mentioned previously. Thus, the trend between
plausibility and the position Ax,…, Ex is not observable from such low average
plausibility rates.

For RTT through NL, we do not observe any trend in terms of how frequently
first (A1, B1, …, E1) or later (Ax, …, Ex, x$\;>1$) patches are plausible or
compilable. This result is in line with TransCoder (Python) in the RTT through
PL. Remarkably, RTT through NL and RTT through PL with TransCoder (Python)
have higher plausibility rates than RTT with other models and through other
PLs, as shown previously. This observation points back at the discussion of
rigidity of code-to-code translation models in the RTT setting: They keep same
variable names, other tokens and logical bugs in place. By contrast, NL models
and code-to-code models with a PL that differs enough from the original PL
show better results in RTT for bug fixing. They abstract and change the input
buggy code enough to obtain a different representation that can in the next
step lead to a bug fix. The uniform estimated distribution of compilable and
plausible patches over the position at which they are generated also supports
the argument that sampling more candidate patches from LLMs in the RTT
pipeline can improve the bug fixing scores.

Figure 6. Percentage of compilable candidate patches in the 25 positions with
PL as intermediate. The percentage is calculated over all patches in four
benchmark datasets generated with RTT using a fixed model

_Other common APR metrics._ Exact Match, BLEU, and CodeBLEU are frequently
used to check whether candidate patches resemble the ground truth in
benchmarks (Lu et al., 2021). Since RTT is aimed at finding functionally
correct patches, not stylistically equivalent ones, we have found these
metrics non-descriptive for RTT. Especially when using NL as intermediate, RTT
can freely deviate from the original buggy code’s style, evidenced by the
average BLEU and CodeBLEU scores ± std over the patches that pass all tests
being less than 40.1 ± 0.09 and 63.7 ± 6.7, respectively.

####  5.3.4. CodeBLEU

We calculate average CodeBLEU values for buggy examples and candidate patches
generated by RTT with each fixed model over all runs and datasets and show the
frequency of observed values in Figure 7. The metric values are scaled to
$[0;100],$ with highest values being the best. The majority of buggy examples
have high CodeBLEU scores, which indicates that target bug fixes are very
similar to original buggy code. High CodeBLEU for the majority of buggy
examples is also explained by the type of bugs: We only consider single-hunk
bugs.

The majority of candidate patches obtained with RTT through PL have CodeBLEU
between 40 and 60, with the outlier value of ca. 26 frequently observed among
candidate patches for Defects4J variants. The most frequently observed average
CodeBLEU values for RTT through NL are between 20 and 50, with a similar
outlier value ca. 26 and an additional outlier of zero CodeBLEU noticeable for
a number of candidate patches for Defects4J versions. The frequency of higher
CodeBLEU values increases with larger model sizes. CodeBLEU values are
considerably lower than 100 for the vast majority of RTT-generated candidate
patches. However, one can observe the trend of regressing towards the mean
type of candidate patches with similar CodeBLEU calculated between targets and
RTT-generated patches.

Figure 7. Histogram on the CodeBLEU scores of candidate patches generated with
RTT through PL or NL (NL is default, if not mentioned). The distribution is
calculated over all patches generated for four APR benchmarks.

_Limitations of RTT for APR._ Studies show that the original style of writing
can be diluted by generative language models (Pedreschi et al., 2023; Grimaldi
and Ehrler, 2023). Furthermore, LLMs have been known to generate code
containing security flaws (Pearce et al., 2022; Tihanyi et al., 2023). These
flaws may compromise the integrity of the application therefore it is
recommended that developers thoroughly audit and review any generated code.
Our experiments show that bugs in code can be corrected via RTT, but we also
see that the original styling of the code is not always retained. Such a
restyle leads to challenges with code maintainability, which can reduce the
willingness of developers to adopt the approach. This issue will be less of a
challenge in projects that enforce a uniform coding style through automated
tools. Moreover, the impact will be lower if RTT is applied in smaller
contexts, for example, in highly modular projects with localized bugs where
restyling will have limited impact on maintainability.

Properties of RTT patches. _(a)_ The average compilability rate of RTT-
generated candidate patches is higher than their average plausibility rate.
_(b)_ RTT can change the code considerably, reducing usefulness of metrics for
ground-truth matching, such as BLEU and CodeBLEU. _(c)_ Because RTT can dilute
the coding style, it is best used in circumstances where rephrasing does not
impact maintainability.

##  6\. Threats to Validity

This section discusses four types of threats to validity for this study,
structured cf. Wohlin et al. (Wohlin et al., 2000, Sec. 6.7 & 6.8).

Internal Validity:  To support the validity of our results, we applied RTT
with two intermediate representations to four benchmarks and tested eight
models. As the benchmarks are publicly available, there is a risk that they
were used during training, also referred to as _data leakage_. This threat can
be mitigated by using models that remove the benchmarks from their training
data. Here, we use HumanEval-Java, which was constructed after the training of
any of the models used in this work. For the other three datasets, we find an
exact match in only 0.03% of the generated candidate patches, which reinforces
the validity of the data and results.

Construct Validity:

To evaluate the RTT performance, we apply widely used APR metrics, including
compilability and plausibility rates. For metrics that depend on test suites,
low-quality or easy-to-pass tests could positively bias the evaluation. We
mitigate this risk by employing four widely used APR benchmarks with different
bug types.

External Validity:

Threats to the external validity concern the generalizability of our approach.
We have validated the approach on a representative sample of APR benchmarks,
but have not extended the results to language pairs that were not covered.
Moreover, we applied the approach using only eight transformer-based models.
Extending the evaluation requires more computational resources and language
models that comply with the RTT requirements in Section 4.1.

Conclusion Validity:  For our experiments, we used off-the-shelf language
models that are publicly available and can be used without retraining. The
four benchmarks are also publicly available and widely used in APR research.
To support open science and enable replication and verification of our work, a
replication package is made available via Zenodo.1

##  7\. Conclusion

In this work, we explore the potential of round-trip translation with LLMs for
automated program repair. Our experiments confirm that RTT can fix various
bugs, including a good amount that are not fixable through other APR
approaches. Our analysis of the results has highlighted the complex relation
between RTT and naturalness of code. While RTT leverages the regression to the
mean to remove bugs, this process may also dilute the original code’s author
style and remove comments. We confirm the viability of RTT as a novel approach
to APR, but also caution to consider potential pitfalls and limitations.
Although RTT does not outperform the NMT and cloze-style APR, we show that
RTT, without any additional fine-tuning costs, is able to repair unique bugs
which were not fixed by the same models fine-tuned for NMT and cloze-type of
APR.

Future Work:  This study opens up several avenues for future research, such as
constrained forms of RTT where changes can only be made in certain masked
areas of the code, as well as reification of comments from the source code
with the fixed code, to ease adoption and ensure future maintainability. Our
replication package can be used as a base to expand the study to new models,
intermediate languages, and datasets. Gaining a deeper understanding of
characteristics and limitations of unique patches, such as model rigidity or
optimal number of candidate patches to generate, can help to develop more
sophisticated translation and repair techniques.

## References

  * (1)
  * Ahmad et al. (2021) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Unified Pre-training for Program Understanding and Generation. In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2655–2668.  https://doi.org/10.18653/v1/2021.naacl-main.211
  * Ahmad et al. (2023) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2023.  Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages. In _17th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, 1528–1542.
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra. 2023.  SantaCoder: Don’t Reach for the Stars!  arXiv:2301.03988 [cs]
  * Allamanis et al. (2018) Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018.  A Survey of Machine Learning for Big Code and Naturalness.  _Comput. Surveys_ 51, 4 (July 2018), 81:1–81:37.  https://doi.org/10.1145/3212695
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models Are Few-Shot Learners. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 33. Curran Associates, Inc., 1877–1901.
  * Buratti et al. (2020) Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, and Giacomo Domeniconi. 2020.  Exploring Software Naturalness through Neural Language Models.  https://doi.org/10.48550/arXiv.2006.12641 arXiv:2006.12641 [cs]
  * Chen et al. (2021b) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021b.  Evaluating Large Language Models Trained on Code.  arXiv:2107.03374
  * Chen et al. (2021a) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2021a.  SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair.  _IEEE Transactions on Software Engineering_ 47 (2021), 1943–1959.  https://doi.org/10.1109/tse.2019.2940179
  * Eghbali and Pradel (2023) Aryaz Eghbali and Michael Pradel. 2023.  CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code. In _37th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ _(ASE ’22)_. ACM, 1–12.  https://doi.org/10.1145/3551349.3556903
  * Fan et al. (2023) Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023.  Automated Repair of Programs from Large Language Models. In _IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. 1469–1481.  https://doi.org/10.1109/icse48619.2023.00128
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023\.  InCoder: A Generative Model for Code Infilling and Synthesis. In _11th International Conference on Learning Representations (ICLR)_. 1–26.
  * Grimaldi and Ehrler (2023) Gianluca Grimaldi and Bruno Ehrler. 2023.  AI et al.: Machines Are about to Change Scientific Publishing Forever.  _ACS Energy Letters_ 8, 1 (2023), 878–880.  https://doi.org/10.1021/acsenergylett.2c02828
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021\.  GraphCodeBERT: Pre-training Code Representations with Data Flow. In _International Conference on Learning Representations_. 1–18.
  * Hermet and Désilets (2009) Matthieu Hermet and Alain Désilets. 2009.  Using First and Second Language Models to Correct Preposition Errors in Second Language Authoring. In _Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications_ _(EdAppsNLP ’09)_. Association for Computational Linguistics, USA, 64–72.
  * Hindle et al. (2012) Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012.  On the Naturalness of Software. In _34th International Conference on Software Engineering (ICSE)_. IEEE, 837–847.
  * Jain et al. (2021) Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021\.  Contrastive Code Representation Learning. In _Conference on Empirical Methods in Natural Language Processing: EMNLP 2021_. Association for Computational Linguistics, 5954–5971.  https://doi.org/10.18653/v1/2021.emnlp-main.482
  * Jiang et al. (2023) Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023\.  Impact of Code Language Models on Automated Program Repair. In _45th International Conference on Software Engineering (ICSE)_. IEEE, 1430–1442.  https://doi.org/10.1109/icse48619.2023.00125
  * Jiang et al. (2021) Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021\.  CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. In _IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. 1161–1173.  https://doi.org/10.1109/icse43902.2021.00107 arXiv:2103.00073 [cs]
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst. 2014\.  Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In _23th International Symposium on Software Testing and Analysis (ISSTA)_ _(ISSTA 2014)_. ACM, 437–440.  https://doi.org/10.1145/2610384.2628055
  * Kang and Yoo (2022) Sungmin Kang and Shin Yoo. 2022.  Language Models Can Prioritize Patches for Practical Program Patching. In _3rd International Workshop on Automated Program Repair @ ICSE 2022_ _(APR ’22)_. ACM, 8–15.  https://doi.org/10.1145/3524459.3527343
  * Kolak et al. (2022) Sophia D. Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. 2022\.  Patch Generation with Language Models: Feasibility and Scaling Behavior. In _Deep Learning for Code Workshop @ ICLR 2022_. 1–11.
  * Le Goues et al. (2019) Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019\.  Automated Program Repair.  _Commun. ACM_ 62, 12 (Nov. 2019), 56–65.  https://doi.org/10.1145/3318162
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  StarCoder: May the Source Be with You!  arXiv:2305.06161 [cs]
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017.  QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge. In _Proceedings Companion of the ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH)_. ACM, 55–56.  https://doi.org/10.1145/3135932.3135941
  * Liventsev et al. (2023) Vadim Liventsev, Anastasiia Grishina, Aki Härmä, and Leon Moonen. 2023.  Fully Autonomous Programming with Large Language Models. In _Genetic and Evolutionary Computation Conference (GECCO)_. ACM, 1146–1155.  https://doi.org/10.1145/3583131.3590481
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021\.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In _Neural Information Processing Systems Track on Datasets and Benchmarks_. 1–16.
  * Mehta et al. (2020) Sneha Mehta, Bahareh Azarnoush, Boris Chen, Avneesh Saluja, Vinith Misra, Ballav Bihani, and Ritwik Kumar. 2020.  Simplify-Then-Translate: Automatic Preprocessing for Black-Box Translation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 05 (April 2020), 8488–8495.  https://doi.org/10.1609/aaai.v34i05.6369
  * Monperrus (2018) Martin Monperrus. 2018\.  Automatic Software Repair: A Bibliography.  _Comput. Surveys_ 51, 1 (Jan. 2018), 1–24.  https://doi.org/10.1145/3105906
  * Niu et al. (2022) Changan Niu, Chuanyi Li, Bin Luo, and Vincent Ng. 2022\.  Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code. In _31st International Joint Conference on Artificial Intelligence_. 5546–5555.  https://doi.org/10.24963/ijcai.2022/775
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs]
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: A Method for Automatic Evaluation of Machine Translation. In _40th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, 311–318.  https://doi.org/10.3115/1073083.1073135
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022\.  Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In _2022 IEEE Symposium on Security and Privacy (SP)_. 754–768.  https://doi.org/10.1109/sp46214.2022.9833571
  * Pedreschi et al. (2023) Dino Pedreschi, Luca Pappalardo, Ricardo Baeza-Yates, Albert-Laszlo Barabasi, Frank Dignum, Virginia Dignum, Tina Eliassi-Rad, Fosca Giannotti, Janos Kertesz, Alistair Knott, Yannis Ioannidis, Paul Lukowicz, Andrea Passarella, Alex Sandy Pentland, John Shawe-Taylor, and Alessandro Vespignani. 2023\.  Social AI and the Challenges of the Human-AI Ecosystem.  https://doi.org/10.48550/arXiv.2306.13723 arXiv:2306.13723 [cs]
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _The Journal of Machine Learning Research_ 21, 1 (Jan. 2020), 140:5485–140:5551.
  * Ray et al. (2016) Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar Devanbu. 2016.  On the ”Naturalness” of Buggy Code. In _38th International Conference on Software Engineering (ICSE)_. ACM, 428–439.  https://doi.org/10.1145/2884781.2884848
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020\.  CodeBLEU: A Method for Automatic Evaluation of Code Synthesis.  arXiv:2009.10297
  * Roziere et al. (2020) Baptiste Roziere, Marie-Anne Lachaux, Guillaume Lample, and Lowik Chanussot. 2020.  Unsupervised Translation of Programming Languages. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , Vol. 33. 20601–20611.  https://doi.org/10.48550/arxiv.2006.03511
  * Silva et al. (2023) André Silva, João F. Ferreira, He Ye, and Martin Monperrus. 2023.  MUFIN: Improving Neural Repair Models with Back-Translation.  https://doi.org/10.48550/arXiv.2304.02301 arXiv:2304.02301 [cs]
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An Analysis of the Automatic Bug Fixing Performance of ChatGPT. In _IEEE/ACM International Workshop on Automated Program Repair (APR)_. 23–30.  https://doi.org/10.1109/apr59189.2023.00012
  * Stahlberg (2020) Felix Stahlberg. 2020\.  Neural Machine Translation: A Review.  _Journal of Artificial Intelligence Research_ 69 (Oct. 2020), 343–418.  https://doi.org/10.1613/jair.1.12007
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014\.  Sequence to Sequence Learning with Neural Networks. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 27. Curran Associates, Inc., 1–9.
  * Tihanyi et al. (2023) Norbert Tihanyi, Tamas Bisztray, Ridhi Jain, Mohamed Amine Ferrag, Lucas C. Cordeiro, and Vasileios Mavroeidis. 2023.  The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verification.  arXiv:2307.02192 [cs.DB]
  * Tufano et al. (2019) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019.  An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation.  _ACM Transactions on Software Engineering and Methodology_ 28, 4 (Sept. 2019), 19:1–19:29.  https://doi.org/10.1145/3340544
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need. In _Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 30. Curran, 1–11.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021\.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In _Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 8696–8708.  https://doi.org/10.18653/v1/2021.emnlp-main.685
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022\.  Emergent Abilities of Large Language Models.  _Transactions on Machine Learning Research_ (June 2022), 1–30.
  * Wohlin et al. (2000) C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell, and A. Wesslen. 2000\.  _Experimentation in Software Engineering: An Introduction_.  Number 6 in The Kluwer International Series in Software Engineering. Springer.
  * Xia et al. (2023) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023\.  Automated Program Repair in the Era of Large Pre-Trained Language Models. In _45th International Conference on Software Engineering (ICSE)_. IEEE, 1482–1494.  https://doi.org/10.1109/icse48619.2023.00129
  * Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022.  Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-Shot Learning. In _30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)_ _(ESEC/FSE 2022)_. ACM, 959–971.  https://doi.org/10.1145/3540250.3549101
  * Xia and Zhang (2023) Chunqiu Steven Xia and Lingming Zhang. 2023.  Keep the Conversation Going: Fixing 162 out of 337 Bugs for $0.42 Each Using ChatGPT.  arXiv:2304.00385 [cs]
  * Yang et al. (2020) Shuoheng Yang, Yuxin Wang, and Xiaowen Chu. 2020.  A Survey of Deep Learning Techniques for Neural Machine Translation.  arXiv:2002.07526 [cs]
  * Zeng et al. (2022) Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022\.  An Extensive Study on Pre-Trained Models for Program Understanding and Generation. In _31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_. ACM, 39–51.  https://doi.org/10.1145/3533767.3534390
  * Zhang et al. (2023) Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023.  A Survey of Learning-based Automated Program Repair.  arXiv:2301.03270 [cs]
  * Zhuo et al. (2023) Terry Yue Zhuo, Qiongkai Xu, Xuanli He, and Trevor Cohn. 2023\.  Rethinking Round-Trip Translation for Machine Translation Evaluation. In _Findings of the Association for Computational Linguistics: ACL 2023_. Association for Computational Linguistics, 319–337.  https://doi.org/10.18653/v1/2023.findings-acl.22