pubstamp pubstamp

# CodeScope: An Execution-based Multilingual Multitask Multidimensional
Benchmark for Evaluating LLMs on Code Understanding and Generation

Weixiang Yan1 Haitian Liu211footnotemark: 1 Yunkun Wang311footnotemark: 1
Yunzhe Li411footnotemark: 1 Qian Chen5 Wen Wang5 Tingyu Lin6 Weishan Zhao7 Li
Zhu2 Shuiguang Deng3 Hari Sundaram4
1 University of California, Santa Barbara 2 Xi’an Jiaotong University 3
Zhejiang University 4 University of Illinois at Urbana-Champaign 5 Speech Lab,
Alibaba Group 6 TU Wien 7 University of Chinese Academy of Sciences
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>{tanqing.cq<EMAIL_ADDRESS>
Equal contribution. Work is supported by Speech Lab, Alibaba Group.
Corresponding to<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have demonstrated remarkable performance on
coding related tasks, particularly on assisting humans in programming and
facilitating programming automation. However, existing benchmarks for
evaluating the code understanding and generation capacities of LLMs suffer
from severe limitations. First, most benchmarks are deficient as they focus on
a narrow range of popular programming languages and specific tasks, whereas
the real-world software development scenarios show dire need to implement
systems with multilingual programming environments to satisfy diverse
requirements. Practical programming practices also strongly expect multi-task
settings for testing coding capabilities of LLMs comprehensively and robustly.
Second, most benchmarks also fail to consider the actual executability and the
consistency of execution results of the generated code. To bridge these gaps
between existing benchmarks and expectations from practical applications, we
introduce CodeScope, an execution-based, multilingual, multi-task, multi-
dimensional evaluation benchmark for comprehensively gauging LLM capabilities
on coding tasks. CodeScope covers 43 programming languages and 8 coding tasks.
It evaluates the coding performance of LLMs from three dimensions
(perspectives): difficulty, efficiency, and length. To facilitate execution-
based evaluations of code generation, we develop MultiCodeEngine, an automated
code execution engine that supports 14 programming languages. Finally, we
systematically evaluate and analyze 8 mainstream LLMs on CodeScope tasks and
demonstrate the superior breadth and challenges of CodeScope for evaluating
LLMs on code understanding and generation tasks compared to other benchmarks.
The CodeScope benchmark and datasets are publicly available at
https://github.com/WeixiangYAN/CodeScope.

##  1 Introduction

Driven by advances in deep learning (DL) and natural language processing
(NLP), large language models (LLMs) have demonstrated outstanding proficiency
in various generation and understanding tasks (OpenAI, 2023; Anil et al.,
2023). However, existing benchmarks (Hendrycks et al., 2020; Zhong et al.,
2023; Zheng et al., 2023a) for evaluating LLMs mainly focus on NLP tasks, such
as common sense reasoning, academic examination, and authenticity
verification. Existing evaluation methods are significantly insufficient in
terms of assessing integrity and comprehensiveness for code understanding and
generation capabilities of LLMs. Firstly, many code LLMs, such as CodeT5+
(Wang et al., 2023b), WizardCoder (Luo et al., 2023), and Code LLaMA (Rozière
et al., 2023), employ their own specific single-task evaluation datasets,
making it infeasible to comprehensively compare the performance of various
LLMs on code understanding and generation tasks on a unified standard.

Secondly, existing datasets designed for evaluating LLMs on code understanding
and generation tasks (Chen et al., 2021; Austin et al., 2021) mainly focus on
a narrow spectrum of popular programming languages, mostly Python only, and
single program synthesis related tasks, while ignoring evaluations of LLMs in
multilingual and multitask settings, that is, scenarios involving multiple
programming languages and multiple code understanding and generation tasks. In
real-world software development scenarios, multiple programming languages are
often employed to address diverse requirements, while these languages may
adhere to various programming paradigms, including object-oriented,
functional, and procedural, among others. Therefore, evaluating LLMs in a
multilingual context can provide insights into their generalization
capabilities across different languages and paradigms. Moreover, the
complementarity between multiple tasks facilitates a comprehensive evaluation
of the overall performance of LLMs, ensuring that an LLM is not over-optimized
for a specific task and can maintain strong performance across diverse tasks.
Importantly, compared to single-task settings, multi-task settings more
accurately simulate the various requirements and challenges faced in real-
world software development practices and hence better test the
generalizability of LLMs, that is, to evaluate whether LLMs can consistently
provide correct and reasonable solutions to previously unseen tasks.

Thirdly and more importantly, most studies (e.g., widely used benchmarks
CodeXGLUE (Lu et al., 2021) and XLCoST (Zhu et al., 2022)) rely only on
matching-based evaluation metrics and overlook the practical applicability of
the generated code, even though the ultimate goal of the generated code is to
successfully complete a specific task through execution. Matching-based
metrics, which solely measure the semantic similarity between generated code
and reference code, could misjudge code that functions identically via
different implementations, or code that appears syntactically correct but
fails to produce the expected results during execution. In contrast,
execution-based metrics consider the generated code successful only if the
generated code can successfully execute and also fulfill its intended
function. Hence, execution-based evaluation methods offer a more authentic and
comprehensive way to determine whether the code output by LLMs is genuinely
applicable and effective. Notably, the recent XCodeEval benchmark (Khan et
al., 2023) considers execution-based evaluations, but it is not fully designed
for evaluating LLMs. For example, its code retrieval task cannot evaluate the
code understanding and generation capabilities of LLMs. Moreover, our
verification also detects significant flaws in the XCodeEval dataset.

To address these limitations, we propose CodeScope, an execution-based
multilingual multitask multidimensional benchmark, which aims to provide a
unified platform for comprehensively evaluating the coding proficiency of
LLMs. Based on a thorough analysis of academic research and real-world
software development processes, we include 8 different tasks for code
understanding and generation in CodeScope, with task descriptions summarized
in Table 1. CodeScope covers 43 programming languages with an average of 13
programming languages per task, effective in evaluating code understanding and
generation abilities of LLMs in multilingual multitask settings. We also
conduct comprehensive evaluations of code understanding and generation
capabilities of a broad range of LLMs spanning three distinct dimensions (that
is, multi-dimensional): Difficulty, Efficiency, and Length, defined as
follows:

  * •

Difficulty: CodeScope sets multiple levels in generation tasks with
progressively increased difficulty, aiming to benchmark the code understanding
and generation ability of LLMs when dealing with programming problems of
different difficulties.

  * •

Efficiency: CodeScope includes a special Code Optimization task that evaluates
the execution speed of code generated by LLMs while factoring in efficiency of
utilizing hardware resources.

  * •

Length: CodeScope sets multiple levels in each task with increased code
lengths, aiming to evaluate the ability of LLMs to understand and generate
long code.

Category Task #Languages #Test Samples Avg. #Tokens/Sample Understanding Code
Summarization 43 4,841 378 Code Smell 2 200 650 Code Review 9 900 857
Automated Testing 4 400 251 Generation Program Synthesis 14 802 494 Code
Translation 14 5,382 513 Code Repair 14 726 447 Code Optimization 4 242 444

Table 1: Summary of our CodeScope. We report the number of samples (#Test
Samples) and the average number of tokens per sample (Avg. #Tokens/Sample) for
test sets of each dataset. Token counts are based on OpenAI’s tiktoken
tokenizer.

CodeScope is a benchmark adopting execution-based evaluation metrics. To
facilitate this goal, we develop a Multilingual Code Execution Engine denoted
MultiCodeEngine. MultiCodeEngine extends the ExecEval engine for XCodeEval
(Khan et al., 2023), which supports 11 programming languages, to facilitate
execution environments for all the 14 programming languages for code
generation tasks in CodeScope. We established 8 strong baselines for each task
respectively to facilitate comprehensive comparisons of coding capabilities of
LLMs. We expect these explorations will provide a deep understanding of the
strengths and limitations of LLMs on code understanding and generation tasks
and provide valuable guidance for future research directions.

Our contributions can be summarized as follows:

  * •

CodeScope benchmark: We built the first-ever comprehensive execution-based
multilingual multitask multidimensional benchmark for evaluating LLMs on code
understanding and generation tasks, CodeScope, which covers the largest number
of programming languages (43 in total) and comprises the broadest spectrum of
diverse code understanding and generation tasks (8 tasks in total) to date.
This benchmark evaluates the actual execution of the generated code,
facilitated by MultiCodeEngine, a multilingual code execution engine
supporting 14 programming languages.

  * •

Multi-dimensional fine-grained evaluation: We comprehensively evaluate the
performance of LLMs on 8 tasks from 3 distinct dimensions, namely, difficulty
(i.e., complexity of programming problems); efficiency (i.e., execution
efficiency of generated code); and length (i.e., length of code required to
solve the problem).

  * •

Comprehensive evaluations and in-depth analyses: We conduct comprehensive
validations and analyses of the utility of the CodeScope benchmark and
establish strong baselines for each task. We evaluate and compare the coding
capabilities of 8 mainstream LLMs. We also investigate the cross-lingual and
cross-task transferability of these LLMs.

##  2 Related Work

Many existing benchmarks for code understanding and generation tasks overlook
execution-based evaluations. Lu et al. (2021) proposed the CodeXGLUE
benchmark, which includes 10 code understanding and generation tasks, yet it
does not evaluate the actual execution of the generated code. Similarly, Zhu
et al. (2022) expanded the variety of programming languages to develop the
XLCoST benchmark but did not adopt an execution-based evaluation metric.
Previous research has shown no strong correlation between code lexical
similarity and execution correctness (Chen et al., 2021; Austin et al., 2021;
Ren et al., 2020). Puri et al. (2021) proposed the semantics-based CodeNet
benchmark, which significantly increased the variety of supported programming
languages, yet the evaluation tasks remain relatively limited as code
similarity and classification, code translation, and code performance (runtime
and memory) improvement techniques. It is noteworthy that experts have found
that about half of the solutions in the CodeNet datasets are incorrect (Zhu et
al., 2022). Recently, Khan et al. (2023) introduced the execution-based
multilingual multitask XCodeEval benchmark. However, some of the tasks of
XCodeEval are unsuitable for evaluating the performance of LLMs, such as its
code retrieval task, since there is still no sufficiently data-supported code
knowledge base for LLMs. Therefore, our validation process also detects
significant flaws in XCodeEval’s dataset.

In addition, most of the related works (Hendrycks et al., 2021; Lai et al.,
2023; Huang et al., 2022; Nijkamp et al., 2023; Chandel et al., 2022) focused
on code generation tasks with target as Python when evaluating model
performance, mainly because setting up a Python execution environment is
relatively straightforward. For instance, Chen et al. (2021) introduced
HumanEval, the most widely used code generation benchmark, which includes 164
Python samples and an average of 6.7 unit tests. Austin et al. (2021)
introduced the MBPP dataset, featuring 974 entry-level Python programming
tasks, while the MathQA-Python dataset (Austin et al., 2021) contains 23,914
more complex problems. APPS (Hendrycks et al., 2021) aimed to elevate the
complexity of programming challenges, featuring a collection of 10,000
problems, each averaging 293.2 words in length, and typically, each problem is
associated with an average of 21.2 test cases. ClassEval (Du et al., 2023)
evaluates LLMs in the complex scenario of class-level code generation,
including 100 class-level Python code generation tasks. The study indicates
that current LLMs still face considerable challenges in effectively handling
class-level code generation.

While recent works (Yu et al., 2023; Li et al., 2022a) tried to expand the
range of supported programming languages for code generation tasks, the
coverage of programming languages remains narrow. For example, Hao et al.
(2022) introduced AiXBench, which includes 175 Java samples. However, since
each sample lacks unit tests, model performance has to be evaluated manually.
MultiPL-E (Cassano et al., 2022), translated the HumanEval and MBPP benchmarks
into eighteen languages through compiler methods. However, the translation
accuracy is not guaranteed. Athiwaratkun et al. (2023) developed the MBXP
dataset using a scalable transformation framework to generate a dataset that
spans ten programming languages. On the other hand, Zheng et al. (2023b)
employed a human translation approach to create HumanEval-X to cover five
programming languages. Concurrently, recent research has pointed out that
HumanEval exhibits marked limitations in gauging a model’s ability to produce
context-dependent code (Yu et al., 2023). Table 2 shows the comparison between
the different code evaluation benchmarks.

Benchmark Execution-Based Multilingual Multitask Multidimensional HumanEval ✓
✗ ✗ ✗ MBPP ✓ ✗ ✗ ✗ CodeXGlue ✗ ✓ ✓ ✗ XLCoST ✗ ✓ ✓ ✗ MathQA ✓ ✗ ✗ ✗ MBXP ✓ ✓ ✗
✗ ClassEval ✓ ✗ ✗ ✗ MultiPL-E ✓ ✓ ✗ ✗ AiXBench ✓ ✗ ✗ ✗ DS-1000 ✓ ✗ ✗ ✗ APPS ✓
✗ ✗ ✗ HumanEval-X ✓ ✓ ✓ ✗ XCodeEval ✓ ✓ ✓ ✗ CodeScope ✓ ✓ ✓ ✓

Table 2: Comparisons between our CodeScope and existing code evaluation
benchmarks.

##  3 The CodeScope Benchmark

CodeScope evaluates the performance of LLMs on both code understanding and
code generation capabilities.

###  3.1 Code Understanding

The code understanding tasks primarily measure LLMs’ proficiency in
understanding and mastery of code. Among the tasks are code summarization,
which evaluates whether the model can succinctly summarize the core
functionality and intent of the code; code smell, which evaluates the model’s
ability to identify potential programming issues and poor practices; code
review, which measures the model’s ability to judge the overall quality of the
code, its coding style, and potential errors; and automated testing, which
evaluates the model’s ability to understand programming logic, the accuracy of
tracking and analyzing data flows, and its insight into the internal
mechanisms during the code execution process.

Figure 1: The task definition of the 4 code understanding tasks.

####  3.1.1 Code Summarization

Overview Code summarization aims to summarize the function and intent of
source code into concise natural language descriptions, assisting developers
in quickly grasping the functionality and behavior of the code. Successful
code summarization depends not only on the model’s recognition and
understanding of the structure and intent of the code, but also requires
accurately capturing the interactions among its components and how they
collaboratively achieve specific functions.

In CodeScope, we define code summarization as a task that transforms from a
programming language to natural language (PL-to-NL), where the input is a
snippet of source code, and the output is the corresponding functional
description of that code. Figure 1 outlines the primary workflow of LLM when
performing the code summarization task. It is worth noting that each
programming language has its own distinct syntax, semantics, and usage
patterns. Evaluating code summarization capabilities based solely on a handful
of mainstream programming languages does not fully demonstrate its
comprehensive understanding of various languages. As such, we conduct code
summarization across 43 high-resource and low-resource programming languages
to thoroughly evaluate LLMs’ understanding of programming languages, and
evaluate the generalizability of LLMs to ensure that they are not overfitted
to specific programming languages.

Dataset According to the TIOBE Programming Community Index111The Rosetta Code
programming website aims to demonstrate the differences in usage between
languages by providing multilingual code solutions to a given set of tasks.,
we collect code summarization data for the 43 most popular programming
languages from the Rosetta Code programming website. To our knowledge, this is
the most comprehensive code summarization dataset spanning multiple
programming languages. In order to keep the difficulty of test samples
consistent across different languages to control the fairness of the
evaluation, we collect code summarization data for the 43 most popular
programming languages from the Rosetta Code programming website. To our
knowledge, this is the most comprehensive code summarization dataset spanning
multiple programming languages. In order to keep the difficulty of test
samples consistent across different languages to control the fairness of the
evaluation, we select 170 high-quality programming tasks and extract 4,841
code samples, which prioritize tasks covering a wider range of programming
languages, aiming to ensure an equivalent level of task difficulty across all
languages. To preserve balance in our dataset, we ensure that the number of
samples for each programming language is at least 30. We revise and craft a
reference summarization for each sample based on the task description, sample
code explanation, and source code. As a result, each sample includes the task
description, programming language, source code, and its reference
summarization.

Metrics We employ four metrics, BLEU (Papineni et al., 2002), METEOR (Banerjee
and Lavie, 2005), ROUGE (Lin and Hovy, 2003), and BERTScore (Zhang et al.,
2020), for comprehensively evaluating code summarization. Specifically, BLEU
measures the degree of overlap between the predicted output and the reference
output, ranging from 1-gram to 4-gram. METEOR considers the alignment of
vocabulary, synonym matching, and morphological variations and also combines
both recall and precision to assess the similarity between the predicted and
reference outputs. ROUGE primarily evaluates based on the longest common
subsequence found between the generated summaries and the human-written
reference ones. BERTScore leverages a pre-trained BERT model and its deep
contextual representations to gauge the semantic similarity between the
generated and reference texts.

####  3.1.2 Code Smell

Overview Code smells refer to situations where the source code functions
correctly, but its structure or implementation may indicate potential design
flaws and quality issues, even violating basic software development
principles. Fowler (1999) first proposed the concept of code smells,
introducing 22 types of code smells that violate design rules, along with
their features and impacts. It is important to emphasize that, unlike code
bugs or errors, code smells are symptoms of poor design patterns affecting the
quality of the software system without directly affecting its functionality or
technical correctness. However, these code smells can lead to reduced system
performance and increased risk of future failures.

In CodeScope, we treat the code smell detection task as a multi-class
classification problem, aiming to evaluate the proficiency of LLMs in code
understanding and their ability to identify code smells. The input to the task
is the smelly code snippet and the source code where it appears, and five
potential code smell categories; the output is the specific code smell
category in the snippet. The code smell detection process of LLMs is shown in
Figure 1. Accurately identifying code smells is a challenging code
understanding task Madeyski and Lewowski (2023), as developers are typically
required to have a thorough analysis and understanding of source code from
both global and local perspectives.

Dataset Madeyski and Lewowski (2023) provided a large dataset of code smells
identified by experienced developers from industry-relevant open source Java
projects. Slivka et al. (2023) proposed a systematic approach for manually
annotating code smells and collected a dataset of C# code smells from active
projects on GitHub. In CodeScope, we integrate the above Java and C# datasets,
comprising three class-level and two method-level code smell categories. We
select 100 representative samples for each language. We manually review each
sample to guarantee the dataset’s balance and high quality, ensuring an equal
number of samples for each code smell type. Each sample includes source code,
smelly code snippets, and potential code smell options.

Metrics We adopt common classification evaluation metrics for the five-class
classification task of code smells, including accuracy, precision, recall, and
weighted F1 score. Each class is treated as a positive class, with metrics
weighted based on the sample size of each class.

####  3.1.3 Code Review

Overview Code review refers to the process by which developers systematically
review source code written by other developers to identify and fix potential
errors while ensuring that the code conforms to the team’s coding standards.
This process is critical to the quality assurance of software development and
open-source projects, preventing defects or errors from having serious
consequences in subsequent development or production environments. In
CodeScope, we employ two reviewer-perspective tasks of quality estimation and
code review generation from Li et al. (2022b) to evaluate the code review
capabilities of large language models. Code refinement is not used because the
task mainly aims at code committers rather than reviewers.

The quality estimation task is a binary classification problem that predicts
whether code changes require further comments or suggestions. The input is
code changes, and the output is comments required or no comments required. The
code review generation task is a sequence generation problem that aims to
generate comments or suggestions for code changes that need refinement. The
input is still the code change, and the output is the predicted natural
language comment. Combining these two tasks can automate code reviews,
aligning closely with actual review workflows. Figure 1 illustrates the
definition of code review. We believe that LLMs hold the potential to automate
code reviews to reduce the burden on reviewers significantly.

Dataset We use the code quality estimation dataset released by Li et al.
(2022b), which includes real-world code changes, quality estimation, and
review comment data in Github, covering nine commonly used programming
languages, including Python, Java, Go, C++, Javascript, C, C#, PHP, and Ruby.
In order to ensure the balance and high quality of the dataset, we filter each
language according to the code length and select 200 representative samples
for each language.

Metrics Quality estimation as a binary classification task, we follow the
evaluation metrics used by Li et al. (2022b), which include accuracy,
precision, recall, and the F1 score. Under this definition, codes that require
further annotation or revision are considered positive categories. For the
evaluation of comment generation, we employ the BLEU, ROUGE, and BERTScore as
our evaluation criteria.

####  3.1.4 Automated Testing

Overview Automated testing refers to executing test cases that are
automatically generated through specific tools or scripts, aiming to quickly
and comprehensively verify code functionality and performance without human
intervention to ensure that it meets expected requirements. Automatically
generated test cases play a key role in identifying and locating defects and
errors in the code, which can effectively ensure the stability and reliability
of the code, while significantly reducing time and labor costs.

The automated testing task of CodeScope requires LLMs to generate test cases
for a given code to achieve optimal pass rate and test coverage. The task
input is the problem description and the corresponding code solution, and the
output is multiple test cases. Figure 1 illustrates the definition of
automated testing. However, automated test case generation is a challenging
task. LLMs must deeply understand the problem and identify potential boundary
conditions and constraints to craft test cases that can capture complex edge
scenarios and abnormal inputs. Therefore, using LLMs for automated testing is
an effective way to evaluate their comprehensive capabilities and is also
expected to reduce developer effort significantly.

Dataset We constructed an automated testing dataset using samples of four
programming languages Python, Java, C, and C++ in the dataset crawled from
Codeforces. Each sample consists of a problem description, time and memory
limits, input and output specifications, input and output samples with
explanations, the source code solution, and multiple test cases. To ensure the
high quality of our dataset, we manually verified and selected 100
representative samples from each language, each exhibiting a 100% pass rate,
line coverage, and branch coverage.

Given that the limited token count of LLMs can critically constrain the
generation of effective test cases, we limited the number of test cases
generated by LLMs to 5 to ensure fairness of evaluation. To establish a
benchmark for manual written test cases, we randomly selected 5 test cases
from each sample and tested their pass rate, line coverage, and branch
coverage on the source code solution. To reduce the bias caused by random
selection, we repeated this process 5 times and took the average of the
results.

Metrics We adopted three metrics to evaluate the quality of test cases
generated by LLMs. The pass rate refers to the percentage of test cases that
successfully pass the test, where the test case is defined as passing if it
meets the format requirements, executes correctly, and produces results
consistent with the expected output. Line coverage refers to the percentage of
code lines covered by test cases as a percentage of the total number of code
lines. Branch coverage is the percentage of branches that have already been
executed out of all possible branches in the code. It is worth noting that
high pass rate and test coverage typically indicate more robust code, which in
turn bolsters our confidence in the quality of the software. To further
optimize the evaluation process, we developed specific execution environments
for the four programming languages and integrated fully automated code
coverage calculation tools.

###  3.2 Code Generation

Compared to code understanding, code generation will focus more on LLMs’
capacity to create target code based on various requirements. In this section,
we mainly developed four subtasks. Program synthesis (Correctness) evaluates
the ability of the LLMs to generate correct code according to the given NL-
description. Code translation (Compatibility) examines whether the LLMs can
maintain functional consistency when translating between different programming
languages. Code repair (Maintainability) focuses on the LLMs’ ability to
identify and fix errors automatically. Code optimization (Efficiency)
evaluates the LLMs’ capability to identify and improve the efficiency of the
code. Additionally, we introduce a new dataset called Codeforces4LLM with two
difficulty levels for the four tasks we mentioned.

Figure 2: The task definition of the 4 code generation tasks.

####  3.2.1 Program Synthesis

Overview The objective of program synthesis is to generate expected code
solutions based on the natural language description of the task. This task has
been widely used to evaluate the code generation capabilities of LLMs
(Chowdhery et al., 2023; Chen et al., 2021; Austin et al., 2021; Khan et al.,
2023). Program synthesis not only requires LLMs to have strong logical
reasoning and problem-solving abilities, but also examines the ability of LLMs
to accurately express logical structures into concrete code at a deeper level.
LLMs are generally considered to have a high level of code generation
capability when they can accurately understand the intent of various complex
problems and generate executable code that meets the expected requirements.

CodeScope defines program synthesis as a generative task from natural language
to programming language (NL-to-PL). The input is a programming scenario
described in natural language, including sample inputs and outputs of the
problem, while the expected output is code that can solve the corresponding
problem. As shown in Figure 2, we demonstrate the process of LLMs in the
program synthesis task. Given that LLMs should demonstrate excellent
generative capabilities in various programming languages to meet the needs of
different programming communities, we have designed the broadest range of
execution-based program synthesis tasks to date, covering 14 high-resource and
low-resource programming languages. Notably, unlike existing benchmarks (Chen
et al., 2021; Austin et al., 2021) that provide clear and straightforward
descriptions of programming requirements, we comprehensively evaluate the
ability of LLMs in solving real-world coding problems by increasing task
difficulty. Therefore, only the real-world scenarios of the task are described
in our data, without specifying specific solutions or providing abstract
programming problems. This requires LLMs not only to understand the task
description, but also to independently design or select appropriate
programming algorithms and generate corresponding solutions.

Dataset We constructed the Codeforces4LLM dataset by collecting data from
Codeforce222https://codeforces.com, a popular online algorithm competition
platform. According to the TIOBE Programming Community Index333TIOBE
Programming Community Index is a metric of the popularity of programming
languages., we collected problem descriptions and correct submissions of
corresponding problems in 14 different programming languages, including C++,
Java, Python, C, C#, Ruby, Delphi, Go, Javascript, Kotlin, PHP, D, Perl, and
Rust. This constitutes the dataset to date covering the broadest variety of
programming languages in code generation tasks.

According to the official difficulty standards of the Codeforces platform, we
set two difficulty levels for each programming language: Easy ([800, 1600))
and Hard ([1600, 2800)). We selected 30 problems for each difficulty level and
ensured that these problems had solutions that passed all test cases.

To ensure the quality of the dataset, we excluded problems with fewer than 10
test cases, as well as non-deterministic problems with multiple potential
outputs for the same test input. When selecting ground truth, we performed
execution validation and excluded submissions that failed to compile in
different environments due to environmental differences, as well as
submissions for brute force solutions exceeding 5k tokens.

Metrics Since code only indeed generates value when it is successfully
executed and meets the expected requirements, we adopt execution-based metrics
Pass@k as the only way to evaluate the code generated by LLM. This implies
that for a given programming problem, if any one of the k generation attempts
satisfies the pre-defined unit test evaluations, we deem the problem to have
been successfully solved. To facilitate this evaluation metric, we developed a
multilingual integrated execution testing environment, called MultiCodeEngine,
which can support 47 compiler/interpreter versions across the 14 programming
languages involved in code generation, partially based on ExecEval (Khan et
al., 2023).

####  3.2.2 Code Translation

Overview The objective of code translation is to convert source code from one
programming language to another, promoting software compatibility across
different platforms, enhancing code maintainability, and supporting the
maintenance and modernization of early software systems. This process requires
LLMs not only to achieve functional equivalence in execution-based evaluation,
but also to identify dependencies and edge cases in different programming
languages to meet strict programming syntax specifications. We consider LLM to
have excellent code-to-code generation capabilities when it can accurately
understand the syntax and semantics of source code and generate executable
code that is fully functionally consistent with source code in other languages
that follow different programming paradigms.

CodeScope defines code translation as a generation task from one programming
language to another (PL-to-PL). Its input includes the source code of a
specific language and the designation of the target programming language, and
the expected output is the corresponding and functionally consistent code in
the target programming language. As shown in Figure 2, we show the main
workflow of LLM in the code translation task. Unlike program synthesis, we
omit the problem description, sample inputs and outputs, and other auxiliary
information, aiming to force LLMs to focus on source code to evaluate their
PL-to-PL code generation capabilities. We followed the same programming
language coverage and task difficulty settings as program synthesis tasks, as
detailed in Section 3.2.1.

Dataset We utilized the Codeforces4LLM dataset constructed in the program
synthesis task. Given that evaluating all permutation combinations across 14
programming languages incurs excessive overhead, we limited the number of code
pairs to 15 at each difficulty level. Furthermore, we maintained the integrity
of the remaining data within the Codeforces4LLM dataset.

Metrics We adopt the same metrics pass@k and execution environment
MultiCodeEngine for code translation tasks as we do for program synthesis. For
more details, please refer to Section 3.2.1.

####  3.2.3 Code Repair

Overview The objective of code repair is to identify and correct errors or
defects in source code to ensure that the code performs correctly and meets
expected functional requirements. In programming practice, code repair can
significantly reduce the burden of program development and maintenance. We
consider LLM to have excellent code repair capabilities when it can accurately
understand the code intent and identify errors or defects in the code, while
generating effective repair solutions that are consistent with the expected
functionality.

CodeScope defines code repair as a generation task from natural language and
programming language to programming language (NL&PL-to-PL). Its input includes
the error code snippet, the corresponding problem description, and the error
message returned by the compiler/interpreter, while the expected output is the
corrected code that solves the corresponding problem. As shown in Figure 2, we
show the main workflow of LLM in the code repair task. Notably, code repair
task integrates a series of complex and diverse challenges, including fine-
grained code understanding, problem diagnosis across NL and PL, and
formulating effective repair strategies. Compared with program synthesis and
code translation tasks, code repair more comprehensively tests LLMs’ analysis
and code generation capabilities in multiple dimensions. We followed the same
programming language coverage and task difficulty settings as program
synthesis tasks, as detailed in Section 3.2.1.

Dataset We expanded the Codeforces4LLM dataset by collecting additional
incorrect code submissions for each problem and executed them in the
MultiCodeEngine to obtain code error information. Furthermore, we maintained
the integrity of the remaining data within the Codeforces4LLM dataset.

Metrics Given that the code repair task primarily assesses the proficiency of
LLMs in identifying and rectifying specific errors/bugs, we employ the
Debugging Success Rate@K (DSR@K) metric proposed by (Yan et al., 2023) to
evaluate the execution-based code repair capabilities of LLMs. DSR@K means
that a code sample that initially does not match expectations can output the
expected execution result after K rounds of debugging, and the code sample is
deemed to be successfully repaired. DSR@K is the percentage of the samples
that successfully execute and produce the expected results among all samples.
Importantly, DSR@K requires interaction between the LLMs and the code
execution environment to obtain error feedback, so we use the MultiCodeEngine
described in Section 3.2.1 to implement interaction with the execution
environment.

####  3.2.4 Code Optimization

Overview Code optimization aims to enhance the time or space complexity of a
program without altering its original functionality. The goal is to improve
execution efficiency, thereby making better use of time and hardware
resources. There are many ways to achieve efficiency optimization, including
algorithmic optimization, compiler optimization, and code syntax structure
optimization. Code optimization in CodeScope pays special attention to how to
enhance code efficiency from the perspective of algorithms, that is, when
there are multiple algorithmic solutions to a problem, how to design or select
more efficient algorithms to markedly upgrade the execution performance of the
code.

The objective of code optimization is distinct from that of program synthesis.
While code optimization aims to enable LLMs to generate more efficient code,
program synthesis is primarily concerned with whether the code generated by
LLMs can be executed smoothly and achieve the expected results. To the best of
our knowledge, CodeScope is the first work to explore the capabilities of LLMs
in code optimization. In this task, the input includes the problem
description, the source code awaiting optimization, the specified programming
language, and representative test case inputs and outputs. The output is the
optimized code by LLM. The workflow of code optimization is illustrated in
Figure 2. LLM ensures that the correct code is generated (that is, it aligns
with the expected output of all test cases) and produces code that executes
more efficiently (that is, it achieves higher efficiency in terms of time or
memory). Given these intricate requirements, code optimization is undeniably a
formidable challenge. However, we believe that evaluating LLMs’ advanced code
understanding and generation capability is critical, as effective code
optimization can significantly reduce resource consumption and enhance the
user experience.

Dataset We screened Codeforces4LLM to construct the code optimization dataset,
specifically selecting 30 programming tasks in each of the four prevalent
programming languages Python 3, C#, C, and C++. To ensure that each task had
diverse solutions from an algorithmic and source code syntactic perspective,
we evaluated the performance of different solutions across various test cases.
Therefore, we selected problem samples with more than 10 correct answer
submissions and over 20 test cases.

In addition, we inspect the execution time and memory usage of code
submissions for each problem in its corresponding test cases. Based on the
inspection results, we identify the code submission samples with the longest
execution time and highest memory usage for each problem. These samples are
deemed to possess considerable optimization potential in terms of time and
memory efficiency, and we calibrate the time and memory efficiency baseline
for each problem. In summary, each data sample includes the problem
description, the type of programming language, the code solution flagged for
optimization potential concerning execution time and memory usage, and an
array of test cases pertinent to the problem.

Metrics Given that code optimization primarily focuses on evaluating the
proficiency of LLMs in identifying and optimizing inefficient code, we
introduce a novel metric, Opt@K, to quantify this aptitude. Opt@K is based on
the assumption that a code sample with potential for efficiency optimization,
if the execution efficiency of any optimized code sample exceeds the
efficiency of the original sample in K optimization attempts, this code sample
will be regarded as successfully optimized. Hence, Opt@K is the percentage of
successfully optimized samples to the total number of samples.

In order to accurately measure the execution efficiency of code samples, we
record their execution time and memory consumption during the code execution
process. Specifically, we define the actual execution time of the code as the
cumulative CPU time in system mode and user mode, and adopt the Unique Set
Size (USS) to measure the actual memory usage of the code solutions.

##  4 Evaluation and Analysis

In this section, we will detail 8 popular large language models and their
performance on various tasks, and analyze the experimental results. We report
in detail the specific configuration information of baseline LLMs, the
parameter setting of the experiment, and the hardware information used for
inference in Appendix A.2. In addition, we report the prompts designed for
each task in the case study in Appendix A.4.

###  4.1 Baselines

Closed-sourced LLMs In recent years, LLMs built on the Transformer
architecture have reached significant milestones in the field of NLP. GPT-4
(OpenAI, 2023), in particular, stands at the forefront of this progression. It
not only generates semantically coherent and logically rigorous natural
language text, but also exhibits excellent performance on code understanding
and generation tasks. Notably, GPT-3.5, building upon GPT-3 (Patel et al.,
2023), adopts the “Reinforcement Learning from Human Feedback” mechanism. This
addition not only enhances the fluency of user interaction but also boosts the
understanding and generation capabilities of natural language and code.
Furthermore, PaLM 2 (Anil et al., 2023), with its 340 billion parameters, has
been trained on 3.6 trillion tokens. It encompasses training in 20 programming
languages, significantly enhancing its capabilities in generating and
debugging code.

Open-sourced LLMs LLaMA 2 (Touvron et al., 2023) is a highly regarded open-
source regression large language model, trained on 2 trillion tokens and
expanded its context length to 4096 tokens. It employs a grouped query
attention mechanism to enhance efficiency and utilizes reinforcement learning
with human feedback (RLHF), not only aligning more closely with human
preferences but also significantly boosting model performance and security.
Additionally, the popular Vicuna (Chiang et al., 2023) fine-tunes LLaMA using
a dialogue corpus, aiming to process dialogue text with greater precision. As
a result, Vicuna possesses over 90% of ChatGPT’s capability to comprehend and
generate natural language dialogues.

Open-sourced Code LLMs StarCoder (Li et al., 2023b), a widely-adopted open-
source Code LLM, is trained on a corpus of 1 trillion tokens from over 80
programming languages and boasts a context length of 8,192 tokens. WizardCoder
(Luo et al., 2023), a fine-grained instruction evolution and incorporation of
code debugging features as well as space-time complexity constraints,
leveraged a new training dataset constructed from Code Alpaca to fine-tune
StarCoder, which makes WizardCoder perform well in program synthesis tasks.
The recently released Code LLaMA (Rozière et al., 2023) is trained further on
a specific code dataset based on LLaMA 2, capable of stably generating up to
100K context tokens and demonstrating outstanding coding performance.

###  4.2 Code Understanding

####  4.2.1 Code Summarization

Table LABEL:table:code_summarization_result shows the comprehensive
performance comparison of various LLMs in code summarization. We report
overall performance by calculating the arithmetic mean of four text
similarity-based metrics. The results show that GPT-4 outperforms other LLMs
on all evaluation metrics, demonstrating its excellent ability to generate
coherent and highly contextual summaries. The performance of GPT-3.5 follows
closely behind. While the other LLMs perform unevenly across these metrics,
they generally fail to reach the level of the GPT series, especially in terms
of semantic understanding and grammatical correctness.

Although PaLM 2 performs competitively on lexical overlap-based metrics (e.g.,
BLEU and ROUGE), it performs poorly on metrics that capture deep semantics
(e.g., METEOR and BERTScore). As shown in the case analysis provided in
Appendix Table 32, this may be because PaLM 2 annotates line by line due to
confusing concepts when generating summaries, thus affecting its overall
performance.

It is worth noting that while all LLMs perform better overall on BERTScore,
they generally score lower on the BLEU metric, which means that while they
perform well on grasping the overall semantics of the code, there remains
significant room for improvement in terms of precise wording and the use of
specific terminology. To provide further insight into our approach to code
summarization, we provide a detailed case analysis in Appendix Table 30. In
addition, we also report the performance of individual LLMs when dealing with
different programming languages and code length variations to provide a more
comprehensive analysis in Appendix Table 6.

####  4.2.2 Code Smell

As shown in Table LABEL:table:code_smell_result, various LLMs show significant
performance differences on the code smell task. We find that WizardCoder
achieves significantly leading performance among all tested LLMs, especially
in processing C# language, while its performance in Java is relatively weak.
LLaMA 2 shows similar performance trends in Java and C# languages. In
contrast, GPT-3.5 and GPT-4 perform mediocrely on code smell classification in
Java and C# languages, but they show more balanced capabilities across
different programming languages. Specifically, Vicuna performs significantly
poorly on the Java code smell classification task, mainly due to its erroneous
classification of up to 28.5% of samples as non-existent code smell, which
affects its overall performance. The overall performance of other LLMs is
generally moderate. However, StarCoder performs extremely poorly on Java code
smell classification. We observe that this is mainly due to the fact that it
misclassifies up to 96% of samples as a large class code smell, resulting in a
significant decline in its performance. To present our code smell task more
clearly, we provide a detailed code smell case study in Appendix Table 34.

####  4.2.3 Code Review

Table LABEL:table:code_review_result shows the performance of various LLMs on
the code review task. We find that the performance differences among different
LLMs are not significant. Among them, Code LLaMA and Vicuna show higher
accuracy on the quality estimation subtask, which may be attributed to their
stronger capability to identify code that requires comments. However, they are
deficient in generating detailed and balanced comments. GPT-4, while not the
best overall performance, maintained a good balance across all metrics,
reflecting its comprehensive performance. PaLM 2 performs best on BLEU,
indicating that it excels in imitating human comments, but its deficiency in
quality estimation affects its overall score. GPT-3.5 stands out in generating
content that is semantically close to human comments, while its tendency to
consider that most samples do not require comments in quality estimation leads
to a lower weighted F1-score. Therefore, GPT-3.5 still has room for
improvement in ensuring comment relevance. StarCoder and WizardCoder show
relatively weak performance. This is mainly due to their tendency to judge
that samples do not require further review, while their review comments are
too verbose, containing much code and less natural language description. To
further demonstrate our code review task, we provide a detailed case study in
Appendix Table 30.

####  4.2.4 Automated Testing

Table LABEL:table:automated_testing_result compares the performance of various
LLMs on automated testing task. We observe that while LLMs excel in line
coverage (LC) and branch coverage (BC) of code, sometimes even surpassing
human baseline, they lag significantly behind humans in test case pass rate
(PR). This indicates that while LLMs can effectively explore and cover
different locations of code, they still fall short in generating precise and
reliable test cases. Specifically, GPT-3.5 performs best among all tested
LLMs, especially demonstrating robust LC and BC in Python and Java language
testing. However, its PR is significantly lower than the human baseline,
indicating that while GPT-3.5 excels in comprehensive coverage of code
structures, it struggles to generate consistently correct test cases. Other
LLMs, while varying in performance, generally have shortcomings in PR and
require further optimization. To comprehensively present our automated testing
task, we provide a detailed case study of automated testing in Appendix Table
37.

###  4.3 Code Generation

####  4.3.1 Program Synthesis

As shown in Table 3, the various LLMs exhibit significant performance
differences in program synthesis across different programming languages. GPT-4
performs exceptionally well at different difficulty levels, significantly
outperforming other models in Pass@5 metrics with scores of 58.57% and 12.01%,
respectively. Although the overall performance of GPT-3.5 is lower than GPT-4,
it still shows considerable ability in solving easy problems. In contrast,
other LLMs perform poorly on easy problems and can barely provide correct
answers to hard problems, exposing significant shortcomings in understanding
and generating complex code. In addition, setting different problem
difficulties can effectively distinguish the advantages and limitations of
different LLMs, demonstrating the importance of reasonably setting task
difficulty when evaluating LLMs. In the Appendix, we provide a detailed case
study in Tables 38 to better understand our program synthesis task.

Model Easy Hard Overall GPT-4 58.57 12.01 35.29 GPT-3.5 39.29 4.96 22.13 Code
LLaMA 7.14 0.26 3.70 WizardCoder 5.95 0.26 3.11 PaLM 2 3.81 0.78 2.30 LLaMA 2
1.43 0.00 0.72 StarCoder 0.95 0.00 0.48 Vicuna 0.71 0.00 0.36

Table 3: Performance comparison of LLMs in program synthesis at varying
difficulty levels, evaluated using Pass@5 testing. Easy and Hard categories
refer to the complexity of the programming tasks.

####  4.3.2 Code Translation

We report the performance of various LLMs on the code translation task at
different difficulty levels in Table LABEL:table:code_translation_result, and
provide detailed translation results for various LLMs at different difficulty
levels in Appendix Tables 12 to 27. In the PL-to-PL task, GPT-4 and GPT-3.5
significantly outperform other models by 31.29% and 21.37%, respectively, in
the Pass@1 metric, due to their advanced training methods and high-quality
training data. In contrast, other LLMs perform significantly inferior, with
none exceeding 7% in the Pass@1 metric. Notably, the performance of LLaMA 2
and Vicuna dropped significantly in the PL-to-PL generation task, with almost
all attempts failing. We speculate that this is due to the fact that the vast
majority of their training corpus is natural language text. To further
demonstrate our code translation task, we provide a case study of code
translation in Appendix Table 39.

####  4.3.3 Code Repair

Table LABEL:table:code_repair_result compares various LLMs’ performance on the
code repair task at different difficulty levels. GPT-4 achieves an overall
success rate of 29.27% in single debugging attempts, which is significantly
better than other LLMs, especially in solving easy problems. In contrast,
GPT-3.5 achieves an overall single debugging success rate of only 13.33%, less
than half that of GPT-4. The performance of other LLMs lags significantly
behind GPT-4 in both easy and hard tasks, indicating their limited assistance
value in practical programming scenarios. Compared to program synthesis, we
observe that repairing existing buggy code is easier for LLMs than generating
solutions from scratch, especially for non-GPT series LLMs. In order to
present the performance of LLMs on code repair in more detail, we list their
specific performance at each difficulty level in Tables 28 and 29, and provide
specific code repair case study in Appendix Tables 40 and 41.

####  4.3.4 Code Optimization

As shown in Table LABEL:table:code_optimization_result, GPT-4 exhibits the
best performance among various LLMs in the comprehensive performance
evaluation of code optimization, especially in saving execution time. Although
GPT-4 does not consistently rank first in memory optimization, it performs
robustly across various programming languages. GPT-3.5 also performs well,
achieving a significant improvement of 76.67% in memory optimization for C
language. However, it fails to show improvement in memory optimization for C#
language, indicating some limitations. It is worth noting that WizardCoder and
Code LLaMA show competitive performance compared to GPT series models in code
optimization, indicating their cognizance of memory usage and time efficiency
during code execution. Although other LLMs show some optimization potential,
they have a large gap compared to the leading models and perform
inconsistently across different languages and optimization dimensions. In
particular, we observe that LLMs achieve the best results in optimizing Python
language, but the weakest performance in optimization of C language,
especially in reducing execution time. This may be due to the low-level
features and strict requirements for details of C language, such as precise
memory management and a deep understanding of pointer operations. Therefore,
LLMs still face significant challenges in execution time optimization for C
language. In addition, we find that most successful optimization cases are
only limited to the syntactic level, where LLMs tend to choose syntactic
improvement schemes. In contrast, they are not mature enough to generate deep-
level changes affecting algorithm efficiency, such as introducing more optimal
algorithms or data structures. To present our code optimization process and
related findings more comprehensively, we provide two case studies of code
optimization in Appendix Tables 42 and 44 of the .

##  5 Discussion

Ranking CodeScope CodeScope CodeScope HumanEval Pass@1 MBPP Pass@1
(Understanding) (Generation) (Overall) 1 WizardCoder (50.00) GPT-4 (31.01)
GPT-4 (38.97) GPT-4 (67.0) GPT-4 (61.8) 2 GPT-3.5 (49.20) GPT-3.5 (20.82)
GPT-3.5 (35.01) WizardCoder (57.3) Code LLaMA (57.0) 3 LLaMA 2 (48.51)
WizardCoder (9.74) WizardCoder (29.87) GPT-3.5 (48.1) GPT-3.5 (52.2) 4 Code
LLaMA (47.99) Code LLaMA (8.02) Code LLaMA (28.01) Code LLaMA (41.5)
WizardCoder (51.8) 5 PaLM 2 (47.28) PaLM 2 (5.59) PaLM 2 (26.44) PaLM 2 (37.6)
PaLM 2 (50.00) 6 GPT-4 (46.93) StarCoder (3.90) LLaMA 2 (25.51) StarCoder
(33.6) LLaMA 2 (45.4) 7 Vicuna (46.54) Vicuna (2.60) Vicuna (24.57) LLaMA 2
(30.5) StarCoder (43.6) 8 StarCoder (44.04) LLaMA 2 (2.50) StarCoder (23.97)
Vicuna (15.2) Vicuna (22.4)

Table 4: Comparison of results of 8 baseline models on CodeScope, HumanEval
and MBPP benchmarks.

Table 4 presents a detailed comparison of the performance of eight widely-used
LLMs across the CodeScope, HumanEval, and MBPP benchmarks. Unlike HumanEval
and MBPP, which primarily focus on a single aspect of evaluation, the
CodeScope benchmark offers a more balanced and comprehensive framework for
evaluating LLMs from code understanding and code generation perspectives.
CodeScope not only evaluates the capability of LLMs to generate new code
snippets but also tests their proficiency in understanding and analyzing
existing complex code, reflecting the diverse challenges encountered in real-
world programming scenarios.

In our evaluation of CodeScope (Understanding), we primarily focused on
evaluating the LLM’s proficiency in interpreting and analyzing code. By
calculating the average performance of each model across four distinct code
understanding tasks, we determined their overall scores in this domain, as
shown in Table 4. Notably, the rankings of these LLMs in code understanding
significantly diverge from their positions in HumanEval and MBPP. For
instance, GPT-4, while leading in HumanEval and MBPP, only ranks sixth in
CodeScope (Understanding). This demonstrates that outstanding performance in
model code generation tasks is not directly equivalent to the ability to
develop a deep understanding of complex code. CodeScope (Understanding)
reveals details that existing benchmarks fail to capture, proving the
necessity of adopting a comprehensive evaluation approach.

In evaluating CodeScope (Generation), we maintained the same overall score
calculation method. As shown in Table 4, we noted significant differences in
rankings. GPT-4 and GPT-3.5 showed significant advantages in code generation,
which were not observed in the evaluations of HumanEval and MBPP. We speculate
that this inconsistency may arise from two factors. Firstly, the diversified
design of CodeScope (Generation) emphasizes evaluating general abilities in
multi-objective, multilingual code generation. Unlike HumanEval and MBPP,
which are limited to NL-to-PL task, CodeScope covers all three scenarios of
NL-to-PL, PL-to-PL, and NL&PL-to-PL in evaluating the correctness, quality,
and actual execution efficiency of the generated code, as well as the
adaptability of LLMs across various programming languages. Secondly, CodeScope
(Generation) presents a higher overall question complexity, offering
challenges across a spectrum of difficulty levels. In stark contrast, both
HumanEval and MBPP are composed of predefined and quite simple programming
problems. For example, the average number of tokens in solutions is 53.8 and
57.6 for HumanEval and MBPP, respectively, compared to a significantly higher
501.8 in CodeScope (Generation). Therefore, some LLMs that perform well in
HumanEval and MBPP, such as WizardCoder, do not perform equally well when
faced with more complex and diverse programming tasks in CodeScope
(Generation).

In evaluating CodeScope (Overall), we observed inconsistencies in the rankings
of LLMs across the CodeScope, HumanEval, and MBPP benchmarks. This finding
emphasizes the advantages of CodeScope in terms of its breadth and challenge.
CodeScope not only focuses on code generation capabilities but also
significantly emphasizes code understanding, aligning more closely with the
complexities of real-world programming scenarios. In addition, by introducing
multilingual, multi-dimensional, multi-task, and execution-based evaluation
strategies, CodeScope increases the difficulty and diversity of the
evaluation. It better simulates the actual programming environment, providing
a more comprehensive and fine-grained framework for evaluating the
comprehensive coding capabilities of LLMs.

##  6 Conclusion

We introduce CodeScope, the first comprehensive benchmark to evaluate LLMs in
coding tasks. CodeScope encompasses 43 programming languages, 8 coding tasks,
and 4 evaluation dimensions, employing a fine-grained, execution-based
evaluation methodology. Through detailed evaluation and analysis of 8
mainstream LLMs using CodeScope, we uncover their strengths and limitations
across various tasks and environments. Moreover, by contrasting CodeScope with
other benchmarks, we demonstrate the necessity of CodeScope in focusing on
real-world programming scenarios across multitasking, multilingual, and
multidimensional evaluation frameworks. In summary, we provide a comprehensive
resource, tool, and benchmark for evaluating LLMs in code understanding and
generation capabilities, aiming to drive future related research.

In future work, we will focus on further augmenting LLMs’ advanced
capabilities in processing and generating complex code. Future research can
progress along two distinct trajectories. Firstly, enhance the programming
capabilities of LLMs to directly solve various challenging problems, aiming to
achieve over 90% performance level on CodeScope. Secondly, explore using
autonomous agents to achieve a more effective collaborative division of labor,
which could help solve complex programming challenges more efficiently.
Through these avenues, we expect to drive the expansion of the frontiers in
the domain of code intelligence with LLMs.

## Limitations

Data independence and fairness are paramount when evaluating large language
models through benchmarks. However, data leakage is a likely problem for
benchmarks for evaluating large language models. While data leakage is
considered an issue that hinders the evaluation of models’ generalization
ability, in this paper, we aim to re-examine the legitimacy and validity of
this issue from the following three perspectives:

Data memorization and recitation represent a unique form of knowledge
capability. Traditional model evaluation tends to pay more attention to the
model’s generalization ability, which is mainly based on the model’s scale and
the training data’s limitations. However, in the current large model
environment, although the model exhibits memorization and recitation when
dealing with vast pre-trained data (Carlini et al., 2019; Yan and Li, 2022),
this behavior actually reflects a special knowledge capability of LLMs. This
is not exactly equivalent to the natural generalization ability, but in some
situations, it can proficiently aid humans in addressing real-world
challenges. Therefore, the unique ability of data memorization and recitation
still has evaluation value.

Constructing a fully zero-leakage evaluation dataset is technically
unfeasible. Given the multitude of LLMs trained on various diverse pre-
training corpora, creating a test dataset that is genuinely independent and
completely untouched by any model is extremely difficult, especially when the
pre-trained data of many models remains closed-source. In addition, even if we
attempt to filter data based on timelines, the knowledge base of LLMs is
constantly evolving444https://platform.openai.com/docs/models/. A zero-leakage
dataset today might be accessible to some models in the future due to model
updates. To mitigate the risk of leakage, we constructed the CodeScope task
dataset using five independent data sources, aiming to minimize reliance on
any single source and diminish the risk of bias in evaluation results.

Furthermore, the community has two distinct ways of handling data leakage in
benchmark tests. On the one hand, most studies tend to ignore the risk of data
leakage, such as AGIEval (Zhong et al., 2023), a recent high-profile bilingual
standardized test evaluation benchmark, the multilingual, multimodal and
multilevel evaluation benchmark M3Exam (Zhang et al., 2023), and the
interdisciplinary comprehensive Chinese evaluation benchmark CMMLU (Li et al.,
2023a). Conversely, some recent benchmarks have recognized the problem of data
leakage, and they generally believe that this challenge is difficult to avoid
completely. For example, SciBench (Wang et al., 2023a), an evaluation
benchmark for complex scientific problems, and C-Eval (Huang et al., 2023), an
evaluation benchmark for multilevel and multi-discipline Chinese, strive to
gather data that is difficult to extract or convert into text to mitigate this
problem.

The ability to generalize downstream tasks beyond data memorization.
Typically, the pre-training of large language models (LLMs) relies on
unsupervised methods, and their performance in various downstream tasks covers
a wide range of scenarios (Li et al., 2023b; Wang et al., 2023b). Even though
LLMs might have encountered certain datasets during the pre-training phase,
the application of these datasets in downstream tasks often differs from the
scenarios during pre-training. Therefore, despite the potential data leakage,
we are essentially still evaluating the capabilities of LLMs to migrate and
generalize across different tasks, rather than just their data memorization
abilities.

While data leakage is an unavoidable challenge, we should have a broader and
more open-minded perspective when evaluating large language models. We must
also re-examine and redefine our evaluation criteria and methods to ensure
their appropriateness and accuracy.

## References

  * Ahmed and Devanbu (2022) Toufique Ahmed and Premkumar T. Devanbu. 2022.  Few-shot training llms for project-specific code-summarization.  In _37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022_ , pages 177:1–177:5. ACM.
  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023.  Palm 2 technical report.  _CoRR_ , abs/2305.10403.
  * Athiwaratkun et al. (2023) Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, and Ramesh Nallapati. 2023.  Multi-lingual evaluation of code generation models.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021.  Program synthesis with large language models.  _CoRR_ , abs/2108.07732.
  * Balog et al. (2017) Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017.  Deepcoder: Learning to write programs.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bastoul (2004) Cédric Bastoul. 2004.  Code generation in the polyhedral model is easier than you think.  In _13th International Conference on Parallel Architectures and Compilation Techniques (PACT 2004), 29 September - 3 October 2004, Antibes Juan-les-Pins, France_ , pages 7–16. IEEE Computer Society.
  * Bondhugula et al. (2008) Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008.  A practical automatic polyhedral parallelizer and locality optimizer.  In _Proceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation, Tucson, AZ, USA, June 7-13, 2008_ , pages 101–113. ACM.
  * Cáceres et al. (2017) Leslie Pérez Cáceres, Federico Pagnozzi, Alberto Franzin, and Thomas Stützle. 2017.  Automatic configuration of GCC using irace.  In _Artificial Evolution - 13th International Conference, Évolution Artificielle, EA 2017, Paris, France, October 25-27, 2017, Revised Selected Papers_ , volume 10764 of _Lecture Notes in Computer Science_ , pages 202–216. Springer.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA, August 14-16, 2019_ , pages 267–284. USENIX Association.
  * Cassano et al. (2022) Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2022.  Multipl-e: A scalable and extensible approach to benchmarking neural code generation.
  * Chandel et al. (2022) Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022.  Training and evaluating a jupyter notebook data science assistant.  _CoRR_ , abs/2201.12901.
  * Chen et al. (2008) Chun Chen, Jacqueline Chame, and Mary Hall. 2008.  Chill: A framework for composing high-level loop transformations.  Technical report, Citeseer.
  * Chen et al. (2016) Dehao Chen, David Xinliang Li, and Tipp Moseley. 2016.  Autofdo: automatic feedback-directed optimization for warehouse-scale applications.  In _Proceedings of the 2016 International Symposium on Code Generation and Optimization, CGO 2016, Barcelona, Spain, March 12-18, 2016_ , pages 12–23. ACM.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _CoRR_ , abs/2107.03374.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023\.  Palm: Scaling language modeling with pathways.  _J. Mach. Learn. Res._ , 24:240:1–240:113.
  * Clement et al. (2020) Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020.  Pymt5: multi-mode translation of natural language and python code with transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 9052–9065. Association for Computational Linguistics.
  * Das et al. (2019) Ananta Kumar Das, Shikhar Yadav, and Subhasish Dhal. 2019.  Detecting code smells using deep learning.  In _TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), Kochi, India, October 17-20, 2019_ , pages 2081–2086. IEEE.
  * Devlin et al. (2017) Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017.  Robustfill: Neural program learning under noisy I/O.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 990–998. PMLR.
  * Du et al. (2023) Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023.  Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation.  _CoRR_ , abs/2308.01861.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  Codebert: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 1536–1547. Association for Computational Linguistics.
  * Fowler (1999) Martin Fowler. 1999.  _Refactoring - Improving the Design of Existing Code_.  Addison Wesley object technology series. Addison-Wesley.
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023.  Incoder: A generative model for code infilling and synthesis.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Hao et al. (2022) Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei. 2022.  Aixbench: A code generation benchmark dataset.  _CoRR_ , abs/2206.13179.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021.  Measuring coding challenge competence with APPS.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _CoRR_ , abs/2009.03300.
  * Huang et al. (2022) Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin B. Clement, Nan Duan, and Jianfeng Gao. 2022.  Execution-based evaluation for data science code generation models.  _CoRR_ , abs/2211.09374.
  * Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _CoRR_ , abs/2305.08322.
  * Khan et al. (2023) Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Long Do, Weishi Wang, Md. Rizwan Parvez, and Shafiq R. Joty. 2023.  xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval.  _CoRR_ , abs/2303.03004.
  * Lai et al. (2023) Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. 2023.  DS-1000: A natural and reliable benchmark for data science code generation.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 18319–18345. PMLR.
  * Li et al. (2023a) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a.  CMMLU: measuring massive multitask language understanding in chinese.  _CoRR_ , abs/2306.09212.
  * Li et al. (2023b) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023b.  Starcoder: may the source be with you! _CoRR_ , abs/2305.06161.
  * Li et al. (2023c) Tsz On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023c.  Finding failure-inducing test cases with chatgpt.  _CoRR_ , abs/2304.11686.
  * Li et al. (2022a) Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022a.  Competition-level code generation with alphacode.  _CoRR_ , abs/2203.07814.
  * Li et al. (2022b) Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022b.  Automating code review activities by large-scale pre-training.  In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022_ , pages 1035–1047. ACM.
  * Lin and Hovy (2003) Chin-Yew Lin and Eduard H. Hovy. 2003.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003_. The Association for Computational Linguistics.
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017.  Quixbugs: a multi-lingual program repair benchmark set based on the quixey challenge.  In _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH 2017, Vancouver, BC, Canada, October 23 - 27, 2017_ , pages 55–56. ACM.
  * Lin et al. (2021) Tao Lin, Xue Fu, Fu Chen, and Luqun Li. 2021.  A novel approach for code smells detection based on deep leaning.  In _Applied Cryptography in Computer and Communications: First EAI International Conference, AC3 2021, Virtual Event, May 15-16, 2021, Proceedings 1_ , pages 171–174. Springer.
  * Ling et al. (2016) Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew W. Senior. 2016.  Latent predictor networks for code generation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Long and Rinard (2015) Fan Long and Martin C. Rinard. 2015.  Staged program repair with condition synthesis.  In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015_ , pages 166–178. ACM.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  Wizardcoder: Empowering code large language models with evol-instruct.
  * Madeyski and Lewowski (2023) Lech Madeyski and Tomasz Lewowski. 2023.  Detecting code smells using industry-relevant data.  _Inf. Softw. Technol._ , 155:107112.
  * Marinescu (2005) Radu Marinescu. 2005.  Measurement and quality in object-oriented design.  In _21st IEEE International Conference on Software Maintenance (ICSM 2005), 25-30 September 2005, Budapest, Hungary_ , pages 701–704. IEEE Computer Society.
  * McIntosh et al. (2014) Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2014.  The impact of code review coverage and code review participation on software quality: a case study of the qt, vtk, and ITK projects.  In _11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 - June 1, 2014, Hyderabad, India_ , pages 192–201. ACM.
  * Moha et al. (2010) Naouel Moha, Yann-Gaël Guéhéneuc, Laurence Duchien, and Anne-Françoise Le Meur. 2010.  DECOR: A method for the specification and detection of code and design smells.  _IEEE Trans. Software Eng._ , 36(1):20–36.
  * Nguyen et al. (2013a) Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. 2013a.  Lexical statistical machine translation for language migration.  In _Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13, Saint Petersburg, Russian Federation, August 18-26, 2013_ , pages 651–654. ACM.
  * Nguyen et al. (2013b) Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. 2013b.  Semfix: program repair via semantic analysis.  In _35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013_ , pages 772–781. IEEE Computer Society.
  * Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023.  Codegen: An open large language model for code with multi-turn program synthesis.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_ , pages 311–318. ACL.
  * Patel et al. (2023) Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. 2023.  Bidirectional language models are also few-shot learners.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Pettis and Hansen (1990) Karl Pettis and Robert C. Hansen. 1990.  Profile guided code positioning.  In _Proceedings of the ACM SIGPLAN’90 Conference on Programming Language Design and Implementation (PLDI), White Plains, New York, USA, June 20-22, 1990_ , pages 16–27. ACM.
  * Plotnikov et al. (2013) Dmitry Plotnikov, Dmitry Melnik, Mamikon Vardanyan, Ruben Buchatskiy, and Roman Zhuykov. 2013.  An automatic tool for tuning compiler optimizations.  In _Ninth International Conference on Computer Science and Information Technologies Revised Selected Papers_ , pages 1–7. IEEE.
  * Popov et al. (2017) Mihail Popov, Chadi Akel, Yohan Chatelain, William Jalby, and Pablo de Oliveira Castro. 2017.  Piecewise holistic autotuning of parallel programs with CERE.  _Concurr. Comput. Pract. Exp._ , 29(15).
  * Prenner and Robbes (2021) Julian Aron Prenner and Romain Robbes. 2021.  Automatic program repair with openai’s codex: Evaluating quixbugs.  _CoRR_ , abs/2111.03922.
  * Puri et al. (2021) Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. 2021.  Project codenet: A large-scale AI for code dataset for learning a diversity of coding tasks.  _CoRR_ , abs/2105.12655.
  * Qi et al. (2014) Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014.  The strength of random search on automated program repair.  In _36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014_ , pages 254–265. ACM.
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020.  Codebleu: a method for automatic evaluation of code synthesis.  _CoRR_ , abs/2009.10297.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code llama: Open foundation models for code.  _CoRR_ , abs/2308.12950.
  * Salehie et al. (2006) Mazeiar Salehie, Shimin Li, and Ladan Tahvildari. 2006.  A metric-based heuristic framework to detect object-oriented design flaws.  In _14th International Conference on Program Comprehension (ICPC 2006), 14-16 June 2006, Athens, Greece_ , pages 159–168. IEEE Computer Society.
  * Sharma et al. (2016) Tushar Sharma, Pratibha Mishra, and Rohit Tiwari. 2016.  Designite: a software design quality assessment tool.  In _Proceedings of the 1st International Workshop on Bringing Architectural Design Thinking into Developers’ Daily Activities, BRIDGE@ICSE 2016, Austin, Texas, USA, May 17, 2016_ , pages 1–4. ACM.
  * Siddiq et al. (2023) Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. 2023.  Exploring the effectiveness of large language models in generating unit tests.  _CoRR_ , abs/2305.00418.
  * Slivka et al. (2023) Jelena Slivka, Nikola Luburic, Simona Prokic, Katarina-Glorija Grujic, Aleksandar Kovacevic, Goran Sladic, and Dragan Vidakovic. 2023.  Towards a systematic approach to manual annotation of code smells.  _Sci. Comput. Program._ , 230:102999.
  * Sun et al. (2023) Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Li, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, Hanwei Qian, Yang Liu, and Zhenyu Chen. 2023.  Automatic code summarization via chatgpt: How far are we? _CoRR_ , abs/2305.12865.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Tufano et al. (2019) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019.  An empirical study on learning bug-fixing patches in the wild via neural machine translation.  _ACM Trans. Softw. Eng. Methodol._ , 28(4):19:1–19:29.
  * Tufano et al. (2022) Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022.  Using pre-trained models to boost code review automation.  In _44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022_ , pages 2291–2302. ACM.
  * Wang et al. (2023a) Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023a.  Scibench: Evaluating college-level scientific problem-solving abilities of large language models.  _CoRR_ , abs/2307.10635.
  * Wang et al. (2023b) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b.  Codet5+: Open code large language models for code understanding and generation.  _CoRR_ , abs/2305.07922.
  * Weimer et al. (2009) Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. 2009.  Automatically finding patches using genetic programming.  In _31st International Conference on Software Engineering, ICSE 2009, May 16-24, 2009, Vancouver, Canada, Proceedings_ , pages 364–374. IEEE.
  * Williams-King and Yang (2019) David Williams-King and Junfeng Yang. 2019.  Codemason: Binary-level profile-guided optimization.  In _Proceedings of the 3rd ACM Workshop on Forming an Ecosystem Around Software Transformation_ , pages 47–53.
  * Xie et al. (2023) Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023.  Chatunitest: a chatgpt-based automated unit test generation tool.  _CoRR_ , abs/2305.04764.
  * Yan and Li (2022) Weixiang Yan and Yuanchun Li. 2022.  Whygen: Explaining ml-powered code generation by referring to training examples.  In _44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24, 2022_ , pages 237–241. ACM/IEEE.
  * Yan et al. (2023) Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023.  Codetransocean: A comprehensive multilingual benchmark for code translation.  _CoRR_ , abs/2310.04951.
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A syntactic neural model for general-purpose code generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 440–450. Association for Computational Linguistics.
  * Yu et al. (2023) Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang. 2023.  Codereval: A benchmark of pragmatic code generation with generative pre-trained models.  _CoRR_ , abs/2302.00288.
  * Yuan et al. (2023) Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023.  No more manual tests? evaluating and improving chatgpt for unit test generation.  _CoRR_ , abs/2305.04207.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhang et al. (2023) Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.  _CoRR_ , abs/2306.05179.
  * Zheng et al. (2023a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023a.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Zheng et al. (2023b) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023b.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.  _CoRR_ , abs/2303.17568.
  * Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023.  Agieval: A human-centric benchmark for evaluating foundation models.
  * Zhu et al. (2022) Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022.  Xlcost: A benchmark dataset for cross-lingual code intelligence.

##  Appendix A Appendix

###  A.1 Related Work

Code Summarization Recently, the emergence of a pre-trained language model has
accelerated the development of code summarization. Many works deemed code
summarization as a translation task and finetuned a pre-trained encoder-
decoder model to a language-specific summarization model (Clement et al.,
2020). More recently, LLM pre-trained with code data such as CodeBert (Feng et
al., 2020), CodeT5+ (Wang et al., 2023b), Incoder (Fried et al., 2023), have
each shown to the open community their competitive performance on summarizing
minor mainstream program languages. Leveraging the few-shot or zero-shot
learning ability that LLMs offer, some work is dedicated to evaluating the
code summarization performance for a specific model under domain-specific
settings. Ahmed and Devanbu (2022) evaluated CodeX on project-specific data.
Sun et al. (2023) evaluated ChatGPT on a Python code dataset.

Code Smell Detecting and repairing code smells early in the development
process is essential to enhance the reliability, scalability, and
maintainability of software systems. Traditional code smell detection mainly
adopts metric-based and rule/heuristic-based approaches. Metric-based
approaches combine metrics such as complexity, coupling, and class size, and
then use thresholds or ranges to determine the presence of code smells
(Marinescu, 2005; Salehie et al., 2006). Rule/Heuristic-based approaches rely
on rules and heuristic criteria set by experienced developers or experts (Moha
et al., 2010; Sharma et al., 2016). In recent years, researchers have used
deep learning to explore code smell detection. Lin et al. (2021) used a fully
convolutional network that focuses on code semantic features for detection,
while the convolutional neural network trained by Das et al. (2019)
demonstrated commendable efficacy in detecting specific code smells.

Code Review McIntosh et al. (2014) demonstrated that code review can
effectively reduce the defect rate of software. With the rapid development of
deep learning, recent studies have focused on using deep learning to automate
code reviews. Tufano et al. (2022) proposed a method based on the T5 model
that can automatically provide code improvement suggestions for reviewers,
while automatically making code changes based on submitted code and natural
language review feedback. This approach has shown great potential in
shortening code review cycles and assisting code submitters. Li et al. (2022b)
designed four pre-trained tasks specifically for code review, thereby
improving the accuracy of code review. At the same time, the performance of
neural networks in code review is evaluated based on three tasks: code change
quality estimation, code review generation, and code refinement.

Automated Testing In recent studies, Siddiq et al. (2023) explored the ability
of LLMs to generate unit tests for software, and evaluated the quality of unit
tests generated by LLMs. Li et al. (2023c) introduced differential prompting,
using ChatGPT to identify test cases that can trigger program errors. Yuan et
al. (2023) proposed ChatTESTER to enhance ChatGPT’s ability to generate high-
quality test cases, and investigated the correctness and usability of test
cases generated by ChatGPT, effectively improving the accuracy and efficiency
of automated testing. Xie et al. (2023) designed a ChatGPT-based automated
unit test Generation-Validation-Repair framework called ChatUniTest, which can
not only generate high-coverage unit tests, but also repair syntactic and
compilation errors.

Program Synthesis Before the boom of LLMs, previous works (Balog et al., 2017;
Ling et al., 2016; Yin and Neubig, 2017) usually focused on synthesizing and
analyzing programs in Domain-specific Language (DSL). Deepcoder (Balog et al.,
2017) leverages an encoder-decoder network to predict program properties with
given input and outputs. Ling et al. (2016); Yin and Neubig (2017) used
recurrent networks and pointer networks to map natural language descriptions
to code segments(e.g., code structure, syntax tree). Devlin et al. (2017)
achieved directly generating target codes by applying a seq2seq generative
network. In addition, LLMs also showed their strong performance in this task,
like BERT-based and GPT-based methods. There are also some benchmarks
developed to evaluate the performance: Austin et al. (2021) focused on
evaluating LLMs on this task with two Python datasets (MBPP and MathQA-
Python). Chen et al. (2021) also focused on the GPT language model’s Python
code-writing capabilities within their Humaneval dataset. Khan et al. (2023)
conducted a broader evaluation scope over nine programming languages.

Code Translation Code translation involves taking source code written in one
programming language (the source language) and generating equivalent code in
another (the target language). Most of the existing works only focus on mutual
translation between two languages. One of the most popular benchmarks,
CodeXGlue (Lu et al., 2021) provides CodeTrans involving the translation
between Java and C#. While Ahmed and Devanbu (2022); Nguyen et al. (2013a)
include Java & Python, and Java& C#, respectively. For enabling translation
among various programming languages, the works by Zhu et al. (2022); Yan et
al. (2023); Khan et al. (2023) primarily focus on supporting 7, 8, and 11
programming languages. For evaluation, most of works (Ahmed and Devanbu, 2022;
Lu et al., 2021; Zhu et al., 2022) still rely on n-gram matching like BLEU
(Papineni et al., 2002) and CodeBLEU (Ren et al., 2020), which suffer from the
heavy dependence on comprehensiveness and accuracy of reference code. Yan et
al. (2023); Khan et al. (2023) adopt executable metrics Debugging successful
rate (DSR) and Pass rate that tests the code by executability and accuracy
under test cases.

Code Repair The earliest tools for code repair are static analysis tools that
check code for basic errors like syntax violations. For automatic code
repairing, the semantic-based repairing techniques develop with the help of a
specification for the intended program behavior (Nguyen et al., 2013b; Weimer
et al., 2009; Long and Rinard, 2015; Qi et al., 2014). With the inspiration of
neural machine translation (NMT), some works leverage the LMs to enhance
automatic code repair. Tufano et al. (2019) leverage the capabilities of NMT
to transform flawed code into corrected code, essentially simulating the
fusion of an Abstract Syntax Tree (AST). Prenner and Robbes (2021) explore the
CodeX’s (Chen et al., 2021) code repairing performance of Python and Java on
QuixBugs (Lin et al., 2017). Recently, Khan et al. (2023) tested the GPT-3.5’s
performance over 11 program languages. Unlike the existing works, we evaluate
8 kinds of powerful LLMs code repairing ability over 14 programming languages,
providing a more comprehensive evaluation.

Code Optimization A wide range of optimization methods have been developed.
Most of them improve the efficiency of developers’ code from the compiler or
source code level. Many optimization techniques have been applied in compilers
during the compiling process, including dead code elimination, inline
expansion, loop optimization, instruction scheduling, automatic
parallelization, and so on. Many researchers worked on using different static
techniques to obtain the best compiler flag combinations to gain performance
(Cáceres et al., 2017; Popov et al., 2017; Plotnikov et al., 2013). Profile-
guided optimization(PGO) approaches (Pettis and Hansen, 1990; Williams-King
and Yang, 2019) obtain a profile(feedback data) by executing the code and
producing an optimized version of the code by analyzing the profile. However,
such a method needs more compiling time and thus needs better usability.
Another group of optimization techniques transforms the source code to make it
more efficient. Much research in this category works on optimizing loops using
the polyhedral model (Bondhugula et al., 2008; Bastoul, 2004). Some work uses
auto-tuning (Chen et al., 2008, 2016) to generate multiple code variants using
alternative algorithms or code transformation(such as loop unrolling, and
blocking scheduling) and search for the best optimization. This paper
emphasizes code optimization at the source code level, as the paradigm of
source code generation is highly compatible with LLMs.

###  A.2 Experimental Setup

To facilitate the replication of our experimental results, we have detailed
the specific configuration information for each LLM and the corresponding
inference environments in Table 5.

Model Model Version Model Size Inference GPU GPT-4 gpt-4 - - GPT-3.5
gpt-3.5-turbo-0613 - - PaLM 2 text-bison-001 - - LLaMA 2 LLaMA-2-70b-chat-hf
70B NVIDIA Tesla A800 * 4 StarCoder starchat-beta 15B NVIDIA Tesla A800 * 1
Code LLaMA Code LLaMA-34b-Instruct-hf 34B NVIDIA GeForce RTX 4090 * 4
WizardCoder WizardCoder-15B-V1.0 15B NVIDIA GeForce RTX 4090 * 2 Vicuna
vicuna-13b-v1.5-16k 13B NVIDIA GeForce RTX 4090 * 2

Table 5: Configuration information for the baseline LLMs, parameters for the
experiment, and hardware information for inference.

###  A.3 Detailed Experimental Result

Language Metric GPT-4 GPT-3.5 Vicuna WizardCoder Code LLaMA LLaMA 2 StarCoder
PaLM 2 Python BLEU 4.85 4.79 3.86 2.75 3.46 2.92 2.48 3.18 METEOR 25.07 20.65
23.71 21.39 22.09 23.08 21.39 15.35 ROUGE 20.16 18.88 19.08 18.04 17.96 18.03
16.91 17.41 BERTScore 85.61 85.11 84.91 82.11 83.6 84.34 83.65 75.8 C BLEU
4.57 4.09 3.64 3.17 2.63 2.68 2.28 4.61 METEOR 25.42 23.63 23.3 22.16 21.23
22.36 21.38 15.01 ROUGE 19.88 19.73 18.68 19.04 16.87 17.16 16.16 18.05
BERTScore 85.49 85.39 84.21 82.57 83.17 83.62 83.92 81.3 C++ BLEU 4.73 3.99
3.23 3.38 2.91 2.61 2.41 4.19 METEOR 25.02 23.77 22.44 22.9 21.37 21.75 22.08
16.17 ROUGE 20.4 19.9 18.37 18.76 16.72 17.43 17.24 19.73 BERTScore 85.65
85.61 84.76 84.58 83.06 84.17 84.37 83.46 Java BLEU 4.77 4.63 3.81 3.08 2.87
3.03 2.93 3.86 METEOR 25.28 24.84 24.4 23.33 23.19 23.09 23.19 13.24 ROUGE
20.17 20.39 19.57 19.52 17.95 18.49 17.97 17.02 BERTScore 85.66 85.91 85.42
83.44 82.72 84.18 84.5 78.21 C# BLEU 4.74 4.07 3.56 3.26 2.9 2.73 1.99 4.16
METEOR 24.55 22.75 22.65 21.93 21.81 21.41 20.95 16.36 ROUGE 20.38 19.64 18.72
19.46 17.89 18.03 16.27 19.96 BERTScore 85.77 85.15 84.02 82.43 82.73 83.46
83.61 83.6 JavaScript BLEU 4.87 4.39 3.01 2.79 2.91 2.63 2.45 2.63 METEOR
24.52 23.91 22.93 21.88 22.56 22.27 21.65 15.51 ROUGE 19.84 20.26 18.07 18.73
18.1 17.56 17.15 17.35 BERTScore 85.39 85.53 84.43 84.35 83.45 84.11 84.06
79.8 Visual Basic BLEU 4.65 3.51 3.55 4.72 3.16 2.52 2.2 4.11 METEOR 20.53
19.74 20.55 21.5 20.36 20.01 19.93 13.68 ROUGE 17.52 16.65 16.4 17.64 17.26
15.87 15.24 16.62 BERTScore 84.93 84.97 83.87 85.3 84.61 84.12 83.98 83.44
Visual Basic (.NET) BLEU 5.17 4.42 3.77 2.64 3.52 2.93 3.16 6.0 METEOR 25.73
23.06 24.06 21.64 21.58 21.92 21.03 16.1 ROUGE 20.6 18.83 18.54 19.68 18.51
18.71 18.33 21.3 BERTScore 86.08 84.12 83.31 82.41 81.71 83.26 84.83 81.5 SQL
BLEU 3.19 4.76 4.1 1.53 2.5 2.44 1.8 3.91 METEOR 22.47 21.35 21.74 19.2 21.53
19.61 18.01 17.74 ROUGE 17.83 19.31 17.08 17.03 17.27 16.38 15.75 20.0
BERTScore 85.07 85.57 85.06 84.95 84.44 83.93 83.64 82.52 PHP BLEU 5.12 5.38
4.29 3.74 4.18 3.58 3.13 5.22 METEOR 24.15 24.12 23.59 22.92 23.09 23.01 21.57
17.18 ROUGE 21.25 21.41 19.2 20.91 19.79 19.24 17.83 21.36 BERTScore 85.94
85.91 84.83 85.56 84.45 84.87 84.35 83.39 MATLAB BLEU 5.73 4.81 3.45 4.39 3.82
3.36 2.97 6.37 METEOR 23.3 22.19 22.64 21.17 21.39 21.43 21.07 15.84 ROUGE
21.03 21.32 19.42 19.85 18.26 18.69 19.1 18.28 BERTScore 85.81 85.69 84.59
83.63 82.87 82.83 85.01 72.77 Fortran BLEU 5.4 4.87 3.93 3.43 3.83 2.49 3.28
3.48 METEOR 24.74 23.76 22.67 22.2 21.99 22.01 22.58 12.73 ROUGE 20.49 20.91
18.64 18.75 18.01 17.04 18.0 16.73 BERTScore 85.74 86.01 83.83 83.55 81.57
83.18 84.69 74.61 Go BLEU 4.49 4.53 3.58 3.98 3.27 2.8 3.1 4.38 METEOR 25.27
23.54 23.72 22.94 22.85 21.5 22.19 14.71 ROUGE 20.35 20.25 18.81 19.35 18.57
17.73 18.57 18.09 BERTScore 85.7 85.78 84.94 85.23 84.18 84.54 85.01 77.45 X86
Assembly BLEU 3.62 3.12 2.82 1.6 2.88 1.55 2.1 4.93 METEOR 23.66 21.78 21.85
19.09 19.7 20.71 19.85 13.51 ROUGE 18.24 18.59 17.86 17.98 15.78 13.51 17.19
17.15 BERTScore 84.89 84.82 84.59 79.62 78.49 82.65 83.82 79.65 ARM Assembly
BLEU 3.35 3.43 2.34 1.29 1.46 1.03 1.8 4.12 METEOR 21.16 19.72 17.15 14.06
12.99 13.54 12.78 11.01 ROUGE 15.36 16.2 13.85 11.36 9.58 7.79 8.96 9.99
BERTScore 84.57 85.29 74.81 64.47 62.5 67.18 78.6 46.53

Table 6: The detailed performance of each LLM in code summarization across all
languages.

Language Metric GPT-4 GPT-3.5 Vicuna WizardCoder Code LLaMA LLaMA 2 StarCoder
PaLM 2 Delphi BLEU 3.98 4.37 3.46 3.35 3.1 2.11 1.63 3.81 METEOR 18.37 23.41
21.88 20.96 20.0 20.57 19.06 15.14 ROUGE 15.99 20.37 17.7 18.65 16.59 16.7
14.81 18.63 BERTScore 84.49 85.86 84.35 84.57 84.19 83.9 83.04 82.64 Ruby BLEU
5.2 4.67 3.33 2.96 2.53 3.61 2.78 4.4 METEOR 24.79 23.08 22.16 22.58 23.61
23.6 20.88 16.94 ROUGE 21.53 20.96 19.34 19.64 19.52 19.59 17.2 19.26
BERTScore 86.02 85.79 84.99 85.05 84.94 84.7 84.3 81.99 Rust BLEU 4.48 3.67
3.15 3.57 3.06 2.75 2.47 5.08 METEOR 24.57 22.69 23.07 22.74 22.08 22.17 21.79
17.88 ROUGE 19.94 19.58 18.61 19.46 18.2 18.53 17.43 20.54 BERTScore 85.42
85.09 84.49 84.56 84.02 83.9 84.67 83.42 Swift BLEU 4.88 3.86 3.35 3.66 2.78
3.33 2.85 5.26 METEOR 24.27 22.3 21.7 22.42 22.13 22.03 21.62 17.99 ROUGE 20.1
19.45 17.78 19.39 17.96 17.91 17.23 19.83 BERTScore 85.42 85.22 84.39 85.21
84.39 84.16 84.36 83.14 R BLEU 5.64 4.57 3.59 2.72 3.68 3.27 2.62 3.95 METEOR
26.08 24.08 22.65 22.25 22.65 22.59 20.19 18.81 ROUGE 21.91 21.11 19.36 19.98
19.49 18.64 17.08 20.33 BERTScore 85.9 85.95 84.61 85.54 84.94 84.4 83.99
84.71 COBOL BLEU 4.17 4.28 3.73 2.69 2.9 2.17 2.58 3.56 METEOR 23.88 22.6
21.23 20.34 20.26 19.47 20.87 13.77 ROUGE 19.03 19.12 16.96 18.32 16.15 13.9
16.84 18.22 BERTScore 85.33 85.63 82.85 82.13 81.35 80.48 83.17 80.21 Ada BLEU
4.78 4.24 3.47 3.15 2.95 2.97 2.56 5.01 METEOR 24.53 23.28 22.59 22.51 21.8
22.37 21.8 16.38 ROUGE 20.12 20.08 18.87 19.92 17.25 17.13 17.02 21.02
BERTScore 85.29 85.73 84.3 85.12 83.3 83.59 84.05 84.76 Julia BLEU 5.14 4.51
2.95 4.05 3.56 3.23 3.03 4.22 METEOR 25.25 23.38 21.87 22.44 22.17 22.0 20.64
18.4 ROUGE 21.3 20.76 18.16 20.29 19.15 18.55 17.32 20.16 BERTScore 85.9 85.67
84.52 84.82 84.5 84.04 83.97 82.46 SAS BLEU 5.6 5.18 4.63 2.13 4.26 3.4 3.51
5.12 METEOR 23.27 21.33 21.65 21.25 22.86 22.75 19.82 19.16 ROUGE 20.46 20.64
18.69 20.7 19.49 18.27 17.99 20.98 BERTScore 85.19 85.71 84.91 85.8 85.15
84.31 83.76 85.42 Kotlin BLEU 4.56 4.01 3.35 3.49 3.02 2.73 2.81 5.71 METEOR
24.52 23.16 22.99 23.47 22.02 22.29 21.97 16.96 ROUGE 20.39 19.79 18.87 19.86
17.62 18.97 17.24 20.51 BERTScore 85.76 85.08 85.04 85.05 84.17 84.97 84.51
83.7 Perl BLEU 5.18 4.45 3.8 3.63 3.2 2.96 2.61 5.4 METEOR 23.93 22.93 22.11
22.08 21.81 21.86 20.6 17.05 ROUGE 20.34 20.71 18.82 19.53 18.19 17.98 16.71
19.85 BERTScore 85.72 85.58 84.98 84.87 84.11 84.26 83.89 79.66 Objective-C
BLEU 4.12 4.59 3.44 4.39 3.69 3.27 3.07 3.61 METEOR 23.01 22.07 21.31 19.67
21.43 22.91 21.79 14.24 ROUGE 19.01 20.01 17.69 19.02 17.88 18.69 18.74 18.58
BERTScore 85.22 85.44 84.34 85.05 84.47 84.08 84.67 78.62 Prolog BLEU 4.25
3.42 3.19 2.08 2.38 2.46 2.54 5.15 METEOR 24.05 23.0 22.96 21.47 21.63 21.98
20.97 16.74 ROUGE 19.88 19.29 18.62 18.74 16.66 16.4 17.4 19.11 BERTScore
84.86 84.81 83.98 84.82 82.14 83.28 84.01 82.34 Lua BLEU 4.81 4.12 3.31 3.8
3.06 2.56 2.2 4.94 METEOR 24.56 21.96 22.32 21.86 22.11 22.21 20.16 16.07
ROUGE 20.21 19.6 18.0 19.91 18.47 17.9 16.85 18.89 BERTScore 85.77 85.71 84.82
85.44 84.76 84.22 84.47 81.13 Scala BLEU 4.33 4.3 2.87 3.76 3.0 2.76 2.47 3.98
METEOR 24.99 23.43 22.76 22.67 21.16 21.64 20.72 17.73 ROUGE 19.21 18.98 17.59
18.65 17.05 17.13 16.73 18.41 BERTScore 85.5 85.28 84.35 84.94 84.26 84.04
83.98 81.72

Table 7: The detailed performance of each LLM in code summarization across all
languages. (Cont. Table 6)

Language Metric GPT-4 GPT-3.5 Vicuna WizardCoder Code LLaMA LLaMA 2 StarCoder
PaLM 2 Dart BLEU 5.06 3.72 3.48 4.17 3.44 3.58 2.66 5.23 METEOR 25.21 22.62
22.97 21.58 22.04 23.62 22.63 15.62 ROUGE 20.26 20.68 18.43 18.23 19.77 20.19
18.68 19.85 BERTScore 85.97 85.8 84.99 84.99 85.27 85.04 85.1 85.99 D BLEU
5.23 4.4 3.84 3.45 3.5 2.97 3.18 5.89 METEOR 24.01 22.34 21.78 22.15 22.58
21.71 21.08 16.4 ROUGE 20.18 20.08 18.24 18.8 17.47 18.09 16.69 21.04
BERTScore 85.52 85.34 84.66 85.06 83.9 84.44 84.42 83.5 Haskell BLEU 5.14 4.06
3.25 2.79 2.93 2.41 2.49 4.15 METEOR 25.79 23.35 22.21 21.77 22.01 21.4 19.14
16.8 ROUGE 20.75 19.88 18.88 18.45 17.49 17.15 16.81 17.86 BERTScore 85.37
85.35 84.24 83.97 83.44 83.62 83.94 79.53 VBScript BLEU 4.14 4.55 3.25 2.99
3.16 2.62 2.91 5.23 METEOR 23.89 22.05 20.77 20.42 21.7 21.24 21.48 16.57
ROUGE 19.14 19.64 16.65 18.67 17.72 16.44 16.97 18.64 BERTScore 85.43 85.57
84.37 85.16 83.6 84.19 84.34 83.54 Scheme BLEU 4.52 4.05 2.77 2.26 3.2 2.86
2.83 5.62 METEOR 24.84 23.16 22.82 21.47 21.67 21.25 21.71 17.34 ROUGE 20.05
20.09 18.21 18.96 17.51 17.44 17.65 20.28 BERTScore 85.39 85.51 83.61 82.66
81.91 82.63 84.45 81.22 PowerShell BLEU 5.03 5.49 4.29 3.93 3.58 3.4 3.05 5.11
METEOR 24.82 23.83 23.88 21.73 23.22 22.33 21.13 15.97 ROUGE 20.62 21.93 19.45
19.76 19.57 18.47 17.63 19.52 BERTScore 85.77 86.04 85.14 85.56 84.94 84.53
84.4 82.48 Logo BLEU 3.45 3.22 2.68 2.07 2.36 2.33 2.64 3.05 METEOR 21.94 20.0
20.41 19.51 19.8 19.86 20.06 14.9 ROUGE 17.88 17.95 16.63 18.32 17.37 15.8
17.14 16.82 BERTScore 84.86 84.8 84.22 84.89 84.15 83.44 84.27 83.31 ABAP BLEU
4.77 3.53 3.12 4.87 2.1 2.61 2.42 4.75 METEOR 24.71 20.29 20.62 20.27 20.68
20.28 19.38 15.28 ROUGE 20.05 18.69 17.04 19.01 16.63 15.88 15.68 18.02
BERTScore 85.17 84.94 81.68 82.75 81.62 81.19 83.57 80.36 F# BLEU 5.84 4.84
3.29 3.92 2.67 2.67 2.96 4.35 METEOR 25.31 23.79 22.91 22.52 22.36 21.88 20.65
17.28 ROUGE 21.56 21.13 18.51 19.2 18.22 17.26 17.7 19.96 BERTScore 85.91
85.79 84.51 85.11 84.69 84.14 84.53 81.76 AWK BLEU 3.97 4.24 3.16 3.83 2.99
2.52 2.15 4.76 METEOR 23.32 23.47 22.5 22.03 22.4 20.97 19.38 16.57 ROUGE
19.54 20.06 17.8 20.23 18.34 17.3 16.09 19.51 BERTScore 85.63 85.83 84.65
85.56 84.8 84.19 83.65 81.82 Groovy BLEU 4.57 3.87 1.97 3.0 2.93 3.06 2.64
4.35 METEOR 24.37 22.76 21.18 21.84 23.01 22.91 20.67 17.26 ROUGE 20.68 20.56
17.96 19.48 18.55 18.51 17.65 19.84 BERTScore 85.67 85.54 84.38 84.75 84.87
84.37 84.08 82.29 ColdFusion BLEU 6.42 4.63 1.59 3.6 4.52 3.73 1.97 6.6 METEOR
23.43 21.22 21.23 22.24 20.92 21.68 15.91 16.58 ROUGE 21.18 20.56 17.71 21.59
19.62 18.76 16.42 21.4 BERTScore 85.72 85.46 83.58 85.68 82.83 84.46 83.44
83.51 Zig BLEU 3.49 3.32 2.45 3.16 2.47 2.58 2.85 4.27 METEOR 23.22 21.91
19.27 22.57 21.3 21.44 20.32 16.76 ROUGE 17.61 17.57 16.25 18.31 15.93 16.56
15.96 19.15 BERTScore 85.22 85.24 84.03 85.34 84.41 84.23 84.5 83.92 Overall
33.66 33.18 32.13 32.10 31.56 31.49 31.16 30.21

Table 8: The detailed performance of each LLM in code summarization across all
languages. (Cont. Table 7)

Language Metric GPT-4 GPT-3.5 Code LLaMA LLaMA 2 PaLM 2 WizardCoder Vicuna
StarCoder C Accuracy 49.00 49.00 52.00 57.00 44.00 50.00 49.00 50.00 Precision
48.90 41.41 54.15 63.73 45.31 50.53 48.49 25.00 Recall 49.00 49.00 52.00 57.00
44.00 50.00 49.00 50.00 F1-score 47.83 34.54 44.85 51.00 44.30 37.04 44.32
33.33 BLEU 1.81 2.76 1.72 1.41 2.11 0.48 1.23 0.05 ROUGE 9.72 10.24 10.65 9.32
10.26 8.67 8.31 1.01 BERTScore 84.34 85.06 81.04 83.94 84.82 83.01 83.08 75.79
C# Accuracy 55.00 51.00 51.00 49.00 46.00 52.00 51.00 49.00 Precision 55.25
52.21 52.55 48.73 45.83 75.51 51.94 24.75 Recall 55.00 51.00 51.00 49.00 46.00
52.00 51.00 49.00 F1-score 54.45 43.23 42.21 46.15 45.45 37.63 46.03 32.89
BLEU 0.40 0.49 0.00 0.38 0.00 0.18 0.77 0.00 ROUGE 8.32 7.85 8.67 7.64 7.53
6.37 7.10 0.93 BERTScore 83.74 84.46 83.48 83.41 83.89 82.13 82.64 75.83 C++
Accuracy 57.00 53.00 48.00 51.00 42.00 51.00 46.00 50.00 Precision 57.23 75.77
45.27 53.84 42.43 56.99 40.53 25.00 Recall 57.00 53.00 48.00 51.00 42.00 51.00
46.00 50.00 F1-score 56.65 39.67 39.22 39.88 42.21 45.27 36.89 33.33 BLEU 1.48
1.62 1.16 0.88 3.10 0.37 0.94 0.08 ROUGE 7.62 7.20 7.65 7.29 10.22 5.56 6.09
0.81 BERTScore 83.61 84.28 81.64 83.32 84.03 81.52 82.68 75.50 Go Accuracy
54.0 49.00 51.00 46.00 57.00 48.00 67.00 50.00 Precision 54.60 24.75 53.05
43.75 57.00 24.74 69.22 25.00 Recall 54.0 49.00 51.00 46.00 57.00 48.00 67.00
50.00 F1-score 52.46 32.89 41.10 40.66 57.00 32.65 66.02 33.33 BLEU 0.76 1.19
0.70 0.45 1.49 0.16 0.47 0.00 ROUGE 8.61 8.06 7.75 7.05 9.50 6.01 7.13 0.67
BERTScore 83.65 84.53 82.00 83.12 84.30 80.22 82.55 75.37 Java Accuracy 46.00
47.00 54.00 52.00 46.00 49.00 52.00 50.00 Precision 45.83 44.12 61.11 53.25
45.89 44.74 52.38 25.00 Recall 46.00 47.00 54.00 52.00 46.00 49.00 52.00 50.00
F1-score 52.46 32.89 41.10 40.66 57.00 32.65 66.02 33.33 BLEU 0.76 1.19 0.70
0.45 1.49 0.16 0.47 0.00 ROUGE 8.61 8.06 7.75 7.05 9.50 6.01 7.13 0.80
BERTScore 83.65 84.53 82.00 83.12 84.30 80.22 82.55 75.44 JavaScript Accuracy
45.00 53.00 51.00 49.00 53.00 50.00 50.00 51.00 Precision 44.16 75.77 51.33
48.59 55.27 25.00 50.00 75.25 Recall 45.00 53.00 51.00 49.00 53.00 50.00 50.00
51.00 F1-score 42.94 39.67 47.73 44.99 53.96 33.33 44.30 35.52 BLEU 1.17 1.16
1.00 0.62 1.55 0.41 0.52 0.00 ROUGE 9.35 10.39 9.56 7.48 11.20 7.23 7.65 0.99
BERTScore 83.93 84.77 83.92 83.19 84.35 82.84 83.05 75.89 PHP Accuracy 44.00
48.00 51.00 48.00 43.00 49.00 52.00 50.00 Precision 42.56 45.27 52.55 47.40
47.27 24.75 53.70 25.00 Recall 44.00 48.00 51.00 48.00 43.00 49.00 52.00 50.00
F1-score 41.15 39.22 42.21 44.82 45.02 32.89 46.77 33.33 BLEU 0.63 1.37 0.00
0.00 0.57 0.38 0.60 0.04 ROUGE 7.11 7.35 7.10 5.78 6.16 6.84 6.42 0.76
BERTScore 83.39 84.46 81.84 82.82 79.84 82.63 82.56 75.43 Python Accuracy
48.00 49.00 50.00 51.00 38.00 46.00 55.00 49.00 Precision 47.77 24.75 50.00
51.96 38.70 36.41 58.86 24.75 Recall 48.00 49.00 50.00 51.00 38.00 46.00 55.00
49.00 F1-score 46.63 32.89 41.56 44.16 38.28 34.43 49.50 32.89 BLEU 0.96 1.54
1.18 0.74 2.69 0.65 1.00 0.00 ROUGE 9.37 9.54 9.29 8.29 10.76 6.93 7.39 0.80
BERTScore 83.56 84.52 83.54 82.80 84.17 81.49 82.57 75.51 Ruby Accuracy 48.00
50.00 66.00 58.00 59.00 51.00 48.00 49.00 Precision 47.7 50.00 70.79 64.88
59.46 55.26 43.21 24.75 Recall 48.00 50.00 66.00 58.00 59.00 51.00 48.00 49.00
F1-score 46.26 36.58 63.92 52.51 58.50 38.56 36.86 32.89 BLEU 0.91 0.97 0.50
1.03 1.78 0.24 0.81 0.05 ROUGE 7.17 6.79 6.40 7.16 7.18 4.87 5.23 0.73
BERTScore 83.09 83.91 81.33 82.69 82.91 79.72 81.90 75.59 Overall 41.37 39.97
42.41 41.77 40.79 38.29 41.92 34.28

Table 9: Detailed experimental results of code review.

Language\Model GPT-4 GPT-3.5 PaLM 2 Code LLaMA WizardCoder LLaMA 2 StarCoder
Vicuna Avg. C++ 86.67 66.67 20.00 13.33 10.00 0.00 0.00 6.67 28.10 Java 90.00
60.00 10.00 23.33 13.33 3.33 0.00 0.00 28.57 Python 53.33 33.33 3.33 3.33 0.00
3.33 0.00 0.00 13.81 C 66.67 33.33 3.33 0.00 6.67 3.33 0.00 0.00 16.19 C#
56.67 33.33 3.33 3.33 0.00 0.00 0.00 0.00 13.81 Ruby 36.67 26.67 3.33 3.33
6.67 0.00 3.33 0.00 11.43 Go 56.67 26.67 6.67 6.67 0.00 0.00 0.00 0.00 13.81
JavaScript 23.33 10.00 3.33 0.00 0.00 6.67 0.00 0.00 6.19 Kotlin 76.67 56.67
6.67 6.67 0.00 0.00 3.33 0.00 21.43 PHP 30.00 16.67 0.00 3.33 0.00 0.00 0.00
0.00 7.14 Rust 73.33 53.33 10.00 10.00 3.33 0.00 0.00 0.00 21.43 Perl 70.00
53.33 23.33 10.00 10.00 3.33 6.67 3.33 25.24 D 33.33 33.33 0.00 0.00 0.00 0.00
0.00 0.00 9.52 Delphi 66.67 46.67 6.67 0.00 3.33 0.00 0.00 0.00 17.62 Avg.
58.57 39.29 7.14 5.95 3.81 1.43 0.95 0.71 -

Table 10: Evaluation result of program synthesis on easy problems, employing
the PASS@5 metric.

Language\Model GPT-4 GPT-3.5 PaLM 2 Code LLaMA WizardCoder LLaMA 2 StarCoder
Vicuna Avg. C++ 10.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.43 Java 20.00 10.00
0.00 0.00 0.00 0.00 0.00 0.00 4.29 Python 10.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 1.43 C 6.67 3.33 0.00 0.00 0.00 0.00 0.00 0.00 1.43 C# 10.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 1.43 Ruby 10.00 3.33 0.00 0.00 0.00 0.00 0.00 0.00
1.90 Go 16.67 3.33 3.33 0.00 0.00 0.00 0.00 0.00 3.33 JavaScript 0.00 9.09
0.00 0.00 0.00 0.00 0.00 0.00 1.30 Kotlin 17.24 3.45 0.00 0.00 0.00 0.00 0.00
0.00 2.96 PHP 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Rust 13.33 6.67
0.00 0.00 0.00 0.00 0.00 0.00 2.86 Perl 36.67 20.00 6.67 3.33 3.33 0.00 0.00
0.00 10.00 D 3.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.48 Delphi 0.00 10.00
0.00 0.00 0.00 0.00 0.00 0.00 1.43 Avg. 10.99 4.94 0.71 0.24 0.24 0.00 0.00
0.00 -

Table 11: Evaluation result of program synthesis on hard problems, employing
the PASS@5 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 60.00 26.67 73.33 40.00 60.00 46.67 13.33 33.33 26.67 53.33 0.00
0.00 6.67 33.85 Java 53.33 - 40.00 46.67 33.33 13.33 26.67 13.33 46.67 13.33
40.00 0.00 0.00 0.00 25.13 Python 40.00 20.00 - 33.33 33.33 53.33 46.67 13.33
20.00 33.33 40.00 6.67 0.00 0.00 26.15 C 73.33 66.67 40.00 - 26.67 20.00 33.33
6.67 46.67 13.33 13.33 0.00 0.00 0.00 26.15 C# 53.33 73.33 66.67 20.00 - 53.33
20.00 33.33 40.00 26.67 40.00 0.00 0.00 0.00 32.82 Ruby 26.67 33.33 53.33 6.67
46.67 - 20.00 0.00 26.67 33.33 6.67 0.00 0.00 0.00 19.49 Go 53.33 66.67 66.67
33.33 46.67 20.00 - 6.67 60.00 0.00 33.33 0.00 6.67 0.00 30.26 JavaScript
33.33 26.67 46.67 53.33 40.00 20.00 26.67 - 33.33 13.33 20.00 33.33 0.00 0.00
26.67 Kotlin 40.00 26.67 46.67 13.33 40.00 13.33 6.67 26.67 - 6.67 46.67 6.67
0.00 0.00 21.03 PHP 46.67 60.00 40.00 26.67 53.33 46.67 26.67 26.67 53.33 -
46.67 6.67 0.00 0.00 33.33 Rust 40.00 40.00 46.67 20.00 20.00 26.67 0.00 6.67
26.67 6.67 - 26.67 0.00 0.00 20.00 Perl 33.33 33.33 26.67 20.00 33.33 46.67
26.67 20.00 46.67 26.67 46.67 - 0.00 0.00 27.69 D 33.33 60.00 60.00 66.67
33.33 33.33 26.67 26.67 40.00 53.33 46.67 46.67 - 0.00 40.51 Delphi 66.67
60.00 40.00 46.67 40.00 20.00 26.67 6.67 33.33 26.67 40.00 33.33 26.67 - 35.90
Avg. 45.64 48.21 46.15 35.38 37.44 32.82 25.64 15.38 38.97 21.54 36.41 12.31
2.56 0.51 28.50

Table 12: Evaluation result of code translation on easy problems using
GPT-3.5, employing the Pass@1 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 26.67 6.67 20.00 13.33 6.67 6.67 6.67 20.00 0.00 6.67 0.00 0.00
0.00 8.72 Java 26.67 - 20.00 33.33 13.33 0.00 0.00 0.00 13.33 0.00 26.67 0.00
0.00 0.00 10.26 Python 20.00 20.00 - 6.67 46.67 20.00 13.33 13.33 33.33 13.33
13.33 13.33 6.67 0.00 16.92 C 80.00 33.33 0.00 - 13.33 6.67 26.67 0.00 6.67
6.67 6.67 0.00 0.00 0.00 13.85 C# 26.67 20.00 13.33 13.33 - 6.67 6.67 20.00
20.00 13.33 0.00 0.00 0.00 0.00 10.77 Ruby 33.33 33.33 33.33 26.67 20.00 -
20.00 0.00 13.33 26.67 13.33 13.33 6.67 0.00 18.46 Go 13.33 20.00 6.67 13.33
20.00 6.67 - 13.33 13.33 0.00 6.67 0.00 0.00 0.00 8.72 JavaScript 27.27 18.18
0.00 18.18 18.18 0.00 0.00 - 9.09 18.18 0.00 18.18 0.00 0.00 9.79 Kotlin 40.00
40.00 13.33 20.00 13.33 20.00 6.67 0.00 - 6.67 6.67 0.00 0.00 0.00 12.82 PHP
53.85 38.46 30.77 23.08 38.46 38.46 0.00 23.08 30.77 - 23.08 7.69 0.00 0.00
23.67 Rust 13.33 33.33 6.67 20.00 13.33 33.33 0.00 6.67 6.67 6.67 - 0.00 0.00
0.00 10.77 Perl 40.00 33.33 53.33 26.67 26.67 13.33 6.67 13.33 26.67 40.00
26.67 - 6.67 6.67 24.62 D 40.00 26.67 26.67 20.00 13.33 13.33 6.67 0.00 20.00
26.67 13.33 6.67 - 0.00 16.41 Delphi 46.67 20.00 0.00 46.67 6.67 6.67 6.67
0.00 0.00 0.00 6.67 0.00 0.00 - 10.77 Avg. 35.47 27.95 16.21 22.15 19.74 13.21
7.69 7.42 16.40 12.17 11.52 4.55 1.54 0.51 14.04

Table 13: Evaluation result of code translation on hard problems using
GPT-3.5, employing the Pass@1 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 86.67 46.67 53.33 53.33 53.33 60.00 33.33 66.67 33.33 80.00 0.00
6.67 0.00 44.10 Java 66.67 - 33.33 46.67 40.00 26.67 33.33 40.00 60.00 20.00
46.67 0.00 0.00 0.00 31.79 Python 66.67 60.00 - 33.33 53.33 60.00 53.33 80.00
46.67 53.33 53.33 6.67 0.00 0.00 43.59 C 66.67 80.00 46.67 - 53.33 33.33 60.00
40.00 46.67 40.00 53.33 0.00 0.00 0.00 40.00 C# 73.33 46.67 60.00 53.33 -
60.00 26.67 40.00 53.33 60.00 53.33 0.00 0.00 0.00 40.51 Ruby 40.00 53.33
40.00 20.00 46.67 - 13.33 53.33 53.33 40.00 40.00 6.67 0.00 0.00 31.28 Go
73.33 53.33 46.67 26.67 46.67 26.67 - 53.33 73.33 13.33 40.00 6.67 6.67 6.67
36.41 JavaScript 26.67 53.33 46.67 20.00 33.33 46.67 46.67 - 53.33 20.00 33.33
26.67 13.33 0.00 32.31 Kotlin 73.33 60.00 66.67 6.67 66.67 40.00 20.00 40.00 -
13.33 33.33 20.00 0.00 13.33 34.87 PHP 66.67 53.33 53.33 33.33 60.00 46.67
60.00 60.00 60.00 - 46.67 20.00 6.67 6.67 44.10 Rust 60.00 73.33 26.67 53.33
46.67 26.67 33.33 13.33 40.00 6.67 - 33.33 0.00 20.00 33.33 Perl 66.67 53.33
26.67 40.00 53.33 53.33 33.33 46.67 66.67 53.33 66.67 - 13.33 13.33 45.13 D
66.67 86.67 53.33 33.33 86.67 60.00 40.00 53.33 80.00 60.00 46.67 66.67 -
20.00 57.95 Delphi 60.00 53.33 40.00 40.00 40.00 66.67 46.67 20.00 60.00 26.67
60.00 86.67 26.67 - 48.21 Avg. 62.05 62.56 45.13 35.38 52.31 46.15 40.51 44.10
58.46 33.85 50.26 21.03 5.64 6.15 40.26

Table 14: Evaluation result of code translation on easy problems using GPT-4,
employing the Pass@1 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 60.00 20.00 40.00 20.00 6.67 33.33 6.67 13.33 6.67 13.33 0.00 0.00
0.00 16.92 Java 40.00 - 33.33 20.00 20.00 13.33 13.33 6.67 13.33 0.00 20.00
0.00 0.00 0.00 13.85 Python 46.67 46.67 - 20.00 46.67 26.67 46.67 26.67 40.00
40.00 33.33 13.33 6.67 6.67 30.77 C 66.67 46.67 6.67 - 26.67 13.33 20.00 13.33
40.00 13.33 6.67 0.00 0.00 0.00 19.49 C# 46.67 13.33 6.67 13.33 - 6.67 13.33
13.33 33.33 26.67 26.67 0.00 0.00 0.00 15.38 Ruby 26.67 26.67 26.67 13.33
33.33 - 20.00 13.33 33.33 26.67 20.00 26.67 0.00 6.67 21.03 Go 20.00 33.33
6.67 6.67 0.00 0.00 - 20.00 33.33 0.00 6.67 0.00 0.00 0.00 9.74 JavaScript
27.27 9.09 27.27 36.36 18.18 9.09 18.18 - 18.18 27.27 0.00 27.27 9.09 0.00
17.48 Kotlin 46.67 46.67 33.33 20.00 33.33 26.67 20.00 26.67 - 13.33 6.67 0.00
0.00 0.00 21.03 PHP 53.85 23.08 61.54 15.38 38.46 53.85 30.77 30.77 30.77 -
30.77 0.00 0.00 0.00 28.40 Rust 40.00 33.33 53.33 13.33 46.67 26.67 13.33
33.33 33.33 13.33 - 0.00 0.00 0.00 23.59 Perl 53.33 53.33 60.00 40.00 60.00
33.33 46.67 60.00 40.00 33.33 60.00 - 20.00 13.33 44.10 D 33.33 40.00 6.67
26.67 26.67 20.00 40.00 20.00 26.67 13.33 26.67 33.33 - 6.67 24.62 Delphi
60.00 53.33 20.00 33.33 13.33 20.00 20.00 13.33 0.00 6.67 20.00 13.33 13.33 -
22.05 Avg. 43.16 37.35 27.86 22.95 29.49 19.71 25.82 21.85 27.36 16.97 20.83
8.76 3.78 2.56 22.03

Table 15: Evaluation result of code translation on hard problems using GPT-4,
employing the Pass@1 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 33.33 13.33 20.00 0.00 0.00 6.67 0.00 6.67 13.33 6.67 0.00 0.00
0.00 7.69 Java 20.00 - 0.00 13.33 0.00 0.00 6.67 0.00 6.67 13.33 0.00 0.00
0.00 0.00 4.62 Python 20.00 13.33 - 0.00 13.33 13.33 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 4.62 C 33.33 13.33 6.67 - 0.00 6.67 0.00 0.00 6.67 13.33 0.00
0.00 0.00 0.00 6.15 C# 20.00 6.67 26.67 6.67 - 13.33 0.00 0.00 0.00 6.67 6.67
0.00 0.00 0.00 6.67 Ruby 0.00 0.00 6.67 0.00 0.00 - 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.51 Go 20.00 6.67 13.33 20.00 6.67 6.67 - 0.00 26.67 0.00 0.00
0.00 0.00 0.00 7.69 JavaScript 13.33 13.33 13.33 6.67 0.00 0.00 0.00 - 6.67
0.00 0.00 6.67 0.00 0.00 4.62 Kotlin 0.00 0.00 6.67 0.00 0.00 6.67 0.00 0.00 -
0.00 6.67 0.00 0.00 0.00 1.54 PHP 13.33 6.67 13.33 20.00 0.00 20.00 0.00 0.00
20.00 - 0.00 0.00 0.00 0.00 7.18 Rust 6.67 20.00 6.67 6.67 6.67 6.67 0.00 0.00
6.67 0.00 - 0.00 0.00 0.00 4.62 Perl 6.67 6.67 13.33 0.00 0.00 6.67 0.00 6.67
6.67 6.67 0.00 - 0.00 0.00 4.10 D 20.00 26.67 13.33 13.33 26.67 6.67 6.67 0.00
13.33 6.67 13.33 6.67 - 0.00 11.79 Delphi 26.67 26.67 26.67 13.33 0.00 0.00
6.67 0.00 6.67 6.67 0.00 0.00 0.00 - 8.72 Avg. 15.38 13.33 12.31 9.23 4.10
6.67 2.05 0.51 8.21 5.13 2.56 1.03 0.00 0.00 5.75

Table 16: Evaluation result of code translation on easy problems using
StarCoder, employing the Pass@1 metric.

from\to C++ Java Python C C# Ruby Go JavaScript Kotlin PHP Rust Perl D Delphi
Avg. C++ - 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.67 0.00 0.00 0.00 0.00 0.00
0.51 Java 6.67 - 0.00 0.00 0.00 0.00 0.00 0.00 13.33 0.00 0.00 0.00 0.00 0.00
1.54 Python 0.00 0.00 - 0.00 0.00 0.00 0.00 0.00 0.00 6.67 0.00 0.00 0.00 0.00
0.51 C 33.33 0.00 0.00 - 0.00 0.00 0.00 0.00 0.00 6.67 0.00 0.00 0.00 0.00
3.08 C# 6.67 6.67 0.00 0.00 - 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1.03 Ruby 6.67 0.00 20.00 0.00 0.00 - 0.00 0.00 0.00 0.00 0.00 6.67 0.00 0.00
2.56 Go 0.00 0.00 0.00 0.00 0.00 0.00 - 0.00 0.00 0.00 6.67 0.00 0.00 0.00
0.51 JavaScript 0.00 0.00 0.00 0.00 0.00 0.00 0.00 - 9.09 0.00 0.00 0.00 0.00
0.00 0.70 Kotlin 13.33 6.67 0.00 6.67 0.00 6.67 0.00 0.00 - 0.00 0.00 0.00