✓✓ ✗✗

# Riemannian Prediction of Anatomical Diagnoses in Congenital Heart Disease
based on 12-lead ECGs

###### Abstract

Congenital heart disease (CHD) is a relatively rare disease that affects
patients at birth and results in extremely heterogeneous anatomical and
functional defects. 12-lead ECG signal is routinely collected in CHD patients
because it provides significant biomarkers for disease prognosis. However,
developing accurate machine learning models is challenging due to the lack of
large available datasets. Here, we suggest exploiting the Riemannian geometry
of the spatial covariance structure of the ECG signal to improve
classification. Firstly, we use covariance augmentation to mix samples across
the Riemannian geodesic between corresponding classes. Secondly, we suggest to
project the covariance matrices to their respective class Riemannian mean to
enhance the quality of feature extraction via tangent space projection. We
perform several ablation experiments and demonstrate significant improvement
compared to traditional machine learning models and deep learning on ECG time
series data.

Index Terms—  CHD, ECG, DL, Riemannian geometry.

##  1 Introduction

12-lead ECG is a very common diagnostic and prognostic tool in cardiac
diseases. The reason behind this is that it is easy to acquire and it reflects
cardiac anatomy and function in high spatio-temporal resolution. In fact,
recent work showed that 12-lead ECGs can be converted into 3D representations
of the direct cardiac activity during a heart beat based on vectorcardiograms
(VCGs) [1, 2]. An example of these representations is shown in Figure 1 and it
reveals signatures of anatomical defects in Congenital Heart Disease (CHD)
based on our data. Several deep learning techniques have been proposed to
classify cardiac rhythms and estimate the risk of adverse effects [3]. These
methods showed impressive results with large datasets that include millions of
patients and ECG recordings. However, it is not clear how they can extend in
relatively rare and extremely heterogeneous cases.

Fig. 1: Average VCGs across anatomical defects in CHD.

In congenital heart disease (CHD) abnormalities in structure and function are
present at birth and affect around $1\%$ of babies. In other words, patients
are born with genetic defects that differ significantly from the cardiac
abnormalities emerging later in life. Therefore, the efficiency of deep
learning methods developed on a broader population is limited due to lack of
large scale representative data and extreme physiological variations in both
anatomy and function.

Inspired by successful work on Riemannian classification [4], we proposed to
use the covariance structure of the 12-lead ECGs to predict anatomic diagnosis
associated with CHD as an initial step toward mortality prediction. Our
contributions are:

  * •

We exploit multiple class-dependent tangent spaces to project covariances
matrices to their class mean and extract more coherent features for
classification.

  * •

We exploit a covariance mixing regularisation technique for augmentation.
Similar to mixup approach that interpolates samples in a linear way based on a
factor $\alpha$, this approach performs the interpolation on a Riemannian
manifold with respect to the underlying covariance matrices.

  * •

We validate our method in ECGs obtained from patients with congenital heart
disease, which represent an example of extremely small and imbalanced dataset.

##  2 Methods

###  2.1 Data

Fig. 2: Multiple tangent space concept.

4153 12-lead ECGs were extracted from 436 patients (194 female) with CHD under
regular follow-up at the Scottish Adult Congenital Cardiac Service based at
the Golden Jubilee National Hospital in Scotland. ECGs in atrial flutter or
atrial fibrillation were excluded, as were atrioventricular paced rhythms, as
one of our primary aims was to use ECGs in sinus rhythm to predict diagnosis.
The most common condition was tetralogy of fallot (ToF, 39.9%), followed by
atrial septal defect (ASD, 17.6%) and pulmonary atresia (PA, 16.7%). Mean ECG
age was 33 years (SD 11.7 and 75-25% IQR(40,23)). Patients with no documented
ECGs or those in atrial flutter or fibrillation or other heart rhythm
abnormalities including being paced, at the time of the ECG were excluded. For
patients with more than one anatomic diagnosis, the dominant diagnosis was
considered the primary diagnosis. Extracted 436 patients are summarised as
follows: 173 patients with ToF, 77 patients with ASD, 73 patients with PA, 66
patients with Fontan and 47 patients with Mustard.

###  2.2 From Common to Multiple Tangent Spaces

ECG signal $Y$ is described as a $12\times n$ time series data. For Riemannian
manifold classifier, the covariance structure of the multichannel (12-lead)
ECG signal is estimated and projected into a flat space. This process allows
more accurate estimation of linear operations. We applied spatial filtering
$F$ to enhance the signal-to-noise ratio and remove the artifacts [5]. Then,
the covariance matrix is estimated as $\mathbf{C}=F(Y)\cdot F(Y)^{\intercal}$
which reflects the correlations between each pair of the leads. Since
covariance matrices are Symmetric Positive Definite (SPD) matrices, they must
be analysed in a Riemannian manifold rather than Euclidean space. Riemannian
metric rather than Euclidean metric is used to project covariance matrices
onto tangent space while respecting their geometry [6]. To achieve this,
typically, the covariance matrices are projected onto a common tangent space
based on Equation 1 [4].

$\mathbf{V_{i}^{C}}={\rm upper}\left({\bf C}^{-{\frac{1}{2}}}{\rm Log}_{{\bf
C}}\left({\bf C}_{i}\right){\bf C}^{-{\frac{1}{2}}}\right)$ (1)

Data: 12-lead ECG signals

Result: Anatomic diagnosis associated with CHD

1 initialise SPLO splits and align R Peaks;

2 while  _there is a 12-lead ECG_ do

3 calculate covariance matrices based on the

4 spatial filtering $F$;

5 if  _mixup_ then

6 mixup on covariance matrices using Eq. 2;

7

8 if  _projection_ then

9 if  _multiple tangent space_ then

10 fit a tangent space for each class;

11 project covariance matrices to those tangent spaces using Eq. 1;

12 combine each output to get a feature vector to train on the model

13 else

14 fit only a single tangent space;

15 project covariance matrices to the

16 tangent space using Eq. 1;

17 get feature vector to train on the model

18

19 make classification;

20

Algorithm 1 Prediction of anatomical diagnoses

Where $\bf C_{i}$ is the covariance matrix to be projected onto the tangent
space at point $\bf C$, which represents the Riemannian mean of all the
covariance matrices. This projection enhances the performance of classifiers
that depend on distance metrics between the sample covariance matrices and it
has been successful in processing high-dimensional neurophysiological data [6,
7, 8]. However, it assumes that $\bf C$ and $\bf C_{i}$ are relatively close.

Here, we hypothesise that projecting each covariance matrix to its
corresponding class mean will improve the quality of the mapping, since the
distance will be smaller compared to the corresponding distance with the
global Riemannian mean. Thus, we fitted a different tangent space for each
class and then combined each of the outputs into a new feature vector. After
tangent space projection, each covariance matrices is represented as a vector
$\mathbf{V}$ of size $n\times(n+1)/2$, where $n$ is the dimension of the
covariance matrices. Each covariance matrix is mapped into tangent space by
keeping the upper triangular part of the resulting symmetric matrix as denoted
in Equation 1.

Also, an illustration of the multiple tangent space concept can be seen in
Figure 2. Each output of the tangent spaces can be combined into a single
enriched feature vector that is fed as input to the classification model. It
also allows us to balance potential data issues by using covariance
augmentation, described below, on feature vectors of the underrepresented
classes. All the algorithmic steps of our proposed approach are demonstrated
in Algorithm 1.

###  2.3 Augmentation of Covariance Matrices

Since we have limited datasets, we use a covariance mixing technique to
generate more samples in a controlled way similar to [9]. To apply mixing, we
sample $\alpha$ from a beta distribution on the interval [0, 1], and compute
the weighted Riemannian mean according to the Riemannian distance metric
between the randomly selected covariance matrices. We also tried to control
mixing by restricting the range of sampled values for $\alpha$, but the best
results were obtained with no restrictions. Riemannian mean that minimises the
sum of squared Riemannian distances to the given two SPD matrices was
calculated to find the weighted Riemannian mean as in Equation 2, where
$w_{i}$ represents a weight matrix generated using the $\alpha$ value, $d_{R}$
represents the Riemannian distances to the SPD matrices.

$\mathbf{C_{aug}}=\arg\min_{\mathbf{C}}\sum_{i}w_{i}\
d_{R}(\mathbf{C},\mathbf{C}_{i})^{2}$ (2)

Instead of mixing all the data we have, we tried to focus only on the classes
that are not easily distinguishable by the model. t-SNE visualisations [10] on
the tangent space using only original data, and mixed data combined with the
original are shown in Figures 3a and 3b, respectively.

(a) t-SNE on tangent space

(b) t-SNE with covariance augmentations on tangent space

Fig. 3: t-SNE visualizations on the tangent space.

##  3 Results

12-lead ECG data were extracted from ECG PDF documents obtained via the
Marquette™ 12SL by GE Healthcare analysis program. Subsequently, ECG data were
digitised and pre-processed to align the R peak points across heart beats for
all patients. The average length of raw ECGs is 2.5 seconds and R peak aligned
ECGs is around 1 second, and they are sampled at a rate of 500 samples per
second. Aligned ECG data gave better results than raw ECG data, regardless of
input or model type.

Training-testing split of data was repeated 100 times based on pseudo-
randomised, stratified patient leave-out (SPLO) evaluation to ensure that the
testing set of patients was representative of all the classes. One patient for
each class is randomly selected for testing. All of testing patients’ ECG data
are removed from training and only data from the rest of patients are used in
the training. This training procedure is repeated 100 times in a pseudo-
randomised manner and the average performance results (accuracy, AUC and F1
macro) along with standard deviation are reported. On average, each patient
has 10 ECG recordings (for a very small number of patients this number can
vary from 2 to 40). Firstly, the data are split into different sets and
subsequently further processing, such as covariance augmentations, is
performed only on the training data.

Three different classifiers were used to compare our feature extraction
framework: Support Vector Machine (SVM), Minimum Distance to Means (MDM), and
Multilayer Perceptron (MLP). We have done ablation studies with MLP model as
well with baseline models SVM and MDM. MDM performs classification by the
nearest centroid. A centroid of the covariance matrices is estimated for each
of the classes and then, for each new covariance sample, the class is
estimated according to the nearest centroid. MLP model was trained with the
different feature extraction frameworks as mentioned in Table 1. Table 1 shows
the different strategies applied for augmentation and covariance projection.
$(DEF)$ denotes application of the classification directly on the R peak
aligned 12-lead ECG data as the default setting. ($VCG$) represents
augmentations based on ECG vectorcardiogram space. Dower transformation was
used to project 12-lead ECG data into a 3-dimensional VCG space. Subsequently,
we applied rotations along all three orthogonal axes from 5 to 45 degrees and
projected the augmented VCG back into 12-lead ECG space [2]. ($VCG$)
augmentation performed poorly in classifying covariance matrices
$(VCG\\_COV)$. In the next ablation steps, $(COV)$ represents that covariance
matrices were used. The final step was the projection of these matrices into
tangent space (single: $TS$ or multiple: $MTS$). The best results were
obtained using multiple tangent spaces with covariance augmentations
$(MTS\\_COV)$.

Table 1: Ablation study: AUC scores for different methods.

Name | Feature Extraction Framework |  AUC (MLP) |  AUC (SVM) |  AUC (MDM)
---|---|---|---|---
| VCG
---
Augmentations
| Covariance
---
Matrix
| Covariance
---
Augmentations
| Tangent
---
Space
$DEF$ | ✗ |  |  |  | 0.73 ± 0.05 | 0.80 ± 0.08 | NA
$VCG$ | ✓ |  |  |  | 0.76 ± 0.08 | 0.59 ± 0.04 | NA
$COV$ | ✗ | ✓ |  |  | 0.77 ± 0.08 | 0.50 ± 0.09 | 0.76 ± 0.09
$VCG\\_COV$ | ✓ | ✓ |  |  | 0.50 ± 0.00 | 0.50 ± 0.00 | 0.50 ± 0.00
$TS$ | ✗ | ✓ | ✗ | Single | 0.82 ± 0.08 | 0.72 ± 0.07 | 0.80 ± 0.07
$TS\\_COV$ | ✗ | ✓ | ✓ | Single | 0.83 ± 0.07 | 0.77 ± 0.06 | 0.82 ± 0.06
$MTS$ | ✗ | ✓ | ✗ | Multiple | 0.81 ± 0.09 | 0.78 ± 0.08 | NA
MTS_COV | ✗ | ✓ | ✓ | Multiple | 0.84 ± 0.06 | 0.82 ± 0.08 | NA

Table 2: Performance metrics of the machine learning models. With bold we
highlight the best six models.

| Accuracy | AUC | F1 macro
---|---|---|---
|  MLP $(MTS\\_COV)$
---
0.71 ± 0.10 | 0.84 ± 0.06 | 0.69 ± 0.13
|  MLP $(MTS)$
---
0.64 ± 0.17 | 0.81 ± 0.09 | 0.63 ± 0.18
|  MLP $(TS\\_COV)$
---
0.69 ± 0.12 | 0.83 ± 0.07 | 0.65 ± 0.13
|  MLP $(TS)$
---
0.66 ± 0.15 | 0.82 ± 0.08 | 0.63 ± 0.17
|  MLP $(VCG\\_COV)$
---
0.17 ± 0.06 | 0.50 ± 0.00 | 0.05 ± 0.01
|  MLP $(VCG)$
---
0.62 ± 0.13 | 0.76 ± 0.08 | 0.54 ± 0.15
|  MLP $(DEF)$
---
0.55 ± 0.12 | 0.73 ± 0.05 | 0.48 ± 0.09
|  SVM $(MTS\\_COV)$
---
0.66 ± 0.17 | 0.82 ± 0.08 | 0.65 ± 0.17
|  SVM $(MTS)$
---
0.61 ± 0.13 | 0.78 ± 0.08 | 0.56 ± 0.16
|  SVM $(TS\\_COV)$
---
0.59 ± 0.10 | 0.77 ± 0.06 | 0.55 ± 0.11
|  SVM $(TS)$
---
0.56 ± 0.12 | 0.72 ± 0.07 | 0.53 ± 0.12
|  SVM $(VCG\\_COV)$
---
0.17 ± 0.06 | 0.50 ± 0.00 | 0.05 ± 0.01
|  SVM $(DEF_{2D})$
---
0.65 ± 0.14 | 0.80 ± 0.08 | 0.62 ± 0.15
|  MDM $(TS\\_COV)$
---
0.67 ± 0.14 | 0.82 ± 0.06 | 0.66 ± 0.11
|  MDM $(TS)$
---
0.65 ± 0.15 | 0.80 ± 0.07 | 0.63 ± 0.13
|  MDM $(VCG\\_COV)$
---
0.17 ± 0.06 | 0.50 ± 0.00 | 0.05 ± 0.01
|  MDM $(COV)$
---
0.63 ± 0.15 | 0.76 ± 0.09 | 0.56 ± 0.17

Table 2 provides a comparison on the performance metrics (Accuracy, AUC, F1
macro) of the approaches described in ablation Table 1 as well as a baseline
model with SVM applied on the aligned ECG time-series data $DEF_{2D}$. Using
tangent space projection on MLP resulted in a 9% increase in the AUC score and
an 11% increase in the accuracy. Using covariance augmentations on the single
tangent space also yielded a further increase of 1% in the AUC score and a 3%
increase in the accuracy. Taking a step further and using multiple tangent
space projections with covariance augmentations resulted in an 11% increase in
the AUC score and a 16% increase in the accuracy. Better results were obtained
by projecting the augmented covariance matrices into multiple tangent space
while respecting their underlying geometry.

Both SVM and MLP models were trained on R peak aligned ECG data $(DEF)$, but
for the SVM model 3D ECG data were reshaped to 2D $(DEF_{2D})$. SVM model
achieved 7% better results at this stage for the AUC score. Also, some 3D
rotations on the ECG data were tried by projecting it to the VCG space with
the help of Dower transformations. Only MLP model was trained on these 3D
augmented ECG data $(VCG)$ and achieved 3% better results for the AUC score.
But, they caused very poor results with the next steps that include covariance
matrices $(VCG\\_COV)$. VCG means without any augmentations for each class can
be seen in Figure 1.

Figure 4 shows how the top six models compare statistically where statistical
significance is based on corrected paired t-test [11]. All the models except
SVM achieved better results on the tangent space and peaked at an AUC score of
82%. There was a statistically significant difference between the AUC results
of the $MLP(MTS\\_COV)$ model compared to the other models in Figure 4. As
shown in Table 2, using multiple tangent spaces provided an improvement of 11%
for the MLP model and 2% for the SVM model in the AUC score.

Fig. 4: AUC scores for the best six approaches.

There was a statistically significant difference between the AUC results of
the $MLP(MTS\\_COV)$ model compared to the other models in Figure 4. The
confusion matrices in Figure 5a, 5b and 5c, reflect the improvement in
performance with the application of the multiple tangent space projection and
covariance augmentation. Furthermore, Figure 6 shows AUC score for each class
on the best model $MTS\\_COV$, evaluated using One-vs-Rest (OvR) strategy.

(a) MLP

(b) MLP on tangent
space

(c) MLP on multiple
tangent space

Fig. 5: Confusion matrices of $DEF$, $TS\\_COV$ and $MTS\\_COV$ approaches
with an MLP classifier.

Fig. 6: AUC for the best model ($MTS\\_COV$).

##  4 Conclusions

We demonstrated promising results on 12-lead ECG classification of anatomical
diagnosis in congenital heart disease. Our proposed projection of the
augmented covariance matrices to multiple Riemannian spaces yields
significantly better results in improving classification performance with
small and extremely imbalanced 12-lead ECG data.

##  5 Compliance with Ethical Standards

This study was performed in line with the principles of the Declaration of
Helsinki. Study approval was obtained from the Institutional Governance
Division of the NHS Golden Jubilee National Hospital. This work was supported
by the School of Computing Science at University of Glasgow and Golden Jubilee
National Hospital leadership and governance teams. F.D. is supported by
funding from grant EP/W01212X/1. None of the authors have anything to disclose
relevant to this work. We will make our code publicly available to enable
reproducibility.

## References

  * [1] Peter M van Dam, Machteld Boonstra, Emanuela T Locati, and Peter Loh,  “The relation of 12 lead ecg to the cardiac anatomy: the normal cineecg,”  Journal of Electrocardiology, vol. 69, pp. 67–74, 2021.
  * [2] Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar,  “3kg: Contrastive learning of 12-lead electrocardiograms using physiologically-inspired augmentations,”  in Machine Learning for Health. PMLR, 2021, pp. 156–167.
  * [3] Antônio H Ribeiro, Manoel Horta Ribeiro, Gabriela MM Paixão, Derick M Oliveira, Paulo R Gomes, Jéssica A Canazart, Milton PS Ferreira, Carl R Andersson, Peter W Macfarlane, Wagner Meira Jr, et al.,  “Automatic diagnosis of the 12-lead ecg using a deep neural network,”  Nature communications, vol. 11, no. 1, pp. 1760, 2020.
  * [4] Florian Yger, Maxime Berar, and Fabien Lotte,  “Riemannian approaches in brain-computer interfaces: a review,”  IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, no. 10, pp. 1753–1762, 2016.
  * [5] Bertrand Rivet, Antoine Souloumiac, Virginie Attina, and Guillaume Gibert,  “xdawn algorithm to enhance evoked potentials: application to brain–computer interface,”  IEEE Transactions on Biomedical Engineering, vol. 56, no. 8, pp. 2035–2043, 2009.
  * [6] Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten,  “Multiclass brain–computer interface classification by riemannian geometry,”  IEEE Transactions on Biomedical Engineering, vol. 59, no. 4, pp. 920–928, 2011.
  * [7] Marco Congedo, Alexandre Barachant, and Rajendra Bhatia,  “Riemannian geometry for eeg-based brain-computer interfaces; a primer and a review,”  Brain-Computer Interfaces, vol. 4, no. 3, pp. 155–174, 2017.
  * [8] Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten,  “Classification of covariance matrices using a riemannian-based kernel for bci applications,”  Neurocomputing, vol. 112, pp. 172–178, 2013.
  * [9] Georgios Zoumpourlis and Ioannis Patras,  “Covmix: Covariance mixing regularization for motor imagery decoding,”  in 2022 10th International Winter Conference on Brain-Computer Interface (BCI). IEEE, 2022, pp. 1–7.
  * [10] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne.,”  Journal of machine learning research, vol. 9, no. 11, 2008.
  * [11] Claude Nadeau and Yoshua Bengio,  “Inference for the generalization error,”  Advances in neural information processing systems, vol. 12, 1999.


# Benchmarking Educational Program Repair

Charles Koutcheme
Aalto University
Espoo, Finland
<EMAIL_ADDRESS>
&Nicola Dainese
Aalto University
Espoo, Finland
<EMAIL_ADDRESS>
&Sami Sarsa
Aalto University
Espoo, Finland
<EMAIL_ADDRESS>&Juho Leinonen
University of Auckland
Auckland, New Zealand
<EMAIL_ADDRESS>
&Arto Hellas
Aalto University
Espoo, Finland
<EMAIL_ADDRESS>
&Paul Denny
University of Auckland
Auckland, New Zealand
<EMAIL_ADDRESS>
###### Abstract

The emergence of large language models (LLMs) has sparked enormous interest
due to their potential application across a range of educational tasks. For
example, recent work in programming education has used LLMs to generate
learning resources, improve error messages, and provide feedback on code.
However, one factor that limits progress within the field is that much of the
research uses bespoke datasets and different evaluation metrics, making direct
comparisons between results unreliable. Thus, there is a pressing need for
standardization and benchmarks that facilitate the equitable comparison of
competing approaches. One task where LLMs show great promise is program
repair, which can be used to provide debugging support and next-step hints to
students. In this article, we propose a novel educational program repair
benchmark. We curate two high-quality publicly available programming datasets,
present a unified evaluation procedure introducing a novel evaluation metric
rouge@k for approximating the quality of repairs, and evaluate a set of five
recent models to establish baseline performance.

##  1 Introduction

In education, the emergence of large language models (LLMs) has led to a
plethora of research evaluating their performance on educationally relevant
tasks (28; 11). LLMs have been explored for educational tasks such as
providing automatic feedback to students (52; 30; 21), generating novel
exercises (59), and producing novice-friendly explanations of source code (45;
37). Within computing education, one long-standing challenge has been
generating automated feedback on students’ programs (29). This has
traditionally been achieved using unit-test-based automated assessment systems
(51; 26), or intelligent tutoring systems that provide hints on the next steps
that students should take (18; 57; 56). A key parallel stream to this research
is ‘automated program repair’, which employs rule-based and machine-learning-
based strategies for automatically fixing bugs in code (55; 34; 27; 2; 53).
Drawing from this extensive body of prior research, we recognize automated
program repair as a crucial intermediary step toward enhancing the accuracy
and reliability of feedback for novice programmers. Currently, substantial
interest lies in harnessing LLMs for generating feedback based on program
repairs (72; 55; 34).

In general, much of the prior work exploring LLMs in education has used
bespoke datasets due to the lack of widely used benchmarks. This makes it
difficult to meaningfully compare results and identify effective techniques.
Benchmarks are an essential tool in computer science and software engineering,
including machine learning. They provide a standardized method for comparing
various techniques and for monitoring progress within an area over time,
driving innovation and improvement. Datasets in particular have served as
benchmarks for comparing algorithms in a reproducible way which is the bedrock
of modern science. For example, within the field of machine learning standard
benchmarks such as ImageNet (9), MNIST (36) and HumanEval (7) have been
crucial for identifying breakthroughs in algorithms for image recognition and
program synthesis.

In this paper, we propose a framework for evaluating program repair techniques
using LLMs. By providing a benchmark for this task, we hope to facilitate
future research by allowing easy replication of results and evaluation of new
models using the same data that older models have been evaluated with. In
particular, this work makes the following contributions: we (1) formalize the
task of educational program repair, (2) describe an evaluation procedure
including a novel metric for the quality of program repairs called rouge@k,
(3) curate two high-quality and publicly available datasets suitable for
serving as a benchmark to evaluate program repairs, and (4) report the
performance of several recent decoder-only transformer models on these
datasets.

##  2 Related Work

##### Benchmarking program synthesis.

In assessing the capabilities of code language models across various contexts,
programming benchmarks have become a common tool (7; 3; 22; 35). However, when
it comes to program synthesis, few of these benchmarks offer a large number of
problems that are suitable for introductory programming courses (3; 22). While
code language models have traditionally been used in industry and research,
the emergence of highly proficient easily accessible language models such as
ChatGPT has sparked interest in Computing Education Research (CER). In line
with program synthesis principles, extensive research in CER has explored
these models’ abilities to solve real-world introductory programming problems
under authentic course conditions, moving beyond artificial examples found in
standard code benchmarks (15; 16; 10; 69; 61). We notice a similar transition
happening with program repair.

##### Benchmarking program repair.

The task of program repair is often approached as a synthesis problem. In our
study, we draw inspiration from this paradigm to design components of our
evaluation procedure. Similarly to program synthesis, there exist program
refinement benchmarks (43; 42) and particularly close to our work is a recent
extension (47) to the human evaluation benchmark (7) that introduces bugs in
code for LLMs to rectify. Although the bugs in all these datasets are relevant
to student programming, they do not fully encapsulate the complexities of
errors encountered in student-written code, which can present a wide array of
issues extending beyond mere bugs. Consequently, while such refinement
benchmarks assume the presence of ground truth annotations, our evaluation
procedure operates without assuming the existence of a single ground truth
repair, and, in fact, did not necessitate manual annotations.

Our work also builds upon recent efforts utilizing large language models to
provide feedback and repair programs for students. Notably, LLMs have shown
great promise in fixing syntax errors (27; 2; 53; 38) and rectifying bugs in
student programs (72), and preliminary work exists on using them to
automatically generate feedback (21; 30; 52; 40; 71). While most of these
endeavours leverage closed-source systems such as OpenAI Codex, some explore
the use of open-language models (32; 34). Many of the methods we introduce
align with these previous efforts. In particular, our proposed evaluation
procedure aims to encompass the different scenarios explored in this prior
research while leveraging recent insights into evaluation methodologies (33).
Although some prior work has integrated human judgment into the evaluation
(54; 38), we focus on a fully automated evaluation approach which is more
scalable, primarily relying on program correctness and NLP scoring metrics as
proxies for quality. In addition to presenting the evaluation procedure, we
also identify two publicly available datasets that we curate in such a way
that they are suitable as benchmarks for evaluating program repair.

##  3 Educational Program Repair

In this section, we formalize the educational program repair task for large
language models and present an evaluation procedure for that task (Figure 1
shows an overview). We also introduce two educational scenarios of interest
and propose two high-quality publicly available datasets that match these
scenarios.

Figure 1: Overview of the program repair task and its evaluation procedure.
(1) For each incorrect program, we pass to a language model a prompt which
contains the programming task, and the incorrect student code. In addition,
the prompt may contain more contextual information, e.g. grader’s feedback.
(2) The model generates multiple feedbacks based on the instructions; each
feedback must contain (at least) a full correction to the incorrect program.
(3) We extract the full code repairs, which we (4) pass to unit tests. The
samples which pass or fail the unit tests allow us to evaluate (5) functional
correctness and (6) closeness to the incorrect program.

###  3.1 Task and Evaluation Procedure

##### Problem setup and motivations.

For this task, we consider a student working on a programming assignment. We
have access to the problem description, associated unit tests, and a grader
that employs these unit tests to provide summative feedback to students. When
presented with an incorrect program that doesn’t pass all unit tests, our goal
is to utilize a language model to generate a repair to the student’s code,
addressing all the issues in the incorrect program. In essence, we view
student program repair as a program synthesis problem, conditioned upon an
existing (non-working) solution (72). Our primary objective is to ensure that
the generated programs are not only functionally correct but also closely
aligned with the original incorrect program. This closeness is akin to the
preference for hint-generation systems to provide hints that are in proximity
to the student’s current state of knowledge (18; 57; 56). The basic assumption
is that enforcing this closeness will enhance the comprehensibility of the
associated feedback (12), such as hints generated by an Intelligent Tutoring
System (ITS), or the natural language explanations generated by an LLM (53).

##### Evaluation procedure.

Generally, we consider each buggy program as an independent “problem”, and use
our language model to generate one or multiple repairs for each problem.
Classically, (educational) program repair has reported performance in terms of
the total number of buggy programs successfully repaired (i.e., the total
number of problems solved) and the average edit distance between the buggy
programs and their associated code repairs (72; 53). In this work, we generate
up to $n=10$ different repairs and introduce two distinctions to the classical
evaluation procedure.

###### pass@k

First, following prior program synthesis research, we assess functional
correctness using the pass@k estimator (7). We first generate $n$ repairs.
Then for a given $k$ ($k\leq n$), we generate all combinations of $k$ elements
from the $n$ repairs, and count those with at least one repair passing all
unit tests. Dividing this count by the total combinations gives the
probability of a random sample of $k$ repairs fixing a buggy program. We
average this probability for each program in our test set.

###### rouge@k

Second, recent research (33) suggests that the Rouge score (41) serves as a
better repair quality indicator compared to sequence edit distance. Building
upon this insight, we introduce a novel evaluation metric, ‘rouge@k’ mirroring
‘pass@k’. For each of the passing $n$ model generations, we calculate
Rouge111Rouge embodies a family of metrics. For this study, we use the Rouge-L
(41) score. scores with the buggy program (using the buggy program as the
reference ‘sentence’). Importantly, candidates that fail unit tests receive a
score of 0. For some $k$, we generate all combinations of $k$ scores from the
$n$ candidates and average the highest scores from each combination. We repeat
this for each buggy program in our test set, and average the values to produce
the final rouge@k score.

###  3.2 Educational Scenarios and Datasets

We propose two versions of the program repair task: a version where prior
educational data is available, and one where no prior data is available. For
both scenarios, we propose to rely on publicly available real-life datasets
collected from various institutions. Due to space limitations, we refer the
reader to the original dataset papers (resp. Appendix A.1) for curation (resp.
processing) details.

##### Prior data available.

In this scenario, we assume having access to prior data. Specifically, we
define “prior” data as a repository of both correct and incorrect solutions
submitted by students in preceding iterations of the same programming
assignments, typically offered on a semester basis. Automated Program Repair
has frequently relied on such prior data to construct repairs for students’
buggy programs (25; 19; 46; 66). In our context, access to prior data offers
significant advantages for large language models, whether for few-shot
prompting (72) or supervised fine-tuning (32). Additionally, it facilitates
the proper division of our datasets into training, validation/development, and
testing subsets. This mirrors a real-life scenario in which educational teams
continually enhance their models each semester as they accumulate more data.
For this scenario, we propose harnessing the recently released FalconCode
dataset (8). Beyond its substantial scale, this dataset distinguishes itself
through the presence of associated unit tests, and the inclusion of pertinent
metadata (e.g. assignment difficulty level). Notably, it contains free-form
assignments where students are not necessarily required to write functions.
This change from the focus of prior work to repair functions (34; 72) allows
for a broader evaluation of LLM-based repair techniques. For evaluation
purposes, we propose to utilize a smaller curated subset of 1,305 solutions
for testing. It’s worth noting that code benchmarks with fewer than 1000
problems are common in the field (3; 22; 35; 7), partly due to the
computational and/or financial resources required for LLM-based evaluation
techniques.

##### No prior data available.

In the second version of the problem, we introduce a setup where models are
tasked with fixing programs in a dataset where no prior data is available.
This scenario aligns with contexts such as new courses (typically small) or
courses where data about students’ previous submissions is not retained. To
address this challenge, we propose leveraging a dataset collected at the
National University of Singapore (25). This dataset comprises five assignments
where students are required to write functions to solve specific tasks
involving loops, conditions, and other standard introductory concepts. The
reasons for selecting this dataset include an appropriate size, the selection
of exercises with intermediate levels of difficulty, the presence of
associated unit tests, and a good variety of issues in the submissions. We use
a curated subset of 1238 incorrect submissions.

##  4 Baseline and Experiments

In this section, we present a set of simple baseline models obtained from
fine-tuning a variety of open-source pretrained language models of code. We
leave the evaluation of instruction-tuned (67) and chat models (50) for future
work.

###  4.1 Models

We experiment with several decoder-only transformer models, using the CodeGen
family of models (49) with 350M, and 2B parameters, as well as models from the
StarCoder family (39) with 164M, 1B, and 3B parameters. We chose these models
for their relatively small size which allows them to fit fully (without
quantization) inside a single consumer GPU. Although we could evaluate the
repair capabilities of these models using zero-shot and few-shot prompting
(27), prior work suggests that their relatively small size (less than several
billion parameters) prevents them from fully taking advantage of in-context
examples (5). For this reason, we choose to finetune these models to repair
student programs via supervised learning. Because most educational datasets do
not contain ground truth repairs to students’ incorrect programs, prior work
in neural program repair (20) suggested creating artificial repair examples by
mapping each incorrect program in a dataset to the closest correct program
submitted to the same assignment. Instead of using the closest correct
program, we follow recent work (32) and map each incorrect program to the
correction found by an Automated Repair Tool applied to the same dataset.
Using an Automated Repair Tool (in our case, Refactory (25)) allows for
efficiently selecting a semantically close program while smoothing out many
irrelevant syntactic differences (e.g., variable name conventions, needless
reformulations) between the incorrect program and the chosen one."

We trained our supervised baselines on the FalconCode dataset with the above-
mentioned methodology, using the first two semesters of data as training and
development sets (1299 and 1339 distinct programs respectively). Appendix A.2
details our exact training procedure including prompting strategy, and
hyperparameter tuning. As an additional contribution, we released the code for
conducting our experiments on
GitHub222https://github.com/KoutchemeCharles/gaied_nips23.

###  4.2 Experiments and Results

We report the performance of the supervised baselines on the test set of the
FalconCode dataset. Following (3), we separate the results for two distinct
assignment difficulty levels: “easy” and “hard” (with 767 and 538 incorrect
programs respectively). We also report the performance of the models fine-
tuned on FalconCode when prompted on the 1238 incorrect submissions of the
Singapore dataset. The results for these three scenarios, including pass rates
and Rouge scores, are presented in Table 1(b). We can make the following
observations.

Table 1: Functional and closeness results for our baselines models (higher is
better).

(a) Pass@k for k = ${1,5,10}$

|  | falconcode_easy | falconcode_hard | singapore
---|---|---|---|---
model | size | k = 1 | k = 5 | k = 10 | k = 1 | k = 5 | k = 10 | k = 1 | k = 5 | k = 10
starcoder | 164M | 6.88 | 12.46 | 14.68 | 20.27 | 35.93 | 41.85 | 2.54 | 7.62 | 10.46
codegen-mono | 350M | 12.08 | 23.19 | 27.88 | 13.68 | 28.28 | 33.77 | 3.72 | 12.92 | 18.90
starcoder | 1B | 16.91 | 30.38 | 36.06 | 26.44 | 44.94 | 50.72 | 7.35 | 21.02 | 28.77
codegen-mono | 2B | 16.10 | 25.99 | 30.67 | 13.85 | 25.92 | 30.38 | 11.24 | 24.97 | 31.63
starcoder | 3B | 19.11 | 31.68 | 37.73 | 14.63 | 29.88 | 36.51 | 12.29 | 29.37 | 36.14

(b) Rouge@k for k = ${1,5,10}$

model | size | k = 1 | k = 5 | k = 10 | k = 1 | k = 5 | k = 10 | k = 1 | k = 5 | k = 10
---|---|---|---|---|---|---|---|---|---|---
starcoder | 164M | 5.91 | 10.66 | 12.57 | 12.14 | 23.43 | 28.32 | 2.18 | 6.58 | 9.07
codegen-mono | 350M | 10.56 | 20.14 | 24.12 | 9.06 | 19.32 | 23.50 | 2.49 | 8.66 | 12.92
starcoder | 1B | 14.32 | 25.41 | 29.93 | 17.02 | 30.91 | 35.90 | 5.60 | 15.95 | 21.68
codegen-mono | 2B | 13.70 | 21.91 | 25.66 | 10.22 | 19.14 | 22.70 | 7.63 | 13.59 | 15.91
starcoder | 3B | 15.89 | 26.18 | 31.12 | 10.48 | 21.59 | 26.54 | 7.85 | 19.97 | 25.73

##### Model performance scales with training compute and model sizes.

Within the same architectural family, performance scales with model sizes and
number of generations. However, between different model families, the quality
of the pretraining can make smaller models (e.g., starcoder 1B) outperform
larger ones (e.g., codegen-mono 2B). The generalization capabilities of
language models have been shown to scale smoothly with the amount of computing
used during pretraining in proportion with model size and dataset size (23;
64).

##### Fine-tuned language models can overfit their datasets.

Some models demonstrate superior performance on the ‘hard’ section of the
Falcon-code dataset compared to the ‘easy’ section. We posit that this
divergence stems from a training data imbalance, leading to ‘overfitting’ on
more challenging assignments (63). Notably, overfitting is more pronounced in
smaller models, which may exhibit better performance than larger counterparts
on the ‘hard’ subset (e.g., codegen-mono 350M outperforming Codegen 2B on
Falcon-code hard). However, this advantage does not extend to the smaller
training subset (FalconCode easy) or to the Singapore dataset.

##### Supervised training can transfer across datasets.

Despite being fine-tuned on the FalconCode dataset, our models managed to fix
some of the buggy programs in another dataset featuring distinct problem types
and other unique issues within the incorrect programs. Interestingly, beyond
1B parameters, the models reach similar performance on the FalconCode and
Singapore datasets.

##  5 Discussion and Conclusion

We have introduced a benchmark for program repair using LLMs. However, the
versatility of language models extends beyond program repair; they are also
capable of generating natural language explanations paired with functioning
solutions. Evaluating the quality of these explanations poses a challenge in
the absence of ground truths or manual assessments. Although we cannot
guarantee the accuracy of generated explanations with certainty, we assert
that the relative quality of repairs can act as a reliable proxy for the
relative quality of associated Natural Language Explanations (NLEs) when these
NLEs are generated prior to the repairs. Previous research suggests that, for
a given model, repairs of higher quality that are closely aligned with the
student’s code are more likely to be derived from in-context, high-quality
explanations (68). Conversely, generating natural language explanations from
in-context, high-quality repairs (53; 31) – first generating the repair and
then generating explanations – reduces the risk of introducing inaccuracies or
inventing non-existent issues, thus mitigating the risk of hallucinations (4;
21). Thus, while our primary focus is on program repair, our work also serves
as a step towards improving the feedback that can be provided to students.

While our baseline models provide valuable insights, it’s essential to
recognize the growing presence of powerful language models capable of program
repair without additional training. In the source code modelling landscape, we
observe a division between permissive (open-source) and non-permissive
(proprietary) models. While prior work in educational program repair primarily
focused on using non-permissive models (72; 53; 54; 2; 38), we forecast a
potential shift occurring with the emergence of robust open-source
alternatives (64; 58). Historically, non-permissive models such as ChatGPT and
OpenAI Codex have demonstrated state-of-the-art performance on benchmarks, but
open-source alternatives are beginning to match or surpass them. However,
running such powerful permissive instruction-tuned and chat models requires
significant computational resources (e.g. custom GPUs) due to their size. This
is becoming less of a barrier due to the rise of open-source hosting services
(70), the ability of smaller models to achieve high performance (23), and the
use of Parameter Efficient Finetuning Techniques (PEFT) (24). While non-
permissive models such as ChatGPT or OpenAI Codex offer high performance
without the need for custom hardware, they also raise significant concerns
about privacy and potential non-reliability due to model changes (6).

Prior studies in program repair have yielded valuable insights into the
application of various methods. However, meaningful comparisons between
methods have been hindered by the lack of accessible, high-quality datasets.
To overcome this limitation, we have curated the FalconCode and Singapore
datasets specifically for program repair tasks relevant to educational
settings. Nevertheless, we acknowledge concerns that any individual
educational benchmarking dataset can not fully capture the diversity of
exercises and instructional approaches present in programming classes (65;
44). We feel it is important to emphasize that the purpose of these benchmarks
should not be to establish a rigid hierarchy of method superiority but to
serve as reference points for comparing methods in common scenarios, and for
identifying issues in prior studies (60). We believe that educational AI
researchers can benefit from presenting and evaluating their novel techniques
on both private datasets and the common benchmark, providing a well-rounded
perspective.

Finally, we acknowledge several limitations to our current work. First, both
the FalconCode and Singapore datasets are limited to Python and represent data
from only two institutions. Furthermore, syntax errors are currently not
included due to our curation processes. We are also aware of the potential
risk of contamination, a common challenge in code benchmarks (17). While the
FalconCode dataset is unlikely to be part of any model’s pretraining dataset
(as access requires a manual request via an online form), the Singapore
dataset is readily available in a GitHub repository. Lastly, we recognize that
assessing qualitative performance based solely on code closeness may introduce
biases and may not always align with instructors’ pedagogical objectives, such
as teaching specific programming composition methods or emphasizing certain
coding approaches (see e.g. (48; 14; 62; 13)). To address these limitations,
our future plans include introducing other high-quality datasets that cover a
broad range of programming languages and scenarios (e.g. from online MOOC
courses). We have also identified datasets suitable for evaluating both syntax
and semantic errors, and we are working on annotating a subset of our datasets
to provide accessible ground truth annotations (33). Looking further ahead, we
intend to expand our benchmark to encompass a broader range of educational
programming tasks (54) (e.g. hint generation), and we aim to establish an open
online leaderboard (70) for result tracking. As educational program feedback
presents distinctive challenges and real-life constraints, we anticipate
cross-domain collaborations between educational program feedback research and
pure NLP research to be essential.

## References

  * (1)
  * Ahmed et al. (2022) Toufique Ahmed, Noah Rose Ledesma, and Premkumar Devanbu. 2022.  SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics.  arXiv:2104.14671 [cs.SE]
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021.  Program Synthesis with Large Language Models.  arXiv:2108.07732 [cs.PL]
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity.  arXiv:2302.04023 [cs.CL]
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * Chen et al. (2023) Lingjiao Chen, Matei Zaharia, and James Zou. 2023.  How is ChatGPT’s behavior changing over time?  arXiv:2307.09009 [cs.CL]
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating language Models Trained on Code.  arXiv:2107.03374 [cs.LG]
  * de Freitas et al. (2023) Adrian de Freitas, Joel Coffman, Michelle de Freitas, Justin Wilson, and Troy Weingart. 2023.  FalconCode: A Multiyear Dataset of Python Code Samples from an Introductory Computer Science Course. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 938–944.  https://doi.org/10.1145/3545945.3569822
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Denny et al. (2023a) Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023a.  Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1136–1142.  https://doi.org/10.1145/3545945.3569823
  * Denny et al. (2023b) Paul Denny, James Prather, Brett A Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023b.  Computing Education in the Era of Generative AI.  _arXiv preprint arXiv:2306.02608_ (2023).
  * Duran et al. (2022) Rodrigo Duran, Albina Zavgorodniaia, and Juha Sorva. 2022.  Cognitive Load Theory in Computing Education Research: A Review.  _ACM Trans. Comput. Educ._ 22, 4, Article 40 (sep 2022), 27 pages.  https://doi.org/10.1145/3483843
  * Edwards et al. (2020) John Edwards, Joseph Ditton, Dragan Trninic, Hillary Swanson, Shelsey Sullivan, and Chad Mano. 2020.  Syntax exercises in CS1. In _Proceedings of the 2020 ACM Conference on International Computing Education Research_. 216–226.
  * Felleisen et al. (2018) Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, and Shriram Krishnamurthi. 2018.  _How to design programs: an introduction to programming and computing_.  MIT Press.
  * Finnie-Ansley et al. (2022) James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022.  The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In _Proceedings of the 24th Australasian Computing Education Conference_ (Virtual Event, Australia) _(ACE ’22)_. Association for Computing Machinery, New York, NY, USA, 10–19.  https://doi.org/10.1145/3511861.3511863
  * Finnie-Ansley et al. (2023) James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023.  My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In _Proceedings of the 25th Australasian Computing Education Conference_ (Melbourne, VIC, Australia) _(ACE ’23)_. Association for Computing Machinery, New York, NY, USA, 97–104.  https://doi.org/10.1145/3576123.3576134
  * Golchin and Surdeanu (2023) Shahriar Golchin and Mihai Surdeanu. 2023.  Time Travel in LLMs: Tracing Data Contamination in Large Language Models.  arXiv:2308.08493 [cs.CL]
  * Gross et al. (2014) Sebastian Gross, Bassam Mokbel, Benjamin Paaßen, Barbara Hammer, and Niels Pinkwart. 2014.  Example-based feedback provision using structured solution spaces.  _International Journal of Learning Technology 10_ 9, 3 (2014), 248–280.
  * Gulwani et al. (2018) Sumit Gulwani, Ivan Radiček, and Florian Zuleger. 2018.  Automated Clustering and Program Repair for Introductory Programming Assignments.  http://arxiv.org/abs/1603.03165 arXiv:1603.03165 [cs].
  * Gupta et al. (2019) Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019.  _Neural Attribution for Semantic Bug-Localization in Student Programs_.  Curran Associates Inc., Red Hook, NY, USA.
  * Hellas et al. (2023) Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023.  Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In _Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1_ (Chicago, IL, USA) _(ICER ’23)_. Association for Computing Machinery, New York, NY, USA, 93–105.  https://doi.org/10.1145/3568813.3600139
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Coding Challenge Competence With APPS.  arXiv:2105.09938 [cs.SE]
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training Compute-Optimal Large Language Models.  arXiv:2203.15556 [cs.CL]
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-Efficient Transfer Learning for NLP.  arXiv:1902.00751 [cs.LG]
  * Hu et al. (2019) Yang Hu, Umair Z. Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019.  Re-factoring based Program Repair applied to Programming Assignments. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE/ACM, 388–398.
  * Ihantola et al. (2010) Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010.  Review of recent systems for automatic assessment of programming assignments. In _Proceedings of the 10th Koli calling international conference on computing education research_. 86–93.
  * Joshi et al. (2022) Harshit Joshi, José Cambronero, Sumit Gulwani, Vu Le, Ivan Radicek, and Gust Verbruggen. 2022.  Repair Is Nearly Generation: Multilingual Program Repair with LLMs.  arXiv:2208.11640 [cs.SE]
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023\.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and individual differences_ 103 (2023), 102274.
  * Keuning et al. (2018) Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018.  A systematic literature review of automated feedback generation for programming exercises.  _ACM Transactions on Computing Education (TOCE)_ 19, 1 (2018), 1–43.
  * Kiesler et al. (2023) Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023.  Exploring the Potential of Large Language Models to Generate Formative Programming Feedback.  _arXiv preprint arXiv:2309.00029_ (2023).
  * Koutcheme (2022) Charles Koutcheme. 2022.  Towards Open Natural Language Feedback Generation for Novice Programmers Using Large Language Models. In _Proceedings of the 22nd Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’22)_. Association for Computing Machinery, New York, NY, USA, Article 29, 2 pages.  https://doi.org/10.1145/3564721.3565955
  * Koutcheme (2023) Charles Koutcheme. 2023.  Training Language Models for Programming Feedback Using Automated Repair Tools. In _Artificial Intelligence in Education_ , Ning Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, and Vania Dimitrova (Eds.). Springer Nature Switzerland, Cham, 830–835.
  * Koutcheme et al. (2023a) Charles Koutcheme, Sami Sarsa, Juho Leinonen, Lassi Haaranen, and Arto Hellas. 2023a.  Evaluating Distance Measures for Program Repair. In _Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1_ (Chicago, IL, USA) _(ICER ’23)_. Association for Computing Machinery, New York, NY, USA, 495–507.  https://doi.org/10.1145/3568813.3600130
  * Koutcheme et al. (2023b) Charles Koutcheme, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny. 2023b.  Automated Program Repair Using Generative Models for Code Infilling. In _Artificial Intelligence in Education_ , Ning Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, and Vania Dimitrova (Eds.). Springer Nature Switzerland, Cham, 798–803.
  * Lai et al. (2022) Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022.  DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation.  arXiv:2211.11501 [cs.SE]
  * LeCun et al. (2010) Yann LeCun, Corinna Cortes, and CJ Burges. 2010.  MNIST handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ 2 (2010).
  * Leinonen et al. (2023a) Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023a.  Comparing Code Explanations Created by Students and Large Language Models. In _Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1_ (Turku, Finland) _(ITiCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 124–130.  https://doi.org/10.1145/3587102.3588785
  * Leinonen et al. (2023b) Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023b.  Using language Models to Enhance Programming Error Messages. In _Proceedings of the 2023 ACM SIGCSE Technical Symposium on Computer Science Education_.  https://doi.org/10.1145/3545945.3569770
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  StarCoder: may the source be with you!  arXiv:2305.06161 [cs.CL]
  * Liffiton et al. (2023) Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023.  CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes.  arXiv:2308.06921 [cs.CY]
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017.  QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge. In _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity_ (Vancouver, BC, Canada) _(SPLASH Companion 2017)_. Association for Computing Machinery, New York, NY, USA, 55–56.  https://doi.org/10.1145/3135932.3135941
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation.  arXiv:2102.04664 [cs.SE]
  * Luxton-Reilly et al. (2018) Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A Becker, Michail Giannakos, Amruth N Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, et al. 2018\.  Introductory programming: a systematic literature review. In _Proceedings companion of the 23rd annual ACM conference on innovation and technology in computer science education_. 55–106.
  * MacNeil et al. (2023) Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023.  Experiences from using code explanations generated by large language models in a web software development e-book. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_. 931–937.
  * Malysheva and Kelleher (2022) Yana Malysheva and Caitlin Kelleher. 2022.  An Algorithm for Generating Explainable Corrections to Student Code. In _Proceedings of the 22nd Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’22)_. Association for Computing Machinery, New York, NY, USA, Article 13, 11 pages.  https://doi.org/10.1145/3564721.3564731
  * Muennighoff et al. (2023) Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023.  OctoPack: Instruction Tuning Code Large Language Models.  arXiv:2308.07124 [cs.CL]
  * Nelson et al. (2017) Greg L Nelson, Benjamin Xie, and Amy J Ko. 2017.  Comprehension first: evaluating a novel pedagogy and tutoring system for program tracing in CS1. In _Proceedings of the 2017 ACM conference on international computing education research_. 2–11.
  * Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023.  CodeGen: An Open language Model for Code with Multi-Turn Program Synthesis.  arXiv:2203.13474 [cs.LG]
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  arXiv:2203.02155 [cs.CL]
  * Paiva et al. (2022) José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022.  Automated assessment in computer science education: A state-of-the-art review.  _ACM Transactions on Computing Education (TOCE)_ 22, 3 (2022), 1–40.
  * Pankiewicz and Baker (2023) Maciej Pankiewicz and Ryan S Baker. 2023.  Large Language Models (GPT) for automating feedback on programming assignments.  _arXiv preprint arXiv:2307.00150_ (2023).
  * Phung et al. (2023a) Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023a.  Generating High-Precision Feedback for Programming Syntax Errors using language Models.  arXiv:2302.04662 [cs.PL]
  * Phung et al. (2023b) Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023b.  Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors.  arXiv:2306.17156 [cs.CY]
  * Prenner and Robbes (2021) Julian Aron Prenner and Romain Robbes. 2021.  Automatic Program Repair with OpenAI’s Codex: Evaluating QuixBugs.  arXiv:2111.03922 [cs.SE]
  * Price et al. (2017) Thomas Price, Rui Zhi, and Tiffany Barnes. 2017.  Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming.  _International Educational Data Mining Society_ (2017).
  * Rivers and Koedinger (2017) Kelly Rivers and Kenneth R Koedinger. 2017.  Data-driven hint generation in vast solution spaces: a self-improving python programming tutor.  _International Journal of Artificial Intelligence in Education_ 27 (2017), 37–64.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code Llama: Open Foundation Models for Code.  arXiv:2308.12950 [cs.CL]
  * Sarsa et al. (2022a) Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022a.  Automatic generation of programming exercises and code explanations using large language models. In _Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1_. 27–43.
  * Sarsa et al. (2022b) Sami Sarsa, Juho Leinonen, Arto Hellas, et al. 2022b.  Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability.  _Journal of Educational Data Mining_ 14, 2 (2022).
  * Savelka et al. (2023) Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023.  Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?  _arXiv preprint_ (2023).  https://doi.org/10.48550/arXiv.2303.09325 arXiv:2303.09325 [cs.AI]
  * Sorva and Seppälä (2014) Juha Sorva and Otto Seppälä. 2014.  Research-based design of the first weeks of CS1. In _Proceedings of the 14th Koli Calling International Conference on Computing Education Research_. 71–80.
  * Tirumala et al. (2022) Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models.  arXiv:2205.10770 [cs.CL]
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  arXiv:2302.13971 [cs.CL]
  * Vihavainen et al. (2014) Arto Vihavainen, Jonne Airaksinen, and Christopher Watson. 2014.  A systematic review of approaches for teaching introductory programming and their influence on success. In _Proceedings of the tenth annual conference on International computing education research_. 19–26.
  * Wang et al. (2017) Ke Wang, RIshabh Singh, and Zhendong Su. 2017.  Data-Driven Feedback Generation for Introductory Programming Exercises.  _arXiv:1711.07148 [cs]_ (Nov. 2017).  http://arxiv.org/abs/1711.07148 arXiv: 1711.07148.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned Language Models Are Zero-Shot Learners.  arXiv:2109.01652 [cs.CL]
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 [cs.CL]
  * Wermelinger (2023) Michel Wermelinger. 2023.  Using GitHub Copilot to Solve Simple Programming Problems. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 172–178.  https://doi.org/10.1145/3545945.3569830
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  arXiv:1910.03771 [cs.CL]
  * Wu et al. (2021) Mike Wu, Noah D. Goodman, Chris Piech, and Chelsea Finn. 2021.  ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback.  _CoRR_ abs/2107.14035 (2021).  arXiv:2107.14035 https://arxiv.org/abs/2107.14035
  * Zhang et al. (2022) Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2022.  Repairing Bugs in Python Assignments Using language Models.  arXiv:2209.14876 [cs.SE]

##  Appendix A Appendix

###  A.1 Dataset details

In this section, we detail the processing steps applied to each of our
datasets.

####  A.1.1 FalconCode

We requested and acquired the FalconCode dataset from the authors’ website
website, which consists of multiple tables as detailed in the paper by (8).
The downloaded dataset encompasses 1,433,323 submissions written by 1,563
students across 478 distinct assignments spanning three semesters.

There are three levels of difficulty of assignments in the dataset: "skill",
"lab" and "project". For this study, we excluded “project” assignments, which
have high complexity, often require extensive code writing across multiple
files, and lack automated tests. For all our analysis, we refer to “skill” and
“lab” assignments as “easy” and “hard” (respectively). Several other
assignments require access to external files (e.g, .csv files or .txt files)
which the authors did not release in the current version of the dataset.
However, we are currently in the process of retrieving this information. We
excluded all assignments that require these external files. We then removed
students’ intermediate runs (not submitted for grading). All other submissions
present were automatically graded and assigned a score comprised between 0 and
the exercise maximum score (typically 100). We removed all submissions which
have a score of 0, as manual inspection reveals that these solutions are often
the results of students’ obvious mistakes (e.g. forgetting to print) or
students’ "trial and error". After removing intermediate runs and zero-passing
codes, we have in total 270,478 submitted programs to 334 problems written by
1557 students.

Following (25), we selected only the final submissions for each student for
each assignment. This resulted in our test set comprising submissions from
students who did not complete the assignments successfully. While we
acknowledge that this selection may not fully capture the range of
difficulties students encounter during their attempts, it aligns with the idea
that a student’s last attempt often reflects their improved understanding of
the problem. Additionally, students are more likely to seek help after several
unsuccessful attempts. Thus, our setup can be viewed as providing feedback to
students as a last resort for elements they may not have grasped. In our
dataset, this selection process yielded 118,764 distinct solutions. We
recognize the potential for future work to explore more refined methods for
automatically selecting meaningful solutions within large datasets.

For dataset splitting, we adopted a methodology similar to (32). We divided
the dataset by semester, designating the first two semesters (fall and spring
2021) for training and development purposes, respectively. The remaining
semester constituted the testing set (spring 2022). Within each split, we
independently removed submissions that shared identical abstract syntax tree
(AST) structures after variable normalization. Table 2 provides an overview of
the dataset statistics after this processing.

Table 2: FalconCode dataset statistics. subset | training | development | testing
---|---|---|---
difficulty | easy | hard | easy | hard | easy | hard
# correct solutons | 4413 | 16463 | 4521 | 13552 | 3678 | 12164
# incorrect solutions | 398 | 901 | 622 | 717 | 538 | 767
# problems | 89 | 104 | 87 | 68 | 83 | 89

Note that we intentionally also limited the size of our training and dev
subsets to match the distribution of the test set. We encourage further
experimentation with larger subsets of training datasets.

####  A.1.2 Singapore

We obtained the Singapore dataset from the original paper’s GitHub repository
(25). The downloaded dataset initially contains 4225 submissions from 361
students. As with the FalconCode data, we removed similar submissions based on
AST comparison on the Singapore dataset, which left us with a total of 2436
submissions. Table 3 displays the dataset statistics. It’s important to note
that the original dataset lacks information regarding which student submitted
which assignment, and that we do not make use of the correct solutions.

Table 3: Singapore dataset statistics. Assignment | 1 | 2 | 3 | 4 | 5 | Total
---|---|---|---|---|---|---
# correct solutions | 480 | 157 | 152 | 186 | 223 | 1196
# incorrect solutions | 296 | 331 | 246 | 261 | 104 | 1238

###  A.2 Experiment details

In this section, we detail how we created artificial code repair examples for
supervised training (fine-tuning) of the used models, and how we trained and
evaluated our models.

####  A.2.1 Neural Automated Repair

Prior to training, we utilized the Refactory Automated Repair Tool (ART) (25)
to generate repairs for each incorrect program in the training and development
datasets from FalconCode (32). Detailed information about Refactory’s
functionality can be found in the original paper and its associated GitHub
repository333https://github.com/githubhuyang/refactory.

We configured Refactory with online refactoring, sampling 100% of correct
student programs from both the training and development subsets, and enabled
structure mutation and block repair phases (i.e., using the best settings
available). Unlike in (32), if ART couldn’t successfully repair a faulty
program, we followed the approach described in (20). In such cases, we
employed the Rouge-L score (33; 41) to map the incorrect program to the
nearest correct program.

####  A.2.2 Supervised training

We finetuned our models for next token prediction using HuggingFaces’
transformers library (70) using separate training and development sets. To
ensure efficient training, we truncated each sequence to a maximum of 512
tokens. We decayed the learning rate with a cosine schedule, using an initial
learning rate of 5e-5. We then performed a hyperparameter search, varying the
number of epochs (1, 2, or 3) and batch sizes (4, 8, 16, or 32); the model
selected for final evaluation with the testing set is the one showing the
lowest validation loss on (a subset of) the development set. Our template
prompt displayed in Figure 2 shows the structure of the prompts that we used
to generate the repairs with the models. Notably, the last part of the prompt
that includes the repair example found with ART is included only for the
training and development set and excluded for testing sets. It’s important to
note that the prompt used did not include the graders’ feedback. In our future
work, we will evaluate separately the effect of adding such feedback on
program repair performance.

⬇

Repair the incorrect program. 1

{PROBLEM_DESCRIPTION} 2

**Incorrect program** 3

{BUGGY PROGRAM} 4

**Repaired code**: 5

{CORRECTED PROGRAM} 6

Figure 2: The prompt template used for model training and evaluation: (1) a
preamble orienting the model to the task of the program repair (although
pretrained models are not instruction tuned, a preamble remains a useful
signal), (2) the problem description extracted from the dataset, (3) an
indication of the start of the buggy program, (4) the buggy program to fix,
(5) an indication of the start of the repaired program, (6) the artificial
repair found for the buggy program (see section A.2.1) – included only in the
training phase.

####  A.2.3 Evaluation

We generated 10 repairs for each incorrect program in our test set using our
prompt template shown in Figure 2. The repaired programs were generated using
top_p nucleus sampling with a fixed temperature of 0.6, similarly as in (7)
(all other parameters are default). We refer the reader to section 3.1 for the
rest of the evaluation procedure.

####  A.2.4 Computational resources.

The training and evaluation pipelines were run individually and separately for
each model on a single Nvidia Tesla A100 GPU using our institution research
cluster.