# On Multi-Modal Learning of Editing Source Code

Saikat Chakraborty  Department of Computer Science
Columbia University
New York, NY, USA
<EMAIL_ADDRESS>Baishakhi Ray  Department of Computer Science
Columbia University
New York, NY, USA
<EMAIL_ADDRESS>
###### Abstract

In recent years, Neural Machine Translator (NMT) has shown promise in
automatically editing source code. Typical NMT based code editor only
considers the code that needs to be changed as input and suggests developers
with a ranked list of patched code to choose from - where the correct one may
not always be at the top of the list. While NMT based code editing systems
generate a broad spectrum of plausible patches, the correct one depends on the
developers’ requirement and often on the context where the patch is applied.
Thus, if developers provide some hints, using natural language, or providing
patch context, NMT models can benefit from them.

As a proof of concept, in this research, we leverage three modalities of
information: edit location, edit code context, commit messages (as a proxy of
developers’ hint in natural language) to automatically generate edits with NMT
models. To that end, we build Modit, a multi-modal NMT based code editing
engine. With in-depth investigation and analysis, we show that developers’
hint as an input modality can narrow the search space for patches and
outperform state-of-the-art models to generate correctly patched code in top-1
position.

###### Index Terms:

Source Code Edit, Neural Networks, Automated Programming, Neural Machine
Translator, Pretraining, Transformers

##  I Introduction

Programmers often develop software incrementally, adding gradual changes to
the source code. In a continuous software development environment, programmers
modify their source code for various reasons, including adding additional
functionality, fixing bugs, refactoring, etc. It turns out that many of these
changes follow repetitive edit patterns [1, 2, 3] resulting in a surge of
research effort to automatically generate code-changes learned from past
examples [2, 3, 4, 5, 6].

In particular, Neural Machine Translation (NMT) models have been successful in
learning automatic code changes [6, 5, 7, 8, 9, 10, 11]. At the core, these
models contain an encoder and a decoder — the encoder encodes the code that
needs to be edited, and the decoder sequentially generates the edited code.
Such NMT models are trained with a large corpus of previous edits to learn
generic code change patterns. In the inference time, given a code fragment
that needs to be edited, a trained NMT model should automatically generate the
corresponding edited code.

⬇ //Guidance: use LinkedList and fix sublist problem ...  public void
addPicture (String picture){ if ((pictures) == null) { - pictures = new
ArrayList<>(); + pictures = new LinkedList<>(); //correct patch + pictures =
new HashSet<>(); //plausible patch } pictures.add(picture); }
---
Figure 1: Example of an identical code (marked in red) changed in two
different ways ( green and blue) in two different contexts, where both can be
correct patches. However, based on developers’ guidance (top line) to fix a
list related problem, green is the correct patch in this context.

However, learning such generic code changes is challenging. A programmer may
change an identical piece of code in different ways in two different contexts,
both can potentially be correct patches (see Figure 1). For example, an
identical code fragment pictures = new ArrayList$<>$() was changed in two
different ways: pictures = new HashSet$<>$(); and pictures = new
LinkedList$<>$() in two different code contexts. Without knowing the
developers’ intention and the edit context, the automated code editing tools
have no way to predict the most intended patches. For instance, in the above
example, LinkedList was used to fix a sublist-related problem. Once such an
intention is known, it is easy to choose a LinkedList-related patch from the
alternate options. Thus, such an additional modality of information can
reinforce the performance of automated code-editing tools.

Figure 2: A motivating example. The guidance provides a brief summary of what
needs to be changes. The underlined tokens are directly copied from guidance
and context into the patched code.

In fact, given just a piece of code without any additional information, it is
perhaps unlikely that even a human developer can comprehend how to change it.
Consider another real-life example shown in Figure 2. If a programmer only
considers the edited expression in line 9, it is difficult to decide how to
modify it. However, with additional information modalities – _i.e.,_ the
guidance (line 1,2) and the context (the whole method before the patch), the
correct patch often becomes evident to the programmer since the guidance
effectively summarises how to change the code and the context provides
necessary ingredients for generating a concretely patched code. We hypothesize
that such multi-modal information could be beneficial to an automated code-
editing tool. To that end, we design Modit, a multi-modal code editing engine
that is based on three information modalities: (i) the code fragment that
needs to be edited, (ii) developers’ intention written in natural language,
and (iii) explicitly given edit context.

In particular, Modit is based on a transformer-based [12] NMT model. As input,
Modit takes the code that needs to be edited (_e.g.,_ the lines that need to
be patched), additional guidance describing developers’ intent, and the
context of the edits that are explicitly identified by the developer (_e.g.,_
the surrounding method body, or surrounding lines of code, etc.). Note that
previous works [6, 11] also provided context and the edit location while
generating edits; however, they are fed together to the model as a unified
code element. Thus, the model had the burden of identifying the edit location
and then generating the patch. In contrast, isolating the context from the
edit location and feeding them to the model as different modalities provides
Modit with additional information about the edits.

Curating developers’ intent for a large number of edits that can train the
model is non-trivial. As a proof of concept, we leverage the commit messages
associated with the edits to simulate developers’ intent automatically. We
acknowledge that commit messages could be noisy and may not always reflect the
change summary [13]. Nonetheless, our extensive empirical result shows that,
even with such noisy guidance, Modit performs better in generating correctly
edited code.

Being a model that encodes and generates source code, Modit needs to both
clearly understand and correctly generate programming languages (PL). While
several previous approaches [6, 14] designed sophisticated tree/grammar-based
models to embed the knowledge of PL into the model, the most recent
transformer-based approaches [15, 16, 17] showed considerable promise with
pre-training with a large volume of source code. Since these models are pre-
trained with billions of source code written by actual developers, and
transformers are known to learn distant dependencies between the nodes, these
models can learn about code structures during the pre-training step. Among
such pre-trained models, PLBART [17] learns jointly to understand and generate
source code and showed much promise in generative tasks. Thus, we chose PLBART
as the starting point to train Modit, i.e., we initialize Modit’s model with
learned parameters from PLBART.

We evaluate Modit on two different datasets ( $B2F_{s}$, and $B2F_{m}$)
proposed by Tufano  _et al._ [8] consisting of an extensive collection of bug-
fix commits from GitHub. Our empirical investigation shows that a summary of
the change written in natural language as additional guidance from the
developer improves Modit’s performance by narrowing down the search space for
change patterns. The code-edit context, presented as a separate information
modality, helps Modit to generate edited code correctly by providing necessary
code ingredients (_e.g.,_ variable names, method names, _etc._). Modit
generates $\sim$3.5 times more correct patches than Codit showing that Modit
is robust enough to learn PL syntax implicitly. Furthermore, Modit generates
two times as many correct patches as a large transformer model could generate.

Additionally, our empirical investigation reveals that when we use one encoder
to encode all information modalities rather than learning from individual
modalities separately, the model learns representation based on inter-modality
reasoning. In contrast, a dedicated encoder for each individual modality only
learns intra-modality reasoning. Our experiment shows that a multi-
modal/single-encoder model outperforms multi-modal/multi-encoder model by up
to 46.5%.

We summarize our main contributions in this paper as follows.

  * •

We propose Modit– a novel multi-modal NMT-based tool for automatic code
editing. Our extensive empirical evaluation shows that Automatic Code Editing
can be vastly improved with additional information modalities like code
context and developer guidance.

  * •

We empirically investigate different design choices for Modit. We provide a
summary of the lessons that we learned in our experiments. We believe such
lessons are valuable for guiding future research.

  * •

We prototype and build Modit and open-source all our code, data in
https://git.io/JOudU.

##  II Background

###  II-A Neural Machine Translation

Neural Machine Translation(NMT) [18] is a very well studied field, which has
been very successful in translating a sentence from one language to another.
At a very high level, input to an NMT model is a sentence
($X=x_{1},x_{2},...,x_{n}$), which is usually a sequence of tokens ($x_{i}$),
and the output is also a sentence ($Y=y_{1},y_{2},...,y_{m})$ – sequence of
tokens ($y_{i}$). While learning to translate from $X$ to $Y$, NMT models
learn conditional probability distribution $P(Y|X)$ . Such probability
distributions are learned _w.r.t._ model parameters $\theta$, where model
training process optimizes $\theta$ in such a way that maximizes the expected
probability distribution of a dataset. An NMT model usually contains an
encoder and a decoder. The encoder processes, understands, and generates
vector representations of the input sentence. The decoder starts after the
encoder and sequentially generates the target sentence by reasoning about the
encoder-generated input representation. While sequentially generating the
target sentence, the decoder usually performs different heuristic searches
(for instance, beam search) to balance exploration and exploitation.

In recent few years, Software Engineering has seen a wide spectrum of
adaptation of NMT. Some prominent application of NMT is SE include Program
Synthesis [19], Code summarization [20, 21], Edit summarization [13], Code
Edit Generation [8, 5, 6], Automatic Program Repair [10, 9, 11], etc. These
research efforts capitalize on NMTs’ capability to understand and generate
complex patterns and establish NMT as a viable tool for SE-related tasks.

###  II-B Transformer Model for Sequence Processing

Transformer [12] model revolutionized sequence processing with attention
mechanism. Unlike the traditional RNN-based model where input tokens are
processed sequentially, the transformer assumes soft-dependency between each
pair of tokens in a sequence. Such dependency weights are learned in the form
of attention weights based on the task of the transformer. While learning the
representation of a token, the transformer learns to attend to all the input
tokens. From a conceptual point of view, the transformer converts a sequence
to a complete graph111https://en.wikipedia.org/wiki/Complete_graph, where each
node is a token. The weights of the edges are attention weights between tokens
which are learned based on the task of the transformer. The transformer
encodes each token’s position in the sequence (positional encoding) as part of
the input. In such a way, the transformer learns long-range dependency. Since
its inception, the transformer is very successful in different NLP
understanding and generation tasks. Transformers’ ability of reasoning about
long-range dependency is proved useful for several source code processing task
including code completions [22], code generation [23], code summarization
[21].

###  II-C Transfer Learning for Source Code

In recent few years, Transfer learning shows promise for a wide variety of SE
tasks. Such transfer learning aims at learning task agnostic representation of
source code and reuse such knowledge for different tasks. One way to learn
such task agnostic representation of input is pre-training a model with a
large collection of source code. The learning objective of such pre-training
is often understanding the code or generating the correct code. A pre-trained
model is expected to embed the knowledge about source code through its
parameters. Such pre-trained models are later fine-tuned for task-specific
objectives. CuBERT [24], CodeBERT [15], GraphCodeBERT [16] are all
transformer-based encoder models which are pre-trained to understand code.
Such models are primarily trained using Masked Language Model [25], replaced
token prediction [15], semantic link prediction [16], etc.

For code generation, CodeGPT [26, 9] pre-trains a transformer-based model to
generate general-purpose code sequentially. More recently, PLBART [17] pre-
trained transformer-based model jointly for understanding and generating code
with denoising auto-encoding [27]. PLBART consists of an encoder and a
decoder. The encoder is presented with slight noise (for instance, token
replacement) induced code, and the decoder is expected to generate noise-free
code. Since code editing task requires both the understanding of code and code
generation, we chose PLBART as the base model for Modit.

##  III Modit

Figure 3: Overview of Modit pipeline

Figure 3 shows an overview of Modit’s working procedure. Modit is a multi-
layer encoder-decoder based model consisting of a Transformer-based encoder
and a Transformer-based decoder. Both the encoder and decoder consist of 6
layers. Modit works on three different modalities of information: (i) Code
that needs to be edited ($e_{p}$), (ii) natural language guidance from the
developer ($\mathcal{G}$), and (iii) the context code where the patch is
applied ($C$). We acknowledge that $e_{p}$ is essentially a substring of $C$.
However, by explicitly extracting and presenting $e_{p}$ to Modit, we provide
Modit with additional information about the change location. Thus, despite
being a part of the context, we consider $e_{p}$ a separate modality.
Nevertheless, Modit consists of three steps. First, the pre-processing step
processes and tokenizes these input modalities (§III-A). Then the encoder in
Modit encodes the processed input, and the decoder sequentially generates the
patched code as a sequence of tokens (§III-B). At final step, Modit post-
processes the decoder generated output and prepares the edited code (§III-C).

###  III-A Pre-processing

Input Consolidation. In the pre-processing step, Modit generates consolidated
multi-modal input ($X$) from the three input modalities (_i.e.,_ $e_{p}$,
$\mathcal{G}$, and $C$). Modit combines these input modalities as a sequence
separated by a special <s> token _i.e.,_ $X=$$e_{p}$ <s> $\mathcal{G}$ <s>
$C$. In the example shown in Figure 2, $e_{p}$ is newJson.charAt(1)) !=
wrappingQuote , $\mathcal{G}$ is fix problem which occurred when the resulting
json is empty, and $C$ is the whole function before the edit (see Input
Modalities in Figure 3). Modit generates a consolidates multi-modal input
sequence as newJson.charAt(1)) ... <s> fix problem which occurred ... <s>
private String ... }.

Tokenization. Modit uses sentence-piece tokenizer [28]. Sentence-piece
tokenizer divides every token into sequence of subtokens. Such subword
tokenization is similar to previously used byte-pair encoding in automatic
code editing literature [29, 9]. We use PLBART [17]’s sentence-piece tokenizer
which is trained on billions of code from GitHub. After tokenizing the
consolidated input $X$ from Figure 2, we get _new Json . char At ( 1 ) ... <s>
_fix _problem _which _oc cur red ... <s> _private _String ... _}.

###  III-B Encoder-Decoder Model

The input to Modit’s encoder-decoder model is a sequence of subtokens
generated in the previous step.

Transformer Encoder. Given an input sequence $X=x_{1},x_{1},...,x_{n}$, the
encoder learns the representation of every token at layer $l$ as
$R^{e}_{l}(x_{i})$ using self-attention computed as

$R^{e}_{l}(x_{i})=\sum_{j=i}^{n}{a_{i,j}*R^{e}_{l-1}(x_{j})}$ (1)

Where $R^{e}_{l-1}(x_{j})$ is the representation of subtoken $x_{j}$ as
generated by layer $l-1$, and $a_{i,j}$ is the attention weight of subtoken
$x_{i}$ to $x_{j}$. Such attention weights are learned by multi-head attention
[12]. Final layer generated representation (_i.e.,_ $R^{e}_{6}(x_{i})$) is the
final representation for every subtoken $x_{i}$ in the input. Note that, the
encoder learns the representation of Equation 1 of a subtoken, using all
subtokens in the sequence. Thus the learned representation of every subtoken
contains information about the whole input sequence. Since we encode all the
information modalities in one sequence, the learned representation of every
subtoken encodes information about other modalities.

Transformer Decoder. The decoder in Modit is a transformer-based sequential
left-to-right decoder consisting of 6 layers. It sequentially generates one
subtoken at a time using previously generated subtokens and the final
representation (${R^{e}_{l}(x_{i})}$) from the encoder. The decoder contains
two modules – (i) self-attention, and (ii) cross-attention. The self-attention
layer work similar to the self-attention in the encoder. First, with self
attention, decoder generates representation $R^{d}{l}(y_{i})$ of last
generated token $y_{i}$ with self attention on all previously generated tokens
$(y_{1},y_{2},...,y_{i})$. This self attention follows same mechanism
described in Equation 1. After learning decoder representation by self
attention, decoder applies attention of encoder generated input representation
using the following equation,

$\mathcal{D}_{l}(y_{i})=\sum_{j=i}^{n}{\alpha^{l}_{i,j}*R^{e}_{6}(x_{j})}$ (2)

Where
$\alpha^{l}_{i,j}=softmax\left(dot\left(R^{e}_{6}\left(x_{j}\right),R^{d}{l}\left(y_{i}\right)\right)\right)$
is the attention weight between output subtoken $y_{i}$ to input subtoken
$x_{j}$. The softmax generates an attention probability distribution over the
length of input tokens. Finally the decoder learned representation,
${D}_{l}(y_{i})$ is projected to the vocabulary to predict maximally likely
subtoken from the vocabulary as next token.

In summary, the encoder learns representation of every subtokens in the input
using all input subtoken, essentially encoding the whole input information in
every input subtoken representation. The decoder’s self-attention mechanism
allows the decoder to attend to all previously generated subtokens allowing
the decoder decide on generating correct token at correct place. The cross-
attention allows the decoder to attend to encoded representation - implicitly
letting the model decide where to copy from the input where to choose from new
tokens in the vocabulary. We initialize the end-to-end encoder-decoder in
Modit using pre-trained weights of PLBART [17].

###  III-C Output Generation

The decoder in Modit continue predicting subtoken until it predicts the end of
sequence </s> token. During inference, Modit uses beam search to generate
sequence of subtokens. Once the decoder finishes, Modit post-processes the top
ranked sequence in the beam search. First, Modit removes the end of sequence
</s> token. It then detokenizes the subtokens sequence to code token sequence.
In this step, Modit merges generated subtokens that are fragments of a code
token into one code token. For the example shown in Figure 2, Modit generates
the subtoken sequence _! _json . is Empty () _&& _( _new Json . char At ( 1 )
_!= _wrap ping Quote _) </s>. After detokenization, Modit generates !
json.isEmpty() && ( newJson.charAt(1) != wrappingQuote ) .

##  IV Experimental Design

###  IV-A Dataset

TABLE I: Statistics of the datasets studied.

2 Dataset | Avg. | Avg. | Avg. tokens |  # examples [t]
---|---|---|---|---
Tokens | Change Size* | in Guidance | Train | Valid |  Test [b]
2 $B2F_{s}$ | 32.27 | 7.39 | 11.55 | 46628 | 5828 | 5831
$B2F_{m}$ | 74.65 | 8.83 | 11.48 | 53324 | 6542 | 6538
2 |  |  |  |  |  |

* Change size measured as token edit distance.

To prove our concept of Modit, we experiment on two different datasets
(_i.e.,_ $B2F_{s}$, and $B2F_{m}$) proposed by Tufano  _et al._ [8]. In these
two datasets, they collected large collections of bug-fix code changes along
with commit messages from Java projects in GitHub. Each example in these
datasets contains the java method before the change ($C_{p}$), the method
after the change ($C_{n}$), and the commit message for the change. There are
some examples ($<100$) with corrupted bytes in the commit message, which we
could not process. We excluded such examples from the dataset. Table I shows
statistics of the two datasets we used in this paper. $B2F_{s}$ contains
smaller methods with maximum token length 50, and $B2F_{m}$ contains bigger
methods with up to 100 tokens in length. The average size of the change (edit
distance) is 7.39, and 8.83 respectively, in $B2F_{s}$ and $B2F_{m}$.

###  IV-B Data Preparation

For the datasets described in Section IV-A, we extract the input modalities
and the expected output to train Modit. For every method pair (_i.e.,_ before
edit - $C_{p}$, after edit - $C_{n}$) in those dataset, we use GumTree [30] to
extract a sequence of tree edit locations. We identify the root of the
smallest subtree of $C_{p}$’s AST that encompasses all the edit operations. We
call the code fragment corresponding to that subtree as code to be
edited($e_{p}$) and used as Modit’s first modality. Similarly, we extract the
code corresponding to the smallest subtree encompassing all the edit
operations from $C_{n}$ and use that as code after edit($e_{n}$). We use the
commit message associated with the function pair as Modit’s second modality,
guidance($\mathcal{G}$). Finally, we use the full method before edit ($C_{p}$)
as Modit’s third modality, context($C$).

###  IV-C Training

After combining every example in the datasets in Modit’s input ($e_{p}$,
$\mathcal{G}$, $C$) and expected output ($e_{n}$), we use this combined
dataset to train Modit. For training Modit, we use Label Smoothed Cross
Entropy [31] as loss function. We use Adam optimizer, with a learning rate of
$5e^{-5}$. We train Modit for 30 epochs, after every epoch, we run beam search
inference on the validation dataset. We stop training if the validation
performance does not improve for five consecutive validations.

###  IV-D Evaluation Metric

We use the top-1 accuracy as the evaluation metric throughout the paper. For
proof-of-concept, we evaluate all techniques with beam size 5. When the
generated patched code matches exactly with the expected patched code $e_{n}$,
it is correct, incorrect otherwise. Note that this is the most stringent
metric for evaluation. Previous approaches [6, 11, 10] talked about filtering
out infeasible patches from a ranked list of top k patches using test cases.
However, we conjecture that such test cases may not always be available for
general purpose code edits. Thus, we only compare top-1 accuracy.

###  IV-E Research Questions

Modit contains several design components: (i) use of multimodal information,
(ii) use of transformer and initializing it with the pre-trained model, and
(iii) use of end-to-end encoder-decoder (using PLBART) to generate patches
instead of separately using pre-trained encoder or pre-trained decoder, as
used by previous tools. First, we are interested in evaluating Modit _w.r.t._
state-of-the-art methods. In particular, we evaluate how these three design
choices effect Modit’s performance. So, we start with investigating,
RQ1. How accurately does Modit generate edited code _w.r.t._ other techniques?

Modit uses three input modalities. Our next evaluation target is how these
individual modalities effect Modit’s performance? Thus we ask,
RQ2. What are the contribution of different input modalities in Modit ’s
performance?

Finally, recall from Section III-A, Modit proposes to encode all the input
modalities as a sequence and use one encoder for the consolidated multi-modal
input. An alternative to this encoding mechanism is to encode individual input
modality with dedicated input encoder. Our next evaluation aims at finding out
the best strategy to encode input modalities. Hence, we investigate,
RQ3. What is the best strategy to encode multiple input modalities?

##  V Empirical Results

In our first research question, we evaluate Modit’s performance _w.r.t._ other
techniques and the effect of Modit’s design components.

RQ1. How accurately does Modit generate edited code _w.r.t._ other techniques?

Experimental Setup. We carefully chose the baselines to understand the
contribution from different design choices of Modit. We evaluated our model in
two experimental settings. First, we train different baseline models where the
full model is trained from scratch. In this setting, the first baseline we
consider is an LSTM with attention [18] NMT model. Various existing code
patching approaches [7, 8, 5, 11] used such settings. Second baseline is
Transformer [12] based $\mathcal{S}2\mathcal{S}$ model. We consider two
different-sized transformers. This enables us to contrast effect of model size
in code-editing performance. The Transformer-base model consists of six
encoder layers and six decoder layers. The Transformer-base model’s
architecture is the same as Modit’s architecture. Furthermore, we consider
another transformer with a much larger architecture. Transformer-large
contains twelve encoder layers and twelve decoder layers with three times as
many learnable parameters as the Transformer-base model. The final baseline in
this group is Codit, which is a tree-based model. Comparison _w.r.t._ Codit
allows us to contrast externally given syntax information (in the form of CFG)
and learned syntax by transformers (_i.e.,_ Modit). We use all three input
modalities (see Figure 3 for example) as input to the LSTM and Transformer.
Using auxiliary modalities is non-trivial with Codit since the input to Codit
must be a syntax-tree. Thus, we use uni-modal input ($e_{p}$) with Codit.

(a) CodeBERT — Consist of bidirectional pretrained encoder and a decoder
trained from scratch.

(b) CodeGPT — One pretrained single decoder processes the input and output
sequentially from left to right.

(c) PLBART — Consist of pretrained bidirectional encoder and pretrained left
to right decoder.

Figure 4: Schematic diagram of the three types of pre-trained models. used to
evaluate Modit.

In the second setting, we consider different pre-trained models, which we used
to fine-tune for patch generation. Figure 4 shows schematic diagrams of the
pre-trained models we compared in this evaluation. First two models we
considered are CodeBERT [15], and GraphCodeBERT [16]. Both of these models are
pretrained encoders primarily trained to understand code. To use these for the
patching task, we add a six-layered transformer-based decoder along with the
encoder. The decoder is trained from scratch (see Figure 4(a)). Another pre-
trained baseline is CodeGPT [26]. GPT is a single left-to-right decoder model
primarily pre-trained to generate code. For the code editing task, a special
token <SEP> combines the input and the output as a sequence separated. Jiang
_et al._ [9] showed the effectiveness of GPT for the source code patching task
(see Figure 4(b)). In contrast to these pre-trained models, Modit uses PLBART,
an end-to-end encoder-decoder model trained to understand and generate code
simultaneously (see Figure 4(c)). To compare from a fairground, we evaluate
these pre-trained models with uni-modal input ($e_{p}$), and multi-modal input
($e_{p}$<s> $\mathcal{G}$<s> $C$), separately.

TABLE II: Top-1 accuracies of different models _w.r.t._ their training type,
model sizes, input modality.

2 Training | Model | # of | Multi- |  Accuracy (%) [t]
---|---|---|---|---
Type | Name | params (M) | Modal |  $B2F_{s}$ |  $B2F_{m}$ [b]
2 | LSTM | 82.89 | ✓ | 6.14 |  1.04
| Transformer-base | 139.22 | ✓ | 11.18 |  6.61
| Transformer-large | 406.03 | ✓ | 13.40 |  8.63
From Scratch | Codit | 105.43 | ✗ | 6.53 |  4.79
2 |  |  | ✗ | 24.28 |  16.76
| CodeBERT | 172.50 | ✓ | 26.05 |  17.13
|  |  | ✗ | 24.44 |  16.85
| GraphCodeBERT | 172.50 | ✓ | 25.67 |  18.31
|  |  | ✗ | 28.13 |  16.35
| CodeGPT | 124.44 | ✓ | 28.43 |  17.64
|  |  | ✗ | 26.67 |  19.79
Fine-tuned | Modit | 139.22 | ✓ | 29.99 |  23.02
2 |  |  |  |  |

Results. Table II shows the accuracy in top 1 predicted patch by Modit along
with different baselines. LSTM based $\mathcal{S}2\mathcal{S}$ model predicted
6.14% and 1.04% correct patches in $B2F_{s}$ and $B2F_{m}$ respectively. The
Transformer-base model achieves 11.18% and 6.61% top-1 accuracy in those
datasets, which improves further to 13.40% and 8.63% with the Transformer-
large model. Codit predicts 6.53% and 4.79% correct patches in $B2F_{s}$ and
$B2F_{m}$, respectively. Note that Codit takes the external information in the
form of CFG; thus, the patches Codit generate are syntactically correct.
Nevertheless, the transformers, even the smaller model, perform better to
predict the correct patch. We conjecture that the transformer model can
implicitly learn the code syntax without direct supervision.

In contrast to the models trained from scratch, when we fine-tune a pretrained
model, it generates significantly more correct patches than models trained
from scratch. For instance, Modit (initialized with pretrained PLBART)
generates 168% and 248% more correct patches than the Transformer-base model
(with randomly initialized parameters), despite both of these models having
the same architecture and the same number of parameters. In fact, the smallest
fine-tuned model (CodeGPT) performs much better than the larger model trained
from scratch (Transformer-large).

All the fine-tuned models exhibit better performance when the input data are
multi-modal with various degrees of improvement. With all three input
modalities, CodeBERT [15] generates 7% and 2.2% more correct patches in
$B2F_{s}$ and $B2F_{m}$, respectively, compared to a unimodal CodeBERT model.
In case of Modit, such improvement is 11.07% in $B2F_{s}$ and 16.23% in
$B2F_{m}$. The $\mathcal{G}$ in the multi-modal data often contains explicit
hints about how to change the code. For instance, consider the example shown
in Figure 2, the guidance explicitly says there is a problem with the json
when it is empty. Furthermore, with the presence of $C$ in the input, the
model can identify different variables, methods used in the method and
potentially copy something from the context. We conjecture that such
additional information from these two additional input modalities (i) reduce
the search space for change patterns, (ii) help models copy relevant
identifiers from the context.

Among the fine-tuned models multi-modalities, Modit generates 15.12% more
correct patches than CodeBERT, 16.82% than GraphCodeBERT, and 5.49% than
CodeGPT in $B2F_{s}$. In the case of $B2F_{m}$ dataset, Modit’s improvement in
performance is 34.38%, 25.72%, 30.50% higher than CodeBERT, GraphCodeBERT, and
CodeGPT, respectively. To understand these results better, let us look at some
of the examples.

⬇ //Guidance: merging of items that aren’t actually equal public static
boolean equals( ItemStack one, ItemStack two) { - return one.isSimilar(two) &&
- (one.toString().equals(two.toString())); + return one.isSimilar(two);
//MODIT generated /* CodeGPT generated */ + return
one.toString().equals(two.toString()); }
---
Figure 5: Example patch where Modit was able to generate correct patch, but
CodeGPT could not. Modit’s patch is shown in green, and CodeGPT generated
patch is shown in blue.

Figure 5 shows an example patch where Modit correctly generated the expected
patch but CodeGPT could not. If we look closely, we can see that the code to
be changed ($e_{p}$) is a boolean expression where the two clauses are
combines with &&. While only the first clause, one.isSimilar(two) is the
expected output, CodeGPT chooses the second clause,
one.toString().equals(two.toString()) from the original. Recall from Figure
4(b), CodeGPT processes the combined input and output sequence (separated by
special <SEP> token) in left-to-right fashion. Thus, encodes representation of
the input tokens do not contain information about the whole input sequence. In
contrast, the Modit uses a pre-trained bi-direction encoder which helps Modit
to understand the input fully. Based on the examples we have seen and the
empirical result, we conjecture that, for code-editing tasks, the model must
fully understand the input in a bi-directional fashion.

⬇ // Guidance: ... code refactoring ... public boolean isEmpty() { -
if((first) == null){ return true;} - return false; + return (first) == null;
//MODIT predicts /* CodeBERT generated */ + return ((first) == null) ||
(first.get()) == null; }
---
Figure 6: Correctly predicted patch by Modit. CodeBERT could not understand
and reason about the textual hint to predict the correct patch.

Figure 6 shows an example where Modit generated correct patch, CodeBERT could
not. Note that the guidance text explicitly asks about code refactoring,
implying that the patched code should be semantically similar to the original
code. Similar to the original code, patched could should return true when
first == null , otherwise it should return false . An automated code change
tool should not add additional code features when doing the refactoring.
However, CodeBERT generated patch which introduced an additional clause
first.get() == null in the return expression, which make CodeBERT’s generate
code semantically different from the original. Modit was able to generate the
correct patch for this example.

Finally, we summarize the empirical lessons we learned in this research
question as

  * •

Multi-modal input improves Code-Editing capability, irrespective of the
underlying model used. The guidance often narrows the edit pattern search
space, and the context narrows down the token generation search space.

  * •

Transformer models (especially larger ones) are robust enough to learn the
code’s syntax information without direct supervision. When a pre-trained model
is used to initialize transformer parameters, the improvement is notably
higher.

  * •

For code-editing task, both understanding the input and correctly generated
output are important. While a pre-trained encoder understands the code and a
pre-trained decoder generates correct code, an end-to-end pre-trained encoder-
decoder model (_e.g.,_ PLBART) the best choice to fine-tune for this task.

Result IV-E:  _Modit generates 29.99%, and 23.02% correct patches in top-1
position for two different datasets outperforming CodeBERT by up to 25.72%,
GraphCodeBERT by up to 34.38%, and CodeGPT by up to 30.50%. Pre-trained models
tend to be more effective than models trained from scratch for code
editing—Modit improves the performance by 167% than the best model trained
from the scratch. _

Modit combines multiple modalities of information to generate patches. Now we
investigate,

RQ2. What are the contribution of different input modalities in Modit ’s
performance?

Experimental Setup. In this experiment, we investigate the contribution of
different input modalities in Modit’s performance. Recall from Section III-A
that we use three inputs in Modit (_i.e.,_ $e_{p}$, $C$, $\mathcal{G}$). Here,
we investigate different combinations of such input modalities. More
precisely, we investigate the influence of three information sources: (i) code
that needs to be changed ($e_{p}$), (ii) context ($C$), and (iii) guidance
($\mathcal{G}$). Note that, by presenting $e_{p}$ as a separate information
modality, we are essentially providing Modit with the information about the
location of the change. To study the effect of such presentation, we study
another alternative experimental setup, where we annotate the change location
inside the context with two unique tokens <START> and <END>.

TABLE III: Contribution of different input modalities in Modit’s performance.
✓ indicates that corresponding input modality is used as encoder input, ✗
indicates otherwise. We report top-1 accuracy as performance measure. Exp. ID
is used later to refer to corresponding experiment result. Exp. ID $\Phi_{*}$
denotes an experiment with $*$ as input modalities.

2 Exp. ID | Inputs |  Accuracy (%)
---|---|---
$e_{p}$ | $C$ | $\mathcal{G}$ |  $B2F_{s}$ |  $B2F_{m}$
2 $\Phi_{c}$ | ✗ | ✓ | ✗ | 13.05 |  4.50
$\Phi_{cg}$ | ✗ | ✓ | ✓ | 17.89 |  4.51
2 $\Phi_{c}^{\dagger}$ | ✗ |  ✓† | ✗ | 13.03 |  4.53
$\Phi_{cg}^{\dagger}$ | ✗ |  ✓† | ✓ | 17.90 |  4.60
2 $\Phi_{e}$ | ✓ | ✗ | ✗ | 26.67 |  19.79
$\Phi_{eg}$ | ✓ | ✗ | ✓ | 28.76 |  21.63
$\Phi_{ec}$ | ✓ | ✓ | ✗ | 29.79 |  21.40
$\Phi_{ecg}$ | ✓ | ✓ | ✓ | 29.99 |  23.02
2 |  |  |  |  |

† $e_{p}$ is surrounded by two special tokens <START> and <END> inside the
context.

Result. Table III shows Modit’s performance with different combination of
input modalities. When we present only the context to Modit, it predicts
13.05% correct patches in $B2F_{s}$ and 4.50% in the $B2F_{m}$, which improves
further to 17.89%, and 4.51% in those two datasets respectively when we add
$\mathcal{G}$. Note that in these two scenarios, the model does not explicitly
know which portion of the code needs to be edited; it sees the whole method
and predicts (only) the patched code ($e_{n}$). In addition to learning how to
patch, the model implicitly learns where to apply the patch in this setup. To
test whether the identification of such location is the performance
bottleneck, we surround the code that needs to be patched with two special
tokens <START> and <END>. SequenceR [11] also proposed such annotation of
buggy code. Surprisingly, such annotation resulted in comparable (slightly
worse in one case) performance by Modit.

In the next set of experiments, we extract the code that needs to be edited
($e_{p}$) and present it as a separate input modality. First, we only present
the $e_{p}$ without the other two modalities. When we only present the $e_{p}$
and generate the edited code ($e_{n}$), it results in 26.67% top-1 accuracy in
the $B2F_{s}$ and 19.79% in the $B2F_{m}$. Ding  _et al._ [32] attributed such
improvement to the reduced search space due to shorter input. Our result
corroborates their empirical findings. Nevertheless, when we add the
$\mathcal{G}$ modality with the $e_{p}$, Modit’s performance improves to
28.76% and 21.63% in $B2F_{s}$ and $B2F_{m}$, respectively.

In our final set of experiments in this research question, we augment $e_{p}$
with the $C$. In this evaluation setup, Modit predicts 29.79% correct patches
in the $B2F_{s}$ and 21.40% in the $B2F_{m}$, which is improved further to
29.99%, and 23.02% correct patches in those two datasets when we add
$\mathcal{G}$.

⬇ // Guidance: fixed some bugs in type checking // improved performance by
caching types of expressions private TypeCheckInfo getType(SadlUnionType
expression){ ... return new TypeCheckInfo( - declarationConceptName,
declarationConceptName /* MODIT generated patch with guidance */ +
declarationConceptName, declarationConceptName,  + this, expression /* MODIT
generated patch without guidance */ + this.declarationConceptName, +
this.declarationConceptName ); }
---
Figure 7: Example showing the effect of textual guidance in Modit’s
performance. Modit produced the correct patch with guidance, without guidance
as input Modit’s produced patch is essentially refactored version of original
input.

Figure 7 shows an example where Modit with all modalities could successfully
generate correct patch. The text guidance ($\mathcal{G}$) provides hint that
variable expression should somehow associate with the construction of
TypeCheckInfo in the patched code. However, without this guidance Modit
generated a wrong patch by accessing existing parameters from this object.
Essentially, without the guidance, Modit refactored the input code.

⬇ // Guidance: Fix bug of sending wrong message public void setPredecessor
(model.Message m) { this.predecessor = Integer.valueOf(m.Content);
model.Message sent = new model.Message(); sent.To = m.Origin; -
sendMessage(m); /* MODIT generates with the context. */ + sendMessage(sent);
/* MODIT generates without context as input. */ + sendMessage(m.toString()); }
---
Figure 8: Example showing the necessity of context information in predicting
the correct patch. Modit’s generated correct patch with the context as input.
Without context, Modit received sendMessage(m) and the guidance as input, did
not know the variable sent could be the parameter of the function sendMessage,
and predicted a wrong patch.

Figure 8 shows the effect of context as input modality to Modit. The before
edit version of the code($e_{p}$) passed the wrong parameter (m) to
sendMessage function. When the context ($C$) is presented to Modit, it saw
another variable (sent) in the context. In contrast, without context($C$),
Modit indeed changed the parameter; but sent m.toString() — resulting in a
wrong patch.

When we extract the buggy code and present the buggy code along with the
context, we see a big performance improvement (see the difference between
$\Phi_{c}$, and $\Phi_{ec}$ in Table III). We hypothesize that, when only
context (_i.e.,_ full code) is presented ($\Phi_{c}$), the model gets confused
to identify which portion from the context needs to be edited since any
portion of the code is a likely candidate for patching. However, when we
extract the exact code that needs to be edited and present as a separate input
modality to Modit, it can focus on patching just that code using other
modalities (including the context) as a supporting source of information. In a
recent study, Ding  _et al._ [32] pointed out the need for effective ways to
include context in the NMT based code editors. Our empirical results show that
Modit’s way of including context as a separate modality is a potential
solution to that problem.

In summary, each of the modalities contribute to the overall performances of
Modit. Lessons learned in these experiments are:

  * •

Additional textual guidance helps the patch generation. Such guidance can
provide important clue about how to modify the code and sometimes provide
ingredients necessary for the change.

  * •

Adding context explicitly in the input enables the model to select appropriate
identifiers for patching.

  * •

Isolating buggy code help the model put proper focus on the necessary part of
the code while leveraging auxiliary information from other modalities.

Result IV-E:  _All three modalities (code to be edited, context, and guidance)
are essential for Modit to perform the best. Without either one of those,
performance decreases. Modit’s performance improves up to 37.37% when
additional textual guidance is used as an input modality. Context modality
improves Modit’s performance up to 6.4%._

We investigate alternative ways to combine multiple input modalities. We ask,
RQ3. What is the best strategy to encode multiple input modalities?

Experimental setup. To validate Modit’s design choice of appending all input
modalities into one sequence, we test alternative ways to combine input
modalities. In particular, we follow the design choice proposed by Lutellier
_et al._ [10], where they used multiple encoders to encode the $e_{p}$ and the
$C$. Tufano  _et al._ [33] also leverages a similar idea to encode input code
and code review messages. Nevertheless, we use a multi-encoder model shown in
Figure 9.

Figure 9: An alternative architecture of code editing with multi-encoder
model. We initialize each of the encoders with pre-trained Encoder model.

In a multi-encoder setting, we first encode each input modality with a
corresponding dedicated encoder. After the encoder finishes encoding, we
concatenate the encoded representations and pass those to the decoder for
generating patched code. To retain maximum effectiveness, we initialize each
individual encoder with pretrained weights from CodeBERT [15]. We consider a
single-encoder model (also initialized with CodeBERT) as a baseline to compare
on the fairground. While presenting the inputs to the single encoder model, we
concatenate input modalities with a unique separator token <s>. Finally, to
test the robustness of our empirical finding, we propose two different
experimental settings. In the first evaluation setup, we use all three input
modalities. We compare a tri-encoder model with a single-encoder model. Next,
we consider bimodal input data – $e_{p}$ and $\mathcal{G}$. We use a dual-
encoder model and compare it with a single-encoder model in this setup.

TABLE IV: Comparison of multi encoder model. 2 # of | # of |  Accuracy (%) [t]
---|---|---
Modalities | Encoders |  $B2F_{s}$ |  $B2F_{m}$ [b]
2 3 ($e_{p}$, $\mathcal{G}$, $C$) | 3 | 20.63 |  11.69
1 | 26.05 |  17.13
2 2 ($e_{p}$, $\mathcal{G}$) | 2 | 23.12 |  15.49
1 | 23.81 |  17.46
2 |  |  |

Result. Table IV shows the result of multi-encoder models. For tri-modal input
data, if we use three different encoders, the model can predict 20.63% correct
patches in the $B2F_{s}$ and 11.69% correct patches in the $B2F_{m}$. In
contrast, if we use a single encoder, the model’s predictive performance
increases to 26.05% and 17.13% top-1 accuracy in the $B2F_{s}$ and the
$B2F_{m}$, respectively.

In the bimodal dataset (where the input modalities are $e_{p}$ and
$\mathcal{G}$), the dual-encoder model predicts 23.12% correct patches in the
top-1 position for the $B2F_{s}$ and 15.49% correct for the $B2F_{m}$. The
single encoder counterpart, in this setup, predicts 23.81% correct patches for
the $B2F_{s}$ and 17.46% for the $B2F_{m}$. The empirical results show that
the single-encoder model performs better in both the experimental setup than
the multi-encoder setup. We find similar results with GraphCodeBERT [16].

(a) Single encoder for encoding multiple-modalities. Encoder can learn
representation _w.r.t._ all modalities.

(b) Dual-encoder for encoding individual modalities separately. Representation
of tokens from a particular modality is learned _w.r.t._ (only) other tokens
from the same modality.

Figure 10: Input token representation generation in single encoder and
multiple encoder.

To explain why single-encoder is performing better than multi-encoder, let us
look at the encoders’ working procedure. Figure 10 depicts how the encoder
generates representation for input tokens. Note that the encoders we used in
this research question are transformer-based, and recall from the Section II,
transformer generates representation for an input token by learning its
dependency on all other tokens in the sequence. When we present all the input
modalities to a single encoder, it generates input representation for those
tokens _w.r.t._ and other tokens in the same modality and tokens from other
modalities. For instance, in Figure 10(a), the encoder generates $X_{2}$’s
representation considering $X_{1}$, $Y_{1}$, and $Y_{2}$. In contrast, in
Figure 10, $X_{2}$’s representation is learned only _w.r.t._ $X_{1}$, since
encoder1 does not see the input modality $Y$. Thus, when we present all the
input modalities to one single encoder, we conjecture that learned
representations are more robust than that of learning with multi-encoder.

Finally, we summarize the lessons we learned in this research question as

  * •

In multi-modal translation, using single encoder results in better performance
than using a separate encoder for each modality.

  * •

Single-encoder generates input representation by inter-modality reasoning
(attention), hence learns more robust representation than that of multi-
encoder.

Result IV-E:  _Encoding all the input modalities by a single encoder is the
best way to learn in a multi-modal setting. A single encoder improves code-
editing performance by up to 46.5% than the corresponding multi-encoder
setting._

##  VI Discussion

###  VI-A Localization of Code Edit Site

An alternative modeling approach for code editing is to generate the sequence
of edit operations (_i.e.,_ INSERT, DELETE, UPDATE) [32, 34, 35, 36], where
the model must know the precise location of an edit operation (often a node in
the AST) before applying it. Throughout this paper, we also assumed that such
edit location is known to Modit. This assumption may pose a threat to the
usefulness of Modit in a real development scenario. To mitigate such a threat,
we perform an experiment where we pass the whole function as input to Modit
and expect the whole edited function to be generated.

TABLE V: Performance of Modit when the input in the full code and the output
is patched full code.

2 Inputs |  Accuracy (%)
---|---
Full Code | Guidance |  $B2F_{s}$ |  $B2F_{m}$
2 ✓ | ✗ | 20.35 |  8.35
✓ | ✓ | 21.57 |  13.18
2 |  |  |

Section VI-A shows the top-1 accuracy in the $B2F_{s}$ and the $B2F_{m}$.
Modit generates correctly patched full code in 20.35% cases for the $B2F_{s}$
and 8.35% cases for the $B2F_{m}$. With additional textual guidance, the
performance is further improved to 21.57% and 13.18% in the $B2F_{s}$ and
$B2F_{m}$, respectively. While textual guidance helps in this experimental
setup, we notice a big drop in performance than the results shown in Table
III. This is because the benchmark datasets we used contain small edits (see
Table I). Thus, while generating the full code, the model wastes a large
amount of effort trying to generate things that did not change. Nevertheless,
our hypothesis external guidance improves code editing holds even when the
model generates full code.

###  VI-B Tokenization for Source Code Processing

TABLE VI: Comparison between concrete tokenization and abstract tokenization alongside pre-trained models. Results are shown as top-1 accuracy of full code generation in $B2F_{s}$/ $B2F_{m}$ datasets. 2 Token type | CodeBERT | GraphCodeBERT |  PLBART
---|---|---|---
2 Abstract | 16.4 / 5.16 |  17.30 / 9.10 |  19.21 / 8.98
Concrete |  17.3 / 8.38 | 16.65 / 8.64 |  20.35 / 8.35
2 |  |  |

The possible number of source code can be virtually infinite. Vocabulary
explosion has been a big challenge while processing source code with Machine
Learning technique [7, 37]. Previous research efforts have addressed this
problem using several different heuristics. For instance, Tufano  _et al._ [7,
8] identifiers abstraction, which drastically reduces the vocabulary size
considered making it easier to learn patterns by the model. Recent studies
[32, 10, 9, 37] found that Byte-Pair Encoding [38] partially solves the open-
vocabulary problem by sub-dividing rare words into relatively less rare sub-
words. Such sub-division is also learned from large corpora of data. All the
pre-trained models used in this paper used sub-word tokenization techniques.
CodeBERT and GraphCodeBERT used RoBERTa tokenizer [39], CodeGPT used GPT
tokenizer [40], and PLBART used sentence-piece tokenizer [28]. The use of such
tokenizers strips away the burden of identifier abstraction. Our investigation
shows that, in some cases, pre-trained models perform better with concrete
tokens than abstract tokens (see Table VI for detailed result). Thus, we
champion using input and outputs with concrete tokens when a pre-trained model
is used.

##  VII Related Works

###  VII-A Automatic Code Change

There are a lot of research efforts to capture repetitiveness of developers’
way of editing source code. These researches show the potential of automatic
refactoring [41, 42], boilerplate code [43] etc. These research efforts
include (semi-)automatic tools involving traditional program analysis
techniques (_e.g.,_ clone detection, dependency analysis, graph matching) [44,
3]. Other research direction aims at learning source code edit from previous
edits and applying those edit patterns in similar context [1, 45]. Some of
these efforts targets very specific code changes; For example, Nguyen  _et
al._ [46] proposed a graph-matching-based approach for automatically updating
API usage. Tansey  _et al._ [47] semantic preserving transformation of java
classes for automated refactoring. Other directions of works address more
general-purpose code change learned from open source repositories [5, 6]. Such
approaches target solving automated code editing tasks in a data-driven
approach, and the edit patterns are learned from example changes. In this
research, we also investigated general purpose source code changes in the
wild. More closely to Modit, Rolim  _et al._ [4]’s proposed technique
constraints source code generation with additional input/output specification
or test cases. Nevertheless, we argue that textual guidance could be a very
good surrogate specification.

###  VII-B NMT for Code Change Modeling

NMT has been studied for past couple of years to learn automatic source code
change modeling. Tufano  _et al._ [7, 8, 5] presented initial investigation of
using NMT in learning general purpose code changes. Chakraborty  _et al._ [6]
proposed a tree based hierarchical NMT model for general purpose source code
change. Instead of viewing code as sequence of tokens, they first generated
syntax tree by sampling from Context Free Grammar, and then another model to
fill up the gaps for identifier. To reduce the search space, they performed
scope analysis to search for suitable identifier. Chen  _et al._ [11] proposed
a copy mechanism based NMT model for APR where the input is the code before
change along with the context, and the output is the code after change. Their
work treated the input as uni-modal way where the whole code is one singe
modality. In this work, we consider multi-modal way of modeling, where we
isolate the code fragment that needs to be changed from its context and
present that code fragment concatenated with context to the model. Lutellier
_et al._ [10] treated code needs to be changed and the context as two
difference modalities and use separate encoders. However, our empirical
evidence showed that using one encoder to encode all the modalities result in
the best performance. More recently, Ding  _et al._ [32] presented empirical
evidence that instead of generating a whole code element (_i.e.,_
context+change) of the target version, only generating the sequence of changes
might perform better for code change modeling. Recent works [35, 36] proposed
models for generating such edit sequence. Such models may augment or
outperform NMT based code editing – we leave such investigation as future
work.

###  VII-C Machine Learning for Source Code Analysis

In recent years, Machine Learning, especially Deep Learning has been widely
adopted across different area of software engineering due to Availability of
large collection of source code in open source platforms (_e.g.,_ GitHub,
Bitbucket, _etc._) Application of ML based source code analysis include bug
detection in code [48], clone detection [49], code completion [50],
vulnerability detection [51], code summarization [21], code translation [52],
etc. Recent works also approached to learn general purpose transferable
representation learning for source code, which can later be used for various
source code related tasks [53, 15, 9]. The approaches for learning such
transferable representations can be broadly categorized in two ways. The first
category of approaches (_e.g.,_ Code2Vec [53]) aims at learning explicit
representation for tokens in the code. Another category of approaches (_e.g.,_
CodeBERT [15]) transfers syntactic and semantic interaction between code
components in the form of pre-trained models. In this approach, a model for a
specific task is initialized with a general-purpose pre-trained model, trained
to understand and generate code. In this paper, we empirically found that such
pre-trained models (PLBART) increase accuracy upto 248% in patch generation.

##  VIII Threats to Validity

###  VIII-A External Validity

Bias in the dataset. Both $B2F_{s}$, and $B2F_{m}$ are collection of bug-fix
commits, and thus there is a threat that these dataset may exhibit specific
bias towards bug-fix patches. While the commits in these datasets are filtered
and classified as bug fix commits, these changes are made by real developers
as part of development life cycle. Unlike other bugfix datasets [54],
$B2F_{s}$ and $B2F_{m}$ do not isolate the bug. Thus, we conjecture that
possibility of existence of any such bias is minimal.

Noise in commit message. We used commit message as a guidance for code
editing. While previous research efforts [55, 56] showed that commit messages
are very useful to summarize the changes in a commit, other research efforts
[57, 58] also elucidated noises present in the commit message. To mitigate
this threat, we carefully chose the dataset we tested Modit on. The original
authors [8] of the the dataset reported that they carefully investigated the
dataset and after manual investigation, they reported that 97.6% of the
commits in their datasets are true positive. Despite this threat, Modit’s
performance seems to improve with commit message as additional input.

###  VIII-B Construct Validity

In general, developers write commit message after they edited the code, in
theory, summarizing the edits they made. In this paper, we assumed an
experimental setup where developer would write the summary before editing the
code. Such assumption may pose a threat to the applicability of Modit in real
world, since in some cases, the developer may not know what edits they are
going to make prior to the actual editing. Regardless, we consider Modit as a
proof-of-concept, where empirically we show that, if a developer had the idea
of change in mind, that could help an automated code editor.

###  VIII-C Internal Threat

All Deep Learning based techniques are sensitive to hyper-parameters. Thus
using a sub-optimal hyper-parameter can pose a threat to the validity of
Modit, especially while comparing with other baselines. As we compared with
other pre-trained models, we cannot really modify the architecture and
dimensions of other pre-trained models. As for other hyper-parameters (_i.e.,_
learning rate, batch size, _etc._), we use the exact same hyper-parameters
described by respective paper. Nevertheless, we open source out code and data
for broader dissemination.

##  IX Conclusion

In this paper, we highlight that an automatic code edit tool should possess
knowledge about the underlying programming language, in general. Also, it can
benefit from additional information such as edit context and developers’
intention expressed in natural language. To that end, we design, present, and
evaluate Modit– a multi-modal NMT-based automated code editor. Our in-depth
evaluation shows that Modit improves code-editing by leveraging knowledge
about programming language through pre-training. In addition, we showed that
leveraging additional modalities of information could benefit the source code
editor. Our empirical evaluation reveals some critical lessons about the
design choices of building an automated code editor that we believe will guide
future research in automatic code editing.

## Acknowledgement

This work is supported in part by NSF grants SHF-2107405, SHF-1845893,
IIS-2040961, IBM, and VMWare. Any opinions, findings, conclusions, or
recommendations expressed herein are those of the authors and do not
necessarily reflect those of the US Government, NSF, IBM or VMWare.

## References

  * [1] B. Ray, M. Nagappan, C. Bird, N. Nagappan, and T. Zimmermann, “The uniqueness of changes: Characteristics and applications,” ser. MSR ’15. ACM, 2015.
  * [2] N. Meng, M. Kim, and K. S. McKinley, “Systematic editing: generating program transformations from an example,” _ACM SIGPLAN Notices_ , vol. 46, no. 6, pp. 329–342, 2011.
  * [3] ——, “Lase: Locating and applying systematic edits by learning from examples,” _In Proceedings of 35th International Conference on Software Engineering (ICSE)_ , pp. 502–511, 2013.
  * [4] R. Rolim, G. Soares, L. D’Antoni, O. Polozov, S. Gulwani, R. Gheyi, R. Suzuki, and B. Hartmann, “Learning syntactic program transformations from examples,” in _Proceedings of the 39th International Conference on Software Engineering_. IEEE Press, 2017, pp. 404–415.
  * [5] M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk, “On learning meaningful code changes via neural machine translation,” _arXiv preprint arXiv:1901.09102_ , 2019.
  * [6] S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray, “Codit: Code editing with tree-based neural models,” _IEEE Transactions on Software Engineering_ , vol. 1, pp. 1–1, 2020.
  * [7] M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “An empirical investigation into learning bug-fixing patches in the wild via neural machine translation,” 2018.
  * [8] M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 28, no. 4, pp. 1–29, 2019.
  * [9] N. Jiang, T. Lutellier, and L. Tan, “Cure: Code-aware neural machine translation for automatic program repair,” _arXiv preprint arXiv:2103.00073_ , 2021.
  * [10] T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “Coconut: combining context-aware neural translation models using ensemble for program repair,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 101–114.
  * [11] Z. Chen, S. J. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end-to-end program repair,” _IEEE Transactions on Software Engineering_ , 2019.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30_ , 2017, pp. 5998–6008.
  * [13] Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural-machine-translation-based commit message generation: How far are we?” in _2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2018, pp. 373–384.
  * [14] W. Wang, G. Li, S. Shen, X. Xia, and Z. Jin, “Modular tree network for source code representation learning,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 29, no. 4, pp. 1–23, 2020.
  * [15] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Nov. 2020, pp. 1536–1547.
  * [16] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, J. Yin, D. Jiang _et al._ , “Graphcodebert: Pre-training code representations with data flow,” in _International Conference on Learning Representations_ , 2021.
  * [17] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in _2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2021.
  * [18] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _International Conference on Learning Representations_ , 2015.
  * [19] P. Yin and G. Neubig, “A syntactic neural model for general-purpose code generation,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , vol. 1, 2017, pp. 440–450.
  * [20] B. Wei, G. Li, X. Xia, Z. Fu, and Z. Jin, “Code generation as a dual task of code summarization,” in _Advances in Neural Information Processing Systems 32_ , 2019, pp. 6563–6573.
  * [21] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “A transformer-based approach for source code summarization,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2020.
  * [22] S. Kim, J. Zhao, Y. Tian, and S. Chandra, “Code prediction by feeding trees to transformers,” _arXiv preprint arXiv:2003.13848_ , 2020.
  * [23] A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 1433–1443.
  * [24] A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Pre-trained contextual embedding of source code,” _arXiv preprint arXiv:2001.00059_ , 2019.
  * [25] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, “Unified language model pre-training for natural language understanding and generation,” _arXiv preprint arXiv:1905.03197_ , 2019.
  * [26] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang _et al._ , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” _arXiv preprint arXiv:2102.04664_ , 2021. [Online]. Available: https://arxiv.org/abs/2102.04664
  * [27] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” _arXiv preprint arXiv:1910.13461_ , 2019.
  * [28] T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , Nov. 2018, pp. 66–71.
  * [29] R.-M. Karampatsis, H. Babii, R. Robbes, C. Sutton, and A. Janes, “Big code!= big vocabulary: Open-vocabulary models for source code,” in _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_. IEEE, 2020, pp. 1073–1085.
  * [30] J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and accurate source code differencing,” in _Proceedings of the 29th ACM/IEEE international conference on Automated software engineering_. ACM, 2014, pp. 313–324.
  * [31] R. Müller, S. Kornblith, and G. Hinton, “When does label smoothing help?” _arXiv preprint arXiv:1906.02629_ , 2019.
  * [32] Y. Ding, B. Ray, P. Devanbu, and V. J. Hellendoorn, “Patching as translation: the data and the metaphor,” in _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2020, pp. 275–286.
  * [33] R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” _arXiv preprint arXiv:2101.02518_ , 2021\.
  * [34] E. Dinella, H. Dai, Z. Li, M. Naik, L. Song, and K. Wang, “Hoppity: Learning graph transformations to detect and fix bugs in programs,” in _International Conference on Learning Representations_ , 2019.
  * [35] D. Tarlow, S. Moitra, A. Rice, Z. Chen, P.-A. Manzagol, C. Sutton, and E. Aftandilian, “Learning to fix build errors with graph2diff neural networks,” in _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ , 2020, pp. 19–20.
  * [36] Z. Yao, F. F. Xu, P. Yin, H. Sun, and G. Neubig, “Learning structural edits via incremental tree transformations,” _arXiv preprint arXiv:2101.12087_ , 2021.
  * [37] R. M. Karampatsis and C. Sutton, “How often do single-statement bugs occur? The ManySStuBs4J dataset,” _arXiv preprint arXiv:1905.13334_ , 2019.
  * [38] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _arXiv preprint arXiv:1508.07909_ , 2015.
  * [39] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
  * [40] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [41] X. Ge, Q. L. DuBose, and E. Murphy-Hill, “Reconciling manual and automatic refactoring,” in _Proceedings of the 34th International Conference on Software Engineering_. IEEE Press, 2012, pp. 211–221.
  * [42] V. Raychev, M. Schäfer, M. Sridharan, and M. Vechev, “Refactoring with synthesis,” in _ACM SIGPLAN Notices_ , vol. 48, no. 10. ACM, 2013, pp. 339–354.
  * [43] N. Meng, L. Hua, M. Kim, and K. S. McKinley, “Does automated refactoring obviate systematic editing?” in _Proceedings of the 37th International Conference on Software Engineering-Volume 1_. IEEE Press, 2015, pp. 392–402.
  * [44] R. Robbes and M. Lanza, “Example-based program transformation,” in _International Conference on Model Driven Engineering Languages and Systems_. Springer, 2008, pp. 174–188.
  * [45] H. A. Nguyen, A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, and H. Rajan, “A study of repetitiveness of code changes in software evolution,” in _Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering_. IEEE Press, 2013, pp. 180–190.
  * [46] H. A. Nguyen, T. T. Nguyen, G. Wilson Jr, A. T. Nguyen, M. Kim, and T. N. Nguyen, “A graph-based approach to API usage adaptation,” in _ACM Sigplan Notices_ , vol. 45, no. 10. ACM, 2010, pp. 302–321.
  * [47] W. Tansey and E. Tilevich, “Annotation refactoring: inferring upgrade transformations for legacy applications,” in _ACM Sigplan Notices_ , vol. 43, no. 10. ACM, 2008, pp. 295–312.
  * [48] B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu, “On the” naturalness” of buggy code,” in _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_. IEEE, 2016, pp. 428–439.
  * [49] W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural network and flow-augmented abstract syntax tree,” in _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 2020, pp. 261–271.
  * [50] M. R. Parvez, S. Chakraborty, B. Ray, and K.-W. Chang, “Building language models for text with named entities,” 2018.
  * [51] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet,” _IEEE Transactions on Software Engineering_ , pp. 1–1, 2021.
  * [52] H. Xu, S. Fan, Y. Wang, Z. Huang, H. Xu, and P. Xie, “Tree2tree structural language modeling for compiler fuzzing,” in _International Conference on Algorithms and Architectures for Parallel Processing_. Springer, 2020, pp. 563–578.
  * [53] U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” in _Proceedings of the ACM on Programming Languages_ , vol. 3. ACM, 2019, p. 40. [Online]. Available: https://doi.org/10.1145/3290353
  * [54] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for java programs,” in _Proceedings of the 2014 International Symposium on Software Testing and Analysis_. ACM, 2014, pp. 437–440.
  * [55] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019, pp. 10 197–10 207.
  * [56] Y. Zhou and A. Sharma, “Automated identification of security issues from commit messages and bug reports,” in _Proceedings of the 2017 11th joint meeting on foundations of software engineering_ , 2017, pp. 914–919.
  * [57] K. Gallaba, C. Macho, M. Pinzger, and S. McIntosh, “Noise and heterogeneity in historical build data: an empirical study of travis ci,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , 2018, pp. 87–97.
  * [58] S. Kim, H. Zhang, R. Wu, and L. Gong, “Dealing with noise in defect prediction,” in _2011 33rd International Conference on Software Engineering (ICSE)_. IEEE, 2011, pp. 481–490.


# Some reflections on the test-negative design

Ronald Meester and Jan Bonte

###### Abstract

We discuss some philosophical, methodological and practical problems
concerning the use of the test-negative design for COVID-19 vaccines. These
problems limit the use of this design considerably.

The test-negative design was developed as a method to estimate influenza
vaccine effectiveness, but has more recently also been used in the context of
COVID-19 vaccines [1, 3, 5, 6, 7, 8, 9, 10, 11]. In this contribution, we
discuss the vaccine effectivity as defined through a test-negative design
study. Clearly, in order to do so, we need to specify what “effectivity”
means, and we discuss this below. The literature on this subject is rather
confusing, as we will explain. First we explain the design.

In the test-negative design, it is registered how many people with a
influenza-like illness request medical care. It may concern people who report
to the general practitioner or at a clinic, but it can also concern people who
have to be admitted to hospital because of their complaints. After people
report to a doctor or health facility, a test is done to determine whether
someone has an infection with influenza or not. The group of people who test
positive for influenza forms the research group, the group of people who test
negative forms the control group. Next, the ratio between vaccinated and
unvaccinated in the group of people who test positive for influenza is
compared with the ratio between vaccinated and unvaccinated in the group of
people who test negative.

The difference between a standard case control and the test-negative design is
that in a test-negative design, the study population is prospectively defined
by the occurrence of respiratory infection and the subsequent seeking of
medical help. In this study design, it is first decided whether or not a
person belongs to the study population, based on the complaints and symptoms,
and only then is it decided by means of a diagnostic test who belongs to the
research group (those with a positive test result), and who belongs to the
control group (those with a negative test). In a case control design on the
other hand, it is the positive test that determines whether someone belongs to
the study group, and those individuals are compared with people who have a
negative test, the control group. Thus, in that design the test result is
leading, and not the disease symptoms.

In the studies [9, 10, 11] mentioned above, the data are collected
retrospectively by querying large databases. Retrospective is a characteristic
of a case control design, not of a test-negative design. Some studies use both
terms, “test-negative” and “case control”, as for instance in [11]. This is
confusing: it is not a test-negative study, it is a case control study, with
all its possible forms of bias of results.

We start the discussion by following [1] from which we copy the table below.
In the table, the population is divided into appropriate cells:111In [1]
writes “infected”, where we write “positive” in the table. Since the decision
is made on the basis of a test, our formulation seems more appropriate, and it
will play a role later in our contribution.

| Do seek care for ARI | Do not seek care for ARI
---|---|---
|  | positive |  |  |  | positive |  |
| positive | with |  |  | positive | with |  |
| with | other | not |  | with | other | not |
| influenza | pathogen | positive | total | influenza | pathogen | positive | total
Vaccinated | A | B | C | $N_{1}$ | D | E | F | $N_{2}$
Not vaccinated | G | H | I | $N_{3}$ | J | K | L | $N_{4}$

The quantity of interest, the vaccine effectivity, is now defined as

$VE=1-\frac{(\frac{A}{N_{1}})}{(\frac{G}{N_{3}})}.$ (1)

Still following [1], this $VE$ could be estimated as (writing lower case
letters to indicate samples from the populations described in the table)

$\hat{VE}=1-\frac{(\frac{a}{n_{1}})}{(\frac{g}{n_{3}})}.$ (2)

According to [1], this quantity provides an unbiased estimate of the parameter
of interest (1).

Now note that the quantities $A,B,C,G,H$ and $I$ are observed in the test-
negative design study: these count the people that seek help, are tested and
subsequently registered according to their test- and vaccination status.
However, this is not appreciated in [1]. Indeed, they write that “[…] this
design is impossible to implement due to the difficulty of distinguishing
persons in cell $C$ from cells $D-F$ and persons in cell $I$ from cells $J-L$
(namely uninfected persons that would have sought care had they developed an
ARI).” Hence, [1] seems to confuse “Do seek care”, with “Would seek care”. If
that were the case, then $N_{1}$ and $N_{3}$ would not be observable, and
$\hat{VE}$ would not be an estimator in any statistical sense. They get around
this by assuming that

$\frac{B}{N_{1}}=\frac{H}{N_{3}},$ (3)

in which case $VE$ can be rewritten as

$VE=1-\frac{AH}{GB},$

and this can be estimated by

$\hat{VE}=1-\frac{ah}{gb}.$

Their assumption is, then, that among persons who would seek care for ARI, the
incidence of ARI due to other respiratory pathogens does not differ between
vaccinated and unvaccinated persons. However, we noticed that this assumption
is not necessary if one wants to estimate $V$ with (2). The authors in [1]
apparently assume that those seeking help for respiratory infection but
testing negative for both influenza and other pathogens cannot be defined, but
they can. Because, when diagnosing for respiratory pathogens, a standard panel
is generally used, with a limited number of pathogens. There is no exhaustive
testing for all possible pathogens in the clinical situation, simply because
that is not cost-effective and has no added value. However, this is not to say
that those people who test negative for both influenza and other pathogens are
not infected. After all, they do have the clinical picture of respiratory
infection? It is quite possible that the pathogen in them was not in the
standard panel. All this means that all the people belonging to groups $N_{1}$
and $N_{3}$ can simply be counted. And so the assumption made by the authors
to arrive at the formula in (3) is not necessary at all.

Furthermore, it is worth remembering that to calculate the $VE$ it matters
which control group one takes. Here one has three choices: one can take the
group that tests positive for a different pathogen, or take the group that
tests negative for both influenza and other pathogens, the “pan-negatives” or
take both groups together as a control. If it turns out that vaccination
against a specific pathogen does affect the probability of becoming ill from
another pathogen, as suggested in [5], then the outcome of the calculation
will be affected by the choice of control group. In [3] this is discussed in
detail. Next we make some observations concerning this design.

Not useful for overall mortality
We first observe that the test-negative design is useless when used as a
measure for the vaccine effect on overall mortality. Indeed, in order to
participate in a research according to the test-negative design , one must, to
begin with, be in principle able to seek medical care. So, if one wants to
investigate mortality as an adverse event related to the vaccine, the test-
negative design cannot possibly be useful, and research into the question
whether or not a vaccine has an effect on the overall mortality rate should
not be carried out with the test-negative design. For instance, elderly and
vulnerable people who fall ill and die at home fall out of the scope of the
study, while this group might be the one most at risk as far as vaccinations
are concerned.

The idealized character of the design
We next point out the idealized character of the population in the table. It
seems highly questionable to assume that such a simple subdivision into sub-
populations is realistic. As also discussed in [1], the care-seeking behavior
of people can most certainly not be described by a simple dichotomy as
discussed above. The question whether or not people seek care may depend,
among many other things, on the severeness of the symptoms, on the social
pressure to do so, on their vaccination status, on the time of the year, and
on many other factors. This observation implies that $VE$ is not a physically
existing quantity that we try to approximate as well as we can. It is not a
characteristic of the vaccine that we try to discover. Instead, (2) can only
be properly understood as an estimator of the quantity in (1) within a very
unrealistic and extremely simplistic model of the population. It tells us what
the $VE$ would be in certain circumstances, but not what it is. This helps
explaining why estimates of $VE$ can strongly differ among various studies.
Any estimate of $VE$ with the test-negative design is a joint effect of
vaccine properties and circumstances like the actual composition of the
sampled population; see also [3].

No truth-status
Perhaps it helps to draw an analogy with another statistical concept: the
likelihood ratio, see [4]. A likelihood ratio tells us which out of two
competing hypotheses explains the data best, by computing the probability of
the data under each of the hypotheses. This probability is computed in a
statistical model with typically many uncertainties about, for instance,
parameter values. The likelihood ratio reflects evidential value of the data
based on our current knowledge. If we would receive more information about the
values of the parameters, then the ensuing likelihood ratio would change.
However, there is no such thing as a “true” likelihood ratio which would
result upon having perfect knowledge of the parameters. The evidential value
is not a quantity that resides somewhere and that we need to estimate as best
as we can. On the contrary, it is a number that reflects what we can possibly
know now, in the given circumstances. What people sometimes call the “true
likelihood ratio”, is a number that we would obtain given perfect knowledge.
There is nothing “true” about this though, since it does not correspond to any
realistic state of affairs. Similarly, the $\hat{VE}$ as obtained in the test-
negative design has no truth value other than that this number says something
about what the vaccine may or may not achieve in a situation that we are not
in. As in the case of the likelihood ratio, the idealized situation is not
interesting if we are far away from it. In this connection, ‘bias’ has to do
with how far we are from such an idealized situation: it is an adverb
connected to the situation, not to the estimator.

These remarks mean that indeed we should be very careful with qualifications
of the estimator in (2) as being ‘(un)biased’, a terminology that is used in
many discussions [1, 6]. Indeed, being (un)biased would imply that the
estimator has or does not have a systematic deviation from the ‘true’ value.
However, we just noticed that such a true value has no meaningful realistic
interpretation. In [3], for instance, there is a discussion concerning the
“true $VE$”, but this is in the context of a simulation. In a simulation, one
can realize a population as described in the table above, but this is only an
indication that such a simulation will have a very limited relation to reality
and that we can not draw strong conclusions from it.

The test-negative design for COVID-19 vaccines
Although the test-negative design was originally developed in the context of
influenza, it has more recently been used in the COVID-19 context as well, as
a study into the effectiveness of the various vaccines (see earlier
references). For such studies, it is important to distinguish between various
outcomes that can be the subject of research when it comes to vaccination
effectiveness. In [2], various outcomes are discussed: infection,
hospitalization, and death. We already discussed the problems around overall
mortality with this design. In view of the discussions around the COVID-19
vaccines, we add transmission of the virus to this list. Let us comment on the
usefulness of the test-negative design for each of these measures.

What about the effect on hospitalization? If the test-negative design is used
to study the effect on hospital and ICU admission, it is essential that the
clinical picture fits COVID-19 and that this clinical picture corresponds in
terms of complaints and symptoms to the people who test negative for SARS-
CoV-2. The indication for admission to the ICU will often be the need for
mechanical or artificial ventilation, by which both the research group and the
control group are fairly easy to define. But also for hospitalization, the
reason for admission must be a clinical picture that fits well with COVID-19
and corresponds well with the clinical picture of people who test negative. As
soon as people with a positive test result for SARS-CoV-2 are included with a
clinical picture that does not fit COVID-19, the study is clearly disabled.

A complicating factor is that, certainly in the case of an ICU admission and
often also in the case of a hospital admission, the test result is already
known before the admission follows. That is a potential source of bias (in the
sense described above), because it can influence the description of the
clinical picture. And the vast majority of vaccine effectiveness studies are
conducted by retrospectively questioning databases, with the description of
the clinical picture being one of the selection criteria.

The research into the effect of vaccination on infection and transmission
using the test-negative design deserves a separate discussion, because here a
number of factors play a role that are not or much less decisive in the other
outcome measures. So far, we haven’t said anything about the diagnostic tool
used. But whether it is the antigen rapid test or the rtPCR, both tests will
perform better in a clinical setting like a hospital than in the general
population, because it concerns people with influenza-like symptoms and not
asymptomatic people. By “performing better” we mean the characteristics of the
test, the sensitivity and specificity. What is conveniently ignored in many
studies on the effect of vaccination on transmission is that these tests in a
largely asymptomatic population are likely to perform much worse than in a
selected group people with influenza-like symptoms.

Another point is that, even if the rtPCR and antigen rapid test would perform
the same in an asymptomatic population as in a clinical setting, with a
decreasing prevalence (the number of infections), the number of false positive
results increases, and can exceed the number of true positive test results.
And it goes without saying that the prevalence of infections with SARS-CoV-2
in an asymptomatic population will be (very) much lower than in a group of
people with a influenza-like illness.

To back-up our argument, here are some numerical examples. In the first
example, we have 10,000 vaccinated and 10,000 unvaccintaed people. The
prevalence of SARS-CoV-19 is 10% among unvaccinated and 1% among vaccinated.
Tests are perfect. This leads to the following table:

| test positive | test negative | total
---|---|---|---
vaccinated |  $A$ = 100 |  $B$ = 9,900 |  $N_{1}$ = 10,000
not vaccinated |  $G$ = 1,000 |  $H$ = 9,000 |  $N_{3}$ = 10,000
total | 1,100 | 18,900 | 20,000

This leads to a $\hat{VE}$ of 90%.

But now assume that the test has a sensitivity of 70% and a specificity of
95%. We first compute the number of positive and negative tests among the
(un)vaccinated people:

vaccinated people | infected | not infected | total
---|---|---|---
test positive | 70 | 495 | 565
test negative | 30 | 9,405 | 9,435
total | 100 | 9,900 | 10,000

unvaccinated people | infected | not infected | total
---|---|---|---
test positive | 700 | 450 | 1,150
test negative | 300 | 8,550 | 8,850
total | 1,000 | 9,000 | 10,000

These numbers are then substituted in the table to compute the vaccine
effectivity:

| test positive | test negative | total
---|---|---|---
vaccinated |  $A$ = 565 |  $B$ = 9,435 |  $N_{1}$ = 10,000
not vaccinated |  $G$ = 1,150 |  $H$ = 8,850 |  $N_{3}$ = 10,000
total | 1,715 | 8,285 | 20,000

The corresponding $\hat{VE}$ is now reduced to 51%, as a simple computation
shows. If instead we take a sensitivity of 95% and a specificity of 70%, then
a similar computation shows that $\hat{VE}$ reduces to a mere 16%. The vaccine
effectiveness can also become negative, which is to say that vaccinated people
become infected more easily than people who have experienced an infection; see
also [2]. This shows that the calculation of vaccine effectiveness using
diagnostic tests with insufficient sensitivity and specificity leads to
unacceptable estimates of vaccine effectiveness. In both cases above, it leads
to an underestimation of the actual effectiveness.

It should be clear by now that $\hat{VE}$ in a test-negative design is
problematic for various reasons. The quantity $\hat{VE}$ in the test-negative
design can only properly be interpreted as a quantity in a highly idealized
situation which has virtually no relation to reality. The effect of
vaccinations, therefore, is difficult to measure or even to define. In any
case, $\hat{VE}$ as estimated in the test-negative design is not a number that
can be seen as a very useful characteristic of the vaccine, since it depends
on many circumstances. This should be taken into account in every study with
the test-negative design.

## References

  * [1] Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013;31(17):2165-2168. doi:10.1016/j.vaccine.2013.02.053.
  * [2] Ioannidis JPA. Factors influencing estimated effectiveness of COVID-19 vaccines in non-randomised studies [published online ahead of print, 2022 Mar 25]. BMJ Evid Based Med. 2022;bmjebm-2021-111901. doi:10.1136/bmjebm-2021-111901.
  * [3] Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine. 2017;35(36):4796-4800. doi:10.1016/j.vaccine.2017.07.003.
  * [4] Meester R, Slooten K. Probability and forensic evidence, CUP 2021.
  * [5] Benn CS, Fisker AB, Rieckmann A, Sorup S, Aaby P. Vaccinology: time to change the paradigm?. Lancet Infect Dis. 2020;20(10):e274-e283. Doi:10.1016/S1473-3099(19)30742-X.
  * [6] Vandenbroucke JP, Pearce N. Test-Negative Designs: Differences and Commonalities with Other Case-Control Studies with “Other Patient” Controls. Epidemiology. 2019;30(6):838-844. doi:10.1097/EDE.0000000000001088.
  * [7] Aaby P, Netea MG, Benn CS. Beneficial non-specific effects of live vaccines against COVID-19 and other unrelated infections [published online ahead of print, 2022 Aug 26]. Lancet Infect Dis. 2022;S1473-3099(22)00498-4. doi:10.1016/S1473-3099(22)00498-4.
  * [8] Gazit S, Shlezinger R, Perez G, et al. Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Naturally Acquired Immunity versus Vaccine-induced Immunity, Reinfections versus Breakthrough Infections: A Retrospective Cohort Study. Clin Infect Dis. 2022;75(1):e545-e551.
  * [9] Skowronski DM, Setayeshgar S, Zou M, et al. Single-dose mRNA Vaccine Effectiveness Against Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), Including Alpha and Gamma Variants: A Test-negative Design in Adults 70 Years and Older in British Columbia, Canada. Clin Infect Dis. 2022;74(7):1158-1165. doi:10.1093/cid/ciab616.
  * [10] Chemaitelly H, Yassine HM, Benslimane FM, et al. mRNA-1273 COVID-19 vaccine effectiveness against the B.1.1.7 and B.1.351 variants and severe COVID-19 disease in Qatar. Nat Med. 2021;27(9):1614-1621. doi:10.1038/s41591-021-01446-y.
  * [11] Lopez Bernal J, Andrews N, Gower C, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ. 2021;373:n1088. Published 2021 May 13. doi:10.1136/bmj.n1088.