#  A Transformer-Based Approach for Smart Invocation of Automatic Code
Completion

Aral de Moor<EMAIL_ADDRESS>0009-0003-5105-0518 Delft University of
TechnologyDelftNetherlands ,  Arie van Deursen<EMAIL_ADDRESS>0000-0003-4850-3312 Delft University of TechnologyDelftNetherlands and
Maliheh Izadi<EMAIL_ADDRESS>0000-0001-5093-5523 Delft University of
TechnologyDelftNetherlands

(2024)

###### Abstract.

Transformer-based language models are highly effective for code completion,
with much research dedicated to enhancing the content of these completions.
Despite their effectiveness, these models come with high operational costs and
can be intrusive, especially when they suggest too often and interrupt
developers who are concentrating on their work. Current research largely
overlooks how these models interact with developers in practice and neglects
to address when a developer should receive completion suggestions. To tackle
this issue, we developed a machine learning model that can accurately predict
when to invoke a code completion tool given the code context and available
telemetry data.

To do so, we collect a dataset of 200k developer interactions with our cross-
IDE code completion plugin and train several invocation filtering models. Our
results indicate that our small-scale transformer model significantly
outperforms the baseline while maintaining low enough latency. We further
explore the search space for integrating additional telemetry data into a pre-
trained transformer directly and obtain promising results.  To further
demonstrate our approach’s practical potential, we deployed the model in an
online environment with 34 developers and provided real-world insights based
on 74k actual invocations.

IDE, Code Completion, Usability, Transformers, Interaction

††copyright: rightsretained††doi: 10.1145/3664646.3664760††journalyear:
2024††submissionid: fsews24aiwaremain-p15-p††isbn:
979-8-4007-0685-1/24/07††conference: Proceedings of the 1st ACM International
Conference on AI-Powered Software; July 15–16, 2024; Porto de Galinhas,
Brazil††booktitle: Proceedings of the 1st ACM International Conference on AI-
Powered Software (AIware ’24), July 15–16, 2024, Porto de Galinhas,
Brazil††submissionid: fsews24aiwaremain-p15-p††ccs: Human-centered
computing††ccs: Computing methodologies

##  1\. Introduction

Transformer-based code completion has become essential in modern software
development (Ziegler et al., 2022). The widespread adoption of Artificial
Intelligence (AI) tools in coding highlights the significance of these models,
particularly those built on the Transformer architecture (Stack Overflow,
2023). These AI-driven tools typically analyse the code preceding the cursor
to suggest the next lines of code (Chen et al., 2021; Izadi et al., 2022).
Recent advancements have expanded the considered context to include not only
the subsequent code (Bavarian et al., 2022; Rozière et al., 2023; Lozhkov et
al., 2024; Izadi et al., 2024; Guo et al., 2024), but also related snippets
from other files to enrich the prediction accuracy (Lu et al., 2022; Wang et
al., 2023b).

This focus on augmenting the content quality of completions has inadvertently
overshadowed a vital aspect of the user experience: the interaction dynamics
between the developers and the AI tools (Prather et al., 2023; Vaithilingam et
al., 2022; Barke et al., 2022). While these models generate high-quality code
suggestions, their operational and environmental costs pose significant
challenges (Chien et al., 2023; Mok, 2024). Moreover, due to their potential
to disrupt the coding workflow of developers, the frequency and timing of
these suggestions is critical for the overall productivity the tools aim to
boost (Prather et al., 2023).

Previous efforts focus on developing a filtering model designed to show a
completion only when there is a high confidence it will be accepted (Mozannar
et al., 2023b; Sun et al., 2023). This reduces inference cost and likely
improves developers’ focus. However, Sun et al. (2023) assume completions are
rejected based on the context before the cursor alone, ignoring the interplay
with developers’ mode of thought. Mozannar et al. (2023b) improves on this by
considering in-IDE telemetry data. However, they propose a relatively complex
ensemble model that can incur additional latency by filtering after a
completion is generated; and do not consider that some completions, despite
being rejected, may help guide the user in their thinking.

In this study, we take a further step to proactively predict when to invoke a
code completion model based on code context and telemetry data. Our
lightweight, transformer-based filtering model, JonBERTa, is designed to
trigger a code completion model only when there’s a strong likelihood that a
developer requires assistance or is likely to accept the suggested completion.
To train our model, we leverage the data we have collected from developers’
real-world interactions with our open-source code completion tool called
Code4Me111https://code4me.me, available for both VSCode and Jetbrains IDEs. We
gather code context and telemetry data from user interactions with the plugin,
subject to their consent. We use two indicators to gauge when a developer
would prefer to receive a suggestion based on usage data: (1) when they accept
a model-suggested completion, and (2) when they manually invoke the model,
irrespective of whether they ultimately accept or reject the suggestion.

Every keystroke made by a developer provides two key types of contextual
information that assist our invocation-filtering model in deciding whether to
trigger the LLM-based completion system. These are: (1) the coding context
surrounding the point of invocation, and (2) telemetry data gathered via the
plugin, e.g. the time since the last completion. First, we use code context
alone and train a transformer-based classifier. Next, we investigate hybrid
transformer architectures integrating telemetry data as additional features.
Based on our promising results, we evaluate our approach in a user-study with
34 developers, to investigate how our filters perform in practice. To this
end, we also propose a new performance metric to help mitigate issues from
optimising for just acceptance rate in previous work (Mozannar et al., 2023b;
Ziegler et al., 2022), which weighs proxies for the quality and timing of a
completion equally as a harmonic mean. We find that our proposed JonBERTa-head
model scores highest in both the offline and online evaluation.

Our contributions are as follows:

  * •

An offline evaluation of a transformer model we fine-tuned on our collected
code completion dataset, demonstrating that code context can considerably
improve filtering accuracy over a baseline trained on telemetry features only.

  * •

JonBERTa, a novel transformer architecture to show the potential of training
jointly on code context and telemetry data; as well as a custom tokenisation
strategy centred on the cursor position.

  * •

An online evaluation of our filters in a code-completion plugin with 74k
invocations, spanning 34 users.

  * •

For reproducibility purposes, we publish our replication
package222https://github.com/ar4l/curating-code-completions with an online
appendix, as well as our fine-tuned
models333https://huggingface.co/collections/AISE-TUDelft/smart-invocation-of-
code-completion-66473ddf6fa6cf6e541f750c. However, in compliance with the
GDPR, we cannot share our training dataset.

##  2\. Background and Related Work

Today, the most prominent AI-powered code completion tool with over one
million active users is GitHub Copilot (GitHub Copilot, 2021). This tool is
powered by a transformer-based (Vaswani et al., 2023) LLM trained on source
code, originally with the objective of predicting a function body given its
documentation (Chen et al., 2021). However, given that the suffix lines below
the function are unavailable to the model, the current model powering Copilot
is presumably trained with a Fill-In-the-Middle (FIM) objective (Bavarian et
al., 2022); where the model is trained to predict a span of arbitrary length
between the prefix and suffix. Several alternatives exist to Copilot, namely
Amazon CodeWhisperer (Amazon CodeWhisperer, 2023), TabNine (Tabnine, 2023),
Codeium (Codeium, 2023), Sourcegraph Cody (SourceGraph Cody, 2023), JetBrains
AI (Jetbrains AI, 2023), and Gemini Code Assist (Gemini Code Assist, 2023).

Since its inception, several user studies and surveys (Barke et al., 2022;
Ziegler et al., 2022; Vaithilingam et al., 2022; Prather et al., 2023;
Mozannar et al., 2023a; Peng et al., 2023; Wang et al., 2023a; Liang et al.,
2023) have been performed on GitHub Copilot444 As of 29 December 2023, GitHub
Copilot, and others, also offer a conversational interface; but we limit the
scope of this paper to generative code completion only. . They highlight that
the transformer models backing such tools excel at providing contextual
suggestions, due to their semantical understanding of code. As a result, this
leads to increased developer satisfaction (Vaithilingam et al., 2022) and
perceived productivity (Ziegler et al., 2022).

###  2.1. Code Completion Pain Points

Nonetheless, such new technology comes with new questions about its usability
and design, arising from developer pain points, such as: distraction due to
the always-on nature of suggestions (Prather et al., 2023), out-of-
distribution generation leading to hallucinated terms (Johnson et al., 2023),
and a lack of personalisation with developers’ mode of thought (Barke et al.,
2022). Copilot’s authors themselves reported that about two-thirds of shown
completions are ignored by their end-user (Ziegler et al., 2022). Furthermore,
GitClear recently released a report empirically describing a strong
correlation between the adoption of AI code completion and code churn in
industry-grade software engineering (Harding and Kloster, 2024), raising
questions about the impact of AI on software maintainability. Other studies
further find that the bugs introduced by LLM-powered code completion are often
more subtle (Prather et al., 2023; Chen et al., 2021), and difficult to revert
(Vaithilingam et al., 2022).

Barke et al. (2022) find that developer interactions with AI-powered code
completion are bimodal: either accelerative, where the developer knows what
they want and uses the tool to get there faster; or explorative, where the
developer relies on the tool to suggest possible approaches. Prather et al.
(2023) observe two additional modes among novices: shepherding, where a novice
slowly accepts a suggestion; and drifting, where they are led down a cyclic
‘debugging rabbit hole’. Not only is this an inefficient use of computational
resources, but this also misaligns the tool with developer intent.

Some studies advocate for allowing users to configure the timing and context
of completions (Wang et al., 2023a; Barke et al., 2022) to address these
concerns. However, we assume that the majority of end-users will likely expect
such tools to work out of the box and adapt to their usage patterns. The issue
of language model alignment is as pressing as the rate of their increasing
capabilities. As LLMs are becoming more integrated into end-user workflows, it
is necessary to think beyond aligning mere content; but, also the interactions
with this content.

###  2.2. Existing Solutions

Sun et al. (2023) aim to address this problem by filtering out completions
that are likely to be rejected. They train a transformer-based classifier on
the code prefix before it is submitted to the completion model, and find it
can hide 5% of suggestions that would have been rejected with 94.5% accuracy.
However, the authors rely on the assumption that rejected completions are due
to insufficient code context alone.

Mozannar et al. (2023b) train a filter consisting of two model ensembles: one
ensemble before and one after a completion is generated. They cite it can hide
53% of completions with the guarantee 91% would have been rejected by the
user. However, despite designing their tool for GitHub Copilot, they do not
include a user-study. Additionally, their filter can depend on a completion
being generated, which is guaranteed to incur additional latency and compute
in practice. Moreover, both of these tools aim to maximise the acceptance rate
of the shown completions, while the need for better alignment with end-users
has been highlighted (Barke et al., 2022; Mozannar et al., 2023b; Vaithilingam
et al., 2022; Russo, 2023).

Furthermore, as detailed in a reverse-engineering blog-post (Thakkar, 2023),
Copilot also has its own logistic-regression classifier to filter out
completions based on telemetry data. We analyse how it weighs each feature in
more detail in our online appendix, and design our baseline in this study as
similar as possible.

A head-runner in this field would be Gmail SmartCompose (Chen et al., 2019),
considering the same problem in a natural-language email setting. The authors
detail a deep consideration for latency, scalability, and personalisation.
However, their approach uses legacy language model architectures; and their
insights, while valuable, may not entirely apply to the programming process.

##  3\. Problem Definition

Code completion aims to improve developer productivity by saving them
keystrokes and keeping them in their flow. Ziegler et al. (2022) proposes the
acceptance rate of suggestions as a proxy for developers’ perceived
productivity; however, this is critiqued by Mozannar et al. (2023b), whom find
that optimising the acceptance rate results in shorter and typically less
useful suggestions.

Therefore, some completions, despite seemingly helpful, can potentially be
detrimental to software quality and programming flow. And, conversely, some
completions, despite being rejected, are not wholly ignored by the developer
and are potentially helpful in guiding their thinking (e.g., a tip-of-the-
tongue function call, but with the wrong arguments). Determining the actual
quality of a completion, based on its functional correctness and human
preferences for style, remains an open question, though research is
progressing in the right direction (Evtikhiev et al., 2023; Zhou et al., 2023;
Zhuo, 2024).

Consequently, we reframe this problem to what we can actually measure. We
consider two plausible reasons why a user may have rejected a suggestion, as
depicted in Figure 1: (1) the model is incapable of generating a good
suggestion, which is mainly code context-dependent; or (2) the user does not
want to see a suggestion, which is mainly user telemetry-dependent. These
reasons are not exclusive and likely have considerable overlap. Thus, we
motivate the need for integrating these feature types to better align with the
end-user’s flow.

Figure 1. Reasons for Rejected Completions.

[Venn Diagram of Rejected Completions] Depicts two sets with overlap: ’model
incapability’, and ’user does not want to see a suggestion’.

###  3.1. Code-Completion Data and Constraints

Throughout this study, we leverage Code4Me, an open-source code completion
plugin with around 100 monthly active users developed at our institution
(Izadi et al., 2024). We note three key differences compared with the popular
code completion plugins mentioned earlier (Section 2):

  * •

Line-completion only. Other tools tend to provide multi-line suggestions in
ghost-text style; while we provide completions up to a newline character.
Additionally, completions are displayed in a completion box along with typical
language-server suggestions.

  * •

Restrictive activation. Contrary to alternative plugins, we provide
completions only on a predefined set of trigger characters; at which IDE-based
autocomplete would typically trigger (e.g., full stop or an opening
parenthesis). Though, we do allow users to manually invoke a completion.

  * •

Model selection. Our line-completions are generated by smaller language models
than the industry-standard, allowing the user to pick one of three completions
provided by InCoder (Fried et al., 2023), UniXCoder (Guo et al., 2022), and
CodeGPT (Lu et al., 2021).

To help keep developers in their flow, we want to leverage language models’
ability to complete code even when not at a predefined trigger point. In
practice, this means that our plugin will query the completions server at any
cursor position, and we would like to filter out those points where it may not
be necessary to generate a completion. Thus, the plugin should filter out
those automatic suggestions that historically ended up being rejected or
ignored by the user. And, additionally, the plugin infers that a user wants a
suggestion, at a historically manual trigger point.

As added context, our filter should prioritise false-positives over false-
negatives. We assume that this is where user preferences lie, given the
constant-suggestion nature of the more popular plugins. However, to avoid
wasted compute and developer distraction, we aim to minimise those completions
that are certain to be ignored; e.g., because the developer is actively
typing, or their intent cannot be accurately inferred at the current cursor
position.

Lastly, in terms of non-functional constraints, we would like our filter to
take at most 10ms to make a decision. This is in preparation for a backend
migration to the vLLM engine (Kwon et al., 2023). For any incoming completion
prompt, this library waits about 10ms for to see if it can be batched together
with other requests. If our filter takes less than 10ms, it means we incur 0
additional latency over the base wait time. And, even if filtering takes
longer, it is still possible to terminate the token-by-token generation
process early, saving considerable compute as our completions take about
300–400ms to generate.

###  3.2. Joint Optimisation Objective

Our goal is to train a filter to invoke the completion model only at the
instant a developer wants to see a completion. We partially mitigate the
issues arising from optimising for acceptance rate, through the observation
that we may actually want to display the rejected completions that were
manually invoked via a key-bind. By training a filter with this objective,
along with any completions that were accepted, we hope to better align the
completions that pass through the filter with what end-users want at that
moment. Our objective is thus perpendicular to existing work in this area (Sun
et al., 2023; Mozannar et al., 2023b).

The objective jointly optimises the following: (1) we aim to minimise the
amount of times an end-user has to manually invoke the model, and (2) we aim
to minimise the amount of rejected completions that were automatically
triggered. In other words, we consider all manual invocations and automatic,
accepted completions to be our positive class (not filtered out), and rejected
automatic completions to be our negative class (should be filtered).

An added bonus of considering manually-invoked, yet rejected completions in
the positive class is that the resulting filter will be less dependent on the
completion-model’s capabilities. We assume that a manual trigger is a strong
indicator that a user would like to see a suggestion, and choose not to depend
on whether the suggestion was accepted in this scenario.

##  4\. Approach

We aim to filter out suggestions by teaching a model to predict, at any cursor
location, whether to invoke the completion model or not. To do this, we
propose to leverage our collected dataset of code completions and accompanying
in-IDE telemetry to better discern the nature of interactions. Noting the
state-of-the-art contextual understanding that transformer models exhibit, we
explore architectures for integrating telemetry features with code context.

###  4.1. JonBERTa Architecture

We augment a code-pretrained RoBERTa architecture (Liu et al., 2019), yielding
the following two models of 84M parameters. As there are many possible
extensions to a transformer model, we limit our search space to parameter-
efficient implementations. Specifically, both our models incur less than 1M
additional parameters. We assume readers’ familiarity with the transformer
architecture (Vaswani et al., 2023).

  * •

JonBERTa-head incorporating telemetry features directly in the classification
head.

  * •

JonBERTa-attn attending to (small) learned feature embeddings in the self-
attention modules.

We use the Jon prefix to refer to Jointly optimised attention, to both code
context and telemetry data. The motivation behind this approach lies in the
state-of-the-art contextual understanding transformer models exhibit, which we
hypothesise can also leverage contextual telemetry data. We further propose a
novel tokenisation strategy centred on the cursor, to capture the most
significant parts of code context.

Figure 2. Telemetry Feature Data in Classification Head.

[ Scalar telemetry features are concatenated with the first token (special
classification token) out of the last transformer block. ] Concatenating
scalar telemetry features gives a vector. The output from the last transformer
block is a matrix of transformed token embeddings. The classification head
uses the first token’s embedding to determine the class. We append the
telemetry vector to the first token’s embedding, directly before it reaches
the classification head.

####  4.1.1. Extended Classification Head

We explore a simple JonBERTa-head model depicted in Figure 2. Given that a
classification head first pools the output of the previous layer to the first
token (<cls>, the classification token), it is trivial to extend this token’s
one-dimensional embedding with additional feature data.

In the scope of this paper, we only consider a JonBERTa-head where the
embedding is concatenated before reaching the dense layer. The dense layer is
a matrix of size $c\times c$, where $c$ is the length of a token embedding.
Dense layers can help the model learn low-rank embeddings (Huh et al., 2023)
of both feature and code context (which can help with train/test
generalisability), while the projection layer afterward serves as a logistic
classifier. We increase only the dense layer’s size, along one axis, to
accommodate the concatenated features, and reinitialise it.

Figure 3. Self-Attention Extended to Telemetry Feature Data

[ We extend the attention module to also pay attention to learned feature
embeddings, which are defined as functions of the scalar telemetry features. ]
To convert any scalar feature to an embedding vector, we learn a linear
function for each entry in the embedding vector. We then normalise our feature
embeddings using LayerNorm. This allows us to generate keys and values from
the feature embeddings, which can communicate with the queries from the
typical tokens.

####  4.1.2. Extended Self-Attention

Figure 3 depicts our JonBERTa-attn model, which learns feature embeddings to
be attended to in the pre-existing self-attention module. Each weight in the
feature embedding matrix is learned as a function of the corresponding scalar
feature. Given this embedding, keys, and values can be produced to be attended
to by code tokens. In practice, the attention module itself is equivalent to
the original model, except that the keys emitted from features can dot-
multiply with the queries from token embeddings, to produce weighted scores
for how much a given feature’s value should be added to that token embedding.

By including telemetry feature embeddings in its self-attention mechanism, we
hypothesise the model is able to combine both modalities to grasp a firmer
picture of the current user intent. We further explore a variety of layer
combinations and feature embedding dimensions in the online appendix of our
replication package, but cannot conclusively state which achieves better
results. We use a feature embedding dimension of $204$ throughout this study
to limit the additional parameters to the model, while ensuring enough
expressivity.

####  4.1.3. Tokenisation Strategy

The code context provided to our JonBERTa-head and JonBERTa-attn models
consists of the prefix (before the cursor), and the suffix (after the cursor).
Commonly, tokenisers truncate such sequence pairs either both on the left or
both on the right in paired sequence-classification tasks (e.g., question-
answer matching). However, we hypothesise that it is optimal to centre the
context window on the cursor location. Perhaps surprisingly, something we have
not yet seen in previous work.

To achieve this within the JonBERTa context window of 512 tokens, we first
tokenise the suffix with right-truncation up to a maximum of 128 tokens.
Denote the number of tokens in the suffix by $n_{s}$, which may be less than
128 if the developer is close to the end of the file. We then tokenise the
prefix, up to a maximum of $512-1-n_{s}$; subtracting one off the total
context window here, to allow us to insert a <sep> separator token at the
cursor position. If the total length happens to be shorter than the context
window, we conventionally right-pad the remainder of the sequence. Note that
this doesn’t perfectly centre the cursor in the context window, and assumes
the prefix holds more weight than the suffix. This decision is substantiated
by the results in our online appendix.

###  4.2. Dataset

We train our models on data collected from a code completion plugin developed
at our institution (Section 3.1). Any given code suggestion is either manually
invoked via a key-bind, or automatically on a predefined set of common trigger
characters (e.g., a full stop, or opening paranthesis) (Izadi et al., 2024).
As stated in our objective Section 3.2, we aim to optimise for those
completions that are either (1) manually invoked, or (2) automatically invoked
and accepted.

While we have over 1M invocations of our tool, after filtering for high-
quality samples containing code context (collected on an opt-in basis,
$\sim$200k samples), and balancing our dataset by under-sampling, we maintain
only about 10k code suggestions for training. Our test set, following the
real-world distribution of manual and automatic invocations, contains about
20k samples, completely separate from the training set. We empirically compare
different dataset distributions in our online appendix, motivating the under-
sampled training distribution in this classification scenario where classes
are not equally represented in practice.

Table 1. Class Distribution in Our Code Completion Train/Validation and Test Datasets. Class | Positive |  | Negative
---|---|---|---
Completion | Manual | Automatic
Type | Accepted | Rejected
Test | 6118 (27.8%) | 431 (1.9%) | 15889 (70.3%)
Train/Validation | 3909 (33.3%) | 3909 (33.3%) | 3909 (33.3%)

Table 1 shows the (sub-)class distribution of our train/validation and test
dataset. We purposefully avoid distinguishing between manual accepted, and
manual rejected invocations of our tool, as we consider both to be part of our
positive class. As added context, however, the manual rejected invocations
constitute a total of 47% of our positive class in the real-world (test)
distribution. Unfortunately, we are unable to share our collected dataset as
it contains user-sensitive code context.

##  5\. Experimental Setup

We laid out the code completion plugin context and problem constraints in
Section 3. And, having established our search space for this problem in
Section 4, we now propose how to navigate it.

###  5.1. Research Questions

As defined in Section 3.2, the objective of our models is to label manual and
automatic invocations which are accepted as positive (helpful); and, consider
the remaining automatic invocations that are rejected as the negative class
(unhelpful). To this end, we pose the following research questions:

  * RQ1

How does a transformer model compare to a baseline logistic regression model
at filtering out unhelpful suggestions (offline evaluation)? We fine-tune a
code-pretrained RoBERTa (Liu et al., 2019) model (CodeBERTa) on code
completion snippets collected from our plugin, and evaluate it against a
logistic regression baseline trained on telemetry and selected textual
features. The baseline is inspired by reverse-engineering GitHub Copilot and
reflects the state-of-the-art.

  * RQ2

Can a pre-trained transformer be extended to incorporate an additional
modality consisting of telemetry feature data? We train our JonBERTa variants
(Section 4.1) with telemetry feature data as an additional modality when
making predictions.

  * RQ3

How effective are the above approaches in a real-world setting (online
evaluation)? We deploy our filters in a code-completion plugin to investigate
whether the filters’ decisions align with 34 users in practice. We further
evaluate the computational feasibility of our approach, and note discrepancies
between the offline and online environments.

###  5.2. Evaluation Settings and Metrics

####  5.2.1. Metrics for Offline Settings (RQ1 & RQ2)

To evaluate whether our models can capture the different invocation types that
determine our classes (see Section 3.2), we compute accuracy per manual,
accepted automatic, and rejected automatic subclass. Based on this, we further
compute the macro average accuracy, to serve as a single metric to compare
models on.

We choose macro average accuracy (across classes), as opposed to micro average
(across all samples), as our positive classes are under-represented in our
code-completion dataset. As a result, it is paramount that completions the
developer wants to see are prioritised against the vast majority of
completions that are ignored. We assume that the mistake of filtering out a
completion when a developer would want to see one is worse than showing a
completion when the developer does not want to see one.

As shown in Table 1, some of our classes are considerably under-represented.
The variance due to such a small dataset can become pronounced when training
transformer models. To capture this variance, we train five models on five
train/eval splits (9:1). Then, we bootstrap our accuracy scores on the test
set by alternatingly taking a sample from each of the five models, for a total
of $n=10,000$ samples.

####  5.2.2. Metrics for Online Setting

For our online evaluation, we no longer have a valuable distribution of
manual/automatic classes as we remove the predefined trigger-point invocation
rule. To remedy this, we evaluate completions that pass the filter via (1)
acceptance rate as a proxy for their timing with developers’ mode of thought;
and (2) score accepted completions using CodeBERTScore (Zhou et al., 2023) as
a proxy for their quality. We also measure the latency in milliseconds.

CodeBERTScore is a recently-proposed measure that correlates closest with both
functional correctness and human preference (Zhou et al., 2023). This is
contrary to the oft-seen CodeBLEU, METEOR, and ROUGE-L measures which are
designed for natural languages and do not work well with the syntactic
structure of programming languages (Evtikhiev et al., 2023). CodeBERTScore is
computed by passing the code completion and ground truth (after 30s) through a
code-pre-trained encoder model, and then computes the F3 score based on the
similarity between token embeddings at a layer that correlates best with human
preference and functional correctness.

We further propose the harmonic mean of these two as a single metric to
compare models by. Specifically, optimising only acceptance rate results in
worse completions (Ziegler et al., 2022; Mozannar et al., 2023b). And,
optimising just the content of a completion, does not make the filter align
well with developers, as evidenced by the number of manual completions we
observe in our dataset. We hope this communicates to the reader how these two
measures should be weighed in our framing of the problem.

###  5.3. Feature Engineering and Baselines

The features extracted from our code-completion data for the Logistic
Regression, CodeBERTa, and JonBERTa models, are shown in Table 2. We purposely
avoid providing JonBERTa with features that can be inferred from the code
context (e.g., whether there is whitespace after the cursor), to assert it is
able to leverage that data implicitly. To this end, we define three types of
feature data:

  * T

Telemetry as those features that cannot be directly extracted from a snippet
of code.

  * C

Code context as those textual features that are explicitly extracted by fixed
rules.

  * S

Snippet as the prompt to the completion model, truncated to the filter model’s
context window (512 tokens) using our centred-on-cursor strategy (Section
4.1.3).

Table 2. Features Used in Classification, per Filter Model. |  | Log. Reg. | Code | Jon
---|---|---|---|---
T1 | Time since last completion | ✓ |  | ✓
T2 | Document length | ✓ |  | ✓
T3 | Cursor offset | ✓ |  | ✓
T4 | Offset as percentage | ✓ |  | ✓
T5 | language (20 options) | ✓ |  | ✓
T6 | IDE ( jetbrains / vscode ) | ✓ |  | ✓
C1 | Length of last prefix line | ✓ |  |
C2 | Above, without whitespace | ✓ |  |
C3 | Whitespace after cursor | ✓ |  |
C4 | Last prefix char (ASCII 32-125) | ✓ |  |
C5 | Above, without whitespace | ✓ |  |
S1 | Prefix |  | ✓ | ✓
S2 | Suffix |  | ✓ | ✓

The features for the logistic regression model are inspired by reverse-
engineering Copilot (Thakkar, 2023), with the exception of one feature that
depends on a pre-existing filter (which we do not have). The languages are the
same as the 20 considered by Copilot. We choose to follow Copilot as, to our
knowledge, this is the only filter currently deployed in practice, and
contains most of the significant features found in previous work (Mozannar et
al., 2023b). An extended explanation of these features can be found in our
online appendix.

###  5.4. Configuration and Implementation Details

We fine-tune all transformer models for six epochs with a $2e^{-5}$ learning
rate and $16$ batch size from a public CodeBERTa-
base-v1555https://huggingface.co/huggingface/CodeBERTa-small-v1 checkpoint,
with each epoch containing about 10k training samples. We also use its
tokeniser for snippet features. For the JonBERTa models, we fine-tune from the
3rd-epoch CodeBERTa checkpoint, for an additional three epochs; as we observe
this results in stabler training than training from the public checkpoint.

All of our transformer models are implemented with
PyTorch666https://pytorch.org, and trained on an NVIDIA GeForce RTX 3080 GPU,
taking about $20$ minutes per model. All metrics are computed using the
functions provided by scikit-learn 777http://scikit-learn.org library. For our
online evaluation, we use an inference server with an NVIDIA GeForce RTX 2080
Ti, separate from our training setup. As this is a relatively older GPU, we
expect slightly higher latency during filter inference.

##  6\. Results

###  6.1. RQ1: Impact of Code Context

To address RQ1, we evaluate the contributions from training on snippet
features (S), against telemetry (T), and fixed-rule textual features (C). To
this end, we first consider logistic regression baselines trained on
iteratively more T and C (see Section 5.3). Lastly, Code4Meis cross-
application, so we include a feature for the IDE. In our dataset, JetBrains
users tend to have a higher suggestion acceptance rate, partially because they
support only the popular languages 888Very recently, JetBrains did release a
preview for a cross-language IDE, Fleet: https://www.jetbrains.com/fleet/
which code completion models tend to perform best on (Katzy et al., 2023).

We do not directly extract the weights from its plugin code. Instead, we
retrain the Copilot-style baselines on our own completion-request dataset.
because the data distribution of our completion plugin is likely different due
to its different invocation methods and completion style as explained in
section 3.1.

Table 3. Filter accuracy for Logistic Regression and CodeBERTa classification models, given per invocation sub-class. | Manual | Auto/acc. | Auto/rej. | Avg.
---|---|---|---|---
Logistic Regr. |  |  |  |
Telemetry ${}_{\text{T}1-5}$ | 99.6 $\pm$0.3 | 99.1 $\pm$0.7 | 1.4 $\pm$0.7 | 66.7
\+ Textual ${}_{\text{C}1-4}$ | 98.6 $\pm$0.3 | 66.8 $\pm$4.5 | 61.9 $\pm$1.2 | 75.8
\+ Copilot ${}_{\text{C}5}$ | 98.6 $\pm$0.3 | 66.1 $\pm$4.6 | 63.9 $\pm$0.9 | 76.2
\+ IDE ${}_{\text{T}6}$ | 98.5 $\pm$0.3 | 66.1 $\pm$4.6 | 65.0 $\pm$0.9 | 76.5
CodeBERTa | 98.5 $\pm$0.4 | 74.7 $\pm$4.6 | 73.1 $\pm$1.2 | 82.1

As shown in Table 3, a model trained on telemetry features (T1–5) alone, while
attaining average accuracy of 66.7%, is completely unable to distinguish
automatic invocations that end up being rejected (1.4% accuracy). Thus, it is
necessary to include some explicit textual features (C1–4) for distinguishing
these classes. This intimates that extended code context can be leveraged. We
refer to our replication package for additional experiments with different
granularities of textual features.

Notably, CodeBERTa, trained on solely code snippet (S) features, can
outperform the best baseline on automatically accepted and automatically
rejected queries by $9.6$ and $3.2$ absolute percentage points, respectively.
This indicates that the semantic understanding of code such a transformer
model exhibits propels it past the classification baseline. And, furthermore,
this snippet modality likely contributes orthogonally to the telemetry data,
as both are distinct features that cannot be inferred from each other. This
motivates our architectural exploration to attend to both these modalities in
one classification model.

###  6.2. RQ2: Hybrid JonBERTa Models

To address RQ2, we train JonBERTa models leveraging both snippet and telemetry
modalities and compare them to our new CodeBERTa baseline. This architectural
search space is especially vast for the JonBERTa-attn model, due to the
inclusion of feature embeddings with tuneable parameters. As such, we defer
most of our experiments to the online appendix in our replication package, as
well as a few JonBERTa-head experiments. We choose to only display the first-
layer configuration here999JonBERTa-attn 0L: Ranking 19/67, Median average
score is 81.7. to give telemetry embedding an equally early chance as token
embeddings at communicating in the attention mechanism; and choose an
embedding dimension of $204$ to not incur too many additional parameters.

Table 4. Filter accuracy for CodeBERTa and JonBERTa classification models, given per invocation sub-class. Error Bounds are for $p<0.5$ via Bootstrapping $n=10\;000$. |  | Manual | Auto/acc. | Auto/rej. | Avg.
---|---|---|---|---|---
CodeBERTa |  | 98.5 $\pm$0.4 | 74.7 $\pm$4.6 |  73.1 $\pm$1.2 | 82.1
JonBERTa |  |  |  |  |
head (dense) |  |  98.6 $\pm$0.4 |  78.0 $\pm$6.2 | 71.4 $\pm$5.1 | 82.7
attn (0L) |  |  98.6 $\pm$0.4 | 75.0 $\pm$6.8 | 72.5 $\pm$3.5 | 82.0

In Table 4, JonBERTa-head promisingly shows it can better discern between
completion classes, though our results are less conclusive than before. While
incorporating the telemetry features in the first attention layer of JonBERTa-
attn matches the performance of JonBERTa-head, implying it can learn to
integrate telemetry features with its semantic understanding of code, this may
equally well be attributable to variance. This could be, in part, due to the
limited training set size as revealed by our bootstrap strategy for computing
error bounds (Section 5.2), which we further discuss as an internal threat to
validity in Section 7.1.1.

Regardless, we present these results motivated by the same argument as in
Section 3: Transformer models are becoming increasingly integrated into
software engineering tasks, but also well outside of the field. As a result,
there is a need to integrate additional modalities into pre-trained models, as
tokens are but one of many sources of information that can be leveraged in AI-
powered tools.

###  6.3. RQ3: Online Evaluation

To evaluate how our filters fare in the real world, we deploy them in a code
completion plugin with 34 developers over a 2 week period, resulting in 74k
requests. To this end, we disable the predefined trigger-point constraint that
was described in Section 3.1, to now automatically invoke the completion model
at the historically manual trigger-points.

We perform an A/B study by assigning each user one of the following five
filters per coding session. We define a session as a sequence of completion
requests where any two are no more than 30 minutes apart, to avoid end-user
confusion from different completion behaviour on every request. For all
filters, requests with a prompt (prefix + suffix around the cursor) less than
10 characters are automatically rejected, as they do not have enough context
for a worthwhile completion.

  * 1

None: all completion requests pass through.

  * 2

Logistic regression using telemetry (T) and context (C) features.

  * 3

CodeBERTa using only snippet (S) features.

  * 4 & 5

JonBERTa-head and -attn using both T and S.

Table 5. Completion statistics for filters deployed in Code4Me. The relative acceptance rate is with respect to None. Latency is computed as the median of all requests. Filters |  |  |  | JonBERTa
---|---|---|---|---
None | LogReg. | Code | head | attn
Requests |  |  |  |  |
Received | 13.5k | 10k | 20k | 15k | 13k
Filtered out | 2.2% | 29.4% | 36.6% | 34.4% | 39.4%
Completions |  |  |  |  |
Shown | 97.8% | 70.5% | 63.2% | 65.6% | 60.5%
Accepted | 1.42% | 0.62% | 1.20% | 1.44% | 1.03%
Accepted |  |  |  |  |
Relative rate | 100.0% | 45.0% | 84.5% | 101.0% | 72.3%
CodeBERTScore | 0.76 | 0.94 | 0.85 | 0.82 | 0.88
Harmonic Mean | 0.864 | 0.609 | 0.847 | 0.905 | 0.794
Latency (ms) | 0.0 | 0.2 | 25.0 | 20.8 | 24.1

Table 5 shows our results. Using our proposed harmonic mean to convey the
balance of suggestion quality and timing, JonBERTa-head performs best. While
our proposed metric maintains ordinality among the CodeBERTa and JonBERTa
models, compared to the offline evaluation, this also highlights our proposed
metric is not perfect, as presumably, all our filters should be performing
better than the no-filter baseline. We defer discussion on alternate
weightings for the harmonic mean to future work (Section 7.1.3), to avoid
conflicting interests from tuning our metric here.

Additionally, we demonstrate the feasibility of our approach in practice. A
lightweight transformer model can be deployed server-side as a filter for
incoming requests, with relatively minimal latency compared to the completion
model itself, which takes 300-400ms in our case. Future work can consider
further optimising these models, through e.g., model compression (Wu et al.,
2022), for client-side deployment.

##  7\. Discussion

We anticipate that the issue of redundant invocation will become even more
noticeable as software engineering tasks increasingly incorporate billion-
parameter transformer models. Our results demonstrate the effectiveness of
using a smaller, lightweight transformer to control when a larger, completion
model is invoked. Moreover, we believe smart invocation filtering models such
as ours not only enhance code completion but also any other transformer-based
interaction with users.

To our knowledge, we are the first work to augment a pre-trained transformer
with additional feature modalities. Transformer models exhibit exceptional
contextual understanding, yet are bottlenecked by the textual medium.
Especially considering that in-app telemetry data is often collected anyway,
we highlight that it is fruitful to leverage this additional input dimension.
By showing promising results in this search space, we hope to inspire others
to venture deeper.

###  7.1. Threats to Validity

####  7.1.1. Internal: Limited Contextual Usage Data

In RQ1 and RQ2, we used a training dataset of just 10k samples. This size is
not optimal for fully examining the potential of hybrid transformer models
enhanced with extra feature data. Expanding the dataset in future research
would likely offer a better understanding of these models’ capabilities,
enabling more meaningful comparisons between different architectures.

####  7.1.2. External: Generalisability to Other Code-Completion Tools

We utilized a code-suggestion plugin that has a smaller user base compared to
larger production systems. This choice introduces several factors that might
impact how our findings can be applied to other code completion tools. These
differences have been detailed in Section 3.1. Our approach, while specific,
offers valuable insights but warrants caution when generalizing to other
contexts.

####  7.1.3. Construct: Limitations of the Proxy Metrics

In our online evaluation, we use the harmonic mean of acceptance rate and
CodeBERTScore as our metric to measure performance. This approach, suggested
for further exploration in future studies, allows for adjustments in how each
component is weighted. Consistent with earlier research (Mozannar et al.,
2023b; Ziegler et al., 2022), we acknowledge that these proxy metrics might
not fully capture the usability of the interaction without potentially
compromising it in some other way. To gain a deeper understanding, we suggest
that future work could benefit from qualitative studies, including interviews
with developers, to complement these quantitative measures.

####  7.1.4. Ethical Considerations

Our research received approval from the institutional ethical board and
explicit user consent for data use. Additionally, we have secured explicit
consent from users before collecting and using their information. We chose not
to deeply investigate privacy issues related to developer data usage, as our
invocation filtering models (classifiers) pose fewer privacy risks compared to
generative models. However, to comply with the GDPR, we are unable to share
our dataset as we cannot guarantee anonymity.

##  8\. Conclusion and Future Work

To summarise, we train a transformer-based invocation-filtering model on a
dataset we collected from an open-source code completion plugin, Code4Me. We
show that code context is especially useful in filtering predictions, and
highlight the potential of integrating this information with the telemetry
data collected in an IDE. Lastly, we deploy our filters in practice and show
their practical effectiveness in both offline and online settings.

Future work can more thoroughly explore the search space we have established,
by utilising a larger dataset. Our limited dataset may not fully represent the
diverse behaviours of developers, and related work shows promising results in
personalising the invocation-filtering system (Mozannar et al., 2023b; Chen et
al., 2019). We choose not to explore this avenue in this study to limit our
architectural search space, though strongly advocate for further exploration
in this area.

Lastly, we note that delivering completions exactly when a developer requests
them might not always match what developers truly need in the long run.
Tracking the long-term impact of these completions presents challenges, yet
understanding this is crucial, especially as the use of AI tools shows a link
to increased code changes. This domain deserves further investigation to
better align for lasting developer benefits.

## References

  * (1)
  * Amazon CodeWhisperer (2023) Amazon CodeWhisperer 2023.  AI Code Generator.  Online.  https://aws.amazon.com/codewhisperer/
  * Barke et al. (2022) Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022.  Grounded Copilot: How Programmers Interact with Code-Generating Models.  _Proceedings of the ACM on Programming Languages_ 7, OOPSLA1 (Oct. 2022), 85–111.  https://doi.org/10.1145/3586030
  * Bavarian et al. (2022) Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022.  Efficient Training of Language Models to Fill in the Middle.  http://arxiv.org/abs/2207.14255 arXiv:2207.14255 [cs].
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating Large Language Models Trained on Code.  http://arxiv.org/abs/2107.03374 arXiv:2107.03374 [cs].
  * Chen et al. (2019) Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019.  Gmail Smart Compose: Real-Time Assisted Writing.  http://arxiv.org/abs/1906.00080 arXiv:1906.00080 [cs].
  * Chien et al. (2023) Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. 2023.  Reducing the Carbon Impact of Generative AI Inference (today and in 2035). In _Proceedings of the 2nd Workshop on Sustainable Computer Systems_. 1–7.
  * Codeium (2023) Codeium 2023.  Codeium - Free AI Code Completions.  Online.  https://codeium.com/
  * Evtikhiev et al. (2023) Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023.  Out of the BLEU: how should we assess quality of the Code Generation models?  _Journal of Systems and Software_ 203 (Sept. 2023), 111741.  https://doi.org/10.1016/j.jss.2023.111741 arXiv:2208.03133 [cs].
  * Feurer et al. (2020) Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. 2020.  Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning.  _arXiv:2007.04074 [cs.LG]_ (2020).
  * Feurer et al. (2015) Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015.  Efficient and Robust Automated Machine Learning. In _Advances in Neural Information Processing Systems 28 (2015)_. 2962–2970.
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023.  InCoder: A Generative Model for Code Infilling and Synthesis.  http://arxiv.org/abs/2204.05999 arXiv:2204.05999 [cs].
  * Gemini Code Assist (2023) Gemini Code Assist 2023.  Gemin Code Assist.  Online.  https://cloud.google.com/products/gemini/code-assist
  * GitHub Copilot (2021) GitHub Copilot 2021.  GitHub Copilot: Your AI Pair Programmer.  Online.  https://github.com/features/copilot
  * Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022.  UniXcoder: Unified Cross-Modal Pre-training for Code Representation.  http://arxiv.org/abs/2203.03850 arXiv:2203.03850 [cs].
  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024.  DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence.  http://arxiv.org/abs/2401.14196 arXiv:2401.14196 [cs].
  * Harding and Kloster (2024) William Harding and Matthew Kloster. 2024.  _Coding on Copilot: 2023 Data Shows Downward Pressure on Code Quality_.  Whitepaper. GitClear. 24 pages.  https://gitclear-public.s3.us-west-2.amazonaws.com/Coding-on-Copilot-2024-Developer-Research.pdf
  * Huh et al. (2023) Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. 2023.  The Low-Rank Simplicity Bias in Deep Networks.  http://arxiv.org/abs/2103.10427 arXiv:2103.10427 [cs].
  * Izadi et al. (2022) Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. 2022.  CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences. In _Proceedings of the 44th International Conference on Software Engineering_. 401–412.  https://doi.org/10.1145/3510003.3510172 arXiv:2202.06689 [cs].
  * Izadi et al. (2024) Maliheh Izadi, Jonathan Katzy, Tim van Dam, Marc Otten, Razvan Mihai Popescu, and Arie van Deursen. 2024.  Language Models for Code Completion: A Practical Evaluation. In _46th International Conference on Software Engineering (ICSE)_. ACM/IEEE.  http://arxiv.org/abs/2402.16197 arXiv:2402.16197 [cs].
  * Jetbrains AI (2023) Jetbrains AI 2023.  Jetbrains AI Service an In-IDE Assistant.  Online.  https://www.jetbrains.com/ai/
  * Johnson et al. (2023) Daniel D. Johnson, Daniel Tarlow, and Christian Walder. 2023.  R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents.  (2023).  https://doi.org/10.48550/ARXIV.2303.00732 Publisher: arXiv Version Number: 2.
  * Katzy et al. (2023) Jonathan Katzy, Maliheh Izadi, and Arie van Deursen. 2023.  On the Impact of Language Selection for Training and Evaluating Programming Language Models.  http://arxiv.org/abs/2308.13354 arXiv:2308.13354 [cs].
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient Memory Management for Large Language Model Serving with PagedAttention.  http://arxiv.org/abs/2309.06180 arXiv:2309.06180 [cs].
  * Liang et al. (2023) Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2023.  Understanding the Usability of AI Programming Assistants.  http://arxiv.org/abs/2303.17125 arXiv:2303.17125 [cs].
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  http://arxiv.org/abs/1907.11692 arXiv:1907.11692 [cs].
  * Lozhkov et al. (2024) Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024.  StarCoder 2 and The Stack v2: The Next Generation.  http://arxiv.org/abs/2402.19173 arXiv:2402.19173 [cs].
  * Lu et al. (2022) Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022.  ReACC: A Retrieval-Augmented Code Completion Framework.  http://arxiv.org/abs/2203.07722 arXiv:2203.07722 [cs].
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation.  http://arxiv.org/abs/2102.04664 arXiv:2102.04664 [cs].
  * McKinzie et al. (2024) Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. 2024.  MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training.  http://arxiv.org/abs/2403.09611 arXiv:2403.09611 [cs].
  * Mok (2024) Aaron Mok. 2024.  Estimated Cost of ChatGPT.  https://www.businessinsider.com/how-much-chatgpt-costs-openai-to-run-estimate-report-2023-4?international=true&r=US&IR=T
  * Mozannar et al. (2023a) Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2023a.  Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming.  http://arxiv.org/abs/2210.14306 arXiv:2210.14306 [cs].
  * Mozannar et al. (2023b) Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2023b.  When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming.  http://arxiv.org/abs/2306.04930 arXiv:2306.04930 [cs].
  * Peng et al. (2023) Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023.  The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.  http://arxiv.org/abs/2302.06590 arXiv:2302.06590 [cs].
  * Prather et al. (2023) James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023.  ”It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers.  (April 2023).  https://doi.org/10.48550/ARXIV.2304.02491 Publisher: arXiv Version Number: 1.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code Llama: Open Foundation Models for Code.  http://arxiv.org/abs/2308.12950 arXiv:2308.12950 [cs].
  * Russo (2023) Daniel Russo. 2023.  Navigating the Complexity of Generative AI Adoption in Software Engineering.  http://arxiv.org/abs/2307.06081 arXiv:2307.06081 [cs].
  * SourceGraph Cody (2023) SourceGraph Cody 2023.  Cody - AI Coding Assistant.  Online.  https://sourcegraph.com/cody
  * Stack Overflow (2023) Stack Overflow. 2023.  Stack Overflow Developer Survey 2023.  https://survey.stackoverflow.co/2023/#section-developer-tools-ai-in-the-development-workflow
  * Sun et al. (2023) Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, and Li Li. 2023.  Don’t Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems. In _2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_. IEEE, Melbourne, Australia, 324–325.  https://doi.org/10.1109/ICSE-Companion58688.2023.00089
  * Tabnine (2023) Tabnine 2023.  Tabnine AI Coding Assistant.  Online.  https://www.tabnine.com/
  * Thakkar (2023) Parth Thakkar. 2023.  Copilot Internals.  https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals Publication Title: Copilot-Explorer.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. ACM, New Orleans LA USA, 1–7.  https://doi.org/10.1145/3491101.3519665
  * Vaswani et al. (2023) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023.  Attention Is All You Need.  http://arxiv.org/abs/1706.03762 arXiv:1706.03762 [cs].
  * Wang et al. (2023a) Ruotong Wang, Ruijia Cheng, Denae Ford, and Thomas Zimmermann. 2023a.  Investigating and Designing for Trust in AI-powered Code Generation Tools.  http://arxiv.org/abs/2305.11248 arXiv:2305.11248 [cs].
  * Wang et al. (2023b) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b.  CodeT5+: Open Code Large Language Models for Code Understanding and Generation.  http://arxiv.org/abs/2305.07922 arXiv:2305.07922 [cs].
  * Wu et al. (2022) Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He. 2022.  Extreme Compression for Pre-trained Transformers Made Simple and Efficient.  https://doi.org/10.48550/arXiv.2206.01859 arXiv:2206.01859 [cs].
  * Zhou et al. (2023) Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023.  CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code.  http://arxiv.org/abs/2302.05527 arXiv:2302.05527 [cs].
  * Zhuo (2024) Terry Yue Zhuo. 2024.  ICE-Score: Instructing Large Language Models to Evaluate Code.  http://arxiv.org/abs/2304.14317 arXiv:2304.14317 [cs].
  * Ziegler et al. (2022) Albert Ziegler, Eirini Kalliamvakou, Shawn Simister, Ganesh Sittampalam, Alice Li, Andrew Rice, Devon Rifkin, and Edward Aftandilian. 2022.  Productivity Assessment of Neural Code Completion.  http://arxiv.org/abs/2205.06537 arXiv:2205.06537 [cs].

##  Appendix A Features used in Copilot’s Filter

Based on (Thakkar, 2023), we investigated the v1.57.7193 (June 2022) version
of Copilot and list the features used in their logistic classifier in Table 6.
The weight is the coefficient in the logistic regression model, and the
scaling indicates the transformation applied before multiplying with the
weight. The language and character maps (last three features) are a one-hot
encoding of a fixed set of languages and characters, given in Figure 4 and
Figure 5 respectively.

Before a prompt reaches the filter, two hard-coded rules prevent (1) prompts
containing fewer than 10 characters, and (2) prompts where the cursor is in
the middle of a line, by checking whether there is whitespace after the
cursor; except if there is a closing character on that line, such as a closing
bracket, quote, or semicolon.

Table 6. Features Used in Copilot’s Filter. Feature | Weight | Input Scaling
---|---|---
Previous filter label | $0.997$ | none
Whitespace after cursor | $0.700$ | none
Time since last label | $-0.174$ | log
Length of last prefix line | $-0.230$ | log
Above, without whitespace | $0.134$ | log
Document length | $-0.007$ | log
Cursor offset | $0.005$ | log
Offset as percentage | $0.419$ | none
Document language (map) | $[-0.654,0.358]$ | none
Last prefix char (map) | $[-1.56,1.15]$ | none
Above, without whitespace | $[-1.12,0.85]$ | none

Figure 4. Copilot’s Language Map. Higher-scoring languages are more likely to
get a completion.

[ TODO: desc ] TODO: long desc

Figure 5. Copilot’s Prefix Character Map. Higher-scoring characters (directly
before the cursor) are more likely to get a completion.

[ TODO: desc ] TODO: long desc

The languages in Figure 4 show that more verbose languages like C and Java are
biased to receive more completions, while design-oriented languages like CSS,
SCSS, and TypeScript-React are less likely to receive completions. Notably,
Markdown is least likely to receive completions, likely because the LLMs that
generate completions are primarily trained on code; and, it may be especially
annoying to developers to receive incorrect completions when they are in a
natural-language flow.

The characters in Figure 5 reveal some specific usage patterns too. Many
letters fall close to 0, indicating that they do not have much influence on
the filtering; with the exception of ‘r’, maybe due to the common print-
statement completions which tend to be easier to infer. Furthermore, if the
developer is currently typing a number, this has a negative weight.
Presumably, programmers don’t want completions while they are writing some
uninferrable number. Lastly, perhaps surprisingly, the full-stop ‘.’ has near-
zero weight, while a comma ‘,’ followed by whitespace has positive weight,
implying developers may rely on Copilot to complete list/object structures.

##  Appendix B Comparison of tokenisation Strategies

To investigate the effectiveness of our joint prefix-and-suffix tokenisation
strategy, we compare it to the two alternatives: prefix-only and suffix-only
tokenisation. To this end, we train RoBERTa models from the
huggingface/CodeBERTa-small-v1 checkpoint
101010https://huggingface.co/huggingface/CodeBERTa-small-v1. We use the same
hyperparameters for all models: a learning rate of $2\times 10^{-5}$, a batch
size of 8, and train for 3 epochs.

Table 7. Comparison of tokenisation Strategies. Tokenisation |  | Manual | Auto/acc. | Auto/rej. | Average
---|---|---|---|---|---
Suffix-only |  | $89.3$ | $\mathbf{93.0}$ | $24.1$ | $68.8$
Prefix-only |  | $97.6$ | $61.0$ | $\mathbf{74.7}$ | $77.8$
Joint |  | $\mathbf{98.6}$ | $73.5$ | $71.7$ | $\mathbf{81.3}$

##  Appendix C The Effect of Dataset Distribution on Model Performance

Our code completion dataset distribution does not equally represent the
invocation types we want to classify, yet they should be weighed roughly
equally. To remedy this, we experiment with several data distributions (Figure
6) for training our models. As shown in Table 8 for CodeBERTa, and Table 9 for
the Logistic Regression model, find that the biased distribution, giving
roughly equal weight to all sub-classes and undersampling to better represent
manual, accepted invocations, yields the best macro average performance across
the sub-classes. We observe a similar pattern for all models detailed in
Appendix F. Thus, we use the biased distribution as our training/validation
dataset.

For CodeBERTa in Table 8, the superior performance on the biased balancing may
also be due to our training hyperparameters. We fine-tune for 8 epochs (as
opposed to 6 in the rest of this paper), and our models are prone to
overfitting due to the small training set size. Future work could perhaps
study a data curriculum, going from biased to unbalanced (real-world
distribution) to help the model learn classes quickly, and then generalise to
data more diverse in textual content.

Figure 6. Dataset Distributions

[ TODO: desc ] TODO: long desc

Table 8. CodeBERTa Models under Different Training Data Distributions. |  | Manual | Auto/acc. | Auto/rej. | Average
---|---|---|---|---|---
Unbalanced |  | 98.9 | 19.4 | 98.1 | 72.1
Classes |  | 99.2 | 32.2 | 95.7 | 75.7
Sub-Classes |  | 98.2 | 71.6 | 74.4 | 81.3
Biased |  | 98.6 | 78.9 | 67.3 | 81.6
Table 9. Logistic Regression Models (with full telemetry), under Different Training Data Distributions. |  | Manual | Auto/acc. | Auto/rej. | Average
---|---|---|---|---|---
Unbalanced |  | 97.4 | 0.0 | 99.8 | 65.8
Classes |  | 97.8 | 0.0 | 99.7 | 65.8
Sub-Classes |  | 97.8 | 35.7 | 88.4 | 74.0
Biased |  | 98.5 | 66.1 | 65.0 | 76.5

##  Appendix D JonBERTa-head Architecture Experiments

We list the 3 jonberta-head variants below, with the 2 options for whether to
reinitialise the module or not. We hypothesise that the dense variant performs
best because dense layers allow the model to learn low-rank embeddings, and
speed up training at this scale. For comparison, we provide the baseline
CodeBERTa model trained on code-context alone.

Table 10. JonBERTa-head Variants. R denotes re-initialised modules. | Manual | Auto/acc | Auto/rej | Avg.
---|---|---|---|---
dense-R | 98.6 $\pm$ 0.4 | 78.0 $\pm$ 6.2 | 71.4 $\pm$ 5.1 | 82.7
proj-R | 98.6 $\pm$ 0.6 | 76.1 $\pm$ 11.0 | 72.4 $\pm$ 3.6 | 82.3
dense | 98.6 $\pm$ 0.5 | 75.6 $\pm$ 6.8 | 72.6 $\pm$ 4.4 | 82.3
dense-proj- | 98.6 $\pm$ 0.3 | 73.5 $\pm$ 6.7 | 74.1 $\pm$ 1.8 | 82.1
dense-proj-R | 98.5 $\pm$ 0.5 | 72.2 $\pm$ 5.8 | 75.4 $\pm$ 1.9 | 82.0
proj | 98.4 $\pm$ 0.5 | 72.9 $\pm$ 7.0 | 74.7 $\pm$ 5.0 | 82.0
CodeBERTa | 98.5 $\pm$0.4 | 74.7 $\pm$4.6 | 73.1 $\pm$1.2 | 82.1

##  Appendix E JonBERTa-attn Architecture Experiments

It may seem natural to consider telemetry data as simply another modality that
can be fed to the transformer akin to multimodal large language models
(McKinzie et al., 2024). We choose not to explore this approach as it
introduces too many new parameters for the model to learn in our limited-data
setting. Furthermore, as far as we know, there are no studies investigating
whether an existing natural-language transformer can be extended with
additional modalities without reinitialising (all) its weights. To this end,
we try learning an embedding for each telemetry feature as a function of that
feature, and extend the attention module to attend to these embeddings as
well.

As shown in Table 11, we train a variety of layer configurations, from
individual layers to triplets of layers. We initialise the model from
CodeBERTa-small-v1, provided by
HuggingFace111111https://huggingface.co/huggingface/CodeBERTa-small-v1, and
fine-tune for 6 epochs on our collected code-completion dataset. We note that
the final performance of all models is lower than that of CodeBERTa, and
hypothesise this may be due to the dual training objective; i.e. we are fine-
tuning both as a code-context classification task, and learning telemetry
embeddings simultaneously.

To further investigate, we first fine-tune the CodeBERTa model on code-
completion classification for only 3 epochs, and then fine-tune an additional
3 epochs with the additional telemetry features. The results are shown in
Table 12, where the worst-performing model attains classification accuracy on
par with the best-performing model from Table 11, indicating that better
performance is achievable in this training paradigm.

Table 11. Performance of JonBERTa-attn Variants initialised from RoBERTa-small-v1. Layers | Manual | Auto/acc | Auto/rej | Avg.
---|---|---|---|---
$1,4,5$ | 98.4 $\pm$ 0.4 | 75.9 $\pm$ 6.4 | 69.9 $\pm$ 5.8 | 81.4
$2,4,5$ | 98.4 $\pm$ 0.5 | 77.1 $\pm$ 6.9 | 68.5 $\pm$ 6.4 | 81.3
$0,4,5$ | 98.2 $\pm$ 0.7 | 76.4 $\pm$ 8.0 | 69.1 $\pm$ 5.0 | 81.3
$0,1,4$ | 98.3 $\pm$ 0.4 | 77.3 $\pm$ 5.8 | 68.1 $\pm$ 2.7 | 81.2
$0,1,5$ | 98.3 $\pm$ 0.4 | 77.3 $\pm$ 5.0 | 68.1 $\pm$ 2.8 | 81.2
$0,2,4$ | 98.3 $\pm$ 0.7 | 77.5 $\pm$ 7.1 | 67.9 $\pm$ 5.6 | 81.2
$1,2,3$ | 98.2 $\pm$ 0.5 | 77.6 $\pm$ 5.0 | 67.9 $\pm$ 2.1 | 81.2
$0,1,3$ | 98.4 $\pm$ 0.4 | 78.0 $\pm$ 6.1 | 67.2 $\pm$ 6.1 | 81.2
$3,4,5$ | 98.4 $\pm$ 0.5 | 76.3 $\pm$ 6.2 | 68.7 $\pm$ 3.3 | 81.1
$0,2,5$ | 98.3 $\pm$ 0.4 | 77.4 $\pm$ 5.8 | 67.6 $\pm$ 3.1 | 81.1
$1,2,4$ | 98.4 $\pm$ 0.5 | 77.3 $\pm$ 7.0 | 67.6 $\pm$ 5.0 | 81.1
$0,1,2$ | 98.5 $\pm$ 0.6 | 80.6 $\pm$ 5.2 | 64.1 $\pm$ 6.2 | 81.1
$0,3,5$ | 98.3 $\pm$ 0.5 | 76.0 $\pm$ 7.5 | 68.7 $\pm$ 6.4 | 81.0
$2,3,4$ | 98.3 $\pm$ 0.5 | 76.2 $\pm$ 6.7 | 68.4 $\pm$ 8.2 | 81.0
$2$ | 98.0 $\pm$ 0.6 | 77.1 $\pm$ 5.7 | 67.4 $\pm$ 4.0 | 80.8
$2,3,5$ | 98.3 $\pm$ 0.5 | 77.5 $\pm$ 7.2 | 66.6 $\pm$ 5.2 | 80.8
$0,3,4$ | 98.4 $\pm$ 0.5 | 76.6 $\pm$ 5.9 | 67.2 $\pm$ 4.3 | 80.7
$0,2,3$ | 98.3 $\pm$ 0.7 | 74.9 $\pm$ 10.6 | 68.8 $\pm$ 5.9 | 80.7
$1,2,5$ | 98.3 $\pm$ 0.3 | 76.8 $\pm$ 6.3 | 66.8 $\pm$ 6.1 | 80.6
$1$ | 98.2 $\pm$ 0.5 | 76.6 $\pm$ 7.7 | 66.9 $\pm$ 3.4 | 80.5
$0$ | 98.1 $\pm$ 0.7 | 76.5 $\pm$ 10.0 | 66.8 $\pm$ 5.8 | 80.5
$1,3,5$ | 98.2 $\pm$ 0.6 | 74.7 $\pm$ 10.2 | 68.3 $\pm$ 7.2 | 80.4
$3$ | 98.1 $\pm$ 0.8 | 77.5 $\pm$ 7.8 | 65.3 $\pm$ 5.7 | 80.3
$1,3,4$ | 98.3 $\pm$ 0.6 | 74.1 $\pm$ 7.6 | 68.5 $\pm$ 6.6 | 80.3
$4$ | 98.2 $\pm$ 0.4 | 77.7 $\pm$ 6.4 | 64.5 $\pm$ 5.3 | 80.1
$5$ | 98.2 $\pm$ 0.5 | 78.7 $\pm$ 8.2 | 63.4 $\pm$ 5.9 | 80.1
CodeBERTa | 98.5 $\pm$0.4 | 74.7 $\pm$4.6 | 73.1 $\pm$1.2 | 82.1
Table 12. JonBERTa-attn variants initialised from our fine-tuned CodeBERTa. Layers | Manual | Auto/acc | Auto/rej | Avg.
---|---|---|---|---
$4,5$ | 98.6 $\pm$ 0.5 | 76.6 $\pm$ 5.7 | 72.1 $\pm$ 4.2 | 82.4
$2,3,4$ | 98.5 $\pm$ 0.4 | 74.8 $\pm$ 5.9 | 73.8 $\pm$ 2.4 | 82.4
$0,1,4$ | 98.5 $\pm$ 0.4 | 75.2 $\pm$ 7.6 | 73.1 $\pm$ 4.1 | 82.3
$0,1,3$ | 98.5 $\pm$ 0.4 | 74.8 $\pm$ 4.6 | 73.4 $\pm$ 2.0 | 82.3
$2,5$ | 98.5 $\pm$ 0.4 | 74.9 $\pm$ 4.6 | 73.2 $\pm$ 3.0 | 82.2
$0,5$ | 98.6 $\pm$ 0.3 | 74.6 $\pm$ 7.5 | 73.3 $\pm$ 6.1 | 82.2
$1,2,5$ | 98.6 $\pm$ 0.3 | 75.5 $\pm$ 7.2 | 72.4 $\pm$ 2.7 | 82.2
$2$ | 98.6 $\pm$ 0.4 | 75.8 $\pm$ 5.9 | 71.9 $\pm$ 3.1 | 82.1
$3,4$ | 98.6 $\pm$ 0.4 | 75.7 $\pm$ 5.3 | 72.0 $\pm$ 2.5 | 82.1
$3,4,5$ | 98.4 $\pm$ 0.4 | 74.4 $\pm$ 5.0 | 73.5 $\pm$ 2.3 | 82.1
$3,5$ | 98.5 $\pm$ 0.3 | 74.2 $\pm$ 6.3 | 73.6 $\pm$ 2.7 | 82.1
$0,4,5$ | 98.5 $\pm$ 0.4 | 75.0 $\pm$ 4.9 | 72.8 $\pm$ 2.3 | 82.1
$0,2$ | 98.7 $\pm$ 0.3 | 77.5 $\pm$ 4.6 | 70.1 $\pm$ 1.9 | 82.1
$0,3,5$ | 98.6 $\pm$ 0.4 | 75.4 $\pm$ 6.2 | 72.3 $\pm$ 2.7 | 82.1
$2,3,5$ | 98.4 $\pm$ 0.4 | 72.4 $\pm$ 6.2 | 75.4 $\pm$ 3.4 | 82.1
$0,2,5$ | 98.5 $\pm$ 0.4 | 74.1 $\pm$ 4.3 | 73.5 $\pm$ 1.5 | 82.1
$0,1$ | 98.7 $\pm$ 0.4 | 75.4 $\pm$ 5.6 | 72.1 $\pm$ 2.1 | 82.0
$0,3$ | 98.7 $\pm$ 0.3 | 76.7 $\pm$ 5.0 | 70.7 $\pm$ 3.0 | 82.0
$0$ | 98.6 $\pm$ 0.4 | 75.0 $\pm$ 6.8 | 72.5 $\pm$ 3.5 | 82.0
$1,2,3$ | 98.6 $\pm$ 0.4 | 74.4 $\pm$ 4.4 | 73.1 $\pm$ 1.3 | 82.0
$1,2,4$ | 98.5 $\pm$ 0.4 | 73.1 $\pm$ 7.8 | 74.4 $\pm$ 2.9 | 82.0
$1,5$ | 98.6 $\pm$ 0.6 | 75.3 $\pm$ 9.1 | 72.1 $\pm$ 4.3 | 82.0
$1,4$ | 98.6 $\pm$ 0.5 | 75.8 $\pm$ 7.1 | 71.6 $\pm$ 3.0 | 82.0
$0,1,2$ | 98.6 $\pm$ 0.5 | 74.3 $\pm$ 6.5 | 73.1 $\pm$ 6.0 | 82.0
$0,2,4$ | 98.5 $\pm$ 0.4 | 73.6 $\pm$ 4.6 | 73.7 $\pm$ 3.5 | 81.9
$0,2,3$ | 98.5 $\pm$ 0.5 | 74.3 $\pm$ 5.4 | 73.0 $\pm$ 5.1 | 81.9
$0,4$ | 98.5 $\pm$ 0.4 | 75.0 $\pm$ 5.3 | 72.2 $\pm$ 1.4 | 81.9
$0,3,4$ | 98.6 $\pm$ 0.4 | 76.6 $\pm$ 7.9 | 70.5 $\pm$ 4.2 | 81.9
$1,2$ | 98.6 $\pm$ 0.4 | 74.3 $\pm$ 8.4 | 72.6 $\pm$ 4.2 | 81.9
$1,3,4$ | 98.5 $\pm$ 0.4 | 74.6 $\pm$ 6.8 | 72.6 $\pm$ 3.4 | 81.9
$0,1,5$ | 98.6 $\pm$ 0.5 | 76.4 $\pm$ 10.1 | 70.6 $\pm$ 4.9 | 81.9
$1,4,5$ | 98.6 $\pm$ 0.4 | 73.7 $\pm$ 6.5 | 73.1 $\pm$ 2.2 | 81.8
$1$ | 98.5 $\pm$ 0.4 | 73.8 $\pm$ 7.3 | 73.0 $\pm$ 7.5 | 81.8
$2,3$ | 98.5 $\pm$ 0.5 | 72.4 $\pm$ 7.6 | 74.4 $\pm$ 2.9 | 81.8
$1,3,5$ | 98.6 $\pm$ 0.5 | 73.0 $\pm$ 8.1 | 73.7 $\pm$ 4.8 | 81.7
$2,4,5$ | 98.5 $\pm$ 0.5 | 73.9 $\pm$ 5.5 | 72.8 $\pm$ 4.9 | 81.7
$4$ | 98.5 $\pm$ 0.5 | 75.5 $\pm$ 9.9 | 71.0 $\pm$ 4.4 | 81.7
$2,4$ | 98.5 $\pm$ 0.4 | 73.7 $\pm$ 6.5 | 72.9 $\pm$ 6.7 | 81.7
$3$ | 98.5 $\pm$ 0.3 | 75.0 $\pm$ 8.6 | 71.4 $\pm$ 6.5 | 81.6
$5$ | 98.4 $\pm$ 0.5 | 74.6 $\pm$ 6.9 | 71.8 $\pm$ 5.6 | 81.6
$1,3$ | 98.5 $\pm$ 0.6 | 73.8 $\pm$ 7.3 | 71.9 $\pm$ 5.8 | 81.4

##  Appendix F Alternative Models

We have exhaustively explored alternative approaches to our completion
classification problem, which have guided and motivated our transformer-based
approach. To this end, we train logistic-regression models on telemetry data,
and extend them with various methods of incorporating code context; as well as
AutoML models (Feurer et al., 2015, 2020). Our main finding, as stated in the
paper, is that code context is significantly more important for classification
accuracy than what can be retrieved from telemetry data alone.

The alternative models are shown in Table 13. We compare by encoding $n$
prefix and $n$ suffix words/tokens/lines surrounding the cursor. It is not
fully fair to compare every method with this granularity, as some methods may
perform better with e.g. 3 prefix and the entire suffix encoded, but we do so
for consistency. If we had any remarkable results, this would not be in the
appendix in the first place.

Autosklearn and AutoSklearn2 are ensembles consisting of 22 and 3 models
respectively. We omit these models from the body of our study as we prefer
simpler solutions over complex ones. We further want to highlight the
potential of integrating pre-trained transformers’ semantical understanding
and contextual capabilities with additional modalities like telemetry feature
data.

Table 13. Filter accuracy for Logistic Regression classification models with different code-context encodings, given per invocation sub-class. Macro denotes the macro average of the three sub-classes. Logistic Regr. w/ | Manual | Auto/acc. | Auto/rej. | Macro
---|---|---|---|---
One-Hot Tokens ($\dim=50821$)
3 Tokens | 99.0 | 77.7 | 54.0 | 77.1
2 Tokens | 99.1 | 77.5 | 53.0 | 76.7
1 Token | 99.1 | 75.2 | 50.7 | 73.9
Tok2Vec ($\dim=100$)
3 tokens | 97.7 | 74.0 | 58.8 | 76.0
2 tokens | 98.3 | 75.4 | 56.2 | 76.3
1 token | 98.5 | 76.3 | 52.7 | 75.5
Word2Vec ($\dim=100$)
3 words | 74.9 | 60.3 | 58.2 | 63.7
2 words | 77.5 | 60.6 | 56.3 | 64.2
1 words | 85.1 | 63.3 | 51.7 | 66.7
SetFit ($\dim=768$)
1 line | 78.6 | 58.3 | 66.9 | 70.7
2 lines | 67.4 | 57.6 | 62.1 | 61.4
3 lines | 61.3 | 57.3 | 60.8 | 59.9
TF.IDF ($\dim\in[15000,18500]$)
3 tokens | 93.0 | 84.5 | 41.6 | 72.9
2 tokens | 95.0 | 84.0 | 41.4 | 72.4
1 token | 97.0 | 80.5 | 38.4 | 69.4
1 line | 89.6 | 91.9 | 24.0 | 69.0
AutoSklearn (Feurer et al., 2015) | 98.9 | 75.9 | 64.6 | 79.4
AutoSklearn2 (Feurer et al., 2020) | 98.7 | 73.5 | 65.7 | 78.9

For our One-Hot Tokens and Tok2Vec classifiers, we use the TikToken p50k-base
tokeniser from OpenAI’s Codex models121212https://github.com/openai/tiktoken.
For SetFit, we experiment with three embedding models, and only report all-
mpnet-base-v2131313https://huggingface.co/sentence-transformers/all-mpnet-
base-v2 here as it marginally outperforms.