# How are We Detecting Inconsistent Method Names?
An Empirical Study from Code Review Perspective

Kisub Kim<EMAIL_ADDRESS>Singapore Management
UniversitySingaporeSingapore ,  Xin Zhou<EMAIL_ADDRESS>Singapore Management UniversitySingaporeSingapore ,  Dongsun Kim
<EMAIL_ADDRESS>Kyungpook National UniversityDaeguKorea ,  Julia Lawall
<EMAIL_ADDRESS>InriaParisFrance ,  Kui Liu<EMAIL_ADDRESS>Huawei Software Engineering Application Technology LabHangzhouChina ,
Tegawendé F. Bissyandé<EMAIL_ADDRESS>University of
LuxembourgLuxembourgLuxembourg ,  Jacques Klein<EMAIL_ADDRESS>University of LuxembourgLuxembourgLuxembourg ,  Jaekwon Lee
<EMAIL_ADDRESS>University of OttawaOttawaCanada and  David Lo
<EMAIL_ADDRESS>Singapore Management UniversitySingaporeSingapore

###### Abstract.

Proper naming of methods can make program code easier to understand, and thus
enhance software maintainability. Yet, developers may use inconsistent names
due to poor communication or a lack of familiarity with conventions within the
software development lifecycle. To address this issue, much research effort
has been invested into building automatic tools that can check for method name
inconsistency and recommend consistent names. However, existing datasets
generally do not provide precise details about why a method name was deemed
improper and required to be changed. Such information can give useful hints on
how to improve the recommendation of adequate method names. Accordingly, we
construct a sample method-naming benchmark, ReName4J, by matching name changes
with code reviews. We then present an empirical study on how state-of-the-art
techniques perform in detecting or recommending consistent and inconsistent
method names based on ReName4J. The main purpose of the study is to reveal a
different perspective based on reviewed names rather than proposing a complete
benchmark. We find that the existing techniques underperform on our review-
driven benchmark, both in inconsistent checking and the recommendation. We
further identify potential biases in the evaluation of existing techniques,
which future research should consider thoroughly.

Method Name Recommendation, Consistency Checking, Code Review, Empirical Study

††booktitle: ACM TOSEM††ccs: Software and its engineering Software version
control††ccs: Software and its engineering Empirical software validation

“There are only two hard things in Computer Science: cache invalidation and
naming things.” —- Philip Lewis Karlton (Karlton, 2022)

##  1\. Introduction

A method is the smallest unit of program behavior in a software project (Høst
and Østvold, 2009; Wang et al., 2021). Accordingly, choosing method names that
are consistent with the conventions of the software project and the behavior
of the method body is crucial for software maintenance (Arnaoudova et al.,
2014; Fakhoury et al., 2018), comprehension (Takang et al., 1996; Lawrie et
al., 2006; Schankin et al., 2018; Hofmeister et al., 2017), and reuse (Butler
et al., 2010, 2011a; Arnaoudova et al., 2016; Liblit et al., 2006).
Nevertheless, software developers are not always successful in choosing
appropriate method names. Figure 1 shows a few examples from open-source
projects. In Figure 1a, the method name register suggests that the method
stores some data in an internal container while the method actually creates a
new object and returns it. Thus, the corresponding code review suggests
generateToken instead. In Figure 1b, the method name transitToRestoreActive
suggests that the method changes a program state selectively, while actually,
the method is executed all the time. The corresponding review recommends
enforceRestoreActive.

⬇

public Token register(...) {

...

try {

return new Token(dbsUrl, authenticationName, Instant.now());

} catch (Exception ex) {

LOGGER.log(...);

}

return null;

}

register should be renamed to be consistent with the current approach (was
more meaningful when it was stateful). Proposal: generateToken.

(a) A method from Jenkins project and its corresponding review
comment.222https://github.com/apache/kafka/pull/8319/files/c7ac051e495a98b6c72a340c24f4b9bb1f25dcdd#r395348873

⬇

public void transitToRestoreActive() {

if (state != ChangelogReaderState.ACTIVE_RESTORING) {

...

pauseChangelogsFromRestoreConsumer(standbyRestoringChangelogs());

}

state = ChangelogReaderState.ACTIVE_RESTORING;

}

Renamed this method to make it clear we aren’t necessarily ”transitioning”, we
actually call it all the time now any time we want to ”be in restoreActive”.

(b) A method from Kafka project and its corresponding review
comment.444https://github.com/elastic/elasticsearch/pull/82639

Figure 1. Motivating examples taken from Apache projects.

To alleviate the problem of inconsistent method names, many researchers have
proposed techniques (Liu et al., 2019; Nguyen et al., 2020; Li et al., 2021c;
Allamanis et al., 2015; Li et al., 2021a) to automate two tasks: (1) method
name consistency checking (MCC) and (2) method name recommendation (MNR). A
challenge, however, is how to evaluate such approaches in a meaningful way.
For example, the dataset of Liu et al. (Liu et al., 2019) comes from revisions
that only change method names. While creating a revision that only changes
method names does suggest that the method names were unsuitable, such a
revision does not provide clear evidence of whether the change reflects the
conventions of a community or the opinion of a single developer. Datasets
furthermore have been balanced, to meet the needs of learning methods and
report results on the recognizability of specific classes. But in practice,
method names tend to stabilize over time, and thus most methods in a well-
maintained software project already have consistent names, so a technique
scanning for naming problems has to be able to cope with imbalanced inputs.
Finally, techniques often rely on similarity thresholds that have been
optimized for specific datasets. These issues indicate that there may be a gap
between the research results and reality.

Code review data can provide insight to address the above method naming
challenges. Alsuhaibani et al. (Alsuhaibani et al., 2021) found that method
names are discussed during code review and that developers change method names
in response to comments from reviewers. Such discussions raise an opportunity
to sample method-renaming pairs, in which method name changes are supported by
discussions among maintainers of the affected code base, and to use this
database to assess MCC and MNR techniques. For our benchmark, ReName4J, we
first collect the reviews from pull requests in GitHub (GitHub, 2022). We
leverage naming-related terms (e.g., “naming”, “name”, “inconsistent”, and
“descriptive”) to identify relevant reviews and collect the corresponding
patches, including the actual code files as well as the metadata (e.g., file
hashes, paths, pull request status, and commit information). To extract the
method names, we parse all the collected patches and files, then we map the
extracted names to the reviews. We include 400 sample pairs of method names
(i.e., buggy and fixed), method bodies, and the corresponding reviews.

Based on ReName4J, this paper investigates existing MCC and MNR techniques and
asks the following research questions:

  * •

RQ1: Can the existing MCC/MNR tools retrieve the correct names for the methods
that are considered in code review scenarios?

  * •

RQ2: Do the existing MCC/MNR tools achieve an acceptable detection rate for
consistent method names as well as for those of inconsistent method names?

  * •

RQ3: What happens when we test the MCC/MNR tools on an imbalanced test dataset
reflecting the properties of real world software projects instead of a
balanced test dataset that is artificially selected?

Answering these questions helps fill the gaps between the research and
practice by providing three insights. First, the effectiveness of existing
techniques for measuring MCC/MNR relies on the validity of the potential
ground truth. Second, it is important to consider how current tools fare in
both identifying inconsistencies and verifying consistency. Third, the impact
of the balance or imbalance of test sets on performance should also be
considered. We believe that these research questions could capture the
attention of researchers who are interested in software refactoring
specifically, method name recommendation and consistency checking. Moreover,
the researchers and practitioners can better comprehend various perspectives
to evaluate the relevant tools.

To address these questions, we present an empirical study of how state-of-the-
art techniques detect/recommend the names for test oracles from our benchmark.
The study involves three recent techniques, Spot (Liu et al., 2019), Cognac
(Wang et al., 2021), and GTNM (Liu et al., 2022), for MCC/MNR tasks. Our main
findings are as follows:

  * •

The datasets employed in previous evaluations encompass methods for which
consistency has not undergone comprehensive verification through reviews,
thereby lacking the assurance that the method names are validated. In
contrast, our benchmark includes method names checked and accepted by the
project code reviewers. By providing the verified method names, the benchmark
could be more practical and valuable for more precise evaluation.

  * •

The existing test dataset (Liu et al., 2019) contains many methods that have
simple names consisting of a verb such as “get”, “take”, and “listen”. These
intuitive method names are easy to recommend as method bodies tend to include
many such tokens. However, the method names in code reviews explicitly show
that reviewers prefer more descriptive and comprehensive names. Indeed, the
average number of characters for the existing test method names from the
existing dataset and those of our benchmark are 13.18 and 24.90 (approx.
88.92% longer), respectively.

  * •

The existing techniques show the performance drops for all the tasks on our
human-reviewed method names as compared to results in the original studies.

  * •

The evaluation design of existing recommendation-based techniques needs to be
carefully re-considered as it may be biased for specific cases.

In summary, this paper contributes the following:

  * •

The first in-depth investigation of MCC/MNR tools, reflecting a code-review
perspective.

  * •

A small, but reliable benchmark, consisting of sample pairs of method names
(i.e., buggy and fixed), method bodies, and review comments. This is also the
first manually labeled naming dataset from a code review perspective. It may
serve as a baseline dataset for future MCC/MNR research as the method names
have been confirmed by project reviewers.

  * •

Delving into advanced evaluation methods for MCC/MNR techniques due to
variations in performance measurement protocols within prior literature.
Moreover, examining strategies to construct an improved benchmark that
minimizes manual effort.

The experimental results show that the review-driven benchmark could provide
insights for better naming by exploiting more context available in the reviews
from the experts. The provided reviews in ReName4J also can give intuitions on
how to improve the classification performance by utilizing more information in
the body of the methods.

(a) A review from Apache/Flink project.66footnotemark: 6

(b) A review from Elastic/Elasticsearch project.88footnotemark: 8

Figure 2. Real world code reviews on method names.

##  2\. Motivating Example

The naming of methods is a critical issue for software engineering and
development as method names have a high impact on the comprehension of the
source code (Takang et al., 1996; Maletic and Marcus, 2001). Consequently,
developers actively discuss naming issues frequently during code reviews in
commercial and open-source software projects (Alsuhaibani et al., 2021). We
first investigate the method names and review pairs in open-source projects to
determine whether the explicitly reviewed method names could be a better
ground truth for MCC and MNR tasks.

Figure 2 illustrates two examples of naming-related reviews from
Apache/Shardingsphere and Elastic/Elasticsearch, respectively. Figure 2a shows
a case in which a code reviewer recommends using a more suitable method name
for a test while giving the justification that the current name sounds the
same as that of another method. Figure 2b illustrates a renaming argument
based on the specific naming conventions of the project. Specifically, the
original name was getRealmRef(), which follows a common Java-Bean naming
convention, but the reviewer suggested instead following the Builder naming
convention, illustrated as name(), which implies that the name should be
realmRef().

Investigating various examples, we discovered that the code reviewers tend to
comment on the method names in many ways such that the developers can directly
apply a proposed renaming or be inspired to correct the names. These changes
are explicitly discussed, changed, and accepted. We consider that these cases
are more solid than other renaming cases that are only changed and accepted
without any discussion in software development.

##  3\. Methodology

This section presents an overview of our empirical study. As shown in Figure
3, our study comprises three steps: (1) constructing a novel method naming
benchmark (ReName4J), (2) retraining existing method-name recommendation
techniques, and (3) testing the performance of the techniques with respect to
the benchmark. Our benchmark, ReName4J collects method renaming instances from
code review data such that every instance is backed by discussions of real
developers and reviewers. Based on the benchmark and obtained models, we
conduct two tasks: (1) inconsistency classification of each pair of method
name and body (MCC), and (2) method name recommendation for a given method
body (MNR).

Figure 3. Overall procedure for our study.

###  3.1. Target Techniques

To answer the research questions described in Section 1, we select three
target techniques, Spot (Liu et al., 2019), Cognac (Wang et al., 2021), and
GTNM (Liu et al., 2022) as (1) they have been published recently at top
software engineering venues and (2) their implementations are completely
reproducible.

Spot: Spot (Liu et al., 2019) is designed with the intuition “Methods
implementing similar behavior in their body code are likely to be consistently
named with similar names, and vice versa”. Notably, Spot adopts unsupervised
learning (Hastie et al., 2009) and lazy learning (Aha, 2013) to embed method
names and bodies into numerical vectors. Given a method name and body, it
compares two sets of names. According to the embedding vectors, the first set
consists of other method names similar to the name of the given method and the
second set consists of the names of methods whose bodies are identified as
similar to the body of the given method. When the intersection of these two
sets is larger than a threshold, the method name is recognized as consistent;
otherwise, Spot classifies it as inconsistent (i.e., the MCC task). If the
name turns out to be inconsistent, Spot recommends potential consistent names
for the method based on the comparison results on method bodies (i.e., the MNR
task).

Cognac: Cognac (Wang et al., 2021) is a program-structure-independent approach
aiming to avoid the out-of-vocabulary problem. Unlike prior techniques, Cognac
not only leverages the local context extracted from target methods themselves,
including code tokens and the associated contexts, but it also considers the
global context, which is the contextual information from other methods such as
call relations (i.e., caller/callee information) with the target method.
Technically, it follows the seq2seq paradigm, using the extracted token
sequences to infer method names.

GTNM: GTNM (Liu et al., 2022) is a transformer-based neural model considering
the information of the whole project (global context). It extracts context
from three different levels, given the target method and the project,
including (1) local context, (2) project-specific context, and (3)
documentation context. Technically, it employs a transformer-based seq2seq
framework (Vaswani et al., 2017) with two encoders and a single decoder to
generate the method name. Particularly, we include GTNM in our study because
its design closely resembles that of Cognac while it misses the evaluation of
MCC. Since Cognac has already showcased that an MNR technique can effectively
serve as an MCC task, it is crucial to undertake a comprehensive performance
assessment in this context. This addition could shed light on the potential
advantages associated with choosing the right approach for tasks that
necessitate a specific tool.

Two other approaches, MNire (Nguyen et al., 2020) and DeepName (Li et al.,
2021c) have been recently proposed and shown to outperform previous
approaches. We had to discard MNire (Nguyen et al., 2020) due to the
unavailability of the source code (Wang et al., 2021). Fortunately, Cognac
(Wang et al., 2021) has been compared against the numbers in the MNire paper
with the same dataset, and Cognac outperforms MNire. We failed to execute
DeepName (Li et al., 2021c) based on its replication package, as the
instructions are not complete, and the author who is in charge of the package
did not reply to our questions. Nevertheless, DeepName is similar to Cognac,
in that both use caller/callee information, and both were released at almost
the same time. Both Cognac and DeepName were evaluated against MNire, and they
show approximately 10% and 7% improvement, respectively, in F1-score with the
same dataset for the MNR task.

###  3.2. ReName4J: Naming Review Benchmark

Our study first builds a benchmark based on code review data, which can be
used to assess method-name recommendation techniques. To the best of our
knowledge, there is no concrete benchmark for method (re)naming that
incorporates code-review data. Existing techniques are evaluated on a dataset
collected from source code revision history without an associated explanation
for the naming changes. Distinguishing between true renaming tasks and
coincidental code changes proves challenging, as developers often disperse a
unified modification across multiple commits (Wen et al., 2020, 2022). In
contrast, ReName4J leverages code review data to collect actual method
renaming tasks, supported by justifications of why the name should be revised.
Notably, the dataset could be applied to both traditional and learning-based
approaches.

ReName4J has four components for each method renaming task: (1) ReviewComment:
the review comment explaining why the name should be revised, (2) MethodBody:
the relevant method body (i.e., source code), (3) BuggyName: the method name
before refactoring (buggy_method_name()), and (4) FixedName: the method name
after refactoring (fixed_method_name()).

To collect the review and method name pair dataset, we leverage GitHub’s REST
API.999https://docs.github.com/en/rest We first crawl the top 100 Java
projects sorted by the stars, considering the project’s popularity (Gu et al.,
2018). Among the top 100 Java projects, we discard the projects where the
discussions are primarily written in languages other than English. This leaves
$66$ projects as our targets. We then crawled a total of $144,759$ pull
requests for these projects. As we only need to consider the pull requests
with naming-related reviews, we extract only those pull requests that contain
naming-related keywords such as “naming”, “name”, “inconsistent”, and
“descriptive”. Based on the process, we collected $11,301$ pull requests. A
pull request tends to include multiple commits, and a commit can be associated
with multiple files. Our dataset thus covers $73,985$ commits and a total of
593,240 associated files.

We preprocess the collected code review data as follows. We break down the
patch files to discover the methods where only the name changes while
excluding those whose names have been changed with their body as changes in
the body can naturally affect the name. This step results in 1,212 changed
method names. Connecting reviews with their corresponding methods is difficult
due to the structure of GitHub’s database, as it does not allow for direct
linking between a review comment and a specific method. Typically, review
comments pertain to an entire pull request. Another obstacle to mapping
reviews to methods is that reviewers often comment on external code links that
cannot be accessed through GitHub’s API but are visible on the web. Finally,
we tackle the issue by establishing a connection between the patch files
containing commit modifications and the metadata of pull requests. This
metadata encompasses crucial information such as commit hashes, parent hashes,
and review comments. The authors manually investigated all the pairs of
reviews and method names and only included those that had a clear connection
between the review and the method. The examples in Section 2 also demonstrate
how this connection can be manually labeled by matching the context of the
reviews and methods.

The manual checking and filtering criteria are as follows:

  * •

We include methods where the pull request was accepted by the code reviewer of
the project.

  * •

We include methods with names that are directly suggested by the reviewer
(e.g., testGetConnectionSession() $\rightarrow$ assertGetConnectionSession() —
Based on the review: Please rename ‘testGetConnectionSession’ to
‘assertGetConnectionSession’101010https://github.com/apache/shardingsphere/pull/18618).

  * •

We include methods with names that are undoubtedly inspired by the review
(e.g., completeRestoration() $\rightarrow$ completeRestorationIfPossible() —
Based on the review: nit: can we rename this to
‘maybeCompleteTaskTypeTransition’ or ‘completeTaskTypeConversionIfNecessary’,
etc? Right now, it kind of sounds like we just randomly attempt to convert it
to a new task type out of
nowhere.111111https://github.com/apache/kafka/pull/8988).

  * •

We include methods with names that are recommended based on the project’s
naming convention (e.g., realmRef() $\rightarrow$ getRealmRef() — Based on the
review: Nit: existing getter methods in this class following the Builder type
naming convention, e.g., name() instead of getName(), while this method uses
the JavaBean naming convention (getXxx). I am OK with either of them but would
prefer consistency within a single
class121212https://github.com/elastic/elasticsearch/pull/82639).

  * •

We include methods with names that are reviewed and commented on for
consistency (e.g., register() $\rightarrow$ getToken() — Based on the review:
‘register’ should be renamed to be consistent with current approach (was more
meaningful when it was
stateful)131313https://github.com/jenkinsci/jenkins/pull/4239).

  * •

We ignore simple typos, as they can be noise in the benchmark, as well as
example methods.

The selected commits satisfy at least one of the defined criteria. As a
result, we identify, as the ground truth, 400 method renaming tasks supported
by the corresponding reviews. In addition, we explicitly eliminate all the
class files associated with the corresponding methods from the training data
to ensure that there is no data leakage (Ribeiro et al., 2016). The absolute
number is smaller than the datasets used in the prior studies (Liu et al.,
2019; Wang et al., 2021; Liu et al., 2022), but our benchmark collects only
obvious renaming tasks confirmed by code reviewers and developers. This allows
a more precise evaluation of method renaming techniques.

To understand the characteristic differences between the existing test oracles
(Liu et al., 2019) and those from our benchmark, we further conduct a brief
preliminary investigation. Based on our observation, the average number of
characters for the existing test method names and those of our benchmark are
13.18 and 24.90 (approx. 88.92% longer), respectively, while those of split
sub-tokens are 2.52 and 4.53 for each method name. This indicates a clear
difference whereas our benchmark contains method names that are more
descriptive, and it suggests that the code reviewers, in practice, prefer to
describe as many parts of the methods as possible.

###  3.3. Experiment Design

To conduct a fair comparison, we employ the following procedure (Wang et al.,
2021) for all three target techniques. Overall, we evaluate the three
techniques described in Section 3.1 with two tasks: Method Name Recommendation
(MNR) addressing RQ1 and Method Name Consistency Checking (MCC) addressing
RQ2, after retraining them by using the datasets. Then, we feed the method
name/body pairs in ReName4J to the (retrained) target techniques and measure
the performance with respect to several metrics. In addition, our study
evaluates the techniques on more realistic data (i.e., a highly unbalanced
dataset including the name/body pairs in ReName4J) to see their effectiveness
in a different scenario.

####  3.3.1. Retraining

To avoid bias from the dataset, we re-train the recommendation-based target
techniques (denoted as $\mathbb{M}$, e.g., $\mathbb{M}_{Cognac}$ and
$\mathbb{M}_{GTNM}$) with a single identical dataset. We leverage the Java-
large dataset, released by Alon et al. (Alon et al., 2019a), as the training
and validation dataset for two, Cognac (Wang et al., 2021) and GTNM (Liu et
al., 2022) of our target techniques. This dataset is the most popular (Alon et
al., 2019a, b; Nguyen et al., 2020; Wang et al., 2021; Li et al., 2021c; Liu
et al., 2022) and well-maintained for the method name recommendation task. It
consists of 10,222 top-ranked Java projects from GitHub, including 14,458,828
methods and 1,807,913 unique files. We randomly shuffled and split all the
projects into 9,772 training and 450 validation projects. In addition, to
avoid any data leakage, we carefully checked and eliminated all methods
included in ReName4J (Section 3.2) from the training dataset so that there is
no intersection between training and testing datasets. Table 1 shows the
training and validation data statistics.

Table 1. Statistics of the training and validation dataset. | Train | Validation | Total
---|---|---|---
Projects | 9,772 | 450 | 10,222
Files | 1,756,282 | 51,631 | 1,807,913
Methods | 13,992,028 | 466,800 | 14,458,828

Exceptionally, we use a different training dataset for the checking-based
approach (denoted as $\mathbb{M}_{Spot}$), Spot, as it has an already-known
scalability issue that we confirmed with its authors (Liu et al., 2019). As
Java-large is almost seven times bigger than the dataset originally used for
training Spot, the authors claimed that it is impossible to train Spot with
the Java-large dataset. Therefore, we utilize their original training dataset
to reproduce the model. The original dataset for Spot includes 2,116,413
methods extracted from 430 Java projects.

####  3.3.2. Method Name Recommendation (MNR) Task

The first task to measure the effectiveness of the target techniques is
recommending method names for a given method implementation (i.e., method
body). In this task, each target technique, $\mathbb{M}$, takes a method body
(MethodBody in ReName4J) and produces a set of candidate method names that
best describe the body, as described in the following equation:

(1) $\mathbb{M}:B^{\ast}\rightarrow N^{\ast}$

where $B$ is an alphabet allowed for a method body and $N$ is an alphabet
allowed for a method name, respectively. $B^{\ast}$ and $N^{\ast}$ are sets of
sequences over $B$ and $N$, respectively. After feeding a method body, $b\in
B^{\ast}$, available in ReName4J, we produce a recommended name by
$\mathbb{M}(b)=n\in N^{\ast}$. We then compare $n$ with our ground truth name,
FixedName, in the benchmark. The results of this task for each technique are
discussed in Section 4.1 as RQ1.

While Cognac (Wang et al., 2021) and GTNM (Liu et al., 2022) were designed for
method name recommendation, Spot (Liu et al., 2019) was originally built for
consistency checking and later added recommendation functionality. To ensure a
fair comparison, we replaced the target method names of our test methods with
meaningless tokens when using Spot, which only predicts method names when they
are classified as inconsistent.

####  3.3.3. Method Name Consistency Checking (MCC) Task

Instead of recommending a name directly, it might be useful if a technique
clarifies whether a given method name is inappropriate to describe the method
body. This consistency check can help developers or code reviewers scan a
project and figure out the overall naming practice. Equation 2 represents the
MCC task, $\mathbb{D_{\mathbb{M}}}$, for a given method body $b\in B^{\ast}$,
method name $n\in B^{\ast}$, and specific name recommendation technique
$\mathbb{M}$.

(2) $\mathbb{D_{\mathbb{M}}}:B^{\ast}\times N^{\ast}\rightarrow\\{C,IC\\}$

where $C$ and $IC$ denotes consistent and inconsistent name. Basically, we
expect that the verdict should be $C$ when feeding a method body (MethodBody)
and corresponding FixedName in ReName4J. Otherwise, it should be $IC$ if
BuggyName is given. The results of this task are discussed in Section 4.2 as
RQ2.

State-of-the-art techniques (Nguyen et al., 2020; Li et al., 2021c) claimed
that it is possible to conduct method name consistency checking based on
method name recommendation by employing a specific similarity checking metric,
described in the following subsection with other metrics. Several prior method
name recommendation approaches (Wang et al., 2021; Li et al., 2021c; Liu et
al., 2022) use this hypothesis to check the consistency of the method names
obtained using their recommendation models.

####  3.3.4. Reflection of the Real World

In the real world, the likelihood of encountering method names that are
inconsistent is significantly lower than those that are consistent. To
accurately reflect this situation, we adopt another dataset. As our target
methods in the benchmark are small parts of multiple class files, there exist
remaining methods from these files (i.e., the majority of Java class files
tend to consist of multiple methods). The quantity of these remaining methods
is substantially greater (13,137) in comparison to our test oracles (400).
These are stable methods that have not been modified, and thus, we consider a
name of a method as StableName and a body as StableBody. Note that these are
not explicitly identified as stable during the code review process, but this
is a common assumption as checking every single method that is not reviewed is
practically infeasible. Also, this is the same process with a prior study (Liu
et al., 2019). This implies that we can use them to simulate a realistic
scenario, and we utilize them for our experimentation in addressing real world
reflection (i.e., RQ3), which aims to reflect reality. To conduct an
experiment on such a dataset, we take the same protocol as the MCC task while
the verdict should be $C$ when feeding a StableName. Otherwise, it should be
$IC$ if a BuggyName is given. The results of this task are reported in Section
4.3 as RQ3.

###  3.4. Performance Metrics

####  3.4.1. Metrics for Method Name Recommendation

As the objective of the MNR task is to recommend the same or similar names to
those that human developers wrote, we compare the ground truth names in the
ReName4J with the names generated by the target techniques. Note that the unit
of comparison is tokenized words (i.e., sub-tokens) of the names rather than
the whole chunk of the names to capture partial matches. We employ Precision,
Recall, F1-score, and EMAcc (Exact Match Accuracy) to evaluate the results of
the MNR task, as represented in Equations 4, 5, 6, and 7, respectively.

We use the followings notations for the equations: $o\in N^{\ast}$ and $p\in
N^{\ast}$ are a ground truth name (i.e., test oracle) in ReName4J and a name
recommended by a targeted technique based on the method body corresponding to
$o$, respectively. $token()$ is a function defined from $N^{\ast}$ to its
power set as follows:

(3) $token:N^{\ast}\rightarrow\mathcal{P}(N^{\ast})$

where $token(name)$ produces a token set after tokenizing $name$. This study
applies the camel case tokenization as it is the naming convention of Java.

  * •

$\mbox{Precision}_{MNR}$ (positive predictive value): is the metric that
represents an estimation of how many tokens are correctly predicted within all
the predicted tokens. It is expressed as follows:

(4) $\mbox{Precision}_{MNR}(o,p)=\frac{\left|token(p)\cap
token(o)\right|}{\left|token(p)\right|}$

  * •

$\mbox{Recall}_{MNR}$ (sensitivity): is the metric that represents an
estimation of how many tokens are correctly predicted within all the oracle
tokens. It is expressed as follows:

(5) $\mbox{Recall}_{MNR}(o,p)=\frac{\left|token(p)\cap
token(o)\right|}{\left|token(o)\right|}$

  * •

$\mbox{F1-score}_{MNR}$: is the harmonic mean of the precision and recall. It
weights the two ratios (precision and recall) in a balanced way:

(6) $\mbox{F1-score}_{MNR}(o,p)=\frac{2\times Precision(o,p)\times
Recall(o,p)}{Precision(o,p)+Recall(o,p)}$

Furthermore, we take $\mbox{EMAcc}(o,p)$, i.e., Exact Match Accuracy, to
assess whether a technique can recommend a name exactly the same as the ground
truth name in ReName4J. The function is defined as:

(7) $\mbox{EMAcc}:N^{\ast}\times N^{\ast}\rightarrow\\{0,1\\}$

where $\mbox{EMAcc}(o,p)=1$ if $o$ and $p$ are identical. Otherwise, the value
is 0.

####  3.4.2. Metrics for Method Name Consistency Checking

To measure the performance on the MCC task, we use metrics from prior studies
(Liu et al., 2019; Wang et al., 2021; Nguyen et al., 2020; Li et al., 2021c):
Precision, Recall, F-score, and Accuracy. MCC can have four possible outcomes:
IC classified as IC (i.e., true positive=TP), IC classified as C (i.e., false
negative=FN), C classified as C (i.e., true negative=TN), and C classified as
IC (i.e., false positive=FP). Accordingly, the metrics are defined as follows:
$\mbox{Precision}_{IC}=\frac{\left|TP\right|}{\left|TP\right|+\left|FP\right|}$,
$\mbox{Recall}_{IC}=\frac{\left|TP\right|}{\left|TP\right|+\left|FN\right|}$,
$\mbox{Precision}_{C}=\frac{\left|TN\right|}{\left|TN\right|+\left|FN\right|}$,
$\mbox{Recall}_{C}=\frac{\left|TN\right|}{\left|TN\right|+\left|FP\right|}$,
the F1-score is computed as $\frac{2\times Precision\times
Recall}{Precision+Recall}$ for each class. Moreover, the $Accuracy$ on the
whole dataset is defined as
$\frac{\left|TP\right|+\left|TN\right|}{\left|TP\right|+\left|FP\right|+\left|TN\right|+\left|FN\right|}$.

Additionally, the recommendation-based approaches need a threshold value $T$,
which is used for the similarity between the predicted name and buggy or fixed
names. It is manually set for the MCC task in prior studies (Wang et al.,
2021; Nguyen et al., 2020; Li et al., 2021c), and following them, we set it as
0.85 in this study.

##  4\. Experimental Results

This section reports all the results of our target techniques and answers the
research questions.

###  4.1. RQ1: Can the target techniques recommend the correct names for a
given method body?

The first research question is whether the target techniques listed in Section
3.1 can perform well for the MNR task when they are evaluated on ReName4J
instead of their original datasets. Following the protocol described in
Section 3.3.2, we feed each MethodBody in ReName4J to each technique and take
the corresponding recommended name as a result. After getting the results, we
compute the performance based on the metrics defined in Section 3.4.1.

The target techniques show lower performance on ReName4J than in their
original studies. Table 2 shows the MNR results as $x/y$, where $x$ is our
result and $y$ (in light grey) is the previously reported result. Note that
Spot (Liu et al., 2019) did not report on the performance of the MNR task with
the metrics described in Section 3.4, but rather mainly uses first token
matching. Thus, we cannot directly compare with the original performance of
Spot. Still, Spot performs worst among the target techniques.

Table 2. Results for the Method Name Recommendation.

|  |  Spot (Liu et al., 2019)
---
|  Cognac (Wang et al., 2021)
---
|  GTNM (Liu et al., 2022)
---
EMAcc |  3.50 / - |  8.57 / 51.80 |  14.73 / 62.01
$\mbox{Precision}_{MNR}$ |  30.79 / - |  38.59 / 71.40 |  47.39 / 77.01
$\mbox{Recall}_{MNR}$ |  22.90 / - |  25.12 / 61.90 |  41.80 / 74.15
$\mbox{F1-score}_{MNR}$ |  26.27 / - |  30.49 / 66.30 |  44.42 / 75.60

  * The gray values (nn.nn) denote the reported results in their papers while ‘-’ indicates that such results are not available.

Thanks to ReName4J, where all renames are backed by code review (i.e., the
ReviewComment component), we can figure out the reason why the techniques fail
to recommend proper method names. Cognac incorrectly recommends size() for a
method, and the following
review141414https://github.com/apache/kafka/pull/1486#discussion_r66461876
corresponds to the case in which BuggyName and FixedName are size() and
approximateNumEntries(), respectively:

Reviewer A: It seems like this method doesn’t return an exact “size” for the
RocksDb implementation so we need to spec it differently if we want to include
it. We should also consider the right name and whether we actually want to
expose it.
…
Reviewer B: “approximate” sounds good. Also to differentiate with number of
bytes as @anotherReviewer pointed out on the ticket, how about use
“approximateNumEntries”?

According to the review comment corresponding to the method name, the
reviewers suggested “approximate…” since the function returns approximated
size instead of the exact “size” of a specific container.

ReName4J can help improve a method name recommendation technique as it
provides the rationale of a method renaming, written by real developers. For
example, the above example suggests that researchers should focus more on
semantic and global context information rather than local context. Here is
another
example:151515https://github.com/apache/kafka/pull/11689#discussion_r788921625
Cognac and GTNM recommends withPartition() and createTopic(), respectively,
for the following MethodBody:

⬇ 1{ 2 return new TopicPartition( 3 record.topic(), record.partition() == null
? 4 RecordMetadata.UNKNOWN_PARTITION : record.partition()); 5}

while the actual BuggyName and FixedName in ReName4J are
createTopicPartition() and extract- TopicPartition(). The corresponding
ReviewComment is:

Reviewer: This (“createTopicPartition”) is not a very good name since
“creating a partition” usually means calling createPartitions. How about
“extractTopicPartition”?

This example motivates that we need to leverage global context such as other
source code in the same project since there are similar method implementations
names (e.g., createPartitions() and createTopics()) in other source files.

On ReName4J, the techniques are less successful at recommending appropriate
method names (MNR) than in previous evaluations. Code review comments may
provide insights for better name recommendations, such as exploiting context
information available in other methods of the same classes or modules.

###  4.2. RQ2: Can the target technique detect inconsistent method names?

This research question aims to examine the effectiveness of each target
technique with respect to the MCC task described in Section 3.3.3. Note that
we follow the hypothesis described in a prior study (Liu et al., 2019) that
the methods that have not been commented on in pull requests to be renamed
have consistent names. We feed a pair of a BuggyName and the corresponding
MethodBody in ReName4J to each technique; we expect that these are classified
as inconsistent names. In addition, we feed a pair of a FixedName and the
corresponding MethodBody; we expect that they are classified as consistent
names.

After constructing a confusion matrix based on the four possible outcomes, we
compute values for the four metrics as described in Section 3.4.2. The
computed values are shown in Table 3. The results follow the form $x/y$ used
in Table 2, where $x$ is the result on ReName4J, and $y$ is the previously
reported result. As there are no previous MCC results for GTNM, the table does
not include the previous values in this case. Nevertheless, it is evident that
GTNM can serve as a valuable tool for consistency checking. This is underlined
by the fact that a similar methodology has been employed in the development of
Cognac, which also aims at name recommendations.

As for the MNR task, most of the values drop from the original studies except
for the case of $\mbox{Recall}_{IC}$ for Spot. The overall degradation is less
than the MNR task, but some cases are still significant. For example, the
$\mbox{Recall}_{C}$ and $\mbox{F1-score}_{C}$ values of Spot and Cognac drop
from 38.20% and 55.60% to 5.00% and 14.25%, respectively. This indicates that
most of the FixedName and MethodBody pairs are incorrectly classified as
inconsistent ones. Although the drop of $\mbox{Precision}_{IC}$ and
$\mbox{Precision}_{C}$ is not as significant as the drop in
$\mbox{Recall}_{C}$ and $\mbox{F1-score}_{C}$, the values are just around
$\approx 50$, which indicates that the performance is not better than random
classification, as the MCC task is fundamentally a binary classification.

Table 3. Results for Method Name Consistency Checking.

|  |  Spot (Liu et al., 2019)
---
|  Cognac (Wang et al., 2021)
---
|  GTNM (Liu et al., 2022)
---
| $\mbox{Precision}_{IC}$ |  50.52 / 56.80 |  48.27 / 68.60 | 43.56 / -
IC | $\mbox{Recall}_{IC}$ |  97.00 / 84.50 |  80.00 / 97.60 | 66.00 / -
| $\mbox{F1-score}_{IC}$ |  64.44 / 67.90 |  60.21 / 80.60 | 52.49 / -
| $\mbox{Precision}_{C}$ |  62.50 / 72.00 |  41.61 / 96.00 | 29.90 / -
C | $\mbox{Recall}_{C}$ |  5.00 / 38.20 |  14.25 / 55.60 | 14.50 / -
| $\mbox{F1-score}_{C}$ |  9.26 / 49.90 |  21.23 / 70.40 | 19.53 / -
Accuracy |  51.00 / 60.90 |  47.12 / 76.60 | 40.25 / -

  * The gray values (nn.nn) denote the reported results under their own test oracle, while ‘-’ indicates that such results are not available.

The following example shows the reasons for some failures. Cognac and GTNM
differently classify the following pair of BuggyName and MethodBody as
consistent and inconsistent, respectively.

  1. (1)

BuggyName: append()

  2. (2)

MethodBody:

⬇ 1{ 2 return append(lastOffset < 0 ? baseOffset : lastOffset + 1, 3
timestamp, key, value); 4}

whereas the FixedName is appendWithOffset() corresponding to the MethodBody.
Both Cognac and GTNM classify the pair of this FixedName and the MethodBody as
inconsistent. The
ReviewComment161616https://github.com/apache/kafka/pull/2282#discussion_r93328221
is as follows:

Reviewer: Hmm, can we think of another name for the methods that increment the
offset automatically?
…
Committer: Or maybe this can be the default and the other can be named
“appendWithOffset”?
…
Reviewer: That works for me.

The techniques fail to correctly classify both the BuggyName and the FixedName
since they recommend append() and addrecord() as method names. This indicates
that the techniques should better utilize the information available in
MethodBody as it mentions “Offset” tokens several times. In addition, the
reviewers talk about the method incrementing the offset automatically.

The target techniques achieve a performance similar to random classification
as they overclassify consistent names as inconsistent names. The reviews in
ReName4J can give intuitions on how to improve the classification performance,
such as utilizing more information in the method body.

###  4.3. RQ3: What happens when we test the effect of balanced vs. imbalanced
data?

The objective of this research question is to assess the efficacy of each
target technique when evaluating them on an imbalanced dataset (a huge number
of consistent pairs and a small number of inconsistent ones), which reflects
the MCC task in real practice, as outlined in Section 3.3.4. Similar to RQ2,
we feed the pairs of method names and bodies. For inconsistent pairs, we
assess all the BuggyName and MethodBody inconsistent pairs in ReName4J using
each technique. For consistent pairs, we assess all the StableName and
StableBody pairs from the dataset described in Section 3.3.4; we expect that
they will all be classified as consistent. This is the ‘imbalanced dataset’
(400 vs. 13,137) in this study.

On the imbalanced dataset, the target techniques mostly fail to classify
consistent names correctly. The results are shown in Table 4 and denoted as
$x/y$, where $x$ is the result of the imbalanced dataset described above,
while $y$ is the result for the balanced dataset (same with Table 3).
$Precision_{IC}$ values are significantly decreased, with a decline of 90.40%,
94.39%, and 94.90% for Spot, Cognac, and GTNM, respectively. As only a few
inconsistent pairs are classified as consistent, $Precision_{C}$ values rise,
with an increase of 55.82%, 131.36%, and 212.41%, respectively.

Table 4. Results for Method Name Consistency Checking with the real world
reflecting dataset.

|  |  Spot (Liu et al., 2019)
---
|  Cognac (Wang et al., 2021)
---
|  GTNM (Liu et al., 2022)
---
| $\mbox{Precision}_{IC}$ |  4.85 / 50.52 |  2.71 / 48.27 |  2.22 / 43.56
IC | $\mbox{Recall}_{IC}$ |  97.00 / 97.00 |  80.00 / 80.00 |  66.00 / 66.00
| $\mbox{F1-score}_{IC}$ |  9.24 / 64.44 |  5.25 / 60.21 |  4.30 / 52.49
| $\mbox{Precision}_{C}$ |  97.39 / 62.50 |  96.27 / 41.61 |  93.41 / 29.90
C | $\mbox{Recall}_{C}$ |  5.56 / 5.00 |  15.25 / 14.25 |  14.24 / 14.50
| $\mbox{F1-score}_{C}$ |  10.52 / 9.26 |  26.32 / 21.23 |  24.72 / 19.53
Accuracy |  9.89 / 51.00 |  17.11 / 47.12 |  15.73 / 40.25

  * The gray values (nn.nn) denote the previous results from Section 4.2.

The results in Table 4 indicate that the techniques are highly biased to
classify method name body pairs as inconsistent ones. The following
pair171717https://github.com/wangpeibin713/flink/blob/748ad82745d9d493922150fe007136e125a50209/flink-
table/flink-table-runtime-
blink/src/test/java/org/apache/flink/table/runtime/operators/deduplicate/ProcTimeDeduplicateKeepLastRowFunctionTest.java
shows an example of incorrect classification. Both Cognac and GTNM classify
the following pair of StableName and StableBody as inconsistent with different
recommendations.

  1. (1)

StableName: testWithStateTtlDisabled()

  2. (2)

StableBody:

⬇ 1{ 2 ProcTimeDeduplicateKeepLastRowFunction func = 3
createFunctionWithoutStateTtl(true, true); 4 ... 5
testHarness.processElement(insertRecord("book", 1L, 12)); 6 ... 7 // Keep
LastRow in deduplicate may send UPDATE_BEFORE 8 ... 9}

Both Cognac and GTNM fail to correctly recommend StableName. They suggest
testKeepLastRow- Function() and testInsertWithUpdate(), respectively. This
result indicates that the techniques need to further focus on the invoked
methods as the key was the method, createFunctionWithoutStateTtl().

Overall, the techniques need to relax the threshold values when detecting
inconsistent pairs. For the following
StableBody,181818https://github.com/open-
beagle/shardingsphere/blob/5be6a2c81c647cdffb1f7f17db9c69204be14b4e/shardingsphere-
proxy/shardingsphere-proxy-
backend/src/test/java/org/apache/shardingsphere/proxy/backend/text/distsql/rdl/resource/AlterResourceBackendHandlerTest.java#L132
Cognac and GTNM recommend getAlterResource() and
createSimpleResourceStatement(), respectively, which are quite similar to the
actual StableName, createAlterResourceStatement():

⬇ 1{ 2 return new AlterResourceStatement(Collections.singleton 3 (new
DataSourceSegment(resourceName, 4 "jdbc:mysql://127.0.0.1:3306/ds_0", 5 null,
null, null, "root", "", new Properties()))); 6}

As the recommended names and StableName share many common tokens, it is
appropriate to classify the above case as consistent, but the techniques fail
to correctly classify it because their threshold values are too tight.

The imbalanced dataset, reflecting the real practice, may reveal the
shortcomings such as too strong threshold values for inconsistent name
classification.

##  5\. Discussion

We mainly discuss the experimental settings for evaluating the target
techniques as well as the differences between the datasets. The following
discussion could also deliver actionable insights for future research
directions.

###  5.1. Threshold T used in the MCC task

When performing the MCC task, most of the techniques compute the similarity
between the ground truth name and the predicted name. This similarity
calculation bears a threshold value $T$, which is heuristically decided in the
previous studies (Wang et al., 2021; Nguyen et al., 2020; Li et al., 2021c).
This threshold value then determines the performance. However, the choice of
$T$ depends on the technique. For example, MNire (Nguyen et al., 2020) takes
various values for $T$ while Cognac (Wang et al., 2021) uses 0.85 as a fixed
value. Moreover, the threshold is chosen to maximize the performance of each
technique.

Although the impact of the threshold $T$ is investigated in a literature
(Nguyen et al., 2020), it was only to maximize the performance of each
technique. This indicates that it needs to be tuned depending on the approach
and the choice of $T$ potentially causes bias in different scenarios.
Furthermore, one of the prior studies (Wang et al., 2021) stated, “we never
know a method name is consistent or not before the detection in practice”
regarding the decision on the threshold. This may imply that their test
oracles cannot guarantee the quality for evaluating MCC/MNR tasks. We
investigate the performance of the recommendation-based techniques with $T$ as
1 as we hypothesize FixedName in ReName4J may be the ground truth.

Table 5. The results of Method Name Consistency Checking with the fixed value
for threshold $T$ as 1.

|  |  Cognac (Wang et al., 2021)
---
|  GTNM (Liu et al., 2022)
---
| $\mbox{Precision}_{IC}$ |  47.98 / 48.27 |  43.49 / 43.56
IC | $\mbox{Recall}_{IC}$ |  80.00 / 80.00 |  66.75 / 66.00
| $\mbox{F1-score}_{IC}$ |  59.98 / 60.21 |  52.66 / 52.49
| $\mbox{Precision}_{C}$ |  39.85 / 41.61 |  28.49 / 29.90
C | $\mbox{Recall}_{C}$ |  13.25 / 14.25 |  13.25 / 14.50
| $\mbox{F1-score}_{C}$ |  19.89 / 21.23 |  18.09 / 19.53
Accuracy |  46.62 / 47.12 |  40.00 / 40.25

  * The gray values (nn.nn) denote the previous results from Section 4.2 as we aim to discover the effects of the threshold value $T$.

The results in Table 5 indicate that the target techniques are slightly
influenced by the threshold value. Cognac’s recommendations for the $IC$ class
do not change at all. However, there are more false positives for the $C$
class, reducing the precision. More importantly, both subjects experience a
decrease in performance for the $C$ class. Intriguingly, there is a slight
performance increase in GTNM’s recommendation on $\mbox{Recall}_{IC}$ class
(i.e., 66.00% to 66.75%). The increase in true positives has resulted in this
phenomenon, which is unexpected as it is commonly believed that the higher the
threshold value, the more difficulty the techniques will encounter. Inspired
by this, we motivate the following discussion.

###  5.2. Measurement of True Positives for the Inconsistent Class

During the MCC task of recommendation-based techniques, i.e., Cognac and GTNM,
we discovered that the initial evaluation design may be biased due to the
following reasons. A binary classification is not suitable for these
approaches, as they initially recommend and then check the inconsistency based
on the predictions. In this way, a recommended method name cannot ensure if a
method name is inconsistent or consistent. Previous studies, such as Cognac
(Wang et al., 2021), define a true positive in their evaluation as a case
where the similarity between the predicted tokens and the oracle tokens from
the inconsistent name is less than the threshold value $T$ (i.e., $Sim(o,p)$ ¡
$T$). This means that the lower the model’s recommendation performance, the
more true positives it has for the inconsistent class. However, this approach
may not accurately reflect the goal of improving method name consistency, as
the predicted names should ultimately match the ground truth names. To address
this, we revised the evaluation setting and only consider a predicted name as
a true positive if its similarity to the oracle tokens from the consistent
name is equal to 1, as this represents a perfect match with the ground truth.

Table 6. The results of Method Name Consistency Checking with corrected
measurement of the true positives for IC class.

|  |  Cognac (Wang et al., 2021)
---
|  GTNM (Liu et al., 2022)
---
| $\mbox{Precision}_{IC}$ |  6.79 / 48.27 |  15.56 / 43.56
IC | $\mbox{Recall}_{IC}$ |  6.25 / 80.00 |  15.75 / 66.00
| $\mbox{F1-score}_{IC}$ |  6.51 / 60.21 |  15.65 / 52.49

  * The gray values (nn.nn) denote the previous results from Section 4.2 as we aim to check the phenomenon after the correction.

The findings presented in Table 6 indicate a marked decrease in performance
compared to the initial results. Cognac decreased by 86%, 92%, and 89% for
$Precision_{IC}$, $Recall_{IC}$, and F1-scoreIC, respectively. GTNM performed
slightly better, with decreases of 64%, 76%, and 70% on each metric. The
results indicate that recommendation-based approaches demonstrate significant
limitations on inconsistency checking. Overall, there is a need for further
improvement in such approaches as the results in Table 6 do not seem to be
promising. Additionally, such an evaluation protocol requires careful
examination.

###  5.3. Size of the Test Dataset

On the method name recommendation. Most recommendation-based studies (Liu et
al., 2022; Li et al., 2021c; Wang et al., 2021; Nguyen et al., 2020) employ
the test dataset from Java-large (Alon et al., 2019b) for the MNR task. This
dataset includes approximately 636K test methods from 61K files which is a
very large number compared to our datasets. Although we understand that the
bigger the dataset is, the more solid and robust the evaluation that can be
conducted, such a dataset has been solely cloned from open-source projects
without any filtering criteria. This suggests that there is no evidence that
such method names are ever changed, let alone whether the changes are reviewed
by others.

On the method name consistency checking. Different from the dataset commonly
used for the MNR task, most prior techniques (Liu et al., 2019; Li et al.,
2021b; Wang et al., 2021; Nguyen et al., 2020; Li et al., 2021c) leverage the
dataset from a checking-based study (Liu et al., 2019), which has been
constructed using specific filtering criteria. The filtering criteria include
methods whose names have been changed in a commit without any alteration to
the body code. Additional conditions, such as disregarding method changes with
typographical errors and those that do not involve changes to the first sub-
token, are also applied as the techniques are sensitive to such cases. These
filtering criteria ensure that the dataset only contains methods whose names
have been explicitly changed by developers. The dataset comprises 2,805
methods from 430 projects for the test phase. However, it cannot guarantee
that the method name changes are associated with the inconsistency of the
method name and its body since there are cases where the developer only
changed the method body first and then fixed the name after (Wen et al., 2020,
2022).

On our benchmark. Our main goal is to conduct an empirical investigation of
the existing tools with different perspectives (i.e., code review). While we
continuously collect and craft more method and review pairs, we acknowledge
that the current number of test oracles in this study is relatively small
compared to the existing ones. As the ultimate goal of the MCC/MNR tasks is to
be as precise and practical as possible, it is vital that the test oracles
closely resemble the ground truth. To achieve this, we propose utilizing a
code review process involving the code reviewers as a means to enhance the
quality of the data. We demonstrate the significant differences in the lengths
of names between the method names from our benchmark and those of existing
datasets. This may imply that the benchmark can be leveraged to evaluate
MCC/MNR techniques with different scenarios as we described in earlier
sections.

In summary, the set of test oracles used for the MNR task is the largest, but
its practical usefulness is questionable. The dataset used in the MCC task
cannot guarantee whether the method name changes are associated with the
inconsistency checking task or coincident code changes as it is collected
without considering the rationale behind them. Meanwhile, our benchmark
dataset is composed of methods that have been reviewed and explicitly pointed
out by code reviewers of open-source projects. As we continually extend the
benchmark, we expect it will allow for a more comprehensive evaluation of
MCC/MNR techniques.

###  5.4. Threats to Validity

Threats to external validity may lie on the targeted three techniques (Liu et
al., 2019; Wang et al., 2021; Liu et al., 2022) that are open-source; thus,
the results may not be representative of other techniques or closed-source
techniques. Furthermore, as we consider approaches targeting Java, our
conclusions are only valid for this language. A threat may lie in the dataset
as well. We utilize a common dataset for training all the recommendation-based
techniques, and such a dataset can possibly be biased, which could lead the
models in the wrong direction. To mitigate this threat, we incorporate a
large-scale dataset (Alon et al., 2019a) that consists of high-quality and
well-maintained open-source projects. The identifier names in these large-
scale projects are known to be mostly consistent (Alon et al., 2019a, b).
Additionally, such approaches focus on learning common features from the
majority of the methods. As a result, during the training phase, a natural
mitigation of this concern could be attained.

Threats to internal validity may include human-reviewed method names in our
benchmark. To address this threat, the authors cross-checked the method names
and bodies. We found that there exist very long method names with huge bodies
which could be ambiguous to decide whether the names are consistent or not.
All the authors who cross-checked the method names agreed on the final version
of the benchmark. In addition, although it is not guaranteed that all positive
items have been identified, appropriate rationales for the name changes could
support the quality of the naming dataset as well as recommendation tasks.

Threats to construct validity relate to the evaluation metrics we employed. We
took the same metrics as target techniques (Liu et al., 2019; Wang et al.,
2021; Liu et al., 2022), which are Accuracy, Precision, Recall, F1-score, over
sub-tokens, and EMAcc. Although there may exist other potential metrics to
measure the performance of MCC/MNR techniques, evaluating such techniques with
the prior metrics over sub-tokens is the approach taken by all the related
literature.

##  6\. Related Work

Empirical Investigation on Identifier Naming. Based on the assumption, poor
names make programs harder to understand and maintain, many researchers
(Takang et al., 1996; Liblit et al., 2006; Butler et al., 2009, 2011b) have
conducted empirical investigations on naming source code identifiers. The
majority of these studies explore the impact of names on source code
readability (Scalabrino et al., 2018; Fakhoury et al., 2018), program
comprehension (Hofmeister et al., 2017; Scalabrino et al., 2019; Wainakh et
al., 2021), and software maintainability (Butler et al., 2010). In addition,
several studies (Higo and Kusumoto, 2012; Kim and Kim, 2016) investigate the
unintended inconsistency of the identifiers. Recently, researchers (Jiang et
al., 2019; Alsuhaibani et al., 2021) have begun to focus on naming methods, as
these are the smallest unit of an aggregated functionality. Such studies
investigate existing method name recommendation techniques or survey
professional developers to understand the impact of naming choices.

MCC/MNR techniques, on the applied models. Approaches were built based on
textual similarity and co-occurrence. Høst and Østvold (Høst and Østvold,
2009) introduced a debugging approach for method names by inferring naming
rules based on the return type, control flow, and method parameters. Lucia et
al. (De Lucia et al., 2010) proposed an information retrieval based approach
using LSI (Deerwester et al., 1990) to compute the textual similarity to
improve program comprehension.

Recently, researchers have proposed learning-based techniques. Allamanis et
al. (Allamanis et al., 2015, 2016) employ deep learning techniques (i.e.,
basic and copy convolutional attention models) to recommend method names. Liu
et al. (Liu et al., 2019) take the same embedding strategy with Paragraph
Vector (Le and Mikolov, 2014) and convolutional neural networks (CNN) (LeCun
et al., 1999) considering additional code nodes at the abstract syntax tree
(AST) level to capture code semantic information (Mou et al., 2016). Code2seq
(Alon et al., 2019a) and Code2vec (Alon et al., 2019b) are well-known code
embedding techniques that are evaluated with the method name recommendation
task. Code2seq (Alon et al., 2019a) follows the standard encoder-decoder
architecture (Cho et al., 2014) with a bi-directional LSTM (Huang et al.,
2015) while Code2vec (Alon et al., 2019b) leverages a newly designed path-
attention network. Wang et al. (Wang et al., 2021) further added the pointer-
generator network (See et al., 2017) and introduced Cognac aiming to avoid the
out-of-vocabulary problem. With a similar concept, Nguyen et al. (Nguyen et
al., 2020) and Li et al. (Li et al., 2021c) proposed MNire and DeepName,
respectively, considering different contexts. These models leverage the RNN-
based Seq2seq (Britz et al., 2017) model with attention mechanism (Bahdanau et
al., 2014; Luong et al., 2014) aiming to capture contextual sentences. Liu et
al. (Liu et al., 2022)’s GTNM is a transformer-based model that leverages the
self-attention mechanism capturing rich semantic dependencies.

MCC/MNR techniques, on the leveraged contexts. To check and recommend the
method names, various related contexts are considered. Most approaches exploit
the features from the local context, such as the method implementation (e.g.,
tokens of the return type, parameters, control flow graph, data flow graph,
AST, and each identifier in the body) (Suzuki et al., 2014; Allamanis et al.,
2015, 2016; Alon et al., 2019a, b; Liu et al., 2019). Recently, researchers
started to explore a wider range for the context, such as the enclosing class
name (Nguyen et al., 2020) and method invocation relations (i.e., caller and
callee) (Wang et al., 2021; Li et al., 2021c). Liu et al. (Liu et al., 2022)
further extended the context to project-specific information (i.e., in-file
methods and cross-file contextual methods) and documentation of the method
(i.e., Javadoc) (Liu et al., 2022). A few studies leverage high-level
artifacts such as software requirement documents (De Lucia et al., 2010) and
semantic profiles of the method implementations (Høst and Østvold, 2009).

##  7\. Conclusion

We investigated existing method name consistency checking and recommendation
approaches with a novel benchmark that contains clear rationales for changing
method names. Investigating the recent empirical studies on developer
activities for naming identifiers, we discovered that developers could
finalize the method names with code review comments. Inspired by this
phenomenon, we construct a benchmark for method name changes by collecting and
mapping with the related review comments, which can be a further layer of
checking the consistency. We hold the perspective that our benchmark could
serve as a reference for method name consistency checking and recommendation
techniques. This is because the enclosed test oracles are derived from
recommendations or insights provided by code reviewers within the projects. To
validate existing studies on the benchmark, we explored whether they show
consistent performance compared to the previous dataset that does not rely on
clear rationales. Although there was an exceptional case due to the
scalability issue with an existing technique, Spot, we carefully selected our
target techniques and design the experimental setting for a fair comparison.
The results of the existing approaches demonstrate a consistent decrease in
our test oracles for most of the metrics they defined. Overall, this indicates
that there still exists room for improvement, especially when they encounter
different scenarios and evaluation protocols. Moreover, we revealed that there
exist potentially biased features in the evaluation of existing techniques
which research should consider to fill the gap between research and practice.
We hope that these actionable insights could be applied to future research
directions.

##  8\. Data Availability

We publicly release a replication package that includes all the code and
datasets to reproduce the experiments of our study at
https://figshare.com/s/8cdb4e3208e01991e45c

## References

  * (1)
  * Aha (2013) David W Aha. 2013\.  _Lazy learning_.  Springer Science & Business Media.
  * Allamanis et al. (2015) Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015.  Suggesting accurate method and class names. In _Proceedings of the 2015 10th joint meeting on foundations of software engineering_. 38–49.
  * Allamanis et al. (2016) Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016\.  A convolutional attention network for extreme summarization of source code. In _International conference on machine learning_. PMLR, 2091–2100.
  * Alon et al. (2019a) Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019a.  code2seq: Generating Sequences from Structured Representations of Code. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=H1gKYo09tX
  * Alon et al. (2019b) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019b.  code2vec: Learning distributed representations of code.  _Proceedings of the ACM on Programming Languages_ 3, POPL (2019), 1–29.
  * Alsuhaibani et al. (2021) Reem Alsuhaibani, Christian Newman, Michael Decker, Michael Collard, and Jonathan Maletic. 2021\.  2021 [ICSE]_On the Naming of Methods: A Survey of Professional Developers. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, Madrid, ES, 587–599.  https://doi.org/10.1109/ICSE43902.2021.00061
  * Arnaoudova et al. (2016) Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2016.  Linguistic antipatterns: What they are and how developers perceive them.  _Empirical Software Engineering_ 21, 1 (2016), 104–158.
  * Arnaoudova et al. (2014) Venera Arnaoudova, Laleh M Eshkevari, Massimiliano Di Penta, Rocco Oliveto, Giuliano Antoniol, and Yann-Gaël Guéhéneuc. 2014\.  Repent: Analyzing the nature of identifier renamings.  _IEEE Transactions on Software Engineering_ 40, 5 (2014), 502–532.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Britz et al. (2017) Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017\.  Massive exploration of neural machine translation architectures.  _arXiv preprint arXiv:1703.03906_ (2017).
  * Butler et al. (2009) Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2009.  Relating identifier naming flaws and code quality: An empirical study. In _2009 16th Working Conference on Reverse Engineering_. IEEE, 31–35.
  * Butler et al. (2010) Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2010.  Exploring the influence of identifier names on code quality: An empirical study. In _2010 14th European Conference on Software Maintenance and Reengineering_. IEEE, 156–165.
  * Butler et al. (2011a) Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2011a.  Improving the tokenisation of identifier names. In _European Conference on Object-Oriented Programming_. Springer, 130–154.
  * Butler et al. (2011b) Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2011b.  Mining java class naming conventions. In _2011 27th IEEE International Conference on Software Maintenance (ICSM)_. IEEE, 93–102.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ (2014).
  * De Lucia et al. (2010) Andrea De Lucia, Massimiliano Di Penta, and Rocco Oliveto. 2010.  Improving source code lexicon via traceability and information retrieval.  _IEEE Transactions on Software Engineering_ 37, 2 (2010), 205–227.
  * Deerwester et al. (1990) Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990\.  Indexing by latent semantic analysis.  _Journal of the American society for information science_ 41, 6 (1990), 391–407.
  * Fakhoury et al. (2018) Sarah Fakhoury, Yuzhan Ma, Venera Arnaoudova, and Olusola Adesope. 2018.  The effect of poor source code lexicon and readability on developers’ cognitive load. In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. IEEE, 286–28610.
  * GitHub (2022) GitHub 2022.  https://github.com/.
  * Gu et al. (2018) Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018\.  Deep code search. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 933–944.
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009\.  Unsupervised learning.  In _The elements of statistical learning_. Springer, 485–585.
  * Higo and Kusumoto (2012) Yoshiki Higo and Shinji Kusumoto. 2012.  How often do unintended inconsistencies happen? Deriving modification patterns and detecting overlooked code fragments. In _2012 28th IEEE International Conference on Software Maintenance (ICSM)_. IEEE, 222–231.
  * Hofmeister et al. (2017) Johannes Hofmeister, Janet Siegmund, and Daniel V Holt. 2017\.  Shorter identifier names take longer to comprehend. In _2017 IEEE 24th International conference on software analysis, evolution and reengineering (SANER)_. IEEE, 217–227.
  * Høst and Østvold (2009) Einar W Høst and Bjarte M Østvold. 2009.  Debugging method names. In _European Conference on Object-Oriented Programming_. Springer, 294–317.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _arXiv preprint arXiv:1508.01991_ (2015).
  * Høst and Østvold (2009) Einar W. Høst and Bjarte M. Østvold. 2009.  Debugging Method Names. In _ECOOP 2009 – Object-Oriented Programming_ , Sophia Drossopoulou (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 294–317.
  * Jiang et al. (2019) Lin Jiang, Hui Liu, and He Jiang. 2019.  Machine learning based recommendation of method names: how far are we. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 602–614.
  * Karlton (2022) Karlton 2022.  https://www.nndb.com/people/400/000031307/.
  * Kim and Kim (2016) Suntae Kim and Dongsun Kim. 2016.  Automatic identifier inconsistency detection using code dictionary.  _Empirical Software Engineering_ 21, 2 (2016), 565–604.
  * Lawrie et al. (2006) Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006.  What’s in a Name? A Study of Identifiers. In _14th IEEE international conference on program comprehension (ICPC’06)_. IEEE, 3–12.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. PMLR, 1188–1196.
  * LeCun et al. (1999) Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. 1999.  Object recognition with gradient-based learning.  In _Shape, contour and grouping in computer vision_. Springer, 319–345.
  * Li et al. (2021a) Kejun Li, Taiming Wang, and Hui Liu. 2021a.  NameChecker: Detecting Inconsistency between Method Names and Method Bodies. In _2021 28th Asia-Pacific Software Engineering Conference (APSEC)_. 22–31.  https://doi.org/10.1109/APSEC53868.2021.00010
  * Li et al. (2021b) Kejun Li, Taiming Wang, and Hui Liu. 2021b.  NameChecker: Detecting Inconsistency between Method Names and Method Bodies. In _2021 28th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 22–31.
  * Li et al. (2021c) Yi Li, Shaohua Wang, and Tien Nguyen. 2021c.  A Context-based Automated Approach for Method Name Consistency Checking and Suggestion. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 574–586.
  * Liblit et al. (2006) Ben Liblit, Andrew Begel, and Eve Sweetser. 2006.  Cognitive Perspectives on the Role of Naming in Computer Programs.. In _PPIG_. Citeseer, 11\.
  * Liu et al. (2022) F. Liu, G. Li, Z. Fu, S. Lu, Y. Hao, and Z. Jin. 2022.  Learning to Recommend Method Names with Global Context. In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_. IEEE Computer Society, Los Alamitos, CA, USA, 1294–1306.  https://doi.org/10.1145/3510003.3510154
  * Liu et al. (2019) Kui Liu, Dongsun Kim, Tegawendé F Bissyandé, Taeyoung Kim, Kisub Kim, Anil Koyuncu, Suntae Kim, and Yves Le Traon. 2019\.  Learning to spot and refactor inconsistent method names. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 1–12.
  * Luong et al. (2014) Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014.  Addressing the rare word problem in neural machine translation.  _arXiv preprint arXiv:1410.8206_ (2014).
  * Maletic and Marcus (2001) Jonathan I Maletic and Andrian Marcus. 2001.  Supporting program comprehension using semantic and structural information. In _Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001_. IEEE, 103–112.
  * Mou et al. (2016) Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016.  Convolutional neural networks over tree structures for programming language processing. In _Thirtieth AAAI conference on artificial intelligence_.
  * Nguyen et al. (2020) Son Nguyen, Hung Phan, Trinh Le, and Tien N Nguyen. 2020\.  Suggesting natural method names to check name consistencies. In _Proceedings of the ACM/IEEE 42nd international conference on software engineering_. 1372–1384.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Scalabrino et al. (2019) Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vasquez, Denys Poshyvanyk, and Rocco Oliveto. 2019.  Automatically assessing code understandability.  _IEEE Transactions on Software Engineering_ 47, 3 (2019), 595–613.
  * Scalabrino et al. (2018) Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. 2018.  A comprehensive model for code readability.  _Journal of Software: Evolution and Process_ 30, 6 (2018), e1958.
  * Schankin et al. (2018) Andrea Schankin, Annika Berger, Daniel V Holt, Johannes C Hofmeister, Till Riedel, and Michael Beigl. 2018.  Descriptive compound identifier names improve source code comprehension. In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. IEEE, 31–3109.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017\.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_ (2017).
  * Suzuki et al. (2014) Takayuki Suzuki, Kazunori Sakamoto, Fuyuki Ishikawa, and Shinichi Honiden. 2014.  An approach for evaluating and suggesting method names using n-gram models. In _Proceedings of the 22nd International Conference on Program Comprehension_. 271–274.
  * Takang et al. (1996) Armstrong A Takang, Penny A Grubb, and Robert D Macredie. 1996\.  The effects of comments and identifier names on program comprehensibility: an experimental investigation.  _J. Prog. Lang._ 4, 3 (1996), 143–167.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wainakh et al. (2021) Yaza Wainakh, Moiz Rauf, and Michael Pradel. 2021.  Idbench: Evaluating semantic representations of identifier names in source code. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 562–573.
  * Wang et al. (2021) Shangwen Wang, Ming Wen, Bo Lin, and Xiaoguang Mao. 2021\.  Lightweight global and local contexts guided method name recommendation with prior knowledge. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 741–753.
  * Wen et al. (2020) Fengcai Wen, Csaba Nagy, Michele Lanza, and Gabriele Bavota. 2020\.  An empirical study of quick remedy commits. In _Proceedings of the 28th International Conference on Program Comprehension_. 60–71.
  * Wen et al. (2022) Fengcai Wen, Csaba Nagy, Michele Lanza, and Gabriele Bavota. 2022\.  Quick remedy commits and their impact on mining software repositories.  _Empirical Software Engineering_ 27 (2022), 1–31.