# Code Reviewer Recommendation Based on a Hypergraph with Multiplex
Relationships

Yu Qiao1, Jian Wang1∗, Can Cheng2∗, Wei Tang1, Peng Liang1, Yuqi Zhao1, Bing
Li1∗ 1 School of Computer Science, Wuhan University, Wuhan, China  2 School of
Artificial Intelligence, Hubei University, Wuhan, China  {qiaoyu_cs, jianwang,
weitang_cs, liangp, yuqizhao<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Code review is an essential component of software development, playing a vital
role in ensuring a comprehensive check of code changes. However, the
continuous influx of pull requests and the limited pool of available reviewer
candidates pose a significant challenge to the review process, making the task
of assigning suitable reviewers to each review request increasingly difficult.
To tackle this issue, we present MIRRec, a novel code reviewer recommendation
method that leverages a hypergraph with multiplex relationships. MIRRec
encodes high-order correlations that go beyond traditional pairwise
connections using degree-free hyperedges among pull requests and developers.
This way, it can capture high-order implicit connectivity and identify
potential reviewers. To validate the effectiveness of MIRRec, we conducted
experiments using a dataset comprising 48,374 pull requests from ten popular
open-source software projects hosted on GitHub. The experiment results
demonstrate that MIRRec, especially without PR-Review Commenters relationship,
outperforms existing state-of-the-art code reviewer recommendation methods in
terms of ACC and MRR, highlighting its significance in improving the code
review process.

###### Index Terms:

open source software, code reviewer recommendation, hypergraph, multiplex
relationships.

##  I Introduction

Modern Code Review (MCR) [1] refers to examining source code systematically to
identify and correct overlooked mistakes. This practice is indispensable in
the pursuit of enhancing software quality. MCR is typically carried out before
code integration and release, varying in granularity from individual lines of
code to entire software modules. During this process, one or more reviewers
assess the code for errors, adherence to coding standards, test coverage, and
more. The proliferation of the pull-request (PR) mechanism has reshaped the
landscape of massive open-source projects, establishing PR as a critical tool
for the implementation of MCR. The shift has made MCR more collaborative and
continuous [2]. Developers now have the ability to create a PR and request
reviewers who possess the expertise and relevant past review experience to
inspect the uploaded code changes. This PR-centric review mechanism helps
improve code quality, ensure code standards, promote team collaboration, and
accelerate the development process.

Despite its advantages, the PR-centric review mechanism can also be
challenging. It requires developers to dedicate time and effort that could be
otherwise spent writing new code. In some cases, reviewers invited to review
the code may decline due to factors like unfamiliarity with the content or
lack of time. This presents a human-related issue in matching PRs to
appropriate reviewers, which can be labor-intensive and time-consuming[3].
Consequently, the task of identifying suitable reviewers, while recognized as
crucial within the OSS community, remains a significant challenge.

Limited reviewer capacity. Reviews on PRs enable collaborators to comment on,
approve, or request further modifications to the proposed changes before
merging. However, this workflow, while effective, increases the workload for
reviewers. The rapid growth of PRs, especially in large and popular projects,
coupled with the need for collaboration among several reviewers for PRs that
affect multiple modules, poses a significant challenge for timely responses.
For instance, Tencent, one of the largest Chinese Internet companies, receives
over $100$K code changes per month in proprietary projects [4]. Moreover,
reviewer teams often find themselves significantly outnumbered by the sheer
volume of incoming PRs. A case in point is React, a popular open-source
project on GitHub, which has received over $12$K PRs since its inception in
March $2023$, with fewer than $1$K developers participating in the review
process. Yang et al. [2] also reported that within three years, $437$
reviewers contributed more than 66K reviews to the LibreOffice project. These
examples highlight the challenges faced by reviewer teams in managing and
responding to PRs efficiently.

High review latency. Previous studies[3][5][6][7] have shown that effective
reviews require knowledgeable or familiar reviewers for the submitted changes.
Inappropriate reviewers can hinder the review process, delay PR merges, and
slow down development. Rigby and Bird[8] observed that 50% of reviews take
nearly 30 days. Similarly, Tsay et al.[9] found that a number of code changes
remain pending for up to two months before being merged. Additionally,
reviewers occasionally decline review requests[10] , further exacerbating
review delays. Another study indicated that 16%-66% of the patches have at
least one invited reviewer who did not respond to the review invitation[3].
The decision of whether a reviewer accepts a PR review invitation depends on
various factors, including their available time, existing workload, review
experience, code authoring experience, and more [3]. Hence, recommending
appropriate reviewers for incoming PRs is considered a practical and
collaborative way to expedite meaningful merges and facilitate the OSS
evolution.

In response to these issues, researchers have conducted numerous studies.
Appropriate reviewers who are more familiar with the code changes may spend
less time reviewing them[11][12][13][14]. However, Thongtanunam et al. [12]
found that about 4%-30% of reviews encounter code-reviewer assignment
problems, and larger projects face greater difficulties in finding suitable
code reviewers. Existing reviewer recommendation approaches focused on
modeling the developer’s expertise or interactions with tasks based on
historical information [4] [15] [16] [17]. However, these methods heavily rely
on historical data to model a developer’s expertise and task interactions. If
the historical data is insufficient or inaccurate, it may negatively impact
the quality of recommendations. Different developers may also possess distinct
expertise and patterns of task interactions. If a model cannot effectively
capture these individual differences, it may affect the accuracy of the
recommendations.

Furthermore, recommendation models based on relation graphs or networks [18]
[19][20][21][22] proposed to alleviate the sparse explicit interactions by
considering various explicit and implicit relationships. Specifically, Yu et
al. [18] constructed a social relation network among PR, creators, and
reviewers, while Rong et al. [22] developed a hypergraph reviewer recommender
based on three types of relations. Although these studies have made important
contributions to reviewer recommendation, there is one notable limitation that
much of the current research primarily treats interactions equally, without
considering the varying roles that developers play. Actually, developers may
transition between these roles across different PRs. For instance, developer
$d_{1}$ might act as a reviewer for $PR_{3}$, while being a committer for
$PR_{2}$, as depicted in Figure 1. These multifaceted roles and behaviors of
developers, which can represent their abilities in various development tasks,
are often overlooked in existing code reviewer recommendation methods. Another
limitation is that these existing recommendation approaches based on relation
networks often combine traditional methods, such as Collaborative Filtering
and Graph Neural Networks, to extract features from a bipartite graph.
However, given the inherent challenges in comprehending developer expertise,
relevance, and the sparsity of explicit developer-task interactions, these
approaches are struggling to comprehensively capture these relations and thus
encounter difficulties in achieving high performance.

Figure 1: Multiplex interactions of developers in PR-based review process

To this end, this work aims to automate the recommendation of potentially
qualified reviewers for PRs. To address the issues aforementioned, we apply
the hypergraph to model the multivariate interactions among developers and
PRs, enabling us to recommend appropriate reviewers for PRs. To evaluate the
effectiveness of the proposed approach, we conducted several experiments on
ten OSS projects and compared it with the state-of-the-art methods. The main
contributions of this paper include:

$(1)$ We propose MIRRec, a hypergraph method accomplished by considering the
PR similarity and the Multiplex Interaction Relationships among developers and
PRs within the various roles that developers play, to Recommend potential
reviewers for PRs.

$(2)$ We validate the effectiveness of MIRRec on ten open-source projects, and
the results indicate that MIRRec outperforms the state-of-the-art approaches.

$(3)$ Investigating the influence of interactions on MIRRec reveals that the
type and overlap of interaction data can influence reviewer recommendation
performance, providing valuable insight for the development of further models.

The rest of the paper is organized as follows. Section II delves into the
details of our method. Section III outlines the experiment design of
evaluation. Section IV scrutinizes the outcomes of our evaluation. Section V
highlights the potential threats to validity. In Section VI, we discuss prior
research and key distinctions in our approach. Finally, Section VII reports
conclusions and puts forward future research.

##  II Approach

###  II-A Approach overview

As depicted in Figure 2, our proposed MIRRec comprises two key steps:
multiplex-relationship hypergraph construction and ranking-based reviewer
recommendation. We begin by identifying PRs and developers as graph nodes,
creating hyperedges among them and assigning different weights based on their
interaction history and task similarity to establish the multiplex-
relationship hypergraph. Next, we update the existing hypergraph with incoming
PR data and employ a hypergraph-based learning strategy to calculate ranking
scores efficiently for ranking and recommending suitable reviewers for the new
PR.

Figure 2: Overview of approach

###  II-B Multiplex-relationship Hypergraph Construction

Developers mainly participate in the pull and review processes of PRs through
create, commit, comment, and review, while playing different roles as creator,
committer, commenter, and reviewer [23]. Consequently, we build the following
six hyperedges, inspired by these behaviors associated with the different
roles developers assume, as well as task (PRs) similarity, and assign the
weights respectively.

PR-Creator hyperedge: We establish a hyperedge between each PR and its
creator. Each PR is created by a single developer, who is also one of the
committers for the PR and is familiar with the files and code modifications
involved in the PR. The creation behavior implies the developer’s preference
and the scope of development tasks in the current time point more accurately.
Therefore, we determine the weight of each PR based on the time elapsed since
its creation relative to the start of the dataset. More recent creations
receive higher weights. The weight is calculated as:

$\displaystyle\leavevmode\resizebox{140.92792pt}{}{
$W(p_{i},u_{j})=\frac{t_{i}-t_{s}}{t_{e}-t_{s}}$},$ (1)

where $t_{s}$ and $t_{e}$ represent the start time and end time of the
dataset, while $t_{i}$ is the creation time of PR $p_{i}$ by creator $u_{j}$.

PR-Committers hyperedge: A PR typically includes several commits, which may be
made by different developers. These committers are very likely to participate
in the review of the relevant file and codes of the PR as they are more
familiar with this PR. Therefore, we establish a hyperedge connecting the PR
with its associated committers. We calculate the weight of the hyperedge
considering the modified code lines, the commit creation time, and the number
of commits[24], as formulated in the following equation:

$\displaystyle\leavevmode\resizebox{422.77661pt}{}{
$W(p_{i},u)_{ct}=\sum_{m=1}^{N_{(p_{i},u)}^{ct}}{\sum_{n=1}^{N_{(p_{i},u_{m})}^{ct}}{\alpha^{n-1}\times\frac{\left|t_{mn}^{ct}-t_{s}\right|}{t_{e}-t_{s}}}}\times\frac{1}{1+e^{0.01\times
l_{code}}}$},$ (2)

where $N_{(p_{i},u)}^{ct}$ is the total count of committers for $p_{i}$.
$N_{(p_{i},u_{m})}^{ct}$ is the commit count by committer $u_{m}$ to $p_{i}$.
$t_{mn}^{ct}$and $l_{code}$ are the $n$-th commit time and the modification
code lines of $u_{m}$. $\alpha\in[0,1]$ is an experience attenuation
coefficient used to adjust the influence of committers with multiple commits
on the hyperedge.

PR-Reviewers hyperedge: PRs often undergo several rounds of revisions and code
reviews before being merged. This is likely accomplished through the
cooperation of different reviewers. These review interactions are essential
for reviewer recommendation tasks. Here, we build a PR-Reviewers hyperedge
between the PR and its associated reviewers. The weight of a PR-Reviewer edge
is set as follows:

$\displaystyle\leavevmode\resizebox{422.77661pt}{}{
$W(p_{i},u)_{r}=\sum_{m=1}^{N_{(p_{i},u)}^{r}}{\sum_{n=1}^{N_{(p_{i},u_{m})}^{r}}\alpha^{n-1}}\times
e^{\frac{t_{mn}^{r}-t_{s}}{t_{e}-t_{s}}-1}$},$ (3)

where $N_{(p_{i},u)}^{r}$ is the reviewers number of $p_{i}$.
$N_{(p_{i},u_{m})}^{r}$ denotes the review times conducted by $u_{m}$ for
$p_{i}$. $t_{mn}^{r}$ is the start time of $n$-th review by $u_{m}$ on
$p_{i}$.

PR-Issue Commenters hyperedge and PR-Review Commenters hyperedge: PR is a kind
of issue with code or file modifications. Developers use comments to suggest
improvements, ask questions, or provide feedback on these changes in the pull
and review processes. These comments help improve code quality and facilitate
collaboration and communication among project members. Additionally, some of
these commenters may transition into code reviewers. There are two types of
comments on PRs: comments directly on the PR, which can also be considered as
issue comments, and comments specifically on code or file modifications within
the PR review, which serve as review comments. Therefore, we establish these
two hyperedges through connections between the PR and its respective
commenters. We use the creation time and the number of comments as the basis
for measuring the interest of developers, and define the formulas for
calculating the hyperedge weights of PR-Issue Commenters and PR-Review
Commenters. as follows:

$\displaystyle\leavevmode\resizebox{422.77661pt}{}{
$W(p_{i},u)_{ic}=\sum_{m=1}^{N_{(p_{i},u)}^{ic}}{\sum_{n=1}^{N_{(p_{i},u_{m})}^{ic}}\alpha^{n-1}}\times
e^{\frac{t_{mn}^{ic}-t_{s}}{t_{e}-t_{s}}-1}$},$ (4)

where $W(p_{i},u)_{ic}$ represents the weight of PR-Issue Commenters
hyperedge. $N_{(p_{i},u)}^{ic}$ is the count of commenters of $p_{i}$.
$N_{(p_{i},u_{m})}^{ic}$ is the number of comments by $u_{m}$ on $p_{i}$.
$t_{mn}$ is the comment time of $n$-th comment by $u_{m}$ on $p_{i}$.
Similarly, the weight of PR-Issue Commenters hyperedge $W(p_{i},u)_{rc}$ is
calculated in the same way.

It is important to note that our hypotheses for (2), (3), and (4) are as
follows: $(1)$ Interactions (commits, reviews, and comments) that developers
engage in across multiple PRs hold greater significance compared to
interactions within a single PR. $(2)$ Recent interactions carry more weight
than older ones. To distinguish the variations in interactions between
multiple PRs and those within a single PR, we introduce an empirical decay
factor $\alpha$ to control their influence. Additionally, we incorporate the
time-sensitive factor (the partition similar as (1)) to prioritize recent
interactions[18][22].

PR-PR hyperedge: Developers are likely to review PRs that are similar to
previously reviewed PRs, as they possess the necessary expertise. Here, we
create PR-PR hyperedges connecting similar PRs, with the weight determined by
the similarity of their modified file path sets[12], which is:

$\displaystyle W(p_{i},p_{j})=$ (5) $\displaystyle\begin{aligned}
\begin{cases}0&F_{i}=0|F_{j}=0\\\ \sum\limits_{f_{m}\in
F_{i}}\sum\limits_{f_{n}\in
F_{j}}\frac{\text{SimFilePath}(f_{m},f_{n})}{|F_{i}||F_{j}|}\times
e^{-\frac{|t_{i}-t_{j}|}{t_{e}-t_{s}}}&F_{i}\geq 0|F_{j}\geq
0,\end{cases}\end{aligned}$

$\displaystyle\small\text{SimFilePath}(f_{m},f_{n})=\frac{\text{LCP}(f_{m},f_{n})}{\max(\text{len}(f_{m}),\text{len}(f_{n}))},$
(6)

where $F_{i}$ and $F_{j}$ represent the modified file path set of $p_{i}$ and
$p_{j}$ respectively. $t_{i}$ and $t_{j}$ are the creation time of $p_{i}$ and
$p_{j}$. $\text{SimFilePath}(f_{m},f_{n})$ is the distance between file
$f_{m}$ and $f_{n}$. $\text{LCP}(f_{m},f_{n})$ denotes the longest common
prefix function.

As the similarity between PRs is affected by the modified file path set and
creation time, the similarity between most PRs in the dataset is low. Thus, we
select Top-K similar PRs for each PR to build a PR-PR hyperedge (cf. following
for details) and preferentially consider the latest PRs.

###  II-C Ranking-based Reviewer Recommendation

We carry out the review recommendation as a ranking task on the constructed
hypergraph. The key is to find an optimal ranking vector
$f^{*}\in\mathbb{R}^{V}$ to rank the matching scores between reviewers and
PRs. The details are elaborated as follows.

####  II-C1 Hypergraph learning

For the hypergraph we constructed $G=\langle V,E,W\rangle$, which includes a
vertex set $V$, a hyperedge set $E$, and each hyperedge is assigned a weight
by $W$. The key of ranking is to minimize the object function as below:

$\displaystyle\psi(f)=\varphi(f)+R_{\text{emp}}(f),$ (7)

where $\psi(f)$ is a regularization on hypergraph, $R_{\text{emp}}(f)$ denotes
the supervised empirical loss. $\varphi(f)$ is defined as:

$\displaystyle\small\varphi(f)$
$\displaystyle=f^{T}(I-D_{v}^{-1/2}HWD_{e}^{-1}H^{T}D_{v}^{-1/2})f$
$\displaystyle=f^{T}(I-A)f$ $\displaystyle=f^{T}Lf$ (8)

where $H$ represents the incidence matrix between $V$ and $E$, $D_{v}$ and
$D_{e}$ denote the vertex degree matrix and hyperedge degree matrix
respectively, and $A$ is the adjacency matrix of $G$.

To recommend reviewers for a newly incoming PR $p_{i}$, the ranking loss
function is represented as:

$\displaystyle\small R_{\text{emp}}(f)=\lambda||f-y_{p_{i}}||^{2},$ (9)

$\lambda$ is the positive parameter to weigh the supervised empirical loss.
Then the object function $(f)$ can be deducted to:

$\displaystyle\psi(f)=f^{T}Lf+\lambda||f-y_{p_{i}}||^{2}$ (10)

The optimal $f^{*}$ can be deducted and transformed into:

$f^{*}=(I-\mu A)^{-1}y_{p_{i}},$ (11)

where $\mu\in[0,1]$ is the hyperparameter and $\mu=1/(1+\lambda)$.

####  II-C2 Ranking strategy

Having ranked on the hypergraph, we can recommend the Top-K reviewers as the
candidates. We define the ranking score by considering the relevance of
reviewer candidates in different roles excluding creators to the PR:

$\text{Score}_{u_{i}}=af^{*}[r_{i}]+bf^{*}[ct_{i}]+cf^{*}[rc_{i}]+df^{*}[ic_{i}],$
(12)

where $f^{*}[r_{i}]$,$f^{*}[ct_{i}]$,$f^{*}[rc_{i}]$,$f^{*}[ic_{i}]$ represent
the relation scores as reviewers, committers, review commenters and issue
commenters of $u_{i}$ respectively. If a developer has had no interaction with
a PR in a specific role mentioned above, the corresponding relation score is
set to $0$. The variables $a$, $b$, $c$, and $d$ are the weights assigned to
each relation. The Top-K code reviewers are then recommended based on the
ranking scores of the code reviewer candidates.

##  III Evaluation

This study aims to propose a relationship-aware and hypergraph-based approach
for code reviewer recommendation and assess the impact of the relationship-
based hypergraph strategy on recommendation performance. To this end, three
research questions (RQs) are defined to evaluate MIRRec.

RQ1: Which hyperparameter settings can enhance the performance of MIRRec?

Rationale: Identifying optimal hyperparameter settings helps improve the
performance of MIRRec. With this RQ, we want to use efficient methods to
determine these settings within limited computational resources and
investigate the model’s behavior influenced by the hyperparameters.

RQ2: Does MIRRec surpass state-of-the-art reviewer recommendation methods in
terms of performance?

Rationale: Different from other state-of-the-art methods, MIRRec incorporates
multiplex relationships and hypergraph learning. With this RQ, we aim to
evaluate the effectiveness of this strategy.

RQ3: How do different types of interaction relationships affect the
recommendation performance of MIRRec?

Rationale: As found by Ruangwan et al. in [3], different relationships may
have varying impacts on reviewer recommendation. Therefore, this RQ aims to
understand the impact of interaction relationships on the recommendation
performance.

RQ1 aims to identify hyperparameter settings that can enhance MIRRec’s
performance. RQ2 aims to validate the effectiveness of the optimal
hyperparameter settings found in RQ1 by comparing them with baseline methods.
RQ3 further explores how different types of relationships influence MIRRec’s
performance, thus providing additional support for the results in RQ2 using
the optimal settings obtained in RQ1.

###  III-A Data Preparation

We focus on popular and active OSS projects, as they tend to have frequent
code updates and a wealth of PR and developer data. For this purpose, we
selected OSS projects from GitHub for dataset based on the following criteria:
$(1)$ projects created before Oct $1$st, $2017$ with last commit posted after
Aug $1$st, $2022$, indicating a sufficiently long lifespan and continuous
updates; $(2)$ projects with more than $10$K stars and $3$K forks, indicating
high popularity; and $(3)$ projects with over $3$K PRs and 8K review
histories, ensuring sufficient data for analysis.

Considering factors like programming language popularity, star and fork
rankings, and the quantity of PR-review interaction histories, we selected a
final set of ten OSS projects spanning six programming languages. To avoid PRs
with early data imbalance (a small number of PRs in the early stages of
project creation) and recent data instability (some PRs not yet closed), we
used data from June $1$st, $2018$, to Nov $30$th, $2021$, for this study.

###  III-B Data Preprocessing

We performed several preprocessing steps on the source data to construct the
dataset, following these steps:

$\bullet$ Unifying developer identities. Developers have four identity
attributes: ‘ID’, ‘Login’, ‘Name’, and ‘Email’. ‘ID’, ‘Login’, and ‘Email’
serve as unique identifiers for developers. However, the ‘Email’ attribute may
be empty, and the ‘Name’ attribute may be updated or duplicated. Additionally,
different developer roles may have incomplete identity attributes. For
instance, committers only have ‘Name’ and ‘Email’ attributes. These variations
make it challenging to accurately map the same developers who play different
roles and exhibit different interaction behaviors based solely on their
identity attributes. To address this issue, we employed the following steps
for unique identifier mapping:

$(1)$ For ‘Name’ attributes containing ‘and’, ‘&’, ‘&&’, ‘+’, or ‘—’, we
utilized regular expressions to split them into separate data elements. $(2)$
We created a mapping table that connects the ‘Login’, ‘Email’, and ‘Name’
attributes of developers involved in the project. This table enabled us to
perform conversions effectively. In most cases, developers can obtain their
unique ‘Login’ based on the mapping table for a given project. $(3)$ If a
developer cannot be successfully mapped using the previous steps, we
calculated the edit distance [25] for ‘Login’, ‘Email’, and ‘Name’ attributes.
We chose the attribute with a shorter edit distance for mutual conversion.
$(4)$ Despite the above processing steps, there may still be approximately 1%
of developers who cannot be identified through the mapping. For these
developers, we considered them as other independent developers.

$\bullet$ Filtering out invalid interaction data. We identify that some
developers are invalid, and their interaction histories can interfere with the
recommendation task. To address this, we removed the invalid interaction data
related to three types of developers: $(1)$ Robot developers: In large OSS
projects, there are robots performing code testing and reviewing. We removed
data associated with developers whose type is listed as ‘Bot’ in the developer
information. $(2)$ Deregistered developers: Historical review records of
logged-out users can interfere with the recommendation. GitHub’s API marks the
status of logged-out users as empty. Therefore, we deleted any data where the
developer information is empty. $(3)$ Developers who reviewed their own PRs:
In most cases, PR creators were directly involved in code changes. If creators
also reviewed their own PRs, their subjectivity could significantly impact the
quality of code reviews. Hence, we discarded code review interaction data
where the reviewer and the creator are the same.

$\bullet$ Removing commit data with bulk file modifications. When developers
make modifications to source code, they may end up modifying a significant
number of files due to changes in widely used classes or functions, as well as
folder renaming. While many of these modifications may be insignificant, they
can result in multiple interactions with these files by committers,
potentially impacting the effectiveness of the reviewer recommendation. To
mitigate this, we excluded commit interaction histories where the number of
modified files equals or exceeds $100$.

$\bullet$ Discarding data after PRs are merged. After PRs are merged, it is
common for developers to continue leaving comments. However, since the merging
of PRs signifies the end of the review phase, the comments generated after the
merge become less relevant for code reviewer recommendations. As a result, we
discarded any relevant data that occurs after the PR has been merged.

$\bullet$ Removing PRs without file modifications or review history. To
construct the hypergraph, it is crucial to consider the value of the
interaction data. Some PRs have not undergone a review process, lack actual
code changes, or do not involve file submissions. These types of PRs include
non-code changes (such as documentation, comments, or README files), branch
merges (integrating changes between branches without explicit file
modifications), and collaborative discussions (primarily for discussions and
reviews without direct code modifications or file submissions). The
relationships captured in these types of PRs are often distant and less
informative. Therefore, we excluded these PRs and their associated data.

After applying the aforementioned preprocessing steps, we obtained the
dataset, and its details are presented in Table I.

TABLE I: Overview of the dataset Language |  Fork |  Star |  Project |  PR |  Commit |  Issue comment |  Review comment |  Review
---|---|---|---|---|---|---|---|---
C++ |  34.3 k |  69.7 k |  Bitcoin |  4544 |  11053 |  26587 |  22106 |  20088
14.7 k |  108 k |  Electron |  10163 |  23058 |  12053 |  15526 |  21050
55.3 k |  69 k |  Opencv |  3158 |  8452 |  5050 |  9886 |  8205
6.2 k |  15.8 k |  XBMC |  2789 |  5422 |  7378 |  8324 |  7760
JavaScript |  43.4 k |  208 k |  React |  3167 |  8371 |  4661 |  6693 |  6773
23.6 k |  88.3 k |  Angular |  9095 |  17370 |  14737 |  29288 |  25883
Python |  29.2 k |  71 k |  Django |  2861 |  4302 |  5892 |  11258 |  8262
PHP |  9.1 k |  28.4 k |  Symfony |  7420 |  9814 |  16084 |  21689 |  25142
Ruby |  21.1 k |  52.9 k |  Rails |  2812 |  4861 |  5029 |  7708 |  6793
Scala |  3.2 k |  14.1 k |  Scala |  2365 |  4537 |  5398 |  5429 |  5866
total |  48374 |  97240 |  102869 |  137907 |  135822

###  III-C Experiment settings

We notice that different projects can exhibit varying numbers of PRs and
levels of developer activity over time. For instance, the Scala project
received approximately $500$ PRs within a year, whereas the Electron project
received a similar number within just two months. Developers may also go
through periods of extended inactivity within a project. In the case of the
Django project, we identify a total of $1039$ developers, with the majority
showing short-term activity. However, only 71 developers displayed a pattern
of short-term activity followed by a prolonged period of inactivity (one year)
before reengaging with the project. To account for these variations, we
divided the data into training and testing sets on an annual basis. For
example, the training set encompasses the period from June $2018$ to June
$2019$, while the testing set consists of data from July $2019$. This strategy
enabled us to generate $30$ rounds of sliding training and testing data, with
each round representing a monthly interval, and calculate the average
performance as the final evaluation result.

To evaluate MIRRec, we compared this method with several state-of-the-art code
reviewer recommenders, which include:

$(1)$ RevFinder [12], which recommends code reviewers based on the similarity
of the file path sets of PR modifications. This approach aims to recommend
code reviewers who have expertise in handling PRs with similar file paths in
historical review records.

$(2)$ cHRev [15], which scores candidate reviewers based on their specific
contributions like expertise, workload, and frequency of past reviews.

$(3)$ CN [18], which recommends developers with similar interests as reviewers
by constructing a social relation network based on the interaction between PR
creators and reviewers in the review history.

$(4)$ HGRec [22], which recommends reviewers using a hypergraph-based approach
that considers the interactions of contributors and reviewers with PR, as well
as the similarity between PRs.

We select these baselines for the following reasons: RevFinder and cHRev have
frequently served as the basis for comparison in many existing studies. CN
adopts a graph as the underlying model, while HGRec is the first to use a
hypergraph in the reviewer recommendation task. For these baseline methods, we
either made minimal modifications to their source code to accommodate our
dataset or reproduced based on their research paper. We use Accuracy (ACC) and
Mean Reciprocal Rank (MRR), which are commonly used [26], to evaluate the
performance of MIRRec. The replication package of this study has been publicly
available [27].

##  IV Results

###  IV-A Results for RQ1: Hyperparameters Study

Due to limited computational resources, we employ a greedy search strategy to
configure the parameters $\mu$, K, and $\alpha$. As for the parameters $a$,
$b$, $c$, and $d$, which play a role in the ranking calculation after graph
learning, we utilize a grid search strategy to determine the optimal parameter
combinations. The specific details are outlined below:

$\bullet$ The hyperparameter $\mu\in[0,1]$ is used to adjust the impact of the
standardized loss function and the empirical loss function. Due to the high
experimental cost of seeking the optimal solution within a continuous value
range, we conduct experimental analysis on discrete values in steps of $0.1$
within the value range on different projects. Different values of $\mu$ have
varying impacts on the performance of MIRRec across different projects.
However, MIRRec consistently exhibits the same trend across various projects,
achieving optimal performance when $\mu$ is $0.9$. Figure 3 shows the change
curves of the Top-$1$~Top-$5$ and MRR of MIRRec on the projects of Scala and
React under different hyperparameter $\mu$ values111Due to space limitations,
we randomly selected several projects to demonstrate the impact of various
hyperparameters on the model. Therefore, we set the value of
hyperparameter$\mu$ to 0.9 for subsequent experiments.

Figure 3:  ACC and MRR performance of MIRRec under different $\mu$

$\bullet$ The parameter K represents the maximum number of connections a PR
can have in the PR-PR hyperedge. Having too few connections can result in
inadequate learning of higher-order relationships between similar PRs and
developers, while too many connections can interfere with recommendations and
increase model training time. MIRRec aims to recommend a list of reviewers
(Top-$1$~Top-$5$) for PRs based on multiple interactions between PRs and
developers. To obtain higher-order relationships involving at least five
reviewers through at least five similar PRs, we set the hyperparameter K
between $5$ and $25$ using a ‘trial-and-error’ approach to achieve optimal
recommendation performance.

Figure 4: Training time and ACC performance of MIRRec under different K

As shown in Figure 4, the line data and column data represent the model
accuracy (Top-$1$) and training time under different K values, respectively.
Within the range of $5$ to $10$, increasing K leads to improvements in both
accuracy and training time. However, when K exceeds $10$, the model’s accuracy
on different projects fluctuates, with either a slight improvement or
significant decrease, while the training time continues to rise. Thus, we
ultimately determined 10 as the optimal value for hyperparameter K and use it
for subsequent experiments.

$\bullet$ The parameter $\alpha$ represents the influence a reviewer poses in
a history review. To understand how different $\alpha$ settings affect the
performance of MIRRec, this article also adopted a ‘trial-and-error’ method.
We observe the ACC for different projects at Top-$1$ to Top-$5$, and the
coefficient of variation[28] for ACC is consistently below $0.01$, as shown in
Table II. This indicates that $\alpha$ has a limited overall impact on the
performance of MIRRec in different projects. Further literature research
reveals that many reviewer recommendation methods based on interaction
relationships use an empirical value of $0.8$ for $\alpha$ [22]. Therefore, we
set $\alpha$ to $0.8$ in our subsequent experiments.

TABLE II: coefficient of variation for ACC Project | Django | React | XBMC | Scala | Rails
---|---|---|---|---|---
Top-1 | 0.002390 | 0.004174 | 0.003025 | 0.009531 | 0.013636
Top-3 | 0.001733 | 0.003600 | 0.001658 | 0.003270 | 0.003438
Top-5 | 0.000753 | 0.002401 | 0.001199 | 0.000858 | 0.003369

$\bullet$ To explore the impact of four interaction relationships that is, PR-
Reviewers, PR-Committers, PR-Review Commenters, and PR-Issue Commenters, on
reviewer recommendation, we conducted multiple experiments on the entire
dataset based on interaction relationship hypergraphs with different weight
combinations ($a$, $b$, $c$, $d$ in (12)) to optimize performance. Statistical
analysis of the dataset reveals that over $75\%$ of developers who served as
reviewers also served as committers. In comparison, only $25\%$ of developers
who acted as issue commenters have served as reviewers. Additionally, a
substantial overlap was observed between reviewers and creators, as well as
between committers and reviewers. This suggests that the PR-Reviewers and PR-
Committers relationships exert a more significant influence on the reviewer
recommendation compared to the other types of comment-related relationships.

Since there was no obvious way to obtain the optimal weight combination, we
conducted a pilot study through ‘trial and error’ experiments. We found that
the overall performance was better when c:d$\approx$1:1, Thus, we further
conducted ‘trial and error’ experiments with the premise of $c$:$d$=$1$:$1$,
and finally found that $a$: $b$: $c$: $d$=$4$:$3$:$1$:$1$ resulted in better
model performance. The partial results are shown in Table III.

TABLE III: MIRRec Performance under different weight combinations(%) a |  b |  c |  d |  $ACC_{1}$ |  $ACC_{3}$ |  $ACC_{5}$ |  $MRR_{1}$ |  $MRR_{3}$ |  $MRR_{5}$
---|---|---|---|---|---|---|---|---|---
1 |  1 |  1 |  1 |  46.5 |  80.3 |  90.4 |  46.5 |  61.7 |  64
2 |  1 |  1 |  1 |  46.7 |  80.4 |  90.6 |  46.7 |  61.9 |  64.3
3 |  1 |  1 |  1 |  47.2 |  80.3 |  90.8 |  47.2 |  62.2 |  64.6
4 |  1 |  1 |  1 |  47.1 |  79.7 |  90.7 |  47.1 |  62.0 |  64.5
5 |  1 |  1 |  1 |  47.4 |  80.2 |  90.8 |  47.4 |  62.3 |  64.7
1 |  2 |  1 |  1 |  46.2 |  80.1 |  90.4 |  46.2 |  61.4 |  63.8
2 |  2 |  1 |  1 |  47.1 |  79.9 |  90.6 |  47.1 |  62.1 |  64.6
3 |  2 |  1 |  1 |  48.0 |  80.0 |  90.7 |  48.0 |  62.5 |  65.0
4 |  2 |  1 |  1 |  47.6 |  80.0 |  90.6 |  47.6 |  62.4 |  64.8
5 |  2 |  1 |  1 |  47.6 |  80.1 |  90.9 |  47.6 |  62.4 |  64.9
1 |  3 |  1 |  1 |  45.0 |  80.9 |  89.7 |  45.0 |  60.9 |  63.0
2 |  3 |  1 |  1 |  46.8 |  80.4 |  90.6 |  46.8 |  61.9 |  64.3
3 |  3 |  1 |  1 |  48.1 |  80.2 |  90.2 |  48.1 |  62.6 |  65
4 |  3 |  1 |  1 |  48.3 |  80.3 |  91 |  48.3 |  62.7 |  65.2
5 |  3 |  1 |  1 |  48.1 |  79.9 |  90.8 |  48.1 |  62.5 |  65.0

Answer: The hyperparameter $\mu$, K, $\alpha$ were set to $0.9$, $10$, $0.8$,
and $a$: $b$: $c$: $d$ was set to $4$:$3$:$1$:$1$ for MIRRec determined by
’trial and error’ experiments, demonstrating consistent and superior
performance across projects.

###  IV-B Results for RQ2: Overall Performance Comparison

TABLE IV: Comparison of ACC across Different Methods (%) Project | RevFinder | cHRev | CN | HGRec | MIRRec
---|---|---|---|---|---
Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5
Bitcoin |  31.1 |  58.1 |  69.7 |  24.5 |  50.7 |  65.7 |  32.0 |  58.2 |  71.3 |  34.0 |  61.8 |  73.1 |  34.8 |  63.2 |  74.8
Electron |  36.4 |  75.1 |  90.5 |  16.3 |  51.4 |  78.1 |  41.7 |  76.5 |  92.8 |  41.7 |  77.1 |  92.7 |  41.5 |  78 |  93.1
Opencv |  58.9 |  83.5 |  90.9 |  47.7 |  75.6 |  84.1 |  69.4 |  88.1 |  92.8 |  65.3 |  88.8 |  94.8 |  68.8 |  89.8 |  95.5
XBMC |  29.0 |  55.5 |  67.6 |  34.0 |  59.3 |  71.5 |  42.8 |  63.5 |  72.9 |  43.2 |  66.6 |  76.8 |  44.9 |  68.7 |  78.7
React |  24.3 |  56.0 |  74.9 |  22.2 |  54.8 |  71.2 |  43.6 |  73.9 |  85.5 |  41.7 |  73.2 |  86.1 |  43.8 |  73.8 |  87.1
Angular |  29.6 |  61.6 |  77.6 |  32.6 |  63.5 |  77.4 |  40.2 |  66.5 |  76.5 |  46.8 |  72.2 |  83.4 |  48.5 |  73.8 |  84.1
Django |  46.9 |  79.3 |  87.6 |  36.0 |  69.5 |  81.6 |  57.0 |  77.8 |  85.1 |  59.1 |  81.7 |  87.9 |  59.9 |  81.9 |  88.1
Symfony |  46.8 |  79.3 |  87.1 |  36.8 |  68.3 |  80.5 |  45.4 |  77.2 |  85.7 |  48.2 |  80.8 |  89.1 |  49.5 |  82.2 |  89.5
Rails |  24.7 |  45.2 |  58.9 |  14.7 |  34.4 |  47.6 |  20.9 |  41.7 |  54.5 |  21.9 |  44.9 |  58.6 |  22.3 |  47.4 |  59.6
Scala |  43.1 |  74.2 |  86.5 |  28.8 |  63.3 |  77.7 |  45.5 |  77.5 |  87.4 |  46.9 |  79.3 |  90.3 |  48.3 |  80.3 |  91.0
Average |  37.1 |  66.8 |  79.1 |  29.4 |  59.1 |  73.5 |  43.9 |  70.1 |  80.5 |  44.9 |  72.6 |  83.3 |  46.2 |  73.9 |  84.2
Improve |  24.53% |  10.63% |  6.45% |  57.14% |  25.04% |  14.56% |  5.24% |  5.42% |  4.60% |  2.90% |  1.79% |  1.08% |  - |  - |  -
Note:‘Average’ indicates the mean performance across the entire dataset.
‘Improve’ signifies the degree to which MIRRec outperforms other methods in
terms of average performance. Bold indicates the optimal result, and three
gray background colors, from light to dark, are used to highlight the top
three results (Third, Second, First) in terms of performance for Top-1, Top-3,
and Top-5, respectively.
TABLE V: Comparison of MRR across Different Methods (%) Project | RevFinder | cHRev | CN | HGRec | MIRRec
---|---|---|---|---|---
Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5
Bitcoin |  31.1 |  42.8 |  45.4 |  24.5 |  35.8 |  39.3 |  32.0 |  43.4 |  46.4 |  34.0 |  46.3 |  48.8 |  34.8 |  47.3 |  49.9
Electron |  36.4 |  53.4 |  56.9 |  16.3 |  31.0 |  37.1 |  41.7 |  56.9 |  60.7 |  41.7 |  57.2 |  60.8 |  41.5 |  57.4 |  60.9
Opencv |  58.9 |  70.3 |  71.9 |  47.7 |  60.0 |  62.0 |  69.4 |  77.8 |  78.9 |  65.3 |  76.1 |  77.5 |  68.8 |  78.2 |  79.6
XBMC |  29.0 |  40.5 |  43.3 |  34.0 |  45.1 |  47.9 |  42.8 |  52.0 |  54.2 |  43.2 |  53.4 |  55.8 |  44.9 |  55.4 |  57.7
React |  24.3 |  38.0 |  42.3 |  22.2 |  36.3 |  40.0 |  43.6 |  57 |  59.6 |  41.7 |  55.5 |  58.6 |  43.8 |  56.9 |  60
Angular |  29.6 |  43.5 |  47.2 |  32.6 |  46.1 |  49.3 |  40.2 |  51.6 |  53.9 |  46.8 |  58.0 |  60.6 |  48.5 |  59.6 |  62.0
Django |  46.9 |  61.8 |  63.8 |  36.0 |  50.8 |  53.6 |  57 |  66.2 |  67.9 |  59.1 |  69.5 |  70.9 |  59.9 |  70.0 |  71.4
Symfony |  46.8 |  61.7 |  63.4 |  36.8 |  50.6 |  53.4 |  45.4 |  60.0 |  62.0 |  48.2 |  62.8 |  64.8 |  49.5 |  64.2 |  65.9
Rails |  24.7 |  33.4 |  36.5 |  14.7 |  23.2 |  26.2 |  20.9 |  29.9 |  32.8 |  21.9 |  31.8 |  34.9 |  22.3 |  33.2 |  36
Scala |  43.1 |  56.6 |  59.4 |  28.8 |  43.6 |  46.9 |  45.5 |  59.7 |  61.9 |  46.9 |  61.7 |  64.2 |  48.3 |  62.7 |  65.2
Average |  37.1 |  50.2 |  53 |  29.4 |  42.3 |  45.6 |  43.9 |  55.5 |  57.8 |  44.9 |  57.2 |  59.7 |  46.2 |  58.5 |  60.9
Improve |  24.53% |  16.53% |  14.91% |  57.14% |  38.30% |  33.55% |  5.24% |  5.41% |  5.36% |  2.90% |  2.27% |  2.01% |  - |  - |  -
Note: ‘Average’, ‘Improve’, font and colors have the same meanings as in Table
IV.

Table IV and V display the overall performance of MIRRec and four baseline
approaches on the dataset. In terms of ACC, MIRRec outperforms most of the
baselines on the majority of projects. Across all projects, MIRRec
consistently achieves higher Top-1, Top-3, and Top-5 recommendation average
ACC metrics. Specifically, MIRRec outperforms RevFinder by 24.53%, 10.63%, and
6.45%, respectively; cHRev by 57.14%, 25.04%, and 14.56%; CN by 5.24%, 5.42%,
and 4.60%; and HGRec by 2.90%, 1.79%, and 1.08%. MRR is an indicator that
reflects the order of recommendation results. The larger the MRR, the higher
the average ranking of recommended real code reviewers. Our results for MRR
mirror those of ACC, with MIRRec consistently leading the performance among
all the recommenders in this study. As shown in Table V, MIRRec achieves, on
average, 24.53%, 16.53%, and 14.91% higher MRR values for Top-$1$, Top-$3$,
and Top-$5$ compared to RevFinder; 57.14%, 38.30%, and 33.55% higher than
cHRev; 5.24%, 5.41%, and 5.36% higher than CN; and 2.90%, 2.27%, and
2.01%higher than HGRec. These results demonstrate that MIRRec can accurately
recommend the first real reviewers at a lower rank than the baselines,
highlighting its superior recommendation accuracy.

The potential reasons behind this phenomenon are summarized as follows: $(1)$
MIRRec, CN, and HGRec are graph-based algorithms that excel at conducting
feature learning for developers and PRs by exploring explicit and implicit
multi-relationships, thereby significantly enhancing recommendation accuracy.
$(2)$ Compared to graph-based methods, hypergraph-based methods, such as HGRec
and MIRRec, have distinct advantages in capturing higher-order relationships
between nodes, leading to improved accuracy over graph-based methods like CN.
Specifically, MIRRec is designed to capture potential correlation features
between PRs and reviewers influenced by multiple relationships. This makes it
superior in terms of accuracy compared to the HGRec method, which only
considers PR-Reviewer and PR-Contributor relationships in interaction. $(3)$
MIRRec learns the representation of developers and PRs through interaction and
similarity relationships. From the perspective of developers, MIRRec learns
their representation from the tasks (PRs) they interact with. This is similar
to how RevFinder learns developers’ expertise from the similarity of file path
sets in tasks, or how cHRev learns developers’ expertise from their code
files. However, unlike cHRev and RevFinder, MIRRec can extract more useful
information and enhance feature learning by aggregating high-order neighbors.
Answer: MIRRec outperforms the baselines in ACC and MRR, demonstrating its
accuracy and superiority in leveraging multi-relationships and higher-order
connections between developer and PR for recommendation.

###  IV-C Results for RQ3: Impact of Relations

TABLE VI: ACC for Variants of MIRRec Based on Different Relations (%) Project | MIRRecct_ic_rc | MIRRecre_ic_rc | MIRRecre_ct_rc | MIRRecre_ct_ic | MIRRecre_ct | MIRRecre_ic | MIRRec
---|---|---|---|---|---|---|---
Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5
Bitcoin |  34.5 |  59.6 |  70.8 |  34.1 |  61.3 |  73.7 |  35.1 |  63.6 |  74.4 |  34.9 |  63.5 |  75.1 |  35.5 |  63.9 |  74.8 |  34.2 |  61.8 |  74.0 |  34.8 |  63.2 |  74.8
Electron |  33.8 |  71.7 |  90.3 |  41.4 |  77.0 |  92.7 |  41.7 |  78.0 |  92.9 |  41.7 |  78.3 |  93.1 |  41.8 |  78.4 |  93.0 |  41.7 |  77.2 |  93.0 |  41.5 |  78.0 |  93.1
Opencv |  61.7 |  88.8 |  94.4 |  67.9 |  89.7 |  95.1 |  67.5 |  89.6 |  95 |  68.8 |  89.7 |  95.5 |  68.1 |  89.7 |  95.1 |  68.4 |  89.7 |  95.1 |  68.8 |  89.8 |  95.5
XBMC |  40.4 |  63.6 |  76.5 |  43.4 |  66.9 |  77.4 |  44.7 |  68.3 |  79.0 |  45.0 |  68.5 |  78.7 |  45.3 |  68.8 |  79.1 |  43.6 |  66.8 |  77.5 |  44.9 |  68.7 |  78.7
React |  42.0 |  71.3 |  84.8 |  42.6 |  73.5 |  86.7 |  43.6 |  74.2 |  87.0 |  43.8 |  74.2 |  87.1 |  43.7 |  74.7 |  86.9 |  42.7 |  73.6 |  86.5 |  43.8 |  73.8 |  87.1
Angular |  44.8 |  70.6 |  82.3 |  47.2 |  72.8 |  83.6 |  48.1 |  73.3 |  84.1 |  48.7 |  73.9 |  84.3 |  48.6 |  73.5 |  84.2 |  47.2 |  72.9 |  83.6 |  48.5 |  73.8 |  84.1
Django |  58.9 |  80.2 |  86.4 |  59.8 |  81.8 |  87.9 |  60.0 |  81.8 |  88.1 |  60.2 |  81.9 |  88.0 |  60.2 |  81.9 |  88.2 |  59.6 |  82.0 |  87.9 |  59.9 |  81.9 |  88.1
Symfony |  45.6 |  80.1 |  88.2 |  49.6 |  81.7 |  89.3 |  48.4 |  81.5 |  89.5 |  50.5 |  82.4 |  89.6 |  48.9 |  81.8 |  89.7 |  49.8 |  81.9 |  89.5 |  49.5 |  82.2 |  89.5
Rails |  16.6 |  42.2 |  56.3 |  22.7 |  47.1 |  59.0 |  21.1 |  46.3 |  59.4 |  22.6 |  47.1 |  60.0 |  21.2 |  46.2 |  59.0 |  22.8 |  46.9 |  59.1 |  22.3 |  47.4 |  59.6
Scala |  43.5 |  78.4 |  89.8 |  47.2 |  80.0 |  90.4 |  47.4 |  80.3 |  90.7 |  48.1 |  80.4 |  91.3 |  47.5 |  80.4 |  90.8 |  47.4 |  80.0 |  90.3 |  48.3 |  80.3 |  91.0
Average |  42.2 |  70.7 |  82.0 |  45.6 |  73.2 |  83.6 |  45.8 |  73.7 |  84.0 |  46.4 |  74.0 |  84.3 |  46.1 |  73.9 |  84.1 |  45.7 |  73.3 |  83.7 |  46.2 |  73.9 |  84.2
Improve |  9.48% |  4.53% |  2.68% |  1.32% |  0.96% |  0.72% |  0.87% |  0.27% |  0.24% |  -0.43% |  -0.14% |  -0.12% |  0.22% |  - |  0.12% |  1.09% |  0.82% |  0.60% |  - |  - |  -
Note: ‘Average’, ‘Improve’, font and colors have the same meanings as in Table
IV.
TABLE VII: MRR for Variants of MIRRec Based on Different Relations (%)  Project | MIRRecct_ic_rc | MIRRecre_ic_rc | MIRRecre_ct_rc | MIRRecre_ct_ic | MIRRecre_ct | MIRRecre_ic | MIRRec
---|---|---|---|---|---|---|---
Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5 |  Top-1 |  Top-3 |  Top-5
Bitcoin |  34.5 |  45.5 |  48.1 |  34.1 |  46.1 |  49.0 |  35.1 |  47.6 |  50.1 |  34.9 |  47.5 |  50.1 |  35.5 | 48.0 | 50.5 |  34.2 | 46.3 | 49.1 |  34.8 |  47.3 |  49.9
Electron |  33.8 |  50.1 |  54.4 |  41.4 |  57.0 |  60.7 |  41.7 |  57.5 |  60.9 |  41.7 |  57.7 |  61.1 |  41.8 |  57.8 |  61.2 |  41.7 |  57.3 |  60.9 |  41.5 |  57.4 |  60.9
Opencv |  61.7 |  74.0 |  75.3 |  67.9 |  77.8 |  79.0 |  67.5 |  77.6 |  78.9 |  68.8 |  78.3 |  79.6 |  68.1 |  77.9 |  79.1 |  68.4 |  78.0 |  79.3 |  68.8 |  78.2 |  79.6
XBMC |  40.4 |  50.5 |  53.5 |  43.4 |  53.6 |  56.0 |  44.7 |  55.1 |  57.6 |  45.0 |  55.4 |  57.7 |  45.3 |  55.6 |  58.0 |  43.6 |  53.7 |  56.2 |  44.9 |  55.4 |  57.7
React |  42.0 |  54.8 |  58.0 |  42.6 |  56.2 |  59.2 |  43.6 |  57.0 |  60.0 |  43.8 |  57.1 |  60.1 |  43.7 |  57.2 |  60.0 |  42.7 |  56.2 |  59.2 |  43.8 |  56.9 |  60.0
Angular |  44.8 |  56.2 |  58.9 |  47.2 |  58.6 |  61.0 |  48.1 |  59.2 |  61.6 |  48.7 |  59.8 |  62.2 |  48.6 |  59.5 |  61.9 |  47.2 |  58.7 |  61.1 |  48.5 |  59.6 |  62.0
Django |  58.9 |  68.4 |  69.8 |  59.8 |  69.8 |  71.2 |  60.0 |  70.0 |  71.4 |  60.2 |  70.1 |  71.5 |  60.2 |  70.1 |  71.5 |  59.6 |  69.8 |  71.2 |  59.9 |  70.0 |  71.4
Symfony |  45.6 |  61.2 |  63.1 |  49.6 |  64.1 |  65.8 |  48.4 |  63.3 |  65.2 |  50.5 |  64.8 |  66.4 |  48.9 |  63.7 |  65.5 |  49.8 |  64.3 |  66.0 |  49.5 |  64.2 |  65.9
Rails |  16.6 |  27.6 |  30.8 |  22.7 |  33.3 |  36.0 |  21.1 |  32.2 |  35.2 |  22.6 |  33.3 |  36.3 |  21.2 |  32.3 |  35.2 |  22.8 |  33.3 |  36.1 |  22.3 |  33.2 |  36.0
Scala |  43.5 |  58.9 |  61.6 |  47.2 |  62.0 |  64.4 |  47.4 |  62.4 |  64.8 |  48.1 |  62.7 |  65.2 |  47.5 |  62.4 |  64.8 |  47.4 |  62.1 |  64.5 |  48.3 |  62.7 |  65.2
Avarage |  42.2 |  54.7 |  57.4 |  45.6 |  57.9 |  60.2 |  45.8 |  58.2 |  60.6 |  46.4 |  58.7 |  61.0 |  46.1 |  58.5 |  60.8 |  45.7 |  58.0 |  60.4 |  46.2 |  58.5 |  60.9
Improve |  9.48% |  6.95% |  6.10% |  1.32% |  1.04% |  1.16% |  0.87% |  0.52% |  0.50% |  -0.43% |  -0.34% |  -0.16% |  0.22% |  - |  0.16% |  1.09% |  0.86% |  0.83% |  - |  - |  -
Note: ‘Average’, ‘improve’, font and colors have the same meanings as in Table
IV.

To further investigate the impact of different hyperedges and the
effectiveness of MIRRec based on the hypergraph models constructed with
different hyperedges, we constructed several models by pruning the
relationships, including $\text{MIRRec}_{ct\\_ic\\_rc}$,
$\text{MIRRec}_{re\\_ic\\_rc}$, $\text{MIRRec}_{re\\_ct\\_rc}$,
$\text{MIRRec}_{re\\_ct\\_ic}$, $\text{MIRRec}_{re\\_ct}$, and
$\text{MIRRec}_{re\\_ic}$, using the weight combination setting mentioned
above. The subscript represents the partial interaction relationships used to
construct each model: $re$ (PR-Reviewers), $ct$ (PR-Committers), $ic$ (PR-
Issue Commenters), and $rc$ (PR-Review Commenters). It is worth noting that we
did not perform an ablation experiment on the PR-creator relationship. The
reason is that the creator is the closest role to the PR compared to the
reviewer and commenter, reflecting developers’ involvement and expertise in
the relevant domain of the PR under review. This observation aligns with the
phenomenon we discovered, that many creators review their own created PRs,
supporting the significance of considering this relationship as fundamental.
Tables VI and VII display the performance of these models in terms of ACC and
MRR, respectively.

As shown in Table VI, MIRRec achieves an average Top-1 ACC that is
approximately 9.48% higher than $\text{MIRRec}_{ct\\_ic\\_rc}$. This reveals
the significant impact of the $re$ relationship on recommendations, aligning
with the regularity of recommendation systems based on historical
interactions. MIRRec also slightly outperforms $\text{MIRRec}_{re\\_ic\\_rc}$
with an improvement of 1.32%, 0.96%, 0.72% for Top-$1$, Top-$3$, and Top-$5$,
indicating a positive influence from the $ct$ relationship, likely due to its
ability to capture familiarity with PRs and potential developer collaboration.
While the ACC of MIRRec is slightly lower than that of
$\text{MIRRec}_{re\\_ct}$ on certain projects, the average ACC remains higher
for MIRRec than for $\text{MIRRec}_{re\\_ct}$. This suggests that the two
comment relationships, $ic$ and $rc$, may have some potential positive impact
but also carry the risk of complex information disruptions. Additional
experiments with $\text{MIRRec}_{re\\_ct\\_rc}$,
$\text{MIRRec}_{re\\_ct\\_ic}$, and $\text{MIRRec}_{re\\_ic}$ also confirm the
potential risk of information interference.

Meanwhile, the performance of the MRR of these models shows a high similar to
the performance of ACC as Table VII shows. From these results we can infer
that $ic$ usually plays a positive role, while $rc$ has a negative impact on
some projects. We speculate that this may be because $rc$ is generated based
on $re$, and thus it may introduce some invalid or redundant information on
top of the already existing review interactions. The relation $ic$, on the
other hand, can somewhat capture the potential review and collaboration
possibilities of other developers as opposed to the review comment
relationship. Therefore, we recommend using MIRRec without the $rc$
relationship as the optimal recommender.

We further analyze the statistics to find the potential reasons for these
phenomena. In the interaction data related to $ct$, $ic$, $re$, and $rc$ of
the datasets, we observe that about 76% to 87% of developers in the projects
played only one role out of $ct$, $ic$, and $re$. Moreover, 7% to 15% took on
two of the four roles, 3% to 7% managed three of the four roles, and 2% to 7%
juggled all four roles simultaneously. It should be noted that many developers
who served as $rc$ also took on other roles within the project. In the
Bitcoin, React, and Rails projects, developers exclusively held the role of
$rc$, and this accounted for only about 1% of the total. This highlights the
importance of learning higher-order relationships between developers and PRs
from the interaction histories related to $ct$, $ic$, and $re$ data. However,
it is important to note that the data from $rc$, in addition to $ct$, $ic$,
and $re$, can be considered redundant since these three categories already
encompass most of the $rc$ data in the projects. This redundancy may lead to
overfitting.

Answer: The $re$ relationship significantly influences recommendations, with
$ct$ and $ic$ contributing positively, while $rc$ may introduce redundancy.
This underscores the importance of considering higher-order relationships but
also raises caution about potential overfitting due to overlapping data,
highlighting the need for meticulous data analysis in recommendation systems.

##  V Threats to validaity

External Validity. The evaluation experiments are conducted on a dataset we
constructed rather than on previously publicly available datasets, such as the
dataset introduced by Rong et al. [22]. The reason is that the available
datasets released may not be sourced directly and lack specific relationships.
Consequently, obtaining the necessary multiplex relational information proves
unfeasible. While the projects in our dataset may not represent all OSS
projects, they were selected based on predetermined criteria to ensure their
relevance and diversity. Furthermore, some of the projects utilized have been
referenced in previous studies and the data is up-to-date. As part of our
future work, we plan to expand the evaluation to a wider range of projects,
both open-source and industrial.

Internal Validity. The internal validity of this work is primarily related to
data preprocessing, experiment setup, and hyperparameter tuning. During data
preprocessing, issues such as inconsistent developer names and invalid
identities were addressed, which could have disrupted the accurate
representation of interaction relationships. To mitigate these threats, we
applied various preprocessing techniques to construct the dataset. In terms of
experimental setup, we partitioned the dataset into sliding training and
testing subsets for multiple rounds of experimentation, using average
performance as the evaluation metric. This approach helps mitigate threats
arising from varying data sample sizes due to different project development
velocities over time. Regarding hyperparameter tuning, while it is challenging
to determine optimal values exhaustively, we conducted a series of ‘trial and
error’ experiments to identify suitable hyperparameters. This approach
effectively reduces the potential threats associated with suboptimal
hyperparameters to a certain extent.

Construct Validity. MIRRec is a recommendation approach that focuses on
relationship-aware reviewer recommendations using a hypergraph of multiplex
relationships. To construct this hypergraph, we implemented a series of data
preprocessing steps to retain crucial information and minimize the impact of
redundant data, enhancing the interpretability of our research findings. We
evaluated the effectiveness of our method by comparing it with four state-of-
the-art baselines using metrics such as ACC and MRR. In terms of workload
assessment for the recommended reviewers of MIRRec, we have not delved into
detailed discussions in this paper. However, we provide specific results on
the Recommendation Distribution [22] of MIRRec and baseline methods in the
replication package [27]. Our study results indicate that, while maintaining
excellent accuracy, MIRRec, in contrast to other baseline methods, is more
adept at considering the balanced distribution of workload, thus providing
more diverse recommendation outcomes. The results from our ablation
experiments further underscore the efficacy of constructing a hypergraph based
on multiplex relationships. However, it is worth noting that our approach
relies on the actual reviewers assigned to tasks as ground truth, without
considering factors such as their expertise, reputation, and workload. This
limitation could potentially lead to recommendations that may not always be
the most suitable.

Conclusion Validity. To ensure the validity of our conclusions, we followed a
meticulous and systematic procedure for experimentation and analysis. Our
proposed method was tested on ten OSS projects with rich interaction
histories, encompassing over $4.8$K PRs and $47.3$K interactions. We have
provided a clear elaboration of all the data used, which is publicly
available. This transparency enhances the reliability of our conclusions, as
they can be traced back to the original data, allowing for replication by
other researchers.

##  VI Related work

Industry and academia have made great efforts in recommending code
reviewers[26] recently, and put forward many approaches to select the
appropriate reviewers.

As the necessity for expertise and prior knowledge in code review, many
studies have noted the importance of recommending suitable reviewers, which
can lead to faster and higher quality software updates. For instance,
Hannebauer et al.[29] found that the code reviewers’ recommendation algorithms
based on review expertise outperform those based solely on modification
expertise. Asthana et al. [30] identified potential reviewers by leveraging
their past experience with the files and directories involved in a code
review. Fejzer et al. [31] suggested reviewers based on the similarity between
developers’ expertise profiles and changes to be reviewed. Rahman et al.[32]
heuristically captured relevant cross-project work history and specialized
technologies used in a PR for reviewer recommendation. However, these methods
made solely based on developers’ individual qualities may result in a focus on
recommending core reviewers. In response to this, Rebai et al. [33] formulated
reviewer recommendation as a multi-objective search problem by constructing an
expertise model, availability model, and a collaboration model to balance the
conflicting objectives of expertise, availability, and history of
collaborations. Some researchers[5][34] employed multi-objective search-based
method to find the optimal reviewers by balancing both expertise and workload.

Most of the reviewer recommenders that we know depend on historical reviews
information, and recommend developers based on matrix factorization or
collaborative filtering [35][36][37]. For example, Chueshev et al. [37]
introduced a recommender-based approach for OSS projects, which uses
collaborative filtering to recommend regular reviewers and expand their
numbers from suitable developers. Some researchers have acknowledged the value
of additional information in update changes to enhance reviewer
recommendation. Xia et al.[38] proposed a hybrid and incremental approach to
recommend code reviewers, combining text mining and file location-based
similarity measures between new and previous files. Ye et al. [39] proposed a
multi-instance-based deep neural network model that utilizes LSTM and CNN to
recommend reviewers for PRs based on information from the PR title, commit
message, and code changes. Li et al.[40] proposed an approach to recommend
code reviewers for architecture violation issues based on the file path
similarity of code commits and the semantic similarity of review comments.
Jiang et al.[41] built approaches based on various attributes, including
activeness, text similarity, file similarity and social relations to recommend
commenters for PRs. Jiang et al.[42] also developed a classifier based on SVM
that considers features such as file paths of modified codes, relationships
between contributors and core members, and the activeness of core members to
recommend core reviewers for PRs. While these methods that utilize historical
and change information have been validated and successful, they may overlook
capable reviewers who have not interacted with these tasks before.

Another line of research focuses more on leveraging graph techniques within
the developers’ network[43][44]. Liao et al. [45] proposed a core-reviewer
recommendation approach that combines PR topic model with collaborators in the
social network. Ying et al. [19] constructed a graph connecting incoming PRs
with potential reviewers, using text similarity of PRs and social relations
among developers to identify suitable reviewers, effectively considering
developer expertise and authority. Sülün et al. [20] introduced the ‘know-
about’ metric to measure familiarity between developers and artifacts and
utilized traceability graphs of software artifacts to recommend reviewers for
specific changes. Zhang et al.[46] employed a graph convolutional neural
network for reviewer recommendation that leverages a socio-technical graph
built from the rich set of entities (developers, repositories, files, PRs,
work items, etc. Yu et al.[18] extended three typical approaches based on
machine learning, information retrieval, and file location techniques to
recommend reviewers to PRs. They also proposed a hybrid approach that combines
the comment network with traditional approaches, achieving significant
improvement over the traditional methods. Rong et al. [22] adopted the
hypergraph to model high-order relationships between PRs and developers, then
recommended reviewers based on the model. However, a common limitation of
these is that they do not sufficiently consider the complex interactions
between developers and tasks in the review process.

Key difference. In contrast to prior work, our objective is to explore the
multiplex relationships between developers and PRs, as well as their high-
order connectivities in the review process, with the goal of providing more
accurate reviewer recommendations. Unlike previous studies, which typically
incorporate explicit historical review interactions (PR-Reviewer) and task
similarity (PR-PR) into their models, and consider contributor interactions
(eg. PR-Committer and PR-Issue Commenter) as a single type relationship beyond
the review interaction. We take a different strategy treat these relationships
separately. While HGRec made an improvement by using hypergraphs to model code
reviews and the intricate relationships among participants, our approach,
MIRRec goes further by measuring different hyperedges using multiple factors,
including commit time, lines of code modified, comment time, comment
frequency, and the behavior recency of developers. Thus, emphasizing the depth
and specificity of different contributions. In summary, MIRRec places a
stronger emphasis on advancing the understanding and learning the different
contributions of these complex relationships based on diverse behaviors under
different roles. The evaluation results on OSS projects suggest that
conducting an in-depth analysis of the implications of various relations and
hypergraph technique contributes to the improved accuracy of recommendations.

##  VII Conclusion and future work

In this paper, we propose MIRRec, a novel relationship-aware hypergraph-based
approach for enhancing reviewer recommendation in OSS projects. MIRRec
formally defines the interaction relationships between developers and PRs and
the similarity relationship between PRs. It constructs a multiplex-
relationship hypergraph to model high-order relationships between developers
and PRs. Comprehensive experiments conducted on ten OSS projects indicate that
MIRRec outperforms state-of-the-art baselines. An ablation study was also
carried out to investigate the influence of different relationships on
reviewer recommendations, further validating the effectiveness of the proposed
method.

Our results reveal that the majority of developers engage in separate types of
interactions, including commits, reviews, and issue comments, within the
project. Some developers exhibit at least two types of interaction behaviors,
and the majority of developers who contribute review comments also participate
in additional types of interactions. Only a tiny percentage of developers
exclusively participate in review comment activities. Consequently, it is
crucial to consider relationships related to commits and issue comments
alongside the historical review interactions. It is worth noting that when the
recommender aims to comprehensively learn the potential high-order
connectivity based on the additional review comment behaviors, addressing the
issue of data redundancy leading to overfitting is necessary.

In the future, we aim to enhance MIRRec by incorporating developers’
individual abilities such as expertise, response quality, contributions, and
impact into their representations. Additionally, we plan to expand our
evaluation to a wider range of OSS projects and deploy MIRRec as a GitHub
plugin to conduct practical experiments with developers to assess its
effectiveness in reviewer recommendations.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China
(Nos. 62032016 and 62172311), the Key Research and Development Program of
Hubei Province (No. 2021BAA031), and the Natural Science Foundation of Hubei
Province (No. 2023AFB374).

## References

  * [1] D. Badampudi, M. Unterkalmsteiner, and R. Britto, “Modern Code Reviews—Survey of Literature and Practice,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 1-61, 2023.
  * [2] X. Yang, R. G. Kula, N. Yoshida, and H. Iida, “Mining the modern code review repositories: a dataset of people, process and product,” in Proceedings of the 13th International Conference on Mining Software Repositories (MSR), pp. 460-463, 2016.
  * [3] S. Ruangwan, P. Thongtanunam, A. Ihara, and K. Matsumoto, “The impact of human factors on the participation decision of reviewers in modern code review,” Empirical Software Engineering, vol. 24, no. 2, pp. 973-1016, 2018.
  * [4] D. Kong, Q. Chen, L. Bao, C. Sun, X. Xia, and S. Li, “Recommending Code Reviewers for Proprietary Software Projects: A Large Scale Study,” in Proceedings of the 29th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 630-640, 2022.
  * [5] M. Chouchen, A. Ouni, M. W. Mkaouer, R. G. Kula, and K. Inoue, “WhoReview: A multi-objective search-based approach for code reviewers recommendation in modern code review,” Applied Soft Computing, vol. 100, 106908, 2021.
  * [6] E. Doğan, E. Tüzün, K. A. Tecimer, and H. A. Güvenir, “Investigating the Validity of Ground Truth in Code Reviewer Recommendation Studies,” in Proceedings of the 13th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1-6, 2019.
  * [7] G. Zhao, D. A. da Costa, and Y. Zou, “Improving the pull requests review process using learning-to-rank algorithms,” Empirical Software Engineering, vol. 24, no. 4, pp. 2140-2170, 2019.
  * [8] P. C. Rigby, and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 202-212, 2013.
  * [9] J Tsay, L. Dabbish, and J. Herbsleb, “Let’s Talk About It: Evaluating Contributions through Discussion in GitHub,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE), pp. 144-154, 2014.
  * [10] A. Lee, J. C. Carver, and A. Bosu, “Understanding the Impressions, Motivations, and Barriers of One Time Code Contributors to FLOSS Projects: A Survey,” in Proceedings of the 39th IEEE/ACM International Conference on Software Engineering (ICSE), pp. 187-197, 2017.
  * [11] V. Balachandran, “Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation,” in Proceedings of the 35th International Conference on Software Engineering (ICSE), pp. 931-940, 2013.
  * [12] P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, and K.-i. Matsumoto, “Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review,” in Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 141-150, 2015.
  * [13] A. Ouni, R. G. Kula, and K. Inoue, “Search-Based Peer Reviewers Recommendation in Modern Code Review,” in Proceedings of the 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 367-377, 2016.
  * [14] Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 1035-1047, 2022.
  * [15] M. B. Zanjani, H. Kagdi, and C. Bird, “Automatically Recommending Peer Reviewers in Modern Code Review,” IEEE Transactions on Software Engineering, vol. 42, no. 6, pp. 530-543, 2016.
  * [16] T. Hirao, S. McIntosh, A. Ihara, and K. Matsumoto, “The review linkage graph for code review analytics: a recovery approach and empirical study,” in Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 578-589, 2019.
  * [17] P. Thongtanunam, and A. E. Hassan, “Review Dynamics and Their Impact on Software Quality,” IEEE Transactions on Software Engineering, vol. 47, no. 12, pp. 2698-2712, 2021.
  * [18] Y. Yu, H. Wang, G. Yin, and T. Wang, “Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment?,” Information and Software Technology, vol. 74, pp. 204-218, 2016.
  * [19] H. Ying, L. Chen, T. Liang, and J. Wu, “EARec:Leveraging Expertise and Authority for Pull-Request Reviewer Recommendation in GitHub,” in Proceedings of the 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE), pp. 29-35, 2016.
  * [20] E. Sülün, E. Tüzün, and U. Doğrusöz, “RSTrace+: Reviewer suggestion using software artifact traceability graphs,” Information and Software Technology, vol. 130, pp. 106455, 2021.
  * [21] X. Xie, X. Yang, B. Wang, and Q. He, “DevRec: Multi-Relationship Embedded Software Developer Recommendation,” IEEE Transactions on Software Engineering, vol. 48, pp. 4357-4379, 2022.
  * [22] G. Rong, Y. Zhang, L. Yang, F. Zhang, H. Kuang, and H. Zhang, “Modeling review history for reviewer recommendation:A Hypergraph Approach,” in Proceedings of the 44th International Conference on Software Engineering (ICSE), pp. 1381-1392, 2022.
  * [23] Pull request reviews in GitHub: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews
  * [24] J. Krüger, J. Wiemann, W. Fenske, G. Saake, and T. Leich, “Do you remember this source code?,” in Proceedings of the 40th International Conference on Software Engineering (ICSE), pp. 764-775, 2018.
  * [25] G. Navarro, “A guided tour to approximate string matching,” ACM Computing Surveys, vol. 33, no. 1, pp. 31-88, 2001.
  * [26] H. A. Çetin, E. Doğan, and E. Tüzün, “A review of code reviewer recommendation studies: Challenges and future directions,” Science of Computer Programming, vol. 208, pp. 102652, 2021.
  * [27] Replication Package of the Paper: https://github.com/cufeinfor/MIRRec
  * [28] E. L. Lehmann, J. P. Romano, and G. Casella, Testing Statistical Hypotheses: Springer, 1986.
  * [29] C. Hannebauer, M. Patalas, S. Stünkel, and V. Gruhn, “Automatically recommending code reviewers based on their expertise: an empirical comparison,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 99-110, 2016.
  * [30] S. Asthana, R. Kumar, R. Bhagwan, C. Bird, C. Bansal, C. Maddila, S. Mehta, and B. Ashok, “WhoDo: Automating Reviewer Suggestions at Scale,” in Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 937-945, 2019.
  * [31] M. Fejzer, P. Przymus, and K. Stencel, “Profile based recommendation of code reviewers,” Journal of Intelligent Information Systems, vol. 50, no. 3, pp. 597-619, 2017.
  * [32] M. M. Rahman, C. K. Roy, and J. A. Collins, “CoRReCT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Technology Experience,” in Proceedings of the 38th International Conference on Software Engineering (ICSE), pp. 222-231, 2016.
  * [33] S. Rebai, A. Amich, S. Molaei, M. Kessentini, and R. Kazman, “Multi-objective code reviewer recommendations: balancing expertise, availability and collaborations,” Automated Software Engineering, vol. 27, no. 3-4, pp. 301-328, 2020.
  * [34] W. H. A. Al-Zubaidi, P. Thongtanunam, H. K. Dam, C. Tantithamthavorn, and A. Ghose, “Workload-aware reviewer recommendation using a multi-objective search-based approach,” in Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), pp. 21-30, 2020.
  * [35] Z. Xia, H. Sun, J. Jiang, X. Wang, and X. Liu, “A Hybrid Approach to Code Reviewer Recommendation with Collaborative Filtering,” in Proceedings of the 6th International Workshop on Software Mining (SoftwareMining), pp. 24-31, 2017.
  * [36] A. Strand, M. Gunnarson, R. Britto, and M. Usman, “Using a context-aware approach to recommend code reviewers,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 1-10, 2020.
  * [37] A. Chueshev, J. Lawall, R. Bendraou, and T. Ziadi, “Expanding the Number of Reviewers in Open-Source Projects by Recommending Appropriate Developers,” in Proceedings of the 36th IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 499-510, 2020.
  * [38] X. Xia, D. Lo, X. Wang, and X. Yang, “Who Should Review This Change?: Putting Text and File Location Analyses Together for More Accurate Recommendations,” in Proceedings of the 31st IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 261-270, 2015.
  * [39] X. Ye, Y. Zheng, W. Aljedaani, and M. W. Mkaouer, “Recommending pull request reviewers based on code changes,” Soft Computing, vol. 25, no. 7, pp. 5619-5632, 2021.
  * [40] R. Li, P. Liang, P. Avgeriou, ”Code Reviewer Recommendation for Architecture Violations: An Exploratory Study,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 42-51, 2023.
  * [41] J. Jiang, Y. Yang, J. He, X. Blanc, and L. Zhang, “Who should comment on this pull request? Analyzing attributes for more accurate commenter recommendation in pull-based development,” Information and Software Technology, vol. 84, pp. 48-62, 2017.
  * [42] J. Jiang, J.-H. He, and X.-Y. Chen, “CoreDevRec: Automatic Core Member Recommendation for Contribution Evaluation,” Journal of Computer Science and Technology, vol. 30, no. 5, pp. 998-1016, 2015.
  * [43] Z. Ye, Z. Feng, J. Xiao, Y. Gao, G. Fan, H. Zhang, and S. Chen, ”Heterogeneous Graph Neural Network-Based Software Developer Recommendation,” in Proceedings of the 18th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), pp. 433-452, 2022.
  * [44] X. Xie, B. Wang, and X. Yang, “SoftRec: Multi-Relationship Fused Software Developer Recommendation,” Applied Sciences, vol. 10, no. 12, pp. 4333, 2020.
  * [45] Z. Liao, Z. Wu, Y. Li, Y. Zhang, X. Fan, and J. Wu, “Core-reviewer recommendation based on Pull Request topic model and collaborator social network,” Soft Computing, vol. 24, no. 8, pp. 5683-5693, 2019.
  * [46] J. Zhang, C. Maddila, R. Bairi, C. Bird, U. Raizada, A. Agrawal, Y. Jhawar, K. Herzig, and A. van Deursen, “Using Large-scale Heterogeneous Graph Representation Learning for Code Review Recommendations at Microsoft,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 162-172, 2023.


# Nonsequential Two-Photon Double Ionization of Atoms: Identifying the
Mechanism

Morten Førre<EMAIL_ADDRESS>Department of Physics and Technology,
University of Bergen, N-5007 Bergen, Norway  Sølve Selstø  Centre of
Mathematics for Applications, University of Oslo, N-0316 Oslo, Norway  Raymond
Nepstad  Department of Physics and Technology, University of Bergen, N-5007
Bergen, Norway

###### Abstract

We develop an approximate model for the process of direct (nonsequential) two-
photon double ionization of atoms. Employing the model, we calculate
(generalized) total cross sections as well as energy-resolved differential
cross sections of helium for photon energies ranging from 39 to 54 eV. A
comparison with results of ab initio calculations reveals that the agreement
is at a quantitative level. We thus demonstrate that this complex ionization
process is fully described by the simple model, providing insight into the
underlying physical mechanism. Finally, we use the model to calculate
generalized cross sections for the two-photon double ionization of neon in the
nonsequential regime.

###### pacs:

32.80.Rm, 32.80.Fb, 42.50.Hz

Correlated dynamical processes in nature poses unique challenges to
experiments and theory. A prime example of this is the double ionization of
helium by one-photon impact, which has been studied for more than 40 years.
However, it is only during the last 15 years or so, that advances in theory,
modeling and experiment have enabled scientists to gain a deeper insight into
the role of electron correlations in this ionization process Briggs and
Schmidt (2000); Avaldi and Huetz (2005); Samson et al. (1998); Schneider et
al. (2002); Foumouo et al. (2006). The corresponding problem of two-photon
double ionization of helium, in the photon energy interval between 39.4 and
54.4 eV, is an outstanding quantum mechanical problem that has been, and still
is, subject to intense research worldwide, both theoretically Colgan and
Pindzola (2002); Feng and van der Hart (2003); Laulan and Bachau (2003);
Piraux et al. (2003); Hu et al. (2005); Foumouo et al. (2006); Shakeshaft
(2007); Ivanov and Kheifets (2007); Horner et al. (2007); Nikolopoulos and
Lambropoulos (2007); Feist et al. (2008); Guan et al. (2008); Foumouo et al.
(2008); Palacios et al. (2009); Nepstad et al.  and experimentally, employing
state-of-the-art high-order harmonic Hasegawa et al. (2005); Nabekawa et al.
(2005); Antoine et al. (2008) and free-electron (FEL) light sources Sorokin et
al. (2007); Rudenko et al. (2008). Despite all the interest and efforts that
have been put into this research, major fundamental issues remain unresolved.
What characterizes this particular three-body breakup process is that the
electron correlation is a prerequisite for the process to occur, i.e., it
depends upon the exchange of energy between the outgoing electrons, and as
such it represents a clear departure from an independent-particle picture.

In this Letter, we present a novel approximate model for the direct or
nonsequential two-photon double ionization process in helium, sketched in Fig.
1 (a). We show that the simple model predicts the essential features of the
process, even at a quantitative level, which is quite surprising given the
very high complexity of the problem. In particular, we find very good
agreement between the model predictions and the results obtained by solving
the time-dependent Schrödinger equation from first principles, regarding
(generalized) total cross sections as well as energy-resolved differential
cross sections for the process. The proposed model may be generalized to
account for direct double ionization processes in multi-electron atoms. We
demonstrate this by calculating the generalized cross section for
nonsequential two-photon double ionization of neon.

Few-photon multiple ionization of noble gases beyond helium have been studied
experimentally in some detail Moshammer et al. (2007); Sorokin et al. (2007);
Benis et al. (2006); Rudenko et al. (2008), but to the best of our knowledge,
the cross section for the nonsequential two-photon double ionization process
has not yet been obtained. Therefore, we hope that our results will encourage
further investigation of nonsequential double ionization processes in various
noble gases.

Figure 1: (color online). a) Sketch of the direct two-photon double ionization
process in helium. The abbreviation SI and DI stands for single and double
ionization continuum, respectively, whereas the arrows illustrate the photons
that are absorbed by the system. b) Sketch of the model process for two-photon
double ionization (see text for details). c) Matrix representation of the
model Hamiltonian, for the case where the outer electron is emitted before the
inner electron (see text for more details). Atomic units (a.u.) are used in
the figure (1 a.u. of energy corresponds to 27.2 eV).

Reducing a complex quantum mechanical problem to a simple and transparent
model problem, while retaining the essential physics, is very useful in order
to access the underlying physics Lein et al. (2000); Schneider et al. (2002);
Watson et al. (1997). With such a goal in mind, we will now outline a possible
physical mechanism for the nonsequential two-photon double ionization process
in an atom, and then proceed to construct a simple quantum mechanical model
which implements these ideas. The idea behind the model is that the electrons
are considered to be distinguishable particles that can absorb one photon
each. However, in order to include the effect of the first emitted electron on
the second one, we impose the additional but important constraint that the
absorption of the second photon, by the second electron, can only occur after
the first photon absorption. In this way, and according to the principle of
conservation of energy, the first electron may transfer energy to the second
electron as it is emitted, allowing for the nonsequential ionization process
to take place.

The starting point of our model is the single-active electron approximation
(SAE) where both electrons are considered to be independent particles and
treated differently in that they are both assumed to move in their respective
ionization potentials. That is, the ’outer’ electron moves in an effective
potential set up by the nucleus of charge $Ze$ ($e$ is the elementary charge),
the ’inner’ electron and the $Z-2$ other electrons. The inner electron sees a
corresponding screened potential given by the nucleus and the $Z-2$ remaining
electrons. We will label these two different cases simply by ’$A$’ and ’$B$’,
respectively. Following this procedure, the wave function of the ground state
may be approximated by the product ansatz

$\Psi(\mathbf{r}_{A},\mathbf{r}_{B})=\psi_{A}(\mathbf{r}_{A})\psi_{B}(\mathbf{r}_{B}),$
(1)

where $\psi_{A}$ and $\psi_{B}$ refer to the one-electron wave function of
electron $A$ and $B$, respectively.

Now, the first ionization event in the direct two-photon double ionization
process can be represented by the one-electron dipole coupling between the
ground state wave function of either $A$ or $B$, i.e., the state
$\left|A,E_{A}^{0}\right>$ or $\left|B,E_{B}^{0}\right>$, and their respective
continuum states, $\left|A,E_{A}\right>$ and $\left|B,E_{B}\right>$, where
$E_{A}^{0}$ and $E_{B}^{0}$, and $E_{A}$ and $E_{B}$ represent the energies of
the ground and continuum states, respectively. In the product basis
representation (1), with the length gauge formulation of the light-matter
interaction, the dipole coupling matrix elements may be written on the
following simple form (for the case where electron $A$ is assumed to be
emitted first),

$\left<A,E_{A}^{0}\right|-e\mathbf{E}(t)\cdot\mathbf{r}_{A}\left|A,E_{A}\right>\left<\left.B,E_{B}^{0}\right|B,E_{B}^{0}\right>,$
(2)

where $\mathbf{E}(t)$ is the time-dependent electric field that defines the
laser pulse, which is assumed to be linearly polarized along the z-axis. These
coupling elements are related to the one-photon (one-electron) photoionization
cross section via the relation Cormier and Lambropoulos (1995)

$\sigma_{A}=4\pi^{2}\alpha\left(E_{A}-E_{A}^{0}\right)\left|\langle
A,E_{A}^{0}|z_{A}|A,E_{A}\rangle\right|^{2},$ (3)

where $\alpha$ is the fine structure constant.

At the instant of ionization of electron $A$, electron $B$ remains unaffected.
However, once electron $A$ has absorbed its photon, we allow for the
possibility that electron $B$ (but not $A$) can be hit by a second photon.
This secondary process is included into the model by introducing additional
dipole couplings between the $B$ ground state and its corresponding one-
electron continuum states in the following way:

$\left<B,E_{B}^{0}\right|-e\mathbf{E}(t)\cdot\mathbf{r}_{B}\left|B,E_{B}\right>\delta(E_{A},E^{{}^{\prime}}_{A})$
(4)

Note here that there are only non-vanishing couplings between SAE states
(system $A$) of the same energy, i.e., the resulting coupling matrix attains a
very simple structure, as shown in Fig. 1, with typically only a few hundred
different couplings. The same procedure may also be followed with $A$ and $B$
interchanged, however, this will neccessarily yield the same result, and
therefore need not explicitly be considered.

Figure 2: (color online). Total integrated (generalized) cross section for the
nonsequential two-photon double ionization of helium. Black line: present
model result obtained with Eq. (5); open (blue) circles: ab initio result of
Feist et al. Feist et al. (2008) obtained with a 4 fs pulse; and open (red)
squares: corresponding ab initio result of Nepstad et al. Nepstad et al. . The
vertical lines define the two-photon direct double ionization region.

The couplings (2) and (4) and the mentioned constraints, along with the
corresponding diagonal energies, constitute the entire model that we propose.
To this end, we would like to add that all excited, bound states have been
left out of the model, as they play no role in the present context. As a
matter of fact, despite the extremely simple form of the model matrix
elements, with no explicit presence of the correlation potential, it actually
allows for the possibility that the two electrons exchange energy in the
excitation process. Thus, both electrons may be emitted into the continuum
even though the energy of the secondary photon may not itself be sufficient to
eject the inner electron into the continuum.

Applying second order perturbation theory to the resulting model Hamiltonian,
one can show that the single-differential cross section for the direct two-
photon double ionization of an atom is simply given by

$\displaystyle\frac{d\sigma}{dE}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left[f(E)+f\left(2\hbar\omega+E^{0}_{A}+E^{0}_{B}-E\right)\right]$
(5) $\displaystyle f(E)$ $\displaystyle\equiv$
$\displaystyle\frac{\hbar^{3}\omega^{2}}{\pi}\frac{\sigma_{A}\\!\left(E-E^{0}_{A}\right)\;\sigma_{B}\\!\left(2\hbar\omega-E+E^{0}_{A}\right)}{\left(E-E^{0}_{A}\right)\left(2\hbar\omega-E+E^{0}_{A}\right)\left(E-E^{0}_{A}-\hbar\omega\right)^{2}},$

where $\sigma_{A}$ and $\sigma_{B}$ now refer to the total one-photon single
ionization cross sections of $A$ and $B$, respectively, $E$ is the excess
energy, and where we have explicitly accounted for the exchange symmetry of
identical particles and the possibility that either the inner or the outer
electron is emitted first. At this point we would like to emphasize that the
only parameters needed in order to calculate the nonsequential two-photon
double ionization cross section within the model framework, is the effective
binding energies of electron $A$ and $B$, as well as their respective one-
photon single ionization cross sections. For instance, for helium all these
parameters are well known. The model may straightforwardly be generalized to
account for e.g. nonsequential three-photon triple ionization processes in
atoms. A more detailed exposition of the model and a derivation of the
perturbation theory expression for the cross section, will be outlined in a
forthcoming communication.

Figure 3: (color online). Electron energy distribution for two-photon double
ionization of helium at photon energies of 44.9 and 51.7 eV. Solid (black)
line: model result; and dashed (red) line: ab initio result.

In Fig. 2 we compare the total cross section obtained using the approximate
model, Eq. (5), (black line in the figure), with the ab initio result of Feist
et al. Feist et al. (2008) (blue circles) and Nepstad et al. Nepstad et al.
(red squares), both of which were obtained by solving the time-dependent
Schrödinger equation of helium from first principles. The model result is
obtained using tabulated values for the absolute one-photon photoionization
cross section of helium, as obtained experimentally by West and Marr West and
Marr (1976). Figure 3 shows corresponding energy-resolved single-differential
cross sections at two selected photon energies, 44.9 and 51.7 eV. As a matter
of fact, the agreement between the model result and the ab initio results is
almost perfect in Figs. 2 and 3, in particular for the lower photon energies,
demonstrating the strength of this extremely simple model in predicting
accurate values for the generalized cross section in direct two-photon double
ionization processes. Formula (5) predicts a sharp rise of the total cross
section in the vicinity of the threshold at 54.4 eV, which is in agreement
with recent ab initio calculations Horner et al. (2007); Shakeshaft (2007);
Horner et al. (2008); Feist et al. (2008); Palacios et al. (2009); Nepstad et
al. .

As mentioned in the introduction, the problem of nonsequential two-photon
double ionization of helium has been subject of intense research in recent
years, and accurate predictions for the generalized cross section remain
elusive Colgan and Pindzola (2002); Feng and van der Hart (2003); Laulan and
Bachau (2003); Piraux et al. (2003); Hu et al. (2005); Foumouo et al. (2006);
Shakeshaft (2007); Ivanov and Kheifets (2007); Horner et al. (2007);
Nikolopoulos and Lambropoulos (2007); Feist et al. (2008); Guan et al. (2008);
Foumouo et al. (2008); Palacios et al. (2009); Nepstad et al. ; Hasegawa et
al. (2005); Nabekawa et al. (2005); Antoine et al. (2008); Sorokin et al.
(2007); Rudenko et al. (2008) as the values obtained for the cross section for
the reaction may differ by as much as an order of magnitude. On the
theoretical side, the great discrepancies that remain between different
approaches are usually ascribed to the different ways electron correlations
are handled in the final state. To this end, we hope that the predictions of
the present model study may shed new light on this controversy.

Having justified the validity of our simple approach, we now turn to a more
complex problem, namely the process of nonsequential two-photon double
ionization of neon. Inserting, in Eq. (5), the correct first and second
ionization energies of neon, i.e., 21.6 and 40.9 eV, as well as experimental
values for the photoionization cross sections of Ne West and Marr (1976) and
Ne+ Covington et al. (2002), obtained using synchrotron radiation, the
resulting model prediction for the double ionization cross section is shown in
Fig. 4 (upper panel). The lower panel shows the corresponding electron energy
distribution at three selected photon energies. Interestingly, at lower photon
energies, the energy distribution exhibits a maximum (negative concavity) when
both electrons are emitted with the same energy, while at higher photon
energies the distribution is U-shaped. In sharp contrast to this trend, for
helium, the model yields a U-shaped energy distribution for all photon
energies (see Fig. 3).

Figure 4:  Upper panel: generalized total cross section for the process of
two-photon double ionization of neon in the direct regime. The model result is
obtained using Eq. (5), inserting available experimental values for the total
one-photon single ionization cross sections of neon West and Marr (1976) and
Ne+ Covington et al. (2002), respectively. The vertical lines define the two-
photon direct double ionization region. Lower panel: normalized energy
distributions at various photon energies.

In conclusion, we have implemented an approximate and very simple model to
study the two-photon double ionization process of helium in the direct regime,
i.e., at photon energies below 54.4 eV where the sequential ionization process
is energetically inaccessible. We have investigated the validity of the model
by calculating generalized total cross sections and energy-resolved
differential cross sections and compared the model results with corresponding
results obtained by accurate ab initio calculations. Quantitative agreement
between model results and the full results was achieved in all considered
cases, demonstrating the general validity of the model for the two-photon
double ionization process. Finally, we have obtained the cross section for
nonsequential two-photon double ionization of neon, demonstrating that the
model has a great potential to be used in studies of nonsequential multiphoton
multiple ionization processes in more complex atomic systems. This is an
avenue of research we plan to pursue in the future.

###### Acknowledgements.

This work was supported by the Bergen Research Foundation (Norway). The ab
initio calculations were performed on the Cray XT4 (Hexagon) supercomputer
installation at Parallab, University of Bergen (Norway).

## References

  * Briggs and Schmidt (2000) J. S. Briggs and V. Schmidt, J. Phys. B 33, R1 (2000).
  * Avaldi and Huetz (2005) L. Avaldi and A. Huetz, J. Phys. B 38, S861 (2005).
  * Samson et al. (1998) J. A. R. Samson et al., Phys. Rev. A 57, 1906 (1998).
  * Schneider et al. (2002) T. Schneider, P. L. Chocian, and J.-M. Rost, Phys. Rev. Lett. 89, 073002 (2002).
  * Foumouo et al. (2006) E. Foumouo et al., Phys. Rev. A 74, 063409 (2006).
  * Colgan and Pindzola (2002) J. Colgan and M. S. Pindzola, Phys. Rev. Lett. 88, 173002 (2002).
  * Feng and van der Hart (2003) L. Feng and H. W. van der Hart, J. Phys. B 36, L1 (2003).
  * Laulan and Bachau (2003) S. Laulan and H. Bachau, Phys. Rev. A 68, 013409 (2003).
  * Piraux et al. (2003) B. Piraux et al., Eur. Phys. J. D 26, 7 (2003).
  * Hu et al. (2005) S. X. Hu, J. Colgan, and L. A. Collins, J. Phys. B 38, L35 (2005).
  * Shakeshaft (2007) R. Shakeshaft, Phys. Rev. A 76, 063405 (2007).
  * Ivanov and Kheifets (2007) I. A. Ivanov and A. S. Kheifets, Phys. Rev. A 75, 033411 (2007).
  * Horner et al. (2007) D. A. Horner et al., Phys. Rev. A 76, 030701(R) (2007).
  * Nikolopoulos and Lambropoulos (2007) L. A. A. Nikolopoulos and P. Lambropoulos, J. Phys. B 40, 1347 (2007).
  * Feist et al. (2008) J. Feist et al., Phys. Rev. A 77, 043420 (2008).
  * Guan et al. (2008) X. Guan, K. Bartschat, and B. I. Schneider, Phys. Rev. A 77, 043421 (2008).
  * Foumouo et al. (2008) E. Foumouo et al., J. Phys. B 41, 051001 (2008).
  * Palacios et al. (2009) A. Palacios, T. N. Rescigno, and C. W. McCurdy, Phys. Rev. A 79, 033402 (2009).
  * (19) R. Nepstad, T. Birkeland, and M. Førre, to appear in Physical Review A.
  * Hasegawa et al. (2005) H. Hasegawa et al., Phys. Rev. A 71, 023407 (2005).
  * Nabekawa et al. (2005) Y. Nabekawa et al., Phys. Rev. Lett. 94, 043001 (2005).
  * Antoine et al. (2008) P. Antoine et al., Phys. Rev. A 78, 023415 (2008).
  * Sorokin et al. (2007) A. A. Sorokin et al., Phys. Rev. A 75, 051402(R) (2007).
  * Rudenko et al. (2008) A. Rudenko et al., Phys. Rev. Lett. 101, 073003 (2008).
  * Moshammer et al. (2007) R. Moshammer et al., Phys. Rev. Lett. 98, 203001 (2007).
  * Benis et al. (2006) E. P. Benis et al., Phys. Rev. A 74, 051402(R) (2006).
  * Lein et al. (2000) M. Lein, E. K. U. Gross, and V. Engel, Phys. Rev. Lett. 85, 4707 (2000).
  * Watson et al. (1997) J. B. Watson et al., Phys. Rev. Lett. 78, 1884 (1997).
  * Cormier and Lambropoulos (1995) E. Cormier and P. Lambropoulos, J. Phys. B 28, 5043 (1995).
  * West and Marr (1976) J. B. West and G. V. Marr, Proc. R. Soc. Lond. A 349, 397 (1976).
  * Horner et al. (2008) D. A. Horner, T. N. Rescigno, and C. W. McCurdy, Phys. Rev. A 77, 030703(R) (2008).
  * Covington et al. (2002) A. M. Covington et al., Phys. Rev. A 66, 062710 (2002).