# SATDBailiff- Mining and Tracking Self-Admitted Technical Debt

Eman Abdullah AlOmar, Ben Christians, Mihal Busho, Ahmed Hamad AlKhalid, Ali
Ouni, Christian Newman, Mohamed Wiem Mkaouer
{eaa6167,bbc7909,mb5185,aa5130,cdnvse,mwmvse}<EMAIL_ADDRESS>Rochester Institute of Technology, NY, USA
ETS Montreal, University of Quebec, Canada

###### Abstract

Self-Admitted Technical Debt (SATD) is a metaphorical concept to describe the
self-documented addition of technical debt to a software project in the form
of source code comments. SATD can linger in projects and degrade source-code
quality, but it can also be more visible than unintentionally added or
undocumented technical debt. Understanding the implications of adding SATD to
a software project is important because developers can benefit from a better
understanding of the quality trade-offs they are making. However, empirical
studies, analyzing the survivability and removal of SATD comments, are
challenged by potential code changes or SATD comment updates that may
interfere with properly tracking their appearance, existence, and removal. In
this paper, we propose SATDBailiff, a tool that uses an existing state-of-the-
art SATD detection tool, to identify SATD in method comments, then properly
track their lifespan. SATDBailiff is given as input links to open source
projects, and its output is a list of all identified SATDs, and for each
detected SATD, SATDBailiff reports all its associated changes, including any
updates to its text, all the way to reporting its removal. The goal of
SATDBailiff is to aid researchers and practitioners in better tracking SATDs
instances, and providing them with a reliable tool that can be easily
extended. SATDBailiff was validated using a dataset of previously detected and
manually validated SATD instances. SATDBailiff is publicly available as an
open source, along with the manual analysis of SATD instances associated with
its validation, on the project website111https://smilevo.github.io/self-
affirmed-refactoring/SCP20_index.html.

###### keywords:

Self-Admitted Technical Debt, Mining Software Repositories

##  1 Introduction

Technical debt (TD) is a metaphor that describes taking shortcuts in software
development that will require additional time to fix (or payback) in the
future [1]. Technical debt commonly occurs when developers conclude
development of a section of code before it is complete and optimized [2]. This
is done with an understanding that this creates a debt that will need
additional time and effort to be managed later on. Developers tend to
understand some quality-related implications of adding this debt to their
projects, which can be seen unmistakably when looking specifically at Self-
Admitted Technical Debt (SATD) [3].

Self-Admitted Technical Debt is a candid form of technical debt in which the
contributor of the debt self-documents the location of the debt. This
admission is typically accompanied by a description of a known or potential
defect or a statement detailing what remaining work must be done. Well-known
and frequently used examples of SATD include comments beginning with TODO,
FIXME, BUG, XXX, or HACK. SATD can also take other forms of more complex
language void of any of the previously mentioned keywords. Any comment
detailing a not-quite-right implementation present in the surrounding code can
be classified as SATD.

Modern Integrated Development Environments (IDEs) have begun recognizing the
utility of SATD. It is common for them to highlight comments containing the
aforementioned keywords, or for them to add SATD to a project when
automatically generating unimplemented stubbed functionality to be implemented
manually at a later time. Developers and IDEs both contribute SATD with the
common assumption that including a self-admission will make their technical
debt easier to pay back, or at least reduce the likelihood of it being
forgotten. The effectiveness of this strategy needs to be brought into
question, as it may have significant impacts on development practices.
Understanding the implications of this assumption is vital to assure high-
quality performance in software development teams.

Since SATD is a written testament of an existing negative manifestation in the
source code, several studies have observed the removal of SATD instances to
better understand how developers manage and resolve TD. Detecting the removal
of TD is of interest to both researchers and practitioners, as, in addition to
indicating the disappearance of a problem, it indicates the changes containing
the fix to that TD. Such TD fixes are important to locate, since they can be
valuable when taking corrective actions against similar TDs. Several recent
studies have been focused on accurately identifying the removal of SATD. For
instance, Bavota and Russo [4] have shown that up to 57% of SATD is addressed,
and those instances are typically addressed by the same developer who
initially contributed to the SATD. However, the identification of SATD removal
is complex when taking into account common occurrences when code changes
overshadow SATD removals [5], such as the renaming of code elements containing
the instance or the accidental deletion of containing source files. Another
challenge that threatens the correctness of empirical studies related to SATD
is the changes that may interfere with properly tracking the survivability of
SATDs. For example, if the class containing the SATD gets renamed or moved,
without properly handling such refactorings, one SATD may look like it was
deleted, and another one appeared, while it is the same SATD. Similarly, if
the SATD text gets updated, without properly handling such change, this can be
detected by a removal of one SATD instance, and the appearance of another one.
While not all accidental disappearances of SATD comments imply the correction
of any associated technical debt, there is a need for a tool that can reliably
track the appearance and removal of SATD comments, along with any potential
changes associated with the text of the SATD. This effort would provide
valuable support for existing studies by clearly capturing these removals
without the need to manually validate them.

To address the above-mentioned challenge, in this tool paper, we propose
SATDBailiff, a tool based on the existing classification model, called SATD
Detector [6]. This tool is able to (i) mine, identify, and track the
additions, removal, and changes to SATD comments, while providing an overview
of their lifespan in the project, (ii) detect all textual changes associated
with the SATD, detect all changes associated with the class containing the
SATD (moving, renaming) that it underwent throughout the later commits up
until the commit in which each instance was removed, if applicable, and (iii)
allow the integration of SATD detection and classification tools, and (iv)
allow the input of software repository link, and the output of all commits
associated with additions, changes, and deletions of identified SATDs.

SATDBailiff was validated using a dataset of previously detected and manually
validated SATD instances [7]. SATDBailiff is challenged in identifying and
tracking those instances throughout the evolution of five long-lived open-
source projects from different application domains, namely Gerrit, Camel,
Hadoop, Log4j, and Tomcat. The detail of the studied projects is summarized in
Table 1. We manually analyzed the ability of SATDBailiff in correctly
identifying SATD changes and removals (SATD additions were provided by the
dataset). Results show that SATDBailiff is efficient by averaging an accuracy
score of 0.97 when tracking SATD instances from their appearance in the
project until their disappearance.

Tool, documentation, docker and demo video. SATDBailiff is publicly available
as an open source tool222https://github.com/smilevo/SATDBailiff, along with a
continuous integration feature with a docker and a demo video. The raw data
and the manual analysis of SATD instances are also available on the project
website.333https://smilevo.github.io/self-affirmed-
refactoring/SCP20_index.html

The rest of the paper is organized as follows. Section 2 gives an overview of
the necessary information related to SATD and summarizes the related work.
Section 3 describes our approach, SATDBailiff. Section 4 describes how
SATDBailiff can be used in practice while Section 5 shows the applicability of
the tool. Section 6 details the results of our experiments to evaluate
SATDBailiff. In Section 7, we report the tool’s limitation. Section 8
discusses the known threats to validity, while Section 9 draws our conclusions
and future investigations.

Table 1: Details of the studied 5 projects Project |  # Java files |  # SLOC |  File versions |  # Contributors
---|---|---|---|---
Camel444https://github.com/apache/camel |  15,091 |  800,488 |  254,920 |  289
Gerrit555https://github.com/GerritCodeReview/gerrit |  3,059 |  222,476 |  53,298 |  270
Hadoop666https://github.com/apache/hadoop |  8,466 |  996,877 |  79,232 |  160
Log4j777https://github.com/apache/log4j |  1,112 |  30,287 |  12,609 |  35
Tomcat888https://github.com/apache/Tomcat |  3,187 |  297,828 |  46,716 |  32

##  2 Background & Related Work

This paper focuses on mining and tracking SATD instances from Git
repositories. Thus, in this section, we are interested in discussing related
work on SATD. A summary these state-of-the-art studies is depicted in Table 2.

The investigation of Self-Admitted Technical Debt began to gain traction in
2014 with the study of Potdar and Shihab [3]. Initial approaches to
classifying source comments as SATD involved intensive manual efforts. Potdar
and Shihab manually classified 101,762 Java code comments and generated a
string matching heuristic based on 62 commonly occurring comment patterns.
This heuristic inspired Maldonado and Shihab [8] to apply this classification
to ten projects by manually analyzing 33 K code comments. They indicated that
SATD items point to five types of debt: design, requirement, defect, test, and
documentation. Their findings show that design debt is the most common type of
debt. Later, Bavota and Russo [4] replicated the work of Potdar and Shihab and
carried out an empirical study across 159 projects to explore the diffusion
and evolution of SATD and its impact on software quality. Moreover, Wehaibi et
al. [5] investigated the relationship between SATD and quality. They show that
technical debt increases the system’s difficulty and may lead to complex
software changes.

Maldonado et al. [7] expanded the classification approach to use Natural
Language Processing (NLP) to identify design and requirement debts from ten
open source projects. They also reported the top-10 words appearing within
design and requirement SATD-based comments. Despite the increased performance
of classification models, little work has been done to develop an empirical
understanding of SATD in Java projects. One of the empirical studies has been
conducted by Maldonado et al. [9] to analyze SATD comments removal by looking
at the change history of five large open source projects. Results of the study
indicate that there is a high percentage of SATD comments removed and their
survivability varies by project. The quality of this dataset has been brought
into question by Zampetti et al. [10], who manually altered and improved the
dataset in an effort to improve its quality. While this filtered dataset is
regarded to be high quality, the filtering process removed a significant
number of entries, there does not exist a means to expand its size. In their
in-depth investigation of SATD removal, they found that between 20% and 50% of
SATD comments are removed when either the whole class or method is removed.
Another study examined the relationship between refactoring and technical
debt. Lammarino et al. [11] particularly studied the co-occurrence of
refactorings and SATD removals. The authors show that refactorings are more
likely to co-occur with SATD removals than with other commits. Huang et al.
[12] developed a new SATD classification model using text mining. They built a
composite classifier that combined multiple classifiers from eight different
source projects which improved the F1-score of classification over Maldonado
et al. by 27.95%. Liu et al. [6] proposed an Eclipse plugin and Java library
tool called SATD detector to automatically detect SATD using text mining and
highlight the detected comments in an integrated development environment
(IDE).

More recently, Farias et al. [13] improved a set of contextualized patterns or
SATD identification vocabulary built to detect SATD by carrying out three
empirical studies. With regard to the technical debt items detection, their
results show that more than half of the new patterns were considered decisive
or very decisive. In a different work, Zampetti et al. [14] developed SAtd
Removal using DEep LEarning (SARDELE) that highlighted developers’ need to
cope with SATD removal. Their tool is capable of recommending six SATD removal
strategies (e.g., telling that a more complex change is needed). Their
evaluation reveals that SARDELE is able to predict the type of change with an
average precision of 55% and recall of 57%. In another study, Xavier et al.
[15] mined SATD in issue tracker systems. They studied a sample of 286 SATD
instances collected from five open source projects. Although SATD instances
are not more complex in terms of code churn, their findings show that SATD
instances take more time to be closed.

There is now an opportunity to take advantage of the improved detection tool
by Huang et al. [12] to enhance research efforts with a highly accurate, large
scale empirical history of SATD instances in Java projects previously
unavailable. This can be accomplished alongside fixing some of the data
quality issues noted with Maldonado et al.’s [9] empirical study. This study
will aim to package these improvements and model in a tool will allow further
efforts to expand past these seven previously available software projects in
terms of size and quality. In addition to the publication of this tool, an
empirical history of SATD instances in 30 open source software projects will
be made available as produced by SATDBailiff.

Table 2: A summary of the literature on SATD

Study Year Focus Detection technique SATD tool SATD type Project size Potdar &
Shihab [3] 2014 SATD identification Manual analysis N/A Not mentioned 4
Maldonado & Shihab [8] 2015 SATD detection Manual analysis N/A
design/requirement/defect/test 10 documentation Bavota & Russo [4] 2016 SATD
identification Mining-based technique N/A design/requirement/code/test 159
architecture/defect/people/build documentation Wehaibi et al. [5] 2016 Impact
of SATD on quality Defect-based measurement N/A Not mentioned 5 Maldonado et
al. [7] 2017 SATD automatic classification NLP-based technique Maximum entropy
classifier design / requirement 10 Maldonado et al. [9] 2017 SATD removal NLP-
based technique Maldonado et al.’s tool Not mentioned 5 Zampetti et al. [10]
2018 SATD removal Mining-based technique N/A Not mentioned 5 Huang et al. [12]
2018 SATD automatic classification Machine learning technique N/A
design/requirement/defect/test 8 documentation Liu et al. [6] 2018 SATD
detection Mining-based technique SATD Detector design/requirement/defect/test
8 documentation Lammarino et al. [11] 2019 SATD removal Mining-based technique
N/A Not mentioned 4 Farias et al. [13] 2020 SATD identification Contextualized
vocabulary technique Not mentioned design/requirement/code/test 3
architecture/defect/people/build documentation Zampetti et al. [14] 2020 SATD
removal Deep learning technique SARDELE Not mentioned 5 Xavier et al. [15]
2020 SATD identification Mining-based technique Liu et al.’s tool [6]
design/requirement/code/test 5 infrastructure/build/security/UI
documentation/performance

##  3 Overview of SATDBailiff

SATDBailiff is a Java tool designed to mine the empirical history of SATD
instances from Java project Git repositories on a large scale. This is done
with the goal of tracking the additions, removals, and changes to SATD
instances that occur during the process of software development. SATDBailiff’s
output can be used to better understand the prominence of SATD in software
projects at different points over the course of those projects’ lifetimes,
while also offering new ways to interpret and visualize those SATD instances.
While SATDBailiff accomplishes its objective using a state-of-the-art
classification model [6] and a scalable output format, the tool was also
designed with modularity in mind to allow for the use of new classification
models and output formats. To collect its data, the tool leverages several
existing tools as shown in Figure 1.

The SATDBailiff command line interface (CLI) (Seen in Figure 1 as 1) has two
main inputs: A CSV file containing a list of GitHub repositories will be mined
by the tool and a MySQL configuration file. The CLI has many other optional
inputs to optimally configure the tool.

The Eclipse JGit library999https://github.com/eclipse/jgit 3 is used to
collect Java source files from GitHub. This library is also used to collect
any commit metadata available, which is output alongside any associated SATD
operations found during mining. As explained by Nugroho et al. [16], Git
offers a diff utility for users to select various diff algorithms like Myers
[17] and Histogram [18], which are utilized to obtain the differences of two
identical files located in two different commits. Each algorithm has its own
procedures for finding the items presented in the original document but absent
in the second one and vice versa. JGit is also used to generate edit scripts
between different versions of a project’s source code. These edit scripts
detail which lines contain removals and additions, and represent all changes
to a file between one version of a file to the next. Examples of edit scripts
can be seen in Figures 1, 1, 1, and 2. SATDBailiff is configured to use both
the Myers [17] and Histogram [18] difference algorithms to generate edit
scripts and the tool provides the ability to change between these two
algorithms.

JavaParser101010https://github.com/javaparser/javaparser 4 is used to extract
source code comments from the Java source files obtained by JGit 3. It also
extracts comment metadata, containing method and class, and line numbers. This
metadata is output alongside any associated SATD operations found during
mining.

The SATD Detector tool presented by Liu et al. [6] 5 is used for the binary
classification of source comments as SATD. This state-of-the-art tool achieved
an average F-score of 0.737 during the classification of comments from 5 major
open source projects. This classification interface was designed with
modularity in mind, and any future higher-performance models can be used with
SATDBailiff as well.

Figure 1: High-level architecture of SATDBailiff

The logic that bridges all of these tools together is located within the
SATDBailiff client 2. The client begins by generating parent-child pairs for
every single parent commit found under a given head of the git repository.In
this pairing, commits with multiple parents (i.e. merge commits) are not
handled, as the current version of the client handles only one branch. Then,
for each of those pairs, all source code differences (edit scripts) are
calculated for each Java file. All SATD instances are recorded from each file
impacted by a source code modification in the parent commit, as well as the
child commit. A mapping approach is taken to identify which SATD instances may
have been impacted by these changes. An SATD instance will map between two
commits if both commits contain the same comment, under the same method
signature (or lack thereof), and the same containing class name (or lack
thereof). SATD instances that share all of those identification properties
(e.g., two identical SATD comments in the same method), a number is assigned
based on the order they occur. All SATD instances that were not mapped between
the two commits are then classified as removed or changed. This classification
is determined by the edit scripts generated earlier, and the logic is further
described in the Section 3. The result of this process is a complete empirical
history of all operations to SATD instances between a given point in a
project’s lifetime and its origination.

The implicit implementation of SATDBailiff outputs to an SQL database 6, but
the tool supports a modular implementation allowing for an extension of other
output formats. A simplified data-point sample from the Apache Tomcat project
is included in Table 3. The data includes some important features:

  * 1.

SATD Id and SATD Instance Id. Each entry has two identifying integers. The
SATD Id is a unique identifier for a single operation to an SATD Instance. An
SATD Instance ID is an overarching identifier used to group many SATD
operations to a single contiguous instance. In Table 3, each entry in this
sample would have a different and unique SATD Id.

  * 2.

Resolution. Each SATD operation has a single resolution that impacts the SATD
between two commits. These operations include: SATD_ADDED, SATD_REMOVED,
SATD_CHANGED, FILE_REMOVED, FILE_PATH_CHANGED, and CLASS_OR_METHOD_CHANGED.
The definitions of these operations are described in detail in Section 3.

  * 3.

Comment Metadata. When each SATD operation is recorded, SATDBailiff also
records the comment’s metadata at the time of the operation. This includes
data such as the comment type (Line, Block, or JavaDoc as recorded by
JavaParser), start and end line, containing class and method, the file name,
and the comment itself.

  * 4.

Commit Metadata. When each SATD operation is recorded, SATDBailiff also
records the metadata of both the child and parent commit. This includes author
name and timestamp, committer name and timestamp, and SHA1 commit hash.

Table 3: Simplified sample data from the Apache Tomcat project SATD_id |  SATD_instance_id |  resolution |  commit |  comment
---|---|---|---|---
13958 |  652915385 |  SATD_ADDED |  09b640e |  TODO: 404
14317 |  652916048 |  FILE_PATH_CHANGED |  decfe2a |  TODO: 404
13665 |  652915615 |  FILE_REMOVED |  a457153 |  None

Operations on SATD

Previously, Maldonado et al. [9] conducted an empirical study on the removal
of SATD that pointed changes to SATD instance incorrectly as removals and
additions. In addition to mistaking file renames, this would detect instances
like the examples in Figures 1 and 2 as having both resolved the original SATD
instance and added the new version to the project, respectively.

SATDBailiff resolves this issue by handling SATD comment and file name changes
as operations in-between additions and removals of SATD. In order to observe a
more fine-grained change in source code changes, the tool observes edit
scripts for changes to specific lines of code made between each commit. Edit
scripts detail the addition and removal of specific lines of code within a
file. An example edit script can be seen in Figure 1 detailed by the red and
green highlighted source text.

Figure 2: SATD operations captured by SATDBailiff

Figure 2 depicts the SATD operations captured by our tool and Figure 3 shows
the distribution of these operations in our five projects. Our tool captures
the following cases of the SATD removal: (1) when the file has been deleted–
that is, when SATD disappears because the related code is no longer in the
system (i.e., accidental removal), and (2) when SATD comments are removed but
the code still exists. It is worth noting that the purpose of the tool is not
to evaluate the effectiveness of SATD removal or to explore whether SATD is
addressed or not. Our study supports the existing studies that focus on these
aspects by providing more details and explanations related to the evolution of
SATD (e.g., method changed or not, file removed or not, track when SATD starts
and when it is removed, etc). The cases that we plan to capture in the
extended version of the tool are: (1) when the SATD comment is removed
entirely, and a new SATD comment is added instead, although we have not found
any such case based on our manual analysis, (2) when SATD comment has dropped
while the code still remains unchanged, and (3) when SATD comments are removed
and there are changes in the method.

42%24.5%14.2%7.4%6.8%3.5%1.6%SATD_ADDEDSATD_REMOVEDFILE_PATH_CHANGEDFILE_REMOVEDCLASS_OR_METHOD_CHANGEDSATD_CHANGEDSATD_MOVED_FILE
Figure 3: Distribution of SATD operations in the studied 5 projects

The next subsections describe the process of identifying each of the
operations that SATDBailiff handles. The sections use the variables:

  * 1.

$C_{a},C_{b}$, the parent commit and the more recent commit, respectively.

  * 2.

$S_{a}$, $S_{b}$, a specific SATD instance in the parent commit and the more
recent commit, respectively. It should be assumed that $S_{a}$ and $S_{b}$ are
intentionally related to each other if not identical.

  * 3.

$SA_{a},SA_{b}$, an arbitrary other SATD instance in the same file unrelated
to $S_{a}$ or $S_{b}$ in commits $C_{a}$ and $C_{b}$ respectively.

  * 4.

$E_{1},E_{2},...E_{n}$, the edit scripts generated when differencing $C_{a}$
and $C_{b}$ that impact the lines of SATD Comment $S_{1}$. Multi-line SATD
comments may have multiple edit scripts that impact it where $n$ is used to
differentiate these line-based edit scripts.

  1. 1.

SATD_ADDED & SATD_REMOVED

A naive comment differencing algorithm determines SATD_ADDED instances would
exist in any $C_{a}$ where $S_{a}$ is not present and the associated $C_{b}$
where $S_{b}$ is present. This satisfies a basic case in Figure 1.

⬇ body = exchange.getOut().getBody(); \+ // TODO: what if exchange.isFailed()?
if (body != null) {
---

Figure 4: A basic case SATD_ADDED instance

However, SATDBaillif needs to account for changes in SATD comments. In Figure
1, these changes would be identified by separate SATD_REMOVED and SATD_ADDED
instances using the naive logic. Instead, SATDBAllif determines that if a
single edit script $E_{n}$ exists such that $E_{n}$ impacts $S_{b}$ without
impacting $SA_{a}$, then $S_{b}$ was added by $C_{b}$.

A naive comment differencing algorithm also determines SATD_REMOVED instances
would exist in any $C_{b}$ where $S_{b}$ is not present and the associate
$C_{a}$ where $S_{a}$ is present. This satisfies the basic case in Figure 1.

⬇ protected void connectIfNecessary() { \- // can we avoid copy-pasting?  if
(!client.isConnected()) {
---

Figure 5: A basic case SATD_REMOVED instance

However, in the case of Figure 1, it is seen that a more robust algorithm must
be used to detect SATD removals. SATDBailiff handles this case such that if a
single edit script $E_{n}$ exists such that $E_{n}$ impacts $S_{a}$ without
impacting $SA_{b}$, then $S_{a}$ was removed by $C_{b}$.

It can also be the case that the previous logic used by SATDBailiff to
classify additions and removals to be false, and for the tool to still
classify an operation as an SATD_ADDED or SATD_REMOVED. This case is better
identified by the logic in the following SATD_CHANGED section.

⬇ logger.log(\say{Init successful}); \- // Moved this config to the bottom  \+
// Moved this config  \+ // to the bottom  super.init();
---

Figure 6: A case of a would-be false SATD_ADDED and SATD_REMOVED instances

  2. 2.

SATD_CHANGED

Changes in an SATD comment can be difficult to determine because they can
remove the SATD comment entirely or replace it with a new non-SATD, or
irrelevant, comment. Ideally, only SATD comments which preserve the original
intent of the SATD comment should be recorded as a SATD_CHANGED. However, this
requires the comprehension of the comment and its relationship with the TD
described. Any updates on the comment text may be related to the addition or
deletion of more details, as seen later in the manual validation. Since we
cannot assess whether an update to an SATD means partially or totally
addressing it, our tool will report any form of textual change to the SATD, as
SATD_CHANGED. To do so, the edit scripts of the file are first observed. If a
single edit script $E_{n}$ exists such that $E_{n}$ impacts both $S_{a}$ and
$SA_{b}$, then it can be determined that a change may have occurred. Our logic
currently flags as SATD_CHANGED additions of newlines (see Figure 1), spelling
corrections (Figure 2), addition or removal of adjacent related and unrelated
comments, and URL updates.

SATDBailiff also checks the updated comments to determine if they still can be
classified as SATD instances. Figure 2 shows an example of an SATD instance
which is recorded in $C_{a}$ as "lets test the receive worked\nTODO" due to
how the tool groups adjacent comments. In $C_{b}$, the removal of the \nTODO
substring of the instance results in the instance no longer being classified
as SATD, and thus SATDBailiff reports this instance as SATD_REMOVED. Without
this additional verification, this SATD instance would be incorrectly reported
as having only been changed. However, this approach has its own limitation: if
a developer removes one SATD and adds an another entirely unrelated SATD, for
the same method, in the same commit, like in Figure 2, our tool would flag
this as SATD_CHANGED. Fortunately, this case does not seem to be frequent, as
shown later in the manual validation, the precision of our model is up to 96%.
Yet, we report the limitation of our tool in Section7, and we discuss our
current explorations to address it.

⬇ try { \- // Maybe this already exitst \+ // Maybe this already exists
success = client.changeDir(dirName);
---

Figure 7: A valid SATD_CHANGED instance

⬇ c = endpoint.createChannel(session); \- // TODO: what if creation fails?  \+
// Bug 1402  c.connect();
---

Figure 8: A possibly valid but false SATD_CHANGED instance

⬇ // lets test the receive worked \- // TODO  \- //
assertMessageRec("???@localhost");  \+ assertMessageRec("copy@localhost");
c.connect();
---

Figure 9: SATD_REMOVED instance removing only part of a comment

  3. 3.

CLASS_OR_METHOD_CHANGED

The final edit script source code change detected by SATDBailiff is
modification to an SATD instance’s containing class or method. These cases are
detected if any $E_{n}$ impacts the class or method containing $S_{a}$ such
that the method signature or the class name are changed. SATDBailiff does not
currently identify when SATD is moved throughout a file by multiple separate
edit scripts.

  4. 4.

FILE_REMOVED & FILE_PATH_CHANGED

File removals are detected implicitly by Git, where a similarity between added
and removed files determines whether a file is removed or renamed when it is
no longer present in the repository when committed. This detection method was
available as part of the JGit library, and was utilized for identifying
FILE_REMOVED and FILE_PATH_CHANGED instances.

##  4 SATDBailiff Usage

This section describes the usage of SATDBailiff and its features.

###  4.1 Installation

The most up-to-date precompiled binaries can be found on the project’s
GitHub111111https://github.com/smilevo/SATDBailiff repository and on the
tool’s website.121212https://smilevo.github.io/self-affirmed-
refactoring/SCP20_index.html The project can be run using a Java version 8+
and is otherwise OS independent as it could be run through Docker.

###  4.2 Usage

SATDBailiff can be used either through its Command Line Interface (CLI) or
Application Program Interface (API), and can be easily modified to support
different output types and classification models.

The SATDBailiff Command Line Interface (CLI) (Seen in Figure 1 as 1) has two
main inputs: A CSV file containing a list of GitHub repositories and a MySQL
configuration file. The CLI has many other optional inputs to optimally
configure the tool.

The CSV file containing repository information details which repositories will
be mined by the tool, and where the mining will terminate. A terminal commit
value can be added next to the repository URI to add an terminal point in time
to which the tool can mine. This is done primarily to assure reproducibility
between datasets mined at different times. If absent, the terminal commit
value will default to the most recently available commit in the repository.
The output format of the data also allows for a manual filtering of SATD
operations by date. However, it should be noted that un-merged branches of the
repository at certain timestamps are likely to cause difference between a pre-
and post-execution commit filtering. Git credentials for private repositories
can be added as a separate program argument.

By default, SATDBailiff outputs to a MySQL database. The tool intends for a
MySQL database to be set up to receive the system’s output. Configuration
fields for this database must be supplied to the program to connect to the
database. A description for the required fields, as well as the required
schema for the database can be found in the GitHub repository.

Other runtime variables available within SATDBailiff include:

     * (a)

File differencing algorithms available through JGit - currently only Histogram
and Myers;

     * (b)

The Normalized Levenshtein distance threshold (between 0 and 1) described in
Section 3;

     * (c)

A toggle for error display;

     * (d)

A help menu display;

When run, SATDBailiff will display using the interface in Figure 10. This
output includes a detailed description of the runtime duration of the tool,
the number of commits differences, and a description of each entry that caused
any sort of detectable error in the system, if errors display is toggled. If
error display is disabled, then only the number of errors encountered will be
displayed.

        $ java -jar SATDBailiff.jar -d mySQL.properties -r repos.csv
    Completed analyzing 78 diffs in 26,103ms (334.65ms/diff, 0 errors) - analogweb/core
    3wks/thundr -- Mining SATD  (13.3%, 46/346, 0 errors) - e048736

Figure 10: Sample runtime snapshot

###  4.3 Interpreting Output

The released implementation of SATDBailiff outputs to a SQL database. The
schema for the output is detailed in Figure 11. Project table contains the
list of open-source Java projects hosted on GitHub that are utilized in the
study. Commits table stores commit level metadata by looping through the
commit log for each studied project. The metadata includes author name and
timestamp, committer name and timestamp, and SHA1 commit hash. A high level
information about SATD is stored in the SATD table, whereas SATD comment’s
metadata including the comment type, start and end line, containing class and
method, the file name, and the comment itself is recorded in SATDInFile table.

SATDBailiff also generates a CSV file and an HTML file that contain the same
information saved in the database. The CSV file is helpful for results
parsing, while the HTML helps with the visualization of the results.

Figure 11: SATDBailiff output schema

##  5 SATDBailiff Applicability

Tracking SATDs are vital for maintaining healthy software systems and reducing
maintenance cost. As reported by Wehaibi et al. [5], SATD has negative
implications on the software development process since it makes it hard to
perform any future changes. Our tool could serve as a guide; helping
developers pay technical debt by recommending the changes they should perform.
Tracking the additions, removals, and changes to SATD instances that occur
during the process of software development can be used to better understand
the prominence of SATD in software projects at different points over the
course of those projects’ lifetimes. Additionally, tracking SATD instances
would provide valuable support for existing studies by capturing the removals
without the need to manually validate them and will help future research to
explore how SATD is addressed and how SATD removal should be measured (e.g.,
evaluate the effectiveness of the SATD removal when a method has changed along
with the removal). Additionally, our tool can assist in exploring ways to
reduce/manage technical debt. For instance, a recent study [11] shows that
there is a higher chance for refactoring actions to occur together with SATD
removals than with other changes. Another recent study on refactoring
documentation [19, 20, 21] shows that developers reported Fix Technical Debt
in their commit messages which shows that refactoring actions could be used to
cope with technical debt.

Martini et al. [22] conducted an industrial case study with practitioners and
their results show that 25.9 % of the average development time and effort is
spent by 215 practitioners to manage technical debt. In their survey, they
found that some respondents report spending more than 40% of their time
managing technical debt. However, only 26% of the participants used a tool to
track technical debt. The respondents mentioned the tools used to track
technical debt are: comments, documentation, issues, backlog, static analyzer,
lint and test coverage. Backlogs (i.e., Jira, Hansoft, and Excel) are the most
used tool among the participants. According to their survey with the
practitioners, respondents show awareness of indicators of technical debt like
comments or documentation. This sheds light on the need of developing tools to
help with automatically tracking technical debt early in the development
stages. Here are some examples that showcase the usefulness of the tool:

     * (a)

Promoting the adoption of tracking SATD in practice. Fixing bugs or adding new
features would be challenging and a time consuming task as the system evolves,
and software engineers might experience a negative impact on the quality.
Adopting technical debt tracking processes by managers or experienced
developers, who understand the importance of technical debt, reduces
maintenance cost. Our tool is one of the methods that helps promote the
adoption of tracking SATD practice. This feature would be fully achieved with
future refinement of the tool, once we gather more feedback from researchers
and practitioners who are using the tool.

     * (b)

Indicating refactoring technique that removed the SATD. Since the management
of technical debt has been heavily correlated with refactoring, the tool shows
the history of SATDs and the refactoring operations that helped remove the
debt. This assists in analyzing the necessary changes needed to remove
technical debt.

##  6 SATDBailiff Validation

###  6.1 Internal Validation:

To verify the accuracy of SATDBailiff, a manual analysis was performed on a
sample of 1,882 entries mined from 5 large open source Java projects described
in Table 1, each as their own strata. The number of samples taken from each
project is determined by the total number of SATD instances mined from the
projects. Each of the SATD instances selected includes SATD operations
(removed or changed) performed on a single instance of SATD. Each instance in
the sample will represent an entirely unique instance of SATD. A simplified
example of a single SATD instance can be seen in Table 3. The results of this
analysis can be seen in Tables 5 and 6.

For the tool validation, we selected five projects that are widely used by the
state-of-the-art research in the context of SATD [9, 10, 11, 14], as these
projects are claimed to have an adequate number of SATD instances. We build on
top of findings of Maldonado et al. [9] and start from their SATD removal
dataset. In addition to the publication of this tool, an empirical history of
SATD instances in 30 open source software projects, produced by SATDBailiff,
are available on our website.

The detail of the studied 30 projects is summarized in Table 4. Figure 12
shows the distribution of SATD operations captured by our tool in the 30
projects. We observe that the dominant category is the SATD_REMOVED with
77,29%. Our future work involves exploring the technique developers used to
remove SATDs from their projects.

SATDBailiff was configured to only mine SATD instances between the original
commit to each project, and the most recent commit reported by the Maldonado
et al. study.

Table 4: Details of the studied 30 projects. Project | # Commits | # Contributors | # detected SATD
---|---|---|---
gitools/gitools 131313https://github.com/gitools/gitools | 2,051 | 4 | 1,704
twilio/twilio-java 141414https://github.com/twilio/twilio-java | 118 | 3 | 89
hbutani/sqlwindowing 151515https://github.com/hbutani/sqlwindowing | 424 | 2 | 26
pulse00/twig-eclipse-plugin 161616https://github.com/pulse00/twig-eclipse-plugin | 499 | 4 | 359
eclipse/vert.x 171717https://github.com/eclipse/vert.x | 4,950 | 196 | 452
ngdata/lilyproject 181818https://github.com/ngdata/lilyproject | 2,717 | 7 | 3,908
hibernate/hibernate-validator 191919https://github.com/hibernate/hibernate-validator | 4,198 | 78 | 885
wocommunity/wonder 202020https://github.com/wocommunity/wonder | 13,885 | 70 | 2,215
apache/maven-plugins 212121https://github.com/apache/maven-plugins | 15,063 | 36 | 2,300
davemckain/qtiworks 222222https://github.com/davemckain/qtiworks | 2,065 | 2 | 635
icy-imaging/icy-kernel 232323https://github.com/icy-imaging/icy-kernel | 665 | 5 | 978
jfxtras/jfxtras-labs 242424https://github.com/jfxtras/jfxtras-labs | 1,834 | 26 | 3,136
romanchyla/montysolr 252525https://github.com/romanchyla/montysolr | 1,628 | 6 | 927
apache/httpclient 262626https://github.com/apache/httpclient | 3,225 | 51 | 690
socialsoftware/blended-workflow 272727https://github.com/socialsoftware/blended-workflow | 849 | 6 | 824
crosswire/jsword 282828https://github.com/crosswire/jsword | 1,865 | 8 | 1,275
motech/motech-whp 292929https://github.com/motech/motech-whp | 2,190 | 14 | 97
eclipse/bpmn2-modeler 303030https://github.com/eclipse/bpmn2-modeler | 1,442 | 6 | 1,072
eclipse/ecf 313131https://github.com/eclipse/ecf | 11,793 | 11 | 1,559
getrailo/railo 323232https://github.com/getrailo/railo | 3,990 | 15 | 862
msbarry/xtest 333333https://github.com/msbarry/xtest | 341 | 1 | 367
projectdanube/xdi2 343434https://github.com/projectdanube/xdi2 | 2,504 | 3 | 254
scribble/scribble-java 353535https://github.com/scribble/scribble-java | 2,139 | 4 | 6,803
tinkerpop/gremlin 363636https://github.com/tinkerpop/gremlin | 1,227 | 15 | 159
adangel/pmd 373737https://github.com/adangel/pmd | 17,691 | 99 | 3,997
qcadoo/mes 383838https://github.com/qcadoo/mes | 14,906 | 30 | 1,440
belaban/jgroups 393939https://github.com/belaban/jgroups | 19,213 | 72 | 1,193
merks/xcore 404040https://github.com/merks/xcore | 118 | 3 | 89,175
rvonmassow/xdoc 414141https://github.com/rvonmassow/xdoc | 151 | 548 | 6
nasa/certware 424242https://github.com/nasa/certware | 237 | 4 | 1,091
10.12%77.29%5.07%3.03%2.05%2.06%0.40%SATD_ADDEDSATD_REMOVEDFILE_PATH_CHANGEDFILE_REMOVEDCLASS_OR_METHOD_CHANGEDSATD_CHANGEDSATD_MOVED_FILE Figure 12: Distribution of SATD operations in the studied 30 projects. Table 5: SATDBailiff manual analysis results for SATD removal Project |  # Entries |  # False Positive |  Precision
---|---|---|---
Camel |  20 |  1 |  0.99
Gerrit |  284 |  6 |  0.99
Hadoop |  608 |  3 |  0.99
Log4j |  7 |  0 |  1.00
Tomcat |  432 |  8 |  0.99
Total |  1351 |  18 |  0.99
Table 6: SATDBailiff manual analysis results for SATD changed Project |  # Entries |  # False Positive |  Precision
---|---|---|---
Camel |  19 |  2 |  0.99
Gerrit |  223 |  25 |  0.92
Hadoop |  112 |  13 |  0.96
Log4j |  9 |  0 |  1.00
Tomcat |  168 |  24 |  0.93
Total |  531 |  64 |  0.96

⬇ protected void processSoapConsumerOut (Exchange exchange) throws Exception {
LOG.info("processSoapConsumerOut:" + exchange); \- // TODO  \+ // TODO check
if the message is oneway message // Get the method name form the soap endpoint
... }
---

Figure 13: An expansion case of SATD_CHANGED instance

⬇ public void onExchange(HttpExchange exchange) { \- // we need an external
HTTP client such as commons-httpclient // TODO \+ // TODO }
---

Figure 14: An abbreviation case of SATD_CHANGED instance

⬇ \- //TODO: YARN-3284 //The containerLocality metrics will be exposed from
AttemptReport \+ // TODO:YARN-3284 private void createContainerLocalityTable
(Block html) { ... }
---

Figure 15: A generalization case of SATD_CHANGED instance

⬇ \- // TODO fix this test \+ // TODO fix this test, it looks like AMQP don’t
support Object message public void xtestJmsRouteWithObjectMessage() throws
Exception { ... }
---

Figure 16: A specialization case of SATD_CHANGED instance

⬇ public void afterPropertiesSet() { \- // TODO: is needed when we add support
for when predicate if (getOutputs().size() == 0) { \+ // no outputs return; }
---

Figure 17: A case of SATD_REMOVED instance when there is no overlap

To perform the analysis, one of the authors was given the set of SATD
instances and asked to locate the exact location of each of the SATD
operations using the GitHub website. A correct entry was identified as an
entry in which every operation made to the SATD instance could be located
using the GitHub interface. Any unnecessary additional, missing, or inaccurate
operations found on GitHub would result in the entire entry being incorrect.
For entries that were not removed from the project, their existence in the
terminal commit supplied to SATDBailiff was confirmed. For transparency of
this analysis, a GitHub link to the exact source modification was recorded in
each of the projects where available. These results are available on the
project’s website.434343https://smilevo.github.io/self-affirmed-
refactoring/SCP20_index.html During validation, it was assumed that all binary
classifications of source code comments as SATD were correct.

The results of the manual analysis (Tables 5 and 6) find SATDBailiff to have a
precision of 0.99 and 0.96 for SATD removal and SATD changes, respectively. It
is infeasible to always achieve a perfect precision of the results extracted
from the tool due to the inconsistent nature of open source projects and their
development practices. Open source projects are varied in size, contributors,
number of comments, and SATDs. Thus, the performance of the tool might differ
based on development practices of the selected projects. While a higher level
of accuracy could have been achieved, it should be noted that many of the
incorrect instances were partially correct. For example, instances frequently
were found to be incorrect because they became dissociated with one another,
where a connection between an SATD instance’s addition to the project and its
deletion from the project was not made by the tool. In cases where only the
additions or removals are observed from the dataset, the accuracy of the data
provided by the tool is much more reliable.

Difficulties in solving many of the tool’s issues came from the imperfect
nature of working with edit scripts produced by Git differencing tools. Edit
scripts are used to show an algorithm’s best guess of changes in files inside
of a Git repository, and do not always reflect the true intentions of the
developer who made them [23]. An example of an edit script can be seen in
Figure 2 depicted as the red and green highlight used to represent a source
code change. JGit uses the Myers [17] and the Histogram [18] diff algorithm to
produce edit scripts. SATDBailiff provides the ability to change between these
two algorithms, and the Myers algorithm was used during performed manual
validation. Both of these algorithms maintain a manually validated accuracy of
less than 0.9 [23]. While, in many cases, an invalid edit script will not
directly invalidate SATDBailiff’s ability to identify operations to SATD
instances, this inaccuracy still serves as a significant limitation in the
upper bound of accuracy achievable by this tool.

To assure that these errors are not able to silently pollute the dataset, the
tool reports any known errors that are encountered during the mining process.
This workaround was taken as an optimistic precaution for an issue that may
not have a perfect alternative solution. For example, SATD that is added
during a merge commit which was not present in either of the merge branches is
not detected with an SATD_ADDED entry. If that SATD is modified or removed
later, then the entry would be added to the project before the SATD_ADDED
entry was found. Because the search occurs chronologically starting with the
oldest commit in the project, the system can detect this as an issue and will
output an error to the terminal during runtime.

When performing the manual validation, we noticed that developers changed SATD
comments as follows:

     * (a)

An expansion form of SATD comment: a comment that has more explanation of the
SATD case. For instance, in Figure 6.1, the comment contains more than just
the TODO tag, as the developer added a couple of functionalities to implement,
as part of the TODO.

     * (b)

An abbreviation form of SATD comment: the opposite of the expansion in which a
shortened form of SATD comment is provided. For instance, in Figure 6.1, the
text explaining the task, tagged with the TODO, has been removed, however, the
TODO has been kept, and therefore, the comment is still flagged as SATD.

     * (c)

A generalization form of SATD comment: a comment has broad SATD context. In
Figure 6.1, the details of the TODO have been removed, and the reference to
the active issue has been kept.

     * (d)

A specialization form of SATD comment: the opposite of the generalization in
which a comment has focused SATD context. For example, in Figure 6.1, details
on how to fix the test and make it support the Object message, have been
added.

As for the SATD removal case, we noticed the case in which there is entirely
no overlap between the SATD comments as shown in Figure 6.1.

###  6.2 External Validation:

To further assess the usefulness of the tool, we perform an external
validation by involving 15 graduate students from the Department of Software
Engineering and Data Science at Rochester Institute of Technology. All
participants volunteered to participate in the experiment. For each
participant, the experiment consists of (1) randomly choosing an Java-based
open-source project and fork it, (2) detecting 10 issues in the open-source
project, using PMD and/or SpotBugs, (3) writing a comment, describing the
issue detected inside the method, while making sure the comment is written in
the form of SATD, then committing it (4) addressing the issues one by one by
making the necessary code changes recommended by PMD and/or SpotBugs, while
making sure to remove the comment describing it, then committing the changes
(5) running SATDBailiff tool to identify these added and removed comments, and
finally (6) filling out the survey to give feedback about the tool. The idea
behind this experiment is to identify potential TD instances, and document
their existence, then addressing them and removing their corresponding
comments. We used PMD and SpotBugs because they are known to be good
indicators for technical debt in the source code [24, 25]. Upon the completion
of the experiment, we calculated the average recall, and we found the average
recall is 0.90, which is considered acceptable. The replication package for
the whole experiment is available in 444444https://smilevo.github.io/self-
affirmed-refactoring/SCP20_index.html.

The survey consisted of 6 questions that are divided into 2 parts. The first
part of the survey includes demographics questions about the participants. In
the second part, we asked about the (1) satisfaction of the tool aspects, (2)
preferred features of the tool, (3) suggested features of the tool to be added
in the future, and (4) any comments about the tool. We constructed the survey
to use 5-point ordered response scale (“Likert scale”) question on the aspects
of the tool, and 3 open-ended questions on the preferred features, suggested
features, and general comments about the tool.

As shown in Figure 18, the experience of these participants with programming
ranged from less than a year to more than 5 years. As for the familiarity with
the concept of technical debt, 66.7% of the participants are familiar with the
technical debt, whereas 33.3% are not familiar with the concept of technical
debt. Prior to the execution process, the participants were provided with a
75-minute tutorial on technical debt along with reference materials.

Figure 19 presents an overview of the aspects of the tool and the satisfaction
of the participants. With respect to tool setup, some of the respondents
reported that they are satisfied with the tool. However, there are a few
participants who are usually not happy with how the tool is set up and we are
planning on improving the tool setup in the future. For the easy to use
aspect, a larger group (10 participants) selected neutral. Some of the
participants found that the tool is not easy to use, so we will work on
improving the usability of the tool. The main feedback from the participants
is to add GUI features, and we think that GUI might be an option to be
considered. Adding GUI is not one of our priorities because we have more
features that we would like to improve, but we will consider adding it in our
future extension of the tool. Regarding the tool documentation, the majority
of the respondents agreed that the documentation is useful; only 2
participants are somewhat unsatisfied. Concerning the execution time, the
participants are happy with it, although we don’t have a consensus between
participants and we believe that execution time depends on the selected
project size. For the format of the input, the vast majority of respondents
agree that the format is acceptable. Likewise, the participants are happy with
the format of the output.

With respect to the preferred features of the tools, the participants listed a
variety of features which are centered around six main topics: (1) tool
correctness, (2) tool accuracy, (3) tool documentation, (4) execution time,
(5) data storage, and (6) Docker integration. We demonstrated some of the
responses:

> The tool performed more smartly and accurately than I had anticipated, since
> one of my SATD resolving commits was actually in 2 parts because its was
> long, I assumed that would cause some inaccuracy and confusion in the tool.
> But surprising and delightfully the tool identified and clubbed the 2
> commits and also connected to the reported SATD beautifully.
>
> The integration with Docker allowed the tool to be setup very easily on my
> system so I would have to say that ’addon’ was fantastic. I found myself
> struggling to 1) Build the tool from the source code and 2) running the
> executable jar. Once I figured out how to download/install the image
> everything began to go smoothly for me. That integration is not necessarily
> a feature of the tool itself, so in that regard I would say I really liked
> the reports generated. The html + csv output were well designed and labels
> were intuitive. On top of that, I didn’t have to do anything crazy to get
> the tool to run on my repo, just changed the repo url in the csv file which
> was quite simple. I appreciated that about the tool.
>
> I liked how fast are the SATD mining process and the correctness of the
> output
>
> I like how the tool is analyze the comments with high accuracy of detect
> each comment that was add or removed .Also , the way of the data is stored ,
> the user have 3 options to analyze the data excel , html report and Sql
> database. In MySql database there are multiple tables with different
> information the user can create customize report based on his requirements.

As for the suggested features of the tool, participants mainly mentioned 3
main features: (1) IDE integration, (2) issue troubleshooting , and (3)
specifying the starting and ending commits. Participants did explicitly share
their concerns during the survey as follows:

I feel that a frontend GUI that would help feeding in the data and also a
simple GUI that did the MYSQL query in the backend and for us it would just be
a download button would make the tool much more friendly and intuitive to
use.” and “A GUI would be good as everything right now is terminal based.

Add as a plugin and add more details about how to use the tool if a particular
error is found.

I think having a way to choose a starting and ending point would be great

26.7%53.3%13.3%6.7%less than 1 year1 - 3 years3 - 5 yearsmore than 5 years
Figure 18: Participant programming experience in years.
$0$$20$$40$$60$$80$$100$Tool setupEase of useTool documentationExecution
timeFormat of inputFormat of outputTime in msVery SatisfiedSomewhat
SatisfiedNeutralSomewhat UnsatisfiedVery Unsatisfied Figure 19: Level of
satisfaction with the aspects of SATDBailiff tool.

##  7 SATDBailiff Limitations and Upcoming Features

In this section, we outline the known limitations to our tool, and the
features we are planning on developing.

     * (a)

As pointed out in Section 3, our current tool does not track comments
introduced by commits with multiple parents. We have not yet performed any
manual analysis to assess the extent to which this affects the tracking of
current comments. In the future, we plan to perform a thorough examination to
determine whether we should prioritize handling multiple parents.

     * (b)

To reduce the false positiveness of our tool, as pointed out by the
experiments, we are currently exploring various text processing techniques to
preprocess the comments and reduce the effects of special characters that may
interfere with our string matching.

     * (c)

We are also investigating the use of pre-trained models that can detect
whether comments are pointing out to the same technical debt. Such a model can
help us avoid detecting two different comments as being the same one, in the
case of removing one SATD and adding another one, for the same method, and the
same commit.

     * (d)

Since the management of technical debt has been heavily correlated with
refactoring, we have already integrated the Refactoring Miner library, through
its API, to run on the commits that we are also detecting the removal of SATD
in. This will support existing and future studies that analyze the necessary
changes needed to remove technical debt. Our tool currently reports all the
refactorings that are associated with the classes and methods containing SATD
comments. This is a recent feature that we are still testing, therefore we do
not have any observations about it in this paper. We anticipate that this
feature will help researchers in developing empirical evidence of the
usefulness of refactoring in terms of managing technical debt.

     * (e)

Upon performing the external validation, we found that the participants
pointed out limitations that are mainly related to tool setup and usage. In
the future, we plan to improve the setup and the usability of the tool by
adding several features mentioned by the participants, including but not
limited to, IDE integration, issue troubleshooting, and specifying the
starting and ending commits. Further, the external validation is currently
conducted with a group of software engineering students. As future work, we
plan to perform another round of external validation with professional
software engineers in industry to hear their perception.

##  8 Threats to Validity

In this section, we identify potential threats to the validity of our approach
and our experiments.

Threats to the validity of this tool include the limited manual evaluation and
general lack of testing. An important potential threat relates to our manual
classification. Since the manual verification of samples is a human intensive
task and it is subject to personal bias, we mitigate this first by selecting
SATD instances from an existing dataset. Then we performed the tracking of
their existence by one author. Only 200 samples of SATD were recorded and
addressed to determine the accuracy of SATDBailiff. Ideally, this number
should be much higher. There was limited time available for other formal and
repeatable forms of programmatic testing, as a majority of the validation
effort was allocated to the manual validation and validation done during
development.

Another threat relates to the SATD instances that are extracted only from open
source Java projects. Our results may not generalize to commercially developed
projects, or to other projects using different programming languages. Another
threat concerns the generalization of SATD patterns used in this study. Since
a method is considered holder of TD when a comment contains SATD, this may not
generalize to other projects if they do not allow inline documentation
(comments).

The reliance on the existing tool (i.e., SATD Detector) is another threat to
validity due to the possibility of introducing false positive results. Thus,
the binary classification of SATD comments resulting from the SATD Detector
might have an impact on our findings.

##  9 Conclusion and Future Work

This paper presents a preview of our tool, SATDBailiff, and discusses its
benefits. The tool aims at offering an unmatched ability to extract SATD
instances and the development operations upon them from Git repositories. This
paper discussed the benefits that a high quality empirical history would have
for the further study of SATD.

An acceptable level of accuracy was achieved with SATDBailiff, however there
is still opportunity for improvement. In their nature, source code
differencing tools may not always record modifications that reflect the true
nature of a developer’s intentions [26]. It is because of these inaccuracies
that solving each and every edge case is infeasible for the purpose of data
generation. Some attempts to fix these edge-cases were made, however their
large numbers and unpredictability made it a relatively futile task. To
compensate, a list of known edge cases are included on the tool’s website to
detail known areas where inaccuracies may appear. As Edit Scripts are more
reliably instantiated from these differences, more accurate detection of SATD-
impacting operations will be possible.

Edit scripts were generated for this tool using the Histogram and Myers
algorithms made available through JGit. Using a differencing tool like GumTree
[26] or a hybrid approach [27] may produce more accurate results. However,
GumTree currently does not offer the tracking of comment changes and does not
plan on implementing that
functionality.454545https://github.com/GumTreeDiff/gumtree/issues/39
Modification of the edit script generation methodology may also have positive
impacts on the project’s runtime which may currently be excessive for larger
project.

Some other issues encountered include the handling of non-English language
source comments. None of the projects used to validate SATDBailiff contained
any known instances of these comments, however the larger dataset of 691
projects released alongside the tool does contain these instances. Ideally, a
tool would be able to detect a non-English comment before classifying it as
SATD, but no attempt was made to solve this issue as it only represents a
small subset of the addressed projects.

## References

     * [1] P. Kruchten, R. L. Nord, I. Ozkaya, Technical debt: From metaphor to theory and practice, IEEE Software 29 (6) (2012) 18–21.
     * [2] W. Cunningham, The wycash portfolio management system, ACM SIGPLAN OOPS Messenger 4 (2) (1992) 29–30.
     * [3] A. Potdar, E. Shihab, An exploratory study on self-admitted technical debt, in: 2014 IEEE International Conference on Software Maintenance and Evolution, IEEE, 2014, pp. 91–100.
     * [4] G. Bavota, B. Russo, A large-scale empirical study on self-admitted technical debt, in: Proceedings of the 13th International Conference on Mining Software Repositories, 2016, pp. 315–326.
     * [5] S. Wehaibi, E. Shihab, L. Guerrouj, Examining the impact of self-admitted technical debt on software quality, in: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1, IEEE, 2016, pp. 179–188.
     * [6] Z. Liu, Q. Huang, X. Xia, E. Shihab, D. Lo, S. Li, Satd detector: A text-mining-based self-admitted technical debt detection tool, in: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, 2018, pp. 9–12.
     * [7] E. d. S. Maldonado, E. Shihab, N. Tsantalis, Using natural language processing to automatically detect self-admitted technical debt, IEEE Transactions on Software Engineering 43 (11) (2017) 1044–1062.
     * [8] E. d. S. Maldonado, E. Shihab, Detecting and quantifying different types of self-admitted technical debt, in: 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 9–15.
     * [9] E. D. S. Maldonado, R. Abdalkareem, E. Shihab, A. Serebrenik, An empirical study on the removal of self-admitted technical debt, in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2017, pp. 238–248.
     * [10] F. Zampetti, A. Serebrenik, M. Di Penta, Was self-admitted technical debt removal a real removal? an in-depth perspective, in: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), 2018, pp. 526–536.
     * [11] M. Iammarino, F. Zampetti, L. Aversano, M. Di Penta, Self-admitted technical debt removal and refactoring actions: Co-occurrence or more?, in: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2019, pp. 186–190.
     * [12] Q. Huang, E. Shihab, X. Xia, D. Lo, S. Li, Identifying self-admitted technical debt in open source projects using text mining, Empirical Software Engineering 23 (05 2017).  doi:10.1007/s10664-017-9522-4.
     * [13] M. A. de Freitas Farias, M. G. de Mendonça Neto, M. Kalinowski, R. O. Spínola, Identifying self-admitted technical debt through code comment analysis with a contextualized vocabulary, Information and Software Technology 121 (2020) 106270.
     * [14] F. Zampetti, A. Serebrenik, M. Di Penta, Automatically learning patterns for self-admitted technical debt removal, in: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2020, pp. 355–366.
     * [15] L. Xavier, F. Ferreira, R. Brito, M. T. Valente, Beyond the code: Mining self-admitted technical debt in issue tracker systems, arXiv preprint arXiv:2003.09418 (2020).
     * [16] Y. S. Nugroho, H. Hata, K. Matsumoto, How different are different diff algorithms in git?, Empirical Software Engineering 25 (1) (2020) 790–823.
     * [17] E. W. Myers, An o(nd) difference algorithm and its variations, Algorithmica 1 (1986) 251–266.
     * [18] B. Cohen, https://alfedenzo.livejournal.com/170301.html.
     * [19] E. A. AlOmar, M. W. Mkaouer, A. Ouni, Can refactoring be self-affirmed? an exploratory study on how developers document their refactoring activities in commit messages, in: Proceedings of the 3nd International Workshop on Refactoring. IEEE, 2019.
     * [20] E. A. AlOmar, M. W. Mkaouer, A. Ouni, Toward the automatic classification of self-affirmed refactoring, Journal of Systems and Software (2020) 110821.
     * [21] E. A. AlOmar, A. Peruma, M. W. Mkaouer, C. Newman, A. Ouni, M. Kessentini, How we refactor and how we document it? on the use of supervised machine learning algorithms to classify refactoring documentation, Expert Systems with Applications 167 (2021) 114176.
     * [22] A. Martini, T. Besker, J. Bosch, Technical debt tracking: Current state of practice: A survey and multiple case study in 15 large organizations, Science of Computer Programming 163 (2018) 42–61.
     * [23] V. Frick, T. Grassauer, F. Beck, M. Pinzger, Generating accurate and compact edit scripts using tree differencing, in: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018, pp. 264–274.
     * [24] F. Zampetti, C. Noiseux, G. Antoniol, F. Khomh, M. Di Penta, Recommending when design technical debt should be self-admitted, in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2017, pp. 216–226.
     * [25] T. Amanatidis, N. Mittas, A. Moschou, A. Chatzigeorgiou, A. Ampatzoglou, L. Angelis, Evaluating the agreement among technical debt measurement tools: building an empirical benchmark of technical debt liabilities, Empirical Software Engineering 25 (5) (2020) 4161–4204.
     * [26] J. Falleri, F. Morandat, X. Blanc, M. Martinez, M. Monperrus, Fine-grained and accurate source code differencing, in: ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014, 2014, pp. 313–324.
URL http://doi.acm.org/10.1145/2642937.2642982

     * [27] J. Matsumoto, Y. Higo, S. Kusumoto, Beyond gumtree: A hybrid approach to generate edit scripts, in: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 2019, pp. 550–554.


#  Low-energy limit of N-photon amplitudes
in a constant field

Naser Ahmadiniaz  Misha A. Lopez-Lopez  Christian Schubert  Helmholtz-Zentrum
Dresden-Rossendorf, Bautzner Landstraße 400, 01328 Dresden, Germany  Institut
für Theoretische Physik, Technische Universität Dresden, 01062 Dresden,
Germany  Facultad de Ciencias Físico-Matemáticas, Universidad Michoacana de
San Nicolás de Hidalgo, Avenida Francisco J. Mújica, 58060 Morelia, Michoacán,
Mexico

###### Abstract

While the QED photon amplitudes at full momentum so far have been calculated
only up to the six-photon level, in the low-energy limit there are explicit
formulas for all helicity components even at the $N$-photon level, obtained by
Martin et al. in 2002. Here we use the worldline formalism to extend that
result to the $N$-photon amplitudes in a generic constant field. For both
scalar and spinor QED, we obtain compact representations for the low-energy
limits of these amplitudes involving only simple algebra and a single global
proper-time integral with trigonometric integrand.

###### keywords:

worldline formalism , photon amplitudes , low energy , constant field

##  1 Introduction

Although the QED photon amplitudes are prototypical for all gauge-boson
correlators and have been studied for almost a century [1, 2, 3, 4, 5, 6, 7]
(for reviews, see [8, 9, 10]), for arbitrary momenta results exist only up to
the six-point level [11]. Only in the low-energy limit, where all photon
energies are small compared to the electron mass, has it been feasible to
derive explicit expressions for all helicity components even at the $N$-photon
level [12]. In the present letter, we generalize these formulas to the
inclusion of a generic constant external field. While in [12] these amplitudes
were derived from the Euler-Heisenberg Lagrangian [2] and its scalar QED
analogue, the Weisskopf Lagrangian [3], for the purpose of the constant-field
generalization we find it more convenient to proceed along the lines of a
later rederivation [13] based on a direct amplitude calculation using the
worldline formalism [14, 15, 16, 17, 18, 19, 20, 21]. We will thus start with
a short review of the calculation of general photon amplitudes in that
formalism, first in vacuum and then in the constant-field background. We then
proceed to the calculation of their low-energy limits, first for scalar and
then for spinor QED.

##  2 Worldline representation of the QED $N$ \- photon amplitudes

We start with a short summary of the computation of photon amplitudes in the
worldline formalism (for details, see [20, 22]). In scalar QED, the formalism
leads to the following path-integral representation of the one-loop $N$ \-
photon amplitude [18],

$\displaystyle\Gamma_{\rm
scal}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N})$ $\displaystyle=$
$\displaystyle(-ie)^{N}\int_{0}^{\infty}\frac{dT}{T}\,\,{\rm
e}^{-m^{2}T}\int_{x(0)=x(T)}Dx\,{\rm
e}^{-\int_{0}^{T}\\!d\tau\frac{\dot{x}^{2}}{4}}$ (1) $\displaystyle\times
V^{\gamma}_{\rm scal}[k_{1},\varepsilon_{1}]\cdots V^{\gamma}_{\rm
scal}[k_{N},\varepsilon_{N}]\;.$

Here $m$ is the mass of the loop scalar, the path integral runs over all
closed loops of periodicity $T$ in (euclidean) spacetime, and each of the
photons is represented by a vertex operator

$\displaystyle V^{\gamma}_{\rm
scal}[k,\varepsilon]=\int_{0}^{T}d\tau\,\varepsilon\cdot\dot{x}\,{\rm
e}^{ik\cdot x}=\int_{0}^{T}d\tau\,\,{\rm e}^{ik\cdot
x+\varepsilon\cdot\dot{x}}\Bigl{|}_{\varepsilon}\,,$ (2)

where the polarisation vector $\varepsilon^{\mu}$ need not necessarily obey
on-shell conditions. A formal gaussian integration of the path integral in (1)
leads to the following master formula for this amplitude [16, 17, 18]:

$\displaystyle\Gamma_{\rm
scal}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N})$ $\displaystyle=$
$\displaystyle{(-ie)}^{N}{\displaystyle\int_{0}^{\infty}}\frac{dT}{T}{(4\pi
T)}^{-\frac{D}{2}}\,{\rm e}^{-m^{2}T}$ (3)
$\displaystyle\times\prod_{i=1}^{N}\int_{0}^{T}d\tau_{i}\exp\biggl{\\{}\sum_{i,j=1}^{N}\Bigl{[}\frac{1}{2}G_{ij}k_{i}\cdot
k_{j}-i\dot{G}_{ij}\varepsilon_{i}\cdot
k_{j}+\frac{1}{2}\ddot{G}_{ij}\varepsilon_{i}\cdot\varepsilon_{j}\Bigr{]}\biggr{\\}}\Bigl{|}_{\varepsilon_{1}\varepsilon_{2}\ldots\varepsilon_{N}}\,.$

Here we have abbreviated $G_{ij}\equiv G(\tau_{i},\tau_{j})$ etc., where $G$
is the “bosonic worldline Green’s function” defined by

$\displaystyle
G(\tau,\tau^{\prime})\equiv\mid\tau-\tau^{\prime}\mid-\frac{(\tau-\tau^{\prime})^{2}}{T}\,,$
(4)

and a ‘dot’ denotes a derivative acting on the first variable,

$\displaystyle\dot{G}(\tau,\tau^{\prime})$ $\displaystyle=$ $\displaystyle{\rm
sgn}(\tau-\tau^{\prime})-2\frac{(\tau-\tau^{\prime})}{T},\quad\ddot{G}(\tau,\tau^{\prime})=2{\delta}(\tau-\tau^{\prime})-\frac{2}{T}\,.$
(5)

The exponential must still be expanded and only the terms be retained that
contain each polarisation vector $\varepsilon_{i}$ linearly:

$\displaystyle\exp\bigl{\\{}\cdot\bigr{\\}}\bigl{|}_{\varepsilon_{1}\varepsilon_{2}\ldots\varepsilon_{N}}\quad\equiv{(-i)}^{N}P_{N}(\dot{G}_{ij},\ddot{G}_{ij})\exp\biggl{[}\frac{1}{2}\sum_{i,j=1}^{N}G_{ij}k_{i}\cdot
k_{j}\biggr{]}\,,$ (6)

with certain polynomials $P_{N}$.

The generalization to the spinor-loop case in the modern approach is done
through the addition of a Grassmann path integral, to be evaluated with the
“fermionic worldline Green’s function” $G_{F}(\tau,\tau^{\prime})\equiv{\rm
sgn}(\tau-\tau^{\prime})$ [23, 16, 17, 18]. Moreover, these spin-induced terms
can be conveniently generated by the following “Bern-Kosower loop-replacement”
rule: after removing all second derivatives $\ddot{G}_{ij}$ from the
polynomial $P_{N}$ by suitable partial integrations, the integrand for the
spinor loop case can be obtained by simultaneously replacing every
“$\tau$-cycle”
$\dot{G}_{i_{1}i_{2}}\dot{G}_{i_{2}i_{3}}\cdots\dot{G}_{i_{n}i_{1}}$ by the
corresponding “super $\tau$-cycle”
$\dot{G}_{i_{1}i_{2}}\dot{G}_{i_{2}i_{3}}\cdots\dot{G}_{i_{n}i_{1}}-G_{Fi_{1}i_{2}}G_{Fi_{2}i_{3}}\cdots
G_{Fi_{n}i_{1}}$ (and multiplying the whole amplitude by a factor of $-2$ for
the difference in statistics and number of degrees of freedom).

Subsequently, it emerged that this whole procedure can be generalized to the
inclusion of a constant external field in a very economical way, namely by a
modification of the path-integral determinant [19] and the introduction of
generalized worldline Green’s functions ${\cal G}_{B},{\cal G}_{F}$ that take
the external field into account [27, 28]

$\displaystyle{\cal G}_{B}(\tau,\tau^{\prime})$ $\displaystyle\equiv$
$\displaystyle\frac{T}{2{\cal Z}^{2}}\biggl{(}{{\cal Z}\over{{\rm sin}{\cal
Z}}}\,{\rm e}^{-i{\cal Z}\dot{G}(\tau,\tau^{\prime})}+i{\cal
Z}\dot{G}(\tau,\tau^{\prime})-1\biggr{)}\,,$ (7) $\displaystyle{\cal
G}_{F}(\tau,\tau^{\prime})$ $\displaystyle\equiv$ $\displaystyle
G_{F}(\tau,\tau^{\prime}){{\rm e}^{-i{\cal
Z}\dot{G}(\tau,\tau^{\prime})}\over{\rm cos}{\cal Z}}\,,$ (8)

where ${\cal Z}_{\mu\nu}\equiv eF_{\mu\nu}T$. These expressions for the
constant-field Green’s functions should be understood as power series in the
Lorentz matrix $\cal Z$. The master formula for the photon amplitudes in
vacuum (3) can then be generalized to the constant field case as follows [27,
28],

$\displaystyle\Gamma_{\rm
scal}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N};F)={(-ie)}^{N}{\displaystyle\int_{0}^{\infty}}{dT\over
T}{(4\pi T)}^{-{D\over 2}}\,{\rm e}^{-m^{2}T}{\rm det}^{{1\over
2}}\biggl{[}\frac{\cal Z}{{\rm sin}{\cal Z}}\biggr{]}$
$\displaystyle\times\prod_{i=1}^{N}\int_{0}^{T}\\!\\!\\!d\tau_{i}\exp\biggl{\\{}\sum_{i,j=1}^{N}\Bigl{[}\frac{1}{2}k_{i}\cdot{\cal
G}_{Bij}\cdot k_{j}-i\varepsilon_{i}\cdot\dot{\cal G}_{Bij}\cdot
k_{j}+\frac{1}{2}\varepsilon_{i}\cdot\ddot{\cal
G}_{Bij}\cdot\varepsilon_{j}\Bigr{]}\biggr{\\}}\Big{|}_{\varepsilon_{1}\varepsilon_{2}\cdots\varepsilon_{N}}\,.$
(9)

This representation has already been tested on the calculations of the tadpole
[29, 30] and vacuum polarization amplitudes [31, 32], as well as on magnetic
photon splitting [33, 34], and found to be vastly more efficient than the
traditional methods based on second quantization.

Similar master formulas have been derived for the $N$ \- photon amplitudes in
plane-wave backgrounds [35] and in combined constant field – plane-wave
backgrounds [36].

##  3 Low-energy limit of the $N$ \- photon amplitudes in vacuum

The low-energy limit of the photon amplitudes is defined by the condition that
all photon energies be small compared to the mass of the loop scalar or
fermion,

$\displaystyle\omega_{i}\ll m,\quad i=1,\ldots,N\,.$ (10)

This condition then justifies truncating all the vertex operators to their
terms linear in the momentum. Noting that the leading, momentum-independent
term in this expansion integrates to zero for a closed loop, and adding a
suitable total-derivative term, we can write the vertex operator of a low-
energy photon as

$\displaystyle V^{\gamma\,({\rm LE})}_{\rm scal}[f]$ $\displaystyle=$
$\displaystyle\frac{i}{2}\int_{0}^{T}d\tau\,x(\tau)\cdot
f\cdot\dot{x}(\tau)=\frac{i}{2}\int_{0}^{T}d\tau\,{\rm e}^{x(\tau)\cdot
f\cdot\dot{x}(\tau)}\Big{|}_{f}\,,$ (11)

where $f_{\mu\nu}=k_{\mu}\varepsilon_{\nu}-\varepsilon_{\mu}k_{\nu}$ is the
photon field-strength tensor. The Wick contraction of a product of such
objects produces products of “Lorentz cycles”

$\displaystyle Z_{n}(i_{1}i_{2}\ldots i_{n})$ $\displaystyle\equiv$
$\displaystyle\Bigl{(}\frac{1}{2}\Bigr{)}^{\delta_{n2}}{\rm
tr}\Bigl{(}\prod_{j=1}^{n}f_{i_{j}}\Bigr{)}\,,$ (12)

with coefficients that, by suitable partial integrations, can be written as
integrals of the “$\tau$ \- cycles”
$\dot{G}_{i_{1}i_{2}}\dot{G}_{i_{2}i_{3}}\cdots\dot{G}_{i_{n}i_{1}}$
introduced above. The result can be further simplified by observing that, in
the one-dimensional worldline theory, each of the resulting “bicycle” factors

$\displaystyle\int_{0}^{T}d\tau_{i_{1}}\cdots\int_{0}^{T}d\tau_{i_{n}}\dot{G}_{i_{1}i_{2}}\dot{G}_{i_{2}i_{3}}\cdots\dot{G}_{i_{n}i_{1}}{\rm
tr}\,(f_{i_{1}}f_{i_{2}}\cdots f_{i_{n}})\,,$ (13)

can be identified with the one-loop $n$ \- point Feynman diagram depicted in
Fig. 1.

Figure 1: Worldline Feynman diagram representing an integrated bicycle factor.

Products of such factors thus in the worldline theory correspond to
disconnected diagrams, and by standard combinatorics can be reduced to the
exponential of the sum of all connected diagrams. In this way, and with a
rescaling $\tau_{i}=Tu_{i}$, we arrive at

$\displaystyle\bigl{\langle}V^{\gamma\,({\rm LE})}_{\rm scal}[f_{1}]\cdots
V^{\gamma\,({\rm LE})}_{\rm
scal}[f_{N}]\bigr{\rangle}=(iT)^{N}\,\exp\biggl{\\{}\sum_{n=1}^{\infty}b_{2n}\sum_{\\{i_{1}\ldots
i_{2n}\\}}Z^{\rm dist}_{2n}(\\{i_{1}i_{2}\ldots
i_{2n}\\})\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}\,,$ (14)

where $Z^{\rm dist}_{k}(\\{i_{1}i_{2}\ldots i_{k}\\})$ denotes the sum over
all distinct Lorentz cycles which can be formed with a given subset of
indices, e.g. $Z^{\rm
dist}_{4}(\\{ijkl\\})=Z_{4}(ijkl)+Z_{4}(ijlk)+Z_{4}(ikjl)$, and $b_{n}$
denotes the basic “cycle integral”

$\displaystyle b_{n}$ $\displaystyle\equiv$
$\displaystyle\int_{0}^{1}du_{1}du_{2}\ldots
du_{n}\,\dot{G}_{12}\dot{G}_{23}\cdots\dot{G}_{n1}\;.$ (15)

This integral can be expressed in terms of the Bernoulli numbers ${\cal
B}_{n}$ [28]:

$\displaystyle
b_{n}=\qquad\left\\{\begin{array}[]{r@{\quad\quad}l}-2^{n}{{\cal B}_{n}\over
n!}&\qquad n{\rm\quad even}\,,\\\ 0&\qquad n{\rm\quad odd}\,.\\\
\end{array}\right.$ (18)

Eq. (14) can be further simplified using the combinatorial fact that

$\displaystyle{\rm tr}\Bigl{[}(f_{1}+\ldots+f_{N})^{n}\Bigr{]}\bigg{|}_{\rm
all\,\,different}$ $\displaystyle=$ $\displaystyle 2n\sum_{\\{i_{1}\ldots
i_{n}\\}}Z^{\rm dist}_{n}(\\{i_{1}i_{2}\ldots i_{n}\\})\,,\qquad$ (19)

($n\neq 0$). Introducing $f_{\rm tot}\equiv\sum_{i=1}^{N}f_{i}$, using all
this in (1) and eliminating the $T$-integral, we arrive at the following
formula for the low-energy limit of the one-loop $N$-photon ($N\geq 4$)
amplitude in scalar QED [13]:

$\displaystyle\Gamma_{\rm scal}^{({\rm
LE})}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N})$ $\displaystyle=$
$\displaystyle\frac{e^{N}\Gamma(N-2)}{(4\pi)^{2}m^{2N-4}}\,\exp\biggl{\\{}\sum_{n=1}^{\infty}\frac{b_{2n}}{4n}{\rm
tr}\,(f_{\rm tot}^{2n})\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}\,,$

(here we have exempted the trivial case $N=2$ to be able to set $D=4$).

Further, the above-mentioned Bern-Kosower loop replacement rule allows us to
generalise this result to spinor QED simply by replacing the cycle integral
(18) by the “super - cycle integral”

$\displaystyle\int_{0}^{1}\\!\\!\\!du_{1}du_{2}\ldots
du_{n}\,\Bigl{(}\dot{G}_{12}\dot{G}_{23}\cdots\dot{G}_{n1}-G_{F12}G_{F23}\cdots
G_{Fn1}\Bigr{)}=(2-2^{n})\,b_{n}\,.$ (21)

The only other change is a global factor of $(-2)$ for statistics and degrees
of freedom. Therefore [13]

$\displaystyle\Gamma_{\rm spin}^{({\rm
LE})}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N})$ $\displaystyle=$
$\displaystyle(-2)\frac{e^{N}\Gamma(N-2)}{(4\pi)^{2}m^{2N-4}}$ (22)
$\displaystyle\times\,\exp\biggl{\\{}\sum_{n=1}^{\infty}(1-2^{2n-1})\frac{b_{2n}}{2n}{\rm
tr}\,(f_{\rm tot}^{2n})\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}\,.$

Formulas equivalent to (LABEL:Nphotlowfin) and (22) were derived in [12] from
the Euler-Heisenberg Lagrangian and its scalar QED equivalent, the Weisskopf
Lagrangian, however as already mentioned we have found the above procedure
more suitable for our present purpose of generalization to the constant-field
background.

Note that in the above derivation on-shell conditions have not yet been used.
In [12] it was further shown how to obtain explicit expressions for all the
helicity components of the on-shell amplitudes, using the spinor helicity
formalism. And for the special case of the “all plus” (or “all minus”)
amplitudes this calculation was carried further to the two-loop level [13],
taking advantage of the explicit expressions obtained in [37] for the two-loop
corrections to the Euler-Heisenberg and Weisskopf Lagrangians in the special
case of a self-dual field.

Two interesting aspects of the on-shell $N$-photon amplitudes in the low-
energy limit (at any loop order) have emerged in that work. First, in the
helicity basis the low-energy amplitudes obey a “double Furry theorem”, that
is, they are non-vanishing only if the number of positive and negative
helicity photons are separately even [12]. Second, in the spinor-helicity
formalism the full dependence of the low-energy amplitudes on the momenta and
polarizations can, for any number of photons and any given helicity component,
be absorbed into a single invariant, effectively reducing the amplitude to a
single number. This holds at any loop order, and made it feasible to use these
amplitudes for a study of the asymptotic properties of the photon S-matrix
[24, 25, 26].

##  4 Low-energy limit of the $N$ \- photon amplitudes in a constant field:
Scalar QED

Let us now return to the scalar QED case, and show how to modify the procedure
of the previous section to take an additional constant external field
$F_{\mu\nu}$ into account. Apart from the global determinant factor exhibited
in (9), the only change is that the Wick contraction of the $N$ low-energy
vertex operators (11) now must be performed using the generalized Green’s
function (7). One still arrives at the same representation in terms of bicycle
factors (13), only that now, since the function replacing
$\dot{G}(\tau,\tau^{\prime})$,

$\displaystyle\dot{\cal G}_{B}(\tau,\tau^{\prime})={i\over{\cal
Z}}\biggl{(}{{\cal Z}\over{{\rm sin}{\cal Z}}}\,{\rm e}^{-i{\cal
Z}\dot{G}(\tau,\tau^{\prime})}-1\biggr{)}\,,$ (23)

is non-trivial in Lorentz space, the Lorentz and $\tau$ \- cycle factors will
not any more factorize, but rather combine to form a single Lorentz trace
$\dot{\mathcal{G}}_{B}(i_{1}i_{2}\dots i_{n})$, defined as

$\displaystyle\dot{\mathcal{G}}_{B}(i_{1}i_{2}\dots i_{n})$
$\displaystyle~{}\equiv~{}$
$\displaystyle\Bigl{(}\frac{1}{2}\Bigr{)}^{\delta_{n1}+\delta_{n2}}\mbox{tr}(f_{i_{1}}\cdot\dot{\mathcal{G}}_{Bi_{1}i_{2}}\cdot
f_{i_{2}}\cdot\dot{\mathcal{G}}_{Bi_{2}i_{3}}\cdots
f_{i_{n}}\cdot\dot{\mathcal{G}}_{Bi_{n}i_{1}}).$

Thus in the presence of the constant field (14) generalizes to

$\displaystyle\bigl{\langle}V^{\gamma\,({\rm LE})}_{\rm scal}[f_{1}]\cdots
V^{\gamma\,({\rm LE})}_{\rm
scal}[f_{N}]\bigr{\rangle}_{F}=(iT)^{N}\\!\exp\biggl{\\{}\sum_{n=1}^{\infty}\sum_{\\{i_{1}\ldots
i_{n}\\}}\prod_{k=1}^{n}\int_{0}^{1}du_{i_{k}}\dot{\mathcal{G}}_{B}^{\rm
dist}(\\{i_{1}i_{2}\ldots i_{n}\\})\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}$
(25)

(note that $n$ is now running over all integers, not only over the even ones)
and the challenge is to compute the generalization of the cycle integral (15),

$\displaystyle I^{\rm cyc}_{\rm
scal}(f_{1},f_{2},\ldots,f_{n};F)\equiv\int_{0}^{1}du_{1}\cdots\int_{0}^{1}du_{n}\,\dot{\mathcal{G}}_{B}(12\dots
n)\,.$ (26)

This can be done in the following way. Restricting ourselves now to the case
of a generic constant field (both Maxwell invariants nonzero), and choosing a
Lorentz frame where both the magnetic and the electric field point along the
$z$ axis, the (euclidean) field strength tensor takes the form111Our
conventions follow [20].

$F=\left(\begin{array}[]{*{4}{c}}0&B&0&0\\\ -B&0&0&0\\\ 0&0&0&iE\\\
0&0&-iE&0\end{array}\right)\;.$ (27)

Defining $z_{+}\equiv eBT$ and $z_{-}\equiv ieET$ and the matrices $g_{+}$,
$g_{-}$, and $r_{+}$, $r_{-}$ by

$\displaystyle g_{+}\equiv\left(\begin{array}[]{*{4}{c}}1&0&0&0\\\ 0&1&0&0\\\
0&0&0&0\\\ 0&0&0&0\end{array}\right),\qquad
g_{-}\equiv\left(\begin{array}[]{*{4}{c}}0&0&0&0\\\ 0&0&0&0\\\ 0&0&1&0\\\
0&0&0&1\end{array}\right),$ (36) (37)
$r_{+}\equiv\left(\begin{array}[]{*{4}{c}}0&1&0&0\\\ -1&0&0&0\\\ 0&0&0&0\\\
0&0&0&0\end{array}\right),\qquad
r_{-}\equiv\left(\begin{array}[]{*{4}{c}}0&0&0&0\\\ 0&0&0&0\\\ 0&0&0&1\\\
0&0&-1&0\end{array}\right),$

one can then show the factorization

$\displaystyle{\rm det}^{{1\over 2}}\biggl{[}\frac{\cal Z}{{\rm sin}{\cal
Z}}\biggr{]}=\frac{z_{+}z_{-}}{{\rm sinh}z_{+}{\rm\sinh}z_{-}}\,,$ (38)

and the decomposition

$\displaystyle\dot{\cal G}_{B}(\tau_{i},\tau_{j})$ $\displaystyle=$
$\displaystyle\sum_{\alpha=\pm}S_{Bij}(z_{\alpha})\,g_{\alpha}^{\mu\nu}-i\sum_{\alpha=\pm}A_{Bij}(z_{\alpha})\,r_{\alpha}^{\mu\nu}\,,$
(39)

with coefficient functions

$\displaystyle S_{Bij}(z)$ $\displaystyle=$
$\displaystyle{\sinh(z\,\dot{G}_{ij})\over\sinh z}\,,$ (40) $\displaystyle
A_{Bij}(z)$ $\displaystyle=$ $\displaystyle{\cosh(z\,\dot{G}_{ij})\over\sinh
z}-{1\over z}\;.$ (41)

For our present purpose, it is essential to note that those can be written in
terms of the single function

$\displaystyle H^{B}_{ij}(z)\equiv\frac{e^{z\dot{G}_{ij}}}{\sinh
z}-\frac{1}{z}$ (42)

as

$\displaystyle S_{Bij}(z)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\Bigl{[}H^{B}_{ij}(z)+H^{B}_{ij}(-z)\Bigr{]}\,,$ (43)
$\displaystyle A_{Bij}(z)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\Bigl{[}H^{B}_{ij}(z)-H^{B}_{ij}(-z)\Bigr{]}\,.$ (44)

Thus, defining

$\displaystyle\quad{\mathfrak{g}}_{\beta}^{\alpha}\equiv\frac{1}{2}(g_{\beta}-\alpha
ir_{\beta}),$ (45)

with $\alpha,\beta=\pm$, we can rewrite (39) as

$\displaystyle\dot{\cal G}_{B}(\tau_{i},\tau_{j})$ $\displaystyle=$
$\displaystyle\sum_{\alpha,\beta=\pm}H^{B}_{ij}(\alpha
z_{\beta}){\mathfrak{g}}_{\beta}^{\alpha}\,.$ (46)

The function $H^{B}$ has the following remarkable property of reproducing
itself under folding,

$\displaystyle H_{ik}^{B(2)}(z,z^{\prime})$ $\displaystyle\equiv$
$\displaystyle\int_{0}^{T}d\tau_{j}H^{B}_{ij}(z)H^{B}_{jk}(z^{\prime})=\frac{H^{B}_{ik}(z)}{z^{\prime}-z}+\frac{H^{B}_{ik}(z^{\prime})}{z-z^{\prime}}\,,$
(47) $\displaystyle H_{il}^{B(3)}(z,z^{\prime},z^{\prime\prime})$
$\displaystyle\equiv$
$\displaystyle\int_{0}^{T}d\tau_{j}\int_{0}^{T}d\tau_{k}H^{B}_{ij}(z)H^{B}_{jk}(z^{\prime})H^{B}_{kl}(z^{\prime\prime})$
$\displaystyle=$
$\displaystyle\frac{H^{B}_{il}(z)}{(z^{\prime}-z)(z^{\prime\prime}-z)}+\frac{H^{B}_{il}(z^{\prime})}{(z-z^{\prime})(z^{\prime\prime}-z^{\prime})}+\frac{H^{B}_{il}(z^{\prime\prime})}{(z-z^{\prime\prime})(z^{\prime}-z^{\prime\prime})}\,,$
$\displaystyle\vdots$ $\displaystyle
H^{B(n)}_{i_{1}i_{n+1}}(z_{1},\ldots,z_{n})$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{n}\frac{H^{B}_{i_{1}i_{n+1}}(z_{k})}{\prod_{l\neq
k}(z_{l}-z_{k})}\,.$ (48)

Using this in (LABEL:12-defbicycle), (26) we obtain

$\displaystyle I^{\rm cyc}_{\rm scal}(f_{1},\ldots,f_{n};F)$ $\displaystyle=$
$\displaystyle\Bigl{(}\frac{1}{2}\Bigr{)}^{\delta_{n1}+\delta_{n2}}\sum_{\alpha_{1},\beta_{1}=\pm}\cdots\sum_{\alpha_{n},\beta_{n}=\pm}{\rm
tr}\,(f_{1}{\mathfrak{g}}_{\beta_{1}}^{\alpha_{1}}f_{2}\cdots{\mathfrak{g}}_{\beta_{n}}^{\alpha_{n}})$
(49)
$\displaystyle\times\sum_{k=1}^{n}\frac{H^{B}_{11}(\alpha_{k}z_{\beta_{k}})}{\prod_{l\neq
k}(\alpha_{l}z_{\beta_{l}}-\alpha_{k}z_{\beta_{k}})}\,,$

where now only the coincidence limit of $H^{B}_{ij}(z)$ appears,

$\displaystyle H^{B}_{ii}(z)={\rm coth}\,z-\frac{1}{z}\,.$ (50)

Note that the product in the denominator can contain zero factors, but those
are spurious and cancelled by zeroes of the numerator (for an example, see the
limit $z^{\prime}\to z$ of the right-hand side of (47)).

Putting the pieces together, we arrive at the following generalization of the
vacuum formula (LABEL:Nphotlowfin),

$\displaystyle\Gamma_{\rm scal}^{({\rm
LE})}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N};F)\\!\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\\!\frac{e^{N}}{(4\pi)^{2}}\int_{0}^{\infty}\frac{dT}{T}T^{N-2}\,{\rm
e}^{-m^{2}T}\frac{z_{+}z_{-}}{{\rm sinh}z_{+}{\rm\sinh}z_{-}}$ (51)
$\displaystyle\times\exp\biggl{\\{}\sum_{n=1}^{\infty}\frac{1}{2n}I_{\rm
scal}^{\rm cyc}(f_{\rm tot},\ldots,f_{\rm
tot};F)\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}\,.$

##  5 Low-energy limit of the $N$ \- photon amplitudes in a constant field:
Spinor QED

The transition to the spinor QED case can be done by the following adaption of
the Bern-Kosower replacement rule to the constant-field case [28],

$\displaystyle\dot{\mathcal{G}}_{B}(i_{1}i_{2}\dots i_{n})$
$\displaystyle\rightarrow$ $\displaystyle\dot{\mathcal{G}}_{B}(i_{1}i_{2}\dots
i_{n})-{\mathcal{G}}_{F}(i_{1}i_{2}\dots i_{n})\,,$ (52)

with

$\displaystyle{\mathcal{G}}_{F}(i_{1}i_{2}\dots i_{n})$ $\displaystyle\equiv$
$\displaystyle\Bigl{(}\frac{1}{2}\Bigr{)}^{\delta_{n1}+\delta_{n2}}\mbox{tr}(f_{i_{1}}\cdot{\mathcal{G}}_{Fi_{1}i_{2}}\cdot
f_{i_{2}}\cdot{\mathcal{G}}_{Fi_{2}i_{3}}\cdots
f_{i_{n}}\cdot{\mathcal{G}}_{Fi_{n}i_{1}})\,,$

where $\mathcal{G}_{F}(\tau,\tau^{\prime})$ was given in (8). It permits a
matrix decomposition analogous to (46),

$\displaystyle{\cal G}_{F}(\tau_{i},\tau_{j})$ $\displaystyle=$
$\displaystyle\sum_{\alpha,\beta=\pm}H^{F}_{ij}(\alpha
z_{\beta}){\mathfrak{g}}_{\beta}^{\alpha}\,,$ (54)

now in terms of the function

$\displaystyle H^{F}_{ij}(z)\equiv G_{Fij}\frac{e^{z\dot{G}_{ij}}}{\cosh
z}\,.$ (55)

Remarkably, this function turns out to obey exactly the same multiple-folding
formula as $H^{B}$, eq. (48). Thus without further ado we can generalize (49)
to

$\displaystyle I^{\rm cyc}_{\rm spin}(f_{1},\ldots,f_{n};F)$
$\displaystyle\equiv$
$\displaystyle\int_{0}^{1}du_{1}\cdots\int_{0}^{1}du_{n}\,\bigl{(}\dot{\mathcal{G}}_{B}(12\dots
n)-{\mathcal{G}}_{F}(12\dots n)\bigr{)}$
$\displaystyle=\Bigl{(}\frac{1}{2}\Bigr{)}^{\delta_{n1}+\delta_{n2}}\sum_{\alpha_{1},\beta_{1}=\pm}\cdots\sum_{\alpha_{n},\beta_{n}=\pm}{\rm
tr}\,(f_{1}{\mathfrak{g}}_{\beta_{1}}^{\alpha_{1}}f_{2}\cdots{\mathfrak{g}}_{\beta_{n}}^{\alpha_{n}})\sum_{k=1}^{n}\frac{H^{B}_{11}(\alpha_{k}z_{\beta_{k}})-H^{F}_{11}(\alpha_{k}z_{\beta_{k}})}{\prod_{l\neq
k}(\alpha_{l}z_{\beta_{l}}-\alpha_{k}z_{\beta_{k}})}\,,$

which now involves only the function

$\displaystyle H^{B}_{ii}(z)-H^{F}_{ii}(z)={\rm coth}\,z-{\rm
tanh}\,z-\frac{1}{z}\,.$ (57)

This brings us to our main result, the spinor QED generalization of the master
formula (51) for the low-energy limit of the $N$ \- photon amplitudes in the
constant field:

$\displaystyle\Gamma_{\rm spin}^{({\rm
LE})}(k_{1},\varepsilon_{1};\ldots;k_{N},\varepsilon_{N};F)\\!\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\\!-2\frac{e^{N}}{(4\pi)^{2}}\int_{0}^{\infty}\frac{dT}{T}T^{N-2}\,{\rm
e}^{-m^{2}T}\frac{z_{+}z_{-}}{{\rm\rm tanh}z_{+}{\rm\rm tanh}z_{-}}$ (58)
$\displaystyle\times\exp\biggl{\\{}\sum_{n=1}^{\infty}\frac{1}{2n}I_{\rm
spin}^{\rm cyc}(f_{\rm tot},\ldots,f_{\rm
tot};F)\biggr{\\}}\bigg{|}_{f_{1}\ldots f_{N}}\,.$

##  6 Summary and Outlook

To summarize, we have applied the worldline formalism to the calculation of
the low-energy limits of the $N$ \- photon amplitudes in a generic constant
field, for both scalar and spinor QED. Our final results for these amplitudes,
(51) and (58), reduce their explicit calculation to simple algebra and a
single integral of Euler-Heisenberg type, which could be easily automatized.
They are still valid off-shell.

In a separate publication [38] we apply these formulas to the study of the on-
shell four-photon amplitudes in a constant magnetic field. Further
generalization from the closed-loop to the open-line case should be feasible
along the lines of [39, 40, 41].

## References

  * [1] H. Euler and B. Kockel, Die Naturwissenschaften 23:15 (1935) 246.
  * [2] W. Heisenberg and H. Euler, Z. Phys. 98 (1936) 714.
  * [3] V. Weisskopf, “Über die Elektrodynamik des Vakuums auf Grund der Quantentheorie des Elektrons”, Kong. Dans. Vid. Selsk. Math-fys. Medd. XIV No. 6 (1936), reprinted in Quantum Electrodynamics, J. Schwinger (Ed.) (Dover, New York, 1958).
  * [4] A. Akhieser, L. Landau and I. Pomeranchouk, Nature 138 (1936) 206.
  * [5] R. Karplus and M. Neumann, Phys. Rev. 80 (1950) 380.
  * [6] R. Karplus and M. Neumann, Phys. Rev. 83 (1951) 776.
  * [7] V. Costantini, B. de Tollis and G. Pistoni, Nuov. Cim. A 2 (1971) 733.
  * [8] Y. Liang and A. Czarnecki, Can. J. Phys. 90 (2012) 11, arXiv: 1111.6126 [hep-ph].
  * [9] K. Scharnhorst, arXiv: 1711.05194 [physics.hist-ph].
  * [10] N. Ahmadiniaz, C. Lopez-Arcos, M. A. Lopez-Lopez and C. Schubert, Nucl. Phys. B 991 (2023) 116216, arXiv: 2012.11791 [hep-th].
  * [11] C. Bernicot and J.P. Guillet, JHEP 0801 (2008) 59.
  * [12] L. C. Martin, C. Schubert and V. M. Villanueva Sandoval, Nucl. Phys. B 668 (2003) 335, arXiv: 0301022 [hep-th].
  * [13] G. V. Dunne and C. Schubert, JHEP 0208 (2002) 053, arXxiv: 0205004 [hep-th].
  * [14] R. P. Feynman, Phys. Rev. 80 (1950) 440.
  * [15] R. P. Feynman, Phys. Rev. 84 (1951) 108.
  * [16] A. M. Polyakov, _Gauge Fields and Strings_ , Harwood (1987).
  * [17] Z. Bern and D. A. Kosower, Nucl. Phys. B 379 (1992) 451.
  * [18] M. J. Strassler, Nucl. Phys. B 385 (1992) 145, arXiv: 9205205 [hep-ph].
  * [19] M. G. Schmidt and C. Schubert, Phys. Lett. B 318 (1993) 438, arXiv: 9309055 [hep-th].
  * [20] C. Schubert, Phys. Rep. 355 (2001) 73, arXiv: 0101036 [hep-th].
  * [21] J. P. Edwards and C. Schubert, arXiv: 1912.10004 [hep-th].
  * [22] O. Corradini, C. Schubert, J. P. Edwards and N. Ahmadiniaz, arXiv: 1512.08694 [hep-th].
  * [23] E. S. Fradkin, Nucl. Phys. 76 (1966) 588.
  * [24] I. Huet, D.G.C. McKeon and C. Schubert, JHEP 1012 036 (2010), arXiv: 1010.5315 [hep-th].
  * [25] I. Huet, M. Rausch de Traubenberg and C. Schubert, Adv. High Energy Phys. 2017 (2017) 6214341, arXiv: 1707.07655 [hep-th].
  * [26] I. Huet, M. Rausch de Traubenberg and C. Schubert, JHEP 1903 (2019) 167, arXiv: 1812.08380 [hep-th].
  * [27] R. Zh. Shaisultanov, Phys. Lett. B 378 (1996) 354, arXiv: 9512142 [hep-th].
  * [28] M. Reuter, M. G. Schmidt and C. Schubert, Ann. Phys. (N.Y.) 259 (1997) 313 , arXiv: 9610191 [hep-th].
  * [29] J. P. Edwards and C. Schubert, Nucl. Phys. B 923 (2017) 339, arXiv: 1704.00482 [hep-th].
  * [30] N. Ahmadiniaz, F. Bastianelli, O. Corradini, J. P. Edwards and C. Schubert, Nucl. Phys. B 924 (2017) 377, arXiv: 1704.05040 [hep-th].
  * [31] W. Dittrich and R. Shaisultanov, Phys. Rev. D 62 (2000) 045024, arxiv: 0001171 [hep-th].
  * [32] C. Schubert, Nucl. Phys. B 585 (2000) 407, arXiv: 0001288 [hep-ph].
  * [33] S.L. Adler and C. Schubert, Phys. Rev. Lett. 77 (1996) 1695, arXiv: 9605035 [hep-th].
  * [34] S. L. Adler, Adventures in Theoretical Physics: Selected Papers With Commentaries, World Scientific 2006, arXiv: 0505177 [hep-ph].
  * [35] James P. Edwards and C. Schubert, Phys. Lett. B 22 (2021) 136696, e-Print: 2105.08173 [hep-th].
  * [36] C. Schubert and R. Shaisultanov, Phys. Lett. B 843 (2023) 137969, arXiv: 2303.08907 [hep-th].
  * [37] G.V. Dunne and C. Schubert, Phys. Lett. B 526 (2002) 55,arXiv: 0111134 [hep-th].
  * [38] M. Ahmadiniaz, M.A. Lopez-Lopez, C. Schubert, R. Schützhold and M.A. Trejo, in preparation.
  * [39] A. Ahmad, N. Ahmadiniaz, O. Corradini, S. P. Kim and C. Schubert, Nucl. Phys. B 919 (2017) 9, arXiv: 1612.02944 [hep-ph].
  * [40] N. Ahmadiniaz, V.M. Banda Guzmán, F. Bastianelli, O. Corradini, J.P. Edwards and C. Schubert, JHEP 08 (2020) 049, arXiv: 2004.01391 [hep-th].
  * [41] N. Ahmadiniaz, V.M. Banda Guzmán, F. Bastianelli, O. Corradini, J.P. Edwards and C. Schubert, JHEP 01 (2022) 050, arXiv: 2107.00199 [hep-th].