tdshort=TD, long=Technical Debt, class=abbrev satdshort=SATD, long=Self-
Admitted Technical Debt, class=abbrev cvmtdshort=CVM-TD, long=Contextualized
Vocabulary Model, class=abbrev nlpshort=NLP, long=Natural Language Processing,
class=abbrev astshort=AST, long=Abstract Syntax Tree, class=abbrev
sbtshort=SBT, long=Structure-Based Traversal, class=abbrev oomshort=OOM,
long=Out Of Memory, class=abbrev rnnshort=RNN, long=Recurrent Neural Network,
class=abbrev lstmshort=LSTM, long=Long Short-Term Memory, class=abbrev
ffnnshort=FFNNs, long=Feed-Forward Neural Networks, class=abbrev cvshort=CV,
long=Cross Validation, class=abbrev nmtshort=NMT, long=Neural Machine
Translation, class=abbrev tpshort=TP, long=True Positive, class=abbrev
tnshort=TN, long=True Negative, class=abbrev fpshort=FP, long=False Positive,
class=abbrev fnshort=FN, long=False Negative, class=abbrev pshort=P,
long=Precision, class=abbrev rshort=R, long=Recall, class=abbrev mnbshort=MNB,
long=Multinomial Naive Bayes, class=abbrev svmshort=SVM, long=Support Vector
Machines, class=abbrev bowshort=BoW, long=Bag of Words, class=abbrev
tfidfshort=TF-IDF, long=Term Frequency-Inverse Document Frequency,
class=abbrev grushort=GRU, long=Gated Recurrent Unit, class=abbrev mlshort=ML,
long=Machine Learning, class=abbrev dlshort=DL, long=Deep Learning,
class=abbrev cplshort=CPL, long=Comment Processing Line, class=abbrev
scplshort=SCPL, long=Source Code Processing Line, class=abbrev vsmshort=VSM,
long=Vector Space Model, class=abbrev cnnshort=CNN, long=Convolutional Neural
Network, class=abbrev tmlshort=TML, long=Traditional Machine Learning,
class=abbrev

# A Framework for Self-Admitted Technical Debt Identification and Description

Abdulaziz Alhefdhi  University of WollongongWollongongNew South WalesAustralia
Prince Sattam bin Abdulaziz University, Wadi Ad-Dawasir, Saudi Arabia
<EMAIL_ADDRESS><EMAIL_ADDRESS>0000-0001-6162-6064 ,  Hoa Khanh
Dam  University of WollongongWollongongNew South WalesAustralia<EMAIL_ADDRESS>,  Yusuf Sulistyo Nugroho  Universitas Muhammadiyah SurakartaSurakartaJawa
TengahIndonesia<EMAIL_ADDRESS>,  Hideaki Hata  Nara Institute of
Science and TechnologyIkomaNaraJapan<EMAIL_ADDRESS>,  Takashi Ishio  Nara
Institute of Science and TechnologyIkomaNaraJapan<EMAIL_ADDRESS>and
Aditya Ghose  University of WollongongWollongongNew South WalesAustralia
<EMAIL_ADDRESS>
###### Abstract.

Technical debt occurs when software engineers favour short-term operability
over long-term stability. Since this puts software stability at risk,
technical debt requires early attention (failing which it accumulates
interest). Most of existing work focus on detecting technical debts through
code comment (i.e. self-admitted technical debt). However, there are many
cases where technical debts are not explicitly acknowledged but deeply hidden
in the code. In this paper, we propose a more comprehensive solution to deal
with technical debt. We design a framework that caters for both cases of the
existence of a comment. If a comment is absent and our framework detects a
technical debt hidden in the code, it will automatically generate a relevant
comment that can be attached with the code. We explore different
implementations of this framework and the evaluation results demonstrate the
applicability and effectiveness of our framework.

Software Documentation, Software Quality, Software Analytics, Self-Admitted
Technical Debt

##  1\. Introduction

Development teams aim at implementing a software project of high quality, on
time, and within budget. In real-world scenarios however, quality sometimes
needs to be sacrificed in order to deliver a software project on time.
Developers tend to apply quick fixes or temporary implementations that are not
necessarily ideal for the long term, which leads to the incurring of td to the
software (Brown et al., 2010; Tom et al., 2013).

Technical Debt is a term introduced by Ward Cunningham (Cunningham, 1993) to
describe the situations where accomplishing short-term goals is chosen over
long-term code quality. Just like financial debt, td can accumulate interest
if it is not delt with quickly. Poor coding practices can cause the presence
of td. For instance, Kruchten et al. put ”quick-and-dirty” conditional
statements (e.g. if-statements) on the top of their td example list (Kruchten
et al., 2019). Previous studies have highlighted the widespread occurrence of
td and its impact on software quality, complexity, and maintenance (Wehaibi et
al., 2016; Zazworka et al., 2011; Lim et al., 2012; Huang et al., 2018). The
presence of td makes changes to the system more frequent (Zazworka et al.,
2011) and more difficult (Wehaibi et al., 2016) to implement. Developers
recognise that td is unavoidable and, therefore, in need of careful management
(Lim et al., 2012). Repaying td comes in the form of re-structuring and re-
factoring the software (Huang et al., 2018).

Despite the importance of td management, especially in its early stages, the
identification of td is a challenge. Moreover, td instances needs to be well-
understood once it is identified in order for it to be properly managed. For
example, if a developer implements a _workaround_ in the code and wraps it
within an if-statement, other developers in the team may not know that this is
a td-carrying statement. To make it visible, the developer writes a comment
that describes the td. Such comments flag Self-Admitted Technical Debt (SATD)
(Potdar and Shihab, 2014).

Most of existing work (e.g. (Potdar and Shihab, 2014; Maldonado and Shihab,
2015; de Freitas Farias et al., 2015; da Silva Maldonado et al., 2017; Huang
et al., 2018; Yan et al., 2018)) has focused on developing tool support for
detecting if a code comment flags satd. All of them assume the existence of
such comments attached to code fragments that contain technical debt.
_However, this assumption does not always hold in practice. There are many
instances where td in code is not explicitly acknowledged via a comment_. In
these cases, there is a need for automated machinery which can: (i) recommend
if a given code fragment has a technical debt; and if so, (ii) generate a
relevant (self-admitting) comment that can be attached to the code. However,
proposals to provide this kind of support is currently missing in the
literature.

In this paper, we propose a framework that provides three levels of support:
satd detection, satd recommendation and satd description. Given a code
fragment as input, if a comment is attached with it, our framework can detect
if the comment flags an satd (_satd detection_). This is the basic support
provided by existing work in the literature. If a comment is not attached with
the code fragment, our framework can recommend if there is a technical debt
hidden in the code (_satd recommendation_), and automatically generate a
comment describing this technical debt (_satd description_). We then explore
and evaluate the capabilities of different machine/deep learning approaches in
implementing this framework and providing all of these types of support.

The rest of the paper is outlined as follows. Section 2 gives a motivating
example. Section 3 illustrates the proposed architectural framework. Section 4
describes the implementation of our approach. We evaluate and discuss our
approach in Sections 5 and 6. We present the related work in Section 7 before
we conclude the paper in Section 8.

##  2\. Motivating example

Previous research (Wehaibi et al., 2016) has studied the impact of satd on
software complexity and changeability. It is inevitable that satd needs
addressing and the ultimate goal is removing satd. In order to manage and
eventually remove satd, we need to identify them in our codebase. This is a
challenging task, especially in large software projects where codebases can
grow to millions of lines of code. Figure 1 depicts several scenarios of a
code fragment to illustrate the challenge of this problem.

Figure 1. In scenario 1.a, the code fragment introduces td and a satd comment
is provided. In scenario 1.b, the code fragment does not introduce td while a
comment is provided. In scenario 2.a, the code fragment introduces td but a
satd comment is not provided. In scenario 2.b, neither the code fragment
introduces td nor a comment is provided.

In scenario 1.a., there is technical debt in the code fragment, which was
explicitly acknowledged in the comment. However, comments can be written for
other purposes, not just only for acknowledging a technical debt (see scenario
1.b.). The challenge here is to detect which comments are satd and which ones
are not. Existing work (e.g. (Potdar and Shihab, 2014; da Silva Maldonado et
al., 2017; Huang et al., 2018)) mostly focuses on addressing this challenge.

The existing approaches rely on the comments provided with the code to detect
satd. However, there are many cases where technical debt is not explicitly
admitted via a comment. Scenario 2.a. in Figure 1 shows an example of a code
fragment extracted from project
NemakiWare111https://github.com/aegif/NemakiWare. This if-statement was
written to address the issue of an empty content. The code only returns null
while it needs better handling, such as logging the incident. Notice that
there was no comment associated with this code fragment. Hence, we need to
analyse the source code to determine if technical debts exist. There are many
cases where the code does not contain debt and there is no comment provided
(see Scenario 2.b.). Automated tool support is thus needed to assist and make
appropriate recommendations to software engineers in doing this task. If a
technical debt is detected, the tool should also be able to generate a
relevant satd comment that can be attached with the code fragment. In the next
section, we present an architectural framework which addresses all the above
scenarios.

Figure 2. Framework Architectural Design

##  3\. Architectural framework

We propose an architectural framework for a tool that addresses the four
possible scenarios presented in Section 2. This framework takes as input a
code fragment and provides different support for detecting satd and generating
satd comments. The framework first checks if the input code fragment contains
a comment, and then determines if that comment is an SATD. If a comment is not
associated with a code fragment, the framework processes and analyses the code
fragment to determine if it contains a technical debt. If a technical debt is
detected, it automatically generates a SATD comment which can be attached with
the code fragment.

The design of this framework consists of multiple components that are
distributed in two _Processing Lines_ and four _Processing Stations_ (see
Figure 2). The two processing lines are the cpl and the scpl. The four
processing stations are Data Processing, Data Vectorisation, satd
Identification, and satd Comment Generation. These framework components
operate in a chronological manner to achieve the main objective of providing
satd Identification and Description services.

The input code fragment is directed to one of the two processing lines
depending on its content. If the input includes a comment, it will be directed
to the cpl. If no comment is provided, the input will be directed to the scpl.

The cpl determines if the comment provided with the input code fragment is a
satd. In this processing line, the comment passes through three processing
stations, namely Data Processing, Data Vectorisation, and satd Identification.
In Data Processing station, the comment goes through Natural Language
Processing tasks, such as converting all letters to lower case, removing
punctuation and stemming. After that, the processed comment is passed to the
Vectorisation station. The user can configure two options: using either
traditional machine learning classifiers (e.g. mnb, svm, or Random Forest) or
deep learning classifiers (e.g. lstm or cnn). Each technology has its own Data
Vectorisation techniques. The Word Embedding component in this station creates
an embedding (i.e. vector) for each word in the comment, While the vsm
component creates a vector that represent the entire comment. The vectors are
then passed to the satd Identification station. This station contains the
essential intelligent learning component for satd comments detection.
Accordingly, the framework raises a warning if it determines that the tested
comment is a satd comment.

The scpl is dedicated for processing code fragments when comments are not
provided, which is a more challenging situation (and is a key novelty of our
framework). This processing line passes through all the four processing
stations, including satd Comment Generation station. In Data Processing
station, the input source code is passed through the component responsible for
Programming Language processing. In the next two processing stations, the user
can configure using either traditional machine learning or deep learning
classifiers. Data Vectorisation technique is performed according to the user’s
configuration. Subsequently, the vectorised source code is passed to the satd
Comment Generation station if the framework determines it is a satd code
fragment. The framework raises a warning and creates a satd comment to be
attached with the input code fragment.

##  4\. Implementation

In this section, we describe how each component in our framework can be
implemented using existing Natural Language Processing, code modelling and
machine learning techniques.

###  4.1. Data Processing

The data processing station consists of two components: one for processing
source code and the other for processing comments. Their details are as
follows.

####  4.1.1. Source Code Processing

We first convert the input source code segment to an Abstract Syntax Tree
(ast) using a method proposed by Hu et al. (Hu et al., 2018) called ast with
sbt. The asts are then translated into a sequences of tokens that can be
readily input to the other components of our framework. Classical traversal
methods that converts asts to sequences (e.g. pre-order traversal) can be
ambiguous in the way that different code segments may produce the same
sequence representation. Hu et al. (Hu et al., 2018) address this problem by
proposing the sbt method to ensure unique sequence representations for
different code inputs. Figure 3 shows an example of parsing a conditional
statement to its ast then applying the sbt method to produce the sequence
representation of the tree.

Figure 3. An example of converting a conditional statement to the sequential
representation of its ast using the sbt method.

####  4.1.2. Comment Processing

For natural language comments that accompany source code, we focus only on
English comments. We ignore numbers, non-English text, and special characters.

A start-of-sentence token, <sos>, is added to the beginning of every comment
and an end-of-sentence token, <eos>, is added to the end of every comment.
This comes in effect in satd Comment Generation. Ideally, the model generates
a sample comment that a developer should use to self-admit and describe the td
introduced by the source code. We will see in Section 4.4 that we need the
start-of-sentence token to signal to the deep learning model to generate the
actual first word in the comment, and we need the end-of-sentence token as a
signal for the model to stop the word generation process.

###  4.2. Data Vectorisation

####  4.2.1. Word Embedding

Text data typically are highly sparse (Bingham and Mannila, 2001) and face the
problem of high dimensionality (Aggarwal and Zhai, 2012). If we use one-hot
encoding to represent a word as vector, we will end up with a vector that is
the size of the entire vocabulary with all 0s except a 1 at the word’s
position, and that is for each word in the vocabulary of our dataset. Note
that “word” here refers to either a _code token_ (in source code) or a
_textual word_ (in a comment). To alleviate this problem, we use a technique
called _word embedding_ (Gal and Ghahramani, 2016) which aims to represent
each word in the vocabulary as a fixed-length continuous vector (also called
an embedding). The values in those embedding vectors are learnt and adjusted
during training. Word embedding has the trait of figuring semantic relations
between words according to how the values inside their embeddings relate to
each other. Word embeddings that have a common semantic relation tend to
cluster together.

Central to this is an embedding matrix
$\mathcal{M}\in\mathbb{R}^{d\times|\mathscr{V}|}$ where $\boldsymbol{d}$ is
the embedding size and $|\mathscr{V}|$ is the number of words in our
vocabulary $\boldsymbol{\mathscr{V}}$. The embedding matrix
$\boldsymbol{\mathcal{M}}$ acts as a lookup table where each row is a vector
representation of a word in our vocabulary. Each word will have an index, and
the word with index $\boldsymbol{i}$ will have its vector representation (i.e.
embedding) in the $i^{th}$ row of the matrix $\boldsymbol{\mathcal{M}}$. Our
implementation only deals with these indices and their vector representations
and does not have access to the actual words/tokens. Our implementation can
also generate indices that then can be converted to their associated words in
order to display the generated comment sentences.

####  4.2.2. Vector Space Model

An alternative approach to word embedding is using Vector Space Model (VSM)
(Salton et al., 1975). While every word/token is represented as a vector in
word embedding, the entire comment/source code segment is represented by a
vector in vsm. Let us call a comment/source code segment a Document
$\boldsymbol{d}$, a word/token a Term $\boldsymbol{t}$. Every Document is
represented by a vector, every vector is a data point, and every Term is a
dimension in these vectors. We calculate the weights of these Terms by using a
scheme called tfidf. This scheme determines the “importance” of a Term
$\boldsymbol{t}$ in a Document $\boldsymbol{d}$. Term’s importance to a
Document increases by two factors: its frequency in that Document and its
rarity in the entire Document set. tfidf is computed as follows:

(1) $idf(t)=\log(\frac{|\mathcal{D}|}{df(t)})+1$

(2) $tfidf(t,d)=tf(t,d)\times idf(t)$

where $\boldsymbol{|\mathcal{D}|}$ is the total number of Documents,
$\boldsymbol{df(t)}$ is the number of Documents the Term $\boldsymbol{t}$
appears in, and $\boldsymbol{tf(t,d)}$ is the number of times the Term
$\boldsymbol{t}$ appears in Document $\boldsymbol{d}$.

###  4.3. SATD Identification

####  4.3.1. Deep Learning Detector

Our implementation for the “Deep Learning Detector” components (and also the
“Deep Learning Generator” component) is based on lstm models . The lstm models
have the embedding layer as their first layer. Let
$\boldsymbol{W_{1}},...,\boldsymbol{W_{n}}$ be an input sequence (a sequence
of source code tokens or comment words). The embedding layer converts the
elements of the input sequence into their word embeddings
$\boldsymbol{V_{1}},...,\boldsymbol{V_{n}}$. The layer(s) following the
embedding layer is an lstm layer (or a stack of layers). An lstm layer
consists of a sequence of lstm units. All of these units share the same model
parameters since lstms are a rnn (Recurrent Neural Network). At a time step
$\boldsymbol{t}$, an LSTM unit reads an input $\boldsymbol{V_{t}}$ and the
previous output state $\boldsymbol{S_{t-1}}$, and returns the current output
state $\boldsymbol{S_{t}}$. Then, the output state $\boldsymbol{S_{t}}$ is
passed into two directions: the next layer (whether it is an lstm layer or a
Dense layer) and the lstm unit in the next time step $\boldsymbol{t+1}$. See
Figures 4 and 5.

There are several variations of the implementation of the deep learning
Detectors which produce different results as we will discuss in Sections 5.4
and 5.5. The variations include three different down-sampling techniques
explained below and different hyper-parameter settings explained in the our
experimental setup (Section 5.2).

Figure 4. The structure of our deep learning Detector (single-layered). The
number of layers is determined by the number of stacked lstm layers. Input can
be code tokens or textual words. “Positive” outcome indicates a technical debt
is detected.

Down-Sampling: The Dense layer accepts one vector as an input. Therefore, we
examine the Detector’s performance with and without using a “pooling”
technique. With pooling, we experiment with _max-pooling_ and _mean-pooling_.

Without pooling, we only consider the “last” output state $\boldsymbol{S_{t}}$
at time step $\boldsymbol{t}$ since it holds information from all the previous
output states $\boldsymbol{S_{1}...S_{t-1}}$, thanks to lstm dynamics, and
pass it to the Dense layer.

Pooling is a down-sampling method that reduces multiple inputs to the desired
size (in our case, the size of the output state). At a time step
$\boldsymbol{t}$, _max-pooling_ consider only the maximum values in every
element position in the output states $\boldsymbol{S_{1}...S_{t}}$ while
_mean-pooling_ averages them. The resulting vector is then passed to the Dense
layer.

Suppose we have an input sequence of three items (tokens or words),
$[W_{1}\quad W_{2}\quad W_{3}]$, and an output state of size 2. Suppose that
the following are the final output states of these three tokens/words:

$\quad S_{1}=[5.2\quad 3.3]\qquad S_{2}=[4.7\quad 7.5]\qquad S_{3}=[9.1\quad
0.6]$

We want to create one vector that captures information from all these three
output states and use it as in input to the Dense layer. If no pooling
technique is used, we consider the last output state $\boldsymbol{S_{3}}$. In
_max-pooling_ , we pool the elements of the same position and consider only
the maximum value. In _mean-pooling_ , we pool the elements of the same
position and take the average:

$\quad NoPool(S_{1},S_{2},S_{3})=[9.1\quad 0.6]$

$\quad MaxPool(S_{1},S_{2},S_{3})=[9.1\quad 7.5]$

$\quad MeanPool(S_{1},S_{2},S_{3})=[6.33\quad 3.8]$

Sigmoid Activation: As explained earlier, every lstm unit is assigned to
process an input item in a timely manner starting from the first input
token/word in a code/comment sequence through the last item. At each time step
$\boldsymbol{t}$, the vector resulting from down-sampling is passed to the
Dense layer. The dense layer has a _sigmoid_ activation function of the
following formula:

(3) $S(x)=\frac{1}{1+e^{-x}}=\frac{e^{x}}{e^{x}+1}$

The sigmoid function return a value between 0 and 1. This value determines
what the Detector thinks regarding the input sequence under investigation. If
the value is closer to 1, it means that the Detector leans more towards
deciding that the input sequence is a satd code/comment. If the value is
closer to 0, it means the Detector votes for the opposite.

####  4.3.2. Traditional Machine Learning Detector

An alternative implementation to deep learning is using traditional machine
learning algorithms. Based on our experimentation, Support Vector Machines
(SVM) and Multinomial Naive Bayes (MNB) provide comparable performance to deep
learning. In fact, McCallum et al. (McCallum et al., 1998) argue that mnb
proves effective with large vocabulary sizes. In addition, svm is a leading
approach for text categorisation problems as suggested by Kibriya et al.
(Kibriya et al., 2004). We refer to (Kibriya et al., 2004; Joachims, 1998) for
futher details on mnb and svm for text classification.

###  4.4. SATD Comment Generation

The purpose of the “SATD Comment Generator” component is to generate a
relevant satd comment which describes the td found in a code fragment. We
implement this component using the deep learning encoder-decoder model that
uses the sequence-to-sequence (_seq2seq_) learning method (Cho et al., 2014;
Sutskever et al., 2014). The encoder and decoder are two lstm networks. We
also incorporate the Attention mechanism into our Generator that is explained
below. We examine the SATD Comment Generator in multiple hyper-parameter
settings explained in Section 5.6. An internal view of our _SATD Comment
Generator_ is depicted in Figure 5 (adapted from (Luong et al., 2015)).

The dynamics of the embedding lstm layers in the satd Comment Generator behave
in the same manner is of the Detector layers described in Section 4.3. In
every time step $\boldsymbol{t}$, the Generator passes the output state
$\boldsymbol{S_{t}}$ to the Attention layer instead of a Dense layer. The
encoder passes the last output state $\boldsymbol{S_{n}}$ to the first lstm
unit in the decoder to accompany the embedding of the pre-first target comment
word $\boldsymbol{W_{out\\_0}}$. The lstm layer produces the first output
state in the decoder $\boldsymbol{S_{1}}$. $\boldsymbol{S_{1}}$ is passed to
the Attention layer. The vector resulting from the Attention layer is passed
to the Dense layer in order to predict the first target word
$\boldsymbol{W_{out\\_1}}$. For predicting every target word
$\boldsymbol{W_{out\\_t}}$, the decoder is fed with the previous target word
$\boldsymbol{W_{out\\_t-1}}$. This technique is called teacher-forcing
(Chollet, 2017), where the decoder trains to generate the same target sequence
but offset by one time step.

Figure 5. The structure of our deep learning Generator (single-layered encoder
and decoder). The number of layers is determined by the number of stacked lstm
layers.

####  4.4.1. Attention Mechanism

The Attention mechanism has demonstrated important improvements in nmt tasks
(Bahdanau et al., 2014). We implement an Attention layer into our _SATD
Comment Generator_ component to align between certain items in the input and
output sequences. When predicting an output word $\boldsymbol{W_{out\\_t}}$ at
a time step $\boldsymbol{t}$, the Attention layer determines the amount of
contribution each token in the input sequence
$\boldsymbol{W_{in\\_1}},...,\boldsymbol{W_{in\\_n}}$ has on the current
output word. Without the Attention layer, all the input tokens would have the
same weight when predicting a new output word, which is less practical since
certain input tokens can map more closely to the current output word than
others. The Attention layer adjusts the mapping weights gradually during
training.

####  4.4.2. Softmax Activation

The last layer in the decoder is a Dense layer with a _softmax_ activation
function:

(4) $\sigma(x)_{i}=\frac{e^{x_{i}}}{\sum_{v=1}^{|\mathscr{V}|}e^{x_{v}}}$

If we have $|\mathscr{V}|$ words in our target vocabulary (in our case, the
vocabulary of the comments), the _softmax_ activation function gives a
probability value between 0 and 1 to each word in the vocabulary for the
current prediction where the sum of all these values is 1. The model then
nominates the word with the highest probability value to be the predicted
output word for the current position $\boldsymbol{t}$ in the comment sentence.

###  4.5. Model training

The training data is fed into the above learning components in batches. For
each batch, the neural networks performs two training tasks: the feed-forward
task and then the back-propagation task. In the feed-forward task, the model
processes the batch input and calculates the predictions. In the back-
propagation task, the model measures the error distance between the actual
outputs and the predicted outputs of the current batch and tweaks its
parameters accordingly. By that, the model completes one training step.
Feeding the data in batches to the model has multiple benefits. First, it
accelerates the training process compared with feeding the model only one
observation at a time. Second, it introduces noise to the model which helps
preventing over-fitting to the training data. Nonetheless, large batch sizes
can be computationally exhaustive and reduce the prediction accuracy. Batch
Size is one of the model’s hyper-parameters that we consider during hyper-
parameter tuning (Section 5.2.1).

We apply an over-fitting prevention strategy called dropout (Srivastava et
al., 2014). This strategy stops a proportion of the neural network’s nodes
from processing the batch’s observations. This is useful because some nodes
can dominate the training weights. By applying a random node dropout rate at
each training step (we set our dropout rate to 20%), we allow the remaining
80% of the nodes to process the current batch, hence help avoid weight
dominance that may cause over-fitting.

In the SATD Identification tasks, the deep learning Detector tries to maximise
the likelihood of predicting the target label. Consider that the target label
for a data observation is $\boldsymbol{y}$ and the label predicted by the
model for that observation is $\boldsymbol{p}$. We already know from the
training data the true value of $\boldsymbol{y}$ (the ground truth, whether it
is 0 or 1). We use this information to learn the model weights. We measure the
accuracy of our prediction $\boldsymbol{p}$ by calculating the log-loss (also
known as the cross entropy) between $\boldsymbol{y}$ and $\boldsymbol{p}$:

(5) $-(y\log(p)+(1-y)\log(1-p))$

In generating SATD comments, _SATD Comment Generator_ tries to maximise the
likelihood of predicting the next target comment word. The target word at each
time step is a word from our vocabulary $\mathscr{V}$. We can look at this
situation as a multi-class classification problem. Let $\boldsymbol{M}$ be the
number of words in our target vocabulary ($M=|\mathscr{V}|$). We treat the
problem as if we have $\boldsymbol{M}$ different classes. Suppose that at the
current time step we are trying to predict the word todo, it is the fifth word
in the vocabulary ($\boldsymbol{c}=5$), and we have only ten words
($\boldsymbol{M}=10$) in our vocabulary. Since it is a multi-class
classification problem, we will have 10 different binary indicators,
$\boldsymbol{y_{c,o}}$, for the current class. Only if the observation
$\boldsymbol{o}$ is the same as the actual class (in this case, todo for both
$\boldsymbol{c}$ and $\boldsymbol{o}$), then $\boldsymbol{y_{c,o}}$ is 1.
$\boldsymbol{y_{c,o}}$ is 0 for the remaining 9 binary indicators
$\boldsymbol{y_{c,o}}$ (e.g. $\boldsymbol{c}$ is todo and $\boldsymbol{o}$ is
hack). Therefore, the log-loss for the current word prediction is calculated
as follows:

(6) $\sum_{c=1}^{M}y_{c,o}\log(p_{c,o})$

Once the cross entropy is computed, a model optimiser is used to update the
model parameters on the opposite direction of the gradient of the log-loss. We
are using the _Adam_ optimiser (Kingma and Ba, 2014) in our Detector and the
_RMSprop_ optimiser (Choetkiertikul et al., 2018) in our Generator to
determine the best model weights (i.e. model parameters) possible.

##  5\. Evaluation

We implement our framework using scikit-learn222http://javaparser.org/, a
machine learning library in Python, and Keras (Chollet et al., 2015), a Python
deep learning library running on top of TensorFlow (Abadi et al., 2015), a
machine learning platform. We also utilise JavaParser333http://javaparser.org/
to help convert conditional statements to their asts. To contribute the
research community, we have published our source code and
dataset444https://drive.google.com/drive/folders/1SrUcUw2ORY8oCXv-
FHtmReaxa2zM80cG?usp=sharing.

###  5.1. Dataset

####  5.1.1. Data collection

Code comments were collected with the same procedure in the previous study
(Hata et al., pear), which had targeted active software development
repositories on GitHub. We targeted repositories written in Java. Active
software development repositories were selected from the GHTorrent
datasets555MySQL database dump 2018-04-01 from
http://ghtorrent.org/downloads.html. (Gousios, 2013) with the following
criteria (Hata et al., pear): (i) more than 500 commits (the same threshold
used in previous work (Aniche et al., 2018)), and (ii) at least 100 commits in
the most active two years (to remove long-term less active projects and short-
term repositories, which may not be software development projects (Munaiah et
al., 2017)).

From the collected 4,995 Java repositories, single comments and just behind
conditional statements were collected as pairs of code and comments. By
analysing the ast of each source file with an ANTLR4-based Java parser,
“outermost” if-statements were identified. We ignored inner conditional
statements enclosed in another if-statement. A sequence of else-if (e.g. if-
else-if-else-if ...else) is regarded as a single if-statement. An if-statement
is linked to a comment if the comment satisfies the following two conditions:
(i) It appears between the if keyword and its previous non-comment token, and
(ii) The character position in line is the same as the if-statement. Although
multiple comments may link to an if-statement, we removed them from our
dataset.

From the extracted comments, we prepare satd and non-satd comments using the
following keywords shown in the previous study (Huang et al., 2018).

  * •

satd comments: including at least one of the common 14 single keywords of
todo, fixme, hack, workaround, yuck, ugly, stupid, nuke, kludge, retarded,
barf, crap, silly, and kaboom.

  * •

Non-satd comments: excluding all the above 14 keywords and other frequently
appearing 22 keywords of implement, fix, ineffici, xxx, broken, ill, should,
need, here, better, why, method, could, work, probabl, not, move, more, make,
code, but, and author.

We obtained 5,313 satd code-comment pairs and 839,431 non-satd code-comment
pairs. In the collected 5,313 satd pairs, there are 2,851 distinct comment
contents.

To understand the characteristics of the collected satd code-comment pairs, a
statistically representative sample of the distinct satd comments was
analysed. The required sample size was calculated so that the ratio of
publication citations would generalise to all comments with a confidence level
of 95% and a confidence interval of
5666https://www.surveysystem.com/sscalc.htm, and we obtained a sample of 339
code-comment pairs.

Three authors independently investigated the same 20 pairs to determine
whether (i) the comment represents technical debt in code, and (ii)
conditional statements are single or multiple.777Kappa agreement was
calculated using http://justusrandolph.net/kappa/. The kappa agreement levels
were (i) 0.90 and (ii) 0.88, which indicate “almost perfect” (Viera et al.,
2005). Based on this encouraging result, the remaining data was then
investigated by a single author. In the statistically representative sample of
339 code-comment pairs, we found (i) 298 (88%) are actually satd pairs. Within
the 298 satd pairs, 270 (91%) code segments are single if-statements. We
consider this result promising for our experiments as we collected satd code-
comment pairs with a small amount of noise and that the obtained conditional
statements were not too complex, which is beneficial for learning satd code-
comment patterns.

####  5.1.2. Data pre-processing

To avoid oom and data noise issues, we set the maximum lengths for input
sequences and comment sentences to 1500 and 150 token/word, respectively. We
do not truncate ast sequences and comments. Truncation could be useful for
classification tasks (e.g. satd Identification). However, it could harm the
satd Comment Generation task since truncated words in the output sentence
could map to tokens in the input sequence and vice versa. Thus, observations
longer than the maximum length are ignored. We reserve an <UNKN/PAD> token for
(i) padding during model training and, (ii) at model validation/testing,
replacing input tokens that have not been seen during model training.

To ensure that there is no bias towards a subset of the dataset, and to avoid
data leakage to the training set (Kaufman et al., 2012), we enforce a strict
rule on the observations of the dataset by removing all duplicate instances in
the dataset so that every observation in the dataset is unique. Additionally,
we apply a data randomisation procedure using the Mersenne Twister
pseudorandom number generator (Matsumoto and Nishimura, 1998) to avoid
observation order biases that potentially occurred during the time of data
collection.

After performing the pre-processing steps, the number of satd observations
shrinks from 5,313 to 3,022. For the satd Source Code Identification
experiment (Section 5.4), we used the same number of non-satd as satd
observations. We used both satd and non-satd observations in order to teach
the Detector to distinguish between satd and non-satd code-comment pairs. We
took advantage of the rest of the non-satd observations that were not used to
train the Detector used then in the _pre-training_ experiment 5.4.2. The
dataset has 27,359 unique code tokens and 9,999 unique comment words.

###  5.2. Experimental Setup

####  5.2.1. Hyper-parameter tuning

This step is performed to search for an optimal set of model hyper-parameters.
This step requires a separate validation set that will not be used during cv.
We refer to it here as a “tuning” set to avoid confusion with cross
validation. The tuning set is a stratified 10% proportion of the entire
dataset. In the hyper-parameter tuning step, we train the model on the
remaining 90% of the dataset multiple times while tuning the hyper-parameters
every time. The best hyper-parameter settings are chosen according to the the
model’s performance results on the tuning set. The nominated hyper-parameter
settings will then be used in the cv step (Section 5.2.2).

We experiment with a set of four hyper-parameters: the batch size, number of
layers, layer size, and embedding size. The batch size is discussed in Section
4.5. The number of layers determines how many layers our lstm network has. We
experiment with one, two, and three layers. For the last two hyper-parameters
(i.e. layer size and embedding size), we combine them in a super hyper-
parameter and call it the _latent dimensionality_. When we experiment with one
and two layers, the size of the embedding and lstm layers remain the same as
the latent dimension. When we experiment with three layers, for the Detector,
the size of the embedding size and the second lstm layer are the same as the
latent dimension, but the first lstm layer is double the latent dimension
while the last lstm layer is half of the latent dimension. For the Generator,
all the layers are equal to the latent dimension.

####  5.2.2. Stratified 10-Fold Cross Validation

We perform 10-fold cv (scikit learn, [n.d.]) on the entire dataset after
removing the tuning set introduced in Section 5.2.1. By that, we guarantee
every observation in the dataset is tested against. For the Detector, cv is
stratified. For the Generator, it is not applicable to have a stratified cv.
10-Fold cv is the main step whose results will be evaluated and discussed.

###  5.3. Evaluation Metrics

Regarding both satd Comment and Code Identification, we treat them as
classification problems. Thus, we use _Precision_ , _Recall_ , and _F1-Score_
to evaluate our explored approaches and compare their performances against the
benchmarks. Regarding satd Comment Generation, we treat it as a translation
problem and use variations of the _BLEU Score_ to evaluate our approach. We
also performed human evaluation of the generated comments using two criteria,
namely acceptability and understandability.

###  5.4. SATD Source Code Identification

Due to space limitations, we report here the result of some hyper-parameter
tuning settings888We refer the readers to the full report at
https://drive.google.com/drive/folders/1SrUcUw2ORY8oCXv-
FHtmReaxa2zM80cG?usp=sharing. They are as follows: _Latent Dimensionality_ :
(8, 16, 32, 64, 128, 256). _Number of Layers_ : (1, 2, 3). _Batch Size_ : (8,
16, 32, 64, 128, 265, 512).

Furthermore, we experimented with _mean-pooling_ , _max-pooling_ , and _last-
vector_ (i.e. no-pooling). We nominated the three best performing hyper-
parameter settings with _mean-pooling_ for the 10-fold cv step. We did the
same for _max-pooling_ and _last-vector_. Table 1 lists the average
_Precisions_ , _Recalls_ , _F1-Scores_ of the 10-fold cv step. The results are
ordered according to the _F1-Score_. If two hyper-parameter settings have the
same _F1-score_ , the one with the higher _Precision_ is listed first.
Different traditional machine learning models and their hyper-parameter
settings were also experimeted in the 10-fold cv step. The best results were
achieved by mnb and svm. Their results are also listed in Table 1.

Table 1. Average _Precisions_ , _Recalls_ , and _F1-scores_ of Stratified 10-fold Cross Validation of Our approach. The highest scores are in bold. | LSTM | P | R | F1
---|---|---|---|---
| Latent | Layers | Batch | Pool
1 | 32 | 2 | 256 | mean | 0.537 | 0.915 | 0.674
2 | 32 | 1 | 256 | max | 0.555 | 0.867 | 0.673
3 | 16 | 2 | 64 | mean | 0.521 | 0.951 | 0.672
4 | 8 | 2 | 16 | mean | 0.518 | 0.960 | 0.671
5 | 32 | 2 | 256 | max | 0.536 | 0.909 | 0.670
6 | 64 | 1 | 64 | max | 0.577 | 0.812 | 0.666
7 | 32 | 1 | 256 | / | 0.514 | 0.912 | 0.655
8 | 8 | 2 | 16 | / | 0.538 | 0.853 | 0.648
9 | 32 | 2 | 256 | / | 0.536 | 0.854 | 0.648
| ML Algorithm | P | R | F1
10 | Multinomial Naive Bayes | 0.612 | 0.665 | 0.637
11 | Support Vector Machines | 0.658 | 0.592 | 0.623

####  5.4.1. The Benchmark

Zampetti et al. (Zampetti et al., 2017) developed a Random-Forest-based
approach called TEDIOuS. When a developer writes a new piece of code, TEDIOuS
provides recommendation to them suggesting when they should self-admit
“design” technical debt. To the best of our knowledge, this is the only
existing work the field of satd that analyses source code. Unlike our
approach, they used source code metrics as features instead of using the raw
source code tokens. We use their approach as a benchmark for our experiment.
Table 2 orders our approach’s results and the benchmark’s results based on the
_F1-Score_ , the highest first. The table orders the pre-training results as
well. Pre-training is explained in Section 5.4.2.

####  5.4.2. Pre-Training

We wanted to see if pre-training can help initialise proper model weights for
the main training time instead of random initialisation. The negligible
difference between the deep learning and traditional machine learning results
shown in Table 1 fuelled our motivation to attempt pre-training.We tried two
pre-training methods: _end2end_ , and embedding pre-training with traditional
machine learning. In _end2end_ , we train an lstm model on predicting the next
source code token. This results in a pre-trained embedding layer and lstm
layers. When we train the model for satd Source Code Identification, the
weights of the layers are not randomly initialised anymore. We also tried
another pre-training method. For pre-training in this style, we also train an
lstm model on predicting the next source code token (just identical to the
_end2end_ pre-training). However, when we train the main model for satd Source
Code Identification, we only use the pre-trained embedding layer. We extract
the vector of each token from that pre-trained embedding layer. We then
represent a source code input by the average pooling of its tokens’
embeddings. The resulting vector is input to the traditional machine learning
model used for satd Source Code Identification training. We tried different
traditional machine learners and found that svm is the best performing one.
Table 2 orders our approach’s results with and without pre-training based on
the _F1-Score_ , the highest first. The table also orders the results of the
benchmark for a wider perspective.

Table 2. Average results of Stratified 10-fold Cross Validation in comparison with 2 pre-training methods and the benchmark. The highest _Precision_ , _Recall_ , and _F1_ scores are in bold. DLM: Deep Learning Model. TMLM: Traditional Machine Learning Model. RF: Random Forest. | Model | P | R | F1
---|---|---|---|---
1 | Our DLM | 0.537 | 0.915 | 0.674
2 | Our Pre-Trained DLM | 0.534 | 0.901 | 0.668
3 | Our TMLM | 0.612 | 0.665 | 0.637
4 | Ours: Pre-Trained | 0.540 | 0.666 | 0.596
Embeddings with TMLM
5 | Benchmark: RF | 0.531 | 0.667 | 0.590

###  5.5. SATD Comment Identification

In this section, we evaluate the performance of the deep learning component
for satd Comment Identification. As there are already multiple studies that
evaluated traditional machine learning approaches, we did not repeat this
experiment here. Instead, we use the state-of-art approaches as benchmarks to
compare our deep learning model against them.

We evaluate and compare our approach on a publicly available dataset of
comments from eight open source software projects (Huang et al., 2018) through
a number of validation rounds. In each validation round, the comments of one
of the eight projects are used for testing. The comments of the other seven
project are used for training. Hence, there will be eight validation rounds
for each project.

Although we have experimented our lstm model with different hyper-parameter
settings, we report here the results of the best performing one999full report
is at https://drive.google.com/drive/folders/1SrUcUw2ORY8oCXv-
FHtmReaxa2zM80cG?usp=sharing. Our deep learning model outperforms the state-
of-the-art consistently in all hyper-parameter tuning settings. The best
hyper-parameter setting for our lstm comment detector is as follows: _Latent
Dimensionality_ : 256, _Number of Layers_ : 2, _Batch Size_ : 512 and _Pooling
technique_ : Max.

We benchmark with the results of three approaches from the state-of-the-art,
namely TM, TM (BesSub) and NLP. TM and TM (BestSub) are two approaches for
satd Comment Identification proposed by Huang et al. (Huang et al., 2018). NLP
is another satd Comment Identification approach proposed by Maldonado et al.
(da Silva Maldonado et al., 2017). Table 3 demonstrates the superiority of our
deep learning approach.

Table 3. _F-1_ scores of our approach compared with benchmark approaches. The highest score for each project is in bold. DLM: Deep Learning Model. TM: Text Mining. BestSub: Best Sub-Classifier. NLP: Natural Language Processing. | Project | Our DLM | TM | TM (BestSub) | NLP
---|---|---|---|---|---
1 | ArgoUML | 0.851 | 0.828 | 0.756 | 0.802
2 | Columba | 0.866 | 0.801 | 0.649 | 0.682
3 | Hibernate | 0.834 | 0.788 | 0.693 | 0.587
4 | jEdit | 0.614 | 0.518 | 0.459 | 0.459
5 | JFreeChart | 0.748 | 0.687 | 0.524 | 0.500
6 | JMeter | 0.794 | 0.781 | 0.617 | 0.557
7 | JRuby | 0.846 | 0.841 | 0.724 | 0.584
8 | SQuirrel | 0.723 | 0.651 | 0.515 | 0.435
| Average | 0.784 | 0.737 | 0.591 | 0.576

###  5.6. SATD Comment Generation

The experimental setup of our lstm Generator differs from the Detector’s due
to two reasons. First, the time and space complexity of the Generator is
higher than that of the Detector. Secondly, the Generator reports distinctive
results with different hyper-parameter settings.We experimented with the
following sets of hyper-parameters: _Latent Dimensionality_ : (512, 1024,
2048), _Number of Layers_ : (1, 2) and _Batch Size_ : (32, 64, 128).

####  5.6.1. Automatic Evaluation - BLEU

Table 4 lists the results from both the hyper-parameter tuning and the 10-fold
cv steps, oredered according to _BLEU-4_ , the highest first. Moreover, Table
5 shows some satd comments generated by our framework in comparison with
human-scripted satd comments. The first example shows a generated comment that
is identical to the human-scripted comment. The second example shows minor
differences, while the third example shows a comment that is totally different
from the human-scripted comment.

Table 4. _BLEU_ scores of the hyper-parameter tuning step followed by the
average _BLEU_ scores of the 10-fold Cross Validation step. The highest _BLEU_
scores are in bold. Hyper-Parameter Tuning
---
| Latent | Layers | Batch | B-1 | B-2 | B-3 | B-4
1 | 1024 | 1 | 64 | 0.156 | 0.107 | 0.091 | 0.083
2 | 1024 | 1 | 32 | 0.145 | 0.103 | 0.088 | 0.082
3 | 2048 | 1 | 32 | 0.133 | 0.091 | 0.077 | 0.070
4 | 512 | 1 | 64 | 0.117 | 0.055 | 0.033 | 0.023
5 | 1024 | 2 | 32 | 0.100 | 0.011 | 0.003 | 0.000
10-Fold cv
| 1024 | 1 | 64 | 0.182 | 0.140 | 0.125 | 0.118
Table 5. Sample model-generated satd comments compared with human-scripted comments Java Code | Comment
---|---
Human-Written Comments
if(number == 0) { PrismObject <UserType> jack... | // todo e g check metadata
if(StringUtilities.isNumeric(itemName)) {... | // workaround issue user types in something like mallsell pail
if(!empty) mView.getControl().handleOwnChatStateEvent... | // workaround clearing the text area is not a key event
Model-Generated Comments
if(number == 0) { PrismObject <UserType> jack... | // todo e g check metadata
if(StringUtilities.isNumeric(itemName)) {... | // workaround issue user types in something like shop reprice pail
if(!empty) mView.getControl().handleOwnChatStateEvent... | // hack for empty arrays

####  5.6.2. Human Evaluation - Acceptability and Understandability

To evaluate if the generated comments could be easy to understand, especially
for inexperienced programmers. We performed a human evaluation between two
authors using two criteria, that is, (1) acceptability, and (2)
understandability (Oda et al., 2015). We assigned a 5-level score (from 1 to
5) to indicate the acceptance of the generated comment, and a 6-level score
(from 0 to 5) to show how well the annotators are in understanding the
generated comments.

Using those criteria, the third and fourth authors evaluated 337 randomly
selected samples independently. The result of the evaluation shows that the
comments generated by our models are relatively acceptable and understandable
by human. This is indicated by the mean score in both acceptability and
understandability at 3.128 and 3.172 respectively. The assessment results
between individual evaluators are quite similar. It can be seen in high
positive correlation values, that is, 0.791 for the acceptability, and 0.783
for the understandability.

##  6\. Discussion

The deep learning component performed only slightly better than the
traditional machine learning component did for satd Source Code
Identification. Thus, we would recommend using the traditional machine
learning component due to its significant small training time (a few seconds
compared to 1.5 hour in training the deep learning component). We also
recommend the use of pooling techniques (e.g. _mean-pooling_ or _max-pooling_)
since they tend to improve the model’s performance.

While pre-training is commonly used in practice, we do not recommend it for
satd Source Code Identification. Pre-training is an alternative to random
weight initialisation. However, we have found that in our setting pre-training
did not help improve the model’s performance although it took a substantial
amount of time.

A key difference between our approach and the benchmark approach in satd
Source Code Identification is the input to the model. The benchmark approach
uses source code metrics as input features. On the other hand, our approach
uses the raw source code tokens as features, hence is able to learn the
semantic and syntactic structure of the code. This benefits the model as can
be seen in the improved performance brought by our approach.

Finally, we would recommend the use of deep learning for satd Comment
Identification since it shows consistent improvement and superiority. This
recommendation is consistent with the finding in a recent work (Ren et al.,
2019) (which uses cnn instead of LSTM) for satd Comment Identification. Hence,
relating to satd Identification, traditional machine learning is our suggested
approach for Source Code Identification while deep learning is our suggested
approach for Comment Identification.

###  6.1. Threats to validity

Threats to internal validity relate to bias in data labelling and errors in
the implementation of the framework components. To mitigate this threat, we
conducted an investigation of the collected dataset (Section 5.1.1). This
investigation is to ensure that the satd comments represent technical debt in
source code.

We have performed extensive hyper-parameter tuning experiments for all of the
framework components. In addition, for the satd Identification experiments, we
have tested multiple hyper-parameter settings in the cross-validation step
instead of only selecting the best performing setting from the hyper-parameter
tuning step.

Threats to external validity relate to the quality and quantity of our
dataset. We have collected code-comment pairs from 4,995 active software
development repositories. From these repositories, we obtained 5,313 satd
code-comment pairs and 839,431 non-satd pairs. Furthermore, we evaluated our
deep learning component on another dataset of 8 open source projects and their
31,870 comments. Thus, we believe that this threat is minimal.

Since all the data is collected from open-source projects and repositories,
there is an open question about the generalisability of our approach to
company projects. However, developers in companies are less prone to self-
admitting technical debt they introduce (Huang et al., 2018; Ren et al.,
2019), which lessens the threat to our approach’s generalisability.
Furthermore, all of the analysed projects and repositories are implemented in
Java. Future work would involve exploring our approach with larger and more
diverse datasets.

##  7\. Related Work

Potdar and Shihab (Potdar and Shihab, 2014) proposed the concept Self-Admitted
Technical Debt (SATD) to describe technical debt that is intentionally
introduced and documented in the comments that accompany source code. They
manually inspected 100k comments and extracted 62 patterns that are actively
used in following studies to detect satd. Research on Self-Admitted Technical
Debt ever since has expanded (Maldonado and Shihab, 2015; Wehaibi et al.,
2016; Bavota and Russo, 2016; de Freitas Farias et al., 2015; de Freitas
Farias et al., 2016; Maldonado et al., 2017; Zampetti et al., 2017; Zampetti
et al., 2018; Yan et al., 2018; da Silva Maldonado et al., 2017; Huang et al.,
2018; Ren et al., 2019). Maldonado and Shihab (Maldonado and Shihab, 2015)
examined 33k source code comments and determined five types of satd: design
debt, defect debt, documentation debt, requirement debt, and test debt. They
found that design debt is the most common one. Farias et al. (de Freitas
Farias et al., 2015; de Freitas Farias et al., 2016) developed a cvmtd to
identify satd in source code comments by analysing code tags and words’ parts
of speech.

Zampetti et al. (Zampetti et al., 2017) developed a Random-Forest-based
approach called TEDIOuS. TEDIOuS recommends to developers when they should
self-admit design debt. Zampetti et al. use static analysis tools to calculate
method-level metrics in order to create the model input features. These
metrics are structural metrics, readability metrics, and warnings raised by
static analysis tools. Our framework is inspired by their Zampetti et al.’s
work. We use their approach as a benchmark for our satd Source Code
Identification components. However, there are key differences between Zampetti
et al.’s approach and ours. Firstly, they calculate source code metrics and
use them as features for the machine learning model to learn from. In our
case, we use the source code tokens themselves rather than calculating metrics
to create the feature vectors. Secondly, their work specialises in detecting
“design” debt. Ours attempts to detect all kinds of debt. Thirdly, Their work
operates on method-level. Ours can operates on any level (whether it is
conditional-statement-level, method-level, class-level, etc.) as long as an
Abstract Syntax Tree can be extracted from the targeted code fragment. Lastly,
We propose a comprehensive framework that also performs other tasks, such as
satd Comment Identification and satd Comment Generation, and leverages deep
learning as well as traditional machine learning.

Maldonado et al. (da Silva Maldonado et al., 2017) propose a nlp approach to
identify the two most common types of satd: design debt and requirement debt.
They employ a maximum entropy classifier to identify the satd comments. Huang
et al. (Huang et al., 2018) used text mining techniques to detect all types of
satd. They utilise feature selection to select useful features for classifier
training. There work outperformed the nlp approach. Note that a limitation of
Maldonado et al.’s work is its speciality in only two satd types rather than
all types. In addition, they did not apply a feature selection technique. A
key difference between Huang et al.’s approaches and ours, other than us
leveraging deep learning, is that they dedicate a sub-classifier for each one
of the eight projects, while we train one comprehensive classifier for all the
projects. Moreover, we propose a comprehensive framework that handles the case
when the satd comment is missing.

In their recent work, Ren et al. (Ren et al., 2019) Proposed a deep learning-
based approach in the form of Convolutional Neural Networks (CNN) for satd
Comment Identification. They argue that the leveraging CNN improves the
detection task over other existing approaches. Furthermore, exploiting the
computational structure of CNN improves the explainability of the results by
identifying key patterns of satd comments. We plan to compare our approach
against theirs and explore the possibility of combining aspects of the two
approaches together and examine the performance. However, as in the other
related work, Ren et al.’s work does not consider the case when the comment is
unavailable. Our comprehensive framework detects source code segments that
need satd comments and generates the appropriate satd comment.

##  8\. Conclusion and Future Work

In this paper, we propose a comprehensive framework for Self-Admitted
Technical Debt handling. This framework takes into account the unavailability
code comments. If the comment is provided, the framework takes advantage of it
and determines if it self-admits technical debt. If the comment is not
provided, the framework analyses the source code to detect potential technical
debt and generates the appropriate satd comment.

We evaluate the framework’s performance on a dataset of code-comment pairs
from 4,995 active software development repositories as well as publicly
available dataset (Huang et al., 2018). Framework Evaluation includes
evaluating its three intelligent features, namely satd Source Code
Identification, satd Comment Identification, and satd Comment Generation. For
the Identification tasks, The results demonstrate the applicability and
advantage of our approach over multiple benchmarks from the state-of-the-art.
For the Generation task, we present the novelty of our approach. The results
demonstrate the acceptability and understandability of the generated satd
comments.

The future work of our work can be directed towards investigating larger and
more diverse datasets. This includes studying other programming languages and
company (private) projects when accessible. Moreover, we plan to develop an
automated tool for satd identification and comment generation. Furthermore, we
plan to benchmark/adopt other deep learning techniques as well as fine-tune
our deep learning approach in order to further validate its usage over
traditional machine learning approaches. Finally, we plan to investigate
machine/deep Learning approaches beyond satd toleration into satd repayment
(i.e. removal).

## References

  * (1)
  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015\.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/ Software available from tensorflow.org.
  * Aggarwal and Zhai (2012) Charu C Aggarwal and ChengXiang Zhai. 2012.  _Mining text data_.  Springer Science & Business Media.
  * Aniche et al. (2018) Maurício Aniche, Gabriele Bavota, Christoph Treude, Marco Aurélio Gerosa, and Arie Deursen. 2018.  Code Smells for Model-View-Controller Architectures.  _Empirical Software Engineering_ 23, 4 (2018), 2121–2157.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Bavota and Russo (2016) Gabriele Bavota and Barbara Russo. 2016.  A large-scale empirical study on self-admitted technical debt. In _Mining Software Repositories (MSR), 2016 IEEE/ACM 13th Working Conference on_. IEEE, 315–326.
  * Bingham and Mannila (2001) Ella Bingham and Heikki Mannila. 2001.  Random projection in dimensionality reduction: applications to image and text data. In _Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining_. 245–250.
  * Brown et al. (2010) Nanette Brown, Yuanfang Cai, Yuepu Guo, Rick Kazman, Miryung Kim, Philippe Kruchten, Erin Lim, Alan MacCormack, Robert Nord, Ipek Ozkaya, et al. 2010\.  Managing technical debt in software-reliant systems. In _Proceedings of the FSE/SDP workshop on Future of software engineering research_. ACM, 47–52.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ (2014).
  * Choetkiertikul et al. (2018) Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Thi Minh Pham, Aditya Ghose, and Tim Menzies. 2018.  A deep learning model for estimating story points.  _IEEE Transactions on Software Engineering_ (2018).
  * Chollet (2017) Francois Chollet. 2017\.  _A ten-minute introduction to sequence-to-sequence learning in Keras_.  https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html
  * Chollet et al. (2015) François Chollet et al. 2015\.  Keras.  https://keras.io.
  * Cunningham (1993) Ward Cunningham. 1993\.  The WyCash portfolio management system.  _ACM SIGPLAN OOPS Messenger_ 4, 2 (1993), 29–30.
  * da Silva Maldonado et al. (2017) Everton da Silva Maldonado, Emad Shihab, and Nikolaos Tsantalis. 2017\.  Using natural language processing to automatically detect self-admitted technical debt.  _IEEE Transactions on Software Engineering_ 43, 11 (2017), 1044–1062.
  * de Freitas Farias et al. (2015) Mário André de Freitas Farias, Manoel Gomes de Mendonça Neto, André Batista da Silva, and Rodrigo Oliveira Spínola. 2015\.  A Contextualized Vocabulary Model for identifying technical debt on code comments. In _Managing Technical Debt (MTD), 2015 IEEE 7th International Workshop on_. IEEE, 25–32.
  * de Freitas Farias et al. (2016) Mário André de Freitas Farias, José Amancio M Santos, André Batista da Silva, Marcos Kalinowski, Manoel G Mendonça, and Rodrigo Oliveira Spínola. 2016\.  Investigating the Use of a Contextualized Vocabulary in the Identification of Technical Debt: A Controlled Experiment.. In _ICEIS (1)_. 369–378.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  A theoretically grounded application of dropout in recurrent neural networks. In _Advances in neural information processing systems_. 1019–1027.
  * Gousios (2013) Georgios Gousios. 2013\.  The GHTorent Dataset and Tool Suite. In _Proceedings of the Working Conference on Mining Software Repositories_. 233–236.
  * Hata et al. (pear) Hideaki Hata, Christoph Treude, Raula Gaikovina Kula, and Takashi Ishio. (to appear).  9.6 Million Links in Source Code Comments: Purpose, Evolution, and Decay. In _Proc. of 41st ACM/IEEE International Conference on Software Engineering (ICSE 2019)_.
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018.  Deep code comment generation. In _Proceedings of the 26th Conference on Program Comprehension_. ACM, 200–210.
  * Huang et al. (2018) Qiao Huang, Emad Shihab, Xin Xia, David Lo, and Shanping Li. 2018.  Identifying self-admitted technical debt in open source projects using text mining.  _Empirical Software Engineering_ 23, 1 (2018), 418–451.
  * Joachims (1998) Thorsten Joachims. 1998\.  Text categorization with support vector machines: Learning with many relevant features. In _European conference on machine learning_. Springer, 137–142.
  * Kaufman et al. (2012) Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitelman. 2012.  Leakage in data mining: Formulation, detection, and avoidance.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 6, 4 (2012), 15.
  * Kibriya et al. (2004) Ashraf M Kibriya, Eibe Frank, Bernhard Pfahringer, and Geoffrey Holmes. 2004.  Multinomial naive bayes for text categorization revisited. In _Australasian Joint Conference on Artificial Intelligence_. Springer, 488–499.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kruchten et al. (2019) Philippe Kruchten, Robert Nord, and Ipek Ozkaya. 2019.  _Managing Technical Debt: Reducing Friction in Software Development_.  Addison-Wesley Professional.
  * Lim et al. (2012) Erin Lim, Nitin Taksande, and Carolyn Seaman. 2012.  A balancing act: What software practitioners have to say about technical debt.  _IEEE software_ 29, 6 (2012), 22–27.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015\.  Effective approaches to attention-based neural machine translation.  _arXiv preprint arXiv:1508.04025_ (2015).
  * Maldonado et al. (2017) Everton da S Maldonado, Rabe Abdalkareem, Emad Shihab, and Alexander Serebrenik. 2017.  An empirical study on the removal of self-admitted technical debt. In _Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on_. IEEE, 238–248.
  * Maldonado and Shihab (2015) Everton da S Maldonado and Emad Shihab. 2015.  Detecting and quantifying different types of self-admitted technical debt. In _2015 IEEE 7th International Workshop on Managing Technical Debt (MTD)_. IEEE, 9–15.
  * Matsumoto and Nishimura (1998) Makoto Matsumoto and Takuji Nishimura. 1998.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator.  _ACM Transactions on Modeling and Computer Simulation (TOMACS)_ 8, 1 (1998), 3–30.
  * McCallum et al. (1998) Andrew McCallum, Kamal Nigam, et al. 1998\.  A comparison of event models for naive bayes text classification. In _AAAI-98 workshop on learning for text categorization_ , Vol. 752. Citeseer, 41–48.
  * Munaiah et al. (2017) Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017.  Curating GitHub for Engineered Software Projects.  _Empirical Software Engineering_ 22, 6 (2017), 3219–3253.
  * Oda et al. (2015) Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015.  Learning to generate pseudo-code from source code using statistical machine translation (t). In _Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on_. IEEE, 574–584.
  * Potdar and Shihab (2014) Aniket Potdar and Emad Shihab. 2014.  An exploratory study on self-admitted technical debt. In _Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on_. IEEE, 91–100.
  * Ren et al. (2019) Xiaoxue Ren, Zhenchang Xing, Xin Xia, David Lo, Xinyu Wang, and John Grundy. 2019\.  Neural Network-based Detection of Self-Admitted Technical Debt: From Performance to Explainability.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 28, 3 (2019), 1–45.
  * Salton et al. (1975) Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975\.  A vector space model for automatic indexing.  _Commun. ACM_ 18, 11 (1975), 613–620.
  * scikit learn ([n.d.]) scikit learn. [n.d.].  _Cross-validation: evaluating estimator performance_.  https://scikit-learn.org/stable/modules/cross_validation.html#stratified-k-fold
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ 15, 1 (2014), 1929–1958.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014\.  Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_. 3104–3112.
  * Tom et al. (2013) Edith Tom, AybüKe Aurum, and Richard Vidgen. 2013\.  An exploration of technical debt.  _Journal of Systems and Software_ 86, 6 (2013), 1498–1516.
  * Viera et al. (2005) Anthony J Viera, Joanne M Garrett, et al. 2005\.  Understanding interobserver agreement: the kappa statistic.  _Fam Med_ 37, 5 (2005), 360–363.
  * Wehaibi et al. (2016) Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. 2016\.  Examining the impact of self-admitted technical debt on software quality. In _Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on_ , Vol. 1. IEEE, 179–188.
  * Yan et al. (2018) Meng Yan, Xin Xia, Emad Shihab, David Lo, Jianwei Yin, and Xiaohu Yang. 2018\.  Automating Change-level Self-admitted Technical Debt Determination.  _IEEE Transactions on Software Engineering_ (2018).
  * Zampetti et al. (2017) Fiorella Zampetti, Cedric Noiseux, Giuliano Antoniol, Foutse Khomh, and Massimiliano Di Penta. 2017\.  Recommending when design technical debt should be self-admitted. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 216–226.
  * Zampetti et al. (2018) Fiorella Zampetti, Alexander Serebrenik, and Massimiliano Di Penta. 2018.  Was self-admitted technical debt removal a real removal? an in-depth perspective. In _2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR)_. IEEE, 526–536.
  * Zazworka et al. (2011) Nico Zazworka, Michele A Shaw, Forrest Shull, and Carolyn Seaman. 2011.  Investigating the impact of design debt on software quality. In _Proceedings of the 2nd Workshop on Managing Technical Debt_. ACM, 17–23.