2018 ACM/IEEE 40th International Conference on Software Engineering: Companion Proceedings

SATD Detector: A Text-Mining-Based Self-Admitted Technical

Debt Detection Tool
Zhongxin Liu Qiao Huang Xin Xia
Zhejiang University, China Zhejiang University, China Monash University, Australia
[email protected] [email protected] [email protected]

Emad Shihab David Lo Shanping Li

Concordia University, Canada Singapore Management University, Zhejiang University, China
[email protected] Singapore [email protected]
[email protected]

ABSTRACT
In software projects, technical debt metaphor is used to describe
the situation where developers and managers have to accept com- ACM Reference Format:
promises in long-term software quality to achieve short-term goals. Zhongxin Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, and Shanping
There are many types of technical debt, and self-admitted technical Li. 2018. SATD Detector: A Text-Mining-Based Self-Admitted Technical
debt (SATD) was proposed recently to consider debt that is intro- Debt Detection Tool. In ICSE ’18 Companion: 40th International Conference
duced intentionally (e.g., through temporaryfi x) and admitted by on Software Engineering , May 27-June 3, 2018, Gothenburg, Sweden. ACM,
developers themselves. Previous work has shown that SATD can New York, NY, USA, 4 pages. https://doi.org/10.1145/3183440.3183478
be successfully detected using source code comments. However,
most current state-of-the-art approaches identify SATD comments 1 INTRODUCTION
through pattern matching, which achieve high precision but very In real-world software projects, developers and managers some-
low recall. That means they may miss many SATD comments and times have to make tradeoffs between long-term code quality and
are not practical enough. In this paper, we propose SATD Detector, short-term revenue due to various reasons (e.g., cost reduction,
a tool that is able to (i) automatically detect SATD comments using market pressure, and tight project schedule). Technical debt, which
text mining and (ii) highlight, list and manage detected comments is introduced by Cunningham [2], is a metaphor used to describe
in an integrated development environment (IDE). This tool consists this kind of situation. It has been shown by prior work that tech-
of a Java library and an Eclipse plug-in. The Java library is the nical debt is common, unavoidable and may degrade code quality
back-end, which provides command-line interfaces and Java APIs and increase software complexity in the future [5, 11]. Moreover,
to re-train the text mining model using users’ data and automati- technical debt is not always visible, i.e., it may only be known to
cally detect SATD comments using either the build-in model or a some specific people but not those who eventually pay for it. There-
user-specified model. The Eclipse plug-in, which is the front-end, fore, many studies have been conducted to enable the detection
first leverages our pre-trained composite classifier to detect SATD and management of technical debt.
comments, and then highlights and marks these detected comments The concept of self-admitted technical debt (SATD) is proposed
in the source code editor of Eclipse. In addition, the Eclipse plug-in by Potdar and Shihab [8], which considers the technical debt that is
provides a view in IDE which collects all detected comments for intentionally introduced (e.g., in the form of temporary workaround)
management. and admitted by developers themselves. In particular, self-admitted
Demo URL: https://youtu.be/sn4gU2qhGm0 technical debt is used to describe the situation where developers
Java library download: https://git.io/vNdnY know that current implementation is not optimal and record this
Eclipse plug-in download: https://goo.gl/ZzjBzp in source code comments. For example, one comment in the open
source project “JEdit” mentions that “Need some format checking
CCS CONCEPTS here”. This comment indicates that developers admitted that the
• Software and its engineering → Software maintenance tools; corresponding code is defective and requires format checking. A
previous study [11] shows that although the percentage of SATD in
KEYWORDS a project is not high, it can negatively impact the maintenance of a
Self-admitted technical debt, SATD detection, Eclipse plug-in project. Detecting and managing SATD can remind developers and
managers about the existence of SATD, help them plan to discharge
Permission to make digital or hard copies of part or all of this work for personal or it and hence result in software quality improvement.
classroom use is granted without fee provided that copies are not made or distributed Prior work also shows that SATD can be successfully detected
for profit or commercial advantage and that copies bear this notice and the full citation
on thefi rst page. Copyrights for third-party components of this work must be honored. using source code comments [8]. However, most of the previous
For all other uses, contact the owner/author(s). studies detected SATD by manually classifying comments [8] or
ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden using the 62 SATD comment patterns [1, 11] which are manually
© 2018 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-5663-3/18/05. derived by Potdar and Shihab [8]. Approaches that involve manual
https://doi.org/10.1145/3183440.3183478 classification of comments require much human effort, and thus

9
 ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Liu et al.

%& 
   




 
 




 




  
 * )
 

  
' ( 





   






  


   
+

  

  

 

 
 
!"# $

Figure 1: Overall Framework of Our Model

are not practical for real-world projects. Although pattern-based outperforms Maldonado and Shihab’s approach [7] by a substantial
approaches can achieve high precision, their recall is often very margin in terms of F1-score.
low since they fail to detect SATD comments which do not match The remainder of the paper is organized as follows. In Section
any known patterns. This is the case since it is difficult to extract all 2, we present the text-mining-based model used to detect SATD
potential SATD comment patterns. Most recently, Maldonado et al. comments by our tool. The details of SATD Detector, including the
proposed an approach based on natural language processing (NLP) usage of the Java library, the workflow, life cycle and user interface
to automatically identify different types of SATD comments [6]. of the Eclipse plug-in, are described in Section 3. Section 4 shows
However, their work only focuses on certain types of SATD (i.e., the experimental results of our evaluation. We conclude our work
design debt, requirement debt or non-SATD), while we care more and mention future work in Section 5.
about whether a comment contains SATD or not, which also in-
cludes other types of SATD (i.e., defect debt, documentation debt
and test debt). Moreover, no prior work provides practical tools to 2 APPROACH
help developers detect and manage SATD in an IDE. 2.1 Overall Framework
In this paper, we present SATD Detector, a tool based on our
In general, SATD Detector leverages a pre-trained text mining
previous work [4]. This tool is able to (i) automatically detect SATD
model to automatically predict whether a comment contains SATD
comments in source code through a text-mining-based approach
or not. The pre-trained model is the composite classifier proposed
and (ii) list and manage detected comments inside an IDE. It con-
in our previous work [4]. Figure 1 presents the overall framework
tains two parts: a Java library and an Eclipse plug-in. The Java
of our model. It contains two phases: a model building phase and
library provides command-line interfaces and Java APIs. Through
a prediction phase. We refer to the projects which are used to
these interfaces, users can train the text mining model using their
build the model as source projects, and the projects we want to
own data and leverage either the build-in model or their own mod-
detect as target projects. In the model building phase, our approach
els to identify SATD comments. The Eclipse plug-in, which is the
builds a sub-classifier using data from each source project. In the
front-end of our tool, uses our pre-trained composite classifier to
prediction phase, all sub-classifiers are combined to jointly predict
make detection after a project is imported into Eclipse. Specifically,
SATD comments in the target project.
whenever a developer opens Eclipse, our plug-in will automatically
Our framework takes as input training comments with known
parse all source codefi les, detect and mark the comments which
labels from different source projects. For each source project, we
contains SATD. Once somefi les are modified, it will re-detect SATD
first preprocesses the text descriptions of comments and extracts
comments and update markers in thesefi les immediately. More-
features (i.e., words) to represent each comment (Step 1). Then,
over, our plug-in also provides an Eclipse view in which all detected
feature selection is applied to select features that are useful for
SATD comments are listed for management. Our tool is easy to
classification and useless features are removed (Step 2). Next, we
deploy and use. With the help of the Eclipse plug-in, it would be
use the selected features to train a sub-classifier for the target
easy for developers and managers to manage SATD and pay back
project (Step 3). Suppose there are n source projects, we end up
it in time. In addition, using our Java library, users can train and
with n classifiers which are combined to form a composite classifier
leverage their own model, and integrate SATD Detector into their
for prediction (Step 4). For each new comment in the target project,
development tools (e.g., other IDEs or continuous integration tools).
wefi rst preprocess the comment to extract features (Step 5) and
To build and evaluate our tool, we use a manually classified
then input features to the composite classifier (Step 6). Finally, each
dataset of source code comments from 8 open source projects with
sub-classifier will predict the label of the comment according to
212,413 comments, provided by Maldonado and Shihab [7]. The
its features, and the label with the largest number of “votes” will
experimental results show that, on every target project, our tool

10
SATD Detector: A Text-Mining-Based SATD Tool ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden

be chosen as thefi nal prediction result of the composite classifier 

(Step 7). 

       

 

2.2 Model Details
Our model mainly contains four steps: text preprocessing, feature Figure 2: Workflow of Our Eclipse Plug-in
selection, sub-classifiers training and classifiers voting. The follow-
ing paragraphs elaborate the details of the four steps:
3 SATD DETECTOR
Text Preprocessing: We preprocess the text description of com- 3.1 The Java Library
ments to extract features (i.e., words) in 3 steps: tokenization, stop- The Java Library in our tool provides command-line interfaces
word removal, and stemming. While tokenizing, we only keep as well as Java APIs. Both of them provide the ability to re-train
English letters in a token and convert all words to lowercase. As the model using user-specified data and detect SATD comments
for stop-word removal, since some stop words are useful for clas- through our pre-trained model or models re-trained by users. The
sification (e.g., “should”), we manually build a list of stop-words detailed manual of this library can be found in our GitHub reposi-
tofi lter stop-words. Words whose lengths are no more than 2 or tory2 .
no less than 20 are also treated as stop-words. Finally, each token
is stemmed (i.e., reduced to its root form) using the well-known 3.2 The Eclipse Plug-in
Porter stemmer1 . 3.2.1 Workflow. Figure 2 shows the workflow of our Eclipse
Feature Selection: After preprocessing and tokenizing the com- plug-in. First of all, it parses the source codefi les in the workspace,
ments, we use the Vector Space Model (VSM) [10] to represent each and extracts comments from them. Then, it leverages regular ex-
comment with a word vector. In total, we have a large number of pressions to remove irrelevant comments which mainly include the
features for each source project (e.g., there are 3,661 features in following two types:
ArgoUML project). Feature selection is applied to identify a subset (1) Automatically generated comments withfi xed format (i.e.,
of features that are most useful in differentiating different classes Auto-generated constructor stubs, auto-generated method
(i.e., SATD comment or not). In this model, we employ Information stubs and auto-generated catch blocks), which are inserted
Gain (IG) [9, 13] to select useful features. Only the features whose as part of code snippets by Eclipse to generate constructors,
feature selection scores are in the top 10% of the ranked list are methods, and try catch blocks.
retained, and the other features are removed. (2) Javadoc and license comments which do not contain any
Sub-classifiers Training: In our tool, we train each sub-classifier task annotation (i.e., “TODO”, “FIXME”, or “XXX”) [9].
using Naive Bayes Multinomial (NBM), which is widely used to Next, the rest of comments are inputted to the pre-trained text
analyze text data in software engineering [12, 14–16]. We use the mining model which is described in Section 2 and implemented in
implementation of NBM in Weka [3] with default settings. Note our back-end library. Each comment will be classified by the model.
that our approach can also work with other classifiers. Finally, for comments which are predicted to contain SATD, our
Classifiers Voting: In our model, the composite classifier is built plug-in will post-process them in the source code editor of Eclipse,
from all the sub-classifiers, and it is responsible for predicting the e.g., highlight them and add markers for them.
label of a new comment in the target project. The prediction process 3.2.2 Life Cycle. After installation, our plug-in will start with
is just like an election, and the prediction result of each sub-classifier Eclipse and then parse source codefi les in the whole workspace
is regarded as a “vote”. The comment label which gets the largest in background. Since in most cases, users only care about SATD
number of “votes” will be thefi nal prediction result of the composite in their own projects, our plug-in ignores thefi les in the third
classifier. party libraries. In our current implementation, the Eclipse plug-in
only supports Java projects, and it will not parse non-Java source
2.3 Dataset codefi les. But our back-end library only cares about source code
comments, so it is not limited to Java projects. Once source code
We use the dataset provided by the authors of [7] to build our model
files are modified, they will be re-parsed and our plug-in will re-
and evaluate its performance. The dataset contains comments ex-
detect SATD in thesefi les immediately. The markers created by
tracted from 8 open source projects, which are ArgoUML, Columba,
our plug-in for SATD comments will not be persisted; hence while
Jmeter, JFreeChart, Hibernate, JEdit, JRuby, SQuirrel, and the label
users are exiting Eclipse, these markers will be deleted and SATD
of each comment, i.e., SATD comment or not. All the labels are
Detector will then stop.
manually labeled by the authors of [7], who reported a high level of
agreement on the classification results. Therefore, we are confident 3.2.3 User Interface. Figure 3 presents the user interface (UI) of
in the quality of the provided dataset. More information about the our Eclipse plug-in. It follows the workflow shown in Figure 2 to
dataset can be found in our previous work [4]. detect SATD comments. Once it identifies one comment with SATD,
it will highlight this comment ( 1 in Figure 3) and add a marker for
this comment ( 2 in Figure 3) in the editor. At the same time, we
can check currently detected SATD comments in an Eclipse view
1 http://tartarus.org/martin/PorterStemmer
2 https://github.com/Tbabm/SATDDetector-Core

11
ICSE ’18 Companion, May 27-June 3, 2018, Gothenburg, Sweden Liu et al.

 The experimental results show that, on every target project

 our approach achieves the best performance in terms of F1-score,


and outperforms the baseline approaches by a substantial margin.
After observing the dataset, wefi nd that different projects write
SATD comments in different ways. Training sub-classifiers and
 combining them through the voting mechanism can reduce the
bias to certain kind of SATD comments, and thus improve the

performance. Readers can refer to our previous work [4] for more
details of our evaluation and experimental results.

5 CONCLUSION & FUTURE WORK

In this paper, we present SATD Detector, a tool that is able to
Figure 3: User Interface of Our Eclipse Plug-in automatically detect SATD comments, and help developers manage
them in an IDE. This tool consists of a back-end Java library and a
front-end Eclipse plug-in. Through the back-end library, users can
re-train the text mining model and integrate SATD Detector into



other development tools easily. The Eclipse plug-in is able to remind
developers and managers of existing SATD and help them pay for
SATD in time. We are also interested in providing visualization tools
to help developers further analyze SATD comments in different
Figure 4: Re-detecting SATD in Background kinds of software projects.
(3 in Figure 3). This Eclipse view displays details of each detected
ACKNOWLEDGMENT
comment, which includes Description (i.e., the text description of a
comment), Resource (i.e., in whichfi le a comment is located), Path This work was partially supported by NSFC Program (No. 61602403
(i.e., the path of a comment’s correspondingfile), Location (i.e., at and 61572426).
which line(s) a comment is located) and Type (i.e., the type of a
comment’s marker). The marker type of SATD comments is set to REFERENCES
[1] Gabriele Bavota and Barbara Russo. 2016. A large-scale empirical study on
“Technical Debt” by default. If a user double clicks some comment in self-admitted technical debt. In MSR.
the view, Eclipse will open thefi le in which this comment is located [2] Ward Cunningham. 1993. The WyCash portfolio management system. ACM
and focus on this comment in the editor ( 4 in Figure 3). This Eclipse SIGPLAN OOPS Messenger (1993).
[3] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann,
view also provides some basic features for SATD management, e.g., and Ian H Witten. 2009. The WEKA data mining software: an update. ACM
filtering and sorting. SIGKDD explorations newsletter (2009).
[4] Qiao Huang, Emad Shihab, Xin Xia, David Lo, and Shanping Li. 2017. Identifying
In addition, our plug-in provides a toolbar button (5 in Figure 3),
self-admitted technical debt in open source projects using text mining. EMSE
which is used to trigger complete SATD detection. This tool will (2017).
re-analyze the comments in the whole workspace if a user clicks [5] Erin Lim, Nitin Taksande, and Carolyn Seaman. 2012. A balancing act: what
software practitioners have to say about technical debt. IEEE software (2012).
this button. The time spent by this detection process depends on [6] Everton Maldonado, Emad Shihab, and Nikolaos Tsantalis. 2017. Using natural
the size of the target project. In order to improve user experience, language processing to automatically detect self-admitted technical debt. TSE
detection process always runs in background and the real-time (2017).
[7] Everton da S Maldonado and Emad Shihab. 2015. Detecting and quantifying
detection progress will be displayed in the Progress view ( 12 in different types of self-admitted technical debt. In MTD.
Figure 4). [8] Aniket Potdar and Emad Shihab. 2014. An exploratory study on self-admitted
technical debt. In ICSME.
[9] J. Ross Quinlan. 1986. Induction of decision trees. Machine learning (1986).
4 EVALUATION [10] Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model
for automatic indexing. Commun. ACM (1975).
The dataset used to evaluate the performance of our tool is de- [11] Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. 2016. Examining the impact
scribed in Section 2.3. We compare our tool with 4 kinds of baseline of self-admitted technical debt on software quality. In SANER.
approaches: [12] Xin Xia, David Lo, Denzil Correa, Ashish Sureka, and Emad Shihab. 2016. It takes
two to tango: Deleted stack overflow question prediction with text and meta
(1) Pattern: In this approach, a comment is regarded as SATD features. In COMPSAC.
comment if and only if it matches one of the 62 patterns [13] Xin Xia, David Lo, Emad Shihab, Xinyu Wang, and Bo Zhou. 2015. Automatic,
high accuracy prediction of reopened bugs. Automated Software Engineering
published by Potdar and Shihab [8]. (2015).
(2) NBM, SVM and kNN: We build simple classifiers using differ- [14] Xin Xia, David Lo, Xinyu Wang, and Xiaohu Yang. 2015. Who should review
ent text mining techniques (i.e., NBM, SVM and kNN), and this change?: Putting text andfi le location analyses together for more accurate
recommendations. In ICSME.
classify comments with these classifiers respectively. [15] Xin Xia, David Lo, Xinyu Wang, and Bo Zhou. 2013. Tag recommendation in
(3) BestSub: For each target project, we choose the sub-classifier software information sites. In MSR.
[16] Xin-Li Yang, David Lo, Xin Xia, Qiao Huang, and Jian-Ling Sun. 2017. High-
with best performance as our baseline. impact bug report identification with imbalanced learning strategies. Journal of
(4) NLP: We follow Maldonado et al.’s work [6] and build a Computer Science and Technology (2017).
maximum entropy classifier to predict whether a comment
contains SATD or not.

12