# Contemporary COBOL: Developers’ Perspectives on Defects and Defect Location

Agnieszka Ciborowska  Virginia Commonwealth University
Richmond, Virginia, USA
<EMAIL_ADDRESS>Aleksandar Chakarov  Phase Change Software
Golden, Colorado, USA
<EMAIL_ADDRESS>Rahul Pandita  Phase Change Software
Golden, Colorado, USA
<EMAIL_ADDRESS>
###### Abstract

Mainframe systems are facing a critical shortage of developer workforce as the
current generation of COBOL developers retires. Furthermore, due to the
limited availability of public COBOL resources, entry-level developers, who
assume the mantle of legacy COBOL systems maintainers, face significant
difficulties during routine maintenance tasks, such as code comprehension and
defect location. While we made substantial advances in the field of software
maintenance for modern programming languages yearly, mainframe maintenance has
received limited attention. With this study, we aim to direct the attention of
researchers and practitioners towards investigating and addressing challenges
associated with mainframe development. Specifically, we explore the scope of
defects affecting COBOL systems and defect location strategies commonly
followed by COBOL developers and compare them with the modern programming
language counterparts. To this end, we surveyed 30 COBOL and 74 modern
Programming Language (PL) developers to understand the differences in defects
and defect location strategies employed by the two groups. Our preliminary
results show that: (1) major defect categories affecting the COBOL ecosystem
are different than defects encountered in modern PL software projects; (2) the
most challenging defect types in COBOL are also the ones that occur most
frequently; and (3) COBOL and modern PL developers follow similar strategies
to locate defective code.

###### Index Terms:

mainframe, COBOL, software defect location, online survey, developers’
perspective.

##  I Introduction

Mainframe systems, although perceived by many as technologically outdated,
stand at the core of the daily operations of many financial, health care, and
governmental institutions. They provide business-essential features that
enable rapid, reliable, and secure processing of extremely large volumes of
transactions. To put the scale and importance of the mainframe development
into perspective, there are “over 220 billion lines of COBOL code being used
in production today, and 1.5 billion lines are written every year” [1], while
“on a daily basis, COBOL systems handle USD 3 trillion in commerce” [2].

These staggering numbers emphasize two important facts. First, mainframe
development is resilient and unlikely to be replaced any time soon due to
significant cost and risk associated with building/migrating to new
transaction processing systems that realize existing, well-established
functionalities. Second, as this generation of mainframe developers retires,
support for these systems is in jeopardy as mainframe development is not part
of current mainstream curricula [3, 4]. We suspect this problem will only
worsen as the new generation of developers trained on modern programming
stacks takes the responsibility of maintaining legacy COBOL systems.

While research efforts in studying maintenance in modern PL stacks resulted in
advanced tooling [5, 6, 7, 8, 9, 10, 11] and guidance to improve developer
productivity [12, 13], investigating maintenance of mainframe systems is rare
with most studies focusing on language migration [14, 15, 16]. Given the low
success rate (less than 50%[17]) of the mainframe migration effort, we posit
that mainframe developers need support and innovation in the most routine
software maintenance activities, such as locating and fixing software defects
[18].

As a step towards better understanding the needs of mainframe developers, in
this study, we explore mainframe developers’ perspectives regarding the scope
of software defects affecting mainframe systems and the strategies commonly
followed to locate the defects in the legacy codebase. We further investigate
and highlight the differences in defects and defect location strategies as
reported from the perspectives of COBOL and non-COBOL developers. Through this
comparison, we aim to: (1) provide insights into the types and frequency of
defects encountered during the development and maintenance of mainframe COBOL
systems; and (2) elicit a key features of a hypothetical defect location tool
targeted towards supporting mainframe developers. In particular, this work
addresses the following research questions:

RQ1: What are the major categories of software defects in COBOL? Are these
defect types specific to COBOL? Previous research identified different defect
categories and analyzed how frequently these defects occur in code [19, 20,
21, 22]. However, most of the analyzed software ecosystems are built around
the object-oriented paradigm, leaving a gap in understanding whether the same
problems affect other environments. This study focuses on mainframe projects
written in COBOL and contrasts frequent defects reported by COBOL and modern
PL developers.

RQ2: Are challenging software defects the same as typical defects? What are
the major challenging software defects in COBOL and non-COBOL environments?
Little is known about the developers’ point of view on the defects that are
most challenging to locate and fix [23, 24]. The goal of RQ2 is to contrast
the types and properties of typical and challenging software defects. This
comparison can shed light on the less studied area of challenging software
defects and pinpoint specific defect types that mainframe developers need the
most assistance with.

RQ3: Does the defect location process vary between software ecosystems? What
are the similarities and differences in developer approaches to locating
defective code? Locating a software defect is a time- and effort-consuming
task. To gain more insights into how developers perform defect location,
researchers performed multiple lab and field studies to observe developers’
work patterns [25, 26, 27, 28, 24]. In this study, we investigate developers’
perceptions of the most effective approaches to locate defects in their
respective environments to observe how working in a specific environment
(e.g., COBOL or non-COBOL) affects the defect location process.

To answer these questions, we asked software developers about their
perspective on software defects and strategies to localize faulty code. We
surveyed 106 professional developers with different job seniority,
specializing in various technologies, including 30 COBOL developers. To
support the survey’s findings and get more insight into developer viewpoints,
we conducted 10 follow-up interviews.

The contributions of this paper are:

  * •

identification of the most frequent and challenging software defects in COBOL,

  * •

investigation of developers’ perception of the most useful defect location
strategies,

  * •

recommendations of the key features of a hypothetical defect location tool to
support developers in maintaining mainframe COBOL systems,

  * •

publicly available anonymized study materials to aid replication and foster
future research in this area[29].

##  II Study design

For this study, we employed an online survey due to the following reasons.
First, the population of COBOL developers is relatively small and scattered,
hence to recruit a sufficient number of participants we could not be limited
by our geographical area. Second, COBOL projects are of crucial value to
companies, so accessing their internal data was highly unlikely. Third, due to
the rapidly evolving COVID-19 pandemic, an online study remained the safest
choice. We employed a mixed-methods approach to ascertain the differences
between the COBOL and non-COBOL developer populations regarding defect types
and defect location strategies. Specifically, we underwent the following
stages, as depicted in Figure 1: (A) Defect types taxonomy synthesis, (B)
Defect location strategy taxonomy synthesis, (C) Survey Design, (D) Survey
Deployment and Participant Recruitment, (E) Result analysis and follow-up
validation, and (F) COBOL Forums Defect Classification. We next describe each
of these stages in detail.

Figure 1: An overview of our study design methodology.

###  II-A Defect Type Taxonomy

Previous research has identified various taxonomies characterizing software
defects [19, 20, 30, 31]. In this study, we leverage the classification schema
proposed by Catolino et al. [20], which provides a concise, yet comprehensive
selection of defect types based on 9 categories: Configuration, Database, GUI,
Network, Performance, Permission/Deprecation, Program Anomaly, Security, and
Test Code. However, we note two major drawbacks of applying this taxonomy in
our study. First, Program Anomaly is a broad category and pertains to the
majority of software defects. This concern was identified during our think-
aloud mock survey sessions (details in Sec. II-C)(also mentioned Catolino et
al.. [20]). Therefore, we decided to further expand Program Anomaly into 3
categories based on defect types indicated during the mock surveys. Input data
refers to issues caused by unexpected or malformed input data, which often
occurs in the COBOL transaction processing system. Concurrency-related defects
involve problems caused by e.g., incorrect access to shared resources or
deadlocks. In contrast, Logic/Flow defects pertain to programming errors,
e.g., incorrect use of data structures. Second, participants of the mock
survey session additionally identified a separate defect category related to
issues arising due to the system operating at the limit of the resources,
Workload/Stress. In total, we use the 11 defect categories shown in Table I.

To capture additional software defect properties, we leverage Orthogonal
Defect Classification (ODC) [19]. ODC captures defect properties from two
perspectives, when the defect is reported and when it is closed. Although ODC
captures fine-grained details of defects, it is not applicable in the context
of a survey since that requires participants to be familiar with extensive ODC
terminology. Therefore, we opted for a hybrid approach where we leverage
Catolino et al. [20] defect categories, which to some extent cover ODC’s
defect reporting perspective, to which we add three of the ODC’s defect
closing perspective: Age, Source, and Qualifier. In particular, Age refers to
the time when the defect was introduced into the code base, Source indicates
which part of code was affected, whereas Qualifier identifies the cause of the
defect. Comparing the options specified in ODC for each of these dimensions,
we decided to introduce two changes. First, we introduce Corrupted data as a
qualifier to capture one of the frequent causes of defects in COBOL projects.
Second, we merge Outsourced and Ported categories in the Source dimension into
one category, Outsourced. The modification was dictated by observing how
during the mock survey session, the participants struggled to differentiate
between the two options. Finally, following Morrison et al. [32], we
introduced a dimension to determine an entity that reported a defect,
Reporter. The final version of defect location taxonomy is presented in Table
I and includes the defects categories and their properties.

TABLE I: Our software defect taxonomy based on [19] and [20]. Defect categories | Defect properties
---|---
Concurrency | Reporter | Developer
Configuration | Quality assurance personnel/Tester
Database | End user
GUI | Cause | Not coded to specification
Input Data | Missing or incomplete specification
Logic/Flow | Corrupted data
Network | Extraneous
Performance | Source | Developed in-house
Permission/Deprecation | External library/API
Security | Outsourced
Test Code | Age | New feature
Workload/Stress | Legacy code
| Re-factored
| Changed to fix a defect

###  II-B Defect Location Taxonomy

There is a rich history of exploratory observational studies investigating the
process of defect location [33, 34, 28, 25, 26]. In general, we note the
following common strategies emerge across different studies: looking for
initial code entities, exploring related program elements, and documenting
relevant code components [33, 28, 34]. In this study, we decided to leverage
the hierarchical feature location model proposed by Wang et al. [28]. The
model comprises three levels of granularity: phases, patterns, and actions.
Each phase represents a high-level stage in a feature location process,
namely, Search for entrance points, Expand the search, Validate, and Document.
Patterns define common strategies developers follow at different phases
related to, e.g., execution or textual search, while actions correspond to
fine-grained physical or mental activities undertaken by developers. Across
all phases, the authors identified 11 physical actions and 6 mental actions.
The physical actions included, e.g., reading code, or exploring source code
files, while mental actions covered, e.g., conceiving an execution scenario or
identifying keywords.

Figure 2: Defect location model based on Wang et al. [28].

We deem this model serves as a good starting point as it: (1) captures feature
location process at different levels of granularity; (2) is based on three
exploratory studies with a large number of developers; and (3) shares a common
set of activities with previous studies. Although the model provides a
comprehensive description of patterns used for locating relevant code
entities, we decided to alter some of its components to include developers’
practices discovered by prior studies. These modifications capture relevant
activities as indicated by the participants of the mock survey. Figure 2
presents the overview of the model with phases containing available patterns,
while physical and mental actions are listed below each phase. We mark
introduced modifications with red color. In general, we reuse all of the
phases defined in the original model. However, we extend available patterns.
In particular, we included the IR-based pattern in the Expand phase for the
symmetry with the Search phase. Researchers noted that developers tend to lose
track of the relevant program elements due to a large number of examined files
and interruptions [33, 28, 34, 18]. To provide more context for the Document
phase and investigate if/how developers preserve the information about
relevant code components, we curated a list of common documenting strategies
based on prior work [33, 28].

We modified the original list of physical actions in the following ways. We
merged Breakpoints operations, Step program and Run program into one succinct
action, Observing runtime. As noted in [35], the location of a specific
feature can be discovered by examining the release log and pull requests, thus
to reflect that we added new physical action, Examine Version Control System
(VCS). We included all mental actions identified by Wang et al. and added one
new action, Thinking about similar defects fixed before, which aims to measure
the importance of developers experience [36]. We decided to refer to mental
actions as reasoning strategies throughout the survey for ease of
understanding. Finally, we associated a set of Motivations/Goals with the
Expanding phase to capture the main objectives of developers at this stage
[37, 34].

###  II-C Survey Design

We leveraged the taxonomies developed in the previous steps to formulate our
survey questions and ask participants to provide answers in the context of
their routine work. We divided the survey into three main sections: Software
Defects, Defect Location, and Demographics.

In the _Software Defects_ section, we focused on the distribution of the
defects taxonomy attributes in the context of typical and challenging defects.
We primed the survey participants with the following definitions:

  * •

_Typical defects_ \- software defects faced frequently by study participants
in their daily work,

  * •

_Challenging defects_ \- software bugs that require a considerable amount of
time and effort when fixing due to, e.g., difficulties in isolating the root
cause, inability to reproduce the defect, or developing a viable solution.

Participants were asked to select between 3 to 5 typical software defect
categories and up to 3 challenging categories. A short explanation accompanied
each defect category. To capture the properties of typical and challenging
bugs, we presented developers with a 5-point Likert scale (Never, Rarely,
Sometimes, Often, Always), asking how frequently the participant observes
specific properties of a defect, e.g., reporter being a tester or an end-user.

In the _Defect Location_ section, we focused on collecting the distribution of
taxonomy attributes in the context of a general defect location process. We
asked questions about the most useful patterns, reasoning strategies or
motivations, and physical actions related to the defect location process.
Moreover, to investigate how developers move through the code, at the end of
each phase, we asked them about their preferred granularity level [23].

Finally, we captured the demographic information in the Demographics section.
In particular, we asked about:

  * •

Years of paid professional experience,

  * •

Preferred programming languages,

  * •

Current country of residence,

  * •

Willingness to participate in a follow-up interview.

Once we had the initial draft of questions, we ran a mock survey with both
COBOL and non-COBOL developers following a think-aloud protocol [38] to
estimate the average response time and further fine-tune our survey. The mock
survey was performed with developers recruited in-house at Phase Change
Software via teleconference calls. We were interested in identifying the
questions that were often: (1) misunderstood \- participants consistently
erred in understanding the intent behind the question; (2) irrelevant \-
participants consistently question the relevance of the question.

We promptly removed the irrelevant questions. Furthermore, we did a
lightweight thematic analysis of the think-aloud recordings to further refine
the questions, which included three changes. First, we refined the language in
questions that caused of the confusion. Second, we reordered questions to
achieve a better transition. Finally, we created two distinct surveys
individually catering to the COBOL and the non-COBOL developer population. The
surveys had semantically equivalent questions, differing, when necessary, to
account for discrepancies in terminology. For instance, classes/files in the
non-COBOL survey translate to modules/procedures in the COBOL survey. Based on
our mock sessions, we estimated that our survey’s average response time is 30
minutes. All of the questions used in our final survey can be found on the
project website [29].

###  II-D Survey Deployment and Participant Recruitment

To recruit COBOL developers, we contacted the mailing list maintained by Phase
Change Software111https://www.phasechange.ai/. We further posted the survey to
various COBOL groups on Linkedin and IBM-maintained COBOL developer forums. To
recruit non-COBOL developers, we reached out to our personal contacts at
several worldwide software companies (Spotify, Google, Facebook, Nokia,
Capgemini), asking to distribute the survey across experienced software
developers. To capture the perspective of the broader audience of software
engineers, we also publicized our survey on professional forums within social
platforms such as LinkedIn, Twitter, and Facebook.

To improve our response rate, we offered rewards in the form of a chance to
win an Amazon gift card. For COBOL developers, we set the reward to USD 25 for
25 participants selected at random. For non-COBOL developers, we set the
reward to USD 10 for 25 participants selected at random. The disproportionate
number of developers influenced the difference in reward in each group, which
is also reflected in these two groups’ response-rates.

###  II-E Analysis and follow-up validation

In this stage, we analyzed collected responses. A summary of these results was
presented to a subset of survey participants as a light-weight mechanism to
confirm our synthesis. Each semi-structured session lasted 30 minutes and
followed a script prepared beforehand. In all, we recruited 6 COBOL and 4 non-
COBOL survey participants at random for the follow-up interviews. These
sessions were conducted on Zoom, an online video conferencing platform, and
recorded with the participants’ permission to be transcribed and analyzed
offline.

###  II-F COBOL Forums Defect Classification

Studying developers’ perspectives can be biased by one’s perception; hence we
decided to compare defect types reported in the survey with results obtained
by data mining. We referred to prior research [32] to triangulate the survey
results of the non-COBOL reported defect types. However, no such baseline
existed for COBOL developers. To address this gap, we manually classified the
queries posted in the COBOL Programming forums of the IBM Mainframe Experts
group [39], which we believe is representative of our target COBOL developers
population.

The forum has over 6000 posts, so to get to a 95% confidence level with a 10%
confidence interval to the target representative population, we needed to
annotate at least 100 posts. However, not all of the posts were defect-
related. Thus, authors ended up coding 400 posts only to discard nearly 300
posts as non-defects. Authors independently classified the defect types of 20
defect-related posts. We captured our agreement with a 0.32 Fleiss’ Kappa
score, which can be interpreted as a “fair agreement”. After discussing and
resolving the differences, the authors again classified the next 20 defect-
related posts to reach 0.64 Fleiss’ Kappa score indicating “substantial
agreement”. The authors then resolved their differences and split the rest of
the posts to code them independently. In all, we coded 107 defect-related
posts.

##  III Results

###  III-A Demographic Information

Overall, we received 106 responses from software developers located across 11
countries, with the majority of participants residing in the United States,
India, and Poland. We discarded the responses from two developers who reported
having less than a year of professional industry experience and considered the
remaining responses for further analysis. Out of 104 respondents, 30 are
currently, or were recently, involved in a COBOL project, while the remaining
74 work with modern programming languages, including Java, Python, and C#. In
the remainder of this section, we refer to the first group as COBOL and the
later as non-COBOL developers.

Among the COBOL developers, we noted 25 males and 5 females. Non-COBOL
developers included 58 males and 10 females, while 6 developers did not
disclose their gender. On average, COBOL developers reported to have 22.3
years of professional programming experience ($std=14.84$, $min=4$, $max=60$).
In contrast, non-COBOL developers reported to have 9.78 years of professional
programming experience ($std=7.68$, $min=1$, $max=51$).

###  III-B Software defects

(a) Q: What defect categories do you work on most frequently?

(b) Q: What are the most challenging defects categories?

Figure 3: Developers’ perspective on typical (left) and challenging software
defects (right).

Typical Software Defects. Figure 3(a) shows the distribution of developers’
perception of typical defect types. Logic/Flow and Input data are the most
frequently occurring defect categories according to both COBOL and non-COBOL
developers. Logic/Flow was selected by 83.3% of COBOL and 59.5% of non-COBOL
developers, while Input data was reported by 80.0% and 48.6% of COBOL and non-
COBOL developers, respectively. Our results are in conformance with previous
work that states that the majority of software defects are related to
programming errors leading to exceptions, crashes, or incorrect behavior [20,
40].

Developer perception varies for the rest of the defect categories. COBOL
developers indicated that other typical defects pertain to Configuration (50%)
and Database (42.9%), whereas Security, Network, or Concurrency related
defects rarely happen. We posit that the lack of focus on modules implementing
network or security features within COBOL mainframe batch processing explains
this distribution. In contrast, non-COBOL developers reported Performance as
their major category of frequently occurring typical defects (52.7%). Overall,
we notice that defects related to the rest of the categories, Configuration,
Database, Test Code, User Interface, Workload/Stress, and Concurrency, are
fairly evenly distributed, being reported as common by 25.7% (Concurrency) to
35.1% (Test Code) of non-COBOL developers.

We ran a chi-squared test to test the null hypothesis: “the distributions of
defect types among COBOL and non-COBOL responses are the same”. However, low
values reported in various defect categories by our COBOL developers required
us to follow the common recommendation to combine some of the rows to obtain
sufficient values for an approximation of a chi-square distribution [41]. We
combined Network, Security, and Permission/Deprecation into a “Miscellaneous”
category. Based on the $p$-value of 0.02, we reject the null hypothesis,
concluding that typical defect types encountered by COBOL developers are
significantly different from typical defect faced by non-COBOL developers.

We investigated whether developers’ perception of typical defect types follows
a distribution reported in previous studies. As a baseline for non-COBOL
responses, we selected data from Morrison et al. [32] that includes a manual
annotation of defect types from large open-source projects. Since our survey
asked developers to select the top-3 typical defect types, we compared the
survey results with the frequencies of top-3 defect categories reported by the
baseline (Performance, Installability, and Capability). Note that we mapped
categories of defects used in the baseline to defect types leveraged by this
study. We ran a chi-square test to test the null hypothesis that “defect
categories reported by the survey and baseline follows the same distribution”.
The test resulted in $p=0.16$; thus, we accept the null hypothesis, which
indicates that non-COBOL developers’ perceptions of typical defects are fairly
accurate.

In the case of COBOL responses, we use annotated defect-related posts from the
IBM mainframe forum as a baseline (described in Section II-F). We observed
that the top-3 most discussed defect types are related to Logic/Flow,
Configuration, and Input data, which broadly confirms the survey results.

RQ1: Defects related to Logic/Flow and Input data are the primary defect types
in COBOL systems. Typical defect types encountered by COBOL developers are
significantly different from typical defects encountered by non-COBOL
developers.

Challenging software defects. The distribution of developers’ perception of
challenging defect types is shown in Fig. 3(b).

According to COBOL developers, challenging defects are mostly related to Input
data (64.3%) and Logic/Flow (53.6%), making those categories typical and
challenging simultaneously. As the top third most challenging defect 35.7% of
COBOL developers selected Workload/Stress. In contrast, only 12.5% of COBOL
developers reported Workload/Stress as a typical defect. Non-COBOL developers
indicated issues related to Performance (60.8%), Concurrency (56.8%), and
Workload/Stress (33.8%) as the most challenging, whereas Logic/Flow and Input
data defects were selected by 27% and 13.5% of the respondents, respectively.

We ran the chi-squared test to compare the distribution of challenging defect
types among COBOL and non-COBOL developer responses. In particular, we tested
the following null hypothesis: “challenging defect types reported by COBOL and
non-COBOL developer responses follow the same distribution”. Our test resulted
in $p<0.01$. We therefore reject the null hypothesis, indicating that
challenging defects reported by COBOL and non-COBOL developers are, in fact,
different. We also performed the chi-squared test to evaluate the differences
in the distribution of typical and challenging defects within each respondent
group. Specifically, for each group, we tested the null hypothesis “if typical
and challenging defect types have the same distribution”. In the case of
COBOL, our test resulted in $p=0.096$, therefore we accept the null hypothesis
and conclude that typical and challenging COBOL defects are not significantly
different. In contrast, challenging defects reported by non-COBOL respondents
are significantly different from the typical defect types with $p\ll 0.01$.

In the follow-up interviews, most COBOL and non-COBOL developers agreed with
the top typical and challenging defect categories reported by their respective
groups. COBOL developers indicated that Logic/Flow and Input data could be
challenging since there is no good IDE for COBOL compared to IntelliJ or
Eclipse tools, and the structure of COBOL code can easily lead to programming
errors. In support, another COBOL developer mentioned one of the hardest
defects he faced was related to a missing null check for file pointer, that
could be easily detected by a modern IDE. Developers also stated that, since
COBOL is primarily a legacy system, they rarely know the entire process, thus
making it difficult to locate the root cause effectively. Finally, developers
also noted that Workload/Stress and Performance defects pose a challenge due
to the difficulty in local reproduction and the unique characteristics of
every defect. Overall, developers listed the following challenges associated
with specific defects: (1) reproducing the defect outside the production
environment is often impossible; (2) defects do not surface immediately;
instead, they happen occasionally, leading to numerous problems across
components; (3) it can take days to find the right solution.

RQ2: Challenging defects encountered by COBOL developers are not different
from typical defects. However, challenging defects encountered by non-COBOL
developers vary significantly from typical defects.

Defect properties. Developers’ perspectives on the properties of software
defects are illustrated in Fig. 4, with each property visualized as a Likert
scale. Properties of typical defects are marked with (T), whereas challenging
defects with (Ch).

When asked about defect reporters, 63.2% of COBOL and 80.7% of non-COBOL
developers agreed that testers are the most likely group to detect typical
defects (Fig. 4(a)). In contrast, only 33.3% of COBOL and 47.3% of non-COBOL
developers expressed positive opinions about testers’ ability to detect
challenging defects. At the same time, both groups indicated that end-users
are more likely to report challenging defects, with 50% for COBOL and 43.2%
for non-COBOL supporting such opinion.

We conducted unpaired Mann-Whitney tests to verify the null hypothesis:
“testers have the same capability to detect typical and challenging defects”.
The tests were conducted for COBOL and non-COBOL separately. We note $p=0.029$
and $p\ll 0.01$ for COBOL and non-COBOL, respectively, indicating a
statistically significant difference in perceived testers’ abilities. We
conducted analogous tests for end-users and obtained $p=0.052$ and $p=0.143$;
thus, we conclude that the observed difference is not significant.

In the follow-up interviews, both COBOL and non-COBOL developers supported
these findings indicating that since end users have a system that has already
been tested, then all the easy defects have been already caught. Hence, end-
users usually find more complicated issues, which the engineering team has not
thought of. Moreover, testing for, e.g., Performance or Workload/Stress, is
rarely doable in the development environment and requires significant time and
effort.

(a) Q: How frequently are software defects reported by:

(b) Q: How likely is a software defect caused by:

(c) Q: How likely is a software defect to occur in code that is:

(d) Q: How likely is a software defect to occur in code that is:

Figure 4: Developers’ perspective on the properties of software defects.

COBOL and non-COBOL developers agree that typical defects are mostly caused by
corrupted data (73.3% and 55.4%) or missing specifications (76.7% and 73%). In
the case of challenging defects, respondents expressed weaker opinions about
the root cause, shifting their opinions from Always towards Often and
Sometimes. Additionally, we note an increased selection of Not coded to
specification as a root cause of challenging defects compared to the results
obtained for typical defects. Developers also agree that both typical and
challenging defects are more likely to be located in the code developed in-
house or outsourced than in external APIs (Fig. 4(c)). The majority of defects
are deemed to originate in a newly developed code. However, developers ranked
legacy code as the second most likely source of issues (Fig. 4(d)).

Testers’ abilities to detect challenging defects are limited due to
environments’ restrictions and lack of tools supporting efficient testing of
e.g., Performance, or Workload/Stress-related scenarios.

###  III-C Defect Location Strategies

We analyzed developers’ responses for the three phases of the defect location
process outlined in the taxonomy in Section II-B: searching for a starting
point, expanding relevant program elements, and documenting [27].

Phase 1: Search. Table II presents reasoning strategies employed by developers
in the first phase of the defect location process, ranked in order of
importance. We note that each group of respondents have their own distinctive
preferences. COBOL developers declare to first focus their efforts on locating
a similar defect that was solved before (47%), while non-COBOL developers
prefer to identify simple steps to reproduce (44%). Moreover, both groups
frequently selected the opposite strategy as their second top choice, further
emphasizing the usefulness of the two approaches. Next, participants indicated
they tend to look into source code files and blocks of code, and finally, they
concentrate on keywords present in a defect report.

The developers further confirmed these rankings in the follow-up interviews.
For instance, COBOL developers mentioned that seeing a defect which is similar
to an issue fixed before gives a good approximation of the potential problem
and its location, therefore their organizations tend to keep defect databases
to retain the knowledge of previous patterns of software failures.

TABLE II: Reasoning strategies undertaken by developers when examining new defect reports. Reasoning strategy | COBOL | Non-COBOL
---|---|---
Mean rank | Mean rank
Thinking about similar defect fixed before | 2.567 (1) | 3.378 (2)
Thinking about simple steps allowing to reproduce the defect | 3.133 (2) | 2.568 (1)
Thinking about source code files that may be related to the defect | 3.200 (3) | 3.689 (4)
Thinking about block of codes that may be related to the defect | 4.167 (4) | 3.500 (3)
Identifying keywords that can be related to defect’s location | 4.367 (5) | 4.014 (5)
Refining or augmenting keywords based on my knowledge | 5.567 (6) | 5.541 (6)
Thinking about not obvious steps that can cause the defect | 5.000 (7) | 5.311 (7)

Developers’ opinion on the effectiveness of physical actions is illustrated in
Fig. 5. In the initial phase, developers rank reading code and reviewing
defect descriptions as the most useful actions for locating an entry-point for
further exploration. Additionally, we note that COBOL developers prioritized
searching for code components and navigating code dependencies, whereas non-
COBOL developers prefer to explore source code files. This difference can be
attributed to the tool capabilities (e.g., providing a summary of related code
components) used by non-COBOL developers during their quick file exploration.

Figure 5: Developers’ perspective on the usefulness of physical actions in the
Search phase.

With respect to the most useful patterns in the Search phase of the defect
location process, the majority of developers in both groups indicated they
prefer to use an execution-based strategy to locate an initial set of entry
points, followed by textual search and navigating static code dependencies.

Phase 2: Expand. After locating a set of entry points, developers move to the
Expand phase. Table III depicts the main goals developers aim to achieve at
this stage. We observe that both groups are mainly focused on reproducing the
defect to confirm whether their location process moves in the direction of the
defect’s cause. Additionally, respondents indicated that they need to
understand the code behavior better. Follow-up interviews revealed that as
developers are often under time pressure, they prefer to focus on the fastest
path leading to resolving the defect. Therefore, they opt for reproducing the
faulty behavior as once the defect gets reproduced, it is very easy to proceed
further.

TABLE III: Developers’ motivations and goals in the Expand phase. Motivation/Goal | COBOL | Non-COBOL
---|---|---
Mean rank | Mean rank
Reproducing a defect | 1.600 (1) | 1.905 (1)
Understanding code behavior | 2.500 (2) | 2.162 (2)
Locating more code components | 2.633 (3) | 2.824 (3)
Depicting dependencies | 3.267 (4) | 3.108 (4)
Figure 6: Developers’ perspective on the usefulness of physical actions in the
Expand phase.

Developers’ perception of the usefulness of physical actions in the Expand
phase is shown in Fig. 6. Like the Search phase, developers rank reading code
as the most beneficial activity, followed by navigating structural code
dependencies and exploring source code files. Further, when analyzing
developers’ patterns in this phase, we note that they prefer to use an
execution-based approach. In contrast to the Search phase, developers reported
leveraging code dependencies more often than performing a textual search.

Phase 3: Document. Preserving information about located relevant files
concludes the defect location process. Fig. 7 illustrates how developers keep
track of important code components. Respondents in both groups indicated they
frequently write notes (e.g., on paper or in a text editor) and use
breakpoints to indicate relevant code pieces. We notice that non-COBOL
developers rely more on “open tabs” and “keep in mind” strategies compared to
COBOL developers. Both preferences can be explained in part by the tooling
support for non-COBOL developers. For instance, a typical modern IDE provides
ample opportunity to keep multiple tabs open and enough visual cues to
facilitate the “keep in mind” strategy.

RQ3: COBOL and non-COBOL developers approach defect location rather similarly.
Minor differences are explainable by the varying level of experience and
available tooling support.

###  III-D Code granularity throughout the defect location process

Figure 7: Developers’ perspective on documentation.

In the case of non-COBOL developers, we observe a consistent preference for
Method/Function level code granularity across all phases with support varying
between 37.8 and 44.6%. We note a slight change in preferences in Phase 3 when
developers mark relevant code components to be fixed. At this stage, non-COBOL
developers reported increased interest in the Block of code (from 17.6% to
30%) and Line of code (from 4.1% to 21.6%) levels of granularity. COBOL
developers tend to follow a top-down approach focusing first on the high
granularity level and progressively going towards lower levels. In the Search
phase, 50% of COBOL developers prefer to work with Files or Methods. In the
Expand phase, they increasingly prefer Methods level, while when documenting,
they operate at the level of Blocks or Lines of code.

All COBOL developers agreed with these results during follow-up interviews and
related the top-down code navigation hypothesis to the COBOL code structure.
On the other hand, non-COBOL developers indicated that while they sometimes
follow a similar top-down approach, overall, they prefer to reason about
functions as a single line of code rarely provides enough context, whereas, in
a well-written code, a method encapsulates a specific sub-task and provides
just enough information to understand and efficiently solve the defect.

COBOL developers typically follow a top-down approach to navigate artifacts
during defect location as reflected by their choices for specific artifacts at
different phases.

Figure 8: Preferred granularity of code components in different phases of
defect location process.

##  IV Discussion and Future Work

Typical defects are challenging. We observed that there is no significant
difference between typical and challenging defects in COBOL. In general, COBOL
developers reported Logic/Flow and Input data as top-two major defect
categories, which are both typical and challenging. This indicates that a
hypothetical defect location tool addressing just these two defect types is
likely to significantly impact the developer productivity and reduce the time
required to locate and fix the defect. As indicated by a COBOL developer in a
follow-up interview, the lack of tools targeting COBOL is one of the key
factors contributing to Logic/Flow being a challenging defect. On the
contrary, non-COBOL developers, who enjoy an wide set of tools, reported that
typical defects such as Logic/Flow or Input Data rarely pose any difficulties.
We believe that if COBOL developers are given supporting tools addressing
their most pressing needs, we would observe a decrease in Logic/Flow and Input
data defects’ perceived difficulty.

COBOL and non-COBOL developers reported that challenging defects are often not
detected by testers; hence they reach production. While undesirable in any
application, it is even worse for COBOL ones as they typically support
critical financial, health care, and government systems. Our respondents
stated that the key reason for challenging defect propagation is related to
differences between testing and production environments’ configuration and
capabilities. Taking a cue from the continuous delivery movement [42], we
recommend tool-smiths to cost-effectively minimize the differences between
COBOL development and production environments.

Defect location strategies are fairly similar. Even though our respondents
work with different software ecosystems, they reported following fairly
similar defect location strategies, indicating that observations and findings
made by prior studies in the context of common patterns followed by non-COBOL
developers can be applied to COBOL developers. Although confirmation requires
further research, initial results are encouraging. Analyzing the survey
results, we also note the benefits that a simple code navigation tool [43]
could provide to COBOL developers. First, the tool could help developers keep
track of the code under analysis as indicated in the code documentation
preferences. Further, it could also be aligned with observed top-down
navigation patterns where COBOL developers start at the highest granularity
(File/Module) and iteratively move to finer granularities (Method/Paragraph).

Respondents in both groups indicated the Execution-based pattern, involving
running or debugging an application, as the most effective strategy for defect
location. However, whereas non-COBOL developers usually have unrestricted
access to the application via an IDE, the same cannot be said for COBOL
developers. Mainframe time is expensive, so the possibility of debugging or
re-running the application is often limited. On the other hand, follow-up
interviews revealed that the sooner developers could reproduce a defect, the
faster they can resolve it. Thus improving the state-of-the-art of debugging
in the mainframe environment would greatly benefit developers.

Emerging research directions. For the researchers, we recommend expanding the
content and scope of this study. One possible research avenue is to conduct an
in-situ study of COBOL developers to verify and refine actionable insights
towards tools supporting mainframe development. We suspect that such an
undertaking would be very fruitful as an industry collaboration since COBOL
projects are typically highly regulated and not accessible to a broader
audience. Another direction can be to offer COBOL courses at universities and
study the student population to get end-user perspectives [44] on tool usage
[45].

While studying developers’ perspectives on software defects gives a good
initial overview of defects in the COBOL ecosystem, analysis of actual COBOL
defects is imperative to validate our findings [46]. Our data is publicly
available at [29]. We hope researchers and practitioners will curate
additional benchmarks to foster future research. We concede that collecting
such information is challenging since enterprise COBOL projects are either
proprietary and regulated.

Finally, we plan to follow previous studies [47, 36] and work with veteran
developers to identify optimal strategies to preserve their knowledge and
investigate how to remove entry-level barriers by passing hard-earned
expertise onto a new generation of developers. With one of the participants
reporting 60 years of programming experience, we deem this research direction
to be particularly exciting and viable in the context of COBOL.

##  V Threats to validity

This study suffers from several limitations that can impact the validity and
generalizability of the results.

Internal validity. Terminology employed in the mainframe environment differs
from that leveraged by non-COBOL developers. Providing developers with
imprecise or misleading wording may increase the interpretation bias, which,
in turn, leads to unreliable results. We developed two independent surveys
with the same content to mitigate this threat, with terminology adjusted to
the target group of respondents. Examples of such adjustments include, e.g.,
using code granularity levels from the mainframe environment such as Job,
Section, Paragraph, or modifying External libraries/API to External
libraries/COTS (Common of the shelf software).

External validity. The survey measures developers’ perception, which is
strictly subjective and affected by interpretation bias; hence it may not
reflect true defect and defect location strategies. We mitigated this threat
with the following steps. First, we conducted a pilot study with a COBOL and
non-COBOL developer to ensure the usage of correct and clear terminology.
Second, we recruited a large number of participating developers to account for
potential noise in the data. Finally, we conducted follow-up interviews to
validate our observations and gather developers’ rationale for them.
Furthermore, we triangulated the non-COBOL defect-types results with an
independent baseline [32] and COBOL defect types by mining defects from online
forums.

##  VI Related work

We contrast our work with prior research, which we grouped into three main
areas: (1) COBOL, (2) developers’ perception, and (3) software defects and
defect location strategies.

COBOL ecosystem. Despite its massive presence in the financial sector, the
COBOL ecosystem has seen little attention in recent software engineering
research. Sellink et al. [48] proposed a set of methods to effectively
restructure COBOL code, while Sneed et al. [14] described the process of
extracting knowledge from a legacy COBOL system. More recently, researchers’
efforts concentrated on the migration of COBOL code to Java. Mossienko [49]
introduced an automated technique translating COBOL to Java. Sneed et al. [50,
51] reported on two industrial COBOL migration projects involving code
restructuring and automated language translation. De Marco et al. [15]
described migration methodology designed for a newspaper company, while
Rodriguez et al. [16] discussed two migration approaches based on a case study
of 32 COBOL transactions. Although these studies made a significant step
towards restructuring and migrating COBOL code, the ways to natively support
large-scale production COBOL systems have not yet been investigated.

As any programming language, COBOL is not defect-free; however, defects and
their properties in the mainframe environment were rarely investigated.
Litecky et al. [52] studied errors commonly made by students learning COBOL
and observed that 20% of error types account for 80% of errors. Veerman et al.
[53] focused specifically on the perform statement and identified programming
constructs leading to unexpected system crashes. In contrast, this study
focuses on studying general defects types and defect location strategies
employed by professional COBOL developers to deepen the understanding of the
problems encountered in the field.

Studying developers’ perceptions. This work leverages online surveys, a common
tool employed by researchers to study developers’ expectations and
perceptions. For instance, the survey methodology has been applied to
investigate debugging tools and approaches to solve the most difficult defects
[54], and developers’ expectations towards fault localization [23]. While our
work is similar in spirit, we explicitly study professional COBOL developers.
Devanbu and colleagues [46] indicate that although developers’ beliefs may
differ from empirical evidence, developers’ perceptions should be taken into
consideration, especially when prior research is limited, and obtaining data
at-scale is challenging. Although we were not explicitly investigating such
discrepancies, we did find outliers in developers’ perceptions. However, they
were few and far between and were discarded in aggregate analysis.

Software defects and defect location strategies. Researchers and practitioners
devised several classification taxonomies to capture and analyze the
characteristics of software defects. Chillarege et al. [19] introduced
Orthogonal Defect Classification (ODC) comprising 8 dimensions describing
defect properties. Researchers at Hewlett-Packard [31] proposed a taxonomy
leveraging three dimensions to explain defects’ type, source, and root cause.
Defect categories and properties identified in these studies have been
successfully applied in various industrial settings [55, 56] and led to
building automatic tools for defect prevention [57] and classification [58].
Recently, Catolino et al. [20] inspected defect reports from 3 large software
projects to build a modern taxonomy consisting of 9 defect categories. Our
work builds upon defect taxonomies proposed by Catolino et al. [20] and ODC
[19].

Researchers conducted numerous lab and field studies observing strategies
leveraged by developers while completing maintenance tasks to delineate the
defect location process. For instance, Ko et al. [33] identified three main
activities frequently interleaved by developers: seeking, relating, and
collecting information. Wang et al. [28, 27] created a hierarchical model
describing the feature location process in terms of phases, patterns, and
actions. Based on exploratory studies, Kevic et al. [25] identified a set of 6
re-occurring activities related to, e.g., understanding or changing source
code. Recently, Chattopadhyay et al. [59] performed an exploratory study with
10 developers to observe how developers maintain the current task context.
This work leverages the model of feature location proposed by Wang et al. [28]
to create the survey.

Building automatic tools to reduce the effort required to localize relevant
code components has been of great interest. However, despite the plethora of
advanced automated techniques for defect location [60, 61, 62, 63, 35, 64],
none of these were evaluated in the context of mainframe development.
Therefore, their applicability in the mainframe environment remains largely
unknown.

##  VII Conclusion

This work compares the COBOL and non-COBOL developers’ perspectives on
software defects and defect-location strategies. We observed that: (1) defects
affecting COBOL belong to two major categories(Logic/Flow and Input Data), (2)
COBOL and non-COBOL developers follow similar defect location strategies, and
(3) adapting debugging tools to the mainframe environment could have a
significant impact in COBOL developer productivity.

## Acknowledgment

We thank our participants for their time and effort, Dan Acheff for his
insights about COBOL ecosystem, and Brad Cleavenger and Greg Brueggeman for
their help in designing the survey. We also thank Phase Change Software for
supporting this work.

## References

  * [1] T. Taulli. (2020) Cobol language: Call it a comeback? [Online]. Available: https://www.forbes.com/sites/tomtaulli/2020/07/13/cobol-language-call-it-a-comeback/#536cd2897d0f
  * [2] D. Cassel. (2017) Cobol is everywhere. who will maintain it? [Online]. Available: https://thenewstack.io/cobol-everywhere-will-maintain/
  * [3] D. Carr and R. J. Kizior, “The case for continued cobol education,” _IEEE Software_ , vol. 17, no. 2, pp. 33–36, 2000.
  * [4] R. J. Kizior, D. Carr, and P. Halpern, “Does cobol have a future?” in _Proc. Information Systems Education Conf_ , vol. 17, no. 126, 2000.
  * [5] D. Shepherd, K. Damevski, B. Ropski, and T. Fritz, “Sando: An extensible local code search framework,” in _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_ , ser. FSE ’12, 2012.
  * [6] M. Martinez, A. Etien, S. Ducasse, and C. Fuhrman, “Rtj: a java framework for detecting and refactoring rotten green test cases,” _arXiv preprint arXiv:1912.07322_ , 2019.
  * [7] X. P. Mai, A. Göknil, F. Pastore, and L. Briand, “Smrl: A metamorphic security testing tool for web systems,” in _IEEE/ACM 42nd International Conference on Software Engineering_ , 2020.
  * [8] B. Buhse, T. Wei, Z. Zang, A. Milicevic, and M. Gligoric, “Vedebug: regression debugging tool for java,” in _IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_. IEEE, 2019.
  * [9] D. Lockwood, B. Holland, and S. Kothari, “Mockingbird: a framework for enabling targeted dynamic analysis of java programs,” in _IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_. IEEE, 2019.
  * [10] R. Meng, B. Zhu, H. Yun, H. Li, Y. Cai, and Z. Yang, “Convul: an effective tool for detecting concurrency vulnerabilities,” in _34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019.
  * [11] D. Beyer and T. Lemberger, “Testcov: Robust test-suite execution and coverage measurement,” in _34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019\.
  * [12] T. Fritz and S. C. Müller, “Leveraging biometric data to boost software developer productivity,” in _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ , 2016.
  * [13] L. Gonçales, K. Farias, B. da Silva, and J. Fessler, “Measuring the cognitive load of software developers: A systematic mapping study,” in _2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)_ , 2019\.
  * [14] H. M. Sneed, “Extracting business logic from existing cobol programs as a basis for redevelopment,” in _Proceedings 9th International Workshop on Program Comprehension. IWPC 2001_ , 2001, pp. 167–175.
  * [15] A. De Marco, V. Iancu, and I. Asinofsky, “Cobol to java and newspapers still get delivered,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2018.
  * [16] J. M. Rodriguez, M. Crasso, C. Mateos, A. Zunino, and M. Campo, “Bottom-up and top-down cobol system migration to web services,” _IEEE Internet Computing_ , 2013.
  * [17] V. Combs. (2020) Everyone wants to retire mainframes but 74% of modernization efforts fail. [Online]. Available: https://www.techrepublic.com/article/everyone-wants-to-retire-mainframes-but-74-of-modernization-efforts-fail/
  * [18] T. D. LaToza, G. Venolia, and R. DeLine, “Maintaining mental models: A study of developer work habits,” in _Proceedings of the 28th International Conference on Software Engineering_ , ser. ICSE ’06, 2006.
  * [19] R. Chillarege, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S. Moebus, B. K. Ray, and M. . Wong, “Orthogonal defect classification-a concept for in-process measurements,” _IEEE Transactions on Software Engineering_ , pp. 943–956, 1992.
  * [20] G. Catolino, F. Palomba, A. Zaidman, and F. Ferrucci, “Not all bugs are the same: Understanding, characterizing, and classifying bug types,” _Journal of Systems and Software_ , 2019.
  * [21] S. Lal and A. Sureka, “Comparison of seven bug report types: A case-study of google chrome browser project,” in _2012 19th Asia-Pacific Software Engineering Conference_ , 2012, pp. 517–526.
  * [22] S. Bruning, S. Weissleder, and M. Malek, “A fault taxonomy for service-oriented architecture,” in _10th IEEE High Assurance Systems Engineering Symposium (HASE’07)_ , 2007, pp. 367–368.
  * [23] P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis_ , ser. ISSTA 2016, 2016\.
  * [24] B. Siegmund, M. Perscheid, M. Taeumel, and R. Hirschfeld, “Studying the advancement in debugging practice of professional software developers,” in _2014 IEEE International Symposium on Software Reliability Engineering Workshops_ , 2014, pp. 269–274.
  * [25] K. Kevic and T. Fritz, “Towards activity-aware tool support for change tasks,” in _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2017.
  * [26] K. Damevski, D. Shepherd, and L. Pollock, “A field study of how developers locate features in source code,” _Empirical Software Engineering_ , 2016\.
  * [27] J. Wang, X. Peng, Z. Xing, and W. Zhao, “An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions,” in _2011 27th IEEE International Conference on Software Maintenance_ , ser. ICSM ’16, 2011.
  * [28] J. Wang, X. Peng, Z. Xing, and W. Zhao, “How developers perform feature location tasks: a human-centric and process-oriented exploratory study,” _Journal of Software: Evolution and Process_ , 2013.
  * [29] (2020) Project data and materials. [Online]. Available: https://zenodo.org/record/4734699
  * [30] B. Freimut, C. Denger, and M. Ketterer, “An industrial case study of implementing and validating defect classification for process improvement and quality management,” in _11th IEEE International Software Metrics Symposium (METRICS’05)_ , 2005, pp. 10 pp.–19.
  * [31] R. B. Grady, _Practical Software Metrics for Project Management and Process Improvement_. USA: Prentice-Hall, Inc., 1992.
  * [32] P. J. Morrison, R. Pandita, X. Xiao, R. Chillarege, and L. Williams, “Are vulnerabilities discovered and resolved like other defects?” _Empirical Softw. Engg._ , p. 1383–1421, 2018.
  * [33] A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks,” _IEEE Transactions on Software Engineering_ , Dec. 2006\.
  * [34] J. Sillito, G. C. Murphy, and K. De Volder, “Asking and answering questions during a programming change task,” _IEEE Transactions on Software Engineering_ , 2008.
  * [35] J. Krüger, W. Gu, H. Shen, M. Mukelabai, R. Hebig, and T. Berger, “Towards a better understanding of software features and their characteristics: A case study of marlin,” in _Proceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems_ , ser. VAMOS 2018, 2018.
  * [36] H. Jordan, J. Rosik, S. Herold, G. Botterweck, and J. Buckley, “Manually locating features in industrial source code: The search actions of software nomads,” in _2015 IEEE 23rd International Conference on Program Comprehension_ , 2015.
  * [37] A. J. Ko, R. DeLine, and G. Venolia, “Information needs in collocated software development teams,” in _Proceedings of the 29th International Conference on Software Engineering_ , ser. ICSE, 2007.
  * [38] E. M. Redmiles, Y. Acar, S. Fahl, and M. L. Mazurek, “A summary of survey methodology best practices for security and privacy researchers,” Department of Computer Science, University of Maryland, Tech. Rep., 2017.
  * [39] (2020) Expert forum cobol programming. [Online]. Available: http://ibmmainframes.com/forum-1.html
  * [40] L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai, “Bug characteristics in open source software,” _Empirical Softw. Engg._ , 2014.
  * [41] L. Ott, “An introduction to statistical methods and data analysis: Boston,” _Mass., PWS-Kent Publishing Company_ , 1988.
  * [42] J. Humble and D. Farley, _Continuous delivery: reliable software releases through build, test, and deployment automation_. Pearson Education, 2010.
  * [43] A. Z. Henley, _Human-centric Tools for Navigating Code_. The University of Memphis, 2018.
  * [44] A. J. Ko, B. A. Myers, and H. H. Aung, “Six learning barriers in end-user programming systems,” in _2004 IEEE Symposium on Visual Languages-Human Centric Computing_. IEEE, 2004, pp. 199–206.
  * [45] R. Pandita, C. Parnin, F. Hermans, and E. Murphy-Hill, “No half-measures: A study of manual and tool-assisted end-user programming tasks in excel,” in _2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 2018, pp. 95–103.
  * [46] P. Devanbu, T. Zimmermann, and C. Bird, “Belief & evidence in empirical software engineering,” in _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_. IEEE, 2016, pp. 108–119.
  * [47] P. Morrison, R. Pandita, E. Murphy-Hill, and A. McLaughlin, “Veteran developers’ contributions and motivations: An open source perspective,” in _2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 2016, pp. 171–179.
  * [48] A. Sellink, H. Sneed, and C. Verhoef, “Restructuring of cobol/cics legacy systems,” _Science of Computer Programming_ , vol. 45, no. 2-3, pp. 193–243, 2002.
  * [49] M. Mossienko, “Automated cobol to java recycling,” in _Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings._ , 2003, pp. 40–50.
  * [50] H. M. Sneed, “Migrating from COBOL to java,” in _2010 IEEE International Conference on Software Maintenance_. IEEE, 2010, pp. 1–7.
  * [51] H. M. Sneed and K. Erdoes, “Migrating as400-cobol to java: A report from the field,” in _17th European Conference on Software Maintenance and Reengineering (ICSME)_ , 2013.
  * [52] C. R. Litecky and G. B. Davis, “A study of errors, error-proneness, and error diagnosis in cobol,” _Commun. ACM_ , vol. 19, no. 1, 1976.
  * [53] N. Veerman and E.-J. Verhoeven, “Cobol minefield detection,” _Software: Practice and Experience_ , vol. 36, no. 14, 2006.
  * [54] B. Siegmund, M. Perscheid, M. Taeumel, and R. Hirschfeld, “Studying the advancement in debugging practice of professional software developers,” in _2014 IEEE International Symposium on Software Reliability Engineering Workshops_ , 2014, pp. 269–274.
  * [55] R. R. Lutz and I. C. Mikulski, “Empirical analysis of safety-critical anomalies during operations,” _IEEE Transactions on Software Engineering_ , vol. 30, no. 3, pp. 172–180, 2004.
  * [56] J. Zheng, L. Williams, N. Nagappan, W. Snipes, J. P. Hudepohl, and M. A. Vouk, “On the value of static analysis for fault detection in software,” _IEEE Transactions on Software Engineering_ , vol. 32, no. 4, pp. 240–253, 2006.
  * [57] A. A. Shenvi, “Defect prevention with orthogonal defect classification,” in _ISEC ’09_ , 2009, p. 83–88.
  * [58] L. Huang, V. Ng, I. Persing, M. Chen, Z. Li, R. Geng, and J. Tian, “Autoodc: Automated generation of orthogonal defect classifications,” _Automated Software Engineering_ , 2015.
  * [59] S. Chattopadhyay, N. Nelson, Y. Ramirez Gonzalez, A. Amelia Leon, R. Pandita, and A. Sarma, “Latent patterns in activities: A field study of how developers manage context,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ , 2019.
  * [60] C. S. Corley, K. Damevski, and N. A. Kraft, “Changeset-based topic modeling of software repositories,” _IEEE Transactions on Software Engineering_ , 2018.
  * [61] M. Wen, R. Wu, and S.-C. Cheung, “Locus: Locating bugs from software changes,” in _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE 2016, 2016.
  * [62] L. Moreno, J. J. Treadway, A. Marcus, and W. Shen, “On the use of stack traces to improve text retrieval-based bug localization,” in _2014 IEEE International Conference on Software Maintenance and Evolution_ , 2014, pp. 151–160.
  * [63] T. Dao, L. Zhang, and N. Meng, “How does execution information help with information-retrieval based bug localization?” in _2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC)_ , 2017, pp. 241–250.
  * [64] A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “Bug localization with combination of deep learning and information retrieval,” in _2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC)_ , 2017.