11institutetext:  Dept. of Computer Science and Eng., Chalmers $\mid$
University of Gothenburg, Gothenburg, Sweden 22institutetext: Zenseact AB,
Gothenburg, Sweden

# Defining Requirements Strategies in Agile:
A Design Science Research Study

Amna Pir Muhammad 11 Eric Knauss 11 Odzaya Batsaikhan 11 Nassiba El Haskouri
11 Yi-Chun Lin 11 Alessia Knauss 22

###### Abstract

Research shows that many of the challenges currently encountered with agile
development are related to requirements engineering. Based on design science
research, this paper investigates critical challenges that arise in agile
development from an undefined requirements strategy. We explore potential ways
to address these challenges and synthesize the key building blocks of
requirements strategies. Our design science research rests on a multiple case
study with three industrial cases in the domains of communication technology,
security services, and automotive. We relied on a total of 20 interviews, two
workshops, participant observation in two cases, and document analysis in each
of the cases to understand concrete challenges and workflows. In each case, we
define a requirements strategy in collaboration with process managers and
experienced engineers. From this experience, we extract guidelines for
defining requirements strategies in agile development.

###### Keywords:

requirements strategy, design science research, requirements engineering,
large-scale agile development

##  1 Introduction

Agile development methodologies aim to shorten the time to market and
incorporate maximum changes during the sprint to meet customer needs [21] and
have been adapted at small-scale as well as large-scale organizations [18].
With its’ focus on interactions and working software over rigid processes and
extensive documentation, traditional well established Requirements Engineering
(RE) processes have been neglected. Research shows that many of the challenges
currently encountered with agile development are related to requirements
engineering [14] for example, misunderstanding customer needs, missing high-
level requirements, and difficulty to achieve having just enough
documentation.

In this study, we identify specific RE-related challenges and related solution
strategies in agile development. Based on this knowledge, we derive necessary
building blocks as different viewpoints that should be considered when
thinking strategically about RE in agile development. In this, we are inspired
by test strategies, which guide testing activities to achieve the quality
assurance objectives [8] and which mandate that each project must have its
test plan document that clearly states the scope, approach, resources, and
schedule for testing activities [20]. We argue that defining a so called
requirements strategy similar to a test strategy for RE can be critical for
successful agile development.

In this paper, we aim to establish the concept of _requirements strategy for
agile development_ by investigating the following research questions based on
iterative design science research in three industrial case studies.

RQ1

Which challenges arise from an undefined requirements strategy?

RQ2

How do companies aim to address these challenges?

RQ3

Which potential building blocks should be considered for defining a
requirements strategy?

Since we particularly target agile development, we aimed to investigate
requirements challenges independent from process phases or specific documents.
Instead, we took the lens of _shared understanding_[7] to investigate
different RE activities (i.e., elicitation, interpretation, negotiation,
documentation, general issues). According to Fricker and Glinz, an
investigation of _shared understanding_ may primarily target how such shared
understanding is _enabled_ in an organization, how it is _built_ , and how it
is _assessed_ for relevance [7].

Therefore, our contribution are guidelines on how requirements strategies
should be described for agile development. Through building three
complementary perspectives, we see that the requirement strategy guidelines
capture relevant information and provide a useful overview. We suggest that a
strategy defines the structure of requirements to create a shared language,
define the organizational responsibilities and ownership of requirements
knowledge, and then map both structure and responsibilities to the agile
workflow.

In the next section, we provide the related work for our study. In Section 3
we elaborate on our design science research method before revealing our
findings in Section 4 in order to answer our research questions. Then, in
Section 5, we present our artifact - guidelines on how to define a
requirements strategy for RE in agile development. Finally, we discuss and
conclude our paper in Section 6.

##  2 Related Work

Literature shows that many companies adopt agile methods [17, 13] due to its
numerous benefits, for example, flexibility in product scope which improves
the success rate of products [3], in contrast to traditional development
methods [27]. Furthermore, agile methods incorporate maximum change and
frequent product delivery [13], encourage changes with low costs, and provide
high quality products in short iterations [21]. Due to its success, agile
methodologies are become widely popular and adopted by both small and large
companies [18]. The term _large-scale agile_ refers to agile development,
which includes large teams and large multi-team projects [4]. Dikert et al.
define large-scale agile development as agile development that includes six or
more teams [3].

However, despite the success of agile methods, large-scale companies also
still face several challenges. Dikert et al. (2016) [3] conducted a systematic
literature review of empirical studies. The authors identified several
challenges and success factors for adopting agile on a large scale. The most
mentioned challenges are change resistance, integrating non-development
functions, difficulty to implement agile methods (misunderstanding, lack of
guidance), requirement engineering challenges (e.g., high-level requirements
management largely missing in agile methods, the gap between long and short
term planning). Based on a literature review, Dumitriu et al. (2019) [5]
identified 12 challenges of applying large-scale agile methods at the
organization level. The most cited challenge is the coordination of several
agile teams. Kasauli et el. (2021) [14] identified 24 challenges through
multiple case studies across seven large-scale companies. Some of the
identified challenges are building long lasting customer value, managing
experimental requirements, and documentation to complete tests and stories.
The authors conclude that strong RE approaches are needed to overcome many
identified challenges.

When it comes to RE in agile development, challenges that have been identified
include lack of documentation, project budget, time estimation, and shared
understanding of customer values [6, 12, 24, 15, 1] First attempts have been
made to tackle some of the challenges of RE in agile development, e.g., Inayat
et al. and Paetsch et al [12, 22]. suggest combining traditional RE with agile
methods and encounter challenges like how much documentation is just enough
documentation [11] to have a shared understanding of customer values.

Considering that there are many challenges related to RE that can be solved
through RE approaches, this paper proposes to use the concept of a
requirements strategy as a method to define requirements engineering practices
to tackle challenges related to requirements engineering in agile development.

##  3 Design Science Research Method

Our research aims to design suitable ways of defining requirements strategies
for organizations with agile software development. Such requirements
strategies should be suitable for addressing real-world needs, incorporating
state-of-the-art knowledge, and ideally being empirically evaluated in
practice. Thus, we decided that design science research [29, 28, 10] is a good
fit.

#### Design Science Research.

Our research questions are targeted towards design science research, with RQ1
focusing on the problem domain, RQ2 investigating potential solutions, and RQ3
targeting towards deriving the artifact. Our artifact are guidelines on how to
define a requirements strategy in agile development. Refining on well-known
challenges with RE in agile development, we needed to gain in-depth insights
into those challenges related to a lack of a clear requirements strategy
throughout the agile development organization (RQ1). Throughout our cases, we
discuss those challenges with respect to potential mitigation strategies (RQ2)
for those challenges. Finally, we systematically synthesize the building
blocks of requirements strategies (RQ3) from solution strategies.

Table 1: Research questions in relation to cases and research cycles | Case 1 | Case 2 | Case 3
---|---|---|---
| Cycle 1 | Cycle 2 | Cycle 3 | Cycle 4 | Cycle 5
Identify challenges (RQ1) | $\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare$ | $\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare$ | $\blacksquare\blacksquare\blacksquare\blacksquare\square\square$ | $\blacksquare\blacksquare\square\square\square\square$ | $\blacksquare\blacksquare\blacksquare\square\square\square$
Identify solutions (RQ2) | $\blacksquare\blacksquare\blacksquare\blacksquare\square\square$ | $\blacksquare\blacksquare\square\square\square\square$ | $\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare$ | $\blacksquare\blacksquare\blacksquare\blacksquare\square\square$ | $\blacksquare\blacksquare\blacksquare\square\square\square$
Building blocks (RQ3) | $\blacksquare\blacksquare\square\square\square\square$ | $\square\square\square\square\square\square$ | $\blacksquare\square\square\square\square\square$ | $\blacksquare\blacksquare\square\square\square\square$ | $\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare\blacksquare$
Data source | 11 Interviews, | 9 Interviews, | Participant
| document | participant observation, | observation,
| analysis | document analysis, | document
|  | 1 workshop | analysis,
|  |  | 1 workshop

Inspired by the regulative cycle [29], the artifact (guidelines for defining a
requirements strategy based on good practices from our cases) has iteratively
evolved, allowing to refine the knowledge with respect to each research
question. Table 1 provides an overview of our research method. As can be seen,
we relied on three case studies over which we distribute a total of five
research cycles. The cycles differ in how much focus is given to each of our
three research questions:

_Case 1 \- Exploring the problem through the lens of requirements engineering
and shared understanding:_ Case 1, an information and communication technology
company, focuses on a strategy to achieve a shared understanding about
customer value throughout the development organization. Our research aims were
two-fold: understand the real world problem and conceptualize a design
artifact that may address this problem. Within a Master’s thesis [2], we
developed an appropriate lens that combined both the concept of shared
understanding (as expressed by Glinz and Fricker through enabling, building,
and assessing shared understanding [7]) and commonly used RE activities (such
as elicitation, interpretation, negotiation, and documentation). We then
relied on 11 interviews to understand customer value and its common
understanding, information sharing, tools and channels for sharing, and tools
and methods for documenting. Since our first cycle focuses on the exploration
of the problem we locally relied on the case study research method for our
research with respect to Case 1 [2]. As Table 1 shows, we complemented the
interviews with document analysis to produce an overview of challenges and
related solution strategies.

_Case 2 \- Refining the requirements strategy artifact iteratively:_ We then
followed up in _Case 2_ , a company producing security smart alarms and
services. In this case, the focus was on a more general requirements strategy
that covers both stakeholder and system requirements. Again, through a
Master’s thesis [8], we investigated concrete requirements challenges of an
agile team, defined a requirements strategy along the lines of the result from
Case 1, and investigated in depth to what extent it could help with the
challenges in practice. At this point, we further focused on investigating
whether there are reusable building blocks for a requirements strategy.

_Case 3 \- Applying and evaluating the artifact:_ Finally, we brought our
experience from the previous two cases into _Case 3_ , an automotive supplier,
focusing on complex safety critical and software intense systems. Here, we
focus less on challenges and solution strategies, in particular, since the
case company already had compiled a good overview. Instead, our focus is to
refine the artifact (guidelines for requirements strategies) by discussing,
applying, and improving our understanding of the building blocks of a
requirements strategy. At the time of the research, the first author of this
paper did an internship with this automotive supplier and helped to make the
requirements strategy explicit. The company did already identify some of the
challenges and making a first step towards implementing their solution
strategies. Thus, she was able to investigate the phenomenon as a participant
observer, contrasting it with documents and ways of working in the practical
context, allowing us to fine-tune our guidelines for requirements strategies
to provide an overview of challenges and solution strategies for continuous
process improvement.

#### Data collection.

We relied on a mix of different methods for data collection, including
interviews, participant observation, document analyses, and workshops.

Interviews - We relied on interviews in Case 1 and Case 2, in particular, to
understand the problem (RQ1) in each specific case. 20 interviews were
conducted using interview guides (details in [8, 2]), relying on a mix of
closed and open-ended questions. Interviews were recorded, transcribed, and
then coded. In both cases, we recruited interviewees through purposeful
sampling [23]. We relied on convenience sampling so that interviewees were
both available and knowledgeable. We employed diversity sampling to capture
the views of multiple relevant roles and stakeholders and similarity sampling
to ensure that we received multiple views for each relevant perspective for
triangulation.

Participant Observation - The fourth author was very familiar with Case 2, in
which she had worked for several years before starting her Master thesis [8].
Her work included defining a testing strategy, which provided intimate
knowledge about the agile ways of working in the case company, which were also
helpful for understanding the requirements-related challenges, defining a
requirements strategy, and conducting the evaluation in Case 2. The first and
last authors did work with RE and the continuous improvement of requirements
processes of Case 3. Through this work, we were able to verify that previously
identified challenges in Case 3, as well as initiatives to address them, were
of similar nature and matched well with our recent work on requirements
strategies. Both co-authors relied on our requirements strategy work to
support the ongoing initiatives on requirements processes and on integrating
RE practices into agile ways of working. This allowed us to evaluate the
suitability of our requirements strategy concept. Knowledge from these
activities was collected through field notes, presentations given at the case
company, and discussions with other co-authors.

Document Analysis - In all three cases, a subset of the authors studied the
documents related to the flow of requirements (Case 1: Author 3 and 5, Case 2:
Author 4, Case 3: Author 1 and 6). Since all three cases embraced agile ways
of working, we considered that not all relevant information might be found in
formal documents. However, we ensured that documentation did match or at least
not contradict our data. We found relevant documentation of requirements,
e.g., as user stories, in all three cases to match and support our other data
sources. Document analysis also allowed us to better understand the implied
requirements strategy and processes.

Workshops - We relied on two workshops in cases 2 and 3 to evaluate the
proposed requirements strategies and, by that, also our requirements strategy
guidelines. In case 2, a workshop was conducted to present the challenges
identified, the proposed solution candidates, and different versions of the
specific requirements strategy of each case. In case 3, a workshop was used to
understand the requirements strategy that was used to address certain
challenges. Expert participants were sampled similarly as for interviews. They
were asked to bring up additional challenges that we may have missed, give
feedback on the criticality of the challenges that we had found, provide their
opinion about the solution candidates, and evaluate the structure,
presentation, and concrete advice on the requirements strategies. Depending on
the circumstances in each case, we recorded and transcribed, took live notes
for all participants to see, or shared our notes after the workshop for
validation.

#### Data analysis.

In order to analyze interview transcripts, field notes from participatory
observation and document analysis, and workshop notes/transcripts, we relied
on typical coding approaches for qualitative research [26]. This allowed us to
report on challenges that relate to a missing or undefined requirements
strategy. For example, the following quote from Case 1 contributed to
identifying the challenge _d) lack of communication with customers_ : _“The
thing that sometimes does not work as it should, is communication with some of
the customer units. It heavily depends on the competence of the customer unit
people.”_

In each case, we had access to an industry champion from the respective
company, who helped to suggest practical solutions. For example, the following
quote suggested a solution for challenge above as _c) ability to initiate on
demand meetings with customer representatives:_ _“The right people to nail
down a requirement should be put together in the meeting to have a requirement
handshake.”_ In addition, the second author was involved as an academic
supervisor in all three cases, providing pointers toward relevant published
knowledge. We regularly presented and discussed our findings at the case
companies, focusing on strong tracing between challenges, solution candidates,
and the proposed requirements strategies. Together with iterative refinements,
this allowed us to analyze the data in depth.

#### Threats to Validity.

_Internal validity_ aims to reveal factors that affect the relationship
between variables, factors investigated, and results. A key threat to internal
validity of this study is the risk of misinterpretations, particularly during
the interviews and observations. _Construct validity_ defines the extent to
which the investigated measures describe what the researchers analyze and what
is studied according to research questions [25]. We mitigated threats to
internal and construct validity through interacting closely with industry
partners in study design and interpretation of results. We also worked
iteratively and triangulated across our iterations and cycles as well as
different data sources. _External validity_ relates to identifying to what
extent our findings can be generalized [19]. We identified common challenges
in all three case companies. Thus, we expect that in particular the structure
and perspectives of our requirements strategy guidelines can be transferred to
other contexts. _Reliability_ reflects to what extent other researchers can
produce the same results repeating the same study methodology. In a
qualitative study, it is always hard to achieve _reliability_ since one cannot
argue based on statistical significance. We mitigate this threat by
elaborating our research method in detail to support other researchers in
replicating our research and in recovering from any possible differences in
results.

##  4 Findings

###  4.1 RQ1: Which challenges arise from an undefined requirements strategy?

The left column of Table 4.1 depicts RE challenges identified based on our
three cases that are encountered without a clear requirements strategy
existing for agile development.The challenges are categorized in RE practices
and related to Glinz and Fricker’s [7] practices of shared understanding,
grouped in three categories, i.e., enable, build, and assess. Enable practices
describe what is needed to form and establish a common foundation of
knowledge. Building practices aim to provide the structured knowledge that can
be communicated within the team or company through explicit artifacts or by
constructing a body of implicit knowledge for shared understanding. Assessing
methods determine how all team members have a shared understanding of a topic
or artifact. Some methods can be used for both building and assessing
practices. Indices indicate in which of the cases a challenge was relevant.

a) Teams struggle to integrate RE in their agile work efficiently1,2,3 - Agile
development enables organizations to respond to change. If there is a change
in code and tests, the requirements should usually be updated. Or if
requirements change, then the code and tests need to be adjusted accordingly.
Teams struggle with this since requirements tools do not integrate well with
agile software development work and do not support parallel changes from
several teams. Thus, it is hard to integrate RE work into the agile work
effectively.

b) No formal event to align on customer value1 - There were no formal events
to create awareness of customer value in Case 1. Even when the customer unit
took the initiative and organized some events, there were only a few
participants. Such events must be better integrated in the organization and
workflow.

c) Insufficient customer feedback1,2 - In Case 1 and 2, developers lack
customer feedback, which is crucial for agile workflows.This can be due to a
lack of formal events, or due to scale and distance to customers. It impacts
the ability of an organization to assess whether shared understanding has been
reached. Customer feedback should be integrated into the workflow across
organizational levels and take into account the specific needs of product
owners and developers.

d) Lack of communication with customer1 - Customer-facing units have a key
role and are on the boundary between development teams and customers. We
encounter difficulties with communication in both directions: between
customer-facing teams and development teams and between customer-facing teams
and the customers. These challenges are mainly due to a lack of systematic
guidance on how such communication should take place, thus depending
completely on the individual skills of those involved. Companies would have to
find a way to ensure good and transparent communication, for example by having
product owners moderating direct meetings between developers and customers.

e) Who owns customer value1 - Requirements enter the development organization
mainly through the hierarchy of product owners (PO) in Case 1. However, a
significant amount of requirements originate from other sources, e.g.,
development teams or system managers, and in those cases, it is less clear who
is able to define or who owns the customer value.

f) Inconsistent elicitation2 - POs or application specialists collect
requirements when needed and apply techniques such as interviews. There is,
however, no systematic strategy to elicitation integrated into the workflow.

g) Lack of feedback on elicitation2 - Without a systematic validation of
elicitation results, misunderstandings will only surface late in the agile
workflow, e.g. during acceptance testing and result in additional costs and
effort.

Table 2: Overview of Challenges in Relation to the Solution Strategies.
Indices (${}^{1},^{2},^{3}$) show in which case study a challenge or strategy
was encountered.

| Shared Understanding |
---|---|---
RE |  _Enable_ |  _Build_ |  _Assess_ |  Solution Strategy
_General issues_ |  a) Teams struggle to integrate RE in their agile work efficiently1,2,3 |  b) No formal event to align on customer value1 |  c) Insufficient customer feedback1,2 |  a) Tools that allow developers to take ownership of req.1,2,3
b) Regular meetings with customer representat.1,2 |  |  |  |
Elicitation |  d) Lack of communication with customer1 |  |  |
e) Who owns customer value1 |  f) Inconsistent elicitation2 |  g) Lack of feedback on elicitation2 |  c) Ability to initiate on demand meetings with customer representatives1,2 |
Interpretation |  h) Unclear why requirement is needed2 |  i) Wrong assumptions about customer value1 |  j) Unclear and volatile customer needs2 |  d) Fast feedback cycles1,2
Negotiation |  k) Decentralized knowledge building3 |  l) Focus on technical details1,2 |  |
m) Req. open for comments3 |  n) No time for stakeholder involvement2 |  e) Req. template includes customer value & goals1,2 |  |
f) Define team respon- sibilities for different parts of req. and review updates regularly2,3 |  |  |  |
Documentation |  o) Customer value description lost between systems1 |  |  |
p) Lack of knowledge about writing requirements1,2,3 |  |  |  |
q) No dedicated time for requirements1,2,3 |  r) Too much/not enough document.1,2 |  |  |
s) Trace the requirements to all levels, (test, and code)3 |  t) Inconsistency b/c of requirements change3 |  |  |
|  g) Rationale must always be provided1 |  |  |
h) Just enough documentation1,2 |  |  |  |
i) Plan time for requirements updates3 |  |  |  |
j) Educate and train the development teams2,3 |  |  |  |
k) Tools need to be setup to support traceability 3 |  |  |  |

h) Unclear why requirement is needed2 - Due to scale, distance to customers,
or because a customer value description is not available for developers (see
Challenge o), application specialists and POs may lack information on why
specific low-level requirements are needed. This can result in a gap between
what product owners want and how the development teams interpret their
requirements.

i) Wrong assumptions about customer value1 - Interviewees highlighted that one
of the significant challenges is that people assume customer value based on
their tacit knowledge, leading to the development of faulty assumptions.

j) Unclear and volatile customer needs2 - Requirements change, for example
when the customer changes their mind or did not have a detailed opinion in the
beginning. When assessing the interpretation of requirements, this can cause
friction, since the team tries to “hit a moving target”.

k) Decentralized knowledge building3 - Different teams develop requirements,
architecture, and also processes at the same time. This decentralized way of
working is needed to yield the benefits of agile work at scale, but requires
some infrastructure to enable knowledge sharing and alignment. Otherwise,
conflicting decisions will be made throughout the organization.

l) Focus on technical details1,2 - Often customer value is not explicitly
described; instead, customer needs and technical solutions are more explicit.
When we asked participants in Case 1 and 2 to describe the customer value of
specific requirements, they explained the technical solutions rather than
customer values. This finding is consistent with documentation, where often
technical details are described instead of linking to a business reason for
motivating the requirement.

m) Requirements open for comments3 - In agile development, everyone who has
access to the system can create issues related to requirements in the
requirements management tool. While it is positive to include as many
stakeholders as possible in discussions, without a defined process that
respects the development lifecycle, this can result in an unstructured
discussion and very late changes.

n) No time for stakeholder involvement2 - Getting stakeholders’ feedback after
interpreting the elicited requirements is challenging since stakeholders do
not have time for several meetings.

o) Customer value description lost between systems1 - At the scale of Case 1,
it is not unusual to use several different tools to manage requirements at
various abstraction layers. Customer-facing units use one tool, in which they
define stakeholder requirements and customer value. Development teams interact
with different tools, and it is the task of the POs to refine and decompose
the stakeholder requirements from tool 1 into work items for the agile teams
in tool 2. At this step, documentation about customer value is often not
transferred and thus not available to the developers.

p) Lack of knowledge about writing requirements1,2,3 - Throughout our cases,
we found that those who are responsible for documenting requirements often do
not have the right training. In addition, we frequently saw a lack of
structure and no requirements information model. Thus, teams mix stakeholder
and system requirements and are challenged with writing high-quality user
stories, system requirements, and in particular quality requirements. In
particular, the quality requirements might not get documented at all and teams
will work on them without making them visible on the sprint dashboard.

q) No dedicated time for requirements1,2,3 - Since agile methods focus on
reducing time to market, spending time on writing formal requirements is not
considered. Instead, agile teams rely on verbal requirements. Dedicated time
to work on requirements should be integrated in the agile workflow, e.g. each
sprint.

r) Too much/not enough documentation1,2 - Because agile focuses on less
documentation, some essential information could be missing (e.g., such as the
“why” part of the requirement). Thus, in agile development, determining the
right amount or sweet spot of documentation is challenging.

s) Trace the requirements to all levels, (test, and code)3 - Due to ISO26262
and ASPICE compliance, the automotive company needs to guarantee full
traceability between all requirements levels, (tests, and code). This places a
big challenge on the entire company, since most teams work on something
related to requirements, tests, or code and those artifacts evolve in
parallel.

t) Inconsistency because of requirements change3 - Agile methods embrace
change and, consequently, teams will make changes on requirements during their
work. However, it is challenging to handle sudden change requests and opinions
from different team members, especially at scale. The consequence can be that
teams inconsistently change related requirements, or that the scope is
increased without central control. The problem is known, yet there is a lack
of guidance on how to handle this in large-scale agile development to avoid
expensive rework.

###  4.2 RQ2: How do companies aim to address these challenges?

The last column of Table 4.1 summarizes the answers to RQ2 on solution
strategies associated with the challenges with each phase of RE in respective
rows, derived from interviews, literature, or workshops and confirmed by
experts in each case.

a) Tools that allow developers to take ownership of requirements1,2,3 - In
order to allow developers to take ownership of requirements, we need to find
requirements tooling that integrates into the mindset and the development
environment of developers to provide an efficient way of manipulating
requirements. For instance, developers work closer to the code, so the
requirements tool that supports commit/git is highly encouraged.

b) Regular meetings with customer representatives1,2 - The customer-facing
unit should arrange regular meetings with customers. These meetings should be
well integrated in the agile workflow and mandatory for team members.

c) Ability to initiate on demand meetings with customer representatives1,2 -
There should be a setup to initiate meetings with customers whenever
developers need feedback. Since access to customer representatives is a sparse
and valuable resource, a strategy for such meetings should be well aligned
with the organizational structure and the agile workflow.

d) Fast feedback cycles1,2 - All teams use direct communication with
stakeholders and fast feedback cycles as a baseline to get the correct
interpretation. Customer insight is abstract knowledge and could be hard to
write down. There is a need to arrange events where people can meet, interact,
and share customer values and feedback.

e) A requirements template that includes customer value and goals1,2 - To
avoid challenges related to a lack of awareness of customer value, there
should be specific fields or tracelinks that show how each requirement adds
customer value. It is important to check their usage regularly.

f) Define team responsibilities for different parts of requirements and review
updates/comments regularly2,3 - In order to yield benefits from agile
workflows, RE must be integrated into the agile workflow. This means that
agile teams need to take responsibility of maintaining requirements and to
monitor changes of requirements that are potentially related. This allows to
manage requirements updates in parallel and at scale. However,
responsibilities have to be carefully delegated and clearly assigned.

g) Rationale must always be provided1 - The rationale for the requirement
should mandatorily be provided by the role/person writing the requirement.
Moreover, it should effectively be passed on from tool to tool.

h) Just enough documentation1,2 - Balancing sufficient communication and
documentation is crucial in agile development. We should not spend too much
time documenting; however, it should have all the necessary information.
Developers need clear guidelines to achieve this balance.

i) Plan time for requirements updates3 - Teams should plan (update, change,
review) the requirements in time to align with the updated scope. Such a plan
should consider that updating requirements in the scope of one team may imply
also requirements updates in other scopes.

j) Educate and train the development teams2,3 - If development teams should
take more responsibility of requirements, they need to be trained in RE as
well as in the specifics of the overall requirements processes in their
organization. A clear requirements strategy can be a good starting point to
plan such training.

k) Tools need to be setup to support traceability3 - Requirements are usually
represented in different forms (e.g., textual requirements, user stories) and
on different levels (e.g., system level and software level). Teams could get
requirements at higher level and then derive the lower level requirements
(e.g., software/technical requirements). Tracing requirements could be hard in
a large complex system. Tools are needed and they should be aligned with a
requirements strategy for agile workflows, i.e. allow parallel work for many
teams.

###  4.3 RQ3: Which potential building blocks should be considered for
defining a requirements strategy?

This section systematically develops the building blocks of a requirements
strategy from our findings in all three cases.

In Case 1, the company was challenged to establish a shared understanding.
Proposed solution strategies for specific challenges in Case 1 can be
categorized as _structural_ , _organizational_ , or related to the _work and
feature flow_. For example, for the challenge _l) focus on technical details)_
, a related solution strategy is _e) requirements template includes customer
value and goals_. This strategy explains that, to avoid the lack of awareness
about customer value, there should be specific fields related to customer
value in the requirements templates. This solution shows that there is a need
for improvement at the _structural level_. In contrast, _b) no formal event to
align on customer value_ is a challenge related to stakeholders’ roles and
responsibilities that needs to be well integrated into the _organization_. The
last column in Table 4.1 provides a solution strategy related to this
challenge as _b) regular meetings with customer representative_ , which
relates not only to the _organizational perspective_ , but also to the _work
and feature flow_.

In Case 2, we found the same perspectives (_structural, organizational_ , as
well as _work and feature flow_) in in solution strategies for their specific
challenges. As in Case 1, the solution strategy to introduce _e) requirements
templates that include customer value and goals_ is a _structural_ example. In
contrast, the challenge _g) lack of feedback on elicitation_ can lead to
misunderstandings late in an agile workflow. The solution strategy is to
establish the _c) ability to initiate on-demand meetings with customer
representatives_. Providing access to a sparse and valuable resources such as
a customer representative relates to the _organizational_ perspective. Another
related solutions strategy, _d) fast feedback cycles_ , for the challenge _j)
unclear and volatile customer needs_ falls into the _work and feature flow_
perspective, by arranging events where people can meet, interact, and share
customer values and feedback.

After looking deep into the concrete solution strategies in Case 1 and Case 2
(see Table 4.1), we found that many of these strategies were already
successfully implemented in Case 3. However, the company still faced some RE
challenges in agile development, allowing us to check whether the same
building blocks are also applicable in Case 3. For example, the challenge _s)
trace the requirements to all levels_ can be addressed with the _structural_
solution strategy _k) tools to set up traceability_. Similarly, the challenge
_k) decentralized knowledge building_ can be addressed by the _organizational_
solution strategy _define team responsibilities for different parts of
requirements and review updates/comments regularly_. Finally, an example of a
_work and feature flow_ related solutions strategy is to _i) plan time for
requirements updates_ in agile sprints to counter the challenge of having _q)
no dedicated time for requirements_.

In summary, in order to address specific challenges related to enabling,
building, and assessing shared understanding of requirements in agile
development, specific solution strategies fall into three distinct categories:
_structure_ , _organization_ , as well as _work and feature flow_. Thus, a
requirements strategy that bundles solution strategies for a concrete case
should cover all three perspectives.

##  5 Artifact: Guidelines for Defining a Requirements Strategy

Our artifact is a set of guidelines for defining a _Requirements Strategy_ as
a means to define RE activities in agile development. As a design science
research study, we built this artifact in parallel to answering our research
questions iteratively. In particular, RQ3 provides empirical validation of the
building blocks. At the time of research, the term “requirements strategy” has
not been widely used. This is in contrast to, for example, “test strategy”,
which has quite widely been accepted to describe how testing practices can be
integrated in development workflows, such as in agile ways of working. In our
work, we refer to “requirements strategy” as a general strategy for including
RE practices in agile methods.

Table 3: Building Blocks of a Requirements Strategy | Support for shared understanding of requirements
---|---
Perspective |  Common language |  Knowledge flow |  Examples
Structural |  Define reqts. levels |  Define structural decomp. |  Stakeholder, System, Component Requirements
|  Define reqts. types |  Define traceability demands |  Requirements and Traceability Information Model
|  Define templates |  |  User stories include customer value and goal
Organizational |  Define ownership of reqts. types |  Define roles and responsibilities |  Training plan per type/role; Team responsibility sheet
Work and feature flow |  Define lifecycle of types |  Map structure to workflow |  Elicitation strategy, definition of done
|  |  Map organization to workflow |  Stakeholder map, requirements review strategy

Definition: Requirements Strategy. A requirements strategy is an outline that
describes the requirements engineering approach in systems or software
development cycles. The purpose of a requirements strategy is to support
decision makers with a rational deduction from organizational, high-level
objectives to actual requirements engineering activities to meet those
objectives and to build a shared understanding about the problem space and
requirements.

The creation and documentation of a requirements strategy should be done in a
systematic way to ensure that all objectives are fully covered and understood
by all stakeholders. It should also frequently be reviewed, challenged, and
updated as the organization, the ways of working, and the product evolve over
time. Furthermore, a requirements strategy should also aim at aligning
different requirements stakeholders in terms of terminology, requirements
types and abstraction levels, roles and responsibilities, traceability,
planning of resources, etc.

Therefore, our contribution is a model of how requirements strategies should
be described for agile development. Through providing three complementary
perspectives, the proposed guidelines help to capture relevant information and
provide an useful overview. Our guidelines are summarized in Table 3,
including reoccurring examples and good practices abstracted from the three
case studies. We propose that a requirements strategy should include the
following building blocks: a structural perspective, an organizational
perspective, and a work and feature flow perspective. Across these
perspectives, a requirement strategy aims to support a shared understanding of
requirements, in particular with respect to establishing a _common language_
(i.e., enabling perspective in Table 4.1) and with respect to facilitating the
exchange and _flow of knowledge_ (i.e., building and assessing perspective in
Table 4.1).

We suggest to start with a structural view to create a common language. A good
starting point can be the artifacts in the development lifecycle model, for
example the requirements information model in the Scaled-Agile Framework SAFe
[16], or to define templates for user stories including customer value. Based
on these initial definitions, refinements can be provided based on experience,
e.g., after sprint reflections.

As a second step, we propose to make the organizational perspective explicit.
Define the roles and responsibilities with respect to the definitions in the
structural view. This can, for example, be done with a one-pager that
describes the responsibilities of a team. Also, state who owns which part of
requirements (e.g., requirements on certain subsystems) to determine specific
training needs.

Finally, the work and feature flow perspective needs to be defined. A good
starting point can be a lifecycle model for each critical type, which is then
mapped to the intended workflow. In agile development, this can partially be
provided by defining done criteria. In particular, it needs to be defined when
and by whom certain information must be provided. If requirements elicitation
efforts are anticipated, guidance should be given on obtaining the information
from stakeholders. The workflow should be related to the roles and
responsibilities as well as ownership. A stakeholder map can provide valuable
information: who owns an artifact, who should be kept informed, and who needs
to review it. An explicit review strategy can be very valuable, affecting not
only the requirements quality but also keeping reviewers informed about recent
changes.

##  6 Discussion and Conclusion

In this design science research study, we identified challenges related to
agile requirements engineering in three case companies. Based on these three
case studies, we identified solution strategies for resolving the identified
challenges and derived building blocks as substantial parts of a requirements
strategy. For each case we investigated a concrete requirements strategy. The
individual requirements strategies have been well received by experts in each
case company. Specifically, we recognize the need to enable, build, and assess
shared understanding of requirements in agile development. As our experience
grew, we noticed reoccurring building blocks on what should be part of such a
requirements strategy. For our design science research, we choose therefore
_guidelines for creating requirements strategies_ as our artifact, which we
develop in parallel to investigating our knowledge questions Our results
suggest that a requirements strategy should describe how requirements are
structured, how work is organized, and how RE is integrated in the agile work
and feature flow.

Buidling on previously published challenges and solution proposals for RE in
agile development (e.g. [1, 14]), our contribution is to enable organizations
to define a holistic approach to RE that integrates with their agile
development. Since our guidelines shall be applicable in agile development,
they do not primarily relate to explicit documentation or a dedicated
requirements phase within a development lifecycle, as for example custom in
waterfall processes. Instead, we rely on the theory of shared understanding to
embrace RE as a knowledge management problem and give suggestions on how
organizations can approach it in their agile development.

Ideally, such a strategy should be documented concisely and made available to
all stakeholders. Our requirements strategy can be interpreted as an instance
of situational method engineering [9] where we focus on the context of agile
system development and requirements methods in particular. By this, we aim to
make it easier for practitioners to integrate RE in their agile workflows.
This supports its evolution through the reflection opportunities built into
agile methods. We hope that our requirements strategy guidelines facilitate
future research on how to manage knowledge related to requirements in agile
development.

## References

  * [1] Alsaqaf, W., Daneva, M., Wieringa, R.: Quality requirements challenges in the context of large-scale distributed agile: An empirical study. Information and software technology 110, 39–55 (2019)
  * [2] Batsaikhan, O., Lin, Y.C.: Building a Shared Understanding of Customer Value in a Large-Scale Agile Organization: A Case Study. Master’s thesis, Dept. of Computer Science and Engineering, Chalmers $\mid$ University of Gothenburg (2018), https://hdl.handle.net/20.500.12380/304465
  * [3] Dikert, K., Paasivaara, M., Lassenius, C.: Challenges and success factors for large-scale agile transformations: A systematic literature review. Journal of Systems and Software 119, 87–108 (2016)
  * [4] Dingsøyr, T., Moe, N.B.: Towards principles of large-scale agile development. In: International Conference on Agile Software Development. pp. 1–8. Springer (2014)
  * [5] Dumitriu, F., Meșniță, G., Radu, L.D.: Challenges and solutions of applying large-scale agile at organizational level. Informatica Economica 23(3), 61–71 (2019)
  * [6] Elghariani, K., Kama, N.: Review on agile requirements engineering challenges. In: Int. Conf. on Computer and Information Sciences. pp. 507–512 (2016)
  * [7] Glinz, M., Fricker, S.A.: On shared understanding in software engineering: an essay. Computer Science-Research and Development 30(3), 363–376 (2015)
  * [8] Haskouri, N.E.: Requirement Strategy in Large-Scale Agile Development: A Design Science Research. Master’s thesis, Dept. of Computer Science and Engineering, Chalmers $\mid$ University of Gothenburg (2021), https://gupea.ub.gu.se/bitstream/2077/69096/1/gupea_2077_69096_1.pdf
  * [9] Henderson-Sellers, B., Ralyté, J.: Situational method engineering: state-of-the-art review. Journal of Universal Computer Science (2010)
  * [10] Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS quarterly 28(1), 75–105 (2004)
  * [11] Hoda, R., Noble, J., Marshall, S.: How much is just enough? some documentation patterns on agile projects. In: Proceedings of the 15th European Conference on Pattern Languages of Programs. pp. 1–13 (2010)
  * [12] Inayat, I., Salim, S.S., Marczak, S., Daneva, M., Shamshirband, S.: A systematic literature review on agile requirements engineering practices and challenges. Computers in human behavior 51, 915–929 (2015)
  * [13] Jorgensen, M.: Relationships between project size, agile practices, and successful software development: results and analysis. IEEE Software 36(2), 39–43 (2019)
  * [14] Kasauli, R., Knauss, E., Horkoff, J., Liebel, G., de Oliveira Neto, F.G.: Requirements engineering challenges and practices in large-scale agile system development. Journal of Systems and Software 172, 110851 (2021)
  * [15] Kasauli, R., Liebel, G., Knauss, E., Gopakumar, S., Kanagwa, B.: Requirements engineering challenges in large-scale agile system development. In: International Requirements Engineering Conference (RE). pp. 352–361 (2017)
  * [16] Knaster, R., Leffingwell, D.: SAFe 4.0 distilled: applying the Scaled Agile Framework for lean software and systems engineering (2017)
  * [17] Lagerberg, L., Skude, T., Emanuelsson, P., Sandahl, K., Ståhl, D.: The Impact of Agile Principles and Practices on Large-Scale Software Development Projects: A Multiple-Case Study of Two Projects at Ericsson. In: International Symposium on Empirical Software Engineering and Measurement. pp. 348–356 (2013)
  * [18] Larman, C.: Practices for scaling lean & Agile development: large, multisite, and offshore product development with large-scale scrum. Pearson Education (2010)
  * [19] Maxwell, J.: Understanding and validity in qualitative research. Harvard educational review 62(3), 279–301 (1992)
  * [20] Méndez, E.M., Pérez, M.A., Mendoza, L.E.: Improving software test strategy with a method to specify test cases (mstc). In: ICEIS (1). pp. 159–164 (2008)
  * [21] Meyer, B.: The ugly, the hype and the good: an assessment of the agile approach. In: Agile! pp. 149–154. Springer (2014)
  * [22] Paetsch, F., Eberlein, A., Maurer, F.: Requirements Engineering and Agile Software Development. In: International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises. pp. 308–313. IEEE (2003)
  * [23] Palinkas, L.A., Horwitz, S.M., Green, C.A., Wisdom, J.P., Duan, N., Hoagwood, K.: Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research 42(5), 533–544 (2015)
  * [24] Ramesh, B., Cao, L., Baskerville, R.: Agile requirements engineering practices and challenges: an empirical study. Information Systems Journal 20(5), 449–480 (2010)
  * [25] Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14(2), 131–164 (2009)
  * [26] Saldaña, J.: The coding manual for qualitative researchers. Sage, 3rd edn. (2015)
  * [27] Serrador, P., Pinto, J.K.: Does Agile work? A quantitative analysis of agile project success. International Journal of Project Management 33(5), 1040–1051 (2015)
  * [28] Vaishnavi, V., Kuechler, W.: Design Science Research Methods and Patterns: Innovating Information and Communication Technology. Taylor & Francis (2007)
  * [29] Wieringa, R.J.: Design science as nested problem solving. In: 4th Intl. Conf. on Design Science Research in Inf. Sys. and Techn. pp. 1–12. Philadelphia (2009)


# Does the First-Response Matter for Future Contributions? A Study of First
Contributions*
*The study was accepted at the MSR 2021 Registered Reports Track.

Noppadol Assavakamhaenghan1  Supatsara Wattanakriengkrai1  Naomichi Shimada1
Raula Gaikovina Kula1  Takashi Ishio1  Kenichi Matsumoto1
1Nara Institute of Science and Technology  Japan
Email: {noppadol.assavakamhaenghan.mt8  wattanakri.supatsara.ws3
shimada.naomichi.sm3
raula-k  ishio<EMAIL_ADDRESS>
###### Abstract

Context: Open Source Software (OSS) projects rely on a continuous stream of
new contributors for sustainable livelihood. Recent studies reported that new
contributors experience many barriers in their first contribution. One of the
critical barriers is the social barrier. Although a number of studies
investigated the social barriers to new contributors, to the best of our
knowledge, the relationship between the first response to the first
contributions and their future contributions has not been studied
comprehensively.

Objective: In this registered report, we introduce the study protocols that
investigate the correlation between the first response given to the first
contributions and the future contribution. First, we performed a preliminary
survey to manually explore the sentiments of the first response. Preliminary
analysis confirms that the first responses are mainly neutral.

Method: Our execution plan includes both qualitative and quantitative
approaches with three research questions. We inspect the first response of the
first contributions, investigate the effects of characteristics of the first
response to the interaction between first-time-contributor and project
contributors, and find the impact of the interactions between other
contributors.

##  I Introduction

GitHub is one of the well-known repository hosting tools that are being used
for Open Source Software (OSS) projects. In 2021, Github reported over 60
million repositories created in recent years and over 56 million contributors;
a massive number of new contributors strive to present their skills to the
world’s largest OSS community through project contributions.
111https://octoverse.github.com/ GitHub strives on social coding, which is the
collaboration of both new contributors and the current community of
contributors [1].

OSS projects truly rely on a continuous stream of new contributors [2]. A
large number of recent studies have provided evidence that new contributors
have encountered various barriers ranging from difficulty in finding
assistance to receiving negative responses from other contributors [3]. In
consequence, the new contributor turnover rate has been increased [4]. For
example, [2] reported that less than 18% of new contributors keep continuing
to contribute to the Hadoop Common Project. This possibly leads to the peril
of OSS projects’ livelihood and sustainability [5]. Zhou and Mockus [6]
revealed that the probability for a new contributor to becoming a long-term
contributor is associated with his/her willingness and environment. An
investigation on specific social coding tactics found that new contributors
who socialized with welcome and assistance messages and constructive criticism
tend to encourage future contributions[6]. Steinmacher et al. [4] mentioned
that new contributors had experienced delayed responses, impolite comments,
and many more issues regarding the social barriers. Furthermore, Gousios et
al. [7] reported that responsiveness is the most reported challenge that new
contributors experience.

Figure 1 shows a screenshot of a negative response to a new contributor on a
pull request in GitHub. In terms of GitHub projects, we consider any pull
request as a contribution, with the comments exchanged during the review of
the pull request as the interactions between developers. Hence, the first
response is the first comment to the pull request submitted by a newcomer to
that repository. In this case:

> ”This behavior seems undesired. I’d rather we kept the current behavior than
> follow npms current semantics.”

We hypothesize that this negative message may cause an unpleasant feeling and
subsequently lead to the discontinuity of their contributions (i.e., such as
further interactions within the same pull request or future contributions),
especially for a novice contribution that can be easily demotivated. Although
a number of studies investigated the social barriers to new contributors, to
the best of our knowledge, the relationship between the first response to the
first contributions and their future contributions has not been studied
comprehensively.

Figure 1: A screenshot example of a negative response to a new contributor.
The new contributor may not further interact with this response or may not
participate in future contributions.

Therefore, in this registered report, we lay a foundation for understanding
the correlation between the first response given to first contributions and
their future contribution. As a preliminary survey, we sampled a set of OSS
projects from the npm ecosystem (NodeJs packages). Although there is a
considerably large portion of the first contribution, i.e., 27.66% with mainly
neutral first response. Our execution plan includes both qualitative and
quantitative approaches with three research questions. First, we inspect the
first response of the first contributions. Second, we investigate the effects
of characteristics of the first response to the interactions between first-
time contributors to find the impact of future contributions.

##  II Preliminary Analysis

The goal of the preliminary analysis is to manually explore whether sentiment
analysis can be useful to detect the sentiment of a first response. Since
communities are more likely to share the same terminologies and communication
culture, we choose projects which belong to the npm ecosystem, which is one of
the largest software ecosystems.

TABLE I: Preliminary Dataset Statistics  Contributions (Pull Requests)
---
Total # of Projects | 20
Total # of Contributions per Project | 43,398
Min. # of Contributions per Project | 674
Max # of Contributions per Project | 5092
Min. # of First Contributions per Project | 179
Max. # of First Contributions per Project | 2024
% First Time Contributions | 27.66% (out of 43,398)
Interactions (Comment)
Total of Interaction in Pull Request | 8,534
Min. # of Interaction per contribution | 0
Max # of Interaction per contribution | 111

##### Collected Dataset

We collected a list of npm projects from two existing datasets
(GHTorrent222https://ghtorrent.org/ and Libraries.io333https://libraries.io/).
GHTorrent provides a mirror of git repositories and developer interactions
gathered from GitHub444https://github.co.jp/, while Libraries.io provides the
meta-data and the relationships among packages that are hosted on popular
software ecosystems, such as npm and Maven. There are over 1.3 million npm
projects. Then, by adding a filter of only active projects (projects that
contain at least one commit, pull request, or issues) in 2020, we are left
with 11,127 projects.

For the preliminary study, we selected the top 20 npm projects with the most
contributors. From this, we obtain 43,398 pull requests. We observed that
12,003 (27.66%) of pull requests are first contributions (i.e., created by the
users who have never create any pull request in the project before), while
only 8,534 (71.10%) of pull requests of first-time-contributions receive
responses. There are 11,268 first-time contributors. Our dataset statistics
are further displayed in Table I.

#####  Sentiment Analysis of First Response

As we plan to find the sentiment of the first responses, a sentiment analysis
tool is needed. We selected SentiStrength-SE [8] as our sentiment analysis
tools since it is the state-of-the-art tool for sentiment analysis of the
software engineering-related text. We calculated the sentiment score of the
first responses using SentiStrength-SE for our preliminary analysis. The tool
receives a text as an input, and it then gives a sentiment polarity score of
the given text. We applied the tool to the first responses and inspected the
responses which their sentiment score are quite high and low. The output is
ranged from a scale of -5, which refers to very negative sentiment, to 5,
which refers to very positive sentiment.

Distribution of sentiments. We found that the sentiment score of the first
responses is ranged from -3 to 4. The distribution of the sentiment score of
the first responses is shown in Figure 2. To explore what kinds of words are
in each score, we use keyword extraction tools to extract keywords from each
sentiment score.555https://monkeylearn.com/keyword-extractor-online/ The
results are shown in Table II.

We found that SentiStrength-SE can appropriately capture a positive sentiment
since it captures many encouraging words, e.g., ’Excellenct,’ ’Nice attempt’
and ’Great job.’ However, in the negative sentiment, keywords do not directly
show displease words. We have also looked into the actual responses and found
some examples of the high and low sentiment scores. We found that encouraging
phrases, e.g., ’Excellent’ and ’Thank for doing this’ is normally in the text
with high sentiment score; and phrases such as ’I don’t like this,’ ’This is
not good,’ etc. are in the text with a low sentiment score.

Figure 2: The distribution of sentiment score of the first response to first contributions TABLE II: Keywords from the First Response Sentiment Score | Keyword
---|---
-3 | movies, odd whitespace changes, lodash, changes, accent colour, git novice, wrong branch, design changes, better way, visual changes
-2 | mithril, user agent reporting, early next week, code, changes, way, minutes, thing, screencap stuff, fork tests
-1 | error, obscure edge cases, data, deserialize, later today, return JSON.parse, mon Magazine, code size, Rookie mistake, contrived example
0 | double colon, docs task, foo, corresponding object key, route parameter syntax, m.request url option
variadic function syntax, super intuitive syntex, minimal test case, keys
1 | Thanks, catch, pull request, nice work, SCSS, origin/next, work, site
2 | Thanks, custom date formatting, first browserify package, minor comments, Thanks Jeremy, good cleanup, first pass, Great work,
pull reqeust, gulp file
3 | Thanks, simple registered elements, something, good catch, async data-dependencies, Nice attempt, extended elements, Great job,
userLand, redaw.strategy
4 | Excellent, Wow

Summary: Preliminary analysis shows that most first responses are neutral. For
keywords and phrases, encouraging phrases, e.g., ’Excellent’ and ’Thank for
doing this’ appear with a high sentiment score. Differently, phrases such as
’I don’t like this’, ’This is not good’, etc. appear with a low sentiment
score.

##  III Study Protocols

In this section, we present the design of our study. This section includes
data selection, sanity check, and the preparation of research questions with
their motivation.

##### Data Selection

To generalize the results, we will perform our full experiment on Hata et al.
[9] dataset, which contains a list of many open-source software projects in
seven well-known programming languages, e.g., C, C++, Java, JavaScript,
Python, PHP, Ruby. There is a total of 29,234 repositories in the dataset. The
pull request will be extracted from the projects. We will select only closed
pull requests as our dataset to ensure the correctness of the result. Since we
investigate the contributors, we also consider the threat of multiple GitHub
alias and bots. Hence, we adopt an identity merging bot detection tool to
mitigate these threats. For identity merging, we follow approaches similar to
Fry et al. [10] and also adopt bot detection techniques Golzadeh et al. [11].

##### Sanity Check

We will use the SentiStrength-SE, Sarker et al. [12] toxicity detection tools,
EmoTxt[13] to investigate the characteristics of the first response. Even
though the tools are well trained on the Software Engineering-related dataset,
we would like to ensure the validity of the tools when applying them to our
dataset. Hence, we will perform a sanity check on the three tools. The
processes of the sanity check are as follows: We will sample the first
responses that are positive, negative, and neutral by the score given by the
sentiments, toxicity, and emotions. To ensure the significance of the sample,
we will determine the sample sizes by using a confidence level of 95% and a
confidence interval of 5%.666https://www.surveysystem.com/sscalc.htm Then,
three of the authors will perform the manual validation of each type of
sentiment score. To calculate the interrater agreement between the manual
classification results of the three authors, the Cohen’s kappa[14] approach
will be adopted. We will then report the precision, recall, F1 score, and
AUROC (Area Under the ROC Curve) of the tools with our manual classification’s
kappa agreements score.

##### Research Questions

To guide our research towards the goal, we designed the following research
questions:

RQ1: What are the characteristics of the first responses toward first
contributors? Our motivation for the first research question is to understand
the first response’s characteristic to the first contributions in terms of
sentiment, responsiveness, toxicity, and emotion of the response. The
definition and extraction procedure of each characteristic will be explained
in Section IV. The first response is the first comment on a pull request. As
for baseline, we will compare the characteristic of the first response of the
first contributions and ones of non-first contributions. We envision that this
analysis will reveal insights on whether there are biases against the first
pull request. Our assumption is that:

H1.1: First contributions are more likely to get positive responses compared
to non-first contributions. Since the community tries to attract more
contributors, we can expect that the community gives positive responses to the
first contributions.

H1.2: First contribution are more likely to get more responsive responses
compared to non-first contributions. Similar to H1.2, we expect that the
community gives responsive responses to the first contributions.

H1.3: There are biases in giving toxic responses between first contribution
and non-first contribution. Similar to H1.1, we expect that the community will
be bias in giving the first response to the first contribution in terms of
toxicity.

H1.4: Each emotion type is not evenly distributed in first contributions and
non-first contributions. Similar to H1.1, we expect that the emotion in the
first responses of the first contribution will be different from the ones of
the non-first contribution. For example, there could be a ’surprise’ emotion
when first-time contributors make a huge contribution to the project while it
seems normal to non-first-time contributors.

RQ2: What is the relationship between receiving a positive response and
further interactions in the same pull request for a first-time contributor?
Our motivation for the second research question is to understand whether the
first responses affect future interactions between first-time contributors and
project contributors. In this study, we consider comments on pull requests as
interactions between contributors. We expect that this analysis will reveal
the effects of the first responses upon the first-time contributors. Our
assumption is that:

H2.1: First-time contributors are more likely to interact with non-first-time
contributors after the first responses when they receive positive responses.
Since receiving positive responses encourage the first-time contributors’
willingness to collaborate [15], we can expect that the first-time
contributors will be likely to interact afterward.

H2.2: First-time contributors are more likely to interact with non-first-time
contributors after the first responses when they receive responsive responses.
As receiving a responsive response can keep the first-time contributors’
attention, i.e., the first-time contributors do not wait too long, we can
expect that the first-time contributors will interact with others when
receiving responsive responses.

RQ3: What is the relationship between first-time contributors’ interactions
and their future contributions? Answering this research question will provide
us with insights into factors that affect the future contributions of first-
time contributors. This possibly leads to potential solutions to avoid first-
time contributors abandon OSS projects.

##  IV Execution Plan

We plan to use both qualitative and quantitative approaches to answer our
research questions.

###  IV-A Research Method for RQ1:

For the first research question, we plan to use a quantitative method. We will
separate contributions into two groups: the first contribution and the non-
first contribution. The characteristics of the first responses of both groups
will be separately extracted. To extract the sentiment of the first responses,
we plan to use SentiStrength-SE [8], the state-of-the-art sentiment analysis
tool for software engineering text. The tool can be used to determine the
scale of the polarity score of the responses. The sentiment score is ranged
from -5 (very negative) to 5 (very positive). For the responsiveness of the
first responses, we will extract the duration between the creation of the pull
requests and the time when the first comment is posted. For toxicity
detection, we will adopt Sarker et al. [12] tool. This tool is a binary
classification tool that receives text as the input, and it then reports
whether the text contains toxicity or not. For emotion detection, we will use
EmoTxt [13]. This tool is a binary classification tool that receives text as
the input. It then classifies the existence of the emotion in the text. The
supported emotion is ”Love,” ”Joy,” ”Surprise,” ”Anger,” ”Sadness,” and
”Fear.”

##### Analysis Plan

To analyze the result of the RQ1, we plan to compare the sentiment and the
responsiveness between the first contribution and non-first contribution group
to answer the H1.1 and H1.2. We will report the result in the box plots. The
box plots will show the difference in the distribution of sentiment
score/responsiveness of the two groups. For the H1.3 and H1.4, we will report
the number of first responses classified as Toxicity and each emotion in a
table.

##### Significant Testing

Before applying the statistical test, we will inspect the data to see whether
it is normally distributed or not. This will allow us to select a suitable
test. To test normality, we will adopt Shapiro-Wilk test [16] with alpha =
0.05. For example, we will perform a Shapiro-Wilk test on the first responses’
sentiment score to first contributions and non-first contributions. If we
receive a p-value greater than 0.05, it will imply that the distribution of
the sentiment score is not significantly different from the normal
distribution. In other words, we can assume normality. If sentiment in the two
groups’ first responses is normally distributed, we will select a two-tailed
independent t-test with alpha = 0.05. This is because our sample group, which
are first contributions and non-first contributions, are independent. In case
when the data of the two groups are not normally distributed, we will adopt a
two-tailed Mann Whitney U test [17] with alpha = 0.05. This is because it is a
non-parametric statistical test, and there is no normality assumption. The
test will be performed on the followings hypothesis to answer H1.1 and H1.2:

$H1.1_{null}$: There is no difference between the sentiment score of the first
response of first contributions and non-first contributions.

$H1.2_{null}$: There is no difference between the responsiveness of the first
response of first contributions and non-first contributions.

We plan to investigate the effect size. The different methods are needed
according to whether the data of the two groups are normally distributed or
not. We will use Hedges g effect size [18], which is a parametric test, to
measure the effect size if the data is normally distributed. Effect size is
analyzed as follows: $|d|<0.2$ as ”negligible”, $|d|<0.5$ as ”small”,
$|d|<0.8$ as ”medium”, otherwise ”large”. If the data are not normally
distributed, we will apply Cliff’s $\delta$, which is a non-parametric effect
size measure (Romano et al, 2006). Effect size is analyzed as follows: (1)
$|\delta|<0.147$ as Negligible, (2) $0.147\leq|\delta|<0.33$ as Small, (3)
$0.33\leq|\delta|<0.474$ as Medium, or (4) $0.474\leq|\delta|$ as Large. To
analyze Hedges g and Cliff’s $\delta$, we use the effsize R
package.777https://cran.r-project.org/web/packages/effsize/

We need different statistical test for H1.3 and H1.4 because the dependent
variables, i.e. toxicity and emotion, are categorical. Thus, we will use
Pearson’s chi-squared test ($X^{2}$) [19] to test the following hypothesis to
answer H1.3 and H1.4:

$H1.3_{null}$: Toxicity is evenly distributed in the first response to the
first contributions and non-first contributions.

$H1.4_{null}$: Each emotion type is evenly distributed in the first responses
to the first contributions and non-first contributions.

###  IV-B Research Method for RQ2:

For RQ2, we plan to use both qualitative and quantitative approaches. The
first-time contributors will be separated into 2 groups: ones that interacted
with project contributors and ones that did not interact with project
contributors in their first contributions. As for the group separation
criteria, we consider the first contributions interacted with project
contributors if they responded at least once after receiving the first
response; otherwise, we consider them as no interaction.

##### Analysis Plan

The sentiment and the responsiveness of the first responses will be used to
compare between two groups. Similar to RQ1, we will report results in the box
plots. Furthermore, we will incorporate a qualitative analysis to reveal
reasons for negative responses. To do this, we will first select a sample of
the negative responses. We will determine the sample sizes using a confidence
level of 95% and a confidence interval of 5%. Then, we perform our qualitative
analysis of the reason for each negative response by following the card
sorting method as in Li et al. [20]. Furthermore, we report the kappa
agreement score.

##### Significant Testing

In order to statistically validate the differences between the two groups,
i.e., ones that interacted with project contributors and ones that did not
interact with project contributors in their first contributions, we plan to
perform the same statistical test as H1.1 and H1.2. The test will be performed
on the followings hypothesis:

$H2.1_{null}$: There is no difference between the sentiment score of the first
response of the first-time contributors that interacted with project
contributors and one that did not interact

$H2.2_{null}$: There is no difference between the responsiveness of the first
response of the first-time contributors that interacted with project
contributors and one that did not interact.

As same as RQ1, we investigate effect size using Cliff’s $\delta$.

###  IV-C Research Method for RQ3:

For RQ3, we plan to use a quantitative method. As we would like to see the
correlation between the interactions between first-time contributors and
project contributors and future contributions. We will train classification
models, e.g., logistic regression, random forest, and SVM. The model will
train on independent variables by leveraging dependent variables. The
followings are the variables that mainly capture the characteristics of the
first-time contributors’ interaction that we focus on: (I) Sentiment of First
Response: We will extract this variable by using SentiStrength-SE [8] as same
as in RQ1 and RQ2. (II) Responsiveness of First Response : This variable will
be calculated as the duration between the pull request creation date and the
first response. (III) Number of interactions : We plan to calculate this
variable by considering the total number of comments on a pull request. More
specifically, we will count the number of comments that were written after the
first response until the last comment of the first-time contributors on the
pull request. (IV) Number of Words in the First Response: This variable is the
total words in the first response. (V) Result of the Pull Request: This is a
binary variable that shows whether the pull request is merged or not. (VI)
Existence of toxicity in the First Response: This is the binary variable that
shows whether the first response contains toxicity or not. (VII) Emotion in
the First Response: As the EmoTxt allows us to detect each emotion separately,
we will consider six binary variables which describe the emotion in the first
response. For example, ”Anger” will be a binary variable in the first
response.

Since contributing to OSS projects contains much more dimensions of factors
than the first response’s dimension, we will incorporate other dimensions of
factors as well. Contributors Dimension variables will be the variables that
capture the contributors-side characteristic, e.g., first-time contributors’
experience. Project Dimension variables will capture the project-side
characteristic, e.g., project-size related features. Contribution Dimension
variables will capture the contribution-side characteristic, e.g., size of the
contribution.

The dependent variable is the Future Contribution. This variable is binary
that shows whether a first-time contributor makes a future contribution. In
order to determine whether the first-time contributor will continuously
contribute or not, we will inspect the time between the first and next
contribution (i.e., submitted pull request) of the first-time contributor in
each dataset. We then select the maximum days as the threshold to consider
that the first-time contributor has left the project.

####  IV-C1 Analysis Plan

Firstly, we will inspect how well each model fits the data. We will apply
10-fold cross-validation and report the precision, recall, F1 score, and AUROC
(Area Under the ROC Curve) of each model. Then, we will select the best
suitable model for the feature importance analysis. We will apply feature
permutation importance to see how much the model depends on each variable.
888https://scikit-learn.org/stable/modules/permutation_importance.html By
doing this, we will be able to compare the feature importance between the
first-time contributors’ interaction variables against the other dimension of
features. To see how each variable is associated with Future Contribution, we
will calculate pairwise correlations. Then, we will report and plot the
correlation matrix in a
heatmap.999https://seaborn.pydata.org/examples/many_pairwise_correlations.html

##  V Implications

We summarize our implications for the key stakeholders:

OSS Projects Our findings will bring insights into how the interaction between
first-time contributors and project contributors in OSS projects affects
future contributions of the first-time contributors. We believe that fast and
appropriate responses to first-time contributors are needed to attract future
contributions to OSS projects. Automated tools, e.g., bots, could promptly
give appropriate responses to contributors when they make pull requests or
could notify project contributors when they are likely to give negative
responses.

Researchers This study will reveal interaction-related factors matter for
future contributors and to what extent does the factors affect OSS projects’
contributions. First-time contributors-related future studies should consider
and could further investigate these factors in other proprietary industries,
such as Nonprofit Organization (NPO) and OSS projects. Implementing tools such
as bots that automatically send the first responses to first contributions and
contributors and measure the sentiment of responses could be useful for future
studies.

Contributors This study helps contributors to understand how important their
interactions are. They should consider the response sentiment and response
time in the review process since the discussion on pull requests can
explicitly affect others’ contribution intention and the number of first
contributions.

##  VI Threats to validity

We summarize four key threats. (I) Multiple Identity. To deal with multiple
GitHub account, we will adopt an identity merging technique to mitigate this
threat similar to Fry et al. [10]. (II) Bots. Bots [21] may threaten our
calculation of text analysis, e.g., sentiment, toxicity, and emotion. We
mitigate this by adopting a tool to identify bots in the OSS projects [11].
(III) No Response Pull-request Selection Process. To mitigate incorrectly no
response pull-requests, we will filter out ongoing pull-request and left only
closed pull-request. (IV) No Future Contribution Selection Process. To
mitigate the incorrectly identified ”no future contribution” status of the
first-time contributors, we plan to leave a time window for a selected pull
request.

## Acknowledgment

This work was supported by JSPS KAKENHI Grant JP20H05706, JP18H03221,
JP20K19774 and JP18H04094.

## References

  * Choi et al. [2010] B. Choi, K. Alexander, R. E. Kraut, and J. M. Levine, “Socialization tactics in wikipedia and their effects,” _Proc. of the ACM Conf. on Computer Supported Cooperative Work_ , pp. 107–116, 2010.
  * Steinmacher et al. [2013] I. Steinmacher, I. Wiese, A. P. Chaves, and M. A. Gerosa, “Why do newcomers abandon open source software projects?” _Proc. of the Inter. Workshop on Cooperative and Human Aspects of Soft. Engr._ , pp. 25–32, 2013.
  * Steinmacher et al. [2015] I. Steinmacher, T. U. Conte, M. A. Gerosa, and D. F. Redmiles, “Social barriers faced by newcomers placing their first contribution in open source software projects,” _Proc. of the ACM Inter. Conf. on Computer-Supported Cooperative Work and Social Computing_ , pp. 1379–1392, 2015\.
  * Steinmacher et al. [2014] I. Steinmacher, I. S. Wiese, T. Conte, M. A. Gerosa, and D. Redmiles, “The hard life of open source software project newcomers,” in _Proc. of the Inter. Workshop on Cooperative and Human Aspects of Soft. Engr._ , 2014, p. 72–78.
  * Valiev et al. [2018] M. Valiev, B. Vasilescu, and J. Herbsleb, “Ecosystem-level determinants of sustained activity in open-source projects: A case study of the pypi ecosystem,” in _Proc. of the ACM Joint Meeting on European Soft. Engr. Conf. and Symposium on the Foundations of Soft. Engr._ , 2018, p. 644–655.
  * Zhou and Mockus [2012] M. Zhou and A. Mockus, “What make long term contributors: Willingness and opportunity in oss community,” in _Proc. of the Inter. Conf. on Soft. Engr._ , 2012, pp. 518–528.
  * Gousios et al. [2016] G. Gousios, M.-A. Storey, and A. Bacchelli, “Work practices and challenges in pull-based development: The contributor’s perspective,” in _Proc. of the Inter. Conf. on Soft. Engr._ , 2016, p. 285–296.
  * Islam and Zibran [2018] M. R. Islam and M. F. Zibran, “Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,” _Journal of Sys. and Soft._ , pp. 125 – 146, 2018.
  * Hata et al. [2019] H. Hata, C. Treude, R. G. Kula, and T. Ishio, “9.6 million links in source code comments: Purpose, evolution, and decay,” in _Proc. of the Inter. Conf. on Soft. Engr._ , 2019, p. 1211–1221.
  * Fry et al. [2020] T. Fry, T. Dey, A. Karnauch, and A. Mockus, “A dataset and an approach for identity resolution of 38 million author ids extracted from 2b git commits,” in _Proc. of the Inter. Conf. on Mining Soft. Repo._ , 2020, p. 518–522.
  * Golzadeh et al. [2021] M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in github issue and pr comments,” _Journal of Sys. and Soft._ , 2021.
  * Sarker et al. [2020] J. Sarker, A. K. Turzo, and A. Bosu, “A benchmark study of the contemporary toxicity detectors on software engineering interactions,” in _Proc. of the Asia-Pacific Soft. Engr. Conf._ , 2020, pp. 218–227.
  * Calefato et al. [2017] F. Calefato, F. Lanubile, and N. Novielli, “Emotxt: A toolkit for emotion recognition from text,” in _Proc. of Inter. Conf. on Affective Computing and Intelligent Interaction Workshops and Demos_ , 2017, pp. 79–80.
  * Viera and Garrett [2005] A. Viera and J. Garrett, “Understanding interobserver agreement: the kappa statistic.” _Family medicine_ , 2005.
  * Grigore and Rosenkranz [2011] M. Grigore and C. Rosenkranz, “Increasing the willingness to collaborate online: an analysis of sentiment-driven interactions in peer content production,” in _Proc. of the Inter. Conf. on Information Sys._ , 2011.
  * SHAPIRO and WILK [1965] S. S. SHAPIRO and M. B. WILK, “An analysis of variance test for normality (complete samples)†,” _Biometrika_ , pp. 591–611, 1965.
  * Mann and Whitney [1947] H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” _Ann. Math. Statist._ , pp. 50–60, 1947.
  * Hedges and Olkin [1985] L. Hedges and I. Olkin, _Statistical Methods in Meta-Analysis_ , 1985.
  * Pearson [1900] K. Pearson, “X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , pp. 157–175, 1900.
  * Li et al. [2021] Z. Li, Y. Yu, T. Wang, G. Yin, S. Li, and H. Wang, “Are you still working on this an empirical study on pull request abandonment,” _IEEE Transactions on Soft. Engr._ , pp. 1–1, 2021.
  * Dey et al. [2020] T. Dey, S. Mousavi, E. Ponce, T. Fry, B. Vasilescu, A. Filippova, and A. Mockus, “Detecting and characterizing bots that commit code,” in _Proc. of the Inter. Conf. on Mining Soft. Repo._ , 2020, p. 209–219.