The decision of which backlog to assess, selection of a representative sample of items from the backlog, team members to involve - as well as the details of the actual assessment are detailed in Fig. 3. The following sub-sections describes each step of BAM in more detail.

#### **4.1 Step 1: Identify Relevant Backlog**

An organization may have several products, thus several product backlogs, or different type of backlogs for the same product, as seen in Step 1, Fig. 3. Having multiple backlogs at several different levels is not an uncommon way to organize the work and requirements in industry. In particular for large organizations where the work requires decisions on multiple levels to select what to realize in the next phase/sprint. The case company Telenor has, in essence, three types of backlogs for a product, one backlog before the feasibility review, one product backlog, and one sprint backlog as illustrated in Fig. 1. In the first backlog all incoming requirements/ideas/features (called items from hereon in) are initially gathered before a feasibility review takes place. An item is then, either dismissed or it progresses to the product backlog, and then to the sprint backlogs. Each of these backlogs contains items that are inherently specified in different ways. That is, the scope in the initial backlog is wider with less details, and in the later backlogs the information is refined and detailed. This is not unlike any engineering process where requirements (items) are refined and evolve as they progress through the organization, from potential idea to on-queue for development. Weather or not there are one or several types of backlogs, Step 1 in BAM is to select a backlog to assess. A general tip here is, not to see the selection as finite as you could study several backlogs over time.

**Fig. 3.** The five main steps in the backlog assessment method (BAM) (Color figure online)

#### **4.2 Step 2: Select Representative Sample of Items from the Backlog**

A backlog consists of items that should be included in the product or action taken in relation to it [3]. Regardless of what items that are present in a backlog, they should be written as user stories [8]. However, in practice, not all items that should be included in the product are included in the backlog, and not all items are written as user stories. Instead, it is common to write items using natural language [24,25]. Telenor Sweden uses several different styles of writing/specifying requirements/backlog items, including use-case diagrams, use-case scenarios, and natural language, as seen in Step 2, Fig. 3. This is not surprising as different specification styles are appropriate at different levels in the decision chain as the abstraction levels differ, and the items are used for different purposes [26].

To have a representative assessment of a backlog, it is important to include items written in all of the existing specification styles that are present in the backlog. Also, it is important to have a good and manageable sample size of items to be able to perform a simple and easy assessment of a backlog - it is better to perform several small assessments than one big one. Whether or not there are items written in one or several specification styles, Step 2 in BAM is to select a representative sample of items from the backlog. To select a representative sample, 10% of all the items, which in the case company was about 100 items, was considered appropriate. However, if the backlog contains thousands of items, then 10% may be too many items. In this case, it is better to select between 50 and 100 items as this size is manageable from an effort perspective (taking one day). In this case representativeness of the selection comes second to the scalability of the assessment.

#### **4.3 Step 3: Identify Relevant Stakeholders**

The next step is to identify the relevant stakeholders, which may include the team(s) working with the backlog, and/or any other role(s) that use the items in the backlog for, e.g. decision-making, planning future products/sprints, estimations, coordination with other teams/products, or prioritization. The identification of stakeholders could be a simple one-to-one mapping, one backlog and one team working with the backlog, or there could be multiple teams working with the same backlog, (Step 3, Fig. 3). At Telenor there is usually one team with various roles from different departments/areas working with the backlog in the feasibility phase, while other teams work with the product and sprint backlogs. That is, it is not the same team/roles that work with the items from potential ideas to on-queue for development. This is an important characteristic of the case in hand, but also for several organizations that try to use backlogs as a coordination mechanism outside development projects. Multiple roles, multiple teams, representing one or several parts of the organization and/or several products might be relevant to involve in the assessment. Regardless of how many teams/roles that are working with the selected backlog, Step 3 in BAM is to identify the relevant stakeholders for the assessment of the backlog. During the assessment of a backlog at Telenor Sweden, four stakeholders were selected to perform the assessment. It is a good start to start with 4–6 stakeholders to get a first quick overview of the backlog. It should be observed that in our case some stakeholders had multiple roles and thus were involved in several decision points.

## **4.4 Step 4: BAM Scales**

At the core, BAM consists of four *BAM perspectives*, as seen in Step 4, Fig. 3. These are used to grade individual items in the backlog. The four *BAM perspectives* are described in below.

*Scope*.  The scope of items in the initial backlog is generally wider, e.g. having the scope of the entire enterprise; while in the later backlogs (e.g. product or sprint) the scope often narrows - a natural consequence of refinement and break down. From a scope perspective, it is important that the stakeholders working with a backlog item can conceptualize the item in relation to the length of the sprint. Otherwise, they may have difficulties to break down the item [27]. The scope is different as breakdown of all items directly would result in wasted resources as many items put in the initial backlog that are compared and then passed on refined, or also often dismissed as other items are relatively more important to realize.

*Abstraction/Level.* Does the item address a local algorithm, a feature, or a vision? The *Abstraction/Level* of an item can differ even if the *Scope* is the same. For example, if an item has the *Scope* of a sub-system, the item can address a local algorithm for the sub-system, or it can address the future vision of the same sub-system.

*Maturity.* How often does an item change? One of the "tools" used in agile to embrace change is actually the backlog itself, which should accommodate easy change to items. Albeit change is to be embraced - maturity is connected to change. In an initial backlog (pre-project, where items are collected for e.g. a feasibility review) item may change on a daily basis. However, stability should increase in later backlogs - even if change is still a reality and has to be embraced.

*Detail.* What is the level of specification of an item. Items in a backlog should have different levels of detail depending on what backlog they are specified in. However, the closer to realization (sprint backlog) an item comes, generally the more detail is present to reflect the analysis and choices made for its realization. Greater detail in later backlogs allow team members to know what they are committing to [27]. However, even in the early backlog it is important to have adequate details about the items (good-enough for decisions and prioritization) otherwise it may result in wrongly dismissed or selected items. Too much detail is also undesirable as it would constitute waste. BAM does not suggest a "correct" level, rather it offers a visual evaluation of current state of items in a backlog, and gives practitioners the ability to judge if the level is appropriate, or not, given their needs in relation to a specific decision point and backlog.

#### **4.5 Step 5: Assessment of the Backlog**

In the fifth and final step of BAM, the identified stakeholders perform the assessment. Each stakeholder individually estimates items by grading them using each of the four perspectives from Step 4. The assessment is performed using predefined templates as seen in Step 4, Fig. 3. The template can be used in two ways, (1) using one template for each item, or (2) adding several items to the same template. The case company used one template for several items (see Fig. 4, Items A - D) by placing the item's ID on the scale (to preserve confidentiality, the two examples of detail for Items A and C are anonymized in terms of feature detail). For example, Item A has the *Scope* of an isolated change, the *Abstraction/Level* is graded as vision, the *Maturity* of the information for Item A is in-flux, and the *Detail* of Item A is specified as a one-liner.

**Fig. 4.** Example of an assessment of two backlog items

The reason for adding the item's ID in the template is to be able to perform a sanity check of the assessment. At Telenor, the sanity check was used for two main reasons. First, to have concrete examples of items when presenting the overall assessment makes it easier to understand and discuss the assessment. Second, the items were used as input if one or more stakeholders had completely different assessments of the same item. Thus, the provided item was used to calibrate the overall assessment. A notable observation is that the actual estimation itself served as a learning experience and coordination effort between different roles and competences, in essence building a shared understanding of not only the perspectives themselves, but more importantly of the needs of fellow co-workers. Thus details in an item that was considered as unnecessary by one stakeholder was important for another. This allowed for an understanding to be built, as well as the realization that one person's "waste" could be another person's critical information.

When all stakeholders have completed their assessment, an overview of the results is constructed (Step 5, Fig. 3). The constructed overview comes in two parts. First, the overview shows the total estimated span of items in the backlog for each of the four perspectives combined with the overlap of all stakeholders. At Telenor Sweden, the four stakeholders had different opinions about the span of items in their backlog (see Step 5, Fig. 3). Each of the colored lines represents one stakeholder. For example, one stakeholder believed that the *Scope* of the items span from an isolated change to the platform (blue line), while another stakeholder believed that the *Scope* spans from an isolated change to the application (green line). The grey boxes represent the shared view among the stakeholders for each scale. Second, the assessment results can also be used to create "typical profiles" of items that exist in the same backlog, as seen in Step 5, Fig. 3. Four typical item profiles were identified in one backlog at Telenor. For example, one typical profile (profile Item A, Step 5 in Fig. 3) was items with a narrow *Scope*, wide *Abstraction/Level*, with constantly changing information (*Maturity*) that was specified with little *Detail*. In the same backlog, other items (profile Item D) had a wider *Scope*, more *Mature* information that was specified in greater *Detail*.

## **5 BAM Case Study Evaluation**

As BAM was completed, it was set to be evaluated in an industrial environment in a real development project. The idea was to use BAM for an actual backlog and backlog items in order to evaluate its steps and components (e.g. the four BAM perspectives), and to check if BAM could help in assessing backlogs as a process assessment activity in order to help agile teams to continuously reflect on how to become more effective, efficient and coordinated.

*Research Design.* The selection of backlog and practitioners for participating in the evaluation was conducted in cooperation with the case company. Seven practitioners representing different roles were selected. The roles chosen were: 1 Process developer, 2 Business analysts, 2 Product managers, 1 Project manager, and 1 Operational manager. The selected practitioners identified 100 backlog items that were used as a representative sample of items from the backlog (Step 2 in BAM, Fig. 3). Then the practitioners used the template for BAM perspectives (Step 4 in BAM, Fig. 3) to perform the assessment of the backlog items (Step 5 in BAM, Fig. 3). After the practitioners had used BAM, data - in terms of BAM's usability and usefulness - was collected. Semi-structured interviews [20] were used to ask questions about how BAM was perceived by the practitioners. The interviews were carried out by one interviewer and one interviewee. For all interviews, we took records in form of written extensive notes. The extensive notes were analyzed using content analysis [20].

## **5.1 Lessons Learned**

Through the evaluation of BAM with Telenor Sweden, we observed experiences summarized below as lessons learned.

*Ease of Use.* In general, the practitioners at Telenor found that BAM is easy to understand, and that the five steps of BAM make sense for assessing a backlog. The practitioners explained that the steps of BAM are helpful and easy to follow, and in particular the provided examples for each step (Fig. 3). In addition, BAM does not seem to take too much time to use in practice. An assessment of about 100 items for one backlog with 4–5 practitioners did not take more than a day, and if the number of items is decreased to 30–40 items, then the time spent on the assessment is only 3–4 h, which could be done on a regular basis by individual teams without formalizing it and getting "permission" or a budget.

*Identification of Gaps.* Using BAM as a tool to perform analysis of items in a specific backlog, and the decision process associated with decision points tied to the backlog, enabled us to identify gaps. This in turn gave input to either put some criteria as to good-enough item analysis and specification, and/or changes to the decision process. Often slight modifications of both are needed as per observations at Telenor Sweden. This gave potential to remove the likelihood of undesired consequences (as explained in Figure 2), but also speed up processing and decrease the clutter in the backlog of lingering items. BAM does not introduce the "correct" levels, rather, allows an estimation of span of the items in a backlog according to the four perspectives - which in turn can be compared to the decision making process and the capabilities of a team responsible for a specific backlog. Just as extensive technical detail might not be necessary for a quick feasibility analysis, abstract business goals are too abstract for a technical development team. BAM thus supports:

BAM will not in-itself solve undesired consequences (e.g. UDC1 - UDC3). However, by assessing the backlog items it is possible to get an overview of the level of information in the backlog, and get the ability to see the typical item profiles and level of span of said items. This can then be used to identify potential issues with the alignment between the backlog and the decision-making process. If the decision-making process is not aligned with the backlog, there is a risk that the teams either spend more time and resources in analyzing the backlog items to make sure that all of them have a similar level of information, or that the team may want to change the decision-making process and criteria to handle all kinds of item profiles by, e.g. changing team member profiles or adding more team members. BAM can also successfully be complemented with a Value Stream Mapping [15] activity to gauge the delays and times of items in play, and the changes as backlog item details and decision processes are tweaked.

#### **5.2 Limitations**

Selection bias (*construct validity*) [21], in terms of selecting participants to interview in relation to the process assessment and the evaluation of BAM, is a threat to this study. Selection bias is always a threat to all studies where the participants are not fully randomly sampled. The threat is that only participants with a negative attitude of the current processes and ways-of-working, and/or having a positive attitude towards BAM were selected. To minimize this threat, the participants were selected based on their experiences and roles by a "gate-keeper" at Telenor Sweden. That is, the researchers did not influence the selection of the participants for the interviews, nor did the researchers influence the selection of which participants/teams that applied BAM in practice. Incorrect data (*internal validity*) [21] is a validity threat to all empirical studies. To minimize this threat, the researchers had the opportunity to validate, both during the interviews and after the interviews by contacting the participants about the answers and interpretations of the answers, which minimizes the risk of misunderstandings (i.e. collecting wrong (incorrect) data). *External validity* [21] is related to the ability to generalize the results from this study beyond the case company. Although Telenor Sweden is a large company developing software and services, it is not representative for all large software developing organizations. However, the aim of qualitative studies is not to generalize beyond the studied setting, instead, qualitative studies aim to explain and understand the phenomenon that is studied [20]. However, some of the issues introduced as motivation behind BAM (see Sect. 3.2) may be generalized for organizations that are using or introducing ASD methods. Moreover, from the concepts, practical application, and evaluation of BAM (as described in Sects. 4 and 5) may give an overview of the specific situation of Telenor Sweden, thus allowing other organizations to judge similarity and potential for BAM being relevant for them.

## **6 Conclusions**

The Backlog Assessment Method was developed and evaluated in close cooperation with Telenor Sweden. BAM is based on the identified needs in industry and the observations that existing agile practices and agile SPI methods only provides guidelines of how backlog items should be written, and guidelines of how to keep the backlog groomed. However, there are no methods or practices for assessing the current backlog and the backlog items. BAM addresses this issue, aiming to enrich the overall picture of the information in the backlog.

As part of BAM's development, evolvement, and refinement, the complete version of BAM was evaluated at Telenor Sweden with seven industry practitioners using a real backlog and actual backlog items from a product. The evaluation was performed to assure BAM's applicability in industry, and that the model is useful as part of agile SPI improvements for individual teams. During the evaluation at Telenor Sweden, the results show that BAM is feasible and relevant in an industrial environment, and the evaluation results indicate that BAM is useful as a tool to perform analysis as to items in a specific backlog, and the decision process associated with the backlog. The main benefit of using BAM is the use of concrete artifacts as the base for the assessment to give visual indications as to how the backlogs are used. The value is in getting results from an evaluation as discussion and decision support material to the practitioners working with the backlogs, thus BAM does not prescribe any correct level or processes.

The next phase that will be undertaken in the validation and evolvement of BAM, as described in this paper, is further evaluations in industry in different domains where the long-term effects, in terms of benefits and challenges of using BAM, need to be investigated to fully validate its feasibility and scalability.

**Acknowledgements.** The work and results presented in this paper were performed and produced in close cooperation between industry and academia. The authors would like to thank everyone involved in the development and evaluation of BAM at Telenor Sweden.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development**

Richard Berntsson Svensson(B) , Robert Feldt, and Richard Torkar

Department of Computer Science and Engineering, Chalmers|University of Gothenburg, Gothenburg, Sweden<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
**Abstract.** With the general trend towards data-driven decision making (DDDM), organizations are looking for ways to use DDDM to improve their decisions. However, few studies have looked into the practitioners view of DDDM, in particular for agile organizations. In this paper we investigated the experiences of using DDDM, and how data can improve decision making. An emailed questionnaire was sent out to 124 industry practitioners in agile software developing companies, of which 84 answered. The results show that few practitioners indicated a widespread use of DDDM in their current decision making practices. The practitioners were more positive to its future use for higher-level and more general decision making, fairly positive to its use for requirements elicitation and prioritization decisions, while being less positive to its future use at the team level. The practitioners do see a lot of potential for DDDM in an agile context; however, currently unfulfilled.

**Keywords:** Data-Driven Decision Making · Agile · Survey

## **1 Introduction**

When developing software-intensive products, agile methods have become the *de facto* way to develop software across almost every industry. The introduction of agile methodologies has changed the way software is developed [1], how Requirements Engineering (RE) is conducted [2], and how decisions are made [3]. In transitioning to Agile Software Development (ASD), learning about the customers, collecting customer/user feedback, and involving a customer representative in development, requirements engineering, and decision making, are important [4]. In addition, ASD teams, due to delivering working software in short iterations, are frequently involved in short-term decisions and need to adopt to a fast decision making process [5].

With digital networks connecting an increasing number of people, devices, and products, a vast amount of diverse data is available. Industries gather data and knowledge from their customers, suppliers, alliance partners, and competitors. For example, mobile phones, cars, transportation vehicles, and automation systems, are developed to generate data about their customers and usage of their activities. This diverse data is not only generated internally within softwareintensive companies, but also from public, proprietary, and purchased sources [6]. Software developing companies need to focus on exploiting the available data to gain competitive advantages [6], which will transform how business are generated, how RE is performed, and how decisions are made [7]. In particular, the recent resurgence of interest in artificial intelligence (AI) and machine learning (ML) accelerates these trends due to their promise of more automated and powerful data analysis.

However, despite the vast amount of data that is available for decision making, the decisions and selection of what to include in the next product release cycle, are commonly based on the product managements and/or stakeholders' previous experiences, opinions, intuitions, various criteria, arguments, or a combination of one or several of these information sources [4,7]. These decisions are typically subjective, frequently inconsistent, and often lack explanations as well as links to which data and evidence they were based on. Moreover, when stakeholders make decisions based on, e.g., opinions, intuitions, and arguments, the decisions are more likely to be influenced by politics and individual agendas [8–10] rather than, e.g., business opportunities or customer value. In addition, even when data is more clearly being taken into account in decisions, too much data and information may distract the decision maker rather then inform them. According to Wnuk et al. [12], irrelevant information is visible in practitioner backlogs to a large extent today, and recent research shows that it can negatively impact decisions [13].

In order to benefit from data-driven decision making (DDDM), not only is the quality of the processing techniques and tools directly related to the quality of the decisions [17], but also the quality of the visualizations used to support decision makers [17]. While visualization of software engineering data has shown promise in supporting practitioners' decisions, the focus has often been on specific phases or problems, e.g., testing and quality assurance [11], rather than throughout development processes and in agile settings. In the literature, most of the attention in DDDM has focused on the development of new techniques, technologies, and tools for data processing [14], while few (if any) have investigated DDDM from the practitioners' perspectives and the specific and important context of agile development has not been in focus.

This paper presents the results of an empirical study that includes data collected through an emailed questionnaire with 84 respondents from 28 agile software developing companies from 9 domains. The study investigate how common the use of data for decision making is in industry today, how often data is used, the respondents opinions about the usage of data in the future, and how data can improve decision making.

The remainder of this paper is organized as follows. In Sect. 2, we outline the background to data-driven decision making. Section 3 describes the research methodology, while Sect. 4 presents an overall statistical analysis of the data. Section 5 presents and discuss the results, and finally Sect. 6 presents the conclusions.

## **2 Background**

Data-driven decision making (DDDM) has become a critical ability for organizational success. Several studies have demonstrated the benefits of DDDM, e.g., Brynjolfsson et al. [16] showed that DDDM is strongly related to higher productivity, higher return on assets, return on equity, and market value.

In the literature, there are several defined steps in DDDM, starting with data capturing and resulting in decision making. For example, Chen and Zhang [14] identify five steps; data recording, data cleaning/integration/representation, data analysis, data visualization/interpretation, and decision making. Although steps are identified, most of the attention in the literature has focused on the development of new techniques, technologies, and tools. Techniques for DDDM involve a number of disciplines with a number of specific techniques and tools in each discipline. For example, fundamental mathematics, statistics, and optimization tools are used as input to data analysis techniques such as data mining, machine learning, neural networks, signal processing, and visualization methods [14]. Current DDDM tools can be divided into three categories: batch processing tools, stream processing tools, and interactive analysis tools [14]. For more details about different techniques, technologies, and tools, we refer to [14]. We also see an increased interest in applying AI and machine learning in a software engineering context [15] and supporting decisions during development is one of the key application types.

The quality of the decisions when using DDDM may improve or degrade based on the quality of the data and the processing techniques and tools [17]. However, the quality of the decisions are not only based on pre-processing techniques, processing techniques and tools, it is also related to the quality of the visualizations of the data to the decision makers, the decision makers' understanding and knowledge about the data sources, the decision makers' ability to interpret data processed data, and the decision makers' knowledge about the relationships of the data [17]. As one example, Feldt et al. [11] showed how visualisation of testing-related data, without any advanced modeling, could foster understanding and support decisions around software quality in an iterative development context. Thus, in order to benefit from DDDM, it is important to focus also on other aspects than just the pre-processing and processing techniques, technologies, and tools.

## **3 Research Method**

The objective of this study was to investigate how common the use of data for decision making is in industry today, how often data is used, and the respondents' opinions about the usage of data in the future, with a special focus on the agile context in which modern-day software is developed. Given the objective, and that the research questions are geared towards the opinions of the respondents, we chose to use a survey as the research method and emailed a questionnaire for data collection. Surveys are an appropriate strategy for getting empirical descriptions about trends, attitude and/or opinions of the studied population [18,19]. In addition, surveys are useful for analyzing large populations, given an adequate response rate [20,21]. The motivation for using an emailed questionnaire was to maximize coverage and participation. The following research questions provided the focus for the empirical investigation:

## **3.1 Survey Study**

The survey was executed through the creation of an emailed questionnaire that was designed based on the research questions using a mix of open-ended and closed questions [19]. In order to test the reliability and validity of the survey instrument, a pilot study was conducted with one industry practitioner. Based on the feedback from the pilot study, the survey instrument was (lightly) revised. The instrument (see Table 1) had three parts. The first part gathered demographic information about the respondents. The second part mainly addressed how, and how often data is used in decision making today, while the third part focused mainly on how data can be used for decision making in the future. Part 1 only contained free-text questions. All of the questions in Parts 2 and 3 contained Likert-type scale and free-text questions. The free-text area was added to allow the respondents to expand and/or explain their answer.

*Data Collection.* Subjects were sampled primarily through personal contacts and previous collaborators in industry and we encouraged them to also spread the survey within their organisations. Hence, the sample can be described as convenience sampling [19]. We provided the contacts with the questionnaire (emailed questionnaires) and information about the goals of the survey, and asked them to answer the questions and to spread the questionnaire to their colleagues. Each contact person reported back how many people they had forwarded the questionnaire to. A total of 124 subjects received the questionnaire, and 84 completed the mandatory questions and returned the questionnaire to the researchers. That is, we obtained a response rate of 67.7%. Without going through personal contacts in industry we likely would not have been able to get this high a response rate.

*Data Analysis.* The data was analyzed using descriptive statistics with diverging stacked bar charts for the graphical visualization. In addition, we built a linear

**Table 1.** Survey instrument

model (ordered logit) using a Bayesian approach [22,23] to statistically analyse the data. The analysis is described in more detail in Sect. 4.

#### **3.2 Validity Threats**

To avoid evaluation apprehension (*construct validity*) [24], we guaranteed the respondents complete anonymity. Another threat is 'hypothesis guessing' [24], which was minimized by clearly expressing the need for honesty in the instructions to the respondents; however, it is not possible to completely dismiss this threat. In addition, the background of the subjects, e.g., experience, may influence the results; however, since the respondents have different competences and roles we believe that this risk is limited. It is not possible to exclude the possibility that the respondents misunderstood the questions (*conclusion validity*) [24]. To minimize this threat, we conducted a pilot study with an industry practitioner, which also minimized the threat of instrumentation (*internal validity*) [24]. One threat that cannot be ignored is the interest of the respondents in the topic, which may influence the representativeness. This is difficult to counter since the willingness to participate and the interest in the topic may be linked. There are also threats to validity based on selection bias and the convenience sampling; even though we sent to most of our contacts in agile software organisations and approached them in a standardised way, the final sample might not be representative for a global population of developers. For example, they were all from organisations in Sweden.

## **4 Analysis**

To plot and assess visually the difference between distributions of responses in Likert scale data is hard. As an example, if we examine Fig. 1, we see that there is a difference between the distribution of answers on two questions (Q16, on top in the figure, and Q17, on bottom) but it is not clear how to judge how large the difference is. Also, if we only use descriptive statistics, which is the default analysis technique for survey data in software engineering, it is difficult to assess the uncertainty of our conclusions. In contrast, a Bayesian statistical analysis does not have the same problem. Thus, in line with recent arguments for use of Bayesian methods in empirical software engineering we thus, first, start with such an analysis [25,26].

**Fig. 1.** Plots of Q16 (top) and Q17 (bottom). Left: Histogram of discrete response in the sample. Middle: Cumulative proportion of each response. Right: Logarithm of cumulative odds of each response. Note that the log-cumulative-odds of Level 5 is infinity if there are responses among all five levels [1,..., 5], as in Q17 (for Q16 there were no responses on Level 5).

response

In order to assess differences in Likert scale data one could assume normality and use a t-test, or make use of some of the non-parametric tests such as Mann-Whitney U or χ<sup>2</sup>.

However, Likert scale data is not only categorical, it is also of an ordered nature but where we cannot assume that the 'distance' between consecutive pairs of answers is the same. Thus it is not clear that we can assume the data is normally distributed or that the distribution of scores for different answers has the same shape (distribution family) [27]. Given these problems, in our view, the most conservative approach to analyze Likert scale data is to build a simple linear model using a Bayesian approach but keeping data categorical [22,23]. This way we will get a posterior distribution with which we can assess uncertainty. To this end we build two overall models to study the general trends in our data:

$$\begin{aligned} R\_i &\sim \text{Ordered}(p\_i) \\ \text{logit}(p\_i) &= \beta\_T \ast \text{temporal}\_i \\ \beta\_T &\sim \mathcal{N}(0, 10) \end{aligned} \tag{1}$$

$$\begin{aligned} R\_i &\sim \text{Ordered}(p\_i) \\ \text{logit}(p\_i) &= \beta\_Q \* \text{req}\_i \\ \beta\_Q &\sim \mathcal{N}(0, 10) \end{aligned} \tag{2}$$

where R*<sup>i</sup>* is the ith response with an ordered categorical outcome, and Model 1 (Eq. 1) compares the answers for questions about the present (Questions 1–9, see Fig. 4) versus future (Questions 10–18, see Fig. 4) use while Model 2 (Eq. 2) compares the non-RE (Questions 13–16, see Fig. 5) versus the RE-specific (Questions 17–18, see Fig. 5) questions. We use the logit link function to translate the linear model's real numbers to probability mass (and hence constrain it to lie between zero and one). The linear model (in Eq. 1) then is simply a parameter β*<sup>T</sup>* that we will estimate given the data at hand (temporal). The data is coded as 0/1, representing 'present' (today) and 'future', respectively. Finally, we assign a prior to β*<sup>T</sup>* , N (0, 10), with a mean of 0 and a large variance of 10. This is a (very) weakly informative prior that only gives a pressure towards realistic parameter values. We also verified that the analysis was not sensitive to the prior selection (i.e., a sensitivity analysis was conducted).

For the other model (Eq. 2) we simply change the parameter. Instead of estimating β*<sup>T</sup>* using 'temporal' data, we estimate β*<sup>Q</sup>* for our variable 'question', which is coded 0/1, representing question with a 'non-RE' (Q13–16) and 'RE' focus (Q17–18), respectively.<sup>1</sup>

Figure 2 visualizes the results from running the first model and drawing 250 samples from the posterior distribution. It is obvious that low Likert scale values are much more common for the 'present' compared to the 'future' category. For example, we see that the number of answers of option 1 ('Strongly disagree') is roughly around 70% for questions about the present (today) state but decreases down to only 5% for the future state. We can also see that the uncertainty is not large with variations only in the range of 1–7% for all the answer alternatives.

<sup>1</sup> The models overall sampled well with mixed chains, <sup>R</sup><sup>ˆ</sup> - 1.1, and an effective sample size of neff 0.2.

When comparing non-RE and RE questions using Model 2 in Fig. 3, we can also see some trends even if they are less clear and the uncertainty is higher as visualized by the, relatively speaking, broader bands of posterior predictions. However, the model clearly shows that we see a difference between non-RE and RE related questions with the average of the β*Q*, being μ = −0.53 HPDI95%[−0.87, −0.19], i.e., the 95% highest posterior density estimate (HPDI) does not cross 0. This indicates that answers to the RE questions are generally lower (i.e. towards more disagreement with the statement in the question) than for the non-RE ones and that this difference is clear.

After this detailed, statistical analysis of the general trends in the responses the following Section will discuss the results in more detail.

**Fig. 2.** Posterior predictions (250 draws) of the ordered categorical model (present vs. future perspective). As is clearly evident, the probability for lower Likert scale values, e.g., 1 or 2, is much higher when the perspective is 'present', compared to 'future', i.e., everything is shifted upwards. This indicates less agreement at present and more agreement for the future, i.e. there is unfulfilled potential since the present state has a higher percentage of low disagreement answers.

## **5 Results and Discussion**

This section presents the results of the survey, organized according to the research questions in Sect. 3.

#### **5.1 Survey Respondent Demographics**

A total of 84 industry practitioners completed the questions of the survey. The respondents come from 28 agile software developing companies varying in size and domain. In total, the respondents came from nine different domains, with the top three being Telecommunication (27%), Consulting (18%), and Transportation (13%), see Table 2. The size of the companies where the respondents

**Fig. 3.** Posterior predictions (250 draws) of the ordered categorical model (non-RE vs. RE perspective). We see some trends, i.e., respondents are more positive in non-RE focused questions, but there is quite much uncertainty visualized here by the broader clusters of lines. However, the parameter β*Q*, which represents the domain focus, indicates that the trend is non-negligible (μ = −0.53 HPDI95%[−0.87, −0.19]).

work, in terms of number of employees, ranges from 25 up to 5,000. With respect to the respondents' roles, see Table 3, the top three are developers (17%), scrum masters (15%), and product owners (14%) with a fairly even distribution of other, common roles also represented. For the development processes used at the companies see Table 4 where Scrum (43%) is the most used, followed by (the general option) Agile (29%), Kanban (15%), and then DevOps (12%). Note that the Agile category means that a respondent did not specify which agile methodology they used. Overall, we consider these respondents representative for a broad set of domains, roles and sizes of companies, even if they are all active in a Swedish context. The one role that is less clearly represented is Requirements Engineer although several of the respondents also partly do work with requirements in one form or another, as is common in agile development.

#### **5.2 View of Data in Decision Making (RQ1)**

In analyzing Research Question 1 (RQ1), this section examines the respondents' view of data as part of decision making in ASD companies. In Fig. 4, we can see the respondents' answers to each question. Each row shows the distribution of answers for that question aligned horizontally so that positive responses are to the right of the mid (zero) line while negative responses are to the left.<sup>2</sup> This makes it possible to compare the answers between different questions.

In general, looking at Fig. 4, we can see that it follows the general trend identified in the statistical analysis above, i.e., respondents disagreed with the

<sup>2</sup> Note that the neutral, mid answer option (on the 5-category Likert scale) is split in half, with half of them shown in a lighter (gray) color to the left and the other half in darker (gray) color to the right of the mid (zero) line.

**Table 2.** Distribution of respondents based on domains

**Table 4.** Distribution of respondents based on development process

**Fig. 4.** Respondents' view of data as part of decision making. Present (Q1–Q3) and future (Q10–Q12).

statements more in questions about the current state while agreeing more in questions about the future. For example, we see that a majority of the respondents disagreed or strongly disagreed that data is important (66% for Q1) and highly valued (79% for Q2) in today's decision making. However, a majority of the respondents agreed or strongly agreed that data should play an important role (71% for Q10) and be highly valued (87% for Q11), when making decisions in the future. Examining if data is treated as an asset today (Q3), 93% of the respondents disagreed or strongly disagreed, while 63% of the respondents agreed or strongly agreed that data should be treated as an asset in the future (Q12). Although the respondents have a positive view of how data should ideally be viewed for decision making, their answers indicate this is not how it is being viewed at present in their organisations.

#### **5.3 Use of Data in Decision Making (RQ2)**

In analyzing Research Question 2 (RQ2), this section examines to what extent data is used (present) and should be used (future) in decision making and requirements engineering in ASD companies, as illustrated in Fig. 5. Figure 5 is constructed in the same way as Fig. 4, with the exception that the zero line, i.e., the neutral answer, is set to the answer 'About half of the time'. In general, Fig. 5 shows that data is seldom (never or sometimes) used in today's decision making or in Requirements Engineering (RE) (Q4–Q9 in Fig. 5). However, a vast majority of the respondents believe that data should be used most of the time or always in future decision making and RE (Q13–Q18 in Fig. 5).

**Fig. 5.** Use of data as part of decision making. Present (Q4–Q9) and future (Q13–Q18)

Looking closely into what extent data is used in today's decision making, for all questions (Q4–Q9), more than 90% of the respondents stated that they never or only sometimes use data in decision making and RE, where more than 73% of the respondents stated that they never use data today. No respondent stated that they always use data. Only 1% of the respondents stated that they use data most of the times for requirements elicitation/identification (Q8) and requirements prioritization (Q9). Instead of using data, the respondents explained in the freetext answer that decisions are mainly based on 'gut-feeling', the decision-makers' experiences, or the value for customers.

That is, the decisions may be subjective [7], politically influenced [8], and/or biases could be involved [13]. Instead of using data when prioritizing requirements, respondents detailed that requirements are prioritized using various criteria (e.g., cost, cost/benefit, customer value, business value), numerical assignment, experiences, 'gut-feeling', or a combination of these. This is inline with other studies on how requirements are prioritized in ASD companies today [28].

When asking the respondents to what extent data should be used in decision making in the future, 93% of the respondents believe that decision makers should always, or most of the time use data for decision making (Q15), 85% believe that data should always, or most of the time be used to identify new business opportunities (Q13), and almost 75% believe that data should always, or most of the time be used to predict future trends and behaviours (Q14). Only 8% of the respondents believe that (agile) teams should always, or most of the time use data for decision making (Q16), while almost half of the respondents (43%) believe the (agile) teams should never, or only sometimes use data when making decisions. No explanation was provided by the respondents in the free-text answers for these questions.

One possible explanation may be that the respondents believe that DDDM is only useful and beneficial for high-level decisions. This is supported by the high confidence in using DDDM for identifying business opportunities (Q13) and to predict future trends and behaviours (Q14). When such high-level decisions are made, including creating product strategies, road-maps, and release plans, the respondents may believe that teams do not need DDDM when, e.g., breaking down high-level requirements to low-level ones. Another explanation may be related to today's development processes and short sprints, which may not be well suited for DDDM at the team level.

To create and rapidly release software-intensive products in the future, it is crucial that the products are based on data and real-time feedback from the customers [7]. Thus, when moving from a subjective decision-making process, mainly based on experiences, to a DDDM process, changes in infrastructure and methodologies are needed in the development processes [7].

For RE, 60% of the respondents believe data should always, or most of the times be used when eliciting/identifying requirements in the future (Q17), while 15% believe data should never, or only sometimes be used for requirements elicitation/identification. Only 35% of the respondents believe data should always, or most of the time be used when prioritizing requirements, 25% believe it should never, or only sometimes be used, while as many as 40% answered that data should be used about half of the times when prioritizing requirements (Q18).

When we analyzed the data by building a simple linear model (Eq. 1) using a Bayesian approach, the results show a difference between today ('present' in Fig. 2) and the future. In Fig. 2, we see that the lower Likert scale values (e.g., answers 'never' and 'sometimes') are more common for Present, while the higher Likert scale values (e.g., answers 'always' and 'most of the time') are more common for the Future. That is, the respondents, with a high certainty, are positive to use DDDM in the future. When comparing RE related questions (Q17 and Q18) with non-RE related questions (Q13–Q16), the Bayesian model (Eq. 2) shows a difference, as shown in Fig. 3. That is, although the respondents are positive to use DDDM in the future in general (as shown in Fig. 2), the respondents are more positive to use DDDM in non-RE related decisions compared to RE-related decisions.

**Reasons for Using (not Using) Data.** We asked the respondents what the reasons for using data in today's decision making is. According to the respondents, the main reason is that DDDM improves the decisions. One respondent explained that when data has been used as input to decision makers, the decisions have been more informed and more transparent. Another reason mentioned by the respondents was, if data is available, then we use it.

A few respondents also gave reasons for partial data use: although the data is there and can improve decisions, it requires a lot of work to filter the data and to present the data in a way that is useful for the decision makers; thus it is only used sometimes for critical/important products/strategies.

Looking at Table 5, we see that *data is not available to us at the company* is the most common reason (82% of the respondents). Most of the respondents who stated that data is not available, also mentioned several other reasons for not using DDDM, including *too much data is available out there* (79% of the respondents), *do not know how to use the data* (73% of the respondents), and *do not know how to make the data relevant to us* (70% of the respondents). Several of the most mentioned reasons for not using DDDM are related to the decision makers' understanding of the data (including the visualization), and how to make use of it. This confirms the findings in [17]. In order to fully benefit from DDDM, the quality of the data is important as it is directly related to the quality of the decisions [17]. Therefore, it is surprising that only 6% of the respondents mentioned that data is not used in today's decision making due to the quality of the data. Either, decision making in agile is different or respondents are less aware of these important considerations.

#### **5.4 How Can Data Improve Decision Making (RQ3)**

We asked the respondents if they believe data could help them in making better decisions (Q19 in Table 6). Eleven percent of the respondents believe data will improve their decisions (answered 'yes'), while a majority (58%) believe that data, in combination with other aspects (described below), will lead to better decisions. Close to a third (29%) of the respondents believe data may help in making better decisions but they weren't sure (i.e., they answered 'maybe'). Their stated reasons were: (1) have not used data hence do not know if it will lead to better decisions, (2) it depends on which data, the quality of the data, and who makes decisions, (3) and what kind of decisions and when the decisions are made. Only 2% of the respondents do not believe data will help in making better decision. One respondent explained this by stating *"data can never replace my own experiences and gut-feeling"*.

**Table 5.** Reasons for not using data in decision making

**Table 6.** Respondents' views if data improves decision making

The respondents identified five aspects that needs to be combined with DDDM in order to make better decisions. The five aspects are: (1) own experience, (2) business value, (3) customer value, (4) input from key stakeholders, and (5) experiences from others.

In order to be able to use the full potential of DDDM and thus truly change how decisions are made in ASD, new approaches to provide and visualise constructive and understandable data (information) to the decision makers are needed. By combining understandable visualizations of data and human expertise, the future of DDDM in ASD looks promising.

## **6 Conclusions**

There is a general trend towards data-driven decision making (DDDM), i.e., basing and driving decision making on and with data. However, there has been a lack of studies on how software practitioners view and use this and, in particular, in an agile context. In this study we thus performed a survey and collected questionnaire responses from 84 software practitioners working with agile software development.

Our main result is that the practitioners see a lot of potential for DDDM but that this potential is currently unfulfilled. While very few respondents indicated more wide-spread data-driven decision making in their current practice, a clear majority saw it as important and highly valued in the future. They were more positive to its future use for higher-level and more general decision making, fairly positive to its use for requirements elicitation and prioritization decisions, while being less positive to its future use at the team level. Multiple reasons were given for data not being used today, in particular it may not be available, be available in too large quantities, or it may not be clear how to use it, make it relevant and link it to decisions. Notably, respondents seemed less concerned about quality and trust issues around data.

Our results show that there is an unfulfilled potential for data-driven decision making in agile software development contexts. Future research should investigate this in more detail and also develop new automated data collection, analysis and visualisations techniques and methodologies that augments existing, agile decision processes by linking relevant data to specific decision contexts.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Empowering Agile Project Members with Accessibility Testing Tools: A Case Study

Viktoria Stray1(&) , Aleksander Bai<sup>2</sup> , Nikolai Sverdrup<sup>1</sup> , and Heidi Mork<sup>3</sup>

<sup>1</sup> Department of Informatics, University of Oslo, Oslo, Norway<EMAIL_ADDRESS><sup>2</sup> Norwegian Computing Center, Oslo, Norway<EMAIL_ADDRESS><sup>3</sup> NRK, Oslo, Norway<EMAIL_ADDRESS>
Abstract. There is a growing interest in making software more accessible for everyone, which is emphasized by the numerous suggestions passed into law in many countries. However, many software organizations that use agile methods postpone or neglect accessibility testing. We aimed to understand how accessibility testing can be better integrated into the daily routine of agile projects by conducting a case study in a Norwegian software company. We investigated three accessibility testing tools: automatic checker, simulation glasses, and a dyslexia simulator. We hosted sessions at which agile project members used the tools while thinking out loud, responded to questionnaires, and were interviewed at the end. Additionally, we observed the project members for 18 workdays. Our results show that all three tools are suitable for agile projects. Especially the automatic checker and simulation glasses worked well in finding accessibility issues and were described as easy to use by the project members. Software organizations should empower their agile project members with low-cost and efficient accessibility testing tools to make their products more accessible for all. Doing this early and often in the development cycle may save the project from potential high costs at a later stage.

Keywords: Accessibility testing Usability Cambridge simulation glasses SiteImprove accessibility checker WCAG Agile software development Universal design

## 1 Introduction

Creating software that is accessible for everyone (including users with impairments) is an important consideration for an increasingly digital society. Accessibility focuses on enabling people with the widest range of capabilities to use a product or removing barriers that can interfere with the user experience [1]. However, putting accessibility into practice remains a challenge in software development [2], and agile methods are especially under scrutiny for not offering enough consideration for accessibility [3, 4].

Testing for accessibility means testing how users with impairments and disabilities (e.g., dyslexia or impaired vision) will experience the software product. Although agile methods highlight principles such as delivering working software and regular testing, incorporating accessibility testing throughout an agile process is complicated and less common [3, 4], and it can be an expensive endeavor [5]. Accessibility testing is often postponed to the late phases of software development which breaks with the principle of delivering working software frequently [6]. Researchers argue for easier methods to make finding accessibility flaws more effective in agile software projects [6, 9, 10]. Agile projects face a growing complexity with a variety of organizational constraints such as universal design, legislation, and security, forcing the cross-functional agile team to adjust their practices and experiment with new team structures (e.g., BizDev teams) [7, 8]. Thus, now might be a good time to include practices and tools that let agile teams test for accessibility issues throughout the project.

In an agile process, which features short iterative cycles, it would be difficult to fit costly and time-consuming accessibility testing into regular testing procedures. Accordingly, there is a need to find methods that integrate well with agile approaches. Methods and tools that can be utilized often without being too resource- and timeintensive will fit well with agile principles. Our motivation for this project was thus to improve accessibility in agile software development by testing different methods that are considered fast and efficient.

The benefits of having an accessible solution are apparent. Poorly developed software with regard to accessibility is aggravating to impaired users and can make the product miss out on potential users, giving the issue both social justice and economic motivation. Research has also shown that software that accommodates accessibility and inclusion will benefit all users, including those without any impairments [11]. Also, for software to be considered working, it needs to be thoroughly tested and ready to be delivered to customers or production. If software fails to be sufficiently accessible, it cannot be considered to be working. For the goal of having working software, accessibility testing needs to be a part of every iteration that involves features which can affect the accessibility of the software. Motivated by this, we set out to investigate how accessibility testing can be integrated into agile software development, by answering the following research question:

#### RQ: How do agile project members experience using accessibility testing tools?

To address this question, we conducted a case study [12] where we had the participants test three different accessibility tools and respond to a questionnaire about the tools they tested. The participants were also interviewed and observed.

The remainder of this paper is organized as follows. Section 2 outlines the relevant background on accessibility testing in agile software development. Section 3 discusses different testing tools. Section 4 describes the data collection. Section 5 presents the results, and Sect. 6 discusses the results and implications for practice. Section 7 concludes and suggests future work.

## 2 Background

Failing to make a product accessible can be costly, as it will increase the probability of having to make late and expensive changes in the development process and further lead to prolonged development time. A product with which the user cannot achieve their goal through the usual procedure will also lead to a reduction in users and increase the need for additional support and assistance to help guide the users, such as help-desk services and on-site support [13]. Those who do include accessibility in their development process will find that the cost for testing procedures can take its fair share out of the development budget, especially since the requirements for accessibility testing often are complex [14].

Investing in accessibility testing early and throughout the development process can yield a substantial return on investment [15]. Detecting mistakes early lowers the cost of fixing the error and avoids the accumulation of big, time-consuming problems at the end of the development process. Accessibility testing, however, has the unfortunate fate of often being conducted at the very end of development [2]. Reduction in the amount of work needing to be done in the maintenance phase can also be a substantial benefit, as a significant portion of the life-cycle expenditure of a software product can congregate in this later stage.

#### 2.1 Accessibility Testing in Agile Software Development

Dissatisfaction surrounding traditional accessibility test methods is a prevalent theme in other research. Researchers have investigated usability issues in agile development (e.g., [16–18]) but few researchers have investigated accessibility testing in agile projects. As Luján-Mora and Masri [9] argue, accessibility is difficult to achieve with traditional software development practices, due mainly to its habit of being implemented too late in the development process. They propose that agile development can significantly help to improve web accessibility due to its focus on regular testing.

Similarly, Eklund and Levingston [6] discuss how usability can benefit from agile development. They propose a set of steps one can use to incorporate usability techniques in agile development, such as conducting smaller tests and reviews to cover more of the development process as well as to rely extensively on external consultants with expert knowledge on accessibility testing. Meszaros and Aston [19] describe the introduction of early accessibility testing in an agile development process using testing methods such as paper prototypes to locate accessibility bugs, wizard of Oz testing, and usability stories. This resulted in a higher acceptance by end users. Kane [3] highlights the lack of accessibility testing in agile development and proposes techniques to integrate more accessibility testing into already established agile practices.

## 3 Testing Tools

Testing for accessibility entails testing for a multitude of different impairments and disabilities in various states. When ensuring a product works for users with eyesightrelated issues, for example, one has to consider aspects such as different levels of visual perception, color blindness, complete blindness, tunnel vision, and so on. All of these affect the user experience in different ways and create different challenges that must be solved. Without any tools, a developer would have to retain an extensive amount of knowledge to meet the demands of accessibility testing, which would be hard to put into practice. One solution is to hire expert help from accessibility consultants or recruit people with different impairments to conduct user testing. However, this is an expensive solution and should be done when the product moves into a more stable phase.

One of the most significant challenges in testing for accessibility is that one has to account for a wide range of people. Using one super-tool that can take account of everything is not realistic due to the diversity of impairments and the different ways they affect users. When testing with accessibility in mind, there is a need to evaluate everything from content, layout, comprehensibility and technical implementation to compatibility with other accessibility tools, such as a screen reader.

There are some tools that we considered but decided not to include. For example, we considered a screen reader and WCAG (Web Content Accessibility Guidelines). However, both tools are intensive regarding the knowledge required to operate them, and this does not fit very well into an agile process with short iterative cycles. A screen reader is a software tool intended mainly for use by people suffering from mildly impaired vision to complete blindness. The biggest challenge of using a screen reader is that it required extensive training in order to operate the tool correctly. It may take years to achieve a proficient use of a screen reader, and advanced users can navigate around an obstacle that novice users cannot. WCAG is the de facto practiced method used to check for accessibility. WCAG is presented as a set of specifications categorized into four broad principles containing 61 numbered paragraphs of success criteria. Each criterion has its own detailed page, specifying the intent, examples, techniques, key terms, and related resources associated with the paragraph. The techniques and success criteria will, altogether, give a total of 379 different pages, each with a description, examples, additional resources, test procedures, and expected test results. From our earlier research, we know that developers perceive WCAG as tedious, cumbersome, and time-consuming to use in agile projects [14].

We chose the testing tools SiteImprove, Cambridge simulation glasses, and a dyslexia simulator (where we developed a Chrome browser extension), briefly presented below, based on other studies that have categorized testing tools according to barrier groups and cost-benefit [14, 20]. Since we wanted testing tools that could be easily integrated into an agile process (low cost and little prior knowledge), we found these to be the best candidates.

#### 3.1 SiteImprove Accessibility Checker

SiteImprove is an automatic checker and browser extension that can analyze a webpage for breaches of many WCAG criteria. When activated, the extension will automatically analyze the currently opened web-page for noncompliance with the different levels of the WCAG technical standard. It distinguishes between the different categories of the WCAG standard and organizes them into groups. Each instance of error can give a direct link to the WCAG manual for a more detailed explanation of why the error exists and includes suggestions about how one can achieve compliance with WCAG. Other notable features are the ability to highlight where the error exists on the site itself and in the public source code by highlighting in the browser's developer tools.

One of the most significant drawbacks with automatic checker tools is that there are many things they cannot check, at least not until we have better artificial intelligence. For instance, an automatic checker can make sure that images have alternative HTML tags, but it cannot check if the description is accurate and meaningful.

## 3.2 Cambridge Simulation Glasses

A person can identify accessibility faults by wearing blurred glasses while interacting with the interface or object that is under examination. For our testing, we used the Cambridge simulation glasses. The glasses are thin enough to stack several pieces of glasses in sequential layers, enabling the tester to be in control of the degree of reduced vision. Before using the glasses, the tester performs a quick eyesight evaluation using a Snellen chart that comes with the glasses or can be printed out. The chart indicates how many glasses the tester should wear to simulate reduced vision (a 95% coverage of the general population). Most people will need two pair of glasses, but those with very good eyesight might require three glasses. While wearing the glasses, the tester interacts with whatever is to be evaluated for accessibility, and any possible shortcomings should be made readily apparent for the tester as they might struggle to use the solution.

## 3.3 Dyslexia Simulation

The dyslexia simulator is a small and simple browser extension that tries to simulate how a dyslectic user or someone with some other reading disorder might experience a website. Dyslexia is a common learning disability, and the condition can severely hamper a user's ability to comprehend a website. Dyslexia impedes the ability to read, and how software is presented can significantly affect the difficulty of interaction for a dyslectic user, especially in sites that feature much text or require the user to write [21]. The tool will test a website by shuffling the letters in words, making it difficult to read.

When testing a solution, the tester operates the interface as normal with the dyslexia software running; if there are areas that become difficult to perceive, the tester can use that as an indication for something that might become difficult for someone with reading difficulties to perceive. The dyslexia simulation tries to highlight issues related to a neurological impairment, which is challenging to test for. It is difficult to make software that can detect areas affected by such disabilities, and it is difficult for a tester to understand what the impaired user might experience. There are several WCAG paragraphs detailing the issues related to reading impairments and steps to help in the matter.

We used a dyslexia simulator in a pilot study that was a script the user had to rerun for each use, which was cumbersome. Therefore, for this study, we made an extension for the Google Chrome web browser (now available on the Chrome web store) that is easier to install and use.

## 4 Data Collection

We conducted this study in a Norwegian software company that makes digital services and solutions for banks. We observed the project members in their daily routines and during agile practices such as daily stand-up meetings [22] and retrospective meetings [23]. We documented their behavior, and conversation topics and were especially looking for matters where accessibility issues could play a part. We also interviewed four members with different roles and responsibilities. The reviews of the three testing methods were conducted by hosting sessions in which the participants tried out accessibility testing tools, answered questionnaires about the tools, and were interviewed at the end of the session, see Table 1 for an overview of the data collection.

Interview questions and testing tasks were made after determining what information was needed and how much time we could have with the participants. As recommended by [24], every tester had filled out a background survey before the session. After a pilot attempt of the testing session, it was decided to use a publicly available website not affiliated with the participants, after we found the participants' knowledge of their products to influence their ability to judge the accessibility.

All the testing sessions were originally scheduled for one hour each, where we had time to test three methods, which involved a quick briefing of the participants about the tools and instructions on how to use them. In the testing sessions, the participants were instructed to solve pre-made tasks that required them to use the tools to locate accessibility faults in a website. As the participants used the tools, they would comment on their experience and be prompted with questions by the interviewers to reflect on the experience. After completing the tasks, they filled out a questionnaire.

Having finished testing, the participants were asked a series of questions regarding their thoughts on each tried method, comparing the methods and identifying which one they preferred and which one they disliked. They were further asked about the ramifications of accessibility testing, and how they thought accessibility testing could be integrated into their routines.

Table 1. Data collection

## 4.1 The Accessibility Testing Tools

We tested three different accessibility testing tools. Five participants tested three tools while two participants tested two tools because of less time available. The participants were encouraged to think aloud. When they forgot to talk while they were using the tools, we reminded them to do so. Below, we will describe how we tested each of the tools.

SiteImprove. Before we began, the participants were instructed to install the extension on their computers, and they were also given a quick overview of what it does at the start of the session. The SiteImprove user test was conducted without any specific tasks or scenarios in mind, as the tool does not need any extraordinary web features to be able to highlight its capabilities. The participants would explore the tool and investigate the different layers of information on a given website. When they had questions about the tool or were stuck, we guided them in the right direction. After exploring the tool on the given website, five of the participants switched to sites they were working on or

were more familiar with to get a better sense of what the tool was telling them.

Simulation Glasses. After a brief introduction and determining how many glasses each tester should wear, the participants who tested the simulation glasses where given tasks to complete while wearing the glasses (Fig. 1). The tasks involved interacting with a web interface and completing two scenarios. The scenarios involved finding and booking airplane tickets online from one of the leading airlines in Scandinavia.

Fig. 1. A participant testing with simulation glasses

Dyslexia Simulation. Before we began, the participants were instructed to install the extension. We described how the tool worked and how it could help them discover accessibility vulnerabilities. Participants were asked to navigate a website with the dyslexia extension turned on and complete two scenarios. The scenarios involved finding information about prices, luggage, and legal requirements. The chosen web pages contained a lot of text to afford the extension something to work with.

#### 4.2 USE Questionnaire for Tool Evaluation

Every participant filled out a questionnaire for each of the methods they tested. The questionnaire for evaluation of the testing methods is a post-session rating metric named the "Usefulness, Satisfaction, and Ease of use (USE) Questionnaire" [25]. The USE Questionnaire is a validated questionnaire that consists of 30 questions divided into four categories. We considered this questionnaire to be helpful in assessing whether each of the methods was suitable for agile projects, by stating general questions about satisfaction with usability and ease of use on a seven-point Likert scale.

#### 4.3 End Interview After Test

At the end of each testing session, we asked the participants a set of prepared questions to gauge what they thought about the testing tools. They were asked to compare the tools in different ways and to talk about how they experienced using each of the tools. We also asked them whether they believed they would use the tools in the next three weeks, which tool they felt discovered most faults, and how the tools could be integrated into the daily agile workflow routines. Also, we had some general questions about how accessibility aspects affected the products they were developing.

## 5 Results

From the interviews, it became evident that many project members did not know much about accessibility testing before they tried the tools. One developer suggested that a reason was that they did not learn it in their software engineering education, and he also claimed it was not a common theme in work discussions. From our observations, we could confirm that there was little talking about accessibility among the project members. In none of the 20 observed meetings was there any discussion of accessibility issues. However, all interviewees were positive to learn about tools to help them test for accessibility throughout the agile project. Even though many had little experience with the tools, all participants managed to install the tools in a short amount of time.

#### 5.1 SiteImprove Accessibility Checker

Seven participants evaluated the SiteImprove tool. As Fig. 2 shows, the method did very well, with a high score for all categories. The percentage reflects how well a category scored on the Likert scale from one to seven, where seven is 100% and one is 0%. The tool received a total score of 4.98 (r = 0.40) with a 95% confidence interval between 4.68 and 5.28.

SiteImprove scored very high in the Usefulness category with a score of 5.43 (r = 0.42), which is well above the average of 4.0. Ease of Learning received a score of 4.82 (r = 0.41) since most of the operations are provided automatically by starting the extension.

The method also scored high in Ease of Use with a score of 4.87 (r = 0.42) even if some had minor complaints and somewhat struggled: "The tool points out that there are faults, but it is hard to spot where the faults are". However, many stated that SiteImprove was much better to use than WCAG. One developer said, "This is close to a developer tool (…). I like that it makes references to WCAG. I have tried to look at the WCAG documentation, but found that difficult".

Fig. 2. SiteImprove evaluation results

#### 5.2 Cambridge Simulation Glasses

Seven participants also evaluated this method. As Fig. 3 shows, the glasses got high scores in all categories and had the highest total score of 6.15 (r = 0.48) among all the methods. The 95% confidence interval for the method was between 5.79 and 6.51.

Not surprisingly, since this method requires the participant only to wear glasses, this method scored very well on both Ease of learning with a score of 6.79 (r = 0.12)

Fig. 3. Cambridge simulation glasses evaluation results

and Ease of use with a score of 6.45 (r = 0.49), One developer said, "The glasses gave quick results and were easy to use. I also like that they are tangible", while another developer said, "I want to have these glasses on my desk and use them often".

Also for the Usefulness category, the method scored high with 5.88 (r = 0.48), and the sub-question "Is it useful" scored 6.57 (r = 0.73), which is well above the average for the category. A designer said, "I will use these glasses in meetings between developers and business developers to make them aware of how the different solutions they are discussing will be perceived by people with reduced sight".

Regarding Satisfaction, the method scored the highest of all with 5.47 (r = 0.84), and this was also reflected in the interviews. Many participants mentioned that the method was fun to use, and many also tried the glasses on their own after the session was completed. During the interviews, many participants mentioned that the method created more awareness of the challenges associated with bad contrast and small fonts.

#### 5.3 Dyslexia Simulation

Five participants evaluated the dyslexia simulation tool, and as Fig. 4 shows, this method was also regarded highly. The tool had the second highest evaluation in the Ease of learning category with a score of 6.35 (r = 0.3). This is not unexpected since the method requires only the push of a button in the browser to be enabled. Both Usefulness and Ease of use also had high scores with 5.33 (r = 0.57) and 5.62 (r = 0.67). A designer said, "The dyslexia simulator makes me realize how important it is to use common and simple words because those are easier to recognize when they are scrambled. I will definitely use this tool to show the management how it will be perceived by people with dyslexia if we use long text with difficult words".

Fig. 4. Dyslexia simulation evaluation results

Overall the Dyslexia simulation method did good, in particular in the category Ease of learning, as Fig. 4 shows. The method itself scored well above neutral (4) and might have received even better scores if not for some bugs in the extension. This is also reflected in the sub-question for Ease of use where the question "I don't notice any inconsistencies as I use it" scored 4.00 (r = 0.89), which is well below all the other sub-questions. This is also probably connected to the fact that the extension did not adapt all the text on the webpage.

The tool scored well overall with a total score of 5.53 (r = 0.53) and a 95% confidence interval between 5.14 and 5.92. In the interviews, many participants said that the method was an eye-opener experience. Participants also liked that it was easy to visualize problems with too much text on a web page and that it emphasizes the importance of writing good and readable text: One developer said, "I like this plugin. I now see that the way you structure the text is very important. When you have too much text close together, it is hopeless for a person with dyslexia". Contrarily, some noted that it could be difficult to understand how good the method was at finding issues.

## 6 Discussion

Good accessibility is difficult to achieve, mainly due to its complex nature and the wide range of people who must be considered. It does not help that accessibility testing is conducted late in the development process when changes are costly to make, and compromises are made instead. However, agile development can significantly help to improve web accessibility due to its focus on regular testing [9]. Eklund and Levingston [6] and Ferreira et al. [10] argue for more frequent testing where more costeffective methods are used several times throughout the project rather than relying on few but large and costly testing activities, such as hiring accessibility experts to do accessibility testing.

There is currently little research that offers concrete solutions for simple accessibility testing in agile settings; most current research discusses just the possibilities for accessibility testing in agile development. In this study, we have focused on giving accessibility testing some much-needed evaluation in an agile project. The fast and low-cost test methods in this study were perceived as useful and easy to use by the agile project members.

Next, as other research also has suggested [9, 26], we found that accessibility needs to be a team effort. Agile principles promote self-organization and communication within the development team and make it possible to inspect even the smallest of details when everyone is involved, instead of having only a few experts working on accessibility. Several of the methods we tested addressed many of these concerns. While some were easier to use and learn than others, they all were usable without extensive training and learning sessions. Our results showed that all three methods we tested, SiteImprove, the Cambridge glasses, and the dyslexia simulator, can be used across the development team and utilized in such a way that it becomes a team effort where testing can be commenced at regular but short intervals.

All three accessibility testing methods we used in this study were low-cost and perceived as very easy and quick to use. Most of them can be used very early in development, some even before any code is written, merely going by design drawings. There is no doubt that accessibility testing can be included early in agile software projects. Using any of the tools will enable testing to be done more often. The interviewees also suggested regular workshops at which the agile team members together could experience the solutions they made with different accessibility testing tools. Also, if there are formal requirements from the organization or customer, these issues could be discussed in planning and retrospective meetings and would probably be a more frequent theme in the daily-standup meetings.

SiteImprove was quick in producing results and is the method that would cover the broadest set of WCAG paragraphs in our study. One key factor in its success is the automation aspect of the tool. It dramatically reduces the time spent finding bugs, and the user is immediately presented with clearly defined statements of what is wrong and how one can improve. It was not surprising that the participants rated SiteImprove as easy to learn since most of the operations are provided automatically by starting the extension. We had expected a lower total score of SiteImprove since the tool uses many complicated terms and advanced terminology. A drawback with SiteImprove is that it cannot be used until a solution has been implemented, rendering it not as useful for designers or in the early phases. It is also limited by being able only to check what is programmatically possible.

The Cambridge simulation glasses were, along with SiteImprove, the method that worked well in many situations. It also worked well in cases where SiteImprove cannot help. Use of the glasses is not dependent on a solution having reached a certain level of development. The glasses can be used early in development on visual concepts, for example on sketches or prototypes. They are also platform-independent and can be used on any software or visual representation, making it a universal accessibility tool. One aspect remarked on by many of the participants who tested the method was that the glasses gave an overview of the product in its entirety. Being able to view the product from a higher perspective rather than fixating on small details also contributes to why so many liked to use the glasses.

We included the dyslexia tool because we wanted methods that covered cognitive impairment, and this method was one of the few testing methods that are low-cost and fast to test for this. The method was rated higher than we had foreseen, particularly because we expected more participants to misunderstand the idea behind the method. The tool received good results on the USE Questionnaire. However, with this tool, the testers have to interpret the results based on their knowledge of dyslexia. Some testers reported that the plugin could be helpful to discover text where long and complicated words were being used, and where the structure and layout were not organized. The improved accessibility by employing clear and concise language is an aspect not addressed by any of the other tools.

#### 6.1 Threats to Validity

As with all empirical studies, there are some threats to validity that we need to consider. First, we used a single-case design and, as a result, have the possibility of bias in the data collection and analysis. Therefore, the general criticisms of single-case studies, such as uniqueness, may apply to our study. However, as the company had been using agile methods for many years, developing bank solutions for a wide variety of people, we found it to be a valuable case for investigating accessibility testing in agile software development. Furthermore, we triangulated the research question from multiple sources of evidence such as interviews, observations, and questionnaires to increase the quality of the case study [12].

Second, when using self-reported data to measure usability of tools, one might have the "social desirability bias" [27] where respondents respond more positively to rating questions because they unconsciously want to satisfy the facilitator giving them the questionnaire. To reduce this bias, following the advice of Albert and Tullis [25], we collected the responses anonymously on the web, and we did not look at the results until afterward.

#### 6.2 Implications for Practice

Our results show that agile accessibility testing methods are not well-known among agile team members. Accordingly, agile projects should invest in simple tools that can be used by all roles, and project members should be informed about the techniques they can use. We found that the three tools tested were useful and should be used regularly in agile projects as they are also cost-efficient. For software developers who consider using accessibility testing tools, we suggest that Cambridge glasses be used early, as they do not require much effort from the agile developers. As the product becomes more stable, SiteImprove can be used regularly.

The methods and tools we have evaluated have been used with the intention of testing software and finding accessibility issues. In the larger scheme of accessible software, we suspect that using these tools, especially the ones that simulate an impairment, might have secondary benefits. Exposure to the type of simulation testing such as with the Cambridge glasses and the dyslexia simulator can give the tester a heightened sense of awareness of issues that could come into conflict with impaired users and foster more empathy and understanding.

Many comments were made that the testers had new insights and understanding. The new knowledge about how different users perceive software can give developers a subconscious ability to shape software in a more accessible direction, outside of directly applying accessibility methods. So, while the methods described in this paper might not be chosen to be used regularly, there can be a significantly insightful experience for agile developers to try these methods, and they can put that experience to use in later development.

There is still a way to go to more easily integrate accessibility testing into agile work processes. The existing tools should be more compatible with agile processes, and agile project members should have a wider variety of accessibility testing tools to aid them in making accessible software. Implementing this would reduce the resources needed for accessibility testing by continuing the work toward more lightweight and comprehensible tools, as well as making the surrounding issues with impairments more visible and the knowledge of how they affect software interaction more obtainable. This will aid in weaving accessibility into the development process.

Another barrier is that many testing tools are platform-dependent. Tools are not uniformly available on all operating systems, browsers, and developer tools. The different tools have their own characteristics, and integration and learning can be made more arduous because of this. Tools that support software not intended for web or mobile are also rare. The lack of diverse tools available on many platforms hampers any attempt to make accessibility a more uniform process as it now has to be individually tailor-made depending on the hardware and software available to developers, and without the right combination, it can be challenging to cover a wide range of impairments with tools.

## 7 Conclusion and Further Work

We have evaluated three different tools for accessibility testing, in addition to interviewing the participants and observing them in their daily work in software development teams. We argue that the methods we tested will help to discover accessibility issues while keeping costs low in an agile project. In our study, when introduced to the tools, the project members became more aware of how to make the software more accessible, and they had a positive attitude toward using the tools throughout their projects. The interviewees stated that they wanted accessibility work to be a team effort, rather than the responsibility of the UX designers or accessibility consultants.

Further work should test other accessibility tools such as screen reader, WCAG, and Personas to try to find an approach for these tools to be integrated into an agile development project. Furthermore, future work should investigate the connection between the cost of doing accessibility work throughout the agile process with the cost saved by fixing accessibility issues early.

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Artifact-Facilitated Communication in Agile User-Centered Design**

Andrei Garcia<sup>1</sup>, Tiago Silva da Silva2(B) , and Milene Selbach Silveira<sup>1</sup>

<sup>1</sup> Faculdade de Inform´atica, PUCRS, Porto Alegre, Brazil<EMAIL_ADDRESS><EMAIL_ADDRESS><sup>2</sup> UNIFESP, Universidade Federal de S˜ao Paulo, S˜ao Jos´e dos Campos, Brazil<EMAIL_ADDRESS>
**Abstract.** One of the main challenges faced while establishing the integration of Agile and User-Centered Design is how to facilitate communication among the invariably distinct involved practitioners. Advocating the use of artifacts as enablers in this scenario, this paper aims to explore and understand the artifacts which can facilitate the communication between developers and designers in an Agile User-Centered Design approach. Drawing upon a netnography of a globally-distributed online community, we carried out community observation, data collection, and data analysis. The data analysis and interpretation pointed out two major themes: artifacts facilitate communication and artifacts support collaboration. Our paper provides an overview of the artifacts used for communication in Agile User-Centered Design and highlights how artifact-facilitated communication ensues in the industry through a perspective from practitioners.

**Keywords:** Agile User-Centered Design *·* Artifacts *·* Communication *·* Netnography *·* Online ethnography

## **1 Introduction**

Agile User-Centered Design (AUCD) evolved from different motivations. On the one hand, Agile aims to satisfy customers through timely releases and responsiveness to change requests without compromising software quality. On the other hand, User-Centered Design (UCD) aims at ensuring appropriate usability of the implemented software, a characteristic that has not been sufficiently considered either in traditional plan-driven approaches nor in agile approaches. UCD addresses this issue but does not consider Agile principles [4]. There is an inherent tension between both schools of thought, and this tension is a core reason why researchers, seeing the value of both arguments, have been investigating how to integrate both approaches [9].

First concrete attempts to integrate Agile and UCD approaches were published about a decade ago. For instance, Sy [30], Fox et al. [12], Ferreira et al. [11], and da Silva et al. [8] came up with very similar proposals about the integration between these two approaches. However, such interaction on a daily basis is still a concern, and one of its main problems is how to facilitate<sup>1</sup> communication between the invariably distinct involved practitioners aiming to build a shared understanding regarding the project context.

This shared understanding among Software Engineering (SE) and Human-Computer Interaction (HCI) individuals is critical to the success of several agile projects, but little has been known about how communication works [5]. Furthermore, the reliance on communication within agile teams is a fundamental characteristic [29]. In AUCD, designers and developers must be prepared to communicate and collaborate.

As Brhel et al. [4], we advocate the idea of artifact-mediated communication in this scenario. Aiming to identify and understand the artifacts used to facilitate the communication between designers and developers in an AUCD approach we carried out a netnographic study in a globally-distributed online community of agile practitioners.

To contextualize our findings, we present a general view of the artifacts used in AUCD (Sect. 2). Then, we detail how the netnographic study was planned and conducted (Sect. 3), and then we present our main findings (Sect. 4 and discuss (Sect. 5). Finally, we draw upon our findings to elaborate our conclusions (Sect. 6) and future work (Sect. 7).

## **2 Artifacts in AUCD**

Salah et al. [24] conducted a systematic review to identify restriction factors regarding Agile and UCD integration and explored practices to deal with them. One of their findings in this review was about the dynamics between developers and designers which addresses the ongoing and continuous communication between the teams. Regarding sharing and understanding design tasks, some practices have been used, such as design studio, developers participating in User Interface (UI) specifications and shared artifacts within the team.

Brhel et al. [4] identified five principles for the integration of Agile and UCD in their study. The fifth one is: "artifact-mediated communication – in AUCD approaches, artifacts should be used to communicate, and document product and design concepts, and should be accessible to all involved stakeholders." For these authors, this principle consists of the use of tangible and up-to-date artifacts – accessible to all involved stakeholders – to document and communicate product and design concepts, which corroborates Sch¨on et al. [26] that state that artifacts can also be used for communication, elaboration, validation, and documentation of requirements in agile environments.

Bearing in mind the importance of the artifacts for the Agile UCD integration, Garcia et al. [13] carried out a systematic mapping to identify which are the artifacts used and in which contexts they have been used to facilitate the

<sup>1</sup> Noteworthy, we are using 'facilitation' as a means of helping people to deal with a process or to reach an agreement or solution without getting directly involved in the process, discussion, etc.

communication in Agile User-Centered Design approaches. In this systematic mapping, the authors identified 20 different artifact groups that play a critical role as a communication facilitator. Prototypes and User Stories stand out as the most used artifacts. Besides, Personas, Sketches, Scenarios, and Wireframes also emerge as essential means for organizing communication and collaboration among AUCD team members.

This mapping study [13] also revealed five distinct events involving artifacts used as communication mediators – Discovery, Planning, Iterative Cycle, Review Meeting, and General Meetings. The Iterative Cycle was the event with the greatest sharing of artifacts, and, in this event, Prototypes, and User Stories once again were the most cited artifacts. Furthermore, artifacts are being used both in physical and electronic formats. Physical artifacts are commonly used throughout Planning and Discovery events; whereas the electronic ones are mostly used during the Iterative Cycle, often as a basis for development.

The importance of the role that artifacts play for AUCD is evident in the literature. However, while their importance is clear, there is no evidence concerning which artifacts are used for the communication between developers and designers, and how and in which ceremonies they are used.

Based on the studies analyzed – AUCD and Artifacts within this context – we can emphasize the importance of specific artifacts to improve teams' communication. Therefore, to understand – from a practitioners' perspective – how and when these artifacts have been used to promote communication within AUCD contexts, we carried out a netnographic study as described as follows.

## **3 Method**

During the last two decades, online environments became rich and vital grounds for ethnographic studies [23]. In the same period, online communities have become one of the most popular forms of online services [21]. Online communities are essentially forums for meeting and communicating with others [2], or in a more detailed definition, online communities are web-based online services with features that make it possible the members to communicate with each other [21]. Along with online environments, the growth of online communities brought by Computer-Mediated Communications (CMC) established a solid research field for online ethnography studies [20].

Online ethnography adopts principles of ethnographic research molded in offline environments and applies them to online environments with necessary adjustments [23]. One of the online ethnography methods is netnography. Developed by Kozinets, netnography is a qualitative research method which adapts ethnography research processes to study cultures and communities that are emerging through CMC [17]. According to Kozinets [18], online ethnography is a generic term used when performing any type of ethnographic research using some sort of online method. Thus it is important to define netnography as a method by referring to a "specific set of related data collection, analysis, ethical and representational research practices" [18].

Thereby, we adopted the netnography method to identify and understand the artifacts used to facilitate communication between developers and designers. The netnography method performed in this study followed the steps of planning and entr´ee, community observation and data collection, data analysis, and reporting as per Kozinets guidance [18] on conducting netnography.

#### **3.1 Planning and Entr´ee**

This step involves the formulation of the research objective and questions, screening and identification of appropriate online communities. Furthermore, it is important to learn about the communities of interest and define the criteria to select the community that will be studied.

**Definition of Research Questions.** The goal of this netnographic study is to understand how artifacts are used to facilitate communication between designer and developer on AUCD approach according to practitioners, which participate in discussions on online communities. Therefore, we defined three research questions: *(RQ1) Which artifacts are used to facilitate communication between designer and developer? (RQ2) In what events are these artifacts being used?* and *(RQ3) What's the role the artifacts play?*

**Identification of Online Communities.** In order to discover appropriate online communities, we took into account the research goal, which led us to search for communities containing the terms "Agile", "User-Centered Design" or "Agile User-Centered Design". Ideally, the community should focus on AUCD, but since this is an approach that can be taken by Agile and UCD, all the three terms were included. Moreover, it is important to mention that in the industry the term User-Centered Design is widely known as User Experience Design, hence we also included this wording to search for communities of interest.

We performed this search using as base Facebook and LinkedIn groups as well as Slack communities. These revealed 36 communities that seemed relevant to the study. Communities containing the research topic but related to job offers, focused in a specific region or country and created for a particular company were not included.

**Criteria Definition to Select the Community.** The selection criteria were composed of seven factors including relevance, activity, interactivity, substantiality, heterogeneity, richness, and experientiality [18]. The relevance is the first and most important factor. For a community to be considered relevant it should have relation to the research focus and questions. The community needs to be active containing recent and regular communications. Interactivity factor is related to the flow of communication between members. Substantial factor regards to the mass of communicators and energetic feel. The heterogeneity factor concerns about either a variety of difference or a consistency of similar type of participants. The community should be rich in data offering more detailed or descriptive data. Finally, there is the experiential factor which is related to the experience that the community offers to the members.

**Community Selection.** Based on the selection criteria, and considering the relevance as the most important factor we selected the *Scrum Practitioners* LinkedIn Group. The decision to cherry-pick this community was based first on the relevance factor, once the community's topic was aligned with the research focus. In addition, the main purpose of this community is to help and spread the knowledge and implementation of agile for practitioners actively practicing the agile product development. Secondly, this community had the most activity containing the average of 2 posts (main threads) and 33 messages per day, the most interactive with the average of 18 comments per post, and therefore contained the most data. Moreover, the community was considered substantial containing 98431 members.

## **3.2 Community Observation and Data Collection**

The second step of the netnographic research involves community observation and data collection [18]. Once we chose the community we could start collecting the data. However, foremost it is imperative to ensure ethical procedures for any online ethnographic study.

**Ethical Procedures.** Prior to the observation and data collection, ensuring ethical research must be an important part of the netnographic research planning. Therefore, to address the ethical issues we followed the procedures defined by Kozinets [18]: identifying the researcher profile and informing members about the research; asking for proper permissions; gaining consent when needed. Thus, we updated our user profiles containing the role as Researcher, additionally, we used the affiliation to openly, and accurately identify ourselves to the community and administrators.

Scrum Practitioners is a closed LinkedIn group with well-defined rules and two administrators. A group administrator is a legitimate gatekeeper who the researcher should approach prior to contacting other group members or collecting any data [18]. In LinkedIn Groups, gatekeepers assume the roles of Owner, Manager or Moderator and they are deemed a 'data controller' as stated by LinkedIn Groups Terms of Service [19].

Because of LinkedIn Groups terms, and considering the administrator as a gatekeeper, we properly asked the group's manager for permissions aimed at data analysis and interaction with the group. Besides, group members were informed about the research with an accurate description of the research focus and interest. No direct quotations were used in the report or publications to ensure the user's anonymity; once direct quotes are increasingly easy to identify through search engines. Also, all data were treated using pseudonyms for the group members.

**Data Collection Strategy.** Netnographic data can assume three forms [18]: archival data, which is already recorded and stored; communicatively co-created or elicited data; and, participative-authored field note data. Additionally, data collection has two elements, which are the data the researcher straight copy from online communities' platforms and the data the researcher writes in from their observations of the community.

Loanzon et al. [20] described that data collection in netnography usually is textual. As per observation, Scrum Practitioners group is mainly focused on textual communication, and their interactions happen through a post, that is usually a question from some agile professional, and the comments on this post from other members trying to collaborate with an answer. Based on LinkedIn Groups post structure, we collected textual data from posts and comments focused on the research questions. Thereby, the collected data derivate from historical posts (archival data), researcher interactions (elicited data), and researcher sketches as field notes (participative-authored data).

We collected archival data bearing in mind the research questions. Thus only posts related to the study topic were downloaded. LinkedIn groups keep all historical posts available, but only posts from March 2015 were gathered to have the most recent discussions. Furthermore, LinkedIn groups provide a search engine inside the community, which supported the archival data exploration. We performed the data collection from August 2017 to January 2018.

**Posters Categorization.** Posters are community members who interact with the group. According to Kozinets [17] they can be sorted in four categories: Tourists, Minglers, Devotees, and Insiders. To match the reality of this study, the description of each category was adapted to be aligned with the study's subject; thus instead of use "consumption activity" wording to define the member engagement, as originally stated by Kozinets, it was adapted to "group topic". Thus, all four categories are detailed next.

Tourists, who lack strong social ties, maintain a superficial interest in the group topic. They have shallow participation in the community, and potentially their participation in the group will not last very long. Tourists are in the community to get information [20]. Minglers, are members who have strong social ties but least interest in the group topic. They have high visibility but limited influence. Devotees are members who maintain a strong interest in group topic but few attachments to the online group. They do not participate actively in all posts, but only in some specific threads that they are interested. Insiders are the leaders of the community, they are both passionate about the group's topic and sensitive to the social welfare of the community. In other words, they have strong social ties and a high group's topic interest. They are well respected by other community members and have frequent and highly visible participation in almost all posts. Therefore, we categorized the members according to these four categories for all collected comments. Besides, we collected information related to the members' industry, job position, region, and skills to help contextualize their comments.

## **3.3 Data Analysis and Interpretation**

Data collection and data analysis is a simultaneous process in qualitative researches [7]. Thus, this netnographic research step occurs concomitantly with community observation and data collection [17]. This step involves organizing, reading, coding, categorizing, and interpreting the data.

**Data Analysis.** We followed the data analysis approach from Creswell [7] through the six defined stages. The first stage commences with data organization and preparation for the analysis. The second stage regards to read and look at all data, in order to have a general sense of information and reflect on its overall meaning on the information collected from participants. Stage 3 addresses the data coding, which is the process of arranging the data by grouping and writing a word representing a category. Stage 4 concerns to generate a description of the setting or people, and also create categories or themes for analysis. Themes or categories can also use the approach of coding, and they are described as the major findings in a qualitative study. In stage 5, it is the moment to define how description and themes will be pictured. Stage 6 is the final step in data analysis, which involves an interpretation of the findings or results. Hence, we took advantage of these six stages approach to analyze the data. Also, we used the qualitative data analysis software QSR International's NVivo 11 [22] to perform the data analysis.

**Data Organization.** Whereas we used the software NVivo to analyze and code all the collected data, it is essential to describe how we organized it. First of all, we imported collected posts into the tool as internal sources and organized them as archival data or elicited data as previously mentioned. Each post was defined as a major case, and each member's comment was considered as an inner case, which was helpful for the coding process. Codes are treated as nodes on NVivo; consequently, we organized and created all codes under the nodes structure. Finally, we classified all inner cases containing the posters information and details according to the posters categorization.

**Data Coding.** Data coding is the process of organizing the data by grouping related data and labeling these categories with a term [7]. Since we have benefited from the results of Garcia et al. [13] we defined a preliminary codebook based on the theory which evolved during data analysis, ending up with 31 codes. From all codes, "Events" and "Artifacts" had sub-nodes that were used to break down the analysis per event and artifact. Furthermore, data coding was performed manually on text-based data since the collected data was textual.

## **4 Findings**

The final set of collected data comprises six main posts, with a total of 134 comments, including archival and elicited data (Table 1). Most of the posts were from 2017, with no related posts between March 2015 and March 2017. Altogether 63 distinct members, from 18 different countries, participated in the posts discussions. The majority of the members who interacted in the collected posts were categorized as Devotees and Insiders due to the level of engagement in Scrum Practitioners LinkedIn group. From the 63 members, 49 were categorized as Devotees, 11 Insiders, and 3 Tourists. Summing up, 95,3% of the members who participated in the collected posts were Insiders and Devotees. According to Kozinets [17], preliminary research reveals that enthusiastic, devoted, energetically involved, and experienced user segments are represented by Insiders and Devotees in online communities.

**Table 1.** Collected data

Our first analytical lens was focused on answering the question *(RQ1) Which artifacts are used to facilitate communication between designer and developer?* Our findings reveal that seven distinct artifacts were mentioned by Scrum Practitioners community members as facilitators in communication between designer and developer. Moreover, the results from Garcia et al. [13] demonstrate that 20 artifacts accomplish this purpose. Therefore, to concentrate on the most used artifacts, we highlighted the intersection between this systematic mapping study and the findings herein presented (Fig. 1). The list of artifacts that were mentioned as communication facilitators includes user story, wireframe, prototype, mockup, sketch, and persona.

**Fig. 1.** Intersection between the Systematic Mapping and the Online Ethnography

All these artifacts may be used in different events of AUCD approach, which led us to answer the second question *(RQ2) In what events are these artifacts* *being used?* Our findings demonstrate that artifacts facilitate the communication throughout the AUCD events, starting with discovery sessions and going to the agile events including planning, iterative cycle, review, and backlog refinement.

For each event, the artifacts can be used for distinct purposes. The outcomes for question *(RQ3) What's the role the artifacts play?* disclosed that during discovery session artifacts as wireframes and prototypes are used to communicate and validate an idea, while personas delineate the user profile and create a shared understanding, as noted by some practitioners. User stories and wireframes were the most cited artifacts on Scrum Practitioners community, consolidating 50% of all artifacts references.

Users story helps to clarify what should be implemented. As Trenton<sup>2</sup> has noted, a well-written user story facilitates the communication, when the designer describes the interactions in the acceptance criteria, and the developers can estimate the effort necessary to implement the story *(Trenton, posted on Scrum Practitioners, March 15, 2017)*. These user story roles are also mentioned by Beyer et al. [3]. They state that user stories are shared with the development team and they can use them to estimate how much effort they require for the next iteration. User stories also facilitate discussions to define if a determined feature will deliver or not value for the product. This discussion also involves the breakdown of large user stories into thin vertical slices, which involves all architectural layers from the user interface to the backend code.

Wireframes support the description of a user story. They are used to support the acceptance criteria and communicate illustratively how a user interface should be structured. Since they illustrate high-level concepts and behaviors [15], wireframes complement user stories description and are necessary to define when a story is ready for development *(Ambrose, posted on Scrum Practitioners, December 17, 2017)*. An overview of all artifacts correlated with the events where they were mentioned and their roles are represented in Table 2.

## **5 Discussion**

The qualitative data analysis and interpretation resulted in two major themes related to our research questions: (1) Artifacts Facilitate Communication and (2) Artifacts Support Collaboration. Nevertheless, before start addressing them, it is essential to discuss team structure. In all collected posts, there were discussions regarding where the designer should work, if as part of the Scrum Team or as part of a "Design Team". This discussion was generated because Scrum Guide [28] states that the Scrum Team consists of a Product Owner, the Development Team, and the Scrum Master, and has no citation to design discipline or designer role. However, the Scrum Guide also explains that Development Teams are crossfunctional, containing all the necessary specialized skills to create a product increment. Thus, the community understanding is that the team must be crossfunctional and the designer should be part of the team. Furthermore, considering

<sup>2</sup> As mentioned in the ethical procedures, we used pseudonyms to refer to members when discussing their comments.

**Table 2.** Overview of artifacts, events, and roles

the approach of AUCD, which has design discipline involved, the team must have a person with great skills in design as part of the team. Donnie has commented about having the designer as part of the team, stating that the designer sits with the team when he/she is not interacting with the end-users *(Donnie, posted on Scrum Practitioners, March 24, 2017)*. Donnie's comment was also supported by other members. Several citations of word "together" referring to the team, were encountered in the analyzed data, as displayed in the word tree (Fig. 2). Moreover, other synonyms were used to state that the designer should be part of the team, such as "along", "alongside", "part of", "sitting with", "whole", "integrate", "embedded", "jointly", and "work with".

#### **5.1 Artifacts Facilitate Communication**

Agile Manifesto asserts in the first sentence: "Individuals and interactions over processes and tools" [27]. Thus, interactions among professionals in the AUCD

**Fig. 2.** Word tree generated from word "together"

approach are essential to conduct the product to the right course and deliver value. A common form of interaction is communication. Designers and developers communicate in different agile events using different artifacts as facilitators. The communication occurs throughout the entire AUCD flow, starting from discovery session, passing through all agile events including planning, iterative cycle, review, and backlog refinement.

As Mathew has noted, during discovery sessions the team, including developers, testers and designers, meet to create low fidelity prototypes to verify the User Interface and interactions feasibility in terms of technical support *(Mathew, posted on Scrum Practitioners, December 20, 2017)*. Thus, the team uses low fidelity prototypes to communicate how the User Interface should be created considering user experience and technical perspectives.

Wireframes and personas were also mentioned as communication facilitators through discovery sessions. Personas are used to delineate the user profile and create a shared understanding between the designer and the developer regarding the product focus. On the other hand, designers use wireframes to validate an idea with users and then employ these same wireframes to validate whether the development team can build it considering how much effort it takes in terms of feasibility and finances. Herman has mentioned that developers can understand what the requirements are, estimated how much effort it will take, and define how they will build them *(Herman, posted on Scrum Practitioners, December 19, 2017)*. Furthermore, Ambrose has commented that these resultant artifacts from discovery sessions are used as foundation to write user stories and/or refine the acceptance criteria of existing user stories that will feed the product backlog, ensuring that the highest priority backlog items are ready to be designed and built *(Ambrose, posted on Scrum Practitioners, December 17, 2017)*. Therefore, wireframes are used as a mean of communication to explain what are the user needs resulting from discovery sessions, as well as to validate and estimate how the requirements should be built.

Likewise, prototypes, as well as user stories, are relevant during planning events where they are used to define what will integrate an iteration. During the estimation process, designers use these artifacts to provide more details to developers for a more accurate estimation [10]. In general, at this event artifacts use to be more flexible and lightweight to facilitate face-to-face communication. Therefore, in the planning, the artifacts are intended for clarifying and sharing the work for the iterative cycle that is starting.

During the iterative cycle, the preliminary prototypes created throughout discovery sessions are used as a reference for designers to generate mockups. All along the iterative cycle, mockups are used to communicate how the user interface should look like, supported by the prototype and user story, which together define the user interface behavior *(Frederick, posted on Scrum Practitioners, March 18, 2017)*. Thus, mockups are important to communicate the user interface details such as colors, typography, and spacing, while prototypes and user stories provide the user interaction definitions. Personas were also mentioned as a communication facilitator during the iterative cycle. This artifact helps both designer and developer to focus and discuss the core product functionalities. Personas are used as means to confirm that the team is going to the right direction to implement the user stories, thus delivering value to the product *(Sophia, posted on Scrum Practitioners, December 15, 2017)*.

By the end of each iteration, there is the sprint review event. During this meeting, while the developers demonstrate the work that has been done, designers can show what is the result of parallel design track. Frederick has posted that wireframes, mockups, and prototypes are shown as result of designer's work in the sprint review, even knowing that these artifacts cannot be deployed as part of the product increment *(Frederick, posted on Scrum Practitioners, March 18, 2017)*. Thus, during sprint review designers can communicate what are the resultant artifacts, and use them to provide an overview of what will be possibly included in the next sprint.

Finally, before the succeeding sprint planning, it is time for product backlog refinement – also known as backlog grooming by practitioners. For the time of refinement, sketches are used to describe the backlog item quickly. They are used to add brief details to a user story, and the designer can easily communicate with the developers about what is expected from the user story even before to have a wireframe or a prototype. *(Tylar, posted on Scrum Practitioners, October 20, 2017)*. Therefore, sketches facilitate quick communication during refinement and help other team members to have a shared understanding.

#### **5.2 Artifacts Support Collaboration**

The collaboration between designers and developers should be supported by facilitating communication of design vision [25]. As mentioned by the authors, sharing an understanding of the design vision can be achieved via design thinking [6], engaging the whole team in design practices and UI specifications, and sharing design artifacts. Design thinking discipline creates an atmosphere of collaboration where the entire team, including designers and developers, can create low fidelity prototypes together to match users needs and technical feasibility *(Benjamin, posted on Scrum Practitioners, December 18, 2017)*. Consequently, the collaborative environment is extremely tied to the community understanding related to the team working as a whole. Figure 3, shows some excerpts considering the code collaboration, which corroborate the idea of close collaboration.

**Fig. 3.** Word tree generated from code "collaboration"

Moreover, InVision [16] and Confluence [1] were mentioned as collaborative tools. InVision provides a collaborative view where developers and designers can interact over shared mockups and prototypes, posting comments and defining the user interface *(Marianna, posted on Scrum Practitioners, October 22, 2017)*. Confluence provides a template where wireframes can be attached, and it also contains a section to add comments and discussions about the artifact *(Marlina, posted on Scrum Practitioners, October 21, 2017)*.

Another collaborative approach mentioned by the community was Lean UX. Lean UX stands on three foundations: design thinking, agile software development, and Lean startup method [14]. Since this approach stands on these three foundations, it also considers the team is working as a single unit, creating a collaborative environment. The author states that the artifacts such as stick notes, sketches, wireframes, and paper prototypes, created during kickoff and ideation sessions – also named discovery stage – are meaningful to the team since they created these artifacts together. Thus, Lean UX also defends the idea of using artifacts to support collaboration and creates a shared understanding necessary to create the team synergy.

## **6 Conclusion**