# How to Measure Performance in Agile Software Development? A Mixed-Method
Study

Kevin Phong Pham  University of Applied Sciences FHDW Hannover
Hannover, Germany
<EMAIL_ADDRESS>Michael Neumann  University of Applied Sciences
and Arts Hannover
Dpt. of Business Information Systems
Hannover, Germany
<EMAIL_ADDRESS>
###### Abstract

Context: Software process improvement (SPI) is known as a key for being
successfull in software development. Measuring quality and performance is of
high importance in agile software development as agile approaches focussing
strongly on short-term success in dynamic markets. Even if software
engineering research emphasizes the importance of performance metrics while
using agile methods, the literature lacks on detail how to apply such metrics
in practice and what challenges may occur while using them. Objective: The
core objective of our study is to identify challenges that arise when using
agile software development performance metrics in practice and how we can
improve their successful application. Method: We decided to design a mixed-
method study. First, we performed a rapid literature review to provide an up-
to-date overview of used performance metrics. Second, we conducted a single
case study using a focus group approach and qualitativ data collection and
analysis in a real-world setting. Results: Our results show that while widely
used performance metrics such as story points and burn down charts are widely
used in practice, agile software development teams face challenges due to a
lack of transparency and standardization as well as insufficient accuracy.
Contributions: Based on our findings, we present a repository of widely used
performance metrics for agile software development. Furthermore, we present
implications for practitioners and researchers especially how to deal with
challenges agile software development face while applying such metrics in
practice.

###### Index Terms:

Agile methods, agile software development, performance metrics, process
improvement

##  I Introduction

In the evolving field of software development, agile methods like Scrum,
Kanban, and eXtreme Programming have become essential [1]. They provide the
flexibility and adaptability agile development teams need to meet changing
requirements and customer demands, which are triggered by the increased
dynanism of the markets [2]. Today, agile methods used worldwide in various
different contexts and are thus often understood as state of the art
approaches in software development.

We know, that the ability of agile software development teams to react to new
or changed circumstances is one of the major objetives for companies to use
agile methods in practice (e.g., [1]). While using agile methods measuring
teams’ performance becomes particularly essential because it fosters
transparency within the team [3, 4]. This transparency is of high importance
for the iterative process of inspection and adaptation that agile approaches
emphasize and on which the process improvement relies on. By continuously
monitoring and evaluating key performance metrics, teams are able to identify
areas where processes can be improved, inefficiencies can be eliminated, and
overall productivity can be increased [3]. This iterative approach of inspect
and adapt allows agile software development teams to foster continuous
improvement (which we also know as Kaizen) leading to central objectives using
agile methods in practice, e.g., stay aligned with customer needs and project
objectives effectively. Performance metrics serve as feedback mechanisms that
inform the team whether their adaptations are moving the project in the right
direction [5].

We know that effective measurement and analysis of these metrics are
fundamental realizing the potential of agile methods in practice, ensuring
that the teams adapt quickly and continuously steers the project towards its
goals [4]. Thus, agile software development necessitates the use of suitable
performance metrics measure and enhance team performance and efficiency.

Previous research [6, 7] shows that performance metrics are crucial for
understanding, predicting, and evaluating software development projects.
Existing software metrics have been widely studied, and recent studies have
discussed their reasons for use and effects within ASD [8]. Choosing and using
the right metrics can facilitate early problem detection and enhance decision-
making within teams. Conversely, using inappropriate metrics can introduce
biases and lead to undesirable behaviors. [4]).

Current knowledge and use of performance metrics in ASD face several
challenges. For example, the high variety in estimation techniques [9] or
methods provides flexibility for agile software development teams [10, 11, 12]
In turn, this situation also leads to an increased complexity ensuring the
selection of the most appropriate metric for a given context. This can lead to
inconsistencies in data quality if the chosen method does not align well with
the specific project requirements or user capabilities. The comparability of
teams is restricted due to the use of different metrics, as the absence of
uniform structures allows for varied approaches to measurement. Additionally,
the variability in methods can complicate training and standardization efforts
across an organization, potentially resulting in misinterpretations and
misalignment in strategic objectives.

Thus, the above motivates the objective of our study, which is refined by the
following research questions:

  * •

RQ1: Which metrics are used in agile software development teams to measure
performance?

  * •

RQ2: What are real-world challenges agile software development teams face when
using performance metrics?

  * •

RQ3: What specific performance metrics can be used and optimized in practice
to measure project success?

The paper at hand is structured as follows: In Section II, we give a brief
introduction on metrics in the area in agile software development and further
provide an overview of the identified work related to our studies topic.
Section III describes our research design. In Section IV, we present the
results of our study followed by a discussion to present practical
implications in Section V. Before the paper closes with a summary in Section
VII, we outline the limitations of the study in Section VI.

##  II Background & Related Work

###  II-A Metrics in Agile Software Development

Metrics are quantitative measures used to assess, quantify, and monitor
various aspects of systems, processes, products, or performances [13, 14]. In
ASD, metrics are of high importance role in measuring the performance,
quality, and progress of development processes [15].

However, agile approaches with their iterative-incremental characteristic
differ significantly from phase-oriented process models like the Waterfall
approach with the result at the end of the project. Thus, several authors
describe agile methods as a reaction to established phase-oriented (or big
design upfront [16]) approaches (e.g., [17, 18]. For the metrics used in
software development, this fundamental led to the situation that established
metrics used in traditional settings are not fully transferable to agile
environments [19].

Phase-oriented approaches often employ rigid, predefined metrics focused on
process adherence and milestone achievement. In contrast, agile methods
utilizes more flexible, iterative metrics that evolve with the project,
emphasizing both product quality and process efficiency [15]). Metrics in
phase-oriented software development are typically quantitative and reviewed at
the end of phases (or the beginning of new ones). In turn, agile metrics
include both qualitative and quantitative data and are updated more frequently
to reflect ongoing feedback and adjustments [15].

This discrepancy highlights the need for developing specific metrics tailored
to an agile method in use, a need driven by the increasing popularity of agile
approaches [20]). As described in literature, metrics provide objective
information that enables precise and comparable assessments [15]. They convert
abstract concepts into measurable units, which helps in identifying successes
or challenges. Metrics also allow continuous monitoring and control of
processes and performance, facilitating the tracking of developments over time
[15]. Additionally, metrics serve as a foundation for informed decision-making
by highlighting problems, identifying improvement opportunities, and
developing optimization strategies [15]. Moreover, different metrics can
challenge team members to achieve better results, potentially leading to
behavioral changes within the team [15].

Metrics can be grouped into several categories, each with a specific purpose
for the tracking and analysis of certain aspects of a project. Some of the
main categories are e.g. process metrics, product metrics, quality metrics or
risk metrics [21]. Each of these main categories can be further distinguished
into more specific sub-categories to provide more detailed insights into
particular areas or processes. In their comprehensive study, Usman et al.
employed a taxonomy design method to systematically and precisely organize
various types of metrics [22]. This taxonomy supports structuring the metrics
to ensure they are categorized accurately, facilitating better analysis and
utilization in software development projects. Performance metrics are
quantifiable measures used to evaluate the effectiveness, efficiency, and
quality of various aspects of an operation, process, or system. These metrics
are crucial for assessing performance, guiding decision-making, and
identifying areas for improvement. Previous research and reviews provide a
wide variety of performance metrics available, reflecting the diverse needs
and goals of different projects and industries [8, 22]).

###  II-B Related Work

ASD has gained significant research interest in recent years as organizations
seek to enhance their adaptability and deliver high-quality software solutions
in rapidly changing environments [3].

Extensive research has examined the benefits of agile methodologies and
metrics across various industries (Misra and Omorodion, 2011 [15]). However,
there are notable gaps in the literature, especially regarding the application
and effects of these methods in regulated sectors such as insurance and
pensions. Research in these areas has been limited and focused specifically on
certain corporate contexts (Choras et al., 2020 [23]; Pichler et al., 2006
[24]).

Systematic literature reviews on agile software development have typically
concentrated on general best practices, often neglecting the unique challenges
of the insurance industry (Kupiainen et al., 2015 [10]; Nguyen and Tran, 2013
[25]). Research primarily focuses on the adoption of agile methods [26] and
the use of metrics to measure their impact [27], with few studies examining
how these practices are applied in real-world scenarios, particularly within
sectors like insurance that require stringent data integrity and security.
This research seeks to address these gaps by specifically analyzing the
application, challenges, and benefits of performance metrics in agile software
development within the insurance sector through a mid-sized german company.

##  III Research Design

We selected a mixed-method research design to gain an in-depth understanding
of metrics in agile software development. Our research design consists mainly
on two different approaches a) a rapid literature review and b) a single case
study using qualitative data collection and analysis methods. Both research
methods were designed, prepared, and conducted using guidelines. In this
section, we first explain our research design, followed by a detailed
explanation of the applied research methods in the subsections below.

Figure 1: Mixed-method research approach

Figure 1 depicts our research design and the connection answering our three
research questions. Initially, a rapid literature review was conducted to
identify, describe and evaluate documented agile metrics answering RQ1. This
phase is essential for the collection of best practices and scientific
findings. To investigate the second and third research question, an assessment
of the current state was then conducted two different focus groups in a mid-
sized company operating in the insurance sector. These interviews are used to
determine satisfaction with current working methods and processes and to gain
industry-specific insights. Recommendations for potential optimizations are
finally developed based on these findings, ultimately leading to suggestions
for improvements that address subsequent research questions.

###  III-A Rapid Literature Review

Rapid Literature Reviews (RLR) are streamlined versions of systematic
literature reviews, designed to quickly synthesize available research within a
condensed timeframe [28]. By focusing on studies and summarizing evidence more
succinctly and efficiently, RLRs provide timely insights into specific topics
of interest often with a dedicated context defined e.g., based on research
questions. This approach can efficiently capture and apply current knowledge
to improve processes and performance metrics.

We designed, prepared, and conducted the RLR mainly based on the guideliens by
Cartaxo et al. [28]. The RLR method followed a three step approach: a)
Defining a search strategy including search strings and select scientific
search engines, b) defining a study selection strategy applying inclusion and
exclusion criteria, and c) data extraction and analysis of the final result
set. However, we took some adaptions of the guidelines [28]. We give more
information below, when we explain the review process on detail.

Key terms were strategically selected based on the research questions to
ensure coverage of relevant aspects. The terms chosen included ”agile”,
”software”, ”metric”, ”performance”, ”mapping”, ”systematic”, and ”taxonomy”.
To optimize the result set by applying a correct search string, we performed
test runs using the search engine from ACM Digital Library to refine the
search string. In total, we created three final search strings:

  * •

”agile AND metric AND performance AND taxonomy”

  * •

”software AND taxonomy AND mapping”

  * •

”agile AND metric AND systematic”

To capture a broad spectrum of relevant primary and secondary studies, we
conducted our search in two scientific databases: ACM Digital Library and
ScienceDirect. The final search runs were conducted on 20.11.2023 resulting in
a total of 43,088 studies. The result set was narrowed down by using filter
settings to apply the inclusion and exclusion criteria related to the
systematic characteristics of the studies such as the article type (research
articles) and the year range filter (max. 10 years old studies; since 2013) to
capture recent trends in the field. After applying the filter settings the
final result set consisted of 24,126 studies.

First and foremost, it is worth to mention that we had to handle a massive set
of potential relevant studies. As this study is designed as mixed-method study
and the RLR was selected because of its characteristic to summarize evident
knowledge in an efficient way, we decided to focus on the most relevant
results. Thus, we focused the literature review including the study selection
process on the first 25 papers per search run (150 studies in total). However,
it is worth to mention that even we screened the other title of the rest of
the studies from the result set, we did not find other relevant studies for
our RLR. For the result set of the 150 included studies, we performed a three-
step selection process—starting with title evaluation, followed by abstract
analysis, and culminating in full-text review based on relevance and depth of
insight—only 5 scientific papers were ultimately selected for detailed
analysis in this study. Further information related to the RLR including the
protocoll of the study and the selection process are made available at Zenodo
[29].

###  III-B Single Case Study

We decided to conduct a single case study using a qualitative research
approach as we want to create an in-depth understanding of the application of
performance metrics in agile software development. For the design and
preparation of the study, we used the guidelines from Runeson and Hoest [30]
and Yin [31]. As shown in Section II, research results emphasize the
importance and benefits of using metrics in agile software development
covering different industries (e.g., presented by Misra and Omorodion [15]).
However, gaps remain in the literature, particularly concerning their
application and impacts in regulated sectors such as insurance and pensions.
Consequently, this study focused on examining the use and effects of these
metrics within a German mid-sized software development company operating in
the insurance sector. Here, we use the anonymized name Dunder Mifflin Inc. for
the case company. Further information about the case company can be provided
upon request and after a proof by the case company. Dunder Mifflin Inc.
provides a unique opportunity to understand how agile practices and metrics
perform in an environment characterized by stringent regulatory requirements
and complex risk management needs.

As mentioned above, the second research method covered qualitative data
collection and analysis using two focus groups applying semi-structured
interview sessions. Here, we aimed gathering the data with key focus groups
within agile software development teams in the case company, selected based on
their central roles using performance metrics. The first focus group included
Software Developers of various experience levels, providing diverse
perspectives on metric application. In the second focus group we interviewed
Product Owners, who offer insights into how metrics influence decision-making
and product planning. An overview of the participants profiles is given in
Table I.

TABLE I: Characteristics of the participants per focus group | Developer | Product Owner
---|---|---
Sample size (n) | 6 | 2
Average experience in years | 11.75 | 12.5
Average employment period |  |
in years at the company | approx. 8.5 | 12.5
Average agile experience in years | approx. 4.5 | 5

This approach allows an in-depth understanding of group dynamics and
individual experiences related to performance metrics in agile settings. The
interview questions were meticulously designed to allow participants from the
case company to openly share their experiences, focusing on issues and
challenges related to the use of performance metrics. The questions covered
various aspects of using performance metrics, including type, frequency and
relevance of metrics used as well as existing challenges and success stories.
Attention was paid to asking open-ended questions that provided scope for
detailed responses and captured the various perspectives of the participants.
The interview guideline used is available at Zenodo [32].

We refrained from recording the focus group dates for three reasons: a)
Regulatory circumstances in the case company and the resulting consequences
for later publication, e.g. through blocking notices, b) Potential effects of
the participants with regard to their openness and transparency during the
interviews and c) Recordings of group discussions are difficult to transcribe
because, for example, it is not always clear who is speaking. Both focus group
appointments were conducted in German and onsite in the office of the case
company. This data collection approach made informal talks possible and we
further were able to consider non-verbal communication. Both focus group were
held in January 2024 and took between 30 to 45 minutes, moderated by the first
author.

We performed the data analysis using the field notes and manually created
protocols of the focus group sessions. All the data was documented using
Microsoft Excel sheets and notes in a textfile. In the first analysis step, we
merged the data to one Microsoft Excel sheet and categorized the data based on
the interview guideline. The data analysis was performed using open coding
aiming to reduce the complexity of the data. The open coding process led to a
comprehensive, but more systematic data basis which we then analyzed in an
coding to identify categories.

This two-step data analysis approach enabled a comprehensive analysis of the
current state of metrics used in the teams in which the focus group
participants operate. Considering ethical reflections related to our
qualitative research approach including humans, their behaviour and beliefs,
we took several actions. First, we guarantee the anonymity of each participant
of the study and the case company including all relevant information such as
organizational units, software development projects, or even products. Pre-
arrangements were made with volunteer participants to ensure structured
interviews and we took the consent by all the participants to be allowed
taking field notes of the interviews. Thus, participants responses were
briefly transcribed by taking notes during the focus group sessions.

##  IV Results

###  IV-A Repository of Performance Metrics in Agile Software Development

In this subsection, we present the findings from our rapid literature review
and answer our first research question: Which metrics are used in agile
software development teams to measure performance?

TABLE II: Overview of the included studies Publication [Ref.] | Research design | Year | Database
---|---|---|---
S1 [11] | Systematic literature review | 2014 | ACM
S2 [33] | Systematic Mapping Study | 2022 | ACM
S3 [34] | Multiple Case Study | 2019 | ACM
S4 [8] | Systematic literature review | 2015 | SD
S5 [12] | Systematic Mapping Study | 2022 | SD

Following the previously described selection process of the current
literature, a total of five studies were retained for detailed analysis. Each
of the identified studies (Table II provides an overview) provides a
comprehensive overview of methods and metrics used in agile software
development. However, there are remarkable differences between the identified
metrics in several aspects. A thorough analysis of the metrics listed in these
studies led to the development of a repository of performance metrics, which
documents the metrics in detail and is available at Zenodo [35]. It provides a
clear understanding and comparison of agile software development metrics
highlighting similarities and differences. Commonly referenced metrics
included Story Points, Function Points, and Velocity, illustrating their
prevalence in agile methods [10, 12, 11]. The variety of metrics presented
demonstrates the adaptability of agile methods to various requirements in
different contexts. Covering established metrics like Story Points or Function
Points, but also more specific agile metrics such as Custom Metrics and Lead
Time, there is a broad spectrum available for performance evaluation. The
studies collectively offer an extensive collection of metrics customized
specifically for measuring work effort, showcasing the flexibility and depth
of metric usage in agile environments.

Based on our analysis, we can highlight Velocity as a key metric for
estimating workload in planning future iterations, emphasizing its role in
measuring the efficiency and performance of agile software development teams
[10, 36]). Its widespread use and examination in these studies underline
Velocity’s important function in planning and managing agile development
processes.

Furthermore, metrics specifically designed to assess the accuracy of
estimations were prominently featured. The study by [11] emphasize the
relevance these metrics serve in evaluating how close initial estimates are to
the current outcome or output. This capability is instrumental in refining the
estimation processes used in iteration planning meetings.

The analysis also highlights that several metrics pertain specifically to the
process of error identification and correction [12]. This focus reflects the
complexity and multifaceted nature of evaluating the performance of agile
software development teams. The variety of metrics related to bug fixing
emphasizes the importance of continuous improvement and adaptability to
unforeseen challenges in agile methods.

Answering RQ1: Based on the identified studies we created a comprehensive
repository of performance metrics used in agile software development,
showcasing a wide range of metrics such as Story Points, Function Points,
Velocity, Custom Metrics, Test Coverage, and Lead Time.

###  IV-B Challenges while Using Performance Metrics in Practice

Based on the results from our focus group study, we answer the second research
question; RQ2: What are real-world challenges agile software development teams
face when using performance metrics?

The interview involved developers from two different teams at the company, so
overlapping responses were observed, especially within the same team where
similar metrics are used, and perceptions of problems and challenges are
alike. These interviews provide valuable insights into the practical
application of metrics from the developers’ perspective, forming a crucial
foundation for further discussion and analysis for both RQ2 and RQ3.

The findings reveal that developers at the company utilize a range of metrics
and tools in descending order of frequency:

  * •

Story Points

  * •

Four-Eyes Principle

  * •

Build Status

  * •

Test Coverage

  * •

Person-Days

Additional metrics such as Runtime Stability, Test Success Rate, Code Quality,
Code Smells, and MQ Service Status are also applied. Metrics are typically
measured initially in the development process. For instance, Story Points are
estimated during the Backlog Refinement every two weeks, where all cards in
the backlog are discussed and estimated through oral Planning Poker. Estimates
are made as soon as a requirement arises and metrics are adjusted only in
exceptional cases throughout the development cycle. The teams primarily use
tools like Grafana Dashboard, SonarQube, Jira, and Confluence to measure
metrics and monitor progress. This setup underscores the structured yet
flexible approach to metric usage in agile environments at the company,
highlighting both commonalities and variations in experiences and
methodologies among the developers.

However, several issues and challenges related to the application of metrics
and estimation methods were identified. A recurring issue was the perceived
irrelevance or inadequacy of Story Points in some teams, which was attributed
by some developers to the lack of commitment to sprints and the absence of
Velocity measurement. The oral Planning Poker method was frequently mentioned
as insufficiently detailed, and there was confusion about the exact process
and estimation method. Developers also reported difficulties using Burn-Down
Charts and estimating Story Points for cards. Additionally, the order
situation was described as unpredictable, leading to disrupted workflows and
changes in prioritization, necessitating adjustments to estimates during the
development cycle. This resulted in discrepancies between the initial
estimates and the actual time and effort required. It was emphasized that the
accuracy of estimates should be reviewed at the end of each sprint to address
these discrepancies effectively.

In addition the interviews with Product Owners (POs) at the company provided a
broader perspective on the organization’s agile methods and practices. POs
core competency rely in defining requirements and prioritizing them, offering
valuable insights into the product management wrt. to the agile software
development within the organization. Thus, they enabling a more comprehensive
assessment of the software processes in use. Technical requirements are
developed through close interactions between various teams and committees
(e.g., communities of practice), with a focus on capturing and evaluating
professional needs. Nevertheless, we identified highlighted issues with the
precise definition and scope of action of POs, particularly regarding their
role and clarification of responsibilities. The evaluation of the use of Story
Points presented a mixed opinion. Generally, the method is considered
understandable and effective for estimating the scope and effort of tasks.
However, concerns about the transparency and accuracy of these estimates were
raised, particularly regarding the influence of oral estimates during Planning
Poker. An overview of the development process is achieved through regular
exchange formats like weekly meetings and the use of the Jira board, with
Story Points serving as a unit of measure to track progress and capacity.
Regarding quality metrics, the focus is on domain tests, system stability, and
performance.

The respondents emphasized the importance of regular interactions with
developers to ensure the quality of software solutions. User feedback and the
continuous improvement of systems also play a crucial role. Challenges in the
agile process were particularly identified in requirements management and the
clarification of responsibilities. Issues such as lack of standardization and
occasional transparency problems during ticket handover were cited as causes
for conflicts and misunderstandings. The POs expressed a desire for clearer
definition of responsibilities and improved information exchange between teams
to enhance efficiency and transparency in the development process.

Answering RQ2: Qualitative interviews with developers and product owners at
the company highlighted the structured yet flexible use of a range of metrics
and tools, including Story Points, SonarQube, and Test Coverage, to monitor
agile development processes. Key issues identified include the perceived
inadequacy of Story Points and challenges with the clarity and accuracy of
estimation methods like oral Planning Poker, leading to discrepancies in
workload estimates and project management. Product owners emphasized the
importance of defining roles clearly and improving communication between teams
to enhance project efficiency and transparency.

##  V Practical Implications

In this section, we first discuss practical implications based on the findings
presented in the previous section aiming to answer our third research
question; RQ3: What specific performance metrics can be used and optimized in
practice to measure project success?

Analyzing the results of the rapid literature review and the interviews allows
to understand the utilization and optimization potential of performance
metrics in agile software development teams, including those at the
investigated company. It was found that while not all metrics described in the
literature are implemented at the company, due to the vast variety available,
relevant metrics are actively integrated into the software development
process. This indicates that the selection and implementation of metrics are
based on a careful balancing process, tailored to meet the specific needs and
contexts of the individual development teams. The analysis of the results of
the rapid review provides a guideline to highlight areas for improvement and
suggest possible adjustments or additions to metric usage. This comparison
between the findings from the literature and the actual practices at the
company serves as a valuable foundation for identifying best practices and
developing recommendations to optimize the use of metrics in agile software
development.

The use of Story Points is a common practice in agile software development
teams for estimating the effort required for tasks and user stories, as
frequently mentioned in the literature and interviews. This method allows for
a relative assessment of tasks, enabling teams to gauge the effort of a task
in comparison to others without relying on absolute time estimates like
person-days. Story Points enhance agility, offering adaptability to changes
and uncertainties, and allowing teams to respond flexibly to new information
or changes in requirements. However, a notable disadvantage is the lack of
objectivity in Story Points, which can lead to inconsistencies and
uncertainties across different teams. This variability challenges the
comparability of Velocity between teams and could potentially hinder the
scalability of agile practices at the enterprise level. Furthermore, if Story
Points do not correlate well with Velocity or are not considered binding, it
can affect the predictability of work progress. Both developers and product
owners at the company have criticized the inadequate determination of
Velocity, highlighting this as a significant issue.

Although Velocity is often identified as a key metric for measuring team
performance in agile software development, it is not actively utilized within
the investigated case company. Velocity is typically used to gauge the amount
of work completed (or also described as the acceleration of a team), e.g.
number of Story Points, within a specific timeframe, like an iteration. Its
measurement can provide planning certainity for future sprints, helping teams
predict their capacity and set realistic goals by understanding the average
amount of work completed per sprint. This may enable efficient resource
utilization and continuous improvement as teams can adjust their used software
development processes and methods based on identified trends. However,
Velocity’s effectiveness can vary significantly between teams due to different
definitions of ”done” and varying levels of efficiency. This variability can
make comparing Velocity between teams challenging and may lead to manipulation
of Velocity figures by teams wanting to appear more or less productive.
Despite these potential drawbacks, Velocity can be a valuable metric if
interpreted correctly and integrated into the broader context of the agile
development process. It should not be the sole indicator of project success
but rather a tool for ongoing improvement.

Planning Poker as described in the literature and also used in agile teams at
the company promotes collaboration and unbiased estimates by allowing open
discussion and maintaining anonymity. The method enhances team collaboration
by facilitating open discussions on estimates, allowing for the integration of
diverse perspectives, which helps in achieving more accurate workload
assessments (Moløkken-Østvold et al., 2008). However, its oral implementation
at the company could lead to uniform estimates influenced by prior responses,
potentially reducing accuracy. The sequential approach could diminish the
impact of varying perspectives and levels of experience. It also allows less
room for individual contemplation and reflection, as members must immediately
respond to their colleagues’ estimations. Despite criticisms from developers
and product owners regarding its execution, the fundamental practice of
Planning Poker is still positively regarded within the team, suggesting that
adjustments to its implementation could enhance its effectiveness. Using
Planning Poker cards could facilitate discussions and enhance all team
members’ involvement in estimation. Structured and moderated discussions
ensure that all team members express their views and that diverse perspectives
are adequately considered, leading to more accurate and agreed-upon
estimations [37].

Expert Estimations is recognized as a common metric in literature and is
frequently used at the investigated company, particularly in release planning.
This method allows experienced experts to use their knowledge and skills to
provide qualitative assessments of work efforts, leading to realistic and
high-quality estimates, especially when the experts are familiar with the
specific requirements and context. However, challenges such as varying
opinions among different experts can lead to disagreements and uncertainties.
Additionally, compared to quantitative methods, Expert Judgement might be less
objective and transparent, potentially affecting the traceability of
estimates. The reliance on individual expertise can also introduce biases,
particularly in the absence of clear guidelines or standardized procedures.
Furthermore, the rapid review revealed various other metrics for estimation,
including advanced techniques such as neural networks.

The literature generally views metrics positively as they help measure,
optimize, and manage various processes in ASD. Metrics are utilized for
several reasons, including enhancing efficiency, tracking progress, planning
future sprints, and measuring customer satisfaction [33]. It is important to
note that implementing improvements through metrics can entail additional
work, including training, process adjustments, and monitoring new metrics.
Thus, a careful balance between the potential benefits and the extra effort is
necessary to ensure that the changes provide genuine value to the ASD teams.

Answering RQ3: The application of Planning Poker and Expert Estimation could
benefit from adjustments to improve objectivity and reduce bias, such as
integrating anonymized estimation processes and standardizing definitions of
completion across teams. Enhancing the use of Velocity by ensuring it is more
consistently applied and reviewed can also aid in better sprint planning and
resource allocation. Ultimately, refining these metrics and their
implementation methods can lead to more accurate estimations and more
efficient project management in agile environments.

##  VI Threats to Validity

Though we followed a systematic research design using guidelines for the rapid
literature review [28] and the focus group [30], some limitations need to be
taken into account. We discuss the limitations of our study using the threats
to validity concept, focusing on the measures we took to address the threats
to validity below.

Construct validity: A major limitation in literature applies related to the
completeness of the identified result set. However, we countered this aspect
using a systematic approach with the applied guidelines for a rapid literature
review. Furthermore, the study selection was performed by applying defined
inclusion and exclusion criteria and a cross-check of the selection by the
second author.

Also, construct validity threats apply for the qualitative focus group
approach. First, the length of the focus group sessions took between 30 and 45
minutes. Such an appointment can be tiring in the long run and thus, affect
the quality of the results. For this reason, we did not hold the focus group
meetings at off-peak times during the working day, but during core working
hours. Sufficient breaks were also planned and implemented.

Internal validity: Although we prepared the focus groups based on a systematic
literature search in the form of a rapid review, there are some limitations
that need to be considered to strengthen the chain of evidence. We took three
actions to reduce the risk of bias. First, the guiding questions for the focus
group were non-leading questions so as not to induce any implicit direction
(e.g., positive or negative affect of metrics application) in the
participants. Second, the flow of the leading questions was semi-structured,
which allowed us to go deeper into the direction the participants were
seeking. Third, we considered a mix of roles, expertise, and experience in the
composition of the focus group participants (see Section III and Table I for
further details).

Furthermore, we did not record the focus group workshops in order to encourage
participation and active involvement during implementation. We are aware that
this may have a negative impact on the data quality, so we systematically
documented the results and took detailed notes also for the informal talks
around the workshops. The focus groups were prepared and moderated by the
first author. The second author was involved in reviewing the analysis
results.

External validity: The external validity of the study could be improved by
including additional focus groups from the case company or additional cases
from other industries and regions. It would also be useful to take agile
software development teams with different agile maturity levels under study,
as we assume that effects on the selection and successful integration of
metrics may apply.

##  VII Conclusion & Future Work

In this paper, we present the results of our mixed-method study dealing with
performance metrics in agile software development. In particular, our study
dealt with three research questions. Below, we conclude our findings based on
the research questions, before we give a brief overview of our planned future
work activities. First, the results from our literature review show that a
wide range of metrics in agile software development is used. For example,
several metrics to measure performance or software quality exist and are
applied in practice. We decided to create a repository of metrics for/in agile
software development to provide a systematic overview of used metrics. Second,
based on the findings from our two focus groups (in a single case study), we
found that some metrics (e.g., Story Points) may be inadequat for the given
context or the underlying objective applying them to a team. Also, metrics for
effort/complexity estimation (e.g., oral Planning Poker) are challenged by
misconceptions or misunderstandings how the metric may be applied. However, we
also identified that the application of metrics is of high importance as they
increase transparency of the teams’ outcome and progress during an iteration.
Third, anlyzing the findings and results from the two other RQs, we found that
it may be valueable approach to apply a metric like the Velocity measuring the
progess of an agile software development team.

As our study is obviously limited by considering a single case company, we
currently plan to expand the study to other highly regulated contexts like
finance or public administration.

## References

  * [1] VersionOne and Collabnet, “17th annual state of agile survey,” 2023. [Online]. Available: stateofagile.com
  * [2] N. Bennett and G. Lemoine, “What a difference a word makes: Understanding threats to performance in a vuca world,” _Business Horizons_ , vol. 57, no. 3, pp. 311–317, 2014.
  * [3] F. Almeida and P. Carneiro, “Perceived importance of metrics for agile scrum environments,” _Information_ , vol. 14, 2023.
  * [4] J. Soini, “A survey of metrics use in finnish software companies,” in _Proc. of the Intl. Symposium on Empirical Software Engineering and Measurement_. IEEE, 2011, pp. 49–57.
  * [5] D. Paulish and A. Carleton, “Case studies of software-process-improvement measurement,” _Computer_ , vol. 27, pp. 50–57, 1994.
  * [6] M. Choraś, T. Springer, R. Kozik, L. López, S. Martínez-Fernández, P. Ram, P. Rodriguez, and X. Franch, “Measuring and improving agile processes in a small-size software development company,” _IEEE Access_ , vol. 8, pp. 78 452–78 466, 2020.
  * [7] M. G. Salido O., G. Borrego, R. R. Palacio Cinco, and L.-F. Rodríguez, “Agile software engineers’ affective states, their performance and software quality: A systematic mapping review,” _Journal of Systems and Software_ , vol. 204, p. 111800, 2023.
  * [8] E. Kupiainen, M. Mäntylä, and J. Itkonen, “Why are industrial agile teams using metrics and how do they use them?” in _Proc. of the Intl. Workshop on Emerging Trends in Software Metrics_. ACM, 2014, pp. 23–29.
  * [9] M. Neumann, “Towards a taxonomy of agile methods: The tree of agile elements,” in _Proc. of the Intl. Conf. in Software Engineering Research and Innovation_ , 2021, pp. 79–87.
  * [10] E. Kupiainen, M. Mäntylä, and J. Itkonen, “Using metrics in agile and lean software development – a systematic literature review of industrial studies,” _Information and Software Technology_ , vol. 62, pp. 143–163, 2015.
  * [11] M. Usman, E. Mendes, F. Weidt, and R. Britto, “Effort estimation in agile software development: A systematic literature review,” in _Proc. of the Intl. Conf. on Predictive Models in Software Engineering_. ACM, 2014, pp. 82–91.
  * [12] L. López, X. Burgués, S. Martínez-Fernández, A. Vollmer, W. Behutiye, P. Karhapää, X. Franch, P. Rodríguez, and M. Oivo, “Quality measurement in agile and rapid software development: A systematic mapping,” _Journal of Systems and Software_ , vol. 186, p. 111187, 2022.
  * [13] D. Heimann, “Metrics and databases for agile software development projects,” IEEE Boston Reliability Society, 2010.
  * [14] C. Kaner and W. Bond, “Software engineering metrics: What do they measure and how do we know,” in _Proc. of the Intl. Software Metrics Symposium_ , 2004.
  * [15] S. Misra and M. Omorodion, “Survey on agile metrics and their inter-relationship with other traditional development metrics,” _ACM SIGSOFT Software Engineering Notes_ , vol. 36, pp. 1–3, 2011.
  * [16] A. Sidky, J. Arthur, and S. Bohner, “A disciplined approach to adopting agile practices: the agile adoption framework,” _Innovation in Systems and Software Engineering_ , vol. 3, no. 3, pp. 203––216, 2007.
  * [17] P. Abrahamsson, O. Salo, J. Ronkainen, and J. Warsta, “Agile software development methods: Review and analysis,” no. 478, pp. 7–94, 2002.
  * [18] L. Williams, “Agile software development methodologies and practices,” in _Advances in computers_. Academic Press, 2010, vol. 80, pp. 1–44.
  * [19] M. Kunz, R. Dumke, and N. Zenker, “Software metrics for agile software development,” in _Proc. of the Australian Conf. on Software Engineering_ , 2008, pp. 673–678.
  * [20] A. Tarhan and S. Yilmaz, “Systematic analyses and comparison of development performance and product quality of incremental process and agile process,” _Information and Software Technology_ , vol. 56, pp. 477–494, 2014.
  * [21] E. Mills, “Metrics in the software engineering curriculum,” _Annals of Software Engineering_ , vol. 6, pp. 181–200, 1998.
  * [22] M. Usman, J. Börstler, and K. Petersen, “An effort estimation taxonomy for agile software development,” _International Journal of Software Engineering and Knowledge Engineering_ , vol. 27, pp. 641–674, 2017.
  * [23] M. Choras, T. Springer, R. Kozik, L. Lopez, S. Martinez-Fernandez, P. Ram, P. Rodriguez, and X. Franch, “Measuring and improving agile processes in a small-size software development company,” _IEEE Access_ , vol. 8, pp. 78 452–78 466, 2020.
  * [24] M. Pichler, H. Rumetshofer, and W. Wahler, “Agile requirements engineering for a social insurance for occupational risks organization: A case study,” pp. 246–251, 2006.
  * [25] D. Nguyen and D. Tran, “A review of effort estimation studies in agile, iterative and incremental software development,” in _Proc. of the Intl. Conf. on Research Challenges in Information Science_. IEEE, 2013.
  * [26] M. Neumann, T. Kuchel, P. Diebold, and E.-M. Schön, “Agile culture clash: Unveiling challenges in cultivating an agile mindset in organizations,” _Computer Science and Information Systems_ , 2024.
  * [27] J. Topp, J.-H. Hille, M. Neumann, and D. Mötefindt, “How a 4-day work week and remote work affect agile software development teams,” in _Proc. of the Intl. Conf. on Lean and Agile Software Development_ , vol. 438, 2022, pp. 61–77.
  * [28] B. Cartaxo, G. Pinto, and S. Soares, “Rapid reviews in software engineering,” in _Contemporary Empirical Methods in Software Engineering_. Springer International Publishing, 2020, pp. 357–384.
  * [29] K. P. Pham, “Rapid review protocol & documentation,” 2024. [Online]. Available: https://doi.org/10.5281/zenodo.12637754
  * [30] P. Runeson and M. Höst, “Guidelines for conducting and reporting case study research in software engineering,” _Empirical Software Engineering_ , vol. 14, no. 2, pp. 131–164, 2009.
  * [31] R. K. Yin, _Case study research: Design and methods_ , 4th ed., ser. Applied social research methods series. Sage, 2009, vol. 5.
  * [32] K. P. Pham, “Interview guideline for the focus groups,” 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11206152
  * [33] D. Chloros, V. Gerogiannis, and G. Kakarontzas, “Use of software and project management metrics in agile software development methodologies: A systematic mapping study,” in _Proc. of the European Symposium on Software Engineering_. ACM, 2022, pp. 25–32.
  * [34] P. Ram, P. Rodriguez, M. Oivo, and S. Martinez-Fernandez, “Success factors for effective process metrics operationalization in agile software development: A multiple case study,” in _Proc. of the Intl. Conf. on Software and System Processes_. IEEE, 2019, pp. 14–23.
  * [35] K. P. Pham, “Performance metrics repo documentation,” 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11206133
  * [36] P. Ram, P. Rodriguez, and M. Oivo, “Software process measurement and related challenges in agile software development: A multiple case study,” in _Proc. of the Intl. Conference on Product-Focused Software Process Improvement_. Springer International Publishing, 2018, pp. 272–287.
  * [37] K. Moløkken-Østvold, N. Haugen, and H. Benestad, “Using planning poker for combining expert estimates in software projects,” _Journal of Systems and Software_ , vol. 81, pp. 2106–2117, 2008.


# An improved paradigm for modeling animal flights at moderate Reynolds
numbers

Kyohei Onoue111Fluids and Thermal Sciences, School of Engineering, Brown
University, Providence, RI 02912, USA.222Kyohei Onoue and Hamid Vejdani
contributed equally to the present work Hamid Vejdani11footnotemark:
122footnotemark: 2 Yunxing Su11footnotemark: 1 Kenneth S.
Breuer11footnotemark: 1333Author for correspondence (email:
<EMAIL_ADDRESS>
## Abstract

We report on experimental and numerical studies aimed at developing an
improved paradigm for modeling avian flights at moderate Reynolds numbers. A
series of experiments were performed to characterize the behaviors of
aerodynamic forces and moment associated with a quasi-steady rectangular wing
over a range of incidence angles, $\alpha$. We demonstrate that, while the
drag coefficient curve, $C_{D}(\alpha)$, can be accurately modeled solely by a
simple trigonometric function, the evolution of lift coefficient curve,
$C_{L}(\alpha)$, is governed by the sum of trigonometric and exponential
functions, where the latter captures the linear variation in lift coefficient
within the small-angle regime, as predicted by the linear inviscid theory. In
addition, we establish an empirical relation between the location of the
center of pressure and the incidence angle, which can be used in conjunction
with the proposed aerodynamic formulas (i.e. $C_{L}$ and $C_{D}$) to evaluate
the pitching moment coefficient, $C_{M}(\alpha)$, about any arbitrary axis.
These quasi-steady formulations are then utilized to simulate forward flight
of a model that possesses dynamical characteristics of a pigeon across various
flight speeds, and the results are compared against previously-reported
experimental data on pigeons. We successfully demonstrate that the proposed
formulas yield accurate predictions of the wingbeat frequency, at least over a
range of Reynolds numbers spanning from approximately 70,000 to 150,000.

## Introduction

There has been a substantial interest in understanding the physics behind
birds and bats flapping flight, both to understand the mechanics of these
natural fliers and also to develop Miro Air Vehicles (MAVs) with similar
physics and size range. The majority of research so far has been done on
relatively small fliers like insects with Reynolds numbers less than 1000
(e.g. Dickinson and Götz, 1993; Dickinson et al., 1999; Sane and Dickinson,
2001, 2002; Berman and Wang, 2007), while for bigger fliers like birds and
bats the Reynolds number would increase to around 40,000. In addition, the
vast majority of current Micro Air Vehicles (MAVs) operate at this medium
range Reynolds numbers and for designing controllers and studying the
stability for these vehicles a computationally-efficient aerodynamic modeling
is required.

The wing aerodynamic model proposed by Dickinson et al. (1999); Sane and
Dickinson (2001, 2002) has been extensively used for modeling aerodynamic
forces (e.g., see, Parslew and Crowther, 2010; Parslew, 2014; Cheng et al.,
2016; Chin and Lentink, 2016) due to its simple nature and low computational
cost, which makes it useful for dynamical modeling, control and optimization
calculations. In Parslew and Crowther (2010) and Parslew (2014), pigeon size
models were analyzed for different flight modes (cruise, climbing and
descending) over a range of forward speeds. Cheng et al. (2016) employed the
quasi-steady modeling for analyzing the mechanics of scape maneuvers in
hummingbirds using the same format for translational lift and drag but with
values adjusted for hummingbirds.

Although the quasi-steady formulation of Dickinson et al. (1999) has been
developed specifically for insects with small Reynolds numbers, it has also
been used for larger Reynolds numbers (Byl, 2010; Chen et al., 2010; Parslew
and Crowther, 2010; Parslew, 2014; Stowers and Lentink, 2015; Cheng et al.,
2016; Read et al., 2016). While for low Reynolds numbers and insect wings the
augmented lift and drag from the stabilized leading-edge vortex (LEV) can be
useful, the increased drag can be seriously deteriorating for large Reynolds
numbers in forward speeds. We intend in this study to propose improvements in
calculating the aerodynamic forces in the quasi-steady formulation for
moderate Reynolds numbers pertinent to avian flights.

## Materials and methods

All experiments were conducted in a closed-return water channel with a test
section of 0.8 m in width and 0.6 m in depth. Fig 1A illustrates a flat plate,
with chord, $c$ = 0.076 m, span, $h$ = 0.456 m and thickness, $\delta$ = 8 mm,
rigidly affixed to the 6-axis force transducer (Delta IP65, ATI Industrial
Automation), in a uniform free-stream with velocity, $U$, and density,
$\rho_{f}$. The free-stream velocity was monitored in real time using an
acoustic doppler velocimeter (Vectrino, Nortek AS). The force transducer
measures the normal, $F_{N}$, and tangential, $F_{T}$, forces, as well as the
pitching moment, $M$, about the rotational axis located at the mid-chord
point; these forces are related to lift, $L$, and drag, $D$, through the
geometry shown in Fig 1B, i.e. $L=F_{N}\cos(\alpha)-F_{T}\sin(\alpha)$ and
$D=F_{N}\sin(\alpha)+F_{T}\cos(\alpha)$, where $\alpha$ denotes the geometric
angle of attack. The angle of attack is modulated by sending an open-loop
command to the rotary stepper motor (Nema23 D600, Applied Motion Products) and
the corresponding value is measured via an optical encoder (E3 optical
encoder, US Digital). In this study, the Reynolds number, based on the chord
length, is defined as $Re=Uc/\nu$, where $\nu$ denotes the kinematic viscosity
of the fluid. Given that the chord length represents less than 10% of the test
section width, the blockage and side-wall boundary layer effects were deemed
minor. Due to the use of a large aspect ratio ($AR=6$) rectangular plate, the
flow associated with the present study is expected to be largely two-
dimensional, and the validity of this assumption will be justified in the
subsequent section.

In the present investigation, the steady-state forces and moment associated
with a steady flat plate immersed in a uniform flow were measured over a broad
range of angles of attack ($\alpha$ = 0∘ $\sim$ 110∘,
$\Delta\alpha=2^{\circ}$) at four different Reynolds numbers varying from
20,000 to 50,000 at an increment of 10,000. At each angle of attack, the
measurements were recorded for 30 seconds, but only the last 20 seconds of the
recording was analyzed during the post-processing in order to isolate the
transient response. The reproducibility of the results was confirmed by
repeating the same set of experiments three times.

## Results and discussion

###  0.1 Part I: quasi-steady aerodynamic formulations

The evolutions of lift and drag coefficient curves as a function of the angle
of attack are illustrated in Figs. 2A and B, respectively, where
$C_{L}=2L/\rho_{f}U^{2}ch$ and $C_{D}=2D/\rho_{f}U^{2}ch$. The lift and drag
coefficients are well approximated by the following empirical relationships in
terms of $\alpha$ (in degrees):

$\begin{split}{C}_{L}(\alpha)=-0.44e^{-0.47\alpha}+1.19\sin(1.74\alpha+20^{\circ})\\\
{C}_{D}(\alpha)=1.04+\sin(1.72\alpha-70^{\circ}),\end{split}$ (1)

where, in each case, the best-fit regression curve with a 95% confidence
interval yielded an R-squared value greater than 0.995. As illustrated in Fig.
2A, the $C_{L}$ curve follows closely the steady thin airfoil theory
prediction with a slope of d$C_{L}$/d$\alpha$ = $2\pi$ at small angles of
attack ($\alpha<8^{\circ}$). One can infer that the laminar boundary layer
separation occurs at approximately $\alpha\approx 8^{\circ}$, and that the
apparent lift augmentation at large angles of attack is presumably due to: (1)
the large-scale steady separation from the sharp leading and trailing edges,
(2) the pressure accumulation on the frontal surface and (3) the viscous
stress induced by the uniform flow. It should be noted that the forces
measured in the present study are “quasi-steady” such that the concomitant
shedding of von Kármán vortices behind the plate does not induce any
measurable time-dependency in the forces and the pitching moment imparted on
the wing. This is presumably because the formation of these large-scale
vortices occurs in the far-wake region approximately 1.5 to 2 chord lengths
downstream of the plate (for Reynolds numbers on the order of
$\mathcal{O}(10^{4})\sim\mathcal{O}(10^{5})$), as discussed in Onoue and
Breuer (2016) and Liu et al. (2017). The validity of the present assertion is
also supported by small standard deviations associated with the force and
moment data presented in Fig. 2, reflecting minor unsteadiness in the
measurements. While the present observation is in contrast to the reports of
Dickinson and Götz (1993) and Wang (2000), where the generation of a von
Kármán street resulted in an oscillatory behavior in the force curves at post-
stall angles, one can attribute this discrepancy to differences in the
Reynolds numbers. Upon the attainment of maximum lift coefficient at
approximately $\alpha\approx 40^{\circ}$, the lift curve decays monotonically
as $\alpha$ is further increased.

As shown in Fig. 2B, the drag coefficient increases monotonically from
approximately 0.1 at $\alpha$ = 0 to a maximum value of 2.0 at $\alpha\approx
90^{\circ}$, conforming very well with the theoretical prediction of 1.95,
wherein the drag coefficient associated with a 2D flat plate facing
perpendicular to the incoming flow can be approximated as $C_{D}=1.95+50/Re$
(Ellington, 1991). In addition, the magnitudes of $C_{L}$ and $C_{D}$ are
comparable to similar steady-state measurements made on a translating 2D
rectangular wing at various fixed angles of attack (Dickinson and Götz, 1993).
A good agreement with the aforementioned studies justifies our previous
statement that the flat plate employed in the current study can, in fact, be
treated as nominally 2D. Interestingly, the drag coefficient curve at high
Reynolds numbers can be well approximated with a simple harmonic relationship
that closely resembles the form derived in the experimental work of Dickinson
et al. (1999), where the authors reported on the steady-state translational
forces (i.e. $C_{L}$ and $C_{D}$) on a dynamically scaled-up $\it{Drosophila}$
wing undergoing a $180^{\circ}$ constant velocity rotation at fixed angles of
attack at a Reynolds number of approximately 140. With the exception of this
similarity, however, the general characteristics of the present force profiles
differ conspicuously from those of Dickinson et al. (1999). First we note
that, while our $C_{L}$ values conform with the thin airfoil theory
predictions for small angles, Dickinson et al. (1999) largely underestimates
the theoretical prediction of d$C_{L}$/d$\alpha$ = $2\pi$ with a value of
approximately 3.8. This discrepancy is attributable to the violation of the
inviscid flow assumption—the basis of the thin airfoil theory (Bisplinghoff et
al., 1957)—in the study of Dickinson et al. (1999), where the experiments were
conducted at a low Reynolds number of $\mathcal{O}(10^{2})$ using the mineral
oil as the working fluid medium. Secondly, the maximum $C_{L}$ and $C_{D}$
obtained in the present study are approximately 45% and 65% smaller,
respectively. Higher steady-state aerodynamic forces associated with a
rotating 3D wing is due, in part, to the formation, growth and subsequent
stabilization of an “attached” leading-edge vortex (LEV) on the upper surface
of the wing during its unsteady motion, as previously noted by a number of
investigators (e.g. Maxworthy, 1981; Ellington et al., 1996; Dickinson et al.,
1999). The general consensus in the archival literature is that the parameter
of rotation (e.g. revolving, flapping and pitching) plays an important role in
the formation and subsequent stabilization of the LEV (e.g. Lentink and
Dickinson, 2009; Wong and Rival, 2015; Onoue and Breuer, 2016), and that this
enhanced vortex stability is directly correlated to the generation of strong
aerodynamic forces (Granlund et al., 2013; Harbig et al., 2013) and pitching
moments (Onoue et al., 2015; Onoue and Breuer, 2016) that far exceed their
static counterparts, easily by a factor of 1.5 to 2.5 in magnitude.

Another useful contribution of the present study is the establishment of a
simple empirical relation between the location of the center of pressure and
the angle of attack. We determined the chordwise location of the center of
pressure (measured from the leading edge) using the formula:
$x^{*}_{cp}=0.5-M/F_{N}c$. Fig 2D illustrates the evolution of $x^{*}_{cp}$ as
a function of $\alpha$ at four distinct Reynolds numbers, where all four cases
collapse nicely. An empirical fit of $x^{*}_{cp}$ is given by Eqn 2, from
which one can evidently infer that the current model predicts that
$x^{*}_{cp}$ approaches the aerodynamic center at 0.25 in the limit as
$\alpha$ tends to zero, which resonates well with the linear inviscid theories
(e.g. Wagner, 1925; Theodorsen, 1935; Garrick, 1936).

$\begin{split}{x}^{*}_{cp}(\alpha)=0.247+0.016\alpha^{0.6}+0.026{\alpha}e^{-0.11\alpha}.\end{split}$
(2)

It should be mentioned that the moment coefficient,
$C_{M}=C_{N}(x^{*}-x^{*}_{cp})$, about any arbitrary point can be easily
approximated using Eqns 1 and 2, where the normal force coefficient is defined
as $C_{N}$ = $C_{L}\cos(\alpha)$ \+ $C_{D}\sin(\alpha)$ and $x^{*}$ denotes
the location of the moment axis with respect to the leading edge (e.g. $x^{*}$
= 0.25 for the quarter-chord point). Using this formulation, we evaluated the
moment coefficient about the mid-chord point ($x^{*}$ = 0.5), and the result
is plotted in Fig 2C with a yellow curve, revealing an excellent agreement
with the experimental measurements (plotted with open circles); note that the
empirical data are normalized as $C_{M}=2M/\rho_{f}U^{2}c^{2}h$. Finally, it
is worthwhile mentioning that the differentiable nature of the proposed
formulas makes them suitable to be used in dynamical simulations that utilize
gradient-based optimization techniques to determine, for example, power-
optimal kinematics of avian wings for different modes of flight.

###  0.2 Part II: simulation

In order to validate the efficacy of the proposed formulas in modeling avian
flights at moderate Reynolds numbers, we simulated forward flight of a model
that replicates dynamical characteristics of a pigeon across various cruise
velocities. The model has two translational degrees of freedom ($x-z$) and one
rotational degree of freedom in pitch ($\psi$), where the latter accounts for
the changes in the body angle, as illustrated in Fig 3A. One of the salient
characteristics of the flight of a pigeon is that the body angle, as well as
the wing’s incidence angle, appear to be a function of the cruise velocity,
which can be appropriately modeled by the pitch degree of freedom, see, e.g.,
Tobalske et al. (2003) and Parslew (2014). In addition to generating flapping
and pronation/supination motions, the model is allowed to modulate its
wingspan during the wingbeat cycle in order to replicate basic wing kinematics
of a pigeon. For aerodynamic modeling, we employed a blade element model and
evaluated the forces exerted on each blade element using the proposed
formulations (Eqns 1 and 2) and those of Dickinson et al. (1999). The
morphometric characteristics and the wing kinematics of the pigeon are assumed
from Tobalske and Dial (1996) with the whole mass $M=0.316$ kg, length of each
wing $l_{0}=27.9$ cm and average wing chord $c=11.2$ cm.

Fig 3B compares the predicted wingbeat frequency associated with the steady-
state forward flight using the proposed formulas and those established by
Dickinson et al. (1999). The results from these simulations are then compared
against the experimental measurements on pigeons by Tobalske and Dial (1996).
For high forward velocities ($U>$ 10 m/s), our formulas perform remarkably
well in predicting the actual wingbeat frequencies, while the formulations of
Dickinson et al. (1999) result in a poor agreement with the empirical
observations. This discrepancy is presumably due to large drag, which can only
be compensated by generating sufficient thrust via an increase in the wingbeat
frequency; note that the incidence angle of the pigeon wing was previously
measured by Hedrick et al. (2002), and estimated to be less than 15∘ at high
cruise speeds (U $>$ 9 m/s). As the forward velocity decreases, the importance
of cruise velocity becomes less pronounced, and the model tries to generate
sufficient lift by either increasing the wingbeat frequency, modulating the
angle of attack ($\alpha$) to a larger value or both; recall that $\psi$ and
$\alpha$ increase as the cruise speed is decreased (see Parslew (2014)).
Therefore, if the maximum lift coefficient is insufficient, the model must
inevitably increase the wingbeat frequency to balance out this deficiency.
Therefore, the fact that our formulation overestimates the actual wingbeat
frequency, which remains nearly constant over a range of forward velocities as
depicted in Fig 3B, suggests that the peak lift coefficient provided by our
formula is inadequate. Furthermore, it is interesting to note that the
predicted wingbeat frequency (based on the formulation of Dickinson et al.
(1999)) appears to converge monotonically to the measurement of Tobalske and
Dial (1996) as the forward velocity decreases; this observation may point to
the importance of rotational movement of the wing in elevating the performance
gains in aerodynamic forces, in particular at lower flight velocities where
the body angle ($\psi$), as well as the wing’s incidence angle ($\alpha$), are
conspicuously large (see, e.g, Fig 5 of Parslew, 2014). The current assertion
is also in line with the fact that the rotational excursion of the wing is
pronounced during hovering, which presumably helps improve the wing’s overall
aerodynamic performance at post-stall angles by promoting the stability, and
thus the attachment, of the LEV over the suction side of the wing, as noted
previously. In essence, we hypothesize that the overprediction of the wingbeat
frequency at lower flight velocities is a manifestation of the fact that a
purely translating wing (as in the current geometry) cannot leverage the
dynamic-stall process (owing to lack of rotational maneuver that stabilizes
and prolongs the vortex attachment) and that the von Kármán vortices
associated with the present measurements are fully detached from the lifting
surface (presumably due to small convective time scales associated with
Reynolds numbers on the order of $\mathcal{O}(10^{4})$), both of which lead to
notably smaller steady-state force coefficients (for large $\alpha$) relative
to those reported by Dickinson et al. (1999) for a revolving $\it{Drosophila}$
wing at a Reynolds number on the order of $\mathcal{O}(10^{2})$.

## Acknowledgements

The authors would like to thank the members of the Energy Harvesting group at
Brown University for their contributions to the development of the
experimental apparatus.

## Competing interests

The authors declare no competing financial interests.

## Author contributions

K.O and H.V. drafted the paper; K.O., H.V. and Y.S. designed and executed the
experiments; K.O. characterized the aerodynamic models; H.V. performed
numerical simulations.

## Funding

This research is supported by the Department of Energy’s Advanced Research
Projects Agency-Energy (ARPA-E).

## References

  * Berman and Wang (2007) Berman, G. J. and Wang, Z. J. (2007).  Energy-minimizing kinematics in hovering insect flight.  _J. Fluid Mech._ 582, 153–168.
  * Bisplinghoff et al. (1957) Bisplinghoff, R. L., Ashley, H. and Halfman, R. L. (1957).  _Aeroelasticity_.  Cambridge, Massachusetts: Addison-Wesley Publishing Co. Inc.
  * Byl (2010) Byl, K. (2010).  A passive dynamic approach for flapping-wing micro-aerial vehicle control.  In _ASME 2010 Dynamic Systems and Control Conference_ , pp. 215–223.
  * Chen et al. (2010) Chen, L., Guan, Y., Zhang, X. and Zhang, H. (2010).  Improvement of an aerodynamic model for biomimetic flapping-wing robots.  In _2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics_ , pp. 49–54.
  * Cheng et al. (2016) Cheng, B., Tobalske, B. W., Powers, D. R., Hedrick, T. L., Wang, Y., Wethington, S. M., Chiu, G. T.-C. and Deng, X. (2016).  Flight mechanics and control of escape manoeuvres in hummingbirds ii. aerodynamic force production, flight control and performance limitations.  _Journal of Experimental Biology_ 219, 3518–3531.
  * Chin and Lentink (2016) Chin, D. D. and Lentink, D. (2016).  Flapping wing aerodynamics: from insects to vertebrates.  _J. Exp. Biol._ 219, 920–932.
  * Dickinson and Götz (1993) Dickinson, M. H. and Götz, K. G. (1993).  Unsteady aerodynamic performance of model wings at low reynolds numbers.  _J. Exp. Biol._ 174, 45–64.
  * Dickinson et al. (1999) Dickinson, M. H., Lehmann, F.-O. and Sane, S. P. (1999).  Wing rotation and the aerodynamic basis of insect flight.  _Science_ 284, 1954–1960.
  * Ellington (1991) Ellington, C. P. (1991).  Aerodynamics and the origin of flight.  _Adv. Insect Physiol._ 23, 171–210.
  * Ellington et al. (1996) Ellington, C. P., van den Berg, C., Willmott, A. and Thomas, A. L. R. (1996).  Leading-edge vortices in insect flight.  _Nature_ 384, 626–630.
  * Garrick (1936) Garrick, I. E. (1936).  Propulsion of a flapping and oscillating airfoil.  _NASA Tech. Rep._ 567.
  * Granlund et al. (2013) Granlund, K. O., Ol, M. V. and Bernal, L. P. (2013).  Unsteady pitching flat plate.  _J. Fluid Mech._ 733, R5.
  * Harbig et al. (2013) Harbig, R. R., Sheridan, J. and Thompson, M. C. (2013).  Reynolds number and aspect ratio effects on the leading-edge vortex for rotating insect wing planforms.  _J. Fluid Mech._ 717, 166–192.
  * Hedrick et al. (2002) Hedrick, T. L., Tobalske, B. W. and Biewener, A. A. (2002).  Estimates of circulation and gait change based on a three-dimensional kinematic analysis of flight in cockatiels (nymphicus hollandicus) and ringed turtle-doves (streptopelia risoria).  _Journal of Experimental Biology_ 205, 1389–1409.
  * Lentink and Dickinson (2009) Lentink, D. and Dickinson, M. H. (2009).  Biofluiddynamic scaling of flapping, spinning, and translating fins and wings.  _J. Exp. Biol._ 212, 2691–2704.
  * Liu et al. (2017) Liu, B., Hamed, A. M., Jin, Y. and Chamorro, L. P. (2017).  Influence of vortical structure impingement on the oscillation and rotation of flat plates.  _J. Fluids and Structures_ 70, 417–427.
  * Maxworthy (1981) Maxworthy, T. (1981).  The fluid dynamics of insect flight.  _Annu. Rev. Fluid Mech._ 13, 329–350.
  * Onoue and Breuer (2016) Onoue, K. and Breuer, K. (2016).  Vortex formation and shedding from a cyber-physical pitching plate.  _J. Fluid Mech._ 793, 229–247.
  * Onoue et al. (2015) Onoue, K., Song, A., Strom, B. and Breuer, K. (2015).  Large amplitude flow-induced oscillations and energy harvesting using a cyber-physical pitching plate.  _J. Fluids and Structures_ 55, 262–275.
  * Parslew (2014) Parslew, B. (2014).  Predicting power-optimal kinematics of avian wings.  _Journal of The Royal Society Interface_ 12.
  * Parslew and Crowther (2010) Parslew, B. and Crowther, W. (2010).  Simulating avian wingbeat kinematics.  _Journal of Biomechanics_ pp. 3191–3198.
  * Read et al. (2016) Read, T. J. G., Segre, P. S., Middleton, K. M. and Altshuler, D. L. (2016).  Hummingbirds control turning velocity using body orientation and turning radius using asymmetrical wingbeat kinematics.  _Journal of The Royal Society Interface_ 13.
  * Sane and Dickinson (2001) Sane, S. P. and Dickinson, M. H. (2001).  The control of flight force by a flapping wing: lift and drag production.  _Journal of Experimental Biology_ 204, 2607–2626.
  * Sane and Dickinson (2002) Sane, S. P. and Dickinson, M. H. (2002).  The aerodynamic effects of wing rotation and a revised quasi-steady model of flapping flight.  _Journal of Experimental Biology_ 205, 1087–1096.
  * Stowers and Lentink (2015) Stowers, A. K. and Lentink, D. (2015).  Folding in and out: passive morphing in flapping wings.  _Bioinspiration & Biomimetics_ 10, 025001.
  * Theodorsen (1935) Theodorsen, T. (1935).  General theory of aerodynamic instability and the mechanism of flutter.  _NASA Tech. Rep._ 496.
  * Tobalske and Dial (1996) Tobalske, B. and Dial, K. (1996).  Flight kinematics of black-billed magpies and pigeons over a wide range of speeds.  _Journal of Experimental Biology_ 199, 263–280.
  * Tobalske et al. (2003) Tobalske, B. W., Hedrick, T. L. and Biewener, A. A. (2003).  Wing kinematics of avian flight across speeds.  _Journal of Avian Biology_ 34, 177–184.
  * Wagner (1925) Wagner, H. (1925).  Über die Entstehung des dynamischer Austrieb von Tragflügeln.  _ZAMM_ 5, 17–35.
  * Wang (2000) Wang, Z. J. (2000).  Vortex shedding and frequency selection in flapping flight.  _J. Fluid Mech._ 410, 323–341.
  * Wong and Rival (2015) Wong, J. G. and Rival, D. E. (2015).  Determining the relative stability of leading-edge vortices on nominally two-dimensional flapping profiles.  _J. Fluid Mech._ 766, 611–625.

## Figure Legends

Figure 1
(A) a schematic of the experimental set-up. (B) a definition sketch of the
forces and moment acting on a flat plate immersed in a uniform free-stream.
Figure 2
(A)-(D), respectively, illustrate the evolutions of CL, CD, CM and
$x^{*}_{cp}$ as a function of the angle of attack, $\alpha$.
Figure 3
(A) a schematic of the simulated pigeon model. (B) The predicted wingbeat
frequencies of a pigeon during its steady-state forward flight over a range of
flight speeds varying from 6 m/s to 20 m/s.

Figure 1: The experimental set-up. (A) schematic of the experimental set-up,
illustrating a flat plate (AR = 6) mounted on a 6-axis load cell about its
mid-chord location. The flat plate is immersed in a uniform flow with a free-
stream velocity, $U$. The present set-up allows for a direct measurement of
normal and tangential forces, as well as the pitching moment exerted on a flat
plate. The angle of attack, $\alpha$, is accurately measured and modulated via
the optical encoder and the rotary stepper motor using an open-loop command.
(B) definition sketch of the forces and moment imparted on a flat plate. Lift
and drag are approximated by decomposing the forces ($F_{N}$ and $F_{T}$) into
components along the streamwise and transverse directions. The normalized
center of pressure, $x^{*}_{cp}$, is measured from the leading edge.

Figure 2: The evolution of normalized lift (A), drag (B), moment (C) and
center of pressure (D) as a function of the angle of attack, $\alpha$, at four
distinct Reynolds numbers. Steady-state forces and moment are appropriately
normalized by the inertial forcing of the fluid flow. The error bar associated
with each data point represents the standard deviation of the time-averaged
measurement. Thin airfoil theory predictions of ${C}_{L}$ (i.e. $2\pi\alpha$)
and ${C}_{M}$ (i.e. $\frac{\pi}{2}\alpha$) are plotted with the gray dotted
lines in (A) and (C), respectively. The regression curves for $C_{L}$, $C_{D}$
and $x^{*}_{cp}$ are plotted with the solid yellow curves, where the
corresponding equations are provided in Eqns 1 and 2. The regression curve for
$C_{M}$ is deduced analytically using the expressions of $C_{L}$, $C_{D}$ and
$x^{*}_{cp}$ (discussed further in the text).

Figure 3: (A) schematic of the simulated pigeon model with two translational
degrees of freedom ($x-z$) and one rotational degree of freedom in pitch for
the body angle ($\psi$). In addition to producing flapping and
pronoation/supination motions, the model can modulate its wingspan during the
wingbeat cycles. (B) predicted wingbeat frequency of the pigeon during its
steady forward-flight over a range of flight velocities varying from 6 m/s to
20 m/s. Red dots represent the wingbeat frequencies evaluated using the quasi-
steady aerodynamic formulations of Dickinson et al. (1999). Blue dots
designate the predicted wingbeat frequencies using the proposed aerodynamic
formulas, revealing an improved agreement with the previously-reported
experimental data on pigeons (green dots) by Tobalske and Dial (1996), in
particular for high forward velocities (i.e. $U>$ 10 m/s).