A concrete adaptation project/programme is defned as a set of activities aimed at addressing the adverse impacts of and risks posed by climate change. The activities shall aim at producing visible and tangible results on the ground by reducing vulnerability and increasing the adaptive capacity of human and natural systems to respond to the impacts of climate change, including climate variability*.*

A CCA project's evaluability will be partly determined by how well a project's evidence base and baseline describe the starting position of the targeted natural systems (e.g., forestry coverage, biodiversity, soil characteristics), the starting position of the targeted human systems (e.g., agricultural practices, economic incentives, government policy, institutions), and the current ways in which the two systems interact (e.g., agricultural practice reducing biodiversity, economic incentives accelerating deforestation).

#### **Findings**

With regard to additionality, the Adaptation Fund's proposal templates ensured that projects, in the main, presented clear, well-referenced descriptions of the preintervention evidence base. This requirement was principally used by Fund applicants to justify the case for a project, but it also served to strengthen evaluability: The evidence base and site-specifc context inherently provided a basis against which project progress and results could be evaluated.

However, the assessment also found that the depth and quality of the preintervention evidence base varied according to whether evidence related to natural systems or to human systems. The use of detailed climatic and environmental baseline data (i.e., natural systems) was especially strong within most proposals. Conversely, proposals devoted less attention to describing and evidencing the nonenvironmental context within which an intervention was to operate, including the human and institutional aspects of a project. Beginning at the design/proposal stage, evaluability tended to be stronger for work relating to natural systems, less so for work relating to human systems (and, by extension, the linkages and interactions between natural and human systems).

While considering how the preintervention evidence base and baselines refected human and natural systems, the evaluability assessment also reviewed the extent to which—once projects were under implementation—monitoring approaches (including results frameworks) measured change across both systems, and the interdependencies between them. Although the majority of projects did support some degree of measurement of both human and natural systems, direct monitoring approaches were very heavily geared toward only measuring change within human systems (e.g., agricultural infrastructure, institutional and individual capacities, legislation). Indeed, many projects that measured some aspect of natural systems did so through only one relatively high-level indicator (e.g., area of land restored). Moreover, only a handful of projects had results frameworks in place that would be capable of measuring both systems and the interdependencies between them. The comparatively strong baseline understanding of natural systems was often not being expanded (or even followed up) during implementation. Perhaps this is because accessing historical natural data (i.e., climate, environment, biodiversity) is comparatively easy, so a strong evidence base can be developed. Conversely, accessing historical human data could be more diffcult (and such data may not be available), thus developing a detailed pre-implementation evidence base could be quite resource intensive.

Aside from the differing treatments of human and natural systems, the fndings also illustrate that ensuring project evaluability is an ongoing process. Evaluability can't be achieved through project design alone: Even where evaluability appears strong at the pre-implementation stage, that level of evaluability has to be maintained throughout project delivery by, for example, designing and applying processes capable of gathering the breadth and quality of data necessary to measure results across both human and natural systems.

#### **Resources Allocated to MEL: Direct vs. Indirect**

#### **Relevance to Evaluability**

Project evaluability is partly dependent on suffcient institutional and/or fnancial resources being allocated toward MEL activity (with the defnition of what comprises "suffcient" resources dependent on the nature of each project and its MEL strategy; see Table 5). To support the assessment of resource adequacy, we identifed the level of fnancial resources allocated toward MEL for every project, recording two fgures:

#### **Adaptation-Specifc Evaluability Considerations**

Many CCA interventions include—and are sometimes focused exclusively on—activity that

**Table 5** Guidance for assessing quality of MEL resources

can be considered indirect MEL. Examples of such activities are development of new climate monitoring approaches, consolidation of historical environmental data, research activities, and capacity development on the monitoring and interpretation of data. Project evaluability can be strengthened where these indirect activities are formally linked with the intervention's MEL strategy, in turn ensuring that project MEL (and evaluability) benefts from the widest possible range of data and resources.

#### **Findings**

During assessment of this criterion, our most notable fnding was that the proportion of indirect fnancial resources associated with MEL-relevant activity was far larger than the proportion of resources directly allocated to MEL. Frequently, indirect MEL activity was a core component of projects, such as in projects focused almost exclusively on developing sub-national climatic monitoring infrastructure. Many projects also were strengthening institutions, capacities, and infrastructure required for longer term (postproject) monitoring and measurement of CCArelevant indicators and results.

However, such obviously MEL-relevant activities were not recognized or presented as such by projects. This disconnect could have been due to project partners not conceptualizing the activity as MEL, or the common perception of MEL as a purely accountability-focused exercise, rather than as a tool that can also support learning and knowledge generation. Regardless of the reasons for the disconnect, where projects did not make the link between their direct MEL work and those indirect activities with clear relevance and value for MEL, the data and learning generated through indirect activities may have provided only a limited contribution or none at all to the project's formal MEL. This could have reduced the potential and strength of a project's evaluability.

Another notable fnding was that the data collected through indirect MEL activities tended to be more focused on natural than on human systems. While some projects were actually gathering extensive data and/or developing infrastructure for the measurement of natural systems, this data was not being used to bridge the above-noted gap whereby formal, direct project MEL placed more (and sometimes exclusive) emphasis on the monitoring of human systems. Thinking beyond individual projects, we also see a risk that data could be overlooked by future research and meta-evaluations if not identifed as being MEL relevant. In turn, this could reduce the potential contribution and value of data to longer-term learning and knowledge generation.

From an evaluability perspective, these fndings highlight the importance of looking beyond a project's self-identifed direct MEL activity, particularly where those projects are working to develop monitoring capacities and infrastructure. Moreover, the fndings also demonstrate how the actual process of evaluability assessment can help to identify and uncover opportunities for strengthening not just evaluability, but a project's MEL more broadly.

#### **Potential for Postcompletion Evaluation**

#### **Relevance to Evaluability**

Although still a relatively underdeveloped approach, postcompletion evaluations are increasingly deployed as a tool to measure the longer term impact of interventions. However, for an evaluation to be undertaken several years after a project's closure, adequate resources must be available and, more important, suffcient foundations must be in place for an evaluation to be even plausible (see Table 6). The prospects for long(er)-term evaluability could be improved by, for example, a project having indicators that can be accurately measured over longer time horizons, a project logic model that extends beyond

implementation, and the likely longer term availability of institutions and individuals that participated in the original intervention.

## **Adaptation-Specifc Evaluability Considerations**

The kind of results and changes targeted by CCA interventions are mostly longer term in nature, only identifable and measurable well after a project has been implemented. Consequently, the justifcation for postcompletion evaluation is particularly strong in the adaptation arena. Therefore, considering the extent to which foundations are in place for longer-term evaluability—and postcompletion evaluation—is important for any assessment of an adaptation project's evaluability.

#### **Findings**

The assessment frst sought to identify whether projects had formal plans in place for postcompletion evaluation. Only 3% of projects confrmed formal plans for postcompletion evaluations, with even the concept of postcompletion evaluation rarely being mentioned across the portfolio. Plans for postcompletion monitoring were far more prevalent, with several projects including components whose entire purpose was to establish organizational structures, systems, and capacities for longer-term (indefnite) monitoring of factors such as local meteorological data, water levels, or land use. However, the funding and institutional arrangements for these longer term systems were infrequently specifed. All of these longer term monitoring systems focused on only one aspect of a project; we identifed no examples of projects that planned broader longer term monitoring through approaches such as continued, wideranging monitoring against their original results framework.

The limited examples of longer term MEL are unsurprising; even within the MEL sector the concept of longer term MEL is still relatively new. Consequently, the assessment also considered the extent to which projects had in place certain foundations that at least strengthened the potential for postcompletion MEL, such as logic models that established pathways and results beyond the project's lifetime, project indicators that could plausibly be measured 5 years after project completion, indicators based on preexisting national data sets that are likely to be maintained over the longer term, and clear descriptions of postcompletion institutional ownership of the project's outcomes. Some instances of projects had promising building blocks in place, with several projects benefting from clear exit strategies and descriptions of postcompletion ownership, whether institutional, community, or individual. A handful of projects also aligned their results frameworks with preexisting national results frameworks and data sets, with the explicit rationale being to support longer term monitoring of project results. However, these were exceptions; the broader portfolio was characterized by generally weak potential for postcompletion MEL.

Again, these fndings are not surprising given the universally low application of longer term MEL and postcompletion evaluation. However, the process of evaluability assessment can help pinpoint gaps and opportunities to at least strengthen the foundations for any potential postcompletion evaluation.

#### **Refections on the EA Tool Development and Implementation**

We undertook a refective and consultative process to develop a tool specifc to CCA programming and its particular challenges, aiming to fulfll our objective of an impartial and accurate assessment of the Adaptation Fund portfolio's evaluability. The tool was informed by a set of principles taking into account the Fund's current and historical approaches to evaluation and an understanding of our evaluand, CCA programs. We adopted the OECD defnition of evaluability and set about a process akin to both Wholey's (1979) eight steps and Trevisan and Walser's (2015) four-stage approach: defning the evaluand, literature review, framework development, consultation and piloting, implementation, and presentation of results.

The process of our EA tool development was consultative, involving peer review by an evaluation consultant and by AF-TERG's advisory board of evaluation and climate change programming experts. We developed a 5-point scale with criteria of equal weighting. We piloted the tool, refected on its results, further revised, and piloted again. Two colleagues undertook the review, aligning their interpretation of criteria and scoring. Although we did not determine an inter-rater reliability score, we systematically compared assessments to calibrate judgment.

The results of the EA identifed challenges that are well documented about CCA programming (Fisher et al., 2015), as discussed above, and resonated with Adaptation Fund stakeholders, serving to promote discussion and decision making about the Fund's evaluation policy and processes for funding and partnership. A consideration for this frst phase of the EA was the sole reliance on written documentation and the lack of opportunity to engage on site with partner agencies. In the second phase of our two-phased approach of assessment and verifcation, we will continue to collect data on site with partners and refect on the tool's merit and utility as part of our effort to engage in formative meta-evaluation. With this positive EA experience and as we fnd additional use for the tool going forward, we may also meta-evaluate following feld verifcation, using relevant criteria found in the Program Evaluation Standards (Yarbrough et al., 2011).

#### **Conclusions**

The framework that was developed and applied for the Adaptation Fund EA proved to be a useful tool for understanding evaluability strengths and gaps both within individual projects and across the Fund's whole portfolio. Moreover, the framework supported the identifcation of CCAspecifc evaluability opportunities and challenges. Results of the assessment supported the three challenges related to climate change activities identifed by Fisher et al. (2015): assessing attribution, creating baselines, and monitoring over longer time horizons.

The assessment confrmed the diffculty in identifying the additionality of CCA programs because of the complexity of such interventions. Clarifying the project logic through an EA helps conceptualize the project results in terms of contribution or attribution in a longer term perspective.

The study also highlighted the diffculty in setting baselines to measure the results of CCA in a comprehensive way, which would encompass impacts on both natural and human systems. It showed that, beyond the usual suspect of unavailability of nature-focused and human-focused data, the failure to integrate this data from both systems in the MEL system was the source of gaps in knowledge during project implementation.

Finally, the study highlighted the need to plan for and allocate resources for longer term M&E to be able to measure the delayed impacts of CCA interventions. The EA process can help pinpoint gaps and opportunities to at least strengthen the foundations for any potential postcompletion evaluation. The EA was incredibly useful in assessing the extent of the structures, processes, and resources in place capable of supporting credible M&E. Although we conducted this assessment retroactively, prospective use of the tool could inform and ensure that projects have the evaluability structures necessary at project design. The study of evaluability in practice, which verifes whether the evaluability is maintained throughout project delivery, could also be complemented with feld verifcation to inform gaps in the processes related to project implementation.

The framework also offered a structured process for systematically thinking through what constitutes sound MEL more broadly. Indeed, a key conclusion of the assessment was that the strength and quality of a project's overall approach to MEL is a major determinant of a project's evaluability: If a project gets its MEL "right," the project is highly likely to also have strong evaluability. This suggests that, where improved evaluability is sought, focusing efforts specifcally on strengthening evaluability may not be effcient or even necessary; instead, improvements to overall MEL strategy and processes should inherently deliver improvements to the quality of evaluability.

#### **References**

(GCF) Independent Evaluation Unit (IEU). https:// ieu.greenclimate.fund/documents/977793/985626/ Working\_Paper\_\_Becoming\_bigger\_\_better\_\_ smarter\_-\_A\_summary\_of\_the\_evaluability\_of\_ GCF\_proposals.pdf

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Evaluating Transformational Adaptation in Smallholder Farming: Insights from an Evidence Review**

Laura Silici, Jerry Knox, Andy Rowe, and Suppiramaniam Nanthikesan

#### **Abstract**

The literature on smallholder farming and climate change adaptation (CCA) has predominantly investigated the barriers to and determinants of farmer uptake of adaptation interventions. Although useful, this evidence fails to highlight the changes or persistence of adaptation responses over time. Studies usually adopt a narrow focus on incremental actions that provide limited insights into transformative adaptation pathways and how fundamental shifts in policy can address the root causes of vulnerability across different sectors and dimensions. Drawing on an evidence synthesis commissioned by the International Fund for Agricultural Development's Independent Offce of Evaluation, this chapter outlines how lessons from CCA interventions can be transferred via three learning domains that are essential for transformational change:

L. Silici (\*) Independent Consultant, Rome, Italy

J. Knox Water Science Institute at Cranfeld University, Cranfeld, UK e-mail<EMAIL_ADDRESS>
A. Rowe ARCeconomics, Maple Bay, BC, Canada

S. Nanthikesan Independent Offce of Evaluation of the International Fund for Agricultural Development, Rome, Italy

scaling-up (in its multiple forms), knowledge management, and the human-environment nexus. We discuss the implications of our fndings on monitoring, evaluation, and learning, highlighting the challenges that evaluators may face in capturing (a) the persistence or durability of transformational pathways, (b) the complexity of "super-wicked" problems, and (c) the relevance of context-dependent dynamics, within a landscape setting. We also address the contribution of evidence reviews to contemporary debates around development policy linked to climate change and agriculture, and the implications and value of such reviews to provide independent scientifc rigor and robustness to conventional programmatic evaluations.

#### **Introduction**

Smallholder agriculture represents 75% of the world's farms (Lowder et al., 2016) and 80% of the source of food consumed in the developing world (International Fund for Agricultural Development [IFAD], 2020). Yet smallholder farmers constitute more than half of the world's undernourished people; they inhabit some of the most vulnerable and marginal landscapes, and many lack secure land tenure and water rights. These factors further exacerbate their exposure to

J. I. Uitto, G. Batra (eds.), *Transformational Change for People and the Planet*, Sustainable Development Goals Series, https://doi.org/10.1007/978-3-030-78853-7\_13

<sup>©</sup> The Author(s) 2022 187

climate change (IFAD, 2020), alongside other sources of vulnerability such as population growth and land fragmentation. Indeed, smallholder agriculture is disproportionately threatened by unpredictable weather patterns, with the impacts of extreme events including foods, droughts, and heat waves having profound impli-

tion, especially for rural communities dependent on rainfed agriculture (United Nations Environment Programme [UNEP], 2018). Although extensive information is readily available on the projected agricultural impacts of

cations on both food security and poverty reduc-

climate change and on adaptation measures that could help minimize those impacts, assessments that specifcally address the vulnerability of smallholder farmers to climate change are very limited (Donatti et al., 2019). Thus, there is a need to better understand how smallholder farmers perceive the risks of climate change, the factors that infuence their decisions to adapt, and what adaptation strategies have been practiced and why (Belay et al., 2017).

This chapter draws on the fndings of an evidence synthesis commissioned by the IFAD Independent Offce of Evaluation (IOE) to inform a thematic evaluation of IFAD's support for smallholder farmers' adaptation to climate change*.* Evidence reviews aim to provide a transparent and robust assessment of what is known (and not known) in the literature regarding a specifc topic, by adopting a systematic methodology to search, screen, and critically appraise a database of scholarly articles, including peerreviewed and grey literature.

Executing the review involved several discrete stages following the well-established guidelines of the Collaboration for Environmental Evidence (CEE; Pullin et al., 2018), ensuring that the outcomes met quality standards with respect to robustness, transparency, and repeatability. The exercise provided a valuable narrative synthesis of the evidence produced over the last 10+ years on smallholder adaptation to climate change, focusing on how lessons from climate change adaptation (CCA) in smallholder farming could be transferred within three learning domains: (a) scaling-up, (b) knowledge management, and (c) the nexus interactions between the human and ecosystems. We briefy summarize the key fndings from the review and discuss their policy and practice implications with respect to monitoring, evaluation, and learning (MEL) under the framework of transformational change, with a focus on the challenges that MEL faces. Specifcally, these included addressing (a) the persistence, or durability, of transformational pathways; (b) the complexity of "super-wicked" problems (Levin et al., 2012) that lead to trade-offs across different intervention dimensions and respective goals; and (c) the relevance of context-dependent dynamics, within a landscape setting.

In smallholder farming, transformational adaptation entails adaptive strategies that address the different root causes of vulnerability, internalize aspects of gender, racial, and intergenerational equity (i.e., social and environmental aspects), and enable changes in the wider food system, including solutions at any point in the value chain and diversifcation into off-farm sectors. Transformative solutions exert infuence beyond the boundaries of specifc interventions, possibly reaching different sectors (outside agriculture) and dimensions such as gender roles and behavioral changes. More generally, they imply a conceptual shift toward transformative changes beyond incremental adaptive actions. Indeed, adaptation is not a discrete measure, but rather a dynamic and iterative process due to a changing context where the climate is one of many risks (Vermeulen et al., 2015; Wise et al., 2014). The pace of climate variability is also increasing alongside other sources of vulnerability such as population growth and land fragmentation and the emergence of alternative responses (off-farm employment, markets).

Successful implementation and expansion of transformation pathways in smallholder agriculture are underpinned by three mutually interacting factors. First, CCA solutions need to integrate the multiple nexuses between the socioeconomic sphere and the surrounding ecosystem; second, transformation entails scaling-up, in the intermediate or longer term; and third, an effective and usable climate adaptation knowledge base is needed to understand the nexus better in order to promote scaling-up.

#### **Methodology**

This review was originally conceived as a rapid evidence assessment (REA), an approach designed to deliver a relatively quick synthesis of evidence of existing research on a defned topic. The REA methodology involves defning an explicit research question(s) and following a strict procedure that is not dissimilar to the development of a protocol for conducting a systematic review. An REA is rigorous and has an approach that is tractable and transparent but generally lacks the depth of analysis often associated with a full systematic review. However, REAs are widely used for informing policy decision making. In this study, the overarching question that guided the review was: What interventions have been successful in building smallholders' adaptive capacity and responses to climate change and how have these been effectively transferred as learning outcomes in the three key dimensions of scaling-up, knowledge management, and ecosystem-human interactions? Our review followed the approach developed by CEE (Pullin et al., 2018) of frst developing a protocol that guides the execution and describes how the literature searches will be conducted, including the bibliographic sources and search strings, the data extraction methods, quality assessment, and evidence synthesis. The review was constrained by a set of inclusion and exclusion criteria and the research question defned above.

We used a range of bibliographic databases including Scopus, Web of Science, and Science Direct, and web search engines (Google Scholar) and institutional websites for various international development organizations, think tanks, and research institutes. Following trial searches, we selected the search string "smallholder, AND agric\* AND climat\* AND change, AND adapt\*". Collectively, the searches returned 806 sources of evidence. After a thorough screening of the titles and abstracts using a set of inclusion criteria, we selected 132 sources for full-text reading; of these, we then reviewed and synthesized 91. For a detailed explanation of the review methodology with information on the resources used, see Silici et al. (in press).

#### **Key Messages Emerging from the Evidence Synthesis**

#### **Scaling Up Transformative Adaptation Pathways**

Scaling up a process or an initiative may take multiple forms and does not imply only bringing an intervention to scale (to more people, larger areas) or adapting it to similar conditions in different locations (horizontal scaling-up). Scaling up a project can also relate to moving it forward into a more developed, complex phase, possibly including new components, confgurations, and stakeholders (diagonal scaling-up). It can also consist of mainstreaming a certain approach into policy, leveraging and catalyzing policy and/or institutional change (vertical scaling-up; Neufeldt et al., 2015). In the latter case, scaling-up processes imply more than just physical or technical dimensions, but also social scaling-up (increasing social inclusiveness) and conceptual scalingup in terms of moving beyond participation to embedding empowerment in the entire development process (Binswanger-Mkhize et al., 2009). With respect to adaptation, social and conceptual dimensions of scaling-up are particularly important to pursue genuinely transformative pathways.

The academic literature on CCA and smallholder farming has predominantly investigated the determinants of (or barriers to) farmer adaptation at a given point in time and for a defned location (Etana et al., 2020). In our review, about one fourth of the publications were quantitative studies addressing barriers and determinants of farmers' adoption of CCA. This evidence base provided valuable insights on the necessary preconditions and the enabling factors (or capacities) that determine smallholders' decisions to act in the face of climate change. Among the former are factors that underpin a farmer's decision to adapt: knowledge regarding climate change, farming and natural resource management, weather information services, and past experience of extreme climate events; collectively, these contribute toward building farmers' consciousness of the need to adapt. However, awareness alone does not trigger change. A series of enabling factors of social, economic, environmental, and institutional natures ultimately determine a smallholder's actual capacity and motivation to take action.

Planned adaptation strategies should thus ensure that the underlying determinants of adoption, such as access to knowledge and information, exist alongside other relevant enabling factors, including endowment with productive assets, human capital (education and skills), and institutional support (e.g., farmer groups and collective action; Atsiaya et al., 2019; Bedeke et al., 2019). Profling different vulnerability categories is also extremely important because smallholders adopt different strategies according to their degree of sensitivity and adaptive capacity beyond their actual exposure to climate change. More well-off farmers, for instance, rely on a stronger economic buffer and usually have stronger connections with formal and informal institutions, which in turn affect their risk perception and adaptive decisions. Policy and institutions must build on this understanding to meet the particular needs and priorities for distinct smallholder groups, with the most notable disparities found in income and gender (Ruijs et al., 2011).

While this branch of research is very useful to understand farmers' decision making and how to support capacity building, it fails to show the changes or persistence of adaptation behavior over time (Etana et al., 2020). Moreover, because of a largely quantitative, linear approach and a predominant interest in mainstreaming technological solutions, these studies have maintained a rather narrow focus on incremental actions and adjustment approaches within the agricultural sector. The outcome is a limited perspective on agriculture-related solutions and a failure to detect innovative adaptation practices that incorporate local wisdom and creativity.

Other studies have adopted a more dynamic, longer term perspective in describing adaptation pathways. Here the concept of a pathway focuses more on an iterative process of decision making, rather than on the outcome, emphasizing the adaptive nature of the decision process itself in the face of high uncertainty and intertemporal complexity, rather than the achievement of a given objective (Wise et al., 2014). Some of the evidence also stresses the need to address the social roots of vulnerability and the necessity for political–economic change to achieve "transformative adaptation" (Bassett & Fogelman, 2013).

Several key points emerged regarding smallholder farming and transformational adaptation patterns. First, evidence of path dependence corroborates the need to provide fnancial and institutional support to overcome the behavioral and economic barriers of less endowed households, who could otherwise become locked in vicious cycles of inaction, thereby undermining overall community ability to adapt over time (Etana et al., 2020). In fact, we found that the sustainability of outcomes is better achieved when their distribution is more equitable and inclusive; this implies that all socioeconomic groups and different types of smallholders can respond to risk and vulnerability according to their adaptive capacity. In terms of planning, the review confrmed that no one blueprint or strategy works across all contexts; instead, adaptation should be perceived as a continuum of approaches, ranging from activities that aim to address the different drivers of vulnerability to measures explicitly targeting climate change impacts, including not only farming but also water and soil conservation and off-farm diversifcation. Planned responses should follow iterative processes, should set out all possible trade-offs (economic, temporal) between alternative strategies, and should be open to assess and envision all possible futures (Vermeulen et al., 2015, 2018). For example, the most viable pathway for some farmers may be to exit agriculture altogether, which itself requires careful management and planning of consequent rural transitions (Stringer et al., 2020).

Although identifying generic solutions is not possible, sustained adaptation and scaling-up do have some recurrent features. These include the need for integrated, multisector interventions; adopting participatory approaches in planning, implementation, and dissemination; and fostering knowledge exchange, peer learning and cocreation of knowledge. Participation and ownership of adaptive solutions are extremely important to ensure equitable scaling-up processes, because inter- and intra-household differences mean individuals react differently to multiple stressors. Institutional aspects are also critically important features in creating successful adaptation pathways. Sustainable adaptation only happens if farmers and their community members have the capacity to organize themselves. Institutional learning, or institutional capacity building, doesn't necessarily entail formalized structures (although linking with formal organizations does confer some advantages), but it is needed to foster adaptation through collective action and social learning. Institutional support, collective action, and participation are also critical in bridging local and scientifc knowledge (Arouna & Akpa, 2019; Stringer et al., 2020; Vermeulen et al., 2018).

#### **Transformative Knowledge Management**

Access to knowledge constitutes one of the most important determinants of smallholders' decisions to respond to risk and is a critical element in building adaptive capacity. Creating the knowledge base that underpins its access and use, however, faces two challenges: Climate solutions have a shelf life, and they are very context specifc. Therefore, the way knowledge regarding climate change and variability is produced, transferred, and exchanged is extremely relevant to securing equitable and inclusive scaling-up pathways (Popoola et al., 2020; Roncoli et al., 2010).

Building adaptive capacity does not rely solely on external, scientifc knowledge. Knowledge embedded in farmers' experience and tradition is also critically important in raising awareness of climate risks and selecting appropriate responses. In fact, autonomous adaptation is based on farmer perception of climate change and variability. However, autonomous adaptations may be limited in scope and may be not entirely effective over the long run. They can even lead to maladaptation because farmers typically react to the actual threat experienced and do not consider likely future threats (Akinyemi, 2017; Makate, 2019). Knowledge based on local practices also may not be suffcient to prompt more transformative actions that take account of intergenerational equity (Derbile et al., 2016), or to embark into more risky activities (Etana et al., 2020). Bridging local and external knowledge helps broaden farmers' knowledge base to include more forward-looking considerations (Makate, 2019; Shackleton et al., 2015). On the other hand, having a supply of scientifc or external information does not imply that it is passed on, understood, and accepted; this depends on how it is communicated and, importantly, if it matches smallholders' needs.

School education, vocational training, and agricultural extension and advisory services (where available) are important enablers of adaptive capacity because farmers with more education and skills are able to understand, trust, and assimilate the information they receive and utilize it to support their needs (Guido et al., 2018; Henriksson et al., 2020). Joining formal groups such as producer organizations, cooperatives, or outgrower programs can allow farmers to gain access to technical skills, market information, weather forecasts, and other relevant information on credit, laws, and policies (Abass et al., 2018). These mechanisms for transferring knowledge help foster adaptive capacity and peer support, but they may not be effective for all smallholders or may even be exclusive of some groups if they have too narrow a technical focus and are conceived as a one-way, top-down transfer. The actual way in which knowledge is transferred to smallholders and how it then interacts with local knowledge is critically important.

Knowledge can be produced and circulated via more participatory ways. Partnerships and social learning (deep understanding and assimilation of concepts through social interaction) were identifed in the literature as promising ways to link science, policy, and practice to tackle multiple and related challenges of agricultural development, food security, and climate change adaptation. The evidence showed that beyond empowering specifc groups, inclusive learning processes (such as pro-poor research and gender transformative approaches) accelerate and improve development outcomes for everyone (Shaw & Kristjanson, 2014). Both formal groups and informal collective action, such as self-help neighborhood networks, work-sharing groups, and other community-based organizations, can foster synergies for social learning and capacity building. However, registered community-based organizations seem to be more signifcantly associated with adaptation (Khanal et al., 2019). Synergies between informal and formal structures also result in more effective organizations because they promote institutional learning, drive innovation and knowledge from external sources, and enlarge participation to more heterogeneous groups (Hulke & Revilla Diez, 2020).

Learning platforms based on participatory action research (PAR) that bring together different actors have been shown to be particularly effective in supporting adaptation strategies. They rely on social learning and collective action and help external actors (researchers, policymakers, and development institutions) assess community-based perspectives and existing adaptation options, using these as a starting point for further action (Asociacion Andes, 2016; Wekesa et al., 2017). This type of research can also help in situations where climate change is not the only externality but one of multiple sources of vulnerability, thereby providing a deeper understanding of the contextual barriers to adaptation. By fostering inclusion, social learning eases access to knowledge by women and people in other vulnerable categories who are not always targeted by institutional information sources in agriculture (Kerr et al., 2018; Mapfumo et al., 2013).

Several examples in the literature reported successful aspects of social learning through collaborative approaches that were broadly formalized into learning platforms or alliances. Farmer feld schools (FFS) may also be considered as a relevant learning platform that integrates adaptation at different levels and scales, albeit the degree of participation in needs assessment and design of training modules is not as high as in PAR (Chandra et al., 2017; de Sousa et al., 2018). For learning platforms to be effective in promoting social learning and collective action in adaptation, strong facilitation is needed that minimizes power imbalances and builds trust among different stakeholders. One or more formal institutions should act as a broker to liaise between different interest groups and to mediate and coordinate partnerships. External actors may also have a role in creating or enhancing the demand for knowledge, thereby helping to overcome the limitations associated with relying only on local knowledge.

Researchers taking part in learning platform have to mind-shift into a learner role and need a deep understanding of local cultural norms and institutions to ensure inclusion of all views (e.g., to separate women and men in consultations, and to look beyond existing community groups that may include only stronger members). Time is also needed to build trust and a safe and comfortable interface environment through appropriate communication before the inception phase. Finally, economic incentives may also be needed to include marginalized categories who may not have time to engage in consultation activities (Shaw & Kristjanson, 2014). To work well, PAR and learning platforms also need a supportive institutional environment, refected in economic support but also in an adequate policy and legislative framework (Chandra et al., 2017).

#### **Ecosystem-Based Adaptation and Landscape Approaches in Smallholder Farming**

Individual smallholder farms are typically small, but their collective share of contributions to the ecosystem burden cannot be ignored. Ecosystem goods and services are the backbone of farmers' agricultural economy. Common natural pool resources such as forests and wetlands provide complementary sources of livelihoods and, more generally, a healthy and productive landscape encourages households to remain in rural areas, slowing down outward migration from the countryside while increasing income opportunities and sustaining local adaptation (Arouna & Akpa, 2019; Food and Agriculture Organization of the United Nations, 2014). The interactions between farming and the environment also have negative connotations, with farmers often exposed to natural disasters and weather extremes. On the other hand, even if smallholder agriculture does not contribute to water and air pollution as much as large-scale intensive farming does, anthropogenic activities such as grazing, encroachment, and deforestation may severely undermine the natural resource base and ultimately smallholders' livelihoods. When taken collectively, these small units can still pose a heavy burden on the ecosystem; at the same time, just due to their large number and small scale, they can play an innovative role in forging benefcial ways of interaction between agriculture and the environment.

Despite these important connections, the review identifed relatively few studies that focused explicitly on the nexus between humans and their surrounding ecosystems, or that used an environmental lens to critique adaptation within smallholder agriculture. These fndings may well be infuenced by the inclusion criteria defned in the review, but the fragmented evidence on this topic nevertheless refects a reality where policy planning in agriculture, environment, and climate change still happen in silos, with limited exchanges between disciplines (El Chami et al., 2020). This is a key concern and should be recognized as a strategic development priority and considered explicitly in program evaluations.

Other areas of intervention can support stronger nexus integration between human and ecological systems in adaptation planning, implementation, and assessment. First, several studies stressed the need for a reframing of smallholder farming and agriculture more generally, and for an integrated system alongside natural resource management, energy, and climate change. Agricultural sustainability is a threedimensional model that requires overcoming disciplinary boundaries. A transdisciplinary approach, which represents a step forward for interdisciplinarity with full integration of different disciplines, is much needed (El Chami et al., 2020). Such an approach should be pursued through policies that promote circular models of the economy and cyclical, rather than linear, growth-oriented systems. The economic analysis undertaken to inform such policies should refect this complexity. That is, analysts should be able to value the costs and benefts beyond the conventional measures of productivity and effciency to include, for instance, the economic value of ecosystem goods and services and GHG mitigation, and thereby identify suboptimal equilibria where the different economic, social, and environmental objectives are consistently achieved (Reid et al., 2013).

Beyond stressing the importance of assessing adaptation responses through an integrated, cross-sectoral approach, the evidence review also highlighted the relevance of local and landscape dimensions. The ecological aspects of adaptation responses and land use changes are closely linked to local socioeconomic conditions: different locations, albeit close to each other, may take different adaptation paths and these should be contextualized within the agro-ecological landscapes in which they reside (López et al. 2020; Marquardt et al., 2020; Newsham & Thomas, 2009). Thus, a holistic approach to adaptation in smallholder agriculture should cut across not only sectors but also across multiple levels of intervention, from the household level through the community, right up to the landscape level.

In this context, ecosystem-based adaptation (EBA) has been proposed as an effective tool to achieve such an integrated vision (Abdelmagied & Mpheshea, 2020; El Chami et al., 2020; Vignola et al., 2015). Agro-forestry, for instance, has shown high potential to enhance smallholders' adaptive capacity (Lasco et al., 2014; Partey et al., 2018; Quandt, 2020). Other studies have highlighted that smallholders were aware of the links between the ecosystem and their economic activities, and many already use practices that could be classifed as EBA (Chain-Guadarrama et al., 2019; Shah et al., 2019). However, the evidence also calls for greater involvement of researchers and policymakers to address some key challenges. These include, for example, integrating the farmers' local knowledge with results from scientifc research and extension services to help manage and minimize the trade-offs at farm and landscape levels, such as competitive interactions between trees and crops, or pest infestations that can result from ecosystem stabilization (Lasco et al., 2014; Quandt, 2020). Scientifc and economic research could also help quantify the economic returns from ecosystem goods and services, such as pest and disease control or the role of trees as buffer strips. Economic incentives based on these analyses are much needed because EBA approaches constitute long-term strategies with generally low returns (and hence incentives) in the short term.

Our review also showed that institutional support to social networks and collective action was very important in smallholders' pursuing adaptation pathways that assimilate the nexus between human and ecological systems. Being part of a network helps farmers to coordinate collective action around common pool resources. Strong social networks, coupled with the presence of environmental champions, can help facilitate the spread of knowledge-intensive practices for adaptation, such as agro-ecology, and lead to higher environmental sustainability in smallholder farming (Nyantakyi-Frimpong et al., 2019; SaintVille et al., 2016).

Some authors advocated for combining EBA and community-based adaptation and mainstreaming them into large-scale planning. According to Reid (2016, p. 5) "good EBA should (but does not always) have a strong community focus." The participatory dimension of EBA is easily explained by the fact that many (although not all) natural resource and environmental practices transcend administrative boundaries and go beyond feld limits, requiring collective agreement and action. Indeed "good" community-based adaptation often considers ecosystems and ecosystem services to a strong degree. Local adaptation activities in the feld tend to combine the two approaches and compensate for the shortcomings of top-down initiatives based on hard infrastructure (Reid, 2016). However, to achieve fruitful integration of these two approaches at scale, an important component is addressing the policy context to assess whether governance is actually inclusive for all community members, whether there is devolution of rights and responsibilities to local institutions, and how this works in practice. Incentives are needed both for individuals and community groups. These could be distributed via external funding for public projects and initiatives, as fnancing tools such as offering microcredit or revolving funds, or as payments for ecosystem services (PES) and direct compensation (Reid, 2016).

#### **Policy Shortcomings**

Policymakers play a key role in supporting smallholders' adaptation to climate change. Policies at national and international levels defne and infuence adaptation pathways, frst and foremost by setting the vision that underlies the legal and the institutional frameworks. High-level, public intervention is needed to ensure that autonomous adaptations (that may be guided by short-term perspectives) and planned adaptations (more forward looking to intergenerational sustainability) work consistently toward virtuous patterns of transformative adaptation. Evidence showed that path dependence in farmers' choices to uptake adaptation calls for higher level coordination and investments across three intervention scales household, community, and landscape. Institutional and fnancial support is necessary to preserve local adaptation efforts (e.g., through infrastructure) and to secure an enabling policy and legal environment (e.g., land tenure, access rights to natural resources). External institutions such as government and development actors should also provide economic incentives to ensure that environmental considerations are fully integrated into human responses (such as compensation for investments in agroforestry that don't deliver immediate returns) and, more generally, to reconcile the unavoidable trade-offs between competing objectives in the economic, social, and environmental domains.

The narrative that emerged from the review highlights that policies would need to undergo some fundamental shifts to support adaptation pathways that are genuinely inclusive and sustainable. First, adaptation strategies should address the root causes of vulnerability, which in the farming sector may lie in complex situations where climate change is just one of multiple stressors. This implies pursuing transformative strategies that have a holistic, intersectoral approach, keeping open all the options for response (including, for example, exiting agriculture). Within the agricultural sector, this means welcoming system-oriented solutions that go beyond technological interventions at plot and farm level; in fact, such interventions may prove to be only coping strategies and, if not well targeted, might also generate or exacerbate inequalities at the expense of poorer, less knowledgeable, or less connected farmers. Indeed, addressing the root causes of vulnerability also requires assessing and, if necessary, rebalancing power relationships and politics of knowledge at the community level to avoid asymmetries in access to knowledge and information, the exclusion of most vulnerable groups, and elite capture of benefts. Finally, policy plays a fundamental role in reorienting science and research toward more participative, multistakeholder, and transdisciplinary approaches, according to the principles discussed above.

However, the evidence presents a largely empirical approach and we often perceived a seeming disconnection between policy, research, and practice. Although extensive empirical literature exists on the impacts of climate change, policymakers do not systematically use the evidence available (Khan & Akhtar, 2015). A further problem is that while many studies identify barriers to adaptation in farming, the appreciation of their interactions and impacts remains scarce. At the policy level, perceptions of climate change mainly as an environmental issue rather than a broad development issue results in it often being sidelined, constituting a barrier to planning and implementing effective actions. Other political barriers include a shortfall of funding and insuffcient coordination (Shackleton et al., 2015). Poor interaction between different disciplines, both in research and in policy, was identifed as a further gap. Although recognition is growing that smallholder farming and the surrounding ecosystem are tightly coupled, most climate change interventions in agriculture have still focused on farm-related technological solutions. A technocentric approach risks neglecting environmental and natural resource dimensions because it does not solve problems from a holistic perspective. It also risks exacerbating income and gender inequalities because access to assets and skills is often gender biased and generally interdicted to the most vulnerable.

Pursuing the idea of adaptive development may help overcome some of these limitations and promote a truly transdisciplinary approach where reducing societal inequality and injustice is a prerequisite for adaptation that works for everyone (Lemos et al., 2013). However, several fundamental barriers need to be overcome. At national levels and in international forums, an original disconnection exists between farmers, technical experts, and policymakers and coordination remains limited. True participation by smallholders in the decision-making process is scarce, with "participation" often reduced to consultation, and the devolution of power is weak. As a result, policies often do not refect smallholders' real needs and preferences, which risks reinforcing vicious circles of power imbalance and inequitable adaptation outcomes (Hameso, 2017; Nigussie et al., 2018; Sova et al., 2015; Taylor, 2018).

Further challenges lie in the diffculties of mainstreaming successful local adaptation solutions into large-scale planning; because climate impacts are often locally specifc, large-scale initiatives to support smallholder farmers must consider local priorities and integrate lessons from successful autonomous adaptation efforts. Scaling up climate adaptation into large-scale agricultural initiatives requires not only integration of lessons from community-based adaptation (i.e., learning local priorities, capacities, and lessons), but also the building of inclusive governance. Mainstreaming community-based adaptation, in particular, is more than scaling-up of specifc adaptation practices or knowledge; it is about mainstreaming institutional and organizational approaches that allow this knowledge to be generated (Wright et al., 2014). According to this vision, securing impact at scale in practice involves providing the lawful circumstances to embed local institutions into broader governance frameworks, to legally devolve rights and responsibilities to local groups and communities (especially with regard to natural resource management) and to reform land tenure and rights to access natural resources as required (Reid, 2016).

One major opportunity to bring social learning on adaptation to the national level is via existing extension and advisory services, but in many lower-middle income countries, extension services are limited and often biased to target relatively wealthier households (Wright et al., 2014). Another way to support effective knowledge sharing is through stakeholder platforms. Stakeholder consultations in policy and in research help consider the multiple and sometimes competing objectives among different sectors and interest groups, and the resulting trade-offs in the distribution of benefts and costs (Abegunde et al., 2019). Stakeholder-centered methods are increasingly used alongside other trade-off analysis methods oriented to better management, that is, with the aim to improve the relative situation for different groups, acknowledging that eliminating trade-offs and generating win-win solutions for all is not possible (Gusenbauer & Franks, 2019).

#### **Implications for Monitoring, Evaluation, and Learning**

Evidence assessment and continuous learning are critical to inform policy makers about what works for whom. In the context of the three learning outcomes of relevance to this study, monitoring, evaluation, and learning (MEL) should be able to capture the success of both autonomous and planned adaptation strategies in pursuing transformative pathways in smallholder farming. The outputs from the review highlighted a number of challenges for MEL in attaining this goal, and showed opportunities for innovation and crosslearning from different disciplines and methodological approaches.

One challenge lies in the need to evaluate the durability and the persistence over time of nonlinear development processes. The evidence in the literature on scaling-up and knowledge management showed that social equity remains a key factor of transformational patterns; on the other hand, the reiteration of inequitable starting conditions coupled with scarce opportunities for building adaptive capacity causes path dependence in farmer decision making, leading to technology lock-in situations and possibly maladaptation. Enabling countries and organizations to better assess and evaluate transformative adaptation thus requires an inclusive approach built on effective knowledge sharing and attention to the behavioral and institutional changes that address the needs of the most vulnerable (Anderson, 2011). Without understanding how vulnerability to climate change is produced and distributed, planned adaptations may inadvertently promote injustice and deepen inequality. Inequality of outcomes may be channeled through wealth, gender, and local relationships that infuence dynamics of power, knowledge fow, and uptake of certain technologies. Therefore, at a higher level, evaluation should seek to consider not only the outcomes of adaptation but also the processes behind them, to understand whose preferences and priorities have guided the decisions and the actions that have been implemented. At the operational level, a fundamental challenge to monitoring is that adaptive strategies also take time to implement and deliver impacts that can be monitored; key milestones are therefore needed to track actual changes over time.

Another related challenge for those engaged in evaluation is that complex, iterative, adaptive processes may often cut across different sectors beyond agriculture. MEL should be able to assess outcomes cutting across sectors (transdisciplinary) and take into account these relationships—and the trade-offs—across different intervention domains with different and sometimes competing goals (e.g., food security and conservation). The evidence showed that failing to fully acknowledge the interactions between the human and environmental systems, and continuing to treat agriculture separately from the environment, may undermine a full understanding of adaptation solutions both in agriculture and beyond. The ecological aspects of adaptation responses in agriculture, and more generally in land use changes, are closely linked to the social and economic dimensions of migration, labor, and shifting production patterns. These interlinkages are highly dependent on context and framed within the scope of landscapes that do not necessarily overlap with administrative boundaries and/or productive areas. These considerations call for MEL in agriculture to redefne both its focus (beyond the farming sector) and its scope (across traditional boundaries): Evaluators should go beyond the project or program scale and use the community and landscape dimensions as the scope for evaluating contextdependent dynamics. At a higher level, a conceptual shift is needed in what is conventionally valued as productive and effcient in agriculture in order to embed considerations of inter- and intragenerational equity (including ecological sustainability) into the assessment of adaptation outcomes.

Not only is a conceptual shift required, but changes and improvements in methodology are also necessary to assess and evaluate adaptation to climate change in agriculture. One operational challenge lies in the fact that adaptive strategies take time to bear outcomes that can be monitored, so evaluators need to identify and track key milestones toward the actual change. Another issue is that current M&E approaches mostly rely on purely quantitative methods, which are not suited to understanding complex processes and interactions or to including more nuanced perspectives on equity and vulnerability. More expertise is needed on evaluation applied to adaptation and on the combined application of quantitative and qualitative approaches (Anderson, 2011). Evaluators should also use innovative approaches in economics to value ecosystem goods and services; to assess trade-offs more systematically across different sectoral, spatial, and time scales; and to better evaluate social costs and benefts in the calculation of PES and other economic incentives for farmers.

Finally, improvements in governance are inherently diffcult to assess, and so too is understanding how benefts are distributed among different groups. Indeed, even agreeing on a defnition of successful adaptation may be diffcult because different interest groups will evaluate the same outcome differently. Although using participatory approaches and mutual learning is critical to overcome these issues, they create diffculty in producing impartial and defnitive assessments (Anderson, 2011). In this context, stakeholder platforms provide a promising tool (alongside other analytical methods) to encourage mutual learning, communication, and governance in adaptation. These platforms also may be useful to inform the development of innovative MEL approaches.

#### **Future Role of Evidence Reviews in Programmatic Evaluation**

Beyond providing valuable insights and new evidence on climate change adaptation in smallholder agriculture, this type of review can also feed into broader discussions on how evidence from a range of sources (including systematic and rapid evidence reviews) can be usefully integrated into thematic evaluations and what added value such syntheses can provide for MEL. The value to MEL can be explained in relation to a pyramid of evidence (Fig. 1) where the base represents the degree of uncertainty and the different levels represent increasing consensus for different types of enquiry. In evaluation exercises, one task is usually to gather all available data and information from a range of sources (including project design and progress reports, and semistructured interviews with key informants and stakeholders) to corroborate and/or triangulate expert opinion. The rationale is to realize the potential of the data gathered to inform the evaluation while minimizing any bias and uncertainty in the fnal judgement. Although evaluation panels can access different target audiences, the information they acquire is subject to uncertainty. This could include, for example, the timing and scale of enquiry; the resources available to conduct in-depth, in-country technical missions; access to independent key informants; the relevance of monitoring data; and personal subjectivity. Conclusions also tend to rely on expert judgement that depends on many factors including the evaluation experience and discipline expertise of individual team members.

Traditional literature reviews also can provide valuable insights for evaluation exercises but are contingent on the sources available (and used), the skills of the researcher in framing and executing the searches, the scope of the review, and the method of analysis with inferences based on only a sample of the available evidence. In contrast,

**Fig. 1** Pyramid of evidence to support project and program evaluation

systematic reviews are often considered to be at the top of the pyramid of evidence hierarchy, providing the highest degree of consensus and lowest amount of uncertainty, given their highly structured and systemic methodology. Rapid evidence assessments (REAs) are consistent with many of the systematic review principles and popular for informing decision making and policy formulation.

This synthesis of evidence was based on the structured methodology for an REA, although it ultimately presented a much deeper and wider ranging level of analysis more usually associated with a systematic review. This narrative synthesis required a hybrid approach articulated across three related yet distinct learning domains, with each characterized by a high degree of complexity. We chose the REA framework because it can be designed and executed in a relatively short time span (1–3 months) depending on the scope of the research question and available literature, compared to 3–6 months for a systematic review. As such, it is more appropriate for informing programmatic evaluations where a relatively quick synthesis of evidence is required. An REA can also provide a deep understanding of the volume and characteristics of evidence that is available on a certain topic and make it more accessible for further scrutiny. Hence, REAs can facilitate answering targeted, research-based evaluation questions by maximizing use of the existing evidence base, while also providing a clear picture of the adequacy of that evidence (Collins et al., 2015).

Evidence reviews can also help evaluators contextualize projects and programs within a broader picture for use in comparing their results with fndings drawn from international evidence. Thus, they can help evaluators understand how conceptual issues are addressed across different disciplines and timeframes, with support from a wider pool of case study material. In some instances (for example, when exploring nexus issues between the human system and ecosystem in the agricultural literature), REAs can help identify existing gaps in knowledge and emerging trends in research, policy, and practice to inform how these gaps and trends might then affect intervention outcomes. Finally, evidence reviews can also help connect research communities with practitioners by providing valuable and informed access to the peer-reviewed scientifc literature that is either overlooked or not known to exist within the project and program evaluation community.

**Acknowledgements** This research was funded by the International Fund for Agricultural Development (IFAD) as part of a broader thematic evaluation of their global development program on smallholder adaptation to climate change. The authors acknowledge colleagues from the thematic evaluation team who provided valuable critique and feedback on this review, including Nurul Alam, James Gasana, Margarita Borzelli Gonzalez, Christian Hergarten, Prashanth Kotturi, Susanne Leloup, Carsten Schwensen, and Horst Weyerhaeuser. The opinions expressed in this chapter are those of the authors and do not constitute a formal view from IFAD.

#### **References**

L. Silici et al.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Part IV**

# **Evaluation Approaches**

#### **Introduction**

Carlo Carugi (\*) Global Environment Facility Independent Evaluation Offce, Washington, DC, USA e-mail<EMAIL_ADDRESS>
This part discusses evaluation approaches for assessing integrated environmental and socioeconomic co-benefts. It focuses on how best to approach the socioeconomic and environment nexus from an evaluation point of view in today's Sustainable Development Goals (SDGs) context. To set the stage of this discussion, the part starts with "Evaluation at the Endgame: Evaluating Sustainability and the SDGs by Moving Past Dominion and Institutional Capture" and Rowe's provocative call for attention to the fact that while the SDGs have brought the social, the economic, and the environmental closer to each other, they still treat them as separate, which does not bode well for sustainability. Evaluation should be aware of that. According to Rowe, there is no disputing the serious threats to sustainability faced by all on this planet, or that humans have contributed signifcantly to this. The good news is that a potential exists to diminish the level of threat. Although thus far evaluation has not shown much interest in environmental sustainability, it can still assume a more activist role and contribute value to checkmating extinction. The chapter frst outlines the current stance of evaluation with respect to sustainability and then turns to how to design evaluations that contribute to efforts to reverse the "end game" environmental trends.

The part continues with Miyaguchi's chapter, "Importance and Utilization of Theory-Based Evaluations in the Context of Sustainable Development and Social-Ecological Systems," in which he emphasizes the importance of evaluation theories and practices considering the nexus between the natural and human systems. It is hard to disagree with him. Miyaguchi rightly notes that the breadth of the partnerships and collaborations brought about by the SDGs among non-state actors, including the private sector, nongovernmental organizations, and nonproft organizations, has been unprecedented. But the inconsistencies and incompatibility among 169 SDG targets and their indicators represent a challenge in light of the abilities and capacities of many states, especially those of developing countries, to adequately monitor and evaluate the current and future SDG achievement status. A related major challenge lies in evaluating the nexus of human and natural systems. The author discusses the utility of theory-based evaluation to address that challenge, and introduces a holistic framework called CHANS (coupled human and natural systems) as a useful analytical framework in evaluating complex, social-ecological systems.

Finland is committed to implementation of the United Nations 2030 Agenda for Sustainable Development and its goals. As a demonstration of this commitment, the Finnish Government established that the implementation of the 2030 Agenda should be evaluated once during every 4-year electoral term. In "Pathway to the Transformative Policy of Agenda 2030: Evaluation of Finland's Sustainable Development Policy," Räkköläinen and Saxén describe the independent evaluation of Finland's national implementation of the 2030 Agenda. The purpose of the evaluation was to support evidence-based decision making by informing the government and the public on the effciency of the 2030 Agenda implementation. This requires an entirely new kind of comprehensive approach and coherence in the policies of the various Finland administrative branches. According to the authors, the evaluation was very timely as it strengthened the knowledge base for updating the implementation plan for the 2030 Agenda after the parliamentary elections in 2019, contributing to the preparation of the new Government program. Another vital role of the evaluation was providing content for the social policy debate that preceded the parliamentary elections.

Sustainable development is characterized by new demands for development actors to mainstream and incorporate in their work as they emerge. Recent noteworthy examples include the environment and climate change. Evaluators must consider how to refect such emerging priorities in their work and do so in a way that allows measuring progress and achievements. In "Evaluating for Resilient and Sustainable Livelihoods: Applying a Normative Framework to Emerging Realities," Kotturi discusses the experience of the Independent Offce of Evaluation (IOE) of the International Fund for Agricultural Development (IFAD) in incorporating concerns on environment and climate within its evaluations. The IOE did this in two ways, namely by adapting its existing methodologies to address environmental concerns while continuing to meet the demands placed on IFAD's evaluation function, and by introducing in its evaluations new methodologies that allow assessment of environmental change, such as geospatial analysis. IFAD has started geotagging its project/program sites more systematically and plans to introduce this practice as a standard part of its operational procedures. Geospatial data are being harmonized across IFAD regional divisions and made available on a web-based platform called IFAD Geonode. To be able to consistently use these data in its evaluations, the IOE is exploring ways to further build its geospatial analysis skills and capacity. This reciprocal adaptation between IFAD operations and evaluation function is an example to follow.

Change in forest cover over time is one of the environmental changes that are most easily observable with geographic information systems (GIS). In "Measuring the Impact of Monitoring: How We Know Transparent Near-Real-Time Data Can Help Save the Forests," Shea of the World Resources Institute describes Global Forest Watch (GFW), an online platform that can support monitoring and evaluation of conservation projects, international commitments to reduce deforestation, and private sector zero-deforestation plans. GFW is based on the idea that transparent, publicly available data can support the greater good, which in this case is reducing deforestation. However, by its very nature, the use of freely available data can be diffcult to track and its impact diffcult to measure. This chapter provides a framework for other open-data platforms to monitor outcomes and measure impact by exploring four different options for measuring GFW's reach and impact. GFW's experience indicates that while quantitative methods are capable of measuring outcomes and impact, they are not suffcient. Qualitative methods are also necessary to understand the mechanisms of adoption and produce lessons for furthering the reach and impact of open-data solutions.

Anand and Batra close the part with "Application of Geospatial Methods in Evaluating Environmental Interventions and Related Socioeconomic Benefts," discussing the extensive and diversifed experience gained by the Independent Evaluation Offce (IEO) of the Global Environment Facility (GEF) in blending geospatial analysis and remote sensing with qualitative analysis in GEF IEO evaluations. Starting from the consideration that environmental interventions such as those the GEF funds are essential for achieving the objectives laid out in the SDGs and the international environmental conventions the GEF is mandated to serve, Anand and Batra convincingly point at the complexities inherent in the assessment of expected environmental results and related socioeconomic benefts. This complexity originates from the interlinkages, be these synergies or trade-offs, between environmental and developmental goals, and the frequent lack of data, especially in developing countries, that characterizes these interventions. Geospatial approaches and tools allow assessment of environmental and socioeconomic outcomes—in other words, measuring the progress of initiatives over time while addressing their complexity. Drawing on GEF interventions in biodiversity (SDG 15), land degradation (SDG 15), and climate change (SDG 13), the authors discuss the application of geospatial approaches to assess GEF interventions' relevance, results, and sustainability in terms of their environmental outcomes as well as observable socioeconomic (SDG 1, 2) and health (SDG 3) co-benefts.

# **Evaluation at the Endgame: Evaluating Sustainability and the SDGs by Moving Past Dominion and Institutional Capture**

Andy Rowe

#### **Abstract**

Three facts underlay this chapter. First, the human system and all our ambitions for improving the human system depend on sustainable natural systems. Second, we do not have much time. On track to fall well short of all sustainability goals, the climate and sustainability crises grow and extinction looms. Third, up to this point evaluation has shown little interest in sustainability, yet evaluation potentially addresses the very questions that are central to informing and guiding rapid adaptation of human behavior to successfully surmounting extinction.

Business-as-usual evaluation will not suffce. At the endgame with extinction looming, we need an evaluation that is more nimble, keeps up with rapidly accelerating knowledge, is relentlessly use-seeking and that guides the way to joined-up approaches. The evaluation we need will systematically mainstream sustainability across all evaluations and interventions, in all evaluation criteria and standards. For this, all evaluations will always address nexus where human and natural systems join and incorporate knowledge and methods from both systems. Existing evaluation knowledge

ARCeconomics, Maple Bay, BC, Canada e-mail<EMAIL_ADDRESS>
is well suited to this task, as are knowledges in biophysical sciences. We know and promote knowledge processes for integrative evaluation and are starting to shift toward the requirements for evaluation at the nexus. As this chapter shows, the anchors holding us back are political, not technical.

#### **Introduction**

Every line of evidence leads us to conclude that the threats to sustainability of the planet and the life it supports are very real, large, multi-faceted and imminent. And yet globally we are falling well short on milestones such as the 2030 Agenda for Sustainable Development and 2050 carbon reduction goals. We have pushed natural systems beyond their capacity to adapt and continue to provide the services on which we depend. We are at the endgame on this planet.1

With some important exceptions, evaluation globally has not recognized the overwhelming evidence that sustainability is a matter worthy of our attention. Sustainability is a materially differ-

A. Rowe (\*)

<sup>1</sup> In revising this chapter, I came across the work of Robert Nadeau that seems to presage some of my own work. He wrote a decade earlier about the shortcomings of neoclassical economics in not addressing the connection between environment and economics that he attributes to a two culture (economics and environment) thinking (Nadeau, 2008) identifed initially by C. P. Snow (Nadeau, 2006).

<sup>©</sup> The Author(s) 2022 207

J. I. Uitto, G. Batra (eds.), *Transformational Change for People and the Planet*, Sustainable Development Goals Series, https://doi.org/10.1007/978-3-030-78853-7\_14

ent matter than those that evaluators are accustomed to addressing because there is a hard stop if we fall short; absent signifcant improvements in our performance, that hard stop is a clear pathway to extinction. Meaning that evaluation at the endgame is different from business-as-usual evaluation. As with chess, the sustainability endgame needs to be fully goal focused and must fully commit all resources to strategies to achieve checkmate.

This chapter is concerned about the character of evaluation that will enable the feld to make useful contributions at the endgame. The most fundamental change is from evaluations' almost monastic focus on the human system to systematic consideration of all interventions (projects, programs, strategies, policies) in their nexus location where both human and natural systems are present, have infuence, provide value, and are affected.

The underlying mechanism for this monastic, human-centered worldview lies in the rootstock of evaluation that is said to be provided by Western social and management sciences with accountability, social inquiry, and social research methods as the trunk of the tree (Christie & Alkin, 2008; Alkin, 2004). That evaluation rootstock is embedded in and draws nutrition from the accumulated soils of Judeo-Christian society strongly infused with dominion, a worldview in which humans have ascendancy over other living and nonliving things, and over other peoples (Rowe, 2018). Humans, and of course especially those of European origin (i.e., white), are at the top of the heap; all else serves. Nonhuman living and nonliving things that constitute the natural systems on which all life depends are regarded as resources to be freely extracted to support humans. And while social sciences and evaluation are adapting to recognize and address how dominion has shaped thought and practice (e.g., gender bias, racism) the presumption that only humans have value and therefore merit consideration continues virtually unchecked in evaluation.

Accountability is one of the stems of the evaluation tree effectively partitioning governance structures from interventions at all levels so that connectivity to public policy goals is truncated (Chelimsky, 2012). It is an important mechanism for the observable, inverse relationship between public expenditures and the status and trends on conditions targeted by public policy such as public health and education (Williams, 2019). Sustainability is about connected systems while accountability is about partitioned systems, making pursuit of sustainability at odds with contemporary approaches to accountability. Accountability is an important authorizing mechanism bringing dominion into evaluation with the unintended effect of imparting a systematic positive bias to evaluation (Rowe, 2019b).

The COVID-19 pandemic provided dramatic evidence that human and natural systems are connected (Patton, 2020b). The virus reached us along pathways created by our relentless incursions into natural systems. The inverse and causal relationship between contemporary forms of economic growth and environmental health have been starkly shown with the slowing of economic and social activities causally linked to reduced incidence of some important health conditions such as asthma and reductions in GHG emissions from economic downturn. The economic downturn has resulted in falling petroleum prices, making it less expensive to produce virgin plastic from fossil fuels as compared to recycling. At the same time, demand for disposable (plastic) protective equipment has increased manyfold. For example, daily single-use plastic medical waste (gloves, masks, and gowns) in Wuhan at the peak of the pandemic there increased sixfold compared to prepandemic averages (Adyel, 2020), all of which is disposed in landflls. Demand for plastic packaging is estimated to have increased by 5.5%, strongly related to the increased consumption of take-out foods; plastic deposits in landflls increased by 1400 tons during the 8-week shutdown in Singapore (Adyel, 2020). That is, the pandemic reduced economic activity, decreasing demand for fossil fuels and lowering their price, leading to increased fossil fuel use to produce single-use plastic commodities. This resulted in increased deposits in landflls and in unmanaged streams of disposal, a good example of connectivity from public health to economy to environment.

The evaluation worldview must shift to acknowledge that human life is intrinsically contingent on healthy natural systems with which we are coupled2 and that we must end the unnecessary harm we cause and move to restoring critical environmental values. Indigenous worldviews are instructive; for example, we might take direction from Daniel Wildcat from Haskell Indian Nations University:

Think of how our worldview changes if we shift from thinking that we live in a world full of resources to a world where we live among relatives. (Zak, 2019)

Evaluation does not address the natural system for social and political reasons, but we have the knowledges, tools, and methods needed to renovate evaluation by drawing on a broad palate including evaluation, social and biophysical sciences, confict resolution, law, and other felds (Patton, 2020b; Rowe, 2018). And emerging efforts by evaluators are starting to build foundations for incorporating sustainability into evaluation, such as in Blue Marble Evaluation (Patton, 2020a) and Better Evaluation (2020). The need for these efforts is amply demonstrated by two recent stocktakings showing evaluation to be only in the early stages of addressing nexus, that development evaluation appears to lead national and sectoral efforts, and that the intellectual infrastructure for nexus evaluation can only be described as weak (Sustainability Working Group, Canadian Evaluation Society [CES], 2020; United Nations Evaluation Group Working Group on Integrating Environmental and Social Impact into Evaluations [UNEG Working Group], 2020).

This chapter's focus is on evaluation at the endgame. I begin with the fndings of the two sustainability stocktakings to describe where evaluation is now with respect to systematically incorporating sustainability into evaluation effectively our starting point for the endgame. The fndings clearly point to evaluation's almost singular focus on human systems and to an intellectual infrastructure that is not ft for the purpose of incorporating sustainability. I then briefy reprise my arguments that the cause for this state of affairs lies in a worldview of dominion whereby humans, and especially white humans, hold dominion over all other living and nonliving things. This worldview is pervasive in social science and evaluation, with accountability serving as a key mechanism authorizing disregard of the natural system in evaluation. To these earlier arguments I add institutional capture as a further mechanism separating human and natural systems in evaluation and use the example of the SDGs to illustrate this. I then return to the endgame, illustrating some fundamental differences between evaluation needed for the endgame and the evaluation we have now.

#### **Taking Stock on Evaluation Practice and Resources on Sustainability**

Two recent and complementary stocktaking efforts have assessed current evaluation practice and resources to incorporate sustainability. The UNEG Working Group on Integrating Environmental and Social Impact into Evaluations completed a stocktaking of evaluation policy and guidance on social and environmental considerations and of practices of UNEG member evaluation offces in addressing social and environmental considerations (UNEG Working Group, 2020).3 The stocktaking is to contribute to deliberations about a common UN-wide approach for incorporating environmental and social considerations into all evaluations (whether or not the evaluand is an environmental program). The second stocktaking was conducted by the Sustainability

<sup>2</sup>I use the concept of *coupled systems* to refer to the dynamic complex relationships across intimately connected human and natural systems (Liu, 2007; Ostrom, 1990; Rowe, 2019b) and *nexus* to refer to evaluation of sustainable development with interlinked social, economic and environmental dimensions (Uitto, 2019).

<sup>3</sup>UNEG is a professional network that brings together the evaluation units in the UN system, including the various UN departments, specialized agencies, funds and programs (http://unevaluation.org/) as well as non-UN organisations such as the GEF IEO.

Working Group of the Canadian Evaluation Society (CES) for two purposes: to assess the extent to which sustainability has been addressed in federal evaluations and by other governments and organizations in Canada and by Canadian evaluators working internationally, and to assess the intellectual infrastructure for evaluating sustainability in Canada and the United States (CES, 2020). The CES stocktaking is informing consideration of how the CES can mainstream sustainability in its own work and in evaluation in Canada. The CES stocktaking report was completed in 2020 with much of the work undertaken on a pro-bono basis by four leading Canadian consulting frms.4

These two undertakings cover a wide swath of global evaluation with UNEG addressing development evaluation and the CES addressing evaluation at national and sub-national levels while also assessing the Canadian and U.S. intellectual infrastructure for mainstreaming sustainability. Together, these two stocktaking efforts provide powerful evidence that the evaluation feld is, at best, mildly and only recently addressing sustainability and that the social dimension is the priority for evaluation.

The two stocktaking efforts clearly showed that sustainability is largely missing in action from evaluation in the UN system and in Canada, and from the intellectual infrastructure for evaluation in the United States and Canada.

• The UNEG stocktaking also revealed that, frst, coverage of the social system is also only partial and, despite heightened awareness of social–natural systems interaction, evaluation guidance on environment is extremely limited; and second, that the over-arching need emerging from documentary analysis and survey responses of UNEG member agencies is for a comprehensive document providing advice on how to evaluate the interactions among social and environmental considerations within the framework of UN activities in support of the SDGs (UNEG Working Group, 2020, p. 6).

• The CES stocktaking showed sustainability and consideration of the natural system to be largely missing from federal evaluations conducted in 2016–2018, with Global Affairs Canada being a notable exception, and that the intellectual infrastructure in Canada and the United States for evaluation in the natural system is very limited.

The Canadian stocktaking is worth highlighting given the strong and long-standing evaluation infrastructure:

For evaluation function and infrastructure, Canada is a global leader. Canada also has signed most international climate and sustainability protocols and agreements and the elected government platform and positions have, since 2015, accorded sustainability and climate a strong priority.

Given the relative strength of evaluation in Canada and wide acceptance of the importance

<sup>4</sup>Baastel, Goss Gilroy, Prairie Research, and Universalia.

of climate and sustainability, it is reasonable to expect more positive observations than the sustainability stocktaking showed. The stocktaking had four elements:

The fndings of the two stocktakings are sobering but also encouraging. They are sobering in their confrmation that the evaluation feld has little or no presence and little existing capacity in contributing to sustainability, the leading issue of the day. But we can fnd encouragement because they clearly point to a growing recognition that sustainability is a top matter and to an interest in addressing sustainability as a priority.

Given the similarity of fndings of the UNEG and Canadian efforts, a search for the systematic origins for the clear prioritization in evaluation of the human over the natural system, and the separation of the two systems, is reasonable. The next section proposes that the origins lie in a dominioninfused worldview asserting that humans are imbued with rights over all else—basically colonization of the planet to serve humans. Accountability structures have served as an important mechanism framing evaluation from a dominion perspective, and global and national governance units have sought to capture the resulting siloed landscape.

#### **Dominion, Accountability, and Institutional Capture**

The two stocktaking efforts clearly show that evaluation strongly prioritizes social matters, has very limited capacity to address natural systems,6 and only rarely, across the vast landscape of evaluations covered by the two stocktaking efforts, are the two systems, human and natural, considered together.

I offer an explanation that evaluation rests on knowledge that itself rests on a worldview of dominion in which humans, and especially humans of European origin, have dominion over all other living and nonliving things and regard these as resources for use as humans see ft. Social inquiry and social research methods are said to be the rootstock of evaluation (Alkin, 2004), but I argue that they draw their nutrition from the terroir of dominion (Rowe, 2019b). The other rootstock of evaluation is said to be accountability. This management construct is layered on top of dominion and is the second causal force that has contributed to an almost monastic focus

<sup>5</sup>The stocktaking was limited to Canada and the United States for purposes of feasibility only.

<sup>6</sup>Natural systems are inclusive of environmental impacts as addressed by the UNEG stocktaking.

on the human system by bounding accountability, and consequently evaluation, to the intent and boundaries of interventions severing or at least loosening connection to the public policy goals for which they exist and to other efforts addressing those goals. Third, without structure that recognizes the connectivity between human and natural system goals and the dependence of the social system on the national systems, the SDGs offer a goal structure, initially and still today, in which the natural system does not need to be considered. This section provides a brief overview of how dominion, accountability, and institutional capture contribute to evaluation's overwhelming focus on the social system and neglect of the natural system, as refected by the UNEG stocktaking.

#### **Dominion**

Evaluating sustainability frst requires systematically recognizing and addressing those elements of both the human and natural systems that infuence and are infuenced by the evaluand. The stocktaking efforts showed evaluation to have an overwhelming focus on the human system, refecting a dominion worldview where humans are ascendant and all other things, living and natural, can be extracted and deployed for human use. This is an implausible position: If human life depends on what we draw from the natural system, then the natural system must have value to the human system. The position that the natural system has no value and need not be considered has deep roots in social science and economics, which in turn are rooted in Judeo Christian worldviews and associated beliefs about dominion. Dominion is quite a simple concept whose existence is undeniable but, like any deeply embedded concept, it can be challenging to recognize and address. Dominion also provides a causal connectivity between the treatment of colonized and subjugated peoples and the treatment of other species and elements in the natural system. Indeed, one of the rationales for the actions of colonizers was the superiority of their worldview over the very different worldviews of many of the colonized peoples who regarded themselves and other living and nonliving things as equal and part of a whole.

Dominion means that other living and natural things do not have value, that they exist to serve humans, and any monetary value ascribed to them results from ownership or regulated rights that provide the ability to control access and use. A classic example of dominion in action was the construction of massive dams for electrical generation in pursuit of industrial and economic development. Early critiques and resulting modifcation of cost beneft and other analysis of dams recognized and evaluated the direct losses to humans above and below the dams. But only recently have the ecosystem losses from fooding above the dam and water loss below the dam begun to be imputed, although on a limited basis. Because living and natural things other than humans were not valued, no mechanism was in place to recognize their importance and scarcity, directly causing relatively unfettered extraction and destruction—the fundamental cause of the sustainability crisis and climate change.

The issue of temporal and spatial scales is another way that dominion and accountability have led to evaluation's monastic focus on the human system. Systems are by their nature coupled, extensive, and dynamic, each with a wide range of temporal and spatial scales and often very diverse units of account (Rowe, 2012). Human temporal and spatial scales differ signifcantly from scales relevant to the natural system, and, of course, with a dominion-infused worldview, the units of account that matters are human. When the natural system is considered, it is usually from an extraction perspective in terms of utility to humans, not as a coupled system meriting its own place in evaluations.

Evaluation is a human system activity usually conducted from temporal scales meaningful to the aspects of the human system that is commissioning and undertaking the evaluation. By their nature, effects of a human or natural intervention have broad reach, well beyond the temporal and spatial reach of the intervention. Evaluations are aligned with the programmatic schedules of interventions and usually extend backward to the start of the intervention and forward to some programmatic or arbitrary time, usually less than 10 years from their start. These temporal scales bear no relevance for the temporal scales of natural system elements that can range from centuries to moments.

The value and function of natural systems is not the only consequence of dominion. Clearly, racism and misogyny are causally linked to the dominion of white, European-origin males. To illustrate, a 1987 synthesis of two national 1986 studies in the United States found that race was the was the most signifcant factor in locating toxic landflls and that 3 of 5 Black and Hispanic Americans, and approximately half of all Asians, Pacifc Islanders, and American Indians, lived in communities with uncontrolled toxic waste sites (Gilio-Whitaker, 2019). And while the roots of racism, misogyny, and extraction are commonly and frmly planted in dominion, actions on these matters are often pitted against one another using class, religion, nationality, and other constructs, and all and each constrained by what is deemed possible within capitalism and not overly deleterious to economic growth.

#### **Accountability**

Accountability is cited as one of the main stems of evaluation (Alkin, 2004); from the perspective of sustainability, accountability can be described as a highly evolved contagion. It is a management construct designed to enable monitoring and improvement of agreed outcomes and is usually linked to program, management, and personnel performance. Managers and programs seek to constrain risk of falling short on accountability metrics by focusing on what they have the authority, resources, and capacities to be able to likely achieve. This provides incentives to narrow the programmatic box for which they are accountable and to resist being accountable for contributing usefully to other boxes.

The two stocktakings observe that the natural system, sustainability, and the nexus are systematically absent from evaluations. However, the remit of some agencies does address the natural system, such as the Global Environment Facility (GEF), UN Environment, national government departments such as environment and natural resources, and environmental NGOs. The evaluation record of these is mixed; while the GEF Independent Evaluation Offce addresses both systems and incorporates nexus, other natural system agencies focus almost exclusively on the natural system. We can generalize this by framing evaluations as single system (either human or natural) or two system (Rowe, 2012).

Evaluations are overwhelmingly single system, a situation to which accountability frames contribute. Since natural system values are infrequently considered, accountability reinforces ignoring the natural system. We know that even within the human system we must recognize and incorporate connectivity to reach to public policy goals. Reinforcing and incentivizing partitions between human and natural systems and within human systems accountability reinforces silos, the opposite of the silo busting required for evaluation at the nexus and for evaluation more generally.

Evaluating sustainability requires evaluation practices and methods that (a) recognize and operate at the nexus where both human and natural systems are present and (b) address the intrinsic coupling between and within human and natural systems. It is bad enough that the natural system is not valued and that systems approaches and understanding are unlikely with political and administrative partitioning. Accountability reinforces and further constrains possibilities of addressing sustainability in programming and evaluation with its focus on "accountability scales" that rarely reach beyond the accountability frame of the intervention.

One result is that the responsibility and remit of the intervention and reach of its direct effects frame the spatial scale for the evaluation within the larger framing of governance structures such as local area, province, or country, or within the remits of the responsible government organization. Ecosystems and landscapes provide more relevant spatial framing for natural systems; there is no reason to expect the boundaries and shapes of ecosystems to align with human system political and administrative boundaries or program areas. And ecosystems are not always appropriate for the territory of an organism—for example, a wolf, whale, or snail function across or entirely within an ecosystem. At a minimum, the relevant spatial scales for the natural system can be thought of as an ecosystem, often highly coupled with other ecosystems. This presumption that boundaries and territories in the natural system will align with the political and administrative boundaries of the human system is not limited to the natural system. For example, the same assumption is made about boundaries of Indigenous traditional lands, and that the Canada/ United States border is relevant or appropriate where it crosses traditional lands. For many Indigenous peoples, the relevant spatial boundaries are their traditional territories from which food; medicine; and spiritual, ceremonial, and community values are drawn (Gilio-Whitaker, 2019). Instead, evaluation is likely to address the spatial scales defned by colonial occupation such as a reserve or First Nation territory; these are always and importantly smaller than traditional territories and often exclude areas of high importance to Indigenous peoples.

Program managers, evaluators, and especially evaluation commissioners often insist that an evaluation be conducted within the frame of the stated goals and operations of (accountability of) the intervention. This severs interventions from each other and limits the reach of evaluation, falling well short of the critically important public policy goals such as ending poverty or achieving sustainability. As Williams (2019) observed, such a frame establishes a program and evaluation ecosystem where programs systematically are assessed as providing positive contributions to the broad goal and where no progress is visible toward achieving the goal itself. It also creates a systematic positive bias in evaluation (Rowe, 2019b).

#### **Institutional Capture**

Institutional capture is the process by which identifed needs and demands for major structural change are captured by existing structures, policies, and approaches. The SDGs were such a moment when sustainability was recognized as an overriding priority requiring major structural change to address. By and large, responsibilities for individual SDGs were assigned without changing the partitioned structure of organizations. But successfully addressing sustainability programmatically or in evaluation requires platforms suited to the task; the partitioned structures, policies, and approaches are not well suited to pursuit and evaluation of sustainability. Understandably, the UN and other multilateral organizations staked claims on specifc SDGs, pursuing the assurance this provided to their futures; some such as the United Nations Development Programme (UNDP), the United Nations Industrial Development Organization (UNIDO), and the International Fund for Agricultural Development (IFAD) now explicitly recognize this connectivity, while others are on the pathway to do so.

The evaluation criteria of the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD DAC) Network on Development Evaluation (2019) address sustainability as sustaining interventions and achievement of impacts, and not, as most think of sustainability, as a nexus concept of human and natural systems together with emphasis on sustaining the capacity of the natural system to enable life. In this, evaluation is somewhat distinct—elsewhere sustainability is recognized as a science "with a room of its own" (Clark, 2007); the 2009 Nobel Prize in Economics was awarded to Elinor Ostrom for her work on the commons and as one of the founders of coupled human and natural systems (CHANS) analysis. And, as the UNEG and CES stocktaking efforts have shown, evaluation has also been largely captured by the institutions it serves.

#### **Sustainability-Ready Evaluation**

Evaluators are good observers and place confdence in good evidence. They will increasingly be persuaded by the emerging knowledge on sustainability and climate, and increasingly recognize how these have affected the human issues and populations that have been evaluators' primary concern. They will also recognize how their long-preferred interventions and methods in the human system can contribute to worsening climate and sustainability. The underlying premise of sustainability-ready evaluation is that evaluators will recognize the need to address effects in the natural as well as the human system and take evaluation to a place where existing capacities are insuffcient. Evaluators will need to, for example: recognize, speak, and hear representatives of natural system knowledge; learn how to feasibly address dynamically coupled systems (Liu, 2007); incorporate effects that have widely differing temporal and spatial scales and very differently framed units of account; and be open to and advocate for shared evaluation functions (see Carugi & Bryant, 2019; Rowe, 2012; Uitto, 2019).

Other felds of inquiry and assessment will be important contributors to developing and implementing evaluation at nexus settings. Evaluation is a cross-disciplinary feld accustomed to drawing from other felds of inquiry, and this is fortunate because evaluating sustainability will require knowledge from and engagement from more system sciences. Climate and materials sciences, ecology, and geography will be important as will knowledge from more focused felds such as energy engineering, biology, agriculture, forestry and fsheries, and areas of public administration such as procurement. Two connected felds concerned with understanding and assessing nexus will likely be critically valuable fellow travelers: sustainability science (Kates, 2011; Clark et al., 2016) and CHANS work and networks (Liu, 2007; Ostrom, 1990).

Strongly siloed culture, structures, and practices of evaluation and programs create challenges to mainstreaming sustainability in the nexus sense of human and natural systems. To truly incorporate the natural system into longstanding and newer interventions whose primary focus is in the human system is proving diffcult; likewise, to get evaluations to address the natural system is challenging, as shown by the UNEG stocktaking. However, the effort does appear to be gaining some momentum, such as in research on environmental effects of refugee camps (Braun et al., 2016), although discarded Covid-19 face masks are already fnding their way to landflls and water bodies. As Fabien Cousteau (2020) wrote recently,

We live in a closed-loop system. We can't actually throw things "away." The plastic we toss in the garbage often just ends up inside the bodies of marine animals, before fnding its way back inside of us. (para. 12)

This means that what are usually classed as unexpected or unintended effects, or effects that were known but ignored because they lay outside the accountability frame of the intervention, now have to be recognized as a direct effect of the intervention. I have shown (Rowe, 2018, 2019a, b) that ignoring direct effects in the natural system imparts a systematic positive bias to evaluations. To make the point clear, evaluation conducted in silos has a systematic positive bias favorable to the intervention and, importantly, arising because of the accountability frames that are applied as discussed above.

Sustainability-ready evaluation is an evaluation function that is ready to recognize these connections and able to cross them. It is an evaluation function with individual evaluators and evaluation organizations that are enthused by contributing to a future we choose (Figueres & Rivett-Carnac, 2020). There are many strongly held visions of what that future should be, with associated and strongly held views of what we need to change to get there. It is not the job of evaluation to pick a pathway or end point; our job is to be enthused and capable of contributing to improvement, including sorting and valuing the competing pathways and desired new ways. Evaluation today is appropriately described as close to sustainability-ignorant and far from sustainability-ready.

#### **How Can Evaluation Contribute to Checkmating Extinction?**

An evaluation able to contribute to the defeat of extinction requires some relatively simple changes in how we frame and undertake our work, but these simple changes will signifcantly alter the stance and thus the politics of evaluation. Here I briefy sketch some important changes in stance for an evaluation ft for purpose for the endgame.

Checkmating extinction will only be possible if evaluation shifts from a singular focus on the human system to mainstreaming nexus in all evaluation. We are right now at a juncture where urgently needed changes seem possible. A second major change in the stance of evaluation relates to expectations of goal achievement: The current standard of progressing toward goals merely draws out a checkmate in favor of extinction. Instead, overcoming extinction requires a stance at the endpoint and assesses achievement of these goals with evaluation providing guidance to improve performance. Of course, achieving these goals requires joined-up, system-wide efforts for which we need to join evaluation stances with systems approaches. Conditions are worsening faster than expected and efforts to understand status and trends in natural systems and options for mitigation and adaptation are generating new knowledge at a rapid pace. This means that the stance of evaluation must be nimble and adaptive to integrate these changes, and be undertaken with suffcient rapidity to align with signifcantly accelerated decision cycles. Together, all of this means that evaluation for the endgame must be relentlessly use seeking and forward looking.

These are but some of the features needed for an evaluation function and practice that is an ally in efforts to checkmate extinction. Consideration of this stance will identify additional necessary features and perhaps diminish the importance of some that are discussed below. This chapter is only an early step in identifying the stance needed for an evaluation that contributes to the endgame.

#### **Recognizing Natural Systems as the Foundation for the Human System Means Adding the Natural System Perspective to All Evaluation Criteria**

*The opponent at the endgame is continued destruction of the natural system by humans, meaning that both systems must be considered and addressed by evaluation at the endgame. That is, nexus is the required position for evaluation at the endgame.*

Think of the relationship between human and natural systems with the natural system as a bank account. The human system has well exceeded its overdraft limit so that now every draw we make must have a repayment schedule that not only matches current withdrawals but also systematically and strategically starts to reduce the overdraft.

Environmental and social safeguards and policies have been enacted by most development donors with the requirement that they are applied in project development, funding, implementation, operation, and assessment (IFAD, 2018; World Bank, 2020). These standards are relatively recent, most enacted in the past decade, and the documents clearly consider human and natural systems as connected. In practice, however, climate and environment/natural resource management are usually treated as additional criteria that must be addressed in project design and assessment, isolated and marginalized rather than imbedded into planning.

We can consider inclusion of the natural system criteria in four phases, defned by requirements to meet the threshold to achieve a "satisfactory" rating:

approach. It was not unusual to see project designs, evaluations, and supervision reports that considered commitments to compliance with donor environmental guidelines and safeguards and with national regulations to warrant a satisfactory rating. To put this in perspective, I have never seen an evaluation of an education or health intervention make a statement such as, "The design of the intervention incorporated government guidelines and a designated body has the authority to inspect and enforce, so we deem the approach satisfactory." Substitute environment for education in the previous sentence and we have a statement that is frequently made about natural resources, sustainability, and climate in supervision and evaluation documents. *Result: Increase in the overdraft on the natural system account.*

#### *fusion provide positive prospects that this will continue and accelerate.*

Mainstreaming sustainability systematically locates evaluation at the nexus and is a frst and essential change in the stance of evaluation; but valuing the natural system evaluation is beginning to address dominion.

#### **Evaluation Standards Will Emphasize Achieving the Larger Goals Identifed as Central to Checkmating Extinction**

*When the end is in sight, when the endgame is what is at play, our focus shifts from playing the game well (admirable evaluation) and from contributing to incremental improvements for benefciaries to an absolute need to provide value to checkmating our destruction of the natural system that sustains us.*

To illustrate the character of absolute evaluation standards, the International Resources Panel (IRP)7 has shown that the planet does not have the material resources to provide for expansion of existing cities and creation of new ones resulting from urbanization, rural-to-urban migration, and population increase (Swilling, 2018). Development projects typically claim they will "contribute to" slowing rural-to-urban migration through improved rural livelihoods. Rural-tourban migration and population growth are complicated and involve a powerful mix of push and pull factors requiring combined programmatic efforts to achieve sustainable fows and levels that will contribute to sustainable development, not undermine it. This is an illustration of a goal important for the endgame; evaluation needs to assess against achievement of that goal. If population and urban growth threaten sustainability, then the standard that needs to be applied in evaluation is achieving the goals that will remove population increase and rural-to-urban migration as important threats to sustainability. This does not mean curtailing migration and mobility, which are important to escaping severe climate

<sup>7</sup> https://www.resourcepanel.org/

and for humanitarian and economic reasons. Achieving levels of rural-to-urban migration sustainable for both urban and rural areas likely hinges on viable rural communities. And evaluation can provide value in moving from current unsustainable fows by adopting a stance that includes an expectation of verifable achievement of important endgame outcomes that will realize specifc migration goals set at sustainable levels. These goals, like the high-end climate goals of a CO2 reduction to limit temperature increase to 1.5°C, should be specifed in absolute terms; for example, the specifc sustainable population of Vancouver or Hanoi.

## **Standards Need to Shift to Evaluating Against Collective Achievement of Sustainability Goals, and Away from Likely Contributions by Partitioned Organizations and Interventions**

*Achieving the results needed to checkmate extinction requires collective and synthesized efforts; this is the required stance of evaluation for the endgame.*

Partitions must be replaced by joined-up action and evaluation must adopt a collaborative focus on system achievement of the larger goals required for sustainability, regardless of whether interventions have adopted this stance. Holding interventions accountable for achieving results for which they are neither resourced nor authorized is inappropriate. However, for the endgame, evaluators should still address the needed result, what is required to achieve it, and the success and contributions of efforts toward collectively addressing this result. Setting goals that are critically important to success in the endgame is one way evaluation can observe shortcomings in collaboration and shared efforts toward achievement. It will also reveal gaps between current and needed achievements that likely span a number of individual organizational remits. This type of evaluation, focusing on what is needed, refects the spirit of a results focus but from a collective, joined-up perspective rather than from partitioned efforts. It promotes collective action and accountability for sustainability goals.

#### **Sustainability Is Imbedded in All Evaluation Criteria Refecting Nexus, Not Isolated as a Free-Standing Criterion**

*An evaluation stance recognizing the complex connectivity of human and natural systems means that all evaluation criteria should be considered from a two-system stance—sustainability and climate should not be isolated in separate and usually marginalized criteria.*

Collective action means that work toward any and all of the SDGs and government and thirdsector initiatives is likely to be drawing from and contributing to the sustainability of the natural system and climate. The previous element brings these into the scope of evaluation for the endgame, and this element addresses how evaluation accomplishes this. Each of the evaluation criteria and standards, e.g. the OECD DAC criteria (relevance, coherence, effectiveness, effciency, impact, and sustainability), needs to be infused with considerations of sustainability by addressing both human and natural systems. Examples include the effect of humanitarian efforts on the physical landscape, and the many effects on the natural system of the use of plastics.

#### **Evaluation Standards at the Endgame: Evaluating with Rapid Change and Uncertainty**

*Relentless rapid learning and brisk adaptation is the temporal scale required for interventions at the endgame and so must also be for evaluation.*

Sustainability and climate are topics where the knowledge and practice base is improving rapidly and still features considerable uncertainty and ambiguity. Where changes in our knowledge are proceeding at a rapid pace and where considerable ambiguity still exists, longer term interventions—such as 4 or more years—will inevitably be suboptimal by the time they are halfway through their remit, perhaps highly suboptimal. Those implementing interventions must adopt vigorous adaptive management practices and be held accountable for this. We need to accelerate the pace of refection and renewal, or else an important portion of our efforts will be applying approaches that are no longer considered effcacious at a time when we can least afford to do so. Evaluation is an important vehicle for this.

At the endgame, knowledge cycles are greatly reduced—we now think that the shelf life of some current climate knowledge is about 2 years. Severe climate events are also accelerating, becoming more frequent and severe and building cumulative effects. Two category 5 storms and resulting fooding within one month, as happened in 2020 in the Caribbean, requires very different responses than two storms of equal strength separated by 10 years.

To illustrate, consider 2030 and 2050 as forecasts of when we will pass irreversible thresholds, which make them key timings for checkmating extinction. A large portion of program and project cycles approach 7 or more years from inception to renewal, with 1–2 years for planning and funding, 1–2 years to mobilize, then operations of 4–5 years. Seven-year program cycles gives us just one program cycle until 2030 and four until 2050. The typical mid-term, end-of-term, and later ex-post evaluation approaches cannot provide information, insights, and advice in time to affect interventions in much more rapid adaptation cycles. Some evaluation approaches and methods will need to adapt rapidly and signifcantly to be relevant to evaluation at the endgame; fortunately, other approaches and methods are more ft for this purpose. Longer term evaluation undertakings will still provide value, such as with longer term impacts and adaptation of interventions to changing conditions, but overall, evaluation at the endgame is a new challenge for the feld, requiring the evaluation stance to immediately become shorter term and employ more rapid approaches that are relentlessly use seeking such as Rapid Impact Evaluation (Rowe, 2019a).

#### **Evaluation for the Endgame Relentlessly Pursues Use**

*We no longer have the luxury to indulge the evaluation agendas and strategies that do not contribute to checkmating extinction. Our work must focus directly and strongly on the rapid adaptation and learning cycles of a proliferating landscape of actions contributing (or not) to checkmating extinction.*

#### **Conclusion: Nexus Requires New Rootstock to Grow Relevant Evaluation Functions**

This chapter recognizes that we have entered the endgame of extinction and identifes what is needed for evaluation to contribute to checkmating extinction. I have sketched a trail from where evaluation is today to where it needs to be to provide value and guidance to efforts to achieve a checkmate favorable to life on the planet.

That trail frst observes that evaluation at global and national levels is monastically focused on the human system and only marginally addresses the natural system. It reaches back to Judeo-Christian concepts of dominion as the origin story for our focus, and identifes narrowly framed accountability structures as an important contemporary mechanism for the exercise of dominion. Reinforcing this is institutional capture of efforts to infuse sustainability and systematically address necessary climate goals in development and associated social ambitions at all levels. The unhappy result is seen in two recent stocktaking efforts illustrating the limited contributions of contemporary evaluation to sustainability.

Evaluation at the endgame is different from the evaluation we have known and practiced up to now. Evaluation will need to take stances that will be challenging, as is any endgame effort. The six characteristics of evaluation for the endgame are:

Adopting these stances at frst appears to be a radical shift for evaluation, one with poor prospects for adoption. However, a growing recognition of the sustainability and climate imperative is underway. Evaluation working with biophysical knowledge partners is able right now to usefully contribute to the endgame. The hard part is recognizing that the prevailing stance of evaluation is contributing to the problem, that we need to turn our backs on forces and institutional arrangements that have provided us comfort in exchange for complicity, and turn to a future we choose, which is to be a valued and useful contributor to checkmating extinction.

#### **References**

Patton, M. Q. (2020a). *Blue Marble evaluation*. Guilford.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Importance and Utilization of Theory-Based Evaluations in the Context of Sustainable Development and Social-Ecological Systems**

Takaaki Miyaguchi

#### **Abstract**

Numerous challenges confront the task of evaluating sustainable development—its complex nature, complementary evaluation criteria, and the diffculty of evaluation at the nexus of human and natural systems. Theorybased evaluation, drawn from critical realism, is well suited to this task. When constructing a program theory/theory of change for evaluating sustainable development, concepts of socioecological systems and coupled human and natural systems are useful. The chapter discusses four modes of inference and the application of different theory-based evaluation approaches. It introduces the CHANS (coupled human and natural systems) framework, a holistic, analytical framework that is useful in evaluating such complex, socialecological systems and resonates with the challenging elements of sustainable development evaluation.