# METHODS AND APPLICATIONS IN IMPLEMENTATION SCIENCE

EDITED BY : Mary E. Northridge, Donna Shelley, Thomas G. Rundall and Ross C. Brownson PUBLISHED IN : Frontiers in Public Health

#### Frontiers Copyright Statement

© Copyright 2007-2019 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88963-113-1 DOI 10.3389/978-2-88963-113-1

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office<EMAIL_ADDRESS>
# METHODS AND APPLICATIONS IN IMPLEMENTATION SCIENCE

Topic Editors:

Mary E. Northridge, New York University (NYU) Langone Dental Medicine – Brooklyn, United States Donna Shelley, NYU School of Medicine, United States Thomas G. Rundall, University of California, Berkeley School of Public Health, United States

Ross C. Brownson, Washington University School of Medicine, United States

Image: Ververidis Vasilis/Shutterstock.com

The Trans Adriatic Pipeline depicted here starts from the Caspian Sea and reaches the coast of southern Italy. It is intended as a tribute to Lawrence Green, DrPH, MPH, who popularized the pipeline graphic that depicts the 17-year odyssey necessary for the production and transfer of knowledge from research to practice and policy.

The purpose of this Research Topic is to share the latest developments in the methods and application of implementation science. Briefly, implementation science is the study of methods to promote the adoption and integration of evidence-based practices, interventions, and policies into routine health care and public health settings. Implementation research plays an important role in identifying barriers to, and enablers of, effective health systems programming and policymaking, and then leveraging that knowledge to implement evidence-based innovations into effective delivery approaches.

Citation: Northridge, M. E., Shelley, D., Rundall, T. G., Brownson, R. C., eds. (2019). Methods and Applications in Implementation Science. Lausanne: Frontiers Media. doi: 10.3389/978-2-88963-113-1

# Table of Contents

### INTRODUCTION TO THE RESEARCH TOPIC

*06 Editorial: Methods and Applications in Implementation Science* Mary E. Northridge, Donna Shelley, Thomas G. Rundall and Ross C. Brownson

#### THEORIES, FRAMEWORKS, AND APPROACHES

Byron J. Powell, Maria E. Fernandez, Nathaniel J. Williams, Gregory A. Aarons, Rinad S. Beidas, Cara C. Lewis, Sheena M. McHugh and Bryan J. Weiner

*33 The Price per Prospective Consumer of Providing Therapist Training and Consultation in Seven Evidence-Based Treatments Within a Large Public Behavioral Health System: An Example Cost-Analysis Metric*

Kelsie H. Okamura, Courtney L. Benjamin Wolk, Christina D. Kang-Yi, Rebecca Stewart, Ronnie M. Rubin, Shawna Weaver, Arthur C. Evans, Zuleyha Cidav, Rinad S. Beidas and David S. Mandell

### PARTNERSHIPS, ENGAGEMENT, AND COLLABORATION

*41 Unpacking Partnership, Engagement, and Collaboration Research to Inform Implementation Strategies Development: Theoretical Frameworks and Emerging Methodologies*

Keng-Yen Huang, Simona C. Kwon, Sabrina Cheng, Dimitra Kamboukos, Donna Shelley, Laurie M. Brotman, Sue A. Kaplan, Ogedegbe Olugbenga and Kimberly Hoagwood

Mary E. Northridge, Sara S. Metcalf, Stella Yi, Qiuyi Zhang, Xiaoxi Gu, Chau Trinh-Shevrin for the Implementing a Participatory, Multi-Level Intervention to Improve Asian American Health Research Team

*84 A Mixed Methods Approach to Evaluate Partnerships and Implementation of the Massachusetts Prevention and Wellness Trust Fund* Rebekka M. Lee, Shoba Ramanadhan, Gina R. Kruse and Charles Deutsch

#### MEASUREMENT ADVANCES

*94 Developing a Survey Tool to Assess Implementation of Evidence-Based Chronic Disease Prevention in Public Health Settings Across Four Countries*

Elizabeth L. Budd, Xiangji Ying, Katherine A. Stamatakis, Anna J. deRuyter, Zhaoxin Wang, Pauline Sung, Tahna Pettman, Rebecca Armstrong, Rodrigo Reis and Ross C. Brownson

#### ADAPTATION ASSESSMENT AND MODELS

*125 Systematic, Multimethod Assessment of Adaptations Across Four Diverse Health Systems Interventions*

Borsika A. Rabin, Marina McCreight, Catherine Battaglia, Roman Ayele, Robert E. Burke, Paul L. Hess, Joseph W. Frank and Russell E. Glasgow

*136 The Family Check-Up 4 Health (FCU4Health): Applying Implementation Science Frameworks to the Process of Adapting an Evidence-Based Parenting Program for Prevention of Pediatric Obesity and Excess Weight Gain in Primary Care*

Justin D. Smith, Cady Berkel, Jenna Rudo-Stern, Zorash Montaño, Sara M. St. George, Guillermo Prado, Anne M. Mauricio, Amanda Chiapa, Meg M. Bruening and Thomas J. Dishion

*153 Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State*

Sapana R. Patel, Paul J. Margolies, Nancy H. Covell, Cristine Lipscomb and Lisa B. Dixon

#### IMPLEMENTATION STRATEGIES RESEARCH

*162 A Pragmatic Approach to Guide Implementation Evaluation Research: Strategy Mapping for Complex Interventions* Alexis K. Huynh, Alison B. Hamilton, Melissa M. Farmer,

Bevanne Bean-Mayberry, Shannon Wiltsey Stirman, Tannaz Moin and Erin P. Finley

*177 Implementation Mapping: Using Intervention Mapping to Develop Implementation Strategies*

Maria E. Fernandez, Gill A. ten Hoor, Sanne van Lieshout, Serena A. Rodriguez, Rinad S. Beidas, Guy Parcel, Robert A. C. Ruiter, Christine M. Markham and Gerjo Kok

*192 Implementation Climate and Time Predict Intensity of Supervision Content Related to Evidence Based Treatment*

Michael D. Pullmann, Leah Lucid, Julie P. Harrison, Prerna Martin, Esther Deblinger, Katherine S. Benjamin and Shannon Dorsey

#### SUSTAINMENT EFFORTS

*208 Using Survival Analysis to Understand Patterns of Sustainment Within a System-Driven Implementation of Multiple Evidence-Based Practices for Children's Mental Health Services*

Lauren Brookman-Frazee, Chanel Zhan, Nicole Stadnick, David Sommerfeld, Scott Roesch, Gregory A. Aarons, Debbie Innes-Gomberg, Lillian Bando and Anna S. Lau

*220 Agency Leaders' Assessments of Feasibility and Desirability of Implementation of Evidence-Based Practices in Youth-Serving Organizations Using the Stages of Implementation Completion* Lawrence A. Palinkas, Mark Campbell and Lisa Saldana

# Editorial: Methods and Applications in Implementation Science

Mary E. Northridge1 \*, Donna Shelley 2 , Thomas G. Rundall 3 and Ross C. Brownson4,5

1 Hansjörg Wyss Department of Plastic Surgery, NYU School of Medicine, New York University (NYU) Langone Dental Medicine - Brooklyn, Brooklyn, NY, United States, 2 Department of Population Health, NYU School of Medicine, New York, NY, United States, 3 The Center for Lean Engagement and Research (CLEAR), University of California, Berkeley School of Public Health, Berkeley, CA, United States, 4 Prevention Research Center in St. Louis, Brown School at Washington University in St. Louis, St. Louis, MO, United States, 5 Division of Public Health Sciences, Department of Surgery, Alvin J. Siteman Cancer Center, Washington University School of Medicine, St. Louis, MO, United States

#### Keywords: adaptation, evaluation, frameworks, implementation strategies, measurement, mechanisms, partnerships, sustainability

#### **Editorial on the Research Topic**

#### **Methods and Applications in Implementation Science**

In a classic review, Green et al. popularized the pipeline graphic that depicts the 17-year odyssey necessary for the production and transfer of knowledge from research to practice and policy (1). Still, the vetting of research through successive scientific filters does little to assure that the populations in need of evidence-based practices ever benefit from scientific advances. This Research Topic is intended to provide insights from implementation science that move beyond the clinical care of individual patients, to also take account of provider, organizational, systems, and policy levels pertaining to health and health care.

#### Edited by:

Marcia G. Ory, Texas A&M University, United States

#### Reviewed by:

Matthew Lee Smith, University of Georgia, United States

\*Correspondence: Mary E. Northridge<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health

> Received: 05 July 2019 Accepted: 18 July 2019 Published: 31 July 2019

#### Citation:

Northridge ME, Shelley D, Rundall TG and Brownson RC (2019) Editorial: Methods and Applications in Implementation Science. Front. Public Health 7:213. doi: 10.3389/fpubh.2019.00213

Testable theories that describe the causal pathways through which implementation strategies effect change are needed to improve the outcomes produced by evidence-based interventions (EBIs). Lewis et al. advance an innovative four-step approach to building causal pathway models that articulates the mediators, moderators, preconditions, and proximal and distal outcomes of implementation processes. Such clarity in causal pathways will allow us to understand better where, when, and why strategies have an effect on outcomes of interest.

The RE-AIM framework (2) provides important guidance for planning and assessing dimensions that influence the implementation process and potential for EBIs to impact population health. Harden et al. articulate how an updated RE-AIM framework addresses emerging implementation science priorities, such as cost and adaptation, and includes a greater focus on contextual and explanatory factors. Powell et al. present a research agenda for five priorities that need to be addressed to increase the public health impact of implementation strategies: (1) enhance methods for designing and tailoring; (2) specify and test mechanisms of change; (3) conduct more effectiveness research on discrete, multifaceted, and tailored strategies; (4) increase economic evaluations; and (5) improve tracking and reporting. For economic evaluations, the range of approaches is vast, from simple costing to full cost-effectiveness analyses. Okamura et al. report on an innovative method for calculating training and consultation costs related to delivering evidence-based treatments (EBT) that may provide insight into how systems should prioritize training efforts.

Partnerships, engagement, and collaboration (PEC) are important strategies for advancing dissemination and implementation of EBIs in clinical and community settings, but conceptual models and methods to guide design and evaluation of PECs is lacking. Huang et al. conducted a scoping review of the PEC literature that identified key domains, processes, mechanisms, and strategies for PEC, and proposed a new multilevel framework to guide future research in this area.

**6**

Mazzucca et al. assessed the research designs and methodologies used in 212 dissemination and implementation (D&I) study protocols recently published in Implementation Science. While a large majority of the protocols (77%) utilized randomized designs, and most protocols (61%) proposed quantitative and qualitative methods, only 52% reported using a theoretical framework to guide the study. Northridge et al. present a protocol for a participatory, multilevel, dynamic intervention to improve the oral health of low-income Chinese Americans, guided by two complementary, multilevel frameworks: Consolidated Framework for Implementation Research (CFIR) (3) and Implementation Outcomes Framework (IOF) (4). Lee et al. utilized a novel multiphase, explanatory sequential mixed methods design to provide deeper understanding of how complex multisector partnerships impact population health outcomes in an evaluation of the Massachusetts Prevention and Wellness Trust Fund.

As per the public health adage, "what gets measured gets done," (5) progress in implementation requires the development of practical measures that are both reliable and valid. Budd et al. developed and tested a tool for measuring the contextual factors related to evidence-based practice across four countries (Australia, Brazil, China, and the United States), and found variability in reliability across domain and country, suggesting that some items are highly generalizable, while others are less so. Dearing conducted a review of 30 available organizational readiness tools, noting that even as most measure capacity, few measure organizational motivation. Helfrich et al. assessed organizational readiness to change over two waves in a workplace health promotion trial, and found that change commitment declined significantly at both intervention and control sites over time, even as wellness-program effort increased significantly at intervention sites.

Adapting EBIs to the local context is a necessary step to facilitate adoption and implementation. Approaches are needed that promote a systematic approach to documenting and evaluating the adaptation process. Rabin et al. make an important contribution by describing a multilevel, multimethod adaptation approach across four health systems, guided by the Stirman framework (6) for adaptation and modification and

#### REFERENCES

expanded using concepts from the RE-AIM framework (2). The modified adaptation model showed promise in capturing adaptation across a range of projects and content areas. To scale-up an evidence-based parenting program for prevention of pediatric obesity, Smith et al. report on the multiyear process of adaptation to a new clinical target and service delivery system. In a study of behavioral health treatment, Patel et al. apply an instructional design framework in the development and evaluation of e-learning modules as either a single component or one strategy in a multifaceted approach for training in evidencebased practices (EBPs).

Detailed specification of implementation strategies is a challenge, especially for complex, multilevel interventions that use multiple strategies. Huynh et al. describe a five-step method for mapping intervention strategies and demonstrate its use with a study of the implementation of a cardiovascular toolkit. Fernandez et al. introduce Implementation Mapping, which provides a systematic process for developing strategies to improve the adoption, implementation, and maintenance of evidence-based interventions in real-world settings. Pullmann et al. report on findings from a study of the impact of clinical supervision to improve the adoption of EBT for child mental health problems. Findings point to the importance of a supportive organizational climate in predicting supervisory EBT intensity.

Brookman-Frazee et al. contribute to the limited research on EBP sustainment in mental health services long after implementation, illustrating a novel application of survival analysis to administrative claims data in system-driven implementation of multiple EBPs. Finally, Palinkas et al. point to opportunities for using agency leader models to develop strategies to facilitate implementation of evidence-based and innovative practices for children and adolescents, guided by the Stages of Implementation Completion framework (7). Our hope is that this collection advances the field.

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Northridge, Shelley, Rundall and Brownson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# From classification to causality: Advancing Understanding of Mechanisms of change in implementation science

*Cara C. Lewis1,2,3\*† , Predrag Klasnja1†, Byron J. Powell4 , Aaron R. Lyon3 , Leah Tuzzio1 , Salene Jones 5 , Callie Walsh-Bailey1 and Bryan Weiner 6*

*1Kaiser Permanente Washington Health Research Institute, Seattle, WA, United States, 2Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, United States, 3Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, United States, 4Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States, 5Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States, 6Department of Global Health, University of Washington, Seattle, WA, United States*

#### *Edited by:*

*Thomas Rundall, University of California, Berkeley, United States*

#### *Reviewed by:*

*Carolyn Berry, New York University, United States Thomas J. Waltz, Eastern Michigan University, United States*

*\*Correspondence:*

*Cara C. Lewis<EMAIL_ADDRESS>† Joint first authorship.*

#### *Specialty section:*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health*

*Received: 01 December 2017 Accepted: 20 April 2018 Published: 07 May 2018*

#### *Citation:*

*Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, Walsh-Bailey C and Weiner B (2018) From Classification to Causality: Advancing Understanding of Mechanisms of Change in Implementation Science. Front. Public Health 6:136. doi: 10.3389/fpubh.2018.00136*

Background: The science of implementation has offered little toward understanding *how* different implementation strategies work. To improve outcomes of implementation efforts, the field needs precise, testable theories that describe the causal pathways through which implementation strategies function. In this perspective piece, we describe a four-step approach to developing causal pathway models for implementation strategies.

Building causal models: First, it is important to ensure that implementation strategies are appropriately specified. Some strategies in published compilations are well defined but may not be specified in terms of its core component that can have a reliable and measureable impact. Second, linkages between strategies and mechanisms need to be generated. Existing compilations do not offer mechanisms by which strategies act, or the processes or events through which an implementation strategy operates to affect desired implementation outcomes. Third, it is critical to identify proximal and distal outcomes the strategy is theorized to impact, with the former being direct, measurable products of the strategy and the latter being one of eight implementation outcomes (1). Finally, articulating effect modifiers, like preconditions and moderators, allow for an understanding of where, when, and why strategies have an effect on outcomes of interest.

Future directions: We argue for greater precision in use of terms for factors implicated in implementation processes; development of guidelines for selecting research design and study plans that account for practical constructs and allow for the study of mechanisms; psychometrically strong and pragmatic measures of mechanisms; and more robust curation of evidence for knowledge transfer and use.

Keywords: implementation, mechanism, mediator, moderator, theory, causal pathway, strategy

### BACKGROUND: WHY BUILD CAUSAL PATHWAY MODELS?

In recent years, there has been growing recognition of the importance of implementing evidencebased practices as a way to improve the quality of health care and public health. However, the results of implementation efforts have been mixed. About two-thirds of efforts fail to achieve the intended change (2), and nearly half have no effect on outcomes of interest (3). Implementation strategies are often mismatched to barriers [e.g., training, a strategy that could affect implementation outcomes through changes in an individual's knowledge (intrapersonal-level), is used inappropriately to address an organizational-level barrier like poor culture] (4), and implementation efforts are increasingly complex and costly without enhanced impact (5). These suboptimal outcomes are due, in large part, to the dearth of tested theory in the field of implementation science (6). In particular, the field has a limited understanding of *how* different implementation strategies work—the specific causal mechanisms through which implementation strategies influence care delivery [7; Lewis et al. (under review)1 ]. As a consequence, implementation science has been limited in its ability to effectively inform implementation practice by providing guidance about when and in what contexts specific implementation strategies should be used and, just as importantly, when they should not.

The National Academy of Science defines "science" as "the use of evidence to construct testable explanations and predictions of natural phenomena, as well as the knowledge generated through this process." (8) The field of implementation has spent the past two decades building and organizing knowledge, but we are far from having testable explanations that afford us the ability to generate predictions. To improve outcomes of implementation efforts, the field needs testable theories that describe the causal pathways through which implementation strategies function (6, 9). Unlike frameworks, which offer a basic conceptual structure underlying a system or concept (10), theories provide a testable way of explaining phenomena by specifying relations among variables, thus enabling prediction of outcomes (10, 11).

Causal pathway models represent interrelations among variables and outcomes of interest in a given context (i.e., the building blocks of implementation theory). Specifying the structure of causal relations enables scientists to empirically test whether the implementation strategies are operating *via* theorized mechanisms, how contextual factors moderate the causal processes through which implementation strategies operate, and how much variance in outcomes is accounted for by those mechanisms. Findings from studies based on causal models can, over time, both help the field develop more robust theories about implementation processes and advance the practice of implementation by addressing key issues. For instance, causal models can do the following: (1) inform the development of improved implementation strategies, (2) identify mutable targets for new strategies, (3) increase the impact of existing strategies, and (4) prioritize which strategies to use in which contexts.

In this perspective piece, we propose an approach to theory development by specifying, in the form of causal pathway models, hypotheses about the causal operation of different implementation strategies in various settings, so that these hypotheses can be tested and refined. Specifically, we offer a four-step process to developing causal pathway models for implementation strategies. Toward this end, we argue the field must move beyond having lists of variables that can rightly be considered determinants [i.e., factors that obstruct or enable change in provider behavior or health-care delivery processes (12)], and toward precise

articulation of mediators, moderators, preconditions, and (proximal versus distal) outcomes (see **Table 1** for definitions).

#### BUILDING CAUSAL PATHWAY MODELS

Our perspective draws upon Agile Science (13, 14)—a new method for developing and studying behavioral interventions that focuses on intervention modularity, causal modeling, and efficient evaluations to generate empirical evidence with clear boundary conditions (in terms of population, context, behavior, etc.) to maximize knowledge accumulation and repurposing. Agile Science has been used to investigate goal-setting interventions for physical activity, engagement strategies for mobile health applications, depression interventions for primary care, and automated dietary cues to promote weight loss (13, 15). Applied to implementation strategies, Agile Science-informed causal pathway diagram modeling consists of at least four steps: (1) specifying implementation strategies; (2) generating strategy-mechanism linkages; (3) identifying proximal and distal outcomes; and (4) articulating moderators and preconditions. To demonstrate this approach, we offer examples of causal pathway models for a set of three diverse implementation strategies (see **Figure 1**). The strategies are drawn from the following example. A community mental health center is planning to implement measurement-based care in which providers solicit patient-reported outcome data [e.g., Patient Health Questionnaire 9-item depression symptom severity measure (16)] prior to clinical encounters to inform treatment (17). The community mental health center plans to use training, financial penalty (disincentives), and audit and feedback as they are common strategies used to support measurement-based care implementation (18).

#### Step 1: Specifying Implementation Strategies

The Expert Recommendations for Implementing Change study yielded a compilation of 73 implementation strategies (19)

1Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas RS, Mittman B, et al. A systematic review of empirical studies examining mechanisms of dissemination and implementation in health. *Implement Sci* (under review).

developed by a multidisciplinary team through a structured literature review (20), Delphi process, and concept mapping exercise (19, 21, 22). Thus, there exists a solid foundation of strategies that are conceptually clear and well defined. However, the compilation was never explicitly linked to mechanisms. Following Kazdin (7), we define "mechanisms" as the processes or events through which an implementation strategy operates to effect desired implementation outcomes. Upon careful examination, it seems many strategies are not well enough specified to be linked to mechanisms in a coherent manner, a key step in causal model building. For instance, the compilation of 73 strategies lists "learning collaboratives," a general approach for which the discrete strategies or core components are underspecified. This makes it difficult to identify their precise mechanisms of action (23). Underspecified strategies also leave the field vulnerable to inappropriately synthesizing data across studies (24, 25).

In our case example, training is a strategy that is underspecified. We adapted procedures from Michie et al. (26) to guide strategy specification recommending that each strategy be assessed for whether it: (1) aims to promote the adoption, implementation, sustainment, or scale-up of an evidence-based practice; (2) is a proposed "active ingredient" of adoption, implementation, sustainment, or scale-up; (3) represents the smallest component while retaining the proposed active ingredient; (4) can be used alone or in combination with other discrete strategies; (5) is observable and replicable; and (6) can have a measureable impact on specified mechanisms of implementation (and, if so, whether putative mechanisms can be listed). If strategies do not meet these criteria, they require revision and further specification. This could involve suggesting alternative definitions, eliminating an implementation strategy altogether, or articulating a new, narrower strategy that is a component or a type of the original strategy. Training would meet all but the third and sixth criteria (listed previously), because training can be comprised of several active ingredients (e.g., didactics, modeling, role play/rehearsal, feedback, shadowing) each of which may operate on an unique mechanism. In this case, training ought to be more narrowly defined to make clear its core components.

### Step 2: Generating Strategy-Mechanism Linkages

Once specified, an implementation strategy needs to be linked to the mechanisms hypothesized to underlie its functioning. Mechanisms explain *how* an implementation strategy has an effect by describing the actions that lead from the administration of the strategy to the implementation outcomes (see **Table 1** for definitions). Statistically speaking, mechanisms are always mediators, but mediators may not be mechanisms. Similarly, moderators can point toward mechanisms but are not themselves reliably mechanisms. Determinants may explain why an implementation strategy did or did not have an effect, but mechanisms explain *how* a strategy had an effect, by, for example, altering the status of a determinant. Determinants are naturally occurring, and often but not always, malleable factors that could prevent or enable the strategy to affect the desired outcomes. Mechanisms are intentionally activated by the application of an implementation strategy and can operate at different levels of analysis, such as at the levels of intrapersonal (e.g., learning), interpersonal (e.g., sharing), organizational (e.g., leading), community (e.g., restructuring), and macro policy (e.g., guiding) (27). For an implementation effort to be successful, chosen strategies should be compatible with and able to act on the local determinants [e.g., provider habit (determinant) is addressed with clinical decision support (strategy) *via* self-reflection/reflecting (mechanism)]. Although commonly used in implementation science, we propose that the notion of a determinant is insufficiently specific as researchers have used it to refer to at least two types of variables in a causal process: proximal outcomes and effect modifiers (see text footnote 1). Our discussion below uses these more precise terms instead.

Most implementation strategies likely act *via* multiple mechanisms, although it remains an empirical question whether one mechanism is primary and others are ancillary. It is also likely that the same mechanism might be involved in the operation of multiple implementation strategies. Initial assessment of strategy-mechanism linkages is made in the context of the broader scientific knowledge base about how a strategy produces an outcome (7). For instance, many strategies have their own literature base (e.g., audit and feedback) (28) that offer theoretical and empirical insights about which mechanisms might be underlying the functioning of those strategies [e.g., reflecting, learning, and engaging (28)]. Effort should always be made to draw upon and test existing theories, but if none offer sufficient guidance, hypothesizing variables that may have causal influence remains critical. In this way, over time, the initially formulated strategy-mechanism linkages can be reassessed and refined as studies begin to test them empirically. While such empirical evaluations are currently rare—across two systematic reviews of implementation mechanisms, only 31 studies were identified and no mechanisms were empirically established (see text footnote 1; 29)—the causal pathway models we propose here are explicitly intended to facilitate evaluations of the mechanistic processes through which implementation strategies operate.

### Step 3: Identifying Proximal and Distal Outcomes

Implementation scientists have isolated eight outcomes as the desired endpoints of implementation efforts: acceptability, feasibility, appropriateness, adoption, penetration, fidelity, cost, and sustainability (1). Many of these outcomes are appropriately construed as latent variables, but others are manifest/observable in nature (30); a recent systematic review offers measures of these outcomes and measure meta-data (31). In terms of the causal processes through which implementation strategies operate, these outcomes are often best conceptualized as *distal outcomes* that the implementation process is intended to achieve, and each of them may be more salient at one phase of implementation than another. For instance, with the Exploration, Preparation, Implementation, Sustainment Framework (32), acceptability of an evidence-based practice may be most salient in the exploration phase, whereas fidelity may be the goal of an implementation phase. Despite the plausible temporal interrelations among the outcomes, mounting evidence indicates that not all implementation strategies influence each of the aforementioned outcomes (e.g., workshop training can influence adoption but not fidelity) (33). To fully establish the plausibility of an implementation mechanism and a testable causal pathway, proximal outcomes must be expounded.

Proximal outcomes are direct, measurable, and typically observable, products of the implementation strategy that occur because of its specific mechanism of action. That is, affecting a proximal outcome in the intended direction can confirm/disconfirm activation of the putative mechanism, offering a low-inference way to establish evidence for a theorized mechanism. Most often, mechanisms themselves cannot be directly measured, forcing (either high-inference assessment or) reliance on the observation of change in a proximal outcome of interest. For instance, didactic education, as an active ingredient of training, acts primarily through the mechanism of learning on the proximal outcome of knowledge to influence the distal implementation outcome of perceived acceptability or even adoption. Practice with feedback acts through the mechanism of reflecting on proximal outcomes of skills and confidence to influence the distal implementation outcome of adoption or even fidelity. To identify proximal outcomes, one must answer the question, "How will I know if this implementation strategy had an effect *via* the mechanism that I think it is activating?" or "What will be different if the hypothesized mechanisms for this strategy is at play?" It is very common for mechanisms and proximal outcomes to be conflated in the literature given that researchers often test mediation models examining the impact of a strategy on a distal implementation outcome *via* a more proximal outcome. The way we are using the terms, a mechanism is a process through which an implementation strategy operates, and a proximal outcome is a measurable effect of that process that is in the causal pathway toward the distal implementation outcomes.

#### Step 4: Articulating Effect Modifiers

Finally, there are two types of effect modifiers that are important to articulate, both of which can occur across multiple levels of analysis: moderators and preconditions. Moderators are factors that increase or decrease the level of influence of an implementation strategy on an outcome. See **Figure 1** in which an example for intra-individual and organizational-level moderators for audit and feedback are articulated. Theoretically, moderators are factors that interact with a strategy's mechanism of action, even if exactly how they interact mechanistically are not understood. Preconditions are factors that are necessary for an implementation mechanism to be activated at all (see **Figure 1**). They are necessary conditions that need to be in place for the causal process that leads from an implementation strategy to its proximal and distal outcomes to take place. Both moderators and preconditions are most often mischaracterized as "determinants" in the implementation science literature base, which may limit our ability to understand the nature of the relations between a strategy and the individual and contextual factors that modify its effects, and, in turn, where, when, and why strategies have an effect on outcomes of interest.

### FUTURE DIRECTIONS: WHAT THE FIELD OF IMPLEMENTATION NEEDS TO FULLY ESTABLISH ITSELF AS A SCIENCE

In order to fully establish itself as a science by offering testable explanations and enabling the generation of predictions, we offer four critical steps for the field of implementation: (1) specify implementation strategies; (2) generate implementation strategymechanism linkages; (3) identify proximal and distal outcomes; and (4) articulate effect modifiers. In addition to these steps, we suggest

### REFERENCES

that future research should strive for the generation of precise terms for factors implicated in implementation processes and use them consistently across studies. In a systematic review of implementation mechanisms, researchers conflated preconditions, predictors, moderators, mediators, and proximal outcomes (see text footnote 1). In addition, there is room for the field to develop guidelines for selecting research designs and study plans that account for practical constraints of the contexts in which implementation is studied *and* allow for mechanism evaluation. The types of causal pathway models that we advocated for here, paired with an understanding of the constraints of a particular study site, would enable researchers to select appropriate methods and designs to evaluate hypothesized relations by carefully considering the temporal dynamics such as how often a mechanism should be measured and how much the outcome is expected to change and when.

In order to truly advance the field, much work needs to be done to identify or develop psychometrically strong and pragmatic measures of implementation mechanisms. Empirically evaluating causal pathway models requires psychometrically strong measures of mechanisms that are also pragmatic, yet none of the seven published reviews of implementation-relevant measures focus on mechanisms. It is likely that measure development will be necessary to advance the field. Finally, implementation science could benefit from the building of more robust curation of evidence for knowledge transfer and use. Other fields house web-based databases for collecting, organizing, and synthesizing empirical findings [e.g., Science of Behavior Change (34)]. In doing so, fields can accumulate knowledge more rapidly and users of knowledge can determine what is working, when, and why, as well as what generalizes and what does not. Such curation of evidence can more efficiently lead to the development of improved implementation strategies (e.g., through strategy specification), identification of mutable targets for new strategies (e.g., mechanisms revealed for existing strategies that may not be pragmatic), and prioritization of strategy use for a given context (e.g., given knowledge of preconditions and moderators).

## AUTHOR CONTRIBUTIONS

CL and PK are co-first authors, who co-led manuscript development. CL and BW are co-PIs on an R01 proposal that led to the inception of this manuscript. All authors (CL, PK, BP, AL, LT, SJ, CW-B, and BW) contributed to idea development, writing, and editing of this manuscript and agreed with its content.

## ACKNOWLEDGMENT

BP would like to acknowledge funding from the National Institute of Mental Health (K01MH113806).

consolidated framework for advancing implementation science. *Implement Sci* (2009) 4:50. doi:10.1186/1748-5908-4-50

Expert Recommendations for Implementing Change (ERIC) study. *Implement Sci* (2015) 10:109. doi:10.1186/s13012-015-0295-0

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer TW declared a past co-authorship with one of the author BP to the handling Editor.

*Copyright © 2018 Lewis, Klasnja, Powell, Lyon, Tuzzio, Jones, Walsh-Bailey and Weiner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

*Samantha M. Harden1 \*, Matthew Lee Smith2,3,4, Marcia G. Ory 2,3, Renae L. Smith-Ray5 , Paul A. Estabrooks 6 and Russell E. Glasgow7*

*1Physical Activity Research and Community Implementation, Human Nutrition, Foods, and Exercise, Virginia Tech, Blacksburg, VA, United States, 2Center for Population Health and Management, Texas A&M University, College Station, TX, United States, 3Department of Environmental and Occupational Health, School of Public Health, Texas A&M University, College Station, TX, United States, 4Department of Health Promotion and Behavior, College of Public Health, The University of Georgia, Athens, GA, United States, 5Walgreens Center for Health and Wellbeing Research, Deerfield, IL, United States, 6Department of Health Promotion, College of Public Health, University of Nebraska Medical Center, Omaha, NE, United States, 7Department of Family Medicine, School of Medicine, University of Colorado, Aurora, IL, United States*

#### *Edited by:*

*Mary Evelyn Northridge, New York University, United States*

#### *Reviewed by:*

*Melissa Bopp, Pennsylvania State University, United States Katie M. Heinrich, Kansas State University, United States*

*\*Correspondence:*

*Samantha M. Harden<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health*

> *Received: 04 January 2018 Accepted: 22 February 2018 Published: 22 March 2018*

#### *Citation:*

*Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks PA and Glasgow RE (2018) RE-AIM in Clinical, Community, and Corporate Settings: Perspectives, Strategies, and Recommendations to Enhance Public Health Impact. Front. Public Health 6:71. doi: 10.3389/fpubh.2018.00071*

The RE-AIM Framework is a planning and evaluation model that has been used in a variety of settings to address various programmatic, environmental, and policy innovations for improving population health. In addition to the broad application and diverse use of the framework, there are lessons learned and recommendations for the future use of the framework across clinical, community, and corporate settings. The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how the RE-AIM concepts and elements within each dimension can be applied by researchers and practitioners in diverse settings, among diverse populations and for diverse health topics.

Keywords: translation, health promotion, knowledge transfer, implementation science, evaluation framework, dissemination and implementation research

## INTRODUCTION

Dissemination and implementation (D&I) research addresses the "how and why" related to strategies for information sharing (dissemination) and intervention integration (implementation) for the purposes of enhancing evidence-based program delivery and population health (1–5). The advancement of D&I science requires a focus on the wide-scale adoption, implementation, and generalizability of program and policy impacts. With well over 100 different models and frameworks utilized in the field (6), researchers and practitioners can become overwhelmed when selecting (and attempting to apply) the most appropriate model/framework for their scientific inquiry or initiative.1

The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of

1www.dissemination-implementation.org.

RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM elements in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how RE-AIM concepts and elements can be applied by researchers and practitioners in diverse settings, among diverse populations, and for diverse health topics.

### THE RE-AIM FRAMEWORK

The RE-AIM Framework (7, 8) is often used in D&I research (9, 10), which encompasses essential translational research elements. RE-AIM was identified as the most frequently used model or framework between 2000 and 2016 for D&I grant applications submitted to the National Institutes of Health and Centers for Disease Control and Prevention (CDC) (11). This widespread use is, in part, due to the flexibility to address different public health concerns in a practical manner understandable by practitioners and policy makers. The acronym RE-AIM stands for *r*each (How do I reach those who need a specific intervention?), *e*fficacy/*e*ffectiveness (How do I know my intervention is working?), *a*doption (How do I design for dissemination and develop organizational support to deliver my intervention?), *i*mplementation (How do I ensure the intervention is feasible and delivered properly?), and *m*aintenance (How do I ensure long-term benefits and institutionalization of the intervention and continued community capacity for D&I?).

Applying RE-AIM challenges researchers and practitioners to ask fundamental questions about complex issues before, during, and after the implementation of a putative program in "real world" settings. Among the many strengths of RE-AIM is its robust structure that facilitates broad use across settings (e.g., organization, regional, rural), populations (e.g., age, race/ ethnicity, occupation/role), topics (e.g., disease, behavior), and interventions (e.g., demonstration, experimental, translational, longitudinal, multi-level). While the basic RE-AIM dimensions have remained constant since its development in the 1990s (7), its use has evolved over time with new applications in clinical (12), community (13), and corporate (14) settings. A recent systematic review (15) reported health-care (49%) and community (46%) settings applied RE-AIM in empirical or evaluative interventions most frequently; however, no such interventions were reported in corporate settings. As such, efforts are needed to understand the use of RE-AIM in multiple settings. Researchers and practitioners are encouraged to use the RE-AIM framework for beginning with the end in mind, designing for dissemination, and evaluating relevant dimensions across intervention and setting factors. Such deliberate RE-AIM application will contribute to the replicability and generalizability of planned interventions and thus yield optimal public health impact.

#### PRAGMATIC USE OF RE-AIM FOR PLANNING AND EVALUATION

The RE-AIM Framework can be used to direct the planning of new or ongoing interventions and systematic evaluations that include a complex interplay of individual and organizational outcomes (10). Fully employing RE-AIM can speed the translation of effective interventions in practice settings, while demonstrating impact and representativeness (9, 10). Yet, utilizing the full framework may require substantial human, data, and analytic resources that may not be available or feasibly acquired across typical clinical, community, or corporate settings (16). This is especially true in settings where decision-making may be based on a small subset of RE-AIM dimensions coupled with organizational priorities and resources.

Settings must consider the temporality of assessment for each RE-AIM dimension, which may need to occur prospectively, concurrently, and/or retrospectively to determine the impact of an initiative. While employing RE-AIM before an intervention begins is ideal to ensure careful and strategic local planning, in some cases this is not possible. Some organizational practices may be the result of opportunistic intervention, rollout from a central administrative site, innovation testing; corporate, policy, or organizational directive; or quality control and enhancement each of which has distinct challenges in aligning the evaluation with initiative strategies.

**Figure 1** illustrates the application of RE-AIM based on the starting temporal stage of an intervention or initiative, that is if the RE-AIM planning and evaluation is initiated before, during, or after an initiative has been completed. Each temporal starting point includes reflective processes in which researchers or practitioners can gather information (assess) and think critically about the relevance of each RE-AIM dimension (plan). Each stage also includes active processes where those applying RE-AIM can initiate and implement plans for interventions or initiatives (do), process gathered information based on predetermined criteria (evaluate), and engage partners and stakeholders in interpretation to support decision-making (report). The bidirectional arrow along the temporal stages indicate the iterative nature of these processes, each building upon one another to provide cumulative input for advancement and refinement based evolving priorities, challenges, and observed impacts (2, 17). The importance of **Figure 1** is to address the iterative nature of applying RE-AIM in planning and evaluation and how new data are taken into consideration and used to engage in a planning and action process.

As the bidirectional arrow suggests, the end of an initiative is the beginning of another (i.e., sustained implementation, adapted implementation, or implementation of an alternative solution), thus the process is cyclical and ongoing. While it is not feasible to always employ RE-AIM before an intervention or initiative begins, this figure indicates that the process can begin at any temporal stage. At all stages, researchers and practitioners are encouraged to APDER: Assess (using relevant RE-AIM dimensions and available data); Plan (based on best science, program priorities, stakeholder and organizational values, and available resources); Do (based on predetermined plans using defined procedures/protocols and supporting appropriate adaptations as needed during implementation); Evaluate (based on criteria necessary for decision-making and iterative adjustment); and Report (to, and plan for follow-up with, key stakeholders).

The RE-AIM website2 hosts a planning and evaluation document, which includes prompts and considerations across all five RE-AIM dimensions by temporal stage within a project,3 which is also available as a supplemental table to this manuscript (see Appendix A in Supplementary Material). Selected examples of common pragmatic considerations are described below.

Engaging key stakeholders (e.g., policy makers, service delivery personnel, members from the population intended to benefit from the work) is important for guiding pragmatic evaluations using RE-AIM. Researchers and practitioners should partner with organizational decision-makers to identify the necessary information required to determine priorities, justify the need for intervention, sustain implementation, and/or broaden adoption. For example, if a strategy is delivered by a single organization with a centralized delivery infrastructure, issues related to reach and effectiveness (as well as implementation costs and sustainability) may be more relevant than adoption (18). Conversely, when attempting to scale-up or scale-out an effective intervention across a number of sites (within or across organizations), issues related to implementation quality/fidelity and adoption may be considered more important than documenting the intervention's effectiveness in new and diverse settings (19, 20).

Pragmatically measuring RE-AIM outcomes (21) includes leveraging data already collected within the organizational setting to reduce evaluation costs and enhance local relevance. For example, imagine a health-care system will employ a multi-leveled intervention to enhance diabetes control by promoting physical activity. The intervention includes screening, brief counseling, referral to internal or external resources for physical activity. A pragmatic evaluation of this approach may include using electronic health records to assess the reach and representativeness of participants, changes in physical activity based on clinical screenings over time, and the number of referrals made (22). Based on priorities and available resources, it may be less pragmatic for the health-care system to assess patients' use of external resources for physical activity or their actual physical activity levels. However, if a similar multi-level intervention were implemented in a community setting, accessing electronic health records may be politically, legally or cost-prohibitive, or less relevant; rather, documenting participants' physical activity with pedometers/ accelerometers and tracking facility utilization are prioritized.

Available resources for evaluation are often limited in "real world" non-academic community and clinical settings. In most settings, resources are allocated to the intervention's delivery and management to maximize enrollment/engagement. Therefore, the pragmatic selection and use of existing measures is helpful to reduce data collection burden. However, the use of existing measures can also introduce resource needs associated with data extraction, case de-identification, and statistical analyses and data management that may exceed organizational skillsets and typical reporting procedures.

### EXAMPLES OF RE-AIM IN DIFFERENT SETTINGS

In this section, we provide examples of RE-AIM application in three major types of settings. In addition to these examples, **Table 1** contains additional recommendations for using RE-AIM by temporal stages of an intervention (i.e., before, during, after) across clinical, community, and corporate settings. The purpose of this table is to document the consistency of topics to be considered when applying RE-AIM across settings, while highlighting

2www.re-aim.org.

3http://re-aim.org/wp-content/uploads/2016/08/Planning-and-Evaluation-Tool. pdf.

(*Continued*)

for long-term maintenance

*\*HIPPA, health insurance portability and accountability act; BAA, business associate agreement; DUA, data use agreement.*

the unique factors framing the contextualization of RE-AIM within settings.

### Clinical Health-care Setting

Esteemed professional organizations and societies (e.g., The Institute of Medicine, National Academies of Medicine, and Society of Behavioral Medicine) have called for health systems to assess key health behaviors, mental health, and social measures, and address an actionable set of social determinants of health. Leveraging these opportunities, the My Own Health Report (MOHR) consortium tested a brief, evidence-based online and interactive health risk assessment and feedback tool (MyOwnHealthReport.org). The online aid included patientreported items on health risk behaviors, mental health, substance use, demographics, and patient preferences (23).

The MOHR project tested the interactive patient-report and feedback system in a cluster randomized trial of 18 primary care clinics across five states. RE-AIM was used to plan, adapt, and evaluate the system using a low-cost pragmatic implementation strategy. RE-AIM was used in the planning stages to develop strategies feasible for low-resource settings with patients most in need (e.g., federally qualified health centers and other diverse clinics including rural, suburban, and urban clinics). Inclusion criteria were purposively broad for clinics and patients, and time demands on patients and staff were kept to a minimum. The implementation plan involved a high degree of flexibility and allowed each clinic to recruit patients, administer the MOHR, simultaneously provide feedback, use assessment/feedback modalities, select languages (English or Spanish), and place in their clinic workflow. In terms of RE-AIM, this plan addressed reach, adoption, and implementation issues.

RE-AIM was used iteratively to monitor and adjust recruitment strategies (*r*each) and feedback and goal setting print-out delivery to patients and health-care team members (*i*mplementation). Content on print-outs were reinforced by practical webinars providing training about motivational interviewing and collaborative goal setting. The intervention was purposefully brief, low-cost (publicly available), and addressed impact (*e*ffectiveness) through standardized assessment and feedback content (23).

Results are summarized elsewhere (24), but in brief, the intervention produced high levels of reach (49% of all eligible patients, including those not contacted), adoption (18 of 30 diverse, lowincome clinics approached participated), implementation (all eight risk factors assessed significantly more often in intervention patients; assessment, and print-outs delivered consistently), and effectiveness (intervention superior to randomized paired control clinics on goal setting for 6 of 8 behaviors and changes on 5 of the 8 health behavior and mental health issues). The program was not, however, *m*aintained in any of the settings following conclusion of the study.

To achieve high levels of reach, adoption, and implementation, it was necessary to allow considerable flexibility and customization about how the MOHR was delivered while keeping the content of the intervention standard (23–25). The study was conducted inexpensively and rapidly by the standards of controlled trials (25) and demonstrated use of RE-AIM for planning, adaptation, and evaluation. The lack of setting maintenance was due to the inability to integrate the intervention into the existing health records (several different EHR systems were used) and intervention costs while modest (primarily staff time) that exceeded reimbursement provided by Medicare for annual wellness exams.

### Community Setting

The RE-AIM framework was adopted in the mid-2000s for use by community-based grantees in the aging services and public health networks funded through the Administration for Community Living (26). Use of RE-AIM was part of the grant solicitation, and state grantees were expected to employ RE-AIM in their planning and evaluation of selected evidence-based interventions for managing chronic conditions. RE-AIM was chosen because of its alignment with funder goals to: "(1) develop the systems necessary to support the ongoing implementation and sustainability of evidence-based programs for older adults; (2) develop multi-sector community partnerships to enhance program accessibility and extend program capacity; (3) reach the maximum number of at-risk older adults who could benefit from the programs; and (4) deliver evidence-based programs with fidelity" (27). Consultants from the CDC Healthy Aging Research Network (28) provided technical assistance to the grantees (spanning 27 states), who were primarily aging services or public health practitioners, about how RE-AIM elements could be incorporated into their grant processes.

A questionnaire was administered to state grantees to assess the utility of the RE-AIM framework and the integration of RE-AIM elements into different planning, implementation, evaluation, and monitoring processes. Grantees reported RE-AIM was useful for planning, implementation, and evaluation and relevant for various stakeholders (e.g., evaluators, providers, community leaders, and policy makers) (26). For example, RE-AIM influenced grantee decisions about program selection, target populations, and assessment/evaluation tools. Despite the availability of technical assistance, some respondents reported difficulties in use of RE-AIM, especially adopting the framework as a whole. It was not clear if findings reflected grantees' preferences for adopting single RE-AIM elements over the framework as a whole or if they lacked resources needed to fully assess and track all RE-AIM dimensions.

Over the past decade, RE-AIM utilization has been encouraged in other national-, state-, and local-level community-based initiatives designed to improve the healthy aging. Examples include the CDC's Initiatives on Assuring Healthy Caregivers (29), Health Foundation of South Florida Healthy Aging Regional Collaborative (30), and the United Way Healthy Aging and Independent Living Initiative (31).

The RE-AIM framework has been valuable for helping community practitioners ask important questions during program planning, implementation, dissemination, and evaluation. However, there is often more use of and adherence to the individual RE-AIM concepts than the model as a whole, which is complicated by the changing lexicon in the field. For example, although the concepts remain consistent, recent federal aging initiatives use terms such as "scalability" and "sustainability" instead of "reach" and "maintenance." Involvement in these aging initiatives reinforces the strong commonality between the study of aging and the RE-AIM framework: both are dynamic processes, evolving over time, and changing with the social context. For continued relevance, frameworks need to be pragmatic, fluid, and adaptable. It is a testimony to RE-AIM that its basic concepts are now mainstreamed and widely integrated into community practice.

### Corporate Setting

While theoretically as relevant and useful to corporations, the uptake of RE-AIM in corporate settings has been less frequent relative to application in clinical and community settings. Similar to other settings, corporate settings are interested in offering evidence-based programs to their consumers because programs with demonstrated *e*fficacy/*ef*fectiveness are most likely to result in positive outcomes, which ultimately satisfies key consumers and stakeholders, and sustains programs (*m*aintenance). Large corporations can have substantial *r*each because of their infrastructure and support resources (*i*mplementation) that enable rapid employment and embedding of the RE-AIM dimensions. This infrastructure allows for systematic program adoption, dissemination, and implementation supported by centralized communication channels and support staff.

The relevance and usefulness of RE-AIM in corporate settings can be demonstrated by closely examining one large US-based corporation, Walgreens. With its 8,175 locations across the US and 87 million rewards account holders, Walgreens has tremendous potential to *r*each consumers and impact public health. Even a program offered only to Walgreens' 250,000 employees can have an impact similar to implementing a program to every resident of a moderate-size city.

With an emphasis on trust, care, and accessibility, Walgreens aims to deliver programs that improve its participants' health and well-being. This is really no different than the goals of many nonprofit, community-based organizations. What is different, however, is that Walgreens' size and geographic dispersion makes the task of D&I somewhat daunting in terms of logistics and capital needed to initiate a system-wide intervention. Cost and perceived value are the primary reasons that health promotion programs are sustained or discontinued at the community- and corporatelevel (Rhodes and Glasgow, unpublished).4 For example, the incentivized digital health program—*Balance Rewards for healthy choices* (BRhc)—was implemented in 2014 as a resource-efficient solution to assist Walgreens patients track health behaviors. The value of BRhc has been demonstrated by higher adherence to hypertension and diabetes medications among its users and has shown to promote physical activity among younger adults with chronic conditions (32–34). This program has a vast reach with over one million users, and the digital format of the program moderates the ongoing costs of implementation.

Based on its unique position and infrastructure (like many large corporations), Walgreens has exceeded the capability of many health care and community organizations to deliver an

4Rhodes WRD, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: addressing economic issues most relevant to patients, providers and clinics. Unpublished.

intervention with grand-scale reach, adoption, impact, and a maintained presence. However, substantial challenges still exist. Corporations need to value the initial investments and be convinced of adequate return-on-investment for thorough, consistent education and training of delivery staff to achieve reliable results over time (both clinical and financial). If programs are not selected, implemented, and evaluated with the utmost care, the potential patient- and organizational-level loss can be quite damaging. This is a powerful reason to advocate for expanding the application of RE-AIM within corporate settings. Utilizing RE-AIM in corporate settings can produce returns on financial investments while providing benefits to intended populations that are sustained over time.

#### DISCUSSION

This article provided a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation while also offering recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings. Further, this article shared perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within different settings. Due to nature and restrictions of perspective articles, we focused on limited examples of clinical, community, and corporate work. However, these detailed examples describe initial decision-making, iterative application of RE-AIM processes, and impact on public health outcomes. Similar processes can be applied in other settings for healthrelated outcomes. Notably, not all evaluations include all RE-AIM dimensions, and there is no right or wrong answer related to which dimensions on which to focus an evaluation. The primary dimensions deserving attention will vary by community, stakeholder and organizational priories and resources as well as the intervention settings, populations, desired outcomes, and topics. While the processes for reflection and action may differ between clinical, community, and clinical settings based on a unique set of priorities and logistics, the general considerations for applying RE-AIM remain common. To conclude, we discuss lessons learned and recommendations for how RE-AIM can be employed across settings to enhance population health in the future.

A fundamental issue across settings is whether to comprehensively apply the full RE-AIM framework or use a more limited and "strategic" approach to include only certain RE-AIM dimensions. This issue of a full versus pragmatic use of RE-AIM has recently been discussed in detail elsewhere (9, 10, 16, 22, 35), but this topic is especially relevant for applied and unfunded (or underfunded) clinical, community, and corporate non-research settings. For applied settings, the full RE-AIM Framework is best used initially at the outset and planning of a project, and then, select dimensions can be used during and after the program to guide implementation, evaluation, and/or reporting. Initial focus should focus on rough estimates of desired impact for each RE-AIM dimension, followed by decisions about: (A) which dimensions are most important for this project; (B) which dimensions should be measured given limited resources; and (C) which dimensions will be targeted for improvement. This type of pragmatic approach can engage key stakeholders through the use of existing data to determine intervention success (36). A pragmatic approach is intended to allow clinical, community, and corporate settings consider the entirety of the framework during planning, but then identify actionable RE-AIM information about the most relevant dimensions to determine if a given initiative should be abandoned, refined, sustained, scaled-up, or scaled-out (16).

Given challenges with funding (e.g., more competition to obtain limited resources) in clinical, community, and corporate settings, it is essential to consider strategies to reduce costs and leverage available resources. An interesting concept, frequent need, and important area of study is the "de-implementation" of programs and program elements that appear ineffective, too expensive, or produce unanticipated negative outcomes. Such issues need to be identified in "real time" so an intervention can be quickly modified or discontinued. The urgency of conserving costs and alleviating unnecessary spending (especially at the detriment of community well-being and health equity) highlights the need for ongoing reflection about the RE-AIM dimensions throughout the temporal stages of the intervention. As the RE-AIM framework is used to drive implementation efforts, the same framework can (and should) be used to guide and evaluate de-implementation efforts (37).

A new area of RE-AIM application involves its iterative use to provide ongoing, rapid assessments of progress, then using these results to guide program adaptations (38, 39). For example, early tracking of enrollment (*r*each) may reveal that key segments of the target population (e.g., low-income patients, those most at risk) are not participating in the intervention. Efforts can then be redirected (and tested) to improve subsequent participation rates. Although RE-AIM was initially used primarily for *post hoc* program evaluation, it was deemed useful for program planning starting in 2005 (40). Iterative uses of brief, practical measures of targeted RE-AIM dimensions are new and anticipated to grow, which warrants additional research in this area (2).

Our collective experience across clinical, community, and corporate settings indicates the need for greater attention to contextual factors. Often, the most efficient ways to assess contextual factors (the "how and why") are qualitative or mixedmethod approaches (41, 42). Such impressionistic approaches can be helpful to identify conditions under which a program is successful and reasons for such results. The Practical, Robust Implementation, and Sustainability Framework (PRISM) (43) extension of the RE-AIM model may be particularly useful for this purpose because it specifies contextual factor types that may be related to results about different RE-AIM dimensions.

The field of public health has evolved to accommodate changes in societal demographics, the environment, and impacts on the social determinants of health. In fact, such changes have caused new health-related issues and complications that spurned the creation of new fields (e.g., nutrigenomics, computational social science, behavioral economics). As fields advance, so do their need for sophisticated implementation and evaluation efforts to account for increasing complexity (e.g., big data from multiple sources/levels, nested influence and integrated variables, innovative intervention designs and statistical methodologies, systems issue and unanticipated consequences). We anticipate that the application of RE-AIM will expand to these new fields and offer a robust framework for advancing research, practice, and policy. For example, as new fields emerge and existing fields advance, the demand for multi-disciplinary collaboration grows. The RE-AIM Framework is recommended for use as a model to promote interprofessional education (using the community as the classroom) to train the next generation of scholars.

Finally, whereas much of the health promotion literature shows a publication bias toward initial effectiveness data only, using the RE-AIM framework increases the likelihood that that population-level public health impact is captured. Specifically, RE-AIM dimensions allow for the investigation of the degree to which an initiative can be *adopted* and delivered broadly, have the ability for *sustained* and consistent *implementation* at a reasonable cost *reach* large numbers of people especially those who can most benefit, produce *replicable* and *long-lasting* behavior changes. To assist with these challenges, there are RE-AIM planning and evaluation guides on the www.re-aim. org website (44).

#### CONCLUSION

Our experience with clinical, community, and corporate initiatives highlights the importance of several factors for promoting the use of RE-AIM dimensions and methods. Calls to action include actions to: (A) recognize that technical assistance will be important for users from clinical, community, corporate, and/or academic settings to understand each RE-AIM element and how the different elements relate to one another; (B) utilize RE-AIM as a whole, but know it is acceptable to track the most relevant individual elements based on local interests and resources; and (C) give attention to common RE-AIM concepts and elements

#### REFERENCES

within each dimension—as well as potential measures—to bridge interventions across various clinical, community and corporate settings.

#### AUTHOR CONTRIBUTIONS

All authors contributed to the conceptualization of the manuscript and its content. All authors contributed to the full manuscript as well as reviewed and approved the final version of the manuscript.

### FUNDING

The authors would like to acknowledge all members of the National Working Group on RE-AIM Planning and Evaluation Framework (www.re-aim.org). The authors would also like to acknowledge funding support for author contributions: MO and MS contributions supported through ACL SUSTAIN for Better Health and Health Care for Older Adults 90CS0065-01. PE contributions supported by Great Plains IDEA CTR U54 GM115458- 01. RE contributions partially supported by IMPlementation to Achieve Clinical Transformation (IMPACT): the Colorado Training Program from the NIH K12 HL137862.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at https://www.frontiersin.org/articles/10.3389/fpubh.2018.00071/ full#supplementary-material.

Appendix A | Planning and evaluation document, which includes prompts and considerations across all five RE-AIM dimensions by temporal stage within a project (http://www.re-aim.org/resources-and-tools/self-rating-quiz/).

**Conflict of Interest Statement:** No financial conflicts of interest to report. All authors are members of the National Working Group on RE-AIM Planning and Evaluation Framework (www.re-aim.org).

*Copyright © 2018 Harden, Smith, Ory, Smith-Ray, Estabrooks and Glasgow. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda

Byron J. Powell 1,2,3 \*, Maria E. Fernandez 4 , Nathaniel J. Williams 5 , Gregory A. Aarons 6 , Rinad S. Beidas 7,8,9, Cara C. Lewis 10, Sheena M. McHugh11 and Bryan J. Weiner 12

*1 Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States, 2 Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States, 3 Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States, 4 Center for Health Promotion and Prevention Research, School of Public Health, University of Texas Health Science Center at Houston, Houston, TX, United States, 5 School of Social Work, Boise State University, Boise, ID, United States, 6 Department of Psychiatry, University of California, San Diego, La Jolla, CA, United States, 7 Department of Psychiatry, Center for Mental Health, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States, 8 Department of Medical Ethics and Health Policy, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States, 9 Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, United States, 10 MacColl Center for Healthcare Innovation, Kaiser Permanente Washington Health Research Institute, Seattle, WA, United States, 11 School of Public Health, University College Cork, Cork, Ireland, 12 Department of Global Health, Department of Health Services, University of Washington, Seattle, WA, United States*

#### Edited by:

*Mary Evelyn Northridge, New York University, United States*

#### Reviewed by:

*Deborah Paone, Independent Researcher, Minneapolis, MN, United States Christopher Mierow Maylahn, New York State Department of Health, United States*

> \*Correspondence: *Byron J. Powell<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health*

Received: *16 October 2018* Accepted: *04 January 2019* Published: *22 January 2019*

#### Citation:

*Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM and Weiner BJ (2019) Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front. Public Health 7:3. doi: 10.3389/fpubh.2019.00003* The field of implementation science was developed to better understand the factors that facilitate or impede implementation and generate evidence for implementation strategies. In this article, we briefly review progress in implementation science, and suggest five priorities for enhancing the impact of implementation strategies. Specifically, we suggest the need to: (1) enhance methods for designing and tailoring implementation strategies; (2) specify and test mechanisms of change; (3) conduct more effectiveness research on discrete, multi-faceted, and tailored implementation strategies; (4) increase economic evaluations of implementation strategies; and (5) improve the tracking and reporting of implementation strategies. We believe that pursuing these priorities will advance implementation science by helping us to understand when, where, why, and how implementation strategies improve implementation effectiveness and subsequent health outcomes.

Keywords: implementation strategies, implementation science, designing and tailoring, mechanisms, effectiveness research, economic evaluation, reporting guidelines

### INTRODUCTION

Nearly 20 years ago, Grol and Grimshaw (1) asserted that evidence-based practice must be complemented by evidence-based implementation. The past two decades have been marked by significant progress, as the field of implementation science has worked to develop a better understanding of implementation barriers and facilitators (i.e., determinants) and generate evidence for implementation strategies (2). In this article, we briefly review progress in implementation science and suggest five priorities for enhancing the impact of implementation strategies. We draw primarily upon the healthcare, behavioral health, and social services literature. While we hope the proposed priorities are applicable to studies conducted in a wide range of contexts, we welcome discussion regarding potential applications and enhancements for contexts outside of healthcare, such as community and public health settings (3) that often involve different types of stakeholders, interventions, and implementation strategies.

Implementation strategies are methods or techniques used to improve adoption, implementation, sustainment, and scaleup of interventions (4, 5). These strategies vary in complexity, from discrete or single component strategies (6, 7) such as computerized reminders (8) or audit and feedback (9) to multifaceted implementation strategies that combine two or more discrete strategies, some of which have been branded and tested using rigorous designs [e.g., (10, 11)]. Implementation strategies can target a range of stakeholders (12) and multilevel contextual factors across different phases of implementation (13– 16). For example, strategies can address patient (17), provider (18), organizational (19), community (20, 21), policy and financing (22), or multilevel (23) factors.

Several taxonomies describe and organize the types of strategies available (6, 7, 24–26). Similarly, taxonomies of behavior change techniques (27) and methods (28) describe components of strategies at a more granular level. Both types of taxonomies promote a common language, inform implementation strategy development and evaluation by facilitating consideration of various "building blocks" or components of multifaceted and multilevel strategies, and improve the quality of reporting in research and practice.

The evidence base for implementation strategies is steadily developing. Initially, single-component, narrowly focused strategies that were effective in earlier studies were selected in subsequent studies despite differences between the clinical problems and contexts in which they were deployed (29). That approach was based on the assumption that strategies would be effective independent of the implementation problems being addressed (29). This "magic bullet" approach has led to limited success (30), prompting recognition that strategies should be selected or developed based upon a thorough understanding of context, including the causes of quality and implementation gaps, an assessment of implementation determinants, and an understanding of the mechanisms and processes needed to address them (29).

Evidence syntheses for discrete, multifaceted, and tailored implementation strategies have been conducted. The Cochrane Collaboration's Effective Practice and Organization of Care (EPOC) group has been a leader in this regard, with 132 systematic reviews of strategies such as educational meetings (31), audit and feedback (9), printed educational materials (32), and local opinion leaders (33). Grimshaw et al. (34) note that while median absolute effect sizes across implementation strategies are similar (see **Table 1**), the variation in observed effects within each strategy category suggests that effects may vary based upon whether or not they address determinants (barriers and facilitators). Indeed, determinants at multiple levels and phases may signal the need for multifaceted and tailored strategies that address key determinants (13).

While the use of multifaceted and tailored implementation strategies is intuitive and has considerable face validity (29), the evidence regarding their superiority to single-component strategies has been mixed (37, 39, 40). A review of 25 systematic reviews (39) found "no compelling evidence that multifaceted interventions are more effective than singlecomponent interventions" (p. 20). Grimshaw et al. (34) provide one possible explanation, emphasizing that the general lack of an a priori rationale for the selection of components (i.e., discrete strategies) in multifaceted implementation strategies makes it difficult to determine how these decisions were made. They may have been selected thoughtfully to address prospectively identified determinants through theoreticallyor empirically-derived change mechanisms, or they may simply be the manifestation of a "kitchen sink" approach. Wensing et al. (41) offer a complementary perspective, noting that definitions of discrete and multifaceted strategies are problematic. A discrete strategy such as outreach visits may include instruction, motivation, planning of improvement, and technical assistance; thus, it may not be accurate to characterize it as a single-component strategy. Conversely, a multifaceted strategy including educational workshops, educational materials, and webinars may only address provider knowledge and fail to address other important implementation barriers. They propose that multifaceted strategies that truly target multiple relevant implementation determinants could be more effective than single-component strategies (41).

A systematic review of 32 studies testing strategies tailored to address determinants concluded that tailored approaches to implementation were more effective than no strategy or a strategy not tailored to determinants; however, the methods used to identify and prioritize determinants and select implementation strategies were not often well-described and no specific method has been proven superior (37). The lack of systematic methods to guide this process is problematic, as evidenced by a review of 20 studies that found that implementation strategies were often poorly conceived, with mismatches between strategies and determinants (e.g., barriers were identified at the team or organizational level, but strategies were not focused on structures and processes at those levels) (42). A multi-national program of research was undertaken to improve the methods of tailoring implementation strategies (43), but tailored strategies had little impact on primary and secondary outcomes (40). Questions remain about the best methods to develop tailored implementation strategies.

Five priorities need to be addressed to increase the public health impact of implementation strategies: (1) enhance methods for designing and tailoring; (2) specify and test mechanisms of change; (3) conduct more effectiveness research on discrete, multifaceted, and tailored strategies; (4) increase economic evaluations; and (5) improve tracking and reporting. **Table 2** provides examples of studies that have pursued each priority with rigor.

TABLE 1 | Evidence for common implementation strategies targeting professional behavior change.

*Table updated from Grimshaw et al. (34), and draws upon Cochrane Reviews from the Effective Practice and Organization of Care (EPOC) group (38).*

TABLE 2 | Five priorities for research on implementation strategies.

### ENHANCE METHODS FOR DESIGNING AND TAILORING IMPLEMENTATION STRATEGIES

Implementation strategies are too often designed in an unsystematic manner and fail to address key contextual determinants (13–16). Stakeholders may rely upon inertia (i.e., "we've always done things this way"), one size fits all approaches, or utilize what Martin Eccles has called the ISLAGIATT principle (i.e., "it seemed like a good idea at the time") (53). Consequently, strategies are not always well-matched to the contexts in which they are deployed, including the interventions to be implemented, settings, stakeholder preferences, and implementation determinants (37, 42, 54). More rational, systematic approaches to identify and prioritize barriers and link strategies to overcome them are needed (37, 42, 55–57). A number of methods have been suggested. Colquhoun and colleagues (56) found 15 articles with replicable methods for designing strategies to change healthcare professionals' behavior, and Powell et al. (55) proposed Intervention Mapping (58), concept mapping (59), conjoint analysis (60), and system dynamics modeling (61) as methods to aid the design, selection, and tailoring of strategies. These methods share common steps (identification of barriers, linking barriers to strategy component selection, use of theory, and user engagement), and have potential to make the process of designing and tailoring implementation strategies more rigorous (55, 56). For example, Intervention Mapping is step-by-step approach to developing implementation strategies using a detailed and participatory needs assessment and the identification of implementers, implementation behaviors, determinants, and ultimately, behavior change methods and implementation strategies that influence determinants of implementation behaviors. Some work has been done to compare different methods for assessing determinants (62); however, several questions remain. How can determinants be accurately and efficiently assessed (ideally leveraging implementation frameworks)? Can perceived and actual determinants be differentiated? What are the best methods for prioritizing determinants that need to be proactively addressed? When should determinant assessment take place given that new challenges are likely to emerge during the course of implementation? Who should be involved in this process? Each of those questions has resource implications. Similarly, questions remain about efficiently linking prioritized determinants to effective and pragmatic implementation strategies. How can causal theory be leveraged or developed to guide the selection of implementation strategies? Can pragmatic tools be developed to systematically link strategies to determinants? Approaches to designing and tailoring implementation strategies should be tested to determine whether they improve implementation and clinical outcomes (55, 56). Given that clinical problems, clinical and public health interventions, settings, individuals, and contextual factors are highly heterogeneous, there is much to gain from developing generalizable processes for designing and tailoring strategies.

### SPECIFY AND TEST MECHANISMS OF CHANGE

Studies of implementation strategies should increasingly focus on establishing the processes and mechanisms by which strategies exert their effects rather than simply establishing whether or not they were effective (29, 63, 64). The National Institutes of Health (64) provides this guidance:

Wherever possible, studies of dissemination or implementation strategies should build knowledge both on the overall effectiveness of the strategies, as well as "how and why" they work. Data on mechanisms of action, moderators, and mediators of dissemination and implementation strategies will greatly aid decision-making on which strategies work for which interventions, in which settings, and for which populations.

Unfortunately, it is not common that mechanisms are even mentioned, much less tested (63, 65, 66). Williams (63) emphasizes the need for trials that test a wider range of multilevel mediators of implementation strategies, stronger theoretical links between strategies and hypothesized mediators, improved design and analysis of multilevel mediation models in randomized trials, and an increasing focus on identifying implementation strategies and behavior change techniques that contribute most to improvement. Developing a more nuanced understanding of mechanisms will require researchers to thoroughly assess the context of implementation and describe causal pathways by which strategies exert their effects, moving beyond a broad identification of determinants and articulating mediators, moderators, preconditions, and proximal and distal outcomes (67). Examples of this type of approach and guidance for their development can be found in Lewis et al. (67), Weiner et al. (23), Bartholomew et al. (58), and Highfield et al. (44). Additionally, drawing more heavily upon theory (66, 68, 69), using research designs that maximize ability to make causal inferences (70, 71), leveraging methods that capture and reflect the complexity of implementation such as systems science (61, 72, 73) and mixed methods (74–76) approaches, and adhering to methods standards for studies of complex interventions (77) will help to sharpen our understanding of how implementation strategies engage hypothesized mechanisms. Work to link implementation strategies and behavior change techniques to hypothesized mechanisms is underway (67, 78), which promises to improve our understanding of how, when, where, and why implementation strategies are effective.

### CONDUCT MORE EFFECTIVENESS RESEARCH ON DISCRETE, MULTI-FACETED, AND TAILORED IMPLEMENTATION STRATEGIES

There is a need for more and better effectiveness research on discrete, multifaceted, and tailored implementation strategies using a wider range of innovative designs (70, 79–82). First, while a number of discrete implementation strategies have been described (6, 7, 24, 25) and tested (38), there are gaps in our understanding about how to optimize these strategies. There are over 140 randomized trials of audit and feedback, but Ivers et al. (83) conclude that there is much to learn about when it will work best and why, and how to design reliable and effective audit and feedback strategies across different settings and providers. Audit and feedback is an example of how complex implementation strategies can be. The ICeBERG group (69) pointed to the fact that even varying five modifiable elements of audit and feedback (content, intensity, method of delivery, duration, and context) produces 288 potential combinations. These variations matter (84), and there is a need for tests of audit and feedback and other discrete implementation strategies that include clearly described components that are theoretically and empirically derived, and well-operationalized. The results of these studies could inform the use of discrete strategies and their inclusion in multifaceted strategies.

Second, there is a need for trials that give insight into the sequencing of multifaceted strategies and what to do if the first strategy fails (39). These strategies could be compared to discrete/single-component implementation strategies or multifaceted strategies of varying complexity and intensity with well-defined components that are theoretically aligned with implementation determinants. These strategies could be tested using MOST, SMART, or other variants of factorial designs that can evaluate the relative impact of various components of multifaceted strategies and inform their sequencing (70, 85).

Finally, tests of strategies that are prospectively tailored to different implementation contexts to address specific implementers, implementation behaviors, or determinants are needed (37). This work could involve comparisons between tailored and non-tailored multifaceted implementation strategies (86), as well as tests of established and innovative methods that could inform the identification, selection, and tailoring of implementation strategies (55, 56).

### INCREASE ECONOMIC EVALUATIONS OF IMPLEMENTATION STRATEGIES

Few studies include economic evaluations of implementation strategies (87, 88). For example, in a systematic review of 235 implementation studies, only 10% provided information about implementation costs (87). The dearth of economic evaluations severely limits our ability to understand which strategies might be feasible for different contexts, as some decision makers might underestimate the resources required to implement and sustain EBPs, while others might over-estimate them and preemptively limit themselves from implementing EBPs that could benefit their communities (89). Incorporating economic analyses into studies of implementation strategies would provide decision makers more complete information to guide strategy selection, and would encourage researchers to be more judicious and pragmatic in their design and selection of implementation strategies, narrowing attention to strategies and mechanisms hypothesized to be most essential. If methods for designing and tailoring strategies can be improved such that complex multifaceted strategies are proven superior to single-component or less complex multifaceted strategies (39) and tailored strategies are proven superior to more standard multifaceted strategies (37, 40, 43, 55), economic evaluations will be instrumental in demonstrating whether improvements in implementation are worth added costs. Practical tools for integrating economic evaluations within implementation studies have been developed, such as the Costs of Implementing New Strategies (COINS) method (89) which was developed to address the need for standardized methods for analyzing cost data in implementation research that extend beyond the cost of the clinical intervention itself (90). For example, the original COINS study presented a head-to-head trial of two implementation approaches; although one approach was significantly more costly, the implementation outcomes achieved were superior enough to warrant the additional resources (91). Increasing the number and relevance of economic evaluations will require the development of a common framework that promotes comparability across studies (88).

### IMPROVE TRACKING AND REPORTING OF IMPLEMENTATION STRATEGIES

Developing a robust evidence base for implementation strategies will require that their use be contemporaneously tracked and that they be reported in the literature with sufficient detail (92). It is often difficult to ascertain which implementation strategies were used and how they might be replicated. Part of the challenge is the iterative nature of implementation. Even if strategies are meticulously described in a study protocol or trial registry, it is often unrealistic to expect that they will not need to be altered as determinants emerge across implementation phases (13, 93, 94). These changes are likely to occur within and between implementing sites in research studies and applied efforts (50, 51), and without rigorous methods for tracking implementation strategy use, efforts to understand what strategies were used and whether or not they were effective are stymied. Even when strategies are reported in study protocols or empirical articles, there are numerous problems with their description, including inconsistent labeling; lack of operational definitions; poor description and absence of manuals to guide their use; and lack of a clear theoretical, empirical, or pragmatic justification for how the strategies were developed and applied (4). Poor reporting clouds the interpretation of results, precludes replication in research and practice, and limits our ability to synthesize findings across studies (4, 92). Findings from systematic reviews illustrate this problem. For example, Nadeem et al. (95) review of learning collaboratives concluded that, "reporting on specific components of the collaborative was imprecise across articles, rendering it impossible to identify active quality improvement collaborative ingredients linked to improved care."

A number of reporting guidelines could be leveraged to improve descriptions of strategies (4, 96–100). Proctor et al. (4) recommend that researchers name and define strategies in ways that are consistent with the published literature, and carefully operationalize the strategy by specifying: (1) actor(s), (2) action(s), (3) action target(s), (4) temporality, (5) dose, (6) implementation outcomes affected, and (7) theoretical, empirical, or pragmatic justification. Specifying strategies in this way has the potential to increase our understanding of not only which strategies are most effective, but more importantly, the processes and mechanisms by which they exert their effects (29, 67). Additional options that provide structured reporting recommendations include the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations (99, 100), the Simplified Framework (96) and its extension [AIMD; (97)], and the Template for Intervention Description and Replication (TIDieR) checklist (98). Though not specific to the reporting of implementation strategies, the Standards for Reporting Implementation Studies (101) and Neta et al. (102) reporting framework emphasizes how critical it is to report on the multilevel context of implementation. The use of any of the existing guidelines would enhance the clarity of strategy description. We believe that developing approaches to tracking implementation strategies (50, 51), and assessing the extent to which they are pragmatic (e.g., acceptable, compatible, easy, and useful) for both research and applied efforts is a high priority. Further, efficient ways of linking empirical studies with study protocols to gauge the degree to which strategies have been adapted or tailored over the course of an implementation effort would be helpful. Failing to improve the quality of reporting will negate other advances in this area by hindering replication.

### CONCLUSION

Implementation science has advanced considerably, yielding a more robust understanding of implementation strategies. Several resources can inform the use of implementation strategies, including established taxonomies of implementation strategies (6, 7, 24, 25) and behavior change techniques (27, 28), repositories of systematic reviews (38, 103, 104), methods for selecting and tailoring implementation strategies (40, 55, 56), and reporting guidelines that promote replicability (4, 98–100). Nevertheless, questions remain and further effectiveness research and methodological development are needed to ensure that evidence is effectively translated into public health impact. Advancing these priorities will lead to a better understanding of when, where, why,

#### REFERENCES

and how implementation strategies exert their effects (29, 63).

#### AUTHOR CONTRIBUTIONS

BP conceptualized the paper and wrote the first draft of the manuscript. All other authors contributed to the writing and approved the final manuscript.

#### FUNDING

BP was supported by grants and contracts from the NIH, including K01MH113806, R25MH104660, UL1TR002489, R01MH106510, R01MH103310, P30A1050410, and R25MH080916. NW was supported by P50MH113840 from the NIMH. RB was supported by grants from the NIMH through R21MH109878 and P50MH113840. CL was supported by R01MH106510 and R01MH103310 from the NIMH. SM was supported by a Fulbright-Health Research Board Impact Award.

intervention for evidence-based practice implementation. Implement Sci. (2015) 10:1–12. doi: 10.1186/s13012-014-0192-y

methodology-standards?utm\_source=Funding+awards%2C+GAO+Board+ deadline&utm\_campaign=Funding+awards%2C+GAO+Board+deadline& utm\_medium=email#Complex

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Powell, Fernandez, Williams, Aarons, Beidas, Lewis, McHugh and Weiner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Price per Prospective consumer of Providing Therapist Training and consultation in seven evidence-Based Treatments within a large Public Behavioral health system: an example cost-analysis Metric

*Kelsie H. Okamura1,2\*, Courtney L. Benjamin Wolk1 , Christina D. Kang-Yi1 , Rebecca Stewart1 , Ronnie M. Rubin3 , Shawna Weaver3 , Arthur C. Evans4 , Zuleyha Cidav1 , Rinad S. Beidas1 and David S. Mandell1*

#### *Edited by:*

*Ross Brownson, Washington University in St. Louis, United States*

#### *Reviewed by:*

*Ana Baumann, Washington University in St. Louis, United States Beth Prusaczyk, Vanderbilt University Medical Center, United States*

#### *\*Correspondence:*

*Kelsie H. Okamura<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health*

*Received: 17 October 2017 Accepted: 15 December 2017 Published: 08 January 2018*

#### *Citation:*

*Okamura KH, Benjamin Wolk CL, Kang-Yi CD, Stewart R, Rubin RM, Weaver S, Evans AC, Cidav Z, Beidas RS and Mandell DS (2018) The Price per Prospective Consumer of Providing Therapist Training and Consultation in Seven Evidence-Based Treatments within a Large Public Behavioral Health System: An Example Cost-Analysis Metric. Front. Public Health 5:356. doi: 10.3389/fpubh.2017.00356*

*1Department of Psychiatry, University of Pennsylvania, Philadelphia, PA, United States, 2State of Hawaii Child and Adolescent Mental Health Division, Honolulu, HI, United States, 3City of Philadelphia Department of Behavioral Health and Intellectual disAbility Services, Philadelphia, PA, United States, 4American Psychological Association, Washington, DC, United States*

Objective: Public-sector behavioral health systems seeking to implement evidence-based treatments (EBTs) may face challenges selecting EBTs given their limited resources. This study describes and illustrates one method to calculate cost related to training and consultation to assist system-level decisions about which EBTs to select.

Methods: Training, consultation, and indirect labor costs were calculated for seven commonly implemented EBTs. Using extant literature, we then estimated the diagnoses and populations for which each EBT was indicated. Diagnostic and demographic information from Medicaid claims data were obtained from a large behavioral health payer organization and used to estimate the number of covered people with whom the EBT could be used and to calculate implementation-associated costs per consumer.

results: Findings suggest substantial cost to therapists and service systems related to EBT training and consultation. Training and consultation costs varied by EBT, from Dialectical Behavior Therapy at \$238.07 to Cognitive Behavioral Therapy at \$0.18 per potential consumer served. Total cost did not correspond with the number of prospective consumers served by an EBT.

conclusion: A cost-metric that accounts for the prospective recipients of a given EBT within a given population may provide insight into how systems should prioritize training efforts. Future policy should consider the financial burden of EBT implementation in relation to the context of the population being served and begin a dialog in creating incentives for EBT use.

Keywords: evidence-based treatment, therapist, training and consultation, cost-analysis, population health

### INTRODUCTION

In recent years, many efforts to improve mental health have focused on increasing the use of evidence-based treatments (EBTs) within public-sector service systems. Therapist training is a necessary—but not sufficient—implementation strategy to increase EBT use (1). For public-sector service systems, large-scale training of therapists is often the first or only EBT implementation strategy. A combination of experiential and active learning (e.g., didactic and case consultation) tends to produce the most favorable therapist behavior change over time (2, 3). As a result, many EBT developers and certifying organizations now require that therapists receive both didactic foundational training and ongoing case consultation to be "certified" in an EBT (e.g., PCIT International).1 Training and consultation require an investment on the part of therapists, their agencies, and, especially in publicly funded systems, the city or state agency that oversees payment for care. For example, therapists and organizations may incur initial direct costs like attending week-long trainings to first learn about the EBT and subsequently participate in weekly consultation calls for 6–12 months to ensure treatment fidelity. Therapists' time required to participate often results in substantial cost to the agencies which they work (4–8). For example, Lang and Connell (6) estimated that an agency participating in a Trauma Focused-Cognitive Behavioral Therapy learning collaborative, which included agency-wide training and ongoing consultation, spent \$89,575 in direct (e.g., training) and indirect (e.g., preparation hours) costs.

While public-sector service systems have typically used other strategies to select EBT, such as stakeholder feedback in combination with federal- and/or state-policy (9, 10), the breadth of the population served, and the associated costs should be important drivers of choice. Utilizing existing service system data is important for strategic decision-making and implementation tailored to the population (11). Information about the population served is needed to make decisions about where to invest their limited resources by understanding the extent to which an EBT provides diagnostic and demographic "coverage" within a service system (9). Costs associated with EBTs are often noted as significant barriers for implementation (12–14) and thus far the cost-analysis metrics that have been used to study implementation have not considered the population coverage relative to the implementation cost (4–8). A metric that considers the potential consumer served allows for population-based and data-informed decisions when selecting the right EBT. This metric can also inform costevaluative decisions on how applicable an EBT will be for each relevant consumer within the service system.

In this study, we introduce a strategy for calculating a cost per prospective consumer metric to determine the extent to which an EBT covers a given service system. To generate this metric, population data derived from that existing service system are needed; and within behavioral health, insurance claims (15), or practice-monitoring data tied to billing (9, 16) have predominantly been used. These large person-period datasets typically contain information regarding consumer age, gender, diagnoses, service utilization, and medication prescribed. This study was conducted to demonstrate the impact of therapist training and consultation costs in a large public behavioral health system and to describe a complimentary metric for system decision-making when selecting EBT for their population. First, training and consultation requirements for certification among seven EBTs were documented. Next, training, consultation, and indirect labor costs for each EBT were calculated. Finally, the total cost of training, consultation, and indirect labor for each EBT was divided across the number of potential consumers based on diagnostic and demographic information.

### MATERIALS AND METHODS

#### Evidence-Based Treatments Identification of EBT

We identified EBTs for this study using registries created by the American Psychological Association (17), which rely on Chambless and Hollon (18) definitions of EBT. The APA's Division 12 (Society for Clinical Psychology)2 and Division 53 (Society for Child and Adolescent Clinical Psychology)3 websites were consulted to determine EBTs that fit the criteria of (a) having an in-person training, (b) ongoing consultation period, and (c) a certifying body through which therapists can become "certified" in the particular EBT. Seven EBTs were identified through these websites: (a) Cognitive Behavioral Therapy/Cognitive Therapy (19), (b) Cognitive Processing Therapy (20), (c) Dialectical Behavior Therapy (21), (d) Parent–Child Interaction Therapy (22, 23), (e) Prolonged Exposure (24, 25), (f) Modular Approach to Therapy for Children with Anxiety, Depression, Trauma, and Conduct Problems (26), and (g) Trauma Focused-Cognitive Behavioral Therapy (27).

#### Training and Consultation Cost

The cost of training and consultation was determined using information from the certifying body for each EBT (see **Table 1** for certifying bodies for each EBT). First, the certifying body's website was referenced for certification requirements, upcoming trainings, and cost associated with training, consultation, and certification. When prices were not listed, we contacted the certifying body to solicit current prices and requirements for training and consultation to obtain certification. Revenue loss was defined as the total amount of therapist hours spent on training and consultation, as opposed to providing therapy (i.e., billable hours). Hourly wage for therapists, as determined by the US Bureau of Labor Statistics for Philadelphia,4 was established as \$38.37 per hour.

#### Diagnostic and Age Applicability

To determine the population to which an EBT was applicable, diagnostic and age profiles were created for each EBT. We referenced APA's Divisions 12 and 53 websites, the credentialing body's website, and PracticeWise Evidence-based Services Database (28) to identify the studies used to establish each EBT's

2http://www.div12.org/psychological-treatments/.

3http://effectivechildtherapy.org/.

4https://www.bls.gov/bls/blswage.htm.

1http://www.pcit.org/.

#### Table 1 | EBT training requirements.

*EBT, evidence-based treatment; DBT, Dialectical Behavior Therapy; PCIT, Parent–Child Interaction Therapy; PE, Prolonged Exposure; CBT/CT, Cognitive Behavioral Therapy/ Cognitive Therapy; CPT, Cognitive Processing Therapy; MATCH, Modular Approach to Therapy for Children with Anxiety, Depression, Trauma, or Conduct Problems; TF-CBT, Trauma Focused-Cognitive Behavior Therapy.*

efficacy. For example, Division 12's website lists DBT as having Strong Research Support for Borderline Personality Disorder,5 with six efficacy trials used to determine that status. The Division 12 website also lists Strong Research Support for CBT/CT for Attention-Deficit/Hyperactivity Disorder, Insomnia, Binge Eating Disorder, Bipolar Disorders, Bulimia Nervosa, Depressive Disorders, Generalized Anxiety Disorder, Obsessive Compulsive Disorder, Social Phobia, Panic Disorder, and Schizophrenia; CPT for Posttraumatic Stress Disorder; and PE for Posttraumatic Stress Disorder. Efficacy trials for PCIT, MATCH, and TF-CBT were identified through comprehensive literature reviews cited by Division 53 (29–31) and the credentialing body's website (PCIT International, PracticeWise, and TF-CBT National Therapist Certification Program, respectively).

Efficacy trials were coded by two independent raters (Kelsie H. Okamura and Courtney L. Benjamin Wolk) for diagnosis and age range used within each trial. Coders met to regularly resolve discrepancies, using clinical judgment and the conservative criteria of only including diagnoses that the EBT was intended to treat. Specifically for youth CBT, the PracticeWise Evidencebased Services Database (32, 33), a searchable database synthesizing more than 800 treatment studies for youth with psychiatric disorders, was referenced to determine a CBT youth diagnostic and age profile. The database was searched for CBT trials to identify diagnoses and age ranges that met well-established criteria proposed by Chambless and Hollon (18).

#### Population-Based Data Source and Study Sample

Philadelphia County behavioral health Medicaid claims (*N* = 903,980) were used to identify a subset of consumers (*N* = 60,391)

who received outpatient behavioral health services during November 2015 through October 2016. This 1-year time period was chosen because of the shift from ICD-9 and DSM-IV-TR diagnoses to ICD-10 and DSM-5 diagnoses. De-identified claims included age at the first claim, sex, race, psychiatric diagnosis, and behavioral health service use. Behavioral health services were categorized based on level of care codes and only claims reflective of outpatient therapy services were retained (i.e., assessment and medication management codes were excluded). The final sample included the consumers with two or more outpatient claims aggregated by ICD-10 diagnosis. Consumers may have been counted more than once across but not within ICD-10 diagnoses. This allowed for more consumer coverage and the ability to account for multiple psychiatric diagnoses. The University of Pennsylvania and the City of Philadelphia Department of Public Health Institutional Review Boards determined that this study was exempt from review due to the masking of identifiable information.

The final study sample included 897,064 claims representing 53,475 unique consumers. There were 6,916 duplicate consumers removed from analyses due to multiple claims being submitted for the same consumer for more than one diagnosis. In instances of multiple claims, the first claim per consumer was retained. Consumers were 53.4% female (*n* = 34,507) and averaged 29.91 (SD = 17.99) years of age. Race included African-American (42.7%, *n* = 27,573), Hispanic (37.8%, *n* = 24,339), White (15.6%, *n* = 10,061), and Other (3.9%, *n* = 2,531).

#### Cost-Analysis Metric

The cost of therapist training, consultation, certification, and revenue loss were summed to calculate a total training and consultation cost for each EBT. This total training and consultation therapist cost was then divided by the number of consumers within Philadelphia County Medicaid claims who matched the

5http://www.div12.org/psychological-treatments/disorders/borderline-personality-disorder/dialectical-behavior-therapy-for-borderline-personality-disorder/.

EBT diagnostic and age profile. This formula resulted in an EBT training and consultation cost per potential consumer:

> TRAINING CONSULTATION CERTIFICATION REVENUE LOSS + + +

NUMBER OF PROSPECTIVE CONSUMERS SERVED.

#### RESULTS

#### Training and Consultation Requirements

Certifying bodies, training hours, consultation hours, training length, and specific criteria related to consultation are detailed in **Table 1**. Across EBTs, 2–5 days of in-person training were required for certification. TF-CBT and CPT both required online training in addition to the in-person training. Trainings were provided by certified trainers in each respective EBT, identified by the certifying body. Regarding consultation, DBT and PCIT required the most ongoing consultation (i.e., bimonthly contact for approximately a year), whereas CBT/CT required fewer hours (i.e., 1 year of clinical experience with 10 h of consultation). Live feedback in the form of tape review or telehealth observation was included in the consultation descriptions for PCIT and PE. Consultation hours typically spanned 6–12 months. MATCH and TF-CBT gave the option of meeting twice per month for 6 months or once per month for 12 months. PCIT and PE consultation were based on completion of two cases rather than a set time frame. CPT was similar in that it required 20 h of group or 12 h of individual consultation. Consultation was provided by a certified supervisor identified by the certifying body.

#### Training and Consultation Cost

Training, consultation, certification, and revenue loss costs were summed to form a total cost in **Table 2**. EBT are rank ordered by their total cost, with DBT being the most expensive to TF-CBT being the least expensive. Training costs ranged from \$585 for CPT to \$4,900 for PCIT per therapist. However, consultation costs are included in the PCIT training cost. In addition to PCIT, MATCH and TF-CBT included the cost of consultation into their training cost. Stand-alone consultation prices ranged from \$2,000 to \$12,500, with consultation costs as either a set rate (i.e., \$2,000 for CBT/CT consultation), per session rate (i.e., \$185 for PE), or an hourly rate (i.e., \$250 per hour for DBT, \$200 per hour for CPT).

#### Cost per Prospective Consumer

Prospective consumer costs were calculated by summing the total cost of training, consultation, certification, and revenue loss, and dividing that among the number of unique consumers fitting each EBT diagnostic and age profile. **Table 3** details the total cost, age range in years, diagnoses, number of unique consumers fitting the diagnostic and age profile, and a cost per consumer (total cost/consumers) and is ordered by the per prospective consumer cost (most to least expensive). Cognitive Behavioral Therapy/ Cognitive Therapy was the least expensive per consumer (\$0.18) and covered the most prospective consumers (*n* = 39,586). In contrast, DBT was the most expensive per consumer (\$238.07) and covered the fewest prospective consumers (*n* = 81).

#### DISCUSSION

The goal of this study was to develop a cost-analysis metric around the specific implementation strategy of EBT training and consultation while considering the population being served. This is particularly important given the financial pressures that large behavioral health services systems face to effectively implement EBT and manage tax-payer dollars and costs to the system, agencies, therapists, and consumers. Our study used seven common EBTs and compared training and consultation hours and prices and calculated per prospective consumer costs in a large behavioral health system. Training and consultation requirements and costs varied widely across EBT. Training and consultation costs ranged from \$600 to \$14,985 per therapist, and when considering certification fees and revenue loss from time spent in training rather than serving consumers, total costs ranged from \$2,231.32 to \$19,283.30. This represents a substantial investment to therapists, organizations, and systems. For some EBTs, consultation emerged as the most time-consuming and costly aspect, which is often emphasized as an important implementation strategy (2). Total cost did not correspond with the number of prospective consumers served by an EBT in our current behavioral health system sample. That is, the most expensive EBTs were not those that the most prospective consumers would benefit. This costanalysis metric utilizing prospective consumer behavioral health outpatient claims appears to be a useful tool for large system decision-making in choosing EBT.

The costliest EBT to train (i.e., DBT) covered the fewest consumers in the system, likely because few consumers had

*EBT, evidence-based treatment; DBT, Dialectical Behavior Therapy; PCIT, Parent–Child Interaction Therapy; PE, Prolonged Exposure; CBT/CT, Cognitive Behavioral Therapy/ Cognitive Therapy; CPT, Cognitive Processing Therapy; MATCH, Modular Approach to Therapy for Children with Anxiety, Depression, Trauma, or Conduct Problems; TF-CBT, Trauma Focused-Cognitive Behavior Therapy.*

Table 3 | Evidence-based treatment (EBT) cost per consumer.