**Texts in Quantitative Political Analysis** *Series Editor:* Justin Esarey

# Alessia Damonte Fedra Negri *Editors*

# Causality in Policy Studies A Pluralist Toolbox

# **Texts in Quantitative Political Analysis**

#### **Series Editor**

Justin Esarey, Dept of Politics, FM Kirby Hall 319 Wake Forest University Winston Salem, NC, USA

This series covers the novel application of quantitative and mathematical methods to substantive problems in political science as well as the further extension, development, and adaptation of these methods to make them more useful for applied political science researchers. Books in this series make original contributions to political methodology and substantive political science, while serving as educational resources for independent practitioners and analysts working in the feld.

This series flls the needs of faculty, students, and independent practitioners as they develop and apply new quantitative research techniques or teach them to others. Books in this series are designed to be practical and easy-to-follow. Ideally, an independent reader should be able to replicate the authors' analysis and follow any in-text examples without outside help. Some of the books will focus largely on instructing readers how to use software such as R or Stata. For textbooks, example data and (if appropriate) software code will be supplied by the authors for readers.

This series welcomes proposals for monographs, edited volumes, textbooks, and professional titles.

Alessia Damonte • Fedra Negri Editors

# Causality in Policy Studies

A Pluralist Toolbox

*Editors* Alessia Damonte Social and Political Sciences University of Milan MILANO, Milano, Italy

Fedra Negri University of Milan Milano, Italy University of Milan- Bicocca Milano, Italy

This book is an open access publication.

ISSN 2730-9614 ISSN 2730-9622 (electronic) Texts in Quantitative Political Analysis ISBN 978-3-031-12981-0 ISBN 978-3-031-12982-7 (eBook) https://doi.org/10.1007/978-3-031-12982-7

© The Editor(s) (if applicable) and The Author(s) 2023

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

## **Preface**

How can we think of causation in policy research? Almost any research tradition provides a different answer. For instance, emphasis can be placed either on the process leading to a policy outcome or on its underlying conditions. A process can be either observable or unobservable, and the underlying relevant conditions can be understood as single factors or complex confgurations. Either samples, populations, or single cases can be invoked as the proper empirical ground for grasping them. Evidence can be arranged to either claim relevance or irrelevance. These differences refect as many distinct assumptions about the shape of causation and build as many research strategies.

*Causality in Policy Studies* equips researchers to meet two related challenges in the feld. First, algorithms for data analysis embed selected assumptions about causation that often remain unspoken. Knowing these assumptions is crucial to understanding how algorithms can be appropriately employed and eventually combined to compensate for their blind spots and weaknesses. Second, policy research is carried out within various disciplines (such as political science, sociology, economics, management, and administration), each often married to particular traditions. The book addresses the technical drive of such differentiation. In doing so, it provides the opportunity for researchers of any stripe to familiarize themselves with the strategies on which other streams build their claims.

In short, the book shows how to learn from different causal techniques, apply them consciously, and possibly make them speak to each other to get a better sense of fndings. For this purpose, it structures the journey into causal knowledge in three stages. First, it introduces the foundational issues of causation (Chaps. 1 and 2). Then, it exposes the inner working of selected techniques for causal analysis (Chaps. 3, 4, 5, 6, 7, 8 and 9). Last, it considers some incompatibilities and complementarities among techniques to improve causal knowledge (Chaps. 10 and 11).

The red thread connecting all chapters is a reasonable realist stance. All share the tenets that causation is factual and entails generative and transfer processes unfolding at different levels of reality. Moreover, the chapters agree that causation can be known. Hypothetical statements about its manifestations, direction, and conditions can be given a testable shape. They also agree that causal statements should be

believed when logically and empirically compelling. The book's commitment to methodological pluralism follows from these tenets. The complexity of causal phenomena is such that no single technique can grasp its entirety. Still, each technique can illuminate particular facets in response to a precise research question. Indeed, asking whether a factor can yield one outcome differs from asking how it happens or under which conditions it obtains, and each response calls for adequate analytic tools. When pieced together, these responses can offer a better account of the phenomena of interest.

Methodological pluralism can deliver on the promise of better knowledge if the strengths and weaknesses of each technique are understood and tackled. To this end, each substantive chapter clarifes the research question a technique can answer, the research design and data treatment the technique requires for credible results, and the domain of validity of its fndings. Wherever possible, a replicable example illustrates the deployment of the analysis as the sequence of operations and actual decisions. Of course, this selection of techniques is far from exhaustive of the methodological variety of policy studies. Nevertheless, this suite provides sharp insight into the different strategies to establish the tenability of a causal statement. As such, it can offer guidance beyond the boundaries of this book.

The edited format of the book aims at providing highly usable and solid knowledge for policy assessment and evaluation to MA students, PhD students, scholars, and practitioners in policy-related felds. Thus, each chapter is authored by a recognized scholar from different backgrounds, generations, and perspectives. Such a diverse yet "close-knit" team is essential to the volume. A single author could hardly have covered such a range of techniques with comparable expertise.

Public policies are tools and governance systems to tackle collective problems. Good policies call for a generation of open-minded scholars and practitioners willing to understand and learn from research conducted in different felds and capable of handling the techniques in their toolbox consciously and carefully. We hope you will have a good time going through the chapters. Enjoy your journey!

MILANO, Milano, Italy Alessia Damonte

Fedra Negri

## **Acknowledgments**

Every book is a collective enterprise and some more than others. Our frst thanks go to the many students who have compelled this project and shaped it during our courses. They have been the real drivers of this effort, and we are more than grateful for how they kept our motivation high over the many months of writing and revising. Heartfelt thanks also go to Licia Papavero, Francesco Zucchini, and the board of the Ph.D. in Political Studies of the University of Milan. Their constant support to the Summer School in "Research Strategies in Policy Studies" (ReSPoS) has proven vital to the maturation of this project, which consolidates an experience dating back to 2013—now, a seemingly distant past. The School and the project, in turn, would not have been possible without the fnancial contributions of the Compagnia di San Paolo (Turin, Italy) through the Network for the Advancement of Social and Political Sciences (NASP) directed by Maurizio Ferrera. To them go our sincere gratitude. We also are greatly indebted to Springer's Senior Editor for Economics, Political Science, and Public Administration, Lorraine Klimowich, and the Editor of the 'Textbook on Political Analysis' series, Justin Esarey. Their precious suggestions and faultless encouragement have been fundamental to fnalizing a project intentionally positioned at the crossroad of many disciplines and research standards. Our further debt of gratitude is owed to Luigi Curini and the Standing Groups "MetRiSP—Research Methods for Political Science" and "Political Science & Public Policy" of the SISP—Italian Society of Political Science. We have treasured their feedbacks on earlier versions and their backing the main idea. Finally, we gratefully acknowledge the fnancial support of Protego—an advanced project funded by the European Research Council—Grant agreement n°694632.

University of Milan Alessia Damonte Milan, Italy Fedra Negri

## **Contents**

# **Chapter 1 Introduction: The Elephant of Causation and the Blind Sages**

**Alessia Damonte and Fedra Negri**

*It was six men of Indostan, To learning much inclined, Who went to see the Elephant (Though all of them were blind), That each by observation Might satisfy his mind. John G. Saxe (1816–1887).*

**Abstract** What does a policy outcome hinge on? The response is vital to policymaking and calls for the best of our knowledge from a variety of disciplines—from economics to sociology and from political science to public administration and management. The response entails a stance about causation, however, and almost every discipline has its own. Researchers are like the blind sages who had never come across the elephant of causation before and who develop their idea of the elephant by "touching" a different part of it. Which part of the elephant will you happen to touch? Will you be able to listen to and understand what the other sages will tell you?

#### **1.1 Policy Decisions and Causal Theories**

The common wisdom about public policy understands them as governments' decisions to tackle a collective problem. These decisions deploy rules, information, taxes, and expenditures to get "people to do things that they might not otherwise do" or "do things that they might not have done otherwise" (Schneider & Ingram, 1990: 513). By inducing a change in people's willingness and capacity to "do things," policy-makers expect the problem to disappear or, at least, take a more bearable shape.

A. Damonte (\*)

University of Milan, Milan, Italy e-mail<EMAIL_ADDRESS>
F. Negri University of Milan - Bicocca, Milan, Italy e-mail<EMAIL_ADDRESS>
Thus, the kernel of policy decisions is the causal theory that they encapsulate: frst, of the behavior at the root of the collective problem; second and relatedly, of the capacity that certain tools have to make such behavior change for the better. The theory connects outcomes to behavior and then identifes the "carrots, sticks, and sermons" (Vedung, 2010) best suited to put or keep such behavior on a desirable track. For example, in their fght against cancer, governments can address smoking as a proven causal factor and assume people smoke if they have the wrong information or are shortsighted about the consequences of their behavior—else, they would reasonably quit. Governments can fund education campaigns to convey the right information, require tobacco products to carry warning labels, or disallow tobacco advertising and sponsorship. Moreover, to compensate for people's shortsightedness, they can levy "sin taxes" upon tobacco products to make prices a better signal of the hidden costs of smoking or enforce smoke bans that protect non-smokers. Whether a government applies none, one, or a mix of these tools, in turn, depends on policy-makers; whether their decisions reach the addressees properly, instead, is an administrative and a governance matter (e.g., McConnell, 2010). Regardless of the point of attack, the issue of policy success and failure inevitably appeals to causal theories on endowments, concerns, constraints, and incentives accounting for behavior (e.g., Ostrom, 2005).

Policy studies offer exemplary illustrations of the twofold stake of causal theories. First, these theories allow us to make sense of the world. Our bewilderment at some diversity in performance dissolves when we are offered satisfying accounts of relevant behaviors. Second, these theories have straightforward practical implications for individual and collective strategies. If we know which factors compel an event and suppress it, we can change the event's odds by controlling these factors. Then, the driving question remains: how can we get to know these factors well enough to build decisions on them?

#### **1.2 The Elephant of Causation**

Across the philosophy of science and social sciences, the responses to this question invite analogies with the blind sages in Saxe's poem (1872), who "prate about an Elephant that / Not one of them has seen."1 Indeed, actual causation is the complex local production of an outcome and it is hard to identify before it unfolds. The usable knowledge of a causal process pinpoints the key factors of its unfolding that allow us to see it coming in the next instance and, eventually, change its odds (e.g., Craver and Kaplan, 2020). Such knowledge requires criteria to identify the key

1The poem tells the story of a group of blind sages who have never come across an elephant before and who learn what the elephant is like by touching it. Each blind sage feels a different part of the elephant's body, but only one part. They then describe the elephant based on their limited experience and "Though each was partly in the right, /And all were in the wrong!" (Saxe, 1872).

causal factors beyond the single case and credibly so. Historically, guidelines for identifying the key causal factors developed along two lines.

#### *1.2.1 Elephants by the Principle*

The most enduring guideline for determining the key causal factors before a process unfolds has come from the Aristotelian philosophy of science. There, causation was tracked back to four kinds of principles, known as "material," "formal," "effcient," and "fnal." The frst two principles capture the structural features of a causal process, namely, its constituent elements and the shape of their arrangement. The latter two refer to agency and locate the key factors in outer stimuli or the drive from inner purposes (e.g., Moravcsik, 1974). The original "doctrine" maintained that adequate responses to any why-question appealed to all the four principles together.

Indeed, convincing accounts still locate actual causation in the interplay of structure and agency, as infuential mechanistic perspectives make clear (e.g., Little, 2011; Craver, 2006). More often, current research streams specialize in single principles. For example, the causal role of "material" ascriptive features is a driving concern of gender and minority studies. The generative power of formal arrangements is the core tenet of, for instance, game theories. Studies on expected utility, values, habits, and emotions take heed of the fnal goals and motivations, providing fundamental assumptions for neo-institutionalist and behavioral approaches of various stripes. Effcient factors are any stimulus, intervention, or treatment that can elicit a response; thus, they are central to theories of policy instruments, regimes, or political communication, among many others.

With some exceptions (e.g., Bache et al., 2012; Kurki, 2006), current theories seldom claim an explicit legacy with the original canon. The doctrine has fallen into disrepute as improperly scientifc, because it invoked a metaphysical reason to justify the causal standing of its four principles. The tenet that individuals with similar features, in a similar situation, with similar motivations, under equivalent stimuli did and will behave in similar ways was justifed by the belief that all embodied the same metaphysical essence. As Aristotle argued in a seminal fragment, planets do not twinkle because planets are near things, and not twinkling was intrinsic to near things. Thus, the next planet will not twinkle, too, in force of its "near-thingness."

This line of reasoning easily lends itself to circular arguments that restate general assumptions instead of probing them. As late as 1673, Molière still had reasons to satirize it. In his comedy *The Hypochondriac*, a "docto doctore" explains in dog Latin that opium makes people sleepy because it embodies a "dormitive virtue." However, the ultimate criticism came from the British Empiricists, who saw in the appeal to essences a mode for preserving beliefs against evidence and a fundamental obstacle to progress and learning.

#### *1.2.2 Elephants by the Rules*

The rejection of metaphysical warrants has called for a different ground for causal inference. Whether a reliable connection exists between being a near thing and not twinkling across cases, so the argument goes, it can only be decided empirically.

Yet, causal evidence does not come to us with labels and numbers attached. Assumptions are still needed about the empirical traces that distinguish between relevant and irrelevant causal factors. In Hume's much-quoted words, causally relevant is:

an object followed by another and where all the objects, similar to the frst, are followed by objects similar to the second. Or, in other words, where, if the frst object had not been, the second never had existed. (Hume, 1748, Section VII, Part II, §60).

In short, a factor is relevant to an outcome in the single case under two warrants: the association of the two conforms to a *regular* pattern, and it supports *counterfactual* reasoning.

#### **<IP_ADDRESS> Regularity**

The regularity warrant—"where all the objects, similar to the frst, are followed by objects similar to the second"—renders the empirical footprint of Aristotelian essences without assuming them and builds on the repeated observation of similar occurrences.

All objects sharing the same feature are similar and constitute a distinct class. Regularity, then, is established between objects in different classes—for instance, in the class of "swan" and in the class of "white." It requires that any observation of the frst class entails one in the second. When the regularity holds, causal knowledge can be circulated through handy formulae such as "if a swan, then white."

To apply to the next instance, these formulae have to prove faultless, which is hardly the case: classes and gauges are human constructs and can prove too strict or liberal to capture actual causation in the next instance. Hence, regularity holds provisionally only until we meet the black swan that forces a revision of the scope of our regularity tenets.

Regularity may also seem perfect just because we measured two consequences of the same process. These relationships are useful for prediction; however, they do not qualify as causal as they do not grant control over the events' odds as desired in public policy. Indeed, a barometric reading can be relied upon to prepare for extreme weather conditions but does not license the belief that the coming storm can be tamed by forcing the barometer's pointer. Thus, regularity can be a necessary trait of usable knowledge but insuffcient to declare the causal standing of a relationship.

#### **<IP_ADDRESS> Counterfactual**

The counterfactual—"where, if the frst object had not been, the second never had existed"*—*enters the picture as the additional warrant to establish causal relevance and ideally applies to the factor in the single case independent of regularity. The warrant borrows from the classical rules of argumentation and the indirect proofs in geometric demonstrations; however, it displays an empirical edge. Counterfactuals link causal relevance to evidence that we could compel a change in the second object by manipulating the frst.

From the Humean defnition, manipulation is usually understood as suppression; more generally, it means switching the observed state of a feature into its opposite. Thus, counterfactual reasoning requires, frst, that we imagine the frst object with the switched feature and, then, that we can only draw impossible or contradictory conclusions from it (e.g., Levi, 2007). An exemplary illustration comes directly from Hume. Despite his deep skepticism toward the human mind's ability to fully understand causation, he conceded that our intuitions must be somehow right. To justify his claim, he reasoned that had our mind always got causation wrong (switching the feature), then humankind would have long gone extinct (drawing a conclusion), which contrasts with us thriving as a species (showing the conclusion absurd). Such counterfactual criterion improves on the regularity test, as regular non-causal features fail it: as a broken barometer cannot stop a storm, it cannot be recognized as having any causal standing.

However, counterfactuals have their limits, too. First, they cannot be established unless all the plausible alternative causes of the same outcome are ruled out. Hume's argument does not exclude that humankind's evolutionary success instead depends on, for instance, sheer luck—and the unaccounted alternative undermines the cogency of its conclusion. The second and related issue is serious to the point of earning the title of "fundamental problem of causal inference" in some quarters (e.g., Holland, 1988). Unless we cast the same causal process in the same unit with and without the feature of interest, we cannot establish whether switching the feature can change the outcome.

#### **1.3 The Blind Sages' Portrayals as the Book's Blueprint**

The criteria to establish causation by regularity and counterfactual evidence seem as straightforward as impossible to meet. Nevertheless, techniques have been developed as strategies to circumvent the Humean paradoxes and provide empirical warrants to the claim of causal relevance. As Little shows in Chap. 2, technical specialization has undermined the dialogue among techniques and their fndings. The appeal to regularity, counterfactual, or mechanistic principles has turned into as many ultimate understandings of causation: "laws" and counterfactuals offered a rival ground for experimental practices; mechanisms took distances from both and licensed causal analysis in actual cases only, under consideration that any conclusion about aggregates necessarily entails an unfaithful reduction—in the end, all models are wrong.

However, the possibility of integration remains when techniques commit to three considerations and are consistent with a reasonable scientifc realism. First, causation is real, but our best knowledge of it remains a useful approximation. Second, regularity and counterfactuals are epistemic criteria to establish whether portrayals qualify as valid causal accounts; mechanisms are ontological assumptions about single actual elephants instead. Third, the difference between mechanistic description, models, and laws is not of kind but degree: when they address a common slice of the world, they provide a map of it with different details, abstraction, and scope. Under these commitments, techniques can be understood as devices to respond to special questions about the elephant.

#### *1.3.1 Can this Single Factor Make Any Difference?*

The family of experimental and quasi-experimental techniques offers the most renowned, successful, and contentious example at once due to the diffusion of randomized controlled trials as the "gold standard" of scientifc knowledge production (e.g., Kabeer, 2020; Deaton & Cartwright, 2018; Dawid, 2000). This family shares the consideration that although we cannot observe a counterfactual directly, we can construe credible "twin worlds" and "treat" one so that the feature of interest provides the only difference to which the difference in responses can be ascribed.

As Battistin and Bertoni show in Chap. 3, this strategy keeps the role of causal assumptions to the minimum required by a stimulus-response model: the treatment is a supposedly effcient cause and connected to performance by a function of a specifc shape—often, linear—without further details. Unsurprisingly, these techniques are a cornerstone of usable public policy knowledge: they can establish the capacity of a change in taxation, expenditure, information, and regulation to elicit some effect of interest, apparently without the need for further knowledge.

The credibility of this strategy's conclusions, however, rests heavily on the research design: fndings are sound if the twin worlds are construed as statistically identical and independent aggregates, the treatment is forced evenly onto all the units of one world only, and the difference in responses is not affected by the treating procedure or unrelated endogenous dynamics. The threats arise as the statistical aggregates with identical parameters can hide a remarkable inner heterogeneity that may bias both groups' responses in unknown directions. As elaborated by Negri in Chap. 4 and Ornstein in Chap. 5, within the family, this heterogeneity is addressed as the result of selection biases that can be reduced by accounting for observed imbalances and crafting "populations of twins." The solution, however, leaves the issue open of the bending effects from unobservable factors.

The (quasi-)experimental family, in short, can provide reliable measures of the net effect of a treatment, but necessarily at the cost of disregarding the reasons for the diversity in the responses of the treated.

#### *1.3.2 Through Which Structures?*

The diversity in responses is instead the driving concern of the second group of techniques. They address it by fipping the experimental balance of model and design and committing themselves to additional assumptions. They conceive of the generative process as patterns of dependence and assign causal relevance to the bundle of factors that ft them.

The reliance on models sidelines the issue of unit selection as, ideally, any unit carries usable information about the tenability of the causal structure of interest. The structure, moreover, provides the fxed points that still make counterfactuals observable. However, models require criteria to select meaningful variables, and structural assumptions provide partial guidance to it. The main decisions can only be made in light of substantive theories about the generation of the outcome hence, of some previous local knowledge. Within this framework, each technique relies on different languages and pursues different goals.

Path analysis develops within a Bayesian mindset and understands causation as ordered dependencies ftting a few known shapes: chains, colliders, and forks. As Röth clarifes in Chap. 6, these shapes explain because they elaborate on the connection between an alleged causal condition and the dependent by displaying the intermediate causal link, the common factor, or the equivalent alternative factors that support the hypothesis about the unfolding of the causal process before the outcome. The technique supports a neater identifcation of the mechanism linking a factor of interest and its outcome, affords counterfactual analysis, and provides specifc suggestions about the "scope conditions" ensuring the mechanisms. Röth contends that these features qualify path analysis as the natural companion of experimental studies for its capacity to establish the contextual requirements that enhance and refne the validity of their fndings.

Qualitative comparative analysis (QCA) instead builds on sets and Boolean algebra and understands causal structures as teams of individually necessary and jointly suffcient factors to an outcome. In Chap. 7, Damonte makes three points about the explanatory import of the technique. First, its assumptions about the shape of causation support complex causal theories about the interactions of triggering, enabling, or shielding conditions of some underlying causal process. Second, its parameters of ft allow diagnosing the underspecifcation of the theory to the cases at hand, while the algorithm provides a pruning counterfactual device that takes care of its overspecifcation. Last, sets remap qualities onto quantities, which warrant meaningful and sound solutions. Thus, QCA can formalize and test theories about the teams of conditions beneath policy success and failure across given cases beyond special processes. As such, the technique especially suits the purpose of systematic *ex-post* evaluation of policy designs.

#### *1.3.3 Through Which Process?*

The knowledge of the dynamics of a causal situation is the missing piece of knowledge and the core concern of two further strategies, aiming to open up the black box of causation. Both share the direct interest in the actors and their interplay as the ultimate ground of causation, although their point of attack within the causal stream of actions is different.

Bayesian process tracing addresses causation within its local context. In Chap. 8, Bennett shows how analysts can rely on this technique to make causal sense of the chain of events to policy success or failure retrospectively. The strategy understands hypotheses as plausible Bayesian beliefs that we can entertain about the causal process and that evidence can confrm or disconfrm. The weight of evidence rests on the assumption that each hypothesis corresponds to a specifc sequence of actions and events that leave empirical traces. When the connection between a piece of evidence and a hypothesis is unique, certain, or both, the actual retrieval of certain traces in a case contributes to ranking hypotheses by their relative likelihood and eventually licenses the ascription of the case to the hypothesis with the best standing.

Last but not the least, agent-based models make it possible to test hypotheses about causal processes as emergent phenomena in silico. As Squazzoni and Bianchi illustrate in Chap. 9, the technique relies on simulation to verify whether a certain alignment of assumptions about actors and their constraints, when translated into conditional rules of individual behavior and recursively played, returns performance values close to the empirical responses of actual systems. The strategy requires regularity and counterfactual assumptions about the options available to each agent, rendered as alternative states, and about the consequence of choosing a state conditional on the states of the relevant neighbors. These models shed light on the tenability of different understandings of the mechanism that alternative policy constraints or endowments activate in the feld.

#### *1.3.4 Considerations and Extensions*

The order of the chapters, as Beach and Siewert reason in their Chap. 10, chimes with the common prescription in mixed method research that a better causal knowledge follows from a succession of techniques zooming into individual cases, where causation unfolds as actual processes and explanations can fnd their ultimate validation. However, they consider the downward path of mixed methods lays knowledge open to heterogeneity threats. The actual heterogeneity is always equal to the number of instances under analysis; cross-case knowledge, however, requires that we dismiss some heterogeneity as irrelevant to afford comparisons and causal inferences. The move to local contexts implies a twofold shift—from a low to a high number of factors in the analysis and from coarse types to fne-grained tokens of evidence—that seldom support cross-case fndings. Hence, they contend that a more fruitful and conventional strategy follows the upward path from local processes over structures to the causal capacity of single triggers. This path allows more conscious decisions about heterogeneity that can improve models and gauges.

In Chap. 11, Damonte and Negri conclude the journey. The chapter recognizes the fragmented image of causation that the previous contributions convey and asks whether such fragmentation is an undesirable state of affairs, as claimed by a longhonored narrative from the history of science, or an eventually valuable situation, as argued in the pluralist quarters of the philosophy of science. The point of contention concerns the inability to yield dovetailing knowledge that would affect strategies built on alternative tenets. The chapter revises these tenets and contends that, whereas ontology offers complementary angles of attack to the causal elephant and epistemology licenses interpretations that can estrange research communities from one another, methodological reasoning about models and designs reconciles the analyses when it emphasizes that causation corresponds to a few recognized shapes. These shapes, the chapter concludes, offer a rough yet common map of the elephant that strategies of any stripe can detail and enrich while pursuing their special research interests—thus contributing to better policy knowledge.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 2 Causation in the Social Realm**

**Daniel Little**

**Abstract** Explanation is at the center of scientifc research, and explanation almost always involves the discovery of causal relations among factors, conditions, or events. This is true in the social sciences no less than in the natural sciences. But social causes look quite a bit different from causes of natural phenomena. They result from the choices and actions of numerous individuals rather than fxed natural laws, and the causal pathways that link antecedents to consequents are less exact than those linking gas leaks to explosions. It is, therefore, a crucial challenge for the philosophy of social science to give a compelling account of causal reasoning about social phenomena that does justice to the research problems faced by social scientists.

#### **Learning Objectives**

By studying this chapter, you will:

#### **2.1 Why Discuss the Ontology of Causation?**

Ontology precedes methodology. We cannot design good methodologies for scientifc research without having reasonably well-developed ideas about the nature of the phenomena that we intend to investigate (Little, 2020). This point is especially important in approaching the idea of social causation. Only when we have a reasonably clear understanding of the logic and implications of the scientifc idea of

D. Little (\*)

University of Michigan-Dearborn, Dearborn, MI, USA e-mail<EMAIL_ADDRESS>
© The Author(s) 2023

A. Damonte, F. Negri (eds.), *Causality in Policy Studies*, Texts in Quantitative Political Analysis, https://doi.org/10.1007/978-3-031-12982-7\_2

causality can we design appropriate methods of inquiry for searching out causal relations. And only then can we give a philosophically adequate justifcation of existing methods—that is, an account of how the research method in question corresponds to a sophisticated understanding of the nature of the social world.

Here I will work within the framework of an "actor-centered" view of social ontology (Little, 2006, 2014, 2016). On this view, the social realm is constituted by individual actors who themselves have been cultivated and developed within ongoing social relations and who conduct their lives and actions according to their understandings and purposes. Social structures, social institutions, organizations, normative systems, cultures, and technical practices all derive their characteristics and causal powers from the socially constituted and situated individuals who make them up (Little, 2006).

This fact about social entities and processes suggests a high degree of contingency in the social world. Unlike chemistry, the social world is not a system of lawgoverned processes; it is instead a mix of different sorts of institutions, forms of human behavior, natural and environmental constraints, and contingent events. The entities that make up the social world at a given time and place have no essential ontological stability; they do not fall into "natural kinds"; and there is no reason to expect deep similarity across a number of ostensibly similar institutions—states, for example, or labor unions. The "things" that we fnd in the social world are heterogeneous and contingent. And the metaphysics associated with classical thinking about the natural world—laws of nature; common, unchanging structures; and fully predictable processes of change—do not provide appropriate building blocks for our understandings and expectations of the social world nor do they suggest the right kinds of social science theories and constructs.

Instead of naturalism, this actor-centered approach to social ontology leads to an approach to social science theorizing that emphasizes agency, contingency, and plasticity in the makeup of social facts. It recognizes that there is a degree of pattern in social life, but emphasizes that these patterns fall far short of the regularities associated with laws of nature. It emphasizes contingency of social processes and outcomes. It insists upon the importance and legitimacy of eclectic use of multiple social theories: social processes and entities are heterogeneous, and therefore, it is appropriate to appeal to different types of social theories as we explain various parts of the social world. It emphasizes the importance of path dependence in social outcomes.

#### **Box 2.1 Defnitions**

**Agency**: The fact that social change and causation derives from the purposive actions of individual social actors.

**Contingency**: Social outcomes depend upon conjunctions of occurrences that need not have taken place, so the outcome itself need not have taken place. Closely related to "path dependency."

#### **Box 2.1** (continued)

**Path dependency**: The feature of social processes according to which minor and underdetermined events in an early stage of a process make later changes more probable. For example, the QWERTY arrangement of the typewriter keyboard was selected in order to prevent typists from jamming the mechanism by typing too rapidly. Fifty years later, after widespread adoption, it proved impossible to adopt a more effcient arrangement of the keys to permit more rapid typing.

**Plasticity**: A feature of an entity or group of entities according to which the properties of the entity can change over time. Biological species demonstrate plasticity through evolution, and social entities demonstrate plasticity through the piecemeal changes introduced into them by a variety of actors and participants.

How does this ontological perspective ft with current work in policy studies? There are several current felds of social research that illustrate this approach particularly well. One is the feld of the "new institutionalism." Researchers in this tradition examine the specifc rules and incentives that constitute a given institutional setting. They examine the patterns of behavior that these rules and incentives give rise to in the participants in the institution, and they consider as well the opportunities and incentives that exist for various powerful actors to either maintain the existing institutional arrangements or modify them. Kathleen Thelen's (2004) study of different institutions of skill formation in Germany, Great Britain, the United States, and Japan is a case in point. This approach postulates the causal reality of institutions and the specifc ensembles of rules, incentives, and practices that make them up; it emphasizes that differences across institutions lead to substantial differences in behavior; and it provides a basis for explanations of various social outcomes. The rules of liability governing the predations of cattle in East Africa or Shasta County, California, create very different patterns of behavior in cattle owners and other landowners in the various settings (Ellickson, 1991). It is characteristic of the new institutionalism that researchers in this tradition generally avoid reifying large social institutions and look instead at the more proximate and variable sets of rules, incentives, and practices within which people live and act.

#### **2.2 Scientifc Realism About the Social World and Social Causation**

We are best prepared for the task of discovering causal relationships in the social world when we adopt a realist approach to the social world and to social causation. We provide an explanation of an event or pattern when we succeed in identifying the real causal conditions and events that brought it about. The central tenet of causal realism is a thesis about causal mechanisms and causal powers. Causal realism holds that we can only assert that there is a causal relationship between X and Y if we can offer a credible hypothesis of the sort of underlying mechanism that connects X to the occurrence of Y. The sociologist Mats Ekström puts the view this way: "the essence of causal analysis is … the elucidation of the processes that generate the objects, events, and actions we seek to explain" (Ekström, 1992: 115). Authors who have urged the centrality of causal mechanisms for explanatory purposes include Roy Bhaskar (1975), Nancy Cartwright (1989), Jon Elster (1989), Rom Harré and Madden (1975), Wesley Salmon (1984), and Peter Hedström (2005).

Scientifc realism about social causes comes down to several simple ideas.

First, there is such a thing as social causation. Causal realism is a defensible position when it comes to the social world: there are real causal relations among social factors (structures, institutions, groups, norms, and salient social characteristics like race or gender). We can give a rigorous interpretation to claims like "racial discrimination causes health disparities in the United States" or "rail networks cause changes in patterns of habitation."

Second, causal relations among factors or events depend on the existence of real social-causal mechanisms linking cause to effect. Discovery of correlations among factors does not constitute the whole meaning of a causal statement. Rather, it is necessary to have a hypothesis about the mechanisms and processes that give rise to the correlation. Hypotheses about the causal mechanisms that exist among factors of interest permit the researcher to exclude spurious correlation (cases where variations in both factors are the result of some third factor) and to establish the direction of causal infuence (cases where it is unclear whether the correlation between A and B results from A causing B or B causing A). So mechanisms are more fundamental than regularities.

Third, the discovery of social mechanisms in policy studies often requires the formulation of mid-level theories and models of these mechanisms and processes for example, the theory of free-riders. For example, an urban policy researcher may observe that racially mixed high-poverty neighborhoods have higher levels of racial health disparities than racially mixed low-poverty neighborhoods. This is an observation of correlation. Researchers like Robert Sampson (2010) would like to know how "neighborhood effects" work in transmitting racial health disparities. What are the mechanisms by which a neighborhood infuences the health status of an individual household? In order to attempt to answer this question, Sampson turns to mid-level hypotheses in urban sociology that contribute to a theory of the mechanisms involved in this apparent causal relationship. By mid-level theory, I mean essentially the same thing that Robert Merton (1963) conveyed when he introduced the term: an account of the real social processes that take place above the level of isolated individual action but below the level of full theories of whole social systems. Marx's theory of capitalism illustrates the latter; Jevons's theory of the individual consumer as a utility maximizer illustrates the former. Coase's theory of transaction costs (Coase, 1988) is a good example of a mid-level theory: general enough to apply across a wide range of institutional settings, but modest enough in its claim of comprehensiveness to admit of careful empirical investigation. Signifcantly, the theory of transaction costs has spawned major new developments in the new institutionalism in sociology (Brinton & Nee, 1998).

And fnally, it is important to recognize and welcome the variety of forms of social scientifc reasoning that can be utilized to discover and validate the existence of causal relations in the social world. Properly understood, there is no contradiction between the effort to use quantitative tools to chart the empirical outlines of a complex social reality, and the use of theory, comparison, case studies, process tracing, and other research approaches aimed at uncovering the salient social mechanisms that hold this empirical reality together.

#### *2.2.1 Critical Realism*

Critical realism is a specifc tradition within the late-twentieth-century analytic philosophy that derives from the work of Rom Harré and Roy Bhaskar (Harré & Madden, 1975; Bhaskar, 1975; Archer et al., 2016). In brief, the view holds that the ontological stance of realism is required for a coherent conception of scientifc knowledge itself. Unqualifed skepticism about "unobservable entities" makes scientifc research and experimentation philosophically incoherent. We are forced to take the view that the entities postulated by our best theories of the world are "real"—whether electrons, viruses, or social structures. For Bhaskar, this ontological premise has much the status of Kant's transcendental arguments for causation and space and time: we cannot make sense of experience without postulating causation and locations in space and time (Bhaskar, 1975).

Concretely in the social sciences, this is taken to mean that we can be confdent in asserting that social entities exist if these concepts play genuine roles in welldeveloped and empirically supported theories of the social world: for example, organizations, markets, institutions, social classes, normative systems, rules, ideologies, and social networks. Further, we can be confdent in attributing causal powers and effects to the various social entities that we have identifed—always to be supported by empirical evidence of various kinds.

#### **2.3 What Is Causation?**

Let us turn now to a more specifc analysis of causation. What do we mean by a cause of something? Generally speaking, a cause is a circumstance that serves to bring about (or renders more probable) its effect, in a given environment of background conditions. Causes *produce* their effects (in appropriate background conditions). A current fruitful approach is to understand causal linkages in terms of the specifc *causal mechanisms* that link cause to effect.

We can provide a preliminary defnition of causation along these lines:

That is, A is necessary and suffcient in conditions Ci for the production of B. This defnition can be understood in either a deterministic version or a probabilistic version. The deterministic version asserts that A in the presence of Ci always brings about B; the probabilistic version asserts that the occurrence of A in the presence of Ci increases the likelihood of the occurrence of B.

There is a fundamental choice to be made when we consider the topic of causation. Are causes real, or are causal statements just summaries of experimental and observational results and the statistical fndings that can be generated using these sets of data? The frst approach is the position described above as causal realism, while the second can be called causal instrumentalism. If we choose causal realism, we are endorsing the idea that there is such a thing as a *real* causal linkage between A and B; that A has the power to produce B; and that there is such a thing as causal necessity. If we choose causal instrumentalism, we are agnostic about the underlying realities of the situation, and we restrict our claims to observable patterns and regularities. The philosopher David Hume (2007) endorsed the second view; whereas many philosophers of science since the 1970s have endorsed the former view.

Most of the contributors to the current volume engage with the premises of causal realism. They believe that social causation is real; there are real social relations among social factors (structures, institutions, groups, norms, and salient social characteristics like race or gender), and there are real underlying causal mechanisms and powers that constitute those causal relations. According to scientifc realists, a key task of science is to discover the causal mechanisms and powers that underlie the observable phenomena that we study.

Causal realists acknowledge a key intellectual obligation that goes along with postulating real social mechanisms: to provide an account of the ontological *substrate* within which these mechanisms operate**.** In the social realm, the substrate is the system of social actors whose mental frameworks, actions, and relationships constitute the social world. This is what is meant by an "actor-centered" ontology of the social world. On this view, every social mechanism derives from facts about individual actors, the institutional context, the features of the social construction and development of individuals, and the factors governing purposive agency in specifc sorts of settings. Different research programs in the social sciences target different aspects of this nexus.

This view of the underlying reality of social causation justifes a conception of causal necessity in the social realm. Do causes make their effects "necessary" in any useful sense? This is the claim that Hume rejected—the notion that there is any "necessary" connection between cause and effect. By contrast, the notion of *natural necessity* is sometimes invoked to capture this idea:

#### 2 Causation in the Social Realm

• A causes B: given the natural properties of A and given the laws of nature and given the antecedent conditions, B necessarily occurs.

This can be paraphrased as follows:

• Given A, B occurs as a result of natural necessity.

So the sense of necessity of the occurrence of the effect in this case is this: given A and given the natural properties and powers of the entities involved, B had to occur. Or in terms of possible worlds and counterfactuals (Lewis, 1973), we can say:

• In any possible world in which the laws of nature obtain, when A occurs, B invariably occurs as well.

Applied to social causation within the context of an ontology of actor-centered social facts, here is what causal necessity looks like:

• Given the beliefs, intentions, values, and goals of various participants and given the constraints, opportunities, and incentives created by the social context, whenever A occurs, the outcome B necessarily occurs [fnancial crisis, ethnic violence, rapid spread of infectious disease …].

This conception aligns with Wesley Salmon's idea of the "causal structure of the world," applied to the social world (1984). And this in turn indicates why causal mechanisms are such an important contribution to the analysis of causation. A causal mechanism is a constituent of this "stream of events" leading from A to B.

Probabilistic causal relations involve replacing exceptionless connections among events with probabilistic connections among events. A has a probabilistic causal relationship to B just in case the occurrence of A increases (or decreases) the likelihood of the occurrence of B. This is the substance of Wesley Salmon's (1984) criterion of causal relevance. Here is Salmon's idea of causal relevance:

• A is causally relevant to B *if and only if* the conditional probability of B given A is different from the absolute probability of B (Salmon, 1984, adapted notation).

 For example, smoking is causally relevant to the occurrence of lung cancer [working through physiological mechanisms X, Y, Z]. And cell physiologists are expected to provide the mechanisms that connect exposure to tobacco smoke to increased risk of malignant cell reproduction.

It is important to emphasize that we can be causal realists about probabilistic causes just as we can about deterministic causes. A causal power or capacity is expressed as a tendency to produce an outcome; but this tendency generally requires facilitating conditions in order to be operative. The causal power is appropriately regarded as being real, whether or not it is ever stimulated by appropriate events and circumstances. A given cube of sugar is soluble, whether or not it is ever immersed in water at room temperature.

These defnitions have logical implications that suggest different avenues of research and inquiry in the social sciences. First, both the deterministic and the probabilistic versions imply the truth of a *counterfactual* statement: If A had not occurred in these circumstances, B would not have occurred. (Or if A had not occurred in these circumstances, the probability of B would not have increased.) The counterfactual associated with a causal assertion suggests an experimental approach to causal inquiry. We can arrange a set of circumstances involving Ci and remove the occurrence of A and then observe whether B occurs (or observe the conditional probability of the occurrence of B).

Another important implication of a causal assertion is the idea of a set of necessary and suffcient conditions for the occurrence of E, the circumstance of explanatory interest. With deterministic causation, the assertion of a causal relationship between A and B implies that A is suffcient for the occurrence of B (in the presence of Ci) and often the assertion implies that A is a necessary condition as well. (If A had not occurred, then B would not have occurred.) On these assumptions, a valid research strategy involves identifying an appropriate set of cases in which A, Ci, and B occur, and then observe whether the appropriate covariances occur or not. J. L. Mackie (1974) provided a more detailed analysis of the logic of necessary and suffcient conditions in complex conjunctural causation with his concept of an INUS condition: "*insuffcient* but *non-redundant* part of an *unnecessary* but *suffcient* condition" (62). Signifcantly, Mackie's formulation provides a basis for a Boolean approach to discovering causal relations among multiple factors.

These defnitions and logical implications give scope to a number of different strategies for investigating causal relationships among various conditions. For probabilistic causal relationships, we can evaluate various sets of conditional probabilities corresponding to the presence or absence of conditions of interest. For deterministic causal relationships, we can exploit the features of necessary and suffcient conditions by designing a "truth table" or Boolean test of the co-occurrence of various conditions (Ragin, 1987). This is the logic of Mill's methods of similarity and difference (Mill, 1988; Little, 1995). For both deterministic and probabilistic causal relationships, we can attempt to discover and trace the workings of the causal mechanisms that link the occurrence of A to the occurrence of B.

#### *2.3.1 Causal Mechanisms*

As noted above, the central tenet of causal realism is a thesis about the real existence of causal mechanisms and causal powers. The fundamental causal concept is that of a mechanism through which A brings about or produces B (Little 2011). According to this approach, we can only assert that there is a causal relationship between A and B if we can offer a credible hypothesis of the sort of underlying mechanism that connects A to the occurrence of B. This is central to our understanding of causation from single-case studies to large statistical studies suggesting causal relationships between two or more variables. Peter Hedström and other exponents of analytical sociology are recent voices for this approach for the social sciences (Hedström, 2005; Hedström & Ylikoski, 2010). An important paper by Machamer et al. (2000) sets the terms of current technical discussions of causal mechanisms, and James Mahoney (2001) surveyed the various theories of causal mechanisms and called for a greater specifcity.

What is a causal mechanism? Consider this formulation: a causal mechanism is a sequence of events, conditions, and processes leading from the explanans to the explanandum (Little, 1991: 15, 2016: 190–192). A causal relation exists between A and B if and only if there is a set of causal mechanisms that lead from A to B.

The theory has received substantial development in the biological sciences. Glennan et al. (2021) put the mechanisms theory in the form of six brief theses:

This defnition is developed for explanations in biology, but it works well with typical examples of social mechanisms.

The idea that there are real mechanisms embodied in a given domain of phenomena provides a way of presenting causal relations that serves as a powerful alternative to the pure regularity view associated with Hume and purely quantitative approaches to causation. Signifcantly, this is the thrust of Judea Pearl's development of structural equation modeling (discussed below): in order to get a basis for causal inference out of a statistical analysis of a large dataset, it is necessary to provide a theory of the causal mechanisms and relations that are at work in this domain (Pearl, 2021).

Mechanisms bring about specifc effects. For example, "over-grazing of the commons" is a mechanism of resource depletion. Whenever the conditions of the mechanism are satisfed, the result ensues. Moreover, we can reconstruct why this would be true for purposive actors in the presence of a public good (Hardin, 1968). Or consider another example from the social sciences: "the mechanism of stereotype threat causes poor performance on standardized tests by specifc groups" (Steele, 2011). This mechanism is a hypothesized process within the cognitive–emotional system of the subjects of the test, leading from exposure to the stereotype threat through a specifed cognitive–emotional mechanism to impaired performance on the test. So we can properly understand a claim for social causation along these lines: "C causes E" rests upon the hypothesis that "there is a set of causal mechanisms that convey circumstances including C to circumstances including E." In the social realm, we can be more specifc. "C causes E" implies the belief that "there is a set of opportunities, incentives, rules, and norms in virtue of which actors in the presence of C bring about E through their actions."

Are there any social mechanisms? There are many examples from every area of social research. For example: "Collective action problems often cause strikes to fail." "Increasing demand for a good causes prices to rise for the good in a competitive market." "Transportation systems cause shifts of social activity and habitation." "Recognition of mutual interdependence leads to medium-term social cooperation in rural settings." In each case, we have a causal claim that depends on a hypothesis about an underlying behavioral, cognitive, or institutional mechanism producing a pattern of collective behavior.

The discovery of social mechanisms often requires the formulation of mid-level theories and models of these mechanisms and processes—for example, the theory of free-riders or the theory of grievance escalation in contentious politics. Mid-level theories in the social sciences can be viewed as discrete components of a toolbox for explanation. Discoveries about specifc features of the workings of institutions, individual-collective paradoxes, failures of individual rationality like those studied in behavioral economics—all of these mid-level theories of social mechanisms can be incorporated into an account of the workings of specifc social ensembles. The response of a university to a sudden global pandemic may be seen as an aggregation of a handful of well-known institutional dysfunctions, behavioral patterns, and cognitive shortcomings on the part of the various actors.

Aage Sørensen summarizes a causal realist position for the social and policy sciences in these terms: "Sociological ideas are best reintroduced into quantitative sociological research by focusing on specifying the mechanisms by which change is brought about in social processes" (Sørensen, 1998: 264). Sørensen argues that social explanation requires better integration of theory and evidence. Central to an adequate explanatory theory, however, is the specifcation of the mechanisms that are hypothesized to underlie a given set of observations. "Developing theoretical ideas about social processes is to specify some concept of what brings about a certain outcome—a change in political regimes, a new job, an increase in corporate performance, … The development of the conceptualization of change amounts to proposing a mechanism for a social process" (Sørensen, 1998: 239–240). If an educational policy researcher fnds that there is an empirical correlation between schools that have high turnover of teaching staff and high dropout rates, it is very important to investigate whether there is a mechanism that leads from teacher turnover to student dropout. Otherwise, both characteristics may be the joint result of a third factor (inadequate school funding, for example). Sørensen makes the critical point that one cannot select a statistical model for analysis of a set of data without frst asking the question, "What in the nature of the mechanisms do we wish to postulate to link the infuences of some variables with others?" Rather, it is necessary to have a hypothesis of the mechanisms that link the variables before we can arrive at a justifed estimate of the relative importance of the causal variables in bringing about the outcome.

Emphasis on causal mechanisms for adequate social explanation has several favorable benefts for policy research. Policy research is always concerned about

causation: what interventions can be made that would bring about different outcomes? When policy researchers look carefully for the social mechanisms that underlie the processes that they study, they are in a much better position to diagnose the reasons for poor outcomes and to recommend interventions that will bring about better outcomes. Emphasis on the need for analysis of underlying causal mechanisms takes us away from uncritical reliance on uncritical statistical models.

#### *2.3.2 Causal Powers*

Some philosophers of science have argued that substantive theories of causal powers and properties are crucial to scientifc explanation. Leading exponents of this view include Rom Harré (Harré & Madden 1975), Nancy Cartwright (1989), and Stephen Mumford (2009). Nancy Cartwright places real causal powers and capacities at the center of her account of scientifc knowledge (1989). As she and John Dupré put the point, "things and events have causal capacities: in virtue of the properties they possess, they have the power to bring about other events or states" (Dupré & Cartwright, 1988). Cartwright argues, for the natural sciences, that the concept of a real causal connection among a set of events is more fundamental than the concept of a law of nature. And most fundamentally, she argues that identifying causal relations requires substantive theories of the causal powers ("capacities", in her language) that govern the entities in question. Causal relations cannot be directly inferred from facts about association among variables. As she puts the point, "No reduction of generic causation to regularities is possible" (1989: 90). The importance of this idea for sociological research is profound; it confrms the notion shared by many researchers that attribution of social causation depends inherently on the formulation of good, middle-level theories about the real causal properties of various social forces and entities.

Cartwright's philosophy of causation points to the idea of a causal power—a set of propensities associated with a given entity that actively bring about the effect. The causal powers theory rests on the claim that causation is conveyed from cause to effect through the active *powers and capacities* that inhere in the entities making up the cause.

The idea of an ontology of causal powers is that certain kinds of things (metals, gases, military bureaucracies) have internal characteristics that lead them to interact causally with the world in specifc and knowable ways. This means that we can sometimes identify dispositional properties that attach to kinds of things. Metals conduct electricity; gases expand when heated; military bureaucracies centralize command functions (Harré & Madden, 1975). Stephen Mumford and Rani Lill Anjum explore the philosophical implications of a powers theory of causation (2011).

The language of causal powers allows us to incorporate a number of typical causal assertions in the social sciences: "Organizations of type X produce lower rates of industrial accidents"; "paramilitary organizations promote fascist mobilization"; "tenure systems in research universities promote higher levels of faculty research productivity." In each case, we are asserting that a certain kind of social organization possesses, in light of the specifcs of its rules and functioning, a disposition to stimulate certain kinds of participant behavior and certain kinds of aggregate outcomes. This is to attribute a specifc causal power to species of organizations and institutions.

Sociologist James Coleman offered the view that we should distinguish carefully between macro-level social factors and micro-level individual action (Coleman, 1990). He held that all social causation proceeded through three distinct paths: social factors that infuence individual behavior, individuals who interact with each other and create new social facts, and the creation of new macro-level social factors that are the aggregate result of individual actions and interactions at the micro-level. Coleman did not believe that there were direct causal infuences from one macrolevel social fact to another macro-level social fact. Coleman offered a diagram of this view, which came to be known as "Coleman's boat" (Fig. 2.1). On this view, when we say that a certain social entity, structure, or institution has a certain power or capacity, we mean something reasonably specifc: given its confguration, it creates an environment in which individuals commonly perform a certain kind of action. This is the downward strut in the Coleman's boat diagram, labeled 1 in Fig. 2.1. This approach has two important consequences. First, social powers are not "irreducible"—rather, we can explain how they work by analyzing the specifc environment of formation and choice they create. And second, they cannot be regarded as deriving from the "essential" properties of the entity. Change the institution even slightly and we may fnd that it has very different causal powers and capacities. Change the rules of liability for open-range grazing and you get different patterns of behavior by ranchers and farmers (Ellickson, 1991).

#### *2.3.3 Manipulability and Invariance*

Several other aspects of the causal structure of the world have been important in recent discussions of causality in the social sciences. Jim Woodward is a leading exponent of the manipulability (or interventionist) account. He develops his views in detail in his recent book, *Making Things Happen: A Theory of Causal Explanation* (2003). The view is an intuitively plausible one: causal claims have to do with judgments about how the world would be if we altered certain circumstances. If we observe that the concentration of sulfuric acid is increasing in the atmosphere leading to acid rain in certain regions, we might consider the increasing volume of H2SO4 released by coal power plants from 1960 to 1990. And we might hypothesize that there is a causal connection between these facts. A counterfactual causal statement holds that if X (increasing emissions) had not occurred, then Y (increasing acid rain) would not have occurred. The manipulability theory adds this point: if we could remove X from the sequence, then we would alter the value of Y. And this, in turn, makes good sense of the ways in which we design controlled experiments and policy interventions.

Woodward extends this analysis to develop the idea of a relationship that is "invariant under intervention." This idea follows the notion of experimental testing of a causal hypothesis. We are interested in the belief that "X causes Y." We look for interventions that change the state of Y. If we fnd that the only interventions that change Y, do so through their ability to change X, then the X–Y relation is said to be invariant under intervention, and X is said to cause Y (Woodward, 2003: 369–370). Woodward now applies this idea to causal mechanisms. A mechanism consists of separate components that have intervention–invariant relations to separate sets of outcomes. These components are modular: they exercise their infuence independently. And, like keys on a piano, they can be separately activated with discrete results. This amounts to a precise and novel specifcation of the meaning of "causal mechanism": "So far I have been arguing that components of mechanisms should behave in accord with regularities that are invariant under interventions and support counterfactuals about what would happen in hypothetical experiments" (374).

A related line of thought on causal analysis is the idea of *difference-making.* This approach to causation focuses on the explanations we are looking for when we ask about the cause of some outcome. Here philosophers note that there are vastly many conditions that are causally necessary for an event but do not count as being explanatory. Lee Harvey Oswald was alive when he fred his rife in Dallas; but this does not play an explanatory role in the assassination of Kennedy. Crudely speaking, we want to know which causal factors were *salient* and which factors made a difference in the outcome. Michael Strevens (2008) provides an innovative explication of this set of intuitions through the idea of "Kairetic" explanation, a formal way of identifying salient causal factors out of a haystack of causally involved factors in the occurrence of an event guided by generality, cohesion, and accuracy. "To this end, I formulate a recipe that extracts from any detailed description of a causal process a higher level, abstract description that specifes only difference-making properties of the process" (Strevens 2008: xiii).

#### **2.4 Pluralism About Causal Inquiry**

This volume is concerned with the problem of causal inquiry and methods for the discovery of causal relations among factors. How can social researchers identify causal relations among social events and structures? The problem of causal inference is fundamental to methodology in the social and policy sciences. A wellinformed and balanced handbook of political science methodology is provided by Box-Steffensmeier et al. (2008). Here I will provide a brief discussion of several approaches to causal inferences in the social sciences that follows the typology offered there. Especially relevant is Henry Brady's contribution to the volume (Brady, 2008).

In their introduction to the volume, Box-Steffensmeier, Brady, and Collier propose that there are three important kinds of questions to answer when we are investigating the idea of causal relations in the social world. First is semantic: what do we mean by statements such as "A causes B"? Second is ontological: what are the features of the world that we intend to identify when we assert a causal relationship between A and B? And third is epistemological: through what kinds of investigations and processes of inference can we establish the likelihood of a causal assertion about the relationship that exists among two or more features of the social world? The last question brings us to scientifc methodology and a variety of techniques of causal inquiry and inference. However, Box-Steffensmeier, Brady, and Collier are correct in asserting the prior importance of the other two families of questions. We cannot design a methodology of inquiry without having a reasonably well-developed idea of what it is that we are searching for, and that means we must provide reasonable answers to the semantic and ontological questions about causation frst. The editors also make a point that is central to the current chapter as well, in favor of a pluralism of approaches to the task of causal inquiry in the social sciences (2008: 29). There is no uniquely best approach to causal inquiry in the social and policy sciences. The editors refer explicitly to a range of approaches that can be used to investigate causation in the social world: qualitative and quantitative investigation, small-n or large-n studies, experimental data, detailed historical narratives, and other approaches.

Henry Brady (2008) provides a useful typology of several families of methods of inquiry and inference that have developed within the social sciences and that fnd a clear place within the semantic and ontological framework of causation that is developed in this chapter. Brady distinguishes among "neo-humean regularity" approaches, counterfactual approaches, manipulation approaches, and mechanism approaches. And he shows how a wide range of common research methods in the social sciences fall within one or the other of these rubrics. Each of these families of approaches derives from a crucial feature of what we mean by a causal relationship: the fact that causes commonly produce their effects, giving rise to observable regularities; the fact that causes act as suffcient and necessary conditions for their effects, giving rise to the possibility of making inferences about counterfactual scenarios; the fact that causes produce or inhibit other events, giving rise to the possibility of intervening or manipulating a sequence of events; and the fact that causal relations are real and are conveyed by specifc (unobservable) sequences of mechanisms leading from cause to effect, giving rise to the importance of attempting to discover the operative mechanisms.

Brady's typology suggests a variety of avenues of causal inquiry that are possible in the social sciences, given the foregoing analysis of social causes. The ideas sketched in previous sections about the ontology of social causation support multiple avenues for discovering causation. Causes produce their effects, causes work through mechanisms, causal relationships should be expected to result in strong associations among events, and causal necessity supports counterfactual reasoning. We can thus design methods of inquiry that take advantage of the various of ontological characteristics of social causation.

First, the primacy of "real underlying causal mechanisms" suggests that direct research aimed at discovery of the social pathways through which a given outcome is produced by the actions of individual actors within given institutional and normative circumstances is likely to be fruitful. Theory formation about the "institutional logics" created by a given institutional setting can be supplemented by direct study of cases to attempt to identify the pathways hypothesized (Thornton et al., 2012). These insights into the ontology of causation provide encouragement for case-based methods of inquiry, including process tracing, comparative studies, and testing of middle-level social theories of mechanisms. This is a set of methodological ideas supporting causal inquiry developed in detail by George and Bennett (2005), Steinmetz (2004, 2007), and Ermakoff (2019).

Second, the logic of necessary and suffcient conditions associated with the concept of a cause implies methods of research based on experimentation and observation. If we hypothesize that X is a necessary condition for the occurrence of Y, we can design a research study that searches for cases in which Y occurs but X does not. Ragin (1987), Mill (1988), and Tarrow (2010) describe the logic of such cases. The logic of necessary and suffcient conditions also supports research designs based on experimental and quasi-experimental methods—research studies in which the researcher attempts to isolate the phenomenon of interest and observes the outcomes with and without the presence of the hypothetical causal factor. Woodward (2003) illustrates the underlying logic of the experimental approach.

John Stuart Mill's methods of similarity and difference (1988) derive from this feature of the logic of causation. If we believe that A1 & A2 are jointly suffcient to produce B, we can evaluate this hypothesis by fnding a number of cases in which A1, A2, and B occur and examine whether there are any cases where A1 & A2 are present but B is absent. If there is such a case, then we can conclude that A1 & A2 are not suffcient for B. Likewise, if we believe that A3 is necessary for the occurrence of B, we can collect a number of cases and determine whether there are any instances where B occurs but A3 is absent. If so, we can conclude that W is not necessary for the occurrence of B.

#### *2.4.1 Case Studies and Process Tracing*

Alexander George and Andrew Bennett (2005) argue for the value of a case study method of social research. The core idea is that investigators can learn about the causation of particular events and sequences by examining the events of the case in detail and in comparison with carefully selected alternative examples. Here is how George and Bennett describe the case study method:

The method and logic of structured, focused comparison is simple and straightforward. The method is "structured" in that the researcher writes general questions that refect the research objective and that these questions are asked of each case under study to guide and standardize data collection, thereby making systematic comparison and cumulation of the fndings of the cases possible. The method is "focused" in that it deals only with certain aspects of the historical cases examined. The requirements for structure and focus apply equally to individual cases since they may later be joined by additional cases. (George & Bennett, 2005: 67)

The case study method is designed to identify causal connections within a domain of social phenomena. How is that to be accomplished? The most important tool that George and Bennett describe is the method of process tracing. "The process-tracing method attempts to identify the intervening causal process—the causal chain and causal mechanism—between an independent variable (or variables) and the outcome of the dependent variable" (206). Process tracing requires the researcher to examine linkages within the details of the case they are studying and then to assess specifc hypotheses about how these links might be causally mediated.

#### *2.4.2 Quantitative Research Based on Observational Data*

Quantitative studies of large populations are supported by this theory of causation, if properly embedded within a set of hypotheses about causal relations among the data. In his presentation of the logic of "structural equation modeling" (SEM) and causal inference, Judea Pearl (2000, 2021) is entirely explicit in stating that pure statistical analysis of covariation cannot establish causal relationships. In particular, Pearl argues that a causal SEM requires:

A set A of qualitative causal assumptions, which the investigator is prepared to defend on scientifc grounds, and a model MA that encodes these assumptions. (Typically, MA takes the form of a path diagram or a set of structural equations with free parameters. A typical assumption is that certain omitted factors, represented by error terms, are uncorrelated with some variables or among themselves, or that no direct effect exists between a pair of variables.) (Pearl, 2021: 71)

Aage Sørensen takes a similar view and describes the underlying methodological premise of valid quantitative causal research in these terms:

Understanding the association between observed variables is what most of us believe research is about. However, we rarely worry about the functional form of the relationship. The main reason is that we rarely worry about how we get from our ideas about how change is brought about, or the mechanisms of social processes, to empirical observation. In other words, sociologists rarely model mechanisms explicitly. In the few cases where they do model mechanisms, they are labeled mathematical sociologists, not a very large or important specialty in sociology. (Sørensen, 2009: 370)

Purely quantitative studies do not establish causation on their own; but when provided with accompanying hypotheses about the mechanisms through which the putative causal infuences obtain, quantitative study can substantially increase our confdence in inferences about causal relationships among factors. Quantitative methods for research on causation advanced signifcantly through the development of structural equation models (SEMs) and the structural causal model methodology described by Judea Pearl and others (Pearl, 2000; Pearl, 2009, 2021). This approach explicitly endorses the notion that quantitative methods require background assumptions about causal mechanisms: "one cannot substantiate causal claims from associations alone, even at the population level—behind every causal conclusion there must lie some causal assumption that is not testable" (Pearl, 2009: 99).

#### *2.4.3 Randomized Controlled Trials and Quasi-experimental Research*

The method of randomized controlled trials (RCT) is sometimes thought to be the best possible way of establishing causation, whether in biology or medicine or social science. An experiment based on random controlled trials can be described simply. It is hypothesized that:

(H) A causes B in a population of units P.

An experiment testing H is designed by randomly selecting a set of individuals from P into Gtest (the test group) and randomly assigning a different set of individuals from P into Gcontrol (the control group). Gtest and Gcontrol are exposed to A (the treatment) under carefully controlled conditions designed to ensure that the ambient conditions surrounding both tests are approximately the same. The status of each group is then measured with regard to B, and the difference in the value of B between the two groups is said to be the "average treatment effect" (ATE). If the average treatment effect is greater than zero, there is prima facie reason to accept H.

This research methodology is often thought to capture the logical core of experimentation and is sometimes thought to constitute the strongest evidence possible for establishing or refuting a causal relationship between A and B. It is thought to represent a purely observational way of establishing causal relations among factors. This is so because of the random assignment of individuals to the two groups (so potentially causally relevant individual differences are averaged out in each group) and because of the strong efforts to isolate the administration of the test so that each group is exposed to the same unknown factors that may themselves infuence the outcome to be measured. As Handley et al. (2018) put the point: "Random allocation minimizes selection bias and maximizes the likelihood that measured and unmeasured confounding variables are distributed equally, enabling any differences in outcomes between the intervention and control arms to be attributed to the intervention under study" (Handley et al., 2018: 6). The social and policy sciences are often interested in discovering and measuring the causal effects of large social conditions and interventions—"treatments", as they are often called in medicine and policy studies. It might seem plausible, then, that empirical social science should make use of random controlled trials whenever possible, in efforts to discover or validate causal connections.

However, this supposed "gold standard" status of random controlled trials has been seriously challenged in the last several years. Serious methodological and inferential criticisms have been raised of common uses of RCT experiments in the social and behavioral sciences, and philosopher of science Nancy Cartwright has played a key role in advancing these criticisms. Cartwright and Hardie (2012) provided a strong critique of common uses of RCT methodology in areas of public policy, and Cartwright and others have offered convincing arguments to show that inferences about causation based on RCT experiments are substantially more limited and conditional than generally believed.

A pivotal debate among experts in a handful of felds about RCT methodology took place in a special issue of *Social Science and Medicine* in 2018. This volume is an essential reading for anyone interested in causal reasoning. Especially important is Deaton and Cartwright (2018). The essence of their critique is summed up in the abstract: "We argue that the lay public, and sometimes researchers, put too much trust in RCTs over other methods of investigation. Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates" (Deaton & Cartwright, 2018). Deaton and Cartwright provide an interpretation of RCT methodology that places it within a range of comparably reliable strategies of empirical and theoretical investigation, and they argue that researchers need to choose methods that are suitable to the problems that they study.

One of the key concerns they express has to do with extrapolating and generalizing from RCT studies (Deaton & Cartwright, 2018: 3). A given RCT study is carried out in a specifc and limited set of cases, and the question arises whether the effects documented for the intervention in this study can be extrapolated to a broader population. Do the results of a drug study, a policy study, or a behavioral study give a basis for believing that these results will obtain in the larger population? Their general answer is that extrapolation must be done very carefully. "We strongly contest the often-expressed idea that the ATE calculated from an RCT is automatically reliable, that randomization automatically controls for unobservables, or worst of all, that the calculated ATE is true [of the whole population]" (Deaton & Cartwright, 2018: 10).

The general perspective from which Deaton and Cartwright proceed is that empirical research about causal relationships—including experimentation—requires a broad swath of knowledge about the processes, mechanisms, and causal powers at work in the given domain. Here their view converges philosophically with that offered by Pearl above. This background knowledge is needed in order to interpret the results of empirical research and to assess the degree to which the fndings of a specifc study can plausibly be extrapolated to other populations.

These methodological and logical concerns about the design and interpretation of experiments based on randomized controlled trials make it clear that it is crucial for social scientists to treat RCT methodology carefully and critically. Deaton and Cartwright agree that RCT experimentation is a valuable component of the toolkit of sociological investigation. But they insist that it is crucial to keep several philosophical points in mind. First, there is no "gold standard" method for research in any feld; rather, it is necessary to adapt methods to the nature of the data and causal patterns in a given feld. Second, Cartwright (like most philosophers of science) is insistent that empirical research, whether experimental, observational, statistical, or Millian, always requires theoretical inquiry into the underlying mechanisms that can be hypothesized to be at work in the feld. Only in the context of a range of theoretical knowledge is it possible to arrive at reasonable interpretations of (and generalizations from) a set of empirical fndings.

Many issues of causation in the social and policy sciences cannot be addressed in a controlled laboratory environment. In particular, in many instances, it is impossible to satisfy the condition of random assignment of individuals to control and treatment groups. Much data available for social science and policy research is gathered from government databases (Medicaid, Department of Education, Internal Revenue Service) and was assembled for statistical and descriptive purposes. Hypotheses about the causes of failing schools, ineffective prison reforms, or faulty regulatory systems are not amenable to the strict requirements of randomized controlled trials. However, social and policy scientists have developed practical methods for probing causation in complex social settings using natural experiments, feld experiments, and quasi-experiments.

Quasi-experiments, feld experiments, and natural experiments are sometimes defned as "randomized controlled trials carried out in a real-world setting" (Teele, 2014: 3). This defnition is misleading, because the crucial feature of RCTs is absent in a quasi-experiment: the random assignment of units to control and treatment groups. What quasi-experiments have in common is an effort to replace random assignments of units to control and treatment groups with some other way of stratifying available data that would permit inference about cause and effect. Quasiexperiments involve making use of observational data about similar populations that have been exposed to different and potentially causally relevant circumstances. The researcher then attempts to discover treatment effects based on statistical properties of the two groups. In this volume, Battistin and Bertoni (Chap. 3) describe an ingenious set of constructs to uncover the effects of cheating on educational performance examination scores in Italy, based on what they refer to as "instrumental variables" and "regression discontinuity design." The former is a component of the composition of the control group that can be demonstrated to be random. The authors show how this randomness can be exploited to discover the magnitude of effects of the non-random components in the composition of the control group. The latter term takes advantage of the fact that some data sets (class size in Italy, for example) are "saw-toothed" with respect to a known variable. The example they use is the government policy in Italy that regulates class size. School populations increase linearly, but government policy establishes the thresholds at which a school is required to create a new class. So class size increases from the minimum to the maximum, then declines sharply, and continues. This fact can be exploited to examine school performance in classes currently near the minimum versus classes currently near the maximum. This approach removes school population size from the selection and therefore succeeds in removing a confounding causal infuence, which is exactly what randomization was intended to do.

The reasoning illustrated in Battistin and Bertoni (Chap. 3) is admirable in the authors' effort to squeeze meaningful causal inferences out of a data set that is awash with non-random elements. However, as Battistin and Bertoni plainly demonstrate, it is necessary to be rigorously critical in developing and evaluating these kinds of research designs and inferences. Stanley Lieberson's *Making It Count* (1985) formulates a series of diffcult challenges for the logic of quasi-experimental design that continues to serve as a cautionary tale for quantitative social and policy research. Lieberson believes that there are almost always unrecognized forms of selection bias in the makeup of quasi-experimental research designs that potentially invalidates any possible fnding. Cartwright and Hardie (2012) extend these critical points by underlining the limitations on generalizability (external validity) that are endemic to experimental reasoning. So selection bias is still a possibility that can interfere with valid causal reasoning in the design of a quasi-experiment.

What conclusions should we draw about experiments and quasi-experiments? What is the status of randomized controlled trials as a way of isolating causal relationships, whether in sociology, medicine, or public policy? The answer is clear: RCT methodology is a legitimate and important tool for sociological research, but it is not fundamentally superior to the many other methods of empirical investigation and inference in use in the social sciences. Methodologies supporting the design and interpretation of quasi-experiments are also subject to important methodological cautions in the social science and policy studies. It is necessary to remain critical and refective in assessing the assumptions that underlie any social science research design, including randomized controlled trials and sophisticated quasi-experiments.

#### *2.4.4 Generative Models and Simulation Methods*

Advances in computational power and software have made simulations of social situations substantially more realistic than in previous decades. An early advance took place in general equilibrium theory, leading to a set of models referred to as "computable general equilibrium models." Instead of using a three-sector model to

illustrate the dynamics of a general equilibrium model of a market economy, it is now feasible to embody assumptions for one hundred or more industries and work out the equilibrium dynamics of this substantially more realistic representation of an economic system using a computable model (Taylor, 1990). Of special interest for political scientists and policy scholars is the increasing sophistication of agentbased models (de Marchi and Page, 2008). Kollman et al. (2003) provide a highly informative overview of the current state of the feld in their *Computational Models in Political Economy*. They describe the chief characteristics of an agent-based model in these terms:

The models typically have four characteristics, or methodological primitives: agents are diverse, agents interact with each other in a decentralized manner, agents are boundedly rational and adaptive, and the resulting patterns of outcomes comes often do not settle into equilibria…. The purpose of using computer programs in this second role is to study the aggregate patterns that emerge from the "bottom up" (Kollman et al. 2003: 3).

An often-cited early application of agent-based models was Thomas Schelling's segregation model. Schelling demonstrated that residential segregation was likely to emerge from a landscape in which two populations had tolerant but fnite requirements for the ethnic composition of their neighborhoods (Schelling, 1978). A random landscape populated with a mix of the two populations almost always develops into a segregated landscape of the populations after a number of iterations. Agentbased models can be devised to provide convincing "generative" explanations of a range of collective phenomena; and when developed empirically by calibrating the assumptions of the model to current empirical data, their results can result in reasonable predictions about the near-term future of a given social phenomenon (Epstein, 2006).

We can look at ABM simulation techniques as a form of "mechanisms" theory. A given agent-based model is an attempt to work out the dynamics of individuallevel actions at the meso- and macro-level; and this kind of result can be interpreted as an empirically grounded account of the mechanisms that give rise to a given kind of social phenomenon. This feature of agent-based model methodology gives researchers yet another tool through which to probe the social world for causal relations among social features.

#### **2.5 Realism and Methodological Pluralism**

Let us draw to a close. Here are some chief features of social science research that proceeds in ways consistent with this realist view of causation in the social world:

Central in these ideas is the value of *methodological pluralism*. The ultimate goal of research in the social and policy sciences is to discover causal relationships and causal mechanisms. We want to know how the social world works and how we might intervene to change outcomes that are socially undesirable. There are a wide range of methods of inquiry and validation that are used in the social sciences: ethnographic methods (interviews and participant observation), case study analysis, comparative case study research, models and simulations of social arrangements of interest, and large-scale statistical studies. The philosophical position of methodological pluralism is the idea that there is a place in social and policy research for all of these tools and more besides. What holds them together is the fact that in each case, our ultimate concern is to discover the causal relationships that appear to hold in the social world and the mechanisms that underlie these relationships.

The central conclusion to be drawn here is that multiple methods of empirical investigation are available, and our research efforts will be most productive when we are able to connect empirical fndings with hypotheses about social-causal mechanisms that are both theoretically and observationally supported. And equally importantly, it is crucial for researchers from different methodological traditions to interact with each other so that their underlying assumptions about causation and causal inference can be refned and validated.

#### **Review Questions**

#### **References**

Archer, M. S., Decoteau, C., Gorski, P., Little, D., Porpora, D., Rutzou, T., Smith, C., Steinmetz, G., & Vandenberghe, F. (2016). What is critical realism? *Perspectives, 38*(2), 4–9. http://www. asatheory.org/current-newsletter-online/what-is-critical-realism

Bhaskar, R. (1975). *A realist theory of science*. Leeds Books.

#### *Suggested Readings*

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 3 Counterfactuals with Experimental and Quasi-Experimental Variation**

**Erich Battistin and Marco Bertoni**

**Abstract** Inference about the causal effects of a policy intervention requires knowledge of what would have happened to the outcome of the units affected had the policy not taken place. Since this counterfactual quantity is never observed, the empirical investigation of causal effects must deal with a missing data problem. Random variation in the assignment to the policy offers a solution, under some assumptions. We discuss identifcation of policy effects when participation to the policy is determined by a lottery (randomized designs), when participation is only partially infuenced by a lottery (instrumental variation), and when participation depends on eligibility criteria making a subset of participant and non-participant units as good as randomly assigned to the policy (regression discontinuity designs). We offer guidelines for empirical analysis in each of these settings and provide some applications of the methods proposed to the evaluation of education policies.

#### **Learning Objectives**

By studying this chapter, you will:

E. Battistin

M. Bertoni (\*) University of Padova, Padova, Italy e-mail<EMAIL_ADDRESS>
University of Maryland, College Park, MD, USA e-mail<EMAIL_ADDRESS>
#### **3.1 Introduction**

Do smaller classes yield better school outcomes? To answer this and many similar questions, one needs to compare the outcome in the *status quo* (a large class) to the outcome that would have been observed if the input of interest was set to a different level (a small class). The comparison of students enrolled in small and large classes is always a tempting avenue to answer this causal question. As this comparison involves different students, its validity rests on the assumption that students currently enrolled in small and large classes would have presented the same outcome, on average, had they been exposed to the same number of classmates. This remains an untestable assumption that must be discussed on a caseby-case basis.

The chapter discusses ways to combine policy designs and data to corroborate the validity of this assumption. Sections 3.2 and 3.3 introduce the counterfactual causal analysis talk. They describe the concepts of treatments, potential outcomes and causal effects, and the attributes characterizing the validity of a research design. Section 3.4 is about the beauty and limitations of randomized assignment to "treatment" (e.g., a small class) and paves the way for the discussion in the following sections. Specifcally, these sections deal with methods for causal reasoning when randomization is not feasible. Section 3.5 provides an example of instrumental variation in treatment assignment arising from a natural experiment. Section 3.6 is devoted to the closest cousin to randomization, the regression discontinuity design. Section 3.7 offers some concluding remarks.

Our discussion of empirical methods for causal reasoning is far from exhaustive. For example, we do not discuss research designs that exploit longitudinal data and rely on assumptions on pre-treatment outcome trends (e.g., difference-in-differences and synthetic control methods). Similarly, we do not cover matching methods (see Chap. 4 of this volume). In addition, our presentation will mostly focus on the reasoning underlying design-based identifcation and will only barely touch issues related with estimation. The interested reader can refer to the book by Angrist and Pischke (2008) for a discussion of these topics.

#### **3.2 Causation and Counterfactual Impact Evaluation: The Jargon**

It is useful to start by clarifying what we mean by "causes" and "treatment effects." We consider a population of units indexed by *i*, with *i* = 1, …, *N*. Although our narrative will often consider individuals as the units of analysis, the same setting extends to other statistical units such as households, villages, schools, or municipalities.

#### *3.2.1 Causes as Manipulable Treatments*

In the population we study, some units are exposed to a cause, which is a treatment or intervention that manipulates factors that may affect a certain outcome. For instance, we might be interested in studying whether class size at primary school affects student performance. Class size here is the treatment and performance is the outcome, which is typically measured using standardized tests. In many countries, class size formation depends on grade enrollment so that, across cohorts, the number of students in the class may change because enrollment changes or because a specifc policy affects the regulation. We will use the words "cause", "treatment", or "intervention" interchangeably.

The avenue we take here has some limitations, as not all causes worth considering are manipulable in practice (consider, for example, gender, ethnicity, or genetic traits). Moreover, the design-based approach we describe below may be coarse at times and aimed at shedding light on one particular aspect of a more articulated model. For example, empirical evidence on the causal effects of class size on achievement bundles up the possible contribution of multiple channels that may lead to a better learning environment in small classes. The investigation of channels and mechanisms behind the uncovered effects calls for theories and structural models. The most relevant question to consider turns on the quality of the design-based strategy and on our faith to prop up a more elaborate theoretical framework.

We focus only on binary treatments, that is, we assume that treatment status is described by a binary random variable *Di* taking value one if unit *i* is exposed to treatment ("treated" or "participant") and zero otherwise ("untreated", "non-participant", or "control"). In the class size example, this amounts to considering a setting in which students can be enrolled in small or large classes. The extension to the case of multi-valued or continuous treatment (for example, the number of classmates) is logically identical but requires a more cumbersome notation. More in general, the binary case is always worth of consideration even in a more general context as it helps understand the main challenges in the quest for detecting causal effects. A related issue concerns public policies that are designed as "bundles" of multiple components. In those cases, policy-makers are often interested in disentangling the effect of every component of the policy. We abstract from this problem in our discussion, but emphasize here that the ability to address this question will depend, in general, on the exposure of subjects to different components.

We must take a stand on the reasons why different units end up having a value of *Di* equal to one or zero. This is the so-called "assignment rule" and is at the core of any evaluation study. Assignment to treatment can be totally random. In our class size example, this happens when students are randomized to a small or a large class with equal probability and independently of socio-economic background or past performance. When randomization is not at work, participation to treatment is most likely the result of choices made by the units themselves, administrators of the program, or policy makers. For example, parents can choose to enroll their children in schools with smaller classes in the hope of a better learning environment. Finally, participation to treatment may depend on admission rules that units must comply with. The case of class size formation based on total enrollment is a good example, as the chance of being enrolled in a small class depends on a school's yearly total recruitment. As we shall see, our ability to assess causal effects grows with knowledge of the assignment rule.

#### *3.2.2 Effects as Differences Between Factual and Counterfactual Outcomes*

It is essential to set the stage for a transparent defnition of the treatment effect. To do so, we defne *Yi*(1) and *Yi*(0) as the potential outcomes experienced if unit *i* is treated (*Di* = 1) or untreated (*Di* = 0), respectively. The unit-level treatment effect of *Di* on *Yi* is the difference between *Yi*(1) and *Yi*(0): Δ*i* = *Yi*(1) − *Yi*(0). Decades of empirical studies using micro-data analyses have taught us that treatment effects most likely vary across units or groups of units with very similar demographics. The notation employed here accommodates for this possibility (the manuals by Angrist & Pischke, 2008, and Imbens & Rubin, 2015, use the same approach).

The defnition of Δ*i* unveils the fundamental problem that we face when we want to estimate this quantity from the data. While the two potential outcomes can be logically defned for each unit, they can never be observed simultaneously for the same unit. This is true regardless of the assignment rule and the richness or sample size of data we will ever work with. Specifcally, the data can reveal only *Yi*(1) for units with *Di* = 1 and *Yi*(0) for units with *Di* = 0. We can, therefore, express the observed outcome *Yi* as follows: *Yi* = *Yi*(1)*Di* + *Yi*(0)(1 − *Di*) = *Yi*(0) + *Di*(*Yi*(1) − *Yi*( 0)). As simple as this can be, lack of observability of both potential outcomes implies lack of observability of the unit-level effect Δ*i*. We can think of the unitlevel causal effect as the difference between an observed (factual) and an unobserved (counterfactual) potential outcome. Factual quantities are those that can be computed from the data. Counterfactual quantities can be logically defned but can never be computed from data. For treated units, we observe *Yi* = *Yi*(1) and *Yi*(0) is the counterfactual. The opposite is true for control units, for whom we observe *Yi* = *Yi*(0) and *Yi*(1) is the counterfactual.

One way to get around this limitation is to settle for less than unit-level effects. We might be interested in considering average treatment effects for the population or only for some sub-groups. For instance, we defne the average treatment effect (ATE) as the average of the individual-level treatment effect in the whole population: *ATE* = *E*(*Yi*(1) − *Yi*(0)). This parameter refects our expectation of what would happen if we were to expose to treatment a randomly chosen unit from the population. Alternatively, we can consider the average treatment effect for the treated (ATT), which describes our expectation for units who have been exposed to treatment: *ATT* = *E*(*Yi*(1) − *Yi*(0)| *Di* = 1). Analogously, the average treatment effect for the non-treated (ATNT) is informative about what would have happened to the untreated if they had been exposed to the intervention:

$$ATNT = E\left(Y\_i\left(1\right) - Y\_i\left(0\right) \mid D\_i = 0\right).$$

Whether any of the above causal parameters can be retrieved from the data will have to be discussed on a case-by-case basis our understanding of the assignment rule plays a key role in this discussion.

#### *3.2.3 What the Data Tell (And When)*

Our journey to learn about treatment effects begins by comparing features of the observed outcome *Yi* for treated and control units. For instance, the data reveal the average outcomes for treated units, *E*(*Yi*|*Di* = 1), and control units, *E*(*Yi*|*Di* = 0). Recalling the defnition of potential outcomes, the naïve comparison of average outcomes by treatment group, *E*(*Yi*|*Di* = 1) − *E*(*Yi*| *Di* = 0) = *E*(*Yi*(1)|*Di* = 1) − *E*(*Yi*(0)| *Di* = 0), conveys the correlation between the treatment, *Di*, and the outcome, *Yi*.

The causal interpretation of such naïve comparison is controversial in most cases. To see why, we can add and subtract from the right-hand side of the previous equation the quantity *E*(*Yi*(0)|*Di* = 1). This is a counterfactual quantity, as the outcome *Yi*(0) cannot be observed for treated units, and represents what would have happened to treated units had they not participated to treatment. We can arrange the terms and write:

$$\begin{aligned} E\left(Y\_i \mid D\_i = 1\right) - E\left(Y\_i \mid D\_i = 0\right) &= E\left(Y\_i(1) - Y\_i\left(0\right) \mid D\_i = 1\right) + E\left(Y\_i(0) \mid D\_i = 1\right) \\ -E\left(Y\_i(0) \mid D\_i = 0\right). \end{aligned} \tag{5.1}$$

It follows that the naïve comparison on the left-hand side of Eq. 3.1 is equal to the sum of the ATT and the term *E*(*Yi*(0)| *Di* = 1) − *E*(*Yi*(0)| *Di* = 0), which is often called "selection bias". It is worth noting that this representation does not hinge on any assumptions. It is the result of a simple algebraic trick and, as such, is always true.

Selection bias is an error in the causal reasoning. It is different from zero when, in the absence of treatment, the group with *Di* = 1 would have performed differently from the group with *Di* = 0. The same concept is conveyed by the "correlation is not causation" *motto*: correlation (the naïve treatment–control comparison) has no causal interpretation (that is, it does not coincide with the ATT) unless the selection bias is zero. This reframes the quest for causal effects as a discussion on the existence of selection bias. A non-zero bias follows from having groups defned by *Di* = 1 and *Di* = 0 that are not representative of the same population, in the sense that participation to treatment depends on non-random selection. At the end of the day, selection bias refects compositional differences between treatment and control

units. Taking up our class size example, parents with a strong preference for smaller classes are most likely selected in terms of socio-economic background and demographics. If this selection translates into a better learning potential of their children, forming classes as a refection of parental preference must create dis-homogenous groups of students. In this case, detecting a correlation between class size and achievement might just reveal dis-homogeneity across classes rather than a true causal effect of class size.

Importantly, for the time being, we are agnostic about whether this dishomogeneity concerns characteristics of units that are observed in the data at hand or not. In fact, any strategy that can adjust for compositional differences between treated and control units also corrects for this bias. One leading example to consider here is randomization. When classes are formed by a coin toss, composition is the same. Even when it is because of sampling variability, differences in composition must be as good as random. We will formalize this idea in Sect. 3.4, below. Instead, Chapters 4 and 5 in this volume present methods to alleviate imbalances along observable dimensions and discuss the identifying assumptions that permit to reach causal conclusions once these differences are eliminated.

#### **3.3 Shades of Validity**

The assessment of a causal channel from treatment to the outcome depends on the properties of the research design. In short, this is the toolbox of empirical methods that allows one to distinguish between correlation and causality. Any strategy falling short on this minimum requirement is not a valid option to consider for a good researcher. On the other hand, a good research design must be able to detect precisely the causal relationship of interest. That is, you do not want your design to be underpowered for the size of the treatment effect. Finally, the ideal research design should be able to provide causal statements that apply to the largest share of units in the population and extend to other contexts and times. The concern here is one of generalizability, which is of fundamental importance for offering evidence-based policy recommendations. Causal talk makes use of these three ideas of validity in the development of a research design. This is what we will discuss briefy next. The seminal textbook by Cook and Campbell (1979) provides a deeper treatment of these topics.

#### *3.3.1 Internal Validity: The Ability to Make a Causal Claim from a Pattern Documented in the Data*

Internal validity concerns the ability of assessing whether the correlation between treatment and outcome depicts a causal relationship or if it could have been observed even in the absence of the treatment. Therefore, internal validity is solely concerned with the presence of selection bias. It is achieved under a *ceteris paribus* comparison of units, when all else but the treatment is kept constant between treated and control units. As we discussed above, this calls for the same composition of treatment (small class) and control (large class) units. An internally valid conclusion is the one without selection bias. One of the main advantages of using randomization is that such *ceteris paribus* condition is met by design. Because of this, a properly conducted randomization yields internally valid causal estimates.

#### *3.3.2 Statistical Validity: Measuring Precisely the Relationship Between Causes and Outcomes in the Data*

Statistical validity refers to the appropriate use of statistical tools to assess the extent of correlation between treatment and outcomes. It is fundamentally concerned with standard errors and accuracy in assessing a statistical relationship. The main question addressed by statistical validity is whether the chosen data and techniques of statistical inference can produce precise estimates of very small treatment effects (a statistically precise zero) or if, instead, the research design will likely produce statistical zeros (a statistically insignifcant effect). An insignifcant effect that is statistically different from zero is a powerful oxymoron to summarize the idea underlying statistical validity.

#### *3.3.3 External Validity: The Ability to Extend Conclusions to a Larger Population, over Time and Across Contexts*

External validity is about the predictive value of a particular causal estimate for times, places, and units beyond those represented in the study that produced it. The concern posed by external validity is one of generalizability and out-of-sample prediction. For example, an internally valid estimate for a given sub-group of the population might not be informative about the treatment effect for other (potentially different and policy-relevant) sub-groups. Similarly, ATT is, in general, different from ATE. Replicability of the same results in other contexts and times is of fundamental interest for providing policy recommendations.

#### **3.4 Random Assignment Strengthens Internal Validity**

As Andrew Leigh puts it in his book "*Randomistas: How Radical Researchers Are Changing the World*," (Leigh, 2018) randomized controlled trials (RCTs) use "the power of chance" to assign the groups. Randomization can be achieved by fipping a coin, drawing the shorter straw, or using a computer to randomly assign statistical units to groups. In any of these cases, the result would be the same: the treatment and the control group are random samples from the same population.

Random assignment ensures that treatment and control units are the same in every respect, including their expected *Yi*(0). It follows that, in RCTs, selection bias must be zero since *E*(*Yi*(0)| *Di* = 1) = *E*(*Yi*(0)| *Di* = 0). In other words, what we observe for control units approximates what would have happened to treated units in the absence of treatment. It is worth noting that random assignment does not work by eliminating individual differences, but it rather ensures that the composition of units being compared is the same.

RCTs ensure a *ceteris paribus* (i.e., without confounds) comparison of treatment and control groups. Because of this, an RCT provides an internally valid research design for assessing causality. Evidence in support of this validity can be obtained using pre-intervention measurements. In fact, it is a good practice to collect this information and test the validity of the design by carrying out a battery of "balancing" tests. In a properly implemented randomization, there are no selective differences in the distribution of pre-intervention measurements between treated and control units. This statement does not rule out the possibility of between-group differences arising from sampling variability, which is a problem concerning the statistical validity (that is, the precision of point estimates) of RCTs.

Finally, under random assignment, the naïve comparison will provide internally valid conclusions about the average treatment effect on the treated (ATT), as we have that *E*(*Yi*|*Di* = 1) − *E*(*Yi*| *Di* = 0) = *E*(*Yi*(1) − *Yi*(0)|*Di* = 1). In addition, under randomization, the groups with *Di* = 1 and *Di* = 0 are representative of the same population so that *E*(*Yi*(1) − *Yi*(0)|*Di* = 1) = *E*(*Yi*(1) − *Yi*(0)). This means that the causal conclusions hold for any unit randomly selected from the population.

Random assignment to treatment is not uncommon in numerous felds of the social sciences. One such example is the lottery-based allocation of pupils to schools that are oversubscribed. This alternative to the traditional priority criterion based on proximity should dampen school stratifcation caused by wealthy parents buying houses in the close vicinity of high-quality schools. As a result, among the pool of applicants to a school where oversubscription is resolved by a lottery, getting a seat or not is completely random. Some researchers (see Cullen et al., 2006, for an example) have exploited this to evaluate the educational effects of attending one's preferred school.

Another example is the Oregon Health Insurance Experiment (see Finkelstein et al., 2012). Medicaid is one of the landmark US public health insurance programs and provides care for millions of low-income families. In 2008, the state of Oregon extended coverage of Medicaid by selecting eligible individuals with a lottery. This gave researchers the unique opportunity to provide credible causal estimates of the effect of health insurance eligibility on health care utilization, medical expenditure, medical debt, health status, earnings, and employment.

Although RCTs are considered as the "gold standard" for providing internally valid estimates of causal effects, they are not without shortcomings (see the excellent surveys by Dufo et al., 2008 and Peters et al., 2018). External validity is often perceived as the main limitation and more so for small-scale experiments on very

specifc subpopulations. Bates and Glennerster (2017) propose a framework to discuss generalizability based on four steps: identify the theory behind the program; check if local conditions hold for that theory to apply; evaluate the strength of the evidence for the required general behavioral change; evaluate whether the implementation process can be carried out well. External validity is granted if these four conditions apply in a context different from the one where the experiment was conducted. Statistical validity as well may challenge the signifcance of many smallscale experiments (see Young, 2019).

RCTs have other limitations. Many RCTs are carried out as small-scale pilots that shall be eventually scaled up to the entire population. Causal reasoning in this context must consider the general equilibrium effects arising from this change in scope. These effects are concerned with the possible externalities for non-participants when the policy is implemented on a larger scale and the implications for market equilibria. An additional concern about RCTs is that the sole fact of being "under evaluation" may generate some behavioral response that has nothing to do with a treatment effect.1 Replicability of experiments also has been called into question in many felds of the social sciences (see Open Science Collaboration, 2015, for psychology and Camerer et al., 2016, for economics).

What happens when randomization is not a feasible option? This is the question to which we turn next.

#### **3.5 Internally Valid Reasoning Without RCTs: Instrumental Variation**

#### *3.5.1 A Tale of Pervasive Manipulation*

Randomizations obtained by design are not the only way to ensure ceteris paribus comparisons. Randomness in the assignment to treatment may arise indirectly from natural factors or events independently of the causal channel of interest. Under assumptions that we shall discuss, these factors can be used instrumentally to pin down a meaningful casual parameter. The most important takeaway message here is that we must use assumptions to make up for the lack of randomization. Because of this, much of the simplicity of the research design is lost, and internal validity must be addressed on a case-by-case basis. We will present an example of the toolbox for good empirical investigations using administrative data on student achievement and, further below, class size.

Our working example makes use of standardized tests from INVALSI (a government agency charged with educational assessment) for second and ffth graders in Italian schools for the years 2009–2011. Italy is an interesting case study as it is

1Such quirky responses are called "Hawthorne" effects for treated subjects and "John Henry" effects for controls.

characterized by a sharp North–South divide along many dimensions, among which school quality. This divide motivates public interventions to improve school inputs in the South. As testing regimes have proliferated in the country, so has the temptation to cut corners or cheat at the national exam.2 As shown in Fig. 3.1, the South is distinguished by widespread manipulation on standardized tests. INVALSI tests are usually proctored and graded by teachers from the same school, and past work by Angrist et al. (2017) has shown that manipulation takes place during the grading process. Classes with manipulated scores are those where teachers did not grade exams honestly.

Consider the causal effect of manipulation on test scores. As scores are infated, the sign of this effect is obvious. However, the size of the causal effect (that is, by

2Cheating or manipulation is not unique to Italy, as discussed in Battistin (2016).