Parallel Experimentation on Advertising Platforms
Caio Waisman Navdeep S. Sahni Harikesh S. Nair Xiliang Lin


0











Parallel Experimentation on Advertising PlatformsThis article was previously circulated under the title “Parallel experimentation in a competitive advertising marketplace.” Lin, Nair and Waisman were part of  when this research was initiated. The views represent that of the authors, and not . Some aspects of the data, institutional context and implementation are masked to address business confidentiality. We thank Lijing Wang for research assistance, and Jun Hao, Jack Lin, Lei Wu, and Paul Yan for their support, collegiality and collaboration during the project. Thanks to Carlos Carrion, Dean Eckles, Günter Hitsch, Carl Mela, Duncan Simester, Stefan Wager; seminar participants at Amazon, Carlson-UMinn, Facebook, Fuqua-Duke, Haas-Berkeley, Kellogg-Northwestern, Kenan-Flagan-UNC, Tepper-CMU, and at the 2020 Joint Statistical Meetings, Marketing Science, Virtual Quant Marketing Seminar and the 2019 Choice Symposium conferences for thoughtful comments and suggestions. Please contact the authors at  (Waisman),  (Sahni),  (Nair) or  (Lin) for correspondence. Previous versions: March 26, May 28, 2019. 
    Caio Waisman   Navdeep S. Sahni    Harikesh S. Nair   Xiliang Lin
    This draft: December 30, 2023
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





This paper studies the measurement of advertising effects on online platforms when parallel experimentation occurs, that is, when multiple advertisers experiment concurrently. It provides a framework that makes precise how parallel experimentation affects this measurement problem: while ignoring parallel experimentation yields an estimate of the average effect of advertising in-place, this estimate has limited value in decision-making in an environment with advertising competition; and, accounting for parallel experimentation provides a richer set of advertising effects that capture the true uncertainty advertisers face due to competition. It then provides an experimental design that yields data that allow advertisers to estimate these effects and implements this design on , a large e-commerce platform that is also a publisher of digital ads. Using traditional and kernel-based estimators, it obtains results that empirically illustrate how these effects can crucially affect advertisers' decisions. Finally, it shows how competitive interference can be summarized via simple metrics that can assist decision-making. 





Keywords: experimentation, A/B/n testing, causal inference, digital advertising, e-commerce, platforms.






Introduction

Experimentation is a fixture of the digital age. Modern technology companies routinely run possibly hundreds or more of randomized controlled trials daily on their digital platforms to measure the effect of various interventions, such as advertising, and to formulate business strategy. Several digital platforms now offer “experimentation as a service” to their participating clients, whereby the platform experiments on behalf of firms, acting as their agent to facilitate measurement. Examples include 's Target A/B Tests, and  and 's Optimize for webpages and Conversion Lift and Brand Lift for advertising.

The proliferation of experimentation on platforms has created pervasive situations where competing firms experiment concurrently. Surprisingly, while the literature on experimentation for measuring the effects of digital interventions has mushroomed (e.g., see ), the impact of multiple overlapping experimentation by competing firms for the measurement of a firm's interventions on platforms, such as advertising, has received very limited attention. The typical approach followed has been to randomize users independently into many experiments, and to assess effects from each experiment separately ignoring the fact that multiple experimentation is occurring (). Implicitly, this approach assumes that cross-experimental interaction effects do not occur, or, if they do, that any such interactions are of limited quantitative consequence for causal measurement. Consequently, in the context of digital ad-experiments, the impact of advertising competition for such experimentation has not been a point of focus of the literature. 

However, ignoring interactions this way is at odds with how we think advertising works in marketplaces, such as competitive platforms. Fundamentally, the effect of a firm's advertising depends on its competitors' actions. The possibility of such competitive interactions has long been recognized in the marketing literature. Examples of such work span over five decades and include <cit.>, <cit.> and <cit.>, and, more recently, <cit.>, <cit.>, <cit.>, and <cit.> to quote a non-exhaustive list. At any point in time, the effectiveness of a firm's advertising may increase or decrease depending on which of its competitors also advertise. For instance, viewing 's ad may decrease the likelihood of consumers thinking of  and shopping at . In such a situation, an ad that reminds the consumer of  may become more effective when  advertises. On the other hand, if 's ad gives an unmatched discount, its presence may prevent 's ad from being effective. Consequently, estimates from a firm's experiment measuring its ad effectiveness depend on which competitors advertised during the experiment. By extension of this rationale, these findings also depend on which competitors conducted their own ad effectiveness experiments simultaneously. 

Given the limited treatment of advertising competition in ad-experiments, some very fundamental questions along these lines remain not well understood, including:


  * What are the nature of competitive interactions that could occur on platforms where competing firms experiment in parallel so that many users are simultaneously in multiple experiments? What are the quantitative significance of such interactions in modern, real-world marketplaces?

  * What are the consequences of ignoring such parallel experimentation for causal inference on the effect of interventions? What causal estimands are measured by the simple difference in means estimator, which ignores parallel experimentation and estimates the Average Treatment Effect (henceforth ATE) of a firm's advertising as the difference in means for its test and control groups? Is this estimate interpretable and policy relevant?

  * What internally consistent causal estimands can be defined in environments where competitive interactions are salient? Which of these estimands can experiments recover? What experimental design and estimators make sense to recover these estimands consistently and precisely? How can they be implemented in practice on modern platforms? 

  * What is the transportability of these estimands to the post-experimentation phase where experiments end, and competing firms have to make decisions about their activities based on what they learned from their experiment? 

  * How are firms' decisions affected by recognition (or lack thereof) of these issues, and how can they be practically leveraged for real-world decision-making?


The goal of this paper is to address these open questions directly. We do so with a specific focus on parallel experimentation by competing firms for the measurement of the effects of advertising on digital publishers. A punchline from the paper is that when the competition is for scarce resources such as, for instance, advertising slots on digital publishers, parallel experimentation can markedly impact the measurement of the intervention of interest; in this case, the measurement of the effects of advertising. Therefore, understanding how it affects the effects we want to measure; defining the required causal estimands precisely; developing effective experimental designs and estimators that can measure these estimands consistently and precisely in real marketplaces; and understanding how they can be used to facilitate practical decision-making following the experiments all become key. This paper outlines solutions for these issues; implements them on a large advertising platform; and illustrates the issues empirically using data from large-scale experiments we implemented on that platform.



  
Contributions

The paper has seven main contributions, which we discuss in sequence.

First, we demonstrate that parallel experimentation can act as a double-edged sword. On the one hand, accounting for parallel experimentation enables the estimation of a rich set of effects of advertising as a function of the presence of a focal firm's competitors, which can be of assistance for future decision-making, such as whether this focal firm should advertise in the first place. Since the advertising platform is the only party that can account for competitors' actions during an experiment, this highlights their important role as providers of relevant information to firms. On the other hand, ignoring parallel experimentation when estimating the effects of advertising as is typically done can lead to estimates of objects that have very limited interpretability and that are of little use for decision-making. We show this formally, and present a new set of causal estimands that have precise interpretability in the presence of competitive interactions.   

Second, we outline a formal framework that explicitly derives a focal firm's advertising effect and its dependence on its competitors' advertising policies. Under this framework, we can show the sources of competitive interference effects, which we refer to as allocation change and externalities. Allocation change refers to the fact that the presence of a competitor can potentially change whether the focal firm's ad is shown due to a change in competition for the ad slot. A change in allocation naturally affects the ATE estimate since an ad affects users only when it is displayed. Externalities are differences in the focal firm's payoff from advertising caused by the presence of a competitor's ads. In our empirical application we investigate the presence of allocation change and externalities.[The more competitive a platform is for advertising, the more likely these effects may manifest. For instance, in a more competitive platform, a competing experimenting campaign may be more likely to affect an  experimenting campaign's ability to show its ad, driving allocation change. In a more competitive platform, it may also be more likely that a competing experimenting campaign's ad is shown to users in an experimenting campaign's control group, driving externalities.]

Third, we outline an experimental design to investigate competitive interference. This is a full factorial design, where each factor is an indicator for whether a firm is advertising. We prove that an experiment that follows this design yields data from which a firm's heterogeneous advertising effects with respect to its competitors' presence can be recovered. This shows that the causal estimands we propose are identified. We also show these estimates are transportable to the post-experiment phase where experimentation ends and firms need to make decisions based on what they learned during the experimentation phase. When experimentation ends, the nature of cross-firm interactions are altered, and we show that typical  estimates that do not explicitly incorporate the impact of the altered interactions are not transportable.

Fourth, we present two estimators that can accomplish this task. The first is a difference in means estimator implemented separately for each factor combination, which can be obtained from a simple linear regression. However, this estimator does not scale well with the number of factor combinations. In our empirical application, we consider an experiment with 16 firms, which yields 2^16=65,536 such combinations, implying that for each firm, we would need to assess how each campaign's effect varies over 2^15=32,768 possible scenarios depending on each of its competitors' presence. This motivates our second estimator, which relies on recent advances in nonparametric econometrics to pool observations from different combinations and borrow information across them instead of treating each combination in isolation. We show that this estimator is effective even in high dimensions, and that the bandwidths associated with it have the attractive advantage of being interpretable: they identify the competitors that most strongly drive competitive interference. To our knowledge, this is the first application of this nonparametric estimator in the context of a full factorial design and for pooling across competitive states, which may be of independent interest.

Fifth, we demonstrate how a focal firm's causal estimands of advertising effects with respect to its competitors' presence can be leveraged for this  firm's decision-making. We propose a simple decision-theoretic problem where a firm's only decision is whether to advertise or not and where it only has knowledge of its own advertising effects, but not of its competitors' and whether they will be present. The solution to this problem can be obtained directly from the causal estimands we can recover. However, one challenge in implementing this solution is that the number of such estimands can be high (e.g., in our empirical exercise there are more than 30,000 such estimands for each firm). This indicates that a firm probably has to be highly sophisticated to be able to implement the solution we obtain. Hence, we suggest a simpler heuristic that performs well in our setting and that only relies on a single object that can obtained by integrating over the different estimands, which can be used in more practical settings with limited computational complexity. By limiting the amount of information that needs to be reported about competitor effects, this heuristic is also favorable to platforms, which may face constraints on how much information they can disclose to firms regarding their competitors' activities.

Sixth, we incorporate these aspects into an advertising experimentation platform we helped build for , an e-commerce company that also is a large publisher of ads in China. We show how the experimental design can be engineered into an auction-driven marketplace and how the relevant causal estimands we outline can be recovered by the platform. This may be useful for researchers and firms who are interested in how to translate the econometric issues discussed here to practical solutions and infrastructure.

Finally, we present data from a large-scale field experiment we conducted on this platform involving parallel experimentation by firms seeking to measure the effect of e-commerce advertising. The field experiment allows us to assess the strength of possible competitive interference and to assess the performance of our proposed estimators and decision rules using real data from a real platform. This serves as an empirical illustration of the framework.



  
Empirical application and results

The field experiment involves 16 experimental ad campaigns that ran parallel experiments for a three-day period in September 2018 on . The experiments include approximately 22 million users.

Our analysis of the experimental data shows that advertising has a sizable effect on advertiser outcomes for several campaigns. Across the experimental campaigns, we find that competitive interactions represent 62%-67% of the main effect of advertising, demonstrating that it can substantively change the effect of a focal campaign's advertising and is quantitatively significant. Unpacking the sources of such interactions, we find evidence for the presence of both allocation change and externalities, which indicates that both channels are relevant on this marketplace. Implementing the nonparametric estimator for a focal campaign, we find that it can recover treatment effects precisely (35% of the the 2^15=32,768 competitive treatment effects are statistically significant at the 5% level). The recovered treatment effects show significant dispersion (the standard deviation is approximately 77% of the mean treatment effect), indicating that the effect of advertising can be substantively impacted by competitor presence or absence.

The analysis of dispersion in the distribution of ATEs is relevant to assess the importance of what we refer as “environmental uncertainty”: it is a reflection of a firm’s uncertainty over which of its competitors will be advertising. Because ATEs are estimated to aid firms in their decision-making, high dispersion across them indicates that consideration of environmental uncertainty is important. Typical practice is to ignore this dispersion. To assess the impact of this typical practice for decision-making, we compute the usual estimator that measures ad effects by taking the difference in means between test and control users for a campaign independently. We assess how much of the environmental uncertainty is captured by the statistical
uncertainty around this estimate of the unconditional ATE as presented by the 95% confidence interval around it. We find that the range of ATEs covered by this confidence interval is small relative to the full range of possible ATEs, covering only around 7% of the full range. Overall, this suggests that even if decision-making accounts for statistical uncertainty around the unconditional ATE, it can still perform poorly by ignoring a substantial portion of the environmental uncertainty. 

To investigate the impact for decision-making, we consider a simple scenario in which a firm only makes the decision of whether to advertise on the platform, for which it uses the estimates of advertising effects reported to it by the platform from the parallel experiment. We consider three simple policies that the firm can use to guide this decision that use more limited information than the entire distribution of ATEs. The first policy (“default-ATE policy”) consists of the firm advertising if the unconditional ATE obtained from the typical difference in means estimator is above the advertising cost. The second policy (“integrated-ATE policy”) is one in which the platform estimates all the competitive treatment effects, but only reports to each firm their average. Finally, the last policy we consider is one in which the platform reports to the firm the probability that the firm's estimate of ATE from the experiment exceeds the cost of advertising, and the firm then randomly advertises according with this distribution (“randomized-ATE policy”).

The default-ATE and randomized-ATE policies are heuristics, while the integrated-ATE policy can be motivated by a Bayesian statistical decision theory problem. Empirically, we find that the integrated-ATE policy strictly dominates the default-ATE policy and further dominates the randomized-ATE policy over the vast majority of scenarios we consider. 

Overall, we have two takeaways from these results. First, relying on the naive unconditional ATE, which ignores competitive interactions, can lead to many mistakes and poor decision-making by experimenting firms. Second, the assessment of the simple policies are encouraging for practical decision-making. They indicate that the information obtained from the nonparametric estimator can be summarized in a single and easy-to-interpret quantity (i.e., the integrated-ATE), which, when computed by platforms and reported to experimenting firms, can lead to much better decision-making by firms. This may represent a practical approach for platforms to implement the theoretical framework developed here in their experimentation reporting and recommendation frameworks.

The rest of the paper proceeds as follows. We first discuss its relationship to the extant literature. The second section presents a simple example to outline our framework, the objects we are interested in recovering, the experimental design we suggest and the decision-theoretic problem we consider. Section 3 contains these features in a general setup. Section 4 presents formal identification results and the fifth section discusses transportability of the estimates obtained from the experiment to the post-experiment setting. Section 6 provides two estimators to recover our objects of interest. Section 7 explains in detail how we implemented our experiment on . Section 8 describes the data we obtained from this experiment and Section 9 presents our main results. Finally, Section 10 concludes.



 §.§ Related literature


This paper relates to several sub-literatures on experimentation, causal inference and digital advertising. We now discuss these relationships in more detail. 

Our setting of interest is one in which experiments are conducted in environments with competitive interactions. Hence, our work relates to the recent work that studies the impact of network externalities in learning from experiments and to ideas regarding the measurement of global effects of interventions from local experiments, such as <cit.>, <cit.>, <cit.>, <cit.> and <cit.>. 

This paper addresses a conceptually different problem from this literature. Broadly speaking, this literature addresses the vexing problem of extrapolating from effects learned from small-scale local experiments to global (market or platform) level effects. The effects of interventions implemented at the global level may differ from effects learned from local experiments because they reflect equilibrium best responses to the intervention. For instance, small-scale interventions studied by local experiments, such as providing coupons or labor market training to a few users or workers, may not induce market responses from other firms during the experiment; but when they are implemented at the global level, they may generate meaningful aggregate price responses or more intense labor market competition. This induces a gap between the effect measured from the experiment versus the realized impact of the intervention implemented at the global level. To address this problem, this literature presents innovative solutions to extrapolate from local effects to bridge the gap to global interventions. For ease of reference, we refer to this is the local-global-gap literature. 

Unlike this literature, the present paper is not focused on the problem of extrapolating from local effects to global effects.[The ideas presented in the local-global-gap literature may be used to extrapolate the effects outlined here to the global level, but this is outside of the scope of this paper.] Rather, the focus of the present study is to understand how the presence of simultaneous local experiments affects what can be learned from the experimentation as a whole. Further, the current study addresses the problem that what can be learned in one local experiment depends on the experimentation policy of competitors. We show that this dependence affects the transportability of the effects to the post-experimental period when experimentation ends and policies have to be implemented. Both these problems are new and not addressed in the current local-global-gap literature. This paper shares a conceptual relationship with the local-global-gap literature in that both aim to address a macro issue in experimentation, that what is measured in a local experiment may not be the right effect that is relevant for decision-making. In this paper, this arises due to the change in the policies of competitors post-experimentation when interventions are implemented, while in the local-global-gap literature this arises due to the global best response that arises when interventions are implemented post-experimentation. In this sense, we consider our paper complementary to this literature and as outlining a new interaction arising from competition that is relevant for platform experimentation.

Our paper is also related to the recent literature on experimentation with interference, such as on social networks and due to spillovers (e.g., ). Broadly speaking, the thrust of this literature is to accommodate situations where there are interactions between experimental units so the stable unit treatment value (SUTVA) assumption is violated. This literature investigates the impact of such SUTVA violations on treatment effects estimation, and develops strategies to define logical causal estimands and estimators in such situations. 
This paper is conceptually distinct from this stream. In the settings considered in this paper, interference occurs not due to interactions between units, but due to the fact that they are simultaneously in multiple experiments, and the presence or absence of firms and their experimentation affects units' behavior with respect to a focal advertising firm. Hence, interactions here happen not due to interactions between experimental units (users), but between experimenting entities (firms), so that the SUTVA condition holds. Therefore, we consider the problems considered here as conceptually different from the problems considered the by interference literature, and, consequently, they require different solutions. Finally, there is a conceptual relationship between this paper and the interference literature in the sense that both make progress on the problem of valid treatment effect estimation by defining new causal estimands that have empirical content in the presence of precisely defined interactions. In this paper, the new estimands are causal effects for a given firm as a function of the presence or absence of its competitors; in the interference literature, the new estimands are group-level causal effects for a given firm defined within a groups of interacting users. Overall, we believe our work is is novel and complementary to this literature.

This paper is also related to the literature on measuring digital advertising effects via experiments and to the empirical literature on measuring advertising effects in competition. Apart from the papers cited in the introduction, examples at this intersection include <cit.>, <cit.>, <cit.> and <cit.>. Unlike our paper, this literature does not focus on the challenges induced by parallel experimentation that we focus on. Broadly, we also contribute to the copious literature on digital ad-experimentation. We keep the review of this literature short for brevity, and point the reader to <cit.> and the references therein for a recent comprehensive overview.

From an experimental design perspective, we rely on advances that have been made in developing infrastructure for implementing overlapping experiments at scale (e.g., ). Our specific design is a full factorial design, which aims to enable the estimation of interaction effects between multiple factors.[For a textbook treatment of full factorial designs, see, for instance, <cit.>.] Full factorial designs have long been used in marketing, dating back at least to <cit.>. This paper leverages such infrastructure and design to develop new estimands with economic content in real marketplaces. In addition, to the extent that we leverage counterfactual policy logging to improve the precision of our estimates, our work is related to the recent literature that has suggested such strategies for improving statistical efficiency (e.g., ). One contribution of this paper is to show how to combine all these aspects together in a coherent practical system that can be implemented at scale on complex, auction-driven ad platforms.



Finally, the problem we study also relates to work from microeconomic theory. Since most digital ads are sold through auctions and since we look at competitive interference, this paper especially relates to the literature on auctions with externalities as in <cit.> and <cit.>. These tools have been applied in the context of online auctions specifically, for example, in <cit.>. At a higher level, the possibility of experimentation by agents in a common environment has been addressed by models of learning in games as surveyed by <cit.>. 

This paper is related to the copious literature on measuring digital advertising effects via randomized controlled trials (e.g., GOLDFARB AND TUCKER, 2011; LEWIS AND REILEY, 2014; BLAKE ET AL., 2015; SAHNI, 2015; SAHNI AND NAIR, 2016; GORDON ET AL., 2019), and to the empirical literature on measuring advertising effects in competition (e.g., ), though these papers have not addressed the issue of parallel experimentation to our knowledge. In addition, to the extent that we leverage counterfactual policy logging to improve the precision of our estimates, our work is related to the recent literature on digital advertising that has suggested such strategies for improving statistical efficiency (e.g., ).


Finally, the setting we study is related to two streams of microeconomic theory. First, the competition we study happens in an auction setting, and our results demonstrate that the effect of displaying a focal advertiser's ad depends on who this advertiser's competitors are. Therefore, our work relates to theoretical models of auctions with externalities (e.g., ), which have been applied in online advertising settings (e.g., ), though not specifically from the goal of experimentation. Second, at a higher level, the possibility of experimentation by agents in a common environment has been addressed by models of learning in games as surveyed by <cit.>. Although we do not utilize a formal theoretical model to explain the nature of the interference effects between advertisers, we hope our results contribute to these streams of the literature, whose work has mostly been confined to theoretical models, lab experiments and observational studies, by providing more credible evidence as to the relevance of externalities obtained from a large-scale field experiment.





A simple example

This section introduces a simple example to illustrate our framework. This example has three goals.  First, using a simpler notation it defines the objects we seek to estimate. In doing so, it characterizes the sources of competitive interference we account for when measuring advertising effects.

Second, it introduces a full factorial design for an experiment that provides data that enable us to estimate advertising effects while taking competitive interference into account when parallel experimentation occurs. Further, it shows the consequences of measuring advertising effects when this interference is ignored. In particular, it demonstrates that the object that would be estimated is difficult to interpret and cannot be seen as a policy relevant treatment effect.

Third, it discusses how the measured advertising effects from the proposed experiment inform firms' decision-making after the experiment takes place. We focus on one specific approach that we see as both practical and reasonable.



 §.§ Objects of interest


Consider an advertising marketplace where two firms, f and g, compete to show their ads to users and only have one opportunity to expose each user to their ad. We take the perspective of firm f, but the analysis for firm g is symmetrical. Firm f seeks to estimate the expected effect of its advertising, ATE, with the intention of using this estimate to better inform its decisions, such as whether it should advertise in the first place. 

Denote firm f's potential outcomes from user i as Y_if(f) if its ad is displayed, Y_if(g) if g's ad is displayed, and Y_if(0) if no ad is displayed. Let A_f and A_g be indicators for whether f and g are advertising, respectively, and W_i be the identity of the ad to which user i is exposed. We define p_w(A_f,A_g)≡ ( W_i=w | A_f, A_g  ). For f's advertising to be relevant, it must hold that p_f(1,A_g)>0. In turn, for g to be a relevant competitor of f it must be the case that p_g(A_f,1)>0. Naturally, p_f(0,A_g)=p_g(A_f,0)=0. This probabilistic structure arises naturally in settings of competitive advertising such as the one we consider here because firms typically compete to show their ads by participating in auctions. 

We now define firm f's ATEs. Let 𝕎 (A_f, A_g  ) be the set of ads users can be exposed to.[For example, if A_f=A_g=1, then 𝕎 (1, 1  )={f,g,0 }, while if A_f=1 and A_g=0, then 𝕎 (1, 0  )={f,0 }.] The observed outcome from user i to firm f is Y_if=∑_w ∈𝕎(A_f,A_g)1{W_i=w}× Y_if(W_i). Firm f's ATE as a function of g's advertising policy, which we denote by τ_f(A_g), is

    τ_f(A_g)   ≡𝔼 [ Y_if| 1, A_g  ] - 𝔼 [ Y_if| 0, A_g  ] 
       = ∑_w∈{f,g,0} (W_i=w | 1,A_g  )×𝔼 [Y_if(W_i)  ] 
            - ∑_w∈{g,0} (W_i=w | 0,A_g  )×𝔼 [Y_if(W_i) ] 
       = ∑_w∈{f,g,0} p_w  (1,A_g  )×𝔼 [Y_if(w)  ]  - ∑_w∈{g,0} p_w  ( 0,A_g  )×𝔼 [Y_if(w) ].


Equation (<ref>) highlights two ways in which interference can arise, each through one of the components within the sums in the right hand side of the equation. First, it can arise if g's presence affects the probability with which f's ad is shown, that is, if p_f(1,1)≠ p_f(1,0). We refer to this effect as allocation change. Second, interference can arise if the outcome for f depends on g's presence when f does not show its ad, that is, if Y_if(g)≠ Y_if(0). We refer to this effect as externalities. If neither allocation change nor externalities are present, then there is no interference, and therefore τ_f(1)=τ_f(0)=τ_f.



 §.§ Experimentation


The next step is to incorporate the possibility of experimentation by g, which can only happen if A_g=1. Let E_g be an indicator for whether g is experimenting and σ_g denote the probability with which i is in g's treatment group, in which case i is eligible to see g's ad. Hence, f's ATE from users in g's treatment group is τ_f(1), while its ATE from users in g's control group is τ_f(0). Firm f's total ATE is thus a convex combination between τ_f(1) and τ_f(0), with weights determined by σ_g. 

To see this, consider first Y_if when E_g=1 with a given σ_g. It follows that:

    𝔼 [Y_if| A_f,σ_g  ]   = (A_g=1)𝔼 [Y_if| A_f,1  ] + (A_g=0)𝔼 [Y_if| A_f,0  ] 
       = σ_g 𝔼 [Y_if| A_f,1  ] + (1-σ_g)𝔼 [Y_if| A_f,0  ].

Since the expression above holds for both values of A_f, we can define f's ATE as a function of g's experimentation policy, which we denote by ξ_f(σ_g), as

    ξ_f(σ_g)   ≡𝔼 [Y_if| 1,σ_g  ] - 𝔼 [Y_if| 0,σ_g  ] 
       =σ_g 𝔼 [Y_if| 1,1  ] + (1-σ_g)𝔼 [Y_if| 1,0  ] 
            - σ_g 𝔼 [Y_if| 0,1  ] - (1-σ_g)𝔼 [Y_if| 0,0  ] 
       = σ_g {𝔼 [Y_if| 1,1  ] - 𝔼 [Y_if| 0,1  ] }
            + (1-σ_g)	{𝔼 [Y_if| 1,0  ] - 𝔼 [Y_if| 0,0  ] }
       =σ_g×τ_f(1) + (1-σ_g)×τ_f(0),

where the second equality uses equation (<ref>) and the last equality uses equation (<ref>).

Notice that ξ_f(·) is a generalized version of τ_f(·). When σ_g=1, firm g allocates all users to its treatment group, which corresponds a situation in which A_g=1 and E_g=0, that is, where g is advertising but not experimenting, so that ξ_f(1)=τ_f(1). On the other hand, if σ_g=0, then firm g allocates all users to its control group, which corresponds to a situation where g is not advertising, so that A_g=E_g=0 and consequently ξ_f(0)=τ_f(0). Finally, when σ_g∈(0,1), A_g=E_g=1 and therefore g is advertising and experimenting, so that f's total ATE becomes a function of σ_g. This environment is summarized in Figure <ref>.






 §.§ Measuring advertising effects
 

Now assume that firm f runs an experiment to obtain data to estimate its ATEs and assume that firm g experiments in parallel. Let D_if and D_ig be indicators for whether i is in f's and g's treatment groups, respectively. We assume that f and g allocate users to their treatment groups independently from each other. This experiment follows a 2^2 factorial design, which is shown in Figure <ref>.


 
Identification of f's ATEs based on data collected from such experiment is straightforward since randomization implies that 𝔼 [Y_if| D_if, D_ig ] = 𝔼 [Y_if| A_f, A_g ]. By focusing on the average outcome of users who belong to f's and g's treatment groups, firm f can estimate 𝔼 [ Y_if|  1,1  ], as illustrated in the northwest quadrant of Figure <ref>. The same logic applies to 𝔼 [ Y_if|  1,0  ], 𝔼 [ Y_if|  0,1  ] and 𝔼 [ Y_if|  0,0  ] using the appropriate groups of users. With these four estimates, firm f can trivially recover τ_f(1) and τ_f(0). 

Notice that this approach presupposes that g's treatment assignment is taken into account, which is often not the case. Assume that f ignores that g is also experimenting. When focusing on its own treatment group, firm f considers users who were eligible to see g's ads with probability σ_g and ineligible with probability 1-σ_g. Hence, the average outcome from these users corresponds to a convex combination of 𝔼 [ Y_if|  1,1  ] and 𝔼 [ Y_if|  1,0  ] as illustrated in the upper rectangle of Figure <ref> and which is shown in equation (<ref>). An analogous result is obtained for users in f's control group.



The usual estimator in such circumstances is the difference in mean outcome of users in f's treatment group and in f's control group. The object this estimator recovers is ξ_f(σ_g), which is shown in equation (<ref>). This object has no intrinsic meaning or interpretation in terms of advertising policies and would only be relevant in case g did not change its conduct, that is, if g kept running the same experiment after f's experiment ended. However, our premise is that the experiments are meant to end, so that g will stop experimenting. This demonstrates a potentially negative consequence of parallel experimentation: if it is conducted but ignored in estimation, the end result of a firm's experiment is an estimated object that is of little use to this firm.



 §.§ Decision-making


We now consider how the objects recovered from the experiment can be used by firm f to make decisions in the post-experimental period. While there are many possible ways f can act upon this information, we consider only one such way, which we believe is both practical and reasonable. More precisely, we consider averaging f's ATEs using its beliefs as weights to obtain a weighted ATE to be used for decision-making.

After the experiment ends, we assume that the platform discloses to f its ATEs, τ_f(1) and τ_f(0), but nothing more. Notice that this approach presupposes that the platform can commit to report the estimates it obtains truthfully. We further assume that firm f has a belief, denoted by π_f(·), which corresponds to its subjective probability that A_g=1 after the experiment ends. We do not take a stand on how these beliefs are formed. Conditional on the estimates provided by the platform and π_f(·), firm f's projected ATE after the experiment ends is given by 

    ξ_f  [ π_f  (· )  ]= ∑_A_g ∈{0,1}π_f (A_g)×τ_f (A_g),
 
which can be computed for any π_f(·) if τ_f(1) and τ_f(0) are known. Assuming that firm f maximizes its expected payoffs from advertising, it then follows that A_f=1 if and only if ξ_f  [ π_f  (· )  ] ≥κ, where κ is the cost of advertising, which we assume f knows.

Firm f can only follow this course of action if it acquires knowledge of how its ATEs vary with its competitor's advertising policy. This does not require f to know g's ATEs, which, in practice, the platform would not disclose to f to preserve g's privacy. Notice that the relevant information can only be acquired through an experiment using a full factorial design, which can only be performed by the platform and therefore demonstrates the critical role it can play in facilitating decision-making by firms. 

However, in most settings firms face more than just one competitor. Even if the platform conducts the appropriate experiment and recovers all ATEs, many firms may not be able to act intelligently upon receiving this information. For example, in our application each firm has more than 30,000 ATEs and it can be expected that only highly sophisticated firms would be able to use all this information. Alternatively, the platform can disclose a summary of the ATEs that is easier for firms to interpret and utilize. Based on the policy relevant treatment effect given in (<ref>), a sensible disclosure policy by the platform is to provide firm f with ξ_f[π_f(·) ], which f can then use to make its advertising decision. The beliefs π_f(·) can be given by f or, as probabilities, be assessed by the platform.





General framework

This section generalizes the framework introduced in Section <ref>. Even though the notation becomes more involved, this achieves two goals. First, it demonstrates how more than two advertisers, more than one impression opportunity per user and more than one target audience can be incorporated to generate objects analogous to those defined in equation (<ref>), which remain the objects we aim to recover.

Second, it allows us to formally study and establish identification results for such objects as well as other quantities that either constitute them or are generated by them. These results, which are given in Section <ref>, help us clearly establish what the experimental design we propose enables us to estimate. 




 §.§ Setup


The advertising marketplace consists of a set 𝕋≡{1,...,T} of mutually exclusive and collectively exhaustive target audiences, which we index by t. We take these target audiences as provided by the platform. The decision of which audience to target in a campaign is determined outside of the model and is made by the advertiser. We assume that target audiences are defined so that within a target audience t there is no systematic selection or targeting. Over the relevant period of time, user i can be exposed to S ads, each of which is indexed by s.[The setup can be easily extended to i- or t-specific number of impression opportunities in the sequence at the expense of a relatively less cumbersome notation. We maintain that all users share the same number of impression opportunities for simplicity.] Finally, the set of firms that compete to show their ads to users is given by 𝔽≡{1,...,F }. For the sake of convenience, we define 𝔽_0 ≡𝔽∪{0}. Firm 0 is not an actual firm, but rather corresponds to the option of showing no ads.

Let A_ft be an indicator for whether firm f is advertising to target audience t. Notice that A_0t=1 for all t. Further, let A_t be a set collecting all such indicators, and A_-f,t be a set collecting all indicators but firm f's. It will also be convenient to define the set A_t^1 ≡{A_ft : f∈𝔽_0 and  A_ft=1}, which is the set of all firms that are advertising for target audience t.

For each ad opportunity s to user i in target audience t, firms are ranked according to a mechanism established by the platform and the one at the top of the ranking displays its ad. We define the set containing the ranked firms by R_ist. We index each position in the ranking by r, so that R_ist≡{R_ist^(1) , ...,R_ist^(r) ,...,R_ist^( |A_t^1  |)}, where R_ist^(r) is the identity of the firm that ranked r^th. Only the firms that are advertising for target audience t are ranked. The rankings for online advertising are typically determined by bids submitted to an auction, but this is not required. All we need is for the firms to be ranked somehow.

We define W_ist as the identity of the s^th ad shown to user i in target audience t, so that W_ist∈𝔽_0. Hence,

    W_ist=R_ist^(1),

where R_ist^(1) is the firm ranked at the top to display its ad. We further define the probabilities p_wst (A_ft,A_-f,t ) ≡ ( W_ist = w | A_ft,A_-f,t ). We state the following.

Ad display possibilities  p_fst(0,A_-f,t)=0 for all s, A_-f,t, f ∈𝔽_0 and t∈𝕋.  

p_fst(1,A_-f,t)≥ 0 for all s, A_-f,t, f∈𝔽_0 and t∈𝕋. 


Remark <ref> states that a firm cannot display its ads if it is not advertising in the first place. In turn, the weak inequality of Remark <ref> allows firms to be irrelevant under some circumstances in the sense that they might not be able to show their ads even when they are advertising.[This could happen, for example, if ad display opportunities are sold via auctions and the firm's bids fail to meet the reserve price.] 


Let W_it≡{W_i1t, W_i2t, ..., W_iSt} be the sequence of ads shown to user i in target audience t. Also define 𝕎 (A_ft, A_-f,t ) as the set of all possible sequences of ads shown to t as a function of firms' advertising policies. We denote the potential outcome from sequence W_it∈𝕎 (A_ft, A_-f,t ) for f from i in t by Y_ift (W_it ). The observed outcome for f from i in t is:

    Y_ift   = ∑_W ∈𝕎 (A_ft, A_-f,t )1{ W_it = W }× Y_ift (W_it) .


We now proceed to define a firm's ATEs as a function of its competitors' policies. Define (W_it=W | A_ft, A_-f,t ) ≡ p_Wt ( A_ft,A_-f,t ). The expected outcome for firm f from user i in target audience t given A_t is:

    𝔼 [ Y_ift| A_ft,A_-f,t ]    = ∑_W_it∈𝕎 (A_ft, A_-f,t )  p_Wt( A_ft,A_-f,t ) ×μ_fWt (W  ),

where μ_fWt (W   )≡𝔼 [ Y_ift (W_it) | W_it= W  ]. Finally, the ATE for firm f in target audience t as a function of its competitors' advertising policies, τ_ft (A_-f,t ), is given by:

    τ_ft(A_-f,t)   =𝔼 [Y_ift| A_ft=1, A_-f,t ] - 𝔼 [Y_ift| A_ft=0, A_-f,t ] 
       = ∑_W_it∈𝕎 (1, A_-f,t )  p_Wt( 1,A_-f,t ) ×μ_fWt (W  )  
            - ∑_W_it∈𝕎 (0, A_-f,t )  p_Wt( 0,A_-f,t ) ×μ_fWt (W  ).

Notice that this defines a collection of 2^F-1 different ATEs for firm f in target audience t. 

These ATEs play a key role for firms' decision-making, as we will demonstrate below. The experimental design we propose is meant to provide data that allow for the estimation of these objects, which we conduct in our empirical application.



 §.§ Experimentation


We now incorporate experimentation into this setup. Let E_ft be an indicator for whether firm f is experimenting on target audience t. By definition, a firm can only experiment when it is advertising, so that E_ft=0 whenever A_ft=0. Further let E_t denote the set of firms experimenting on target audience t and O_t denote the set of firms not advertising for target audience t. Consequently, the set or firms advertising for but not experimenting on target audience t is given by 𝔽_0 ∖ (E_t ∪ O_t  ). 

Define D_ift as the indicator for whether user i in target audience t is allocated to firm f's treatment group. When in firm f's treatment group, i is eligible to see f's ads; otherwise, i is ineligible to see f's ads. Firm f allocates user i in target audience t to its treatment group with probability σ_ft. We note the following.

Treatment allocation probabilitites  If f∈ E_t, so that A_ft=E_ft=1, then σ_ft∈(0,1).  

If f ∈ O_t, so that A_ft=E_ft=0, then σ_ft=0 and D_ift=0 for all i∈ t.  

If f ∉ E_t  ∪ O_t, so that A_ft=1 and E_ft=0, then σ_ft=1 and D_ift=1 for all i∈ t.  

0∉ E_t  ∪ O_t for all t∈𝕋. 


These conditions are straightforward. Remark <ref> states that experimenting firms allocate users to their treatment and control groups with strictly positive probabilities. Remark <ref> states that if f is not advertising for t, no user in t is eligible to see its ads, which is equivalent to a situation where all users in t are allocated to f's control group in t. In turn, Remark <ref> considers the opposite case: if f is advertising for but not experimenting on t, all users in t are eligible to see f's ads, corresponding to a situation in which all users in t are allocated to f's treatment group in t. Finally, Remark <ref> simply states that option 0 of showing no ads is always available to all users in all target audiences. We maintain the following assumption throughout.

Independent treatment allocation Users from all target audiences are allocated to treatment or control groups independently from one another and independently across firms.


Assumption <ref> is self-explanatory. Since the experiments we consider are conducted by the platform where firms compete to display their ads, the platform can guarantee that this condition is satisfied. 

We collect all treatment allocation probabilities on target audience t from all f ∈𝔽_0 in σ_t and all such probabilities except for firm f's in σ_-f,t. We further collect all treatment indicators for user i belonging to target audience t in D_it, which we refer to as user i's full treatment assignment, and all such indicators except for firm f's in the vector D_i,-f,t, which we refer to as user i's partial treatment assignment.

Before proceeding, we need to specify more precisely the mechanism through which ads are displayed when firms experiment. We do so below.

Displaying ads when firms experiment Instead of as in equation (<ref>), it now follows that

    W_ist={min_r=1,..., | A_t^1  | R_ist^(r): D_i,R_ist^(r),t=1 },

where R_ist is determined as if there was no experimentation. 


Assumption <ref> states that user i receives the ad of the highest ranked firm that is advertising for target audience t such that i belongs to its treatment group. Importantly, the way firms are ranked is identical to how they would be ranked if there was no experimentation. As above, this ensures that the correct counterfactual ad is displayed in a way analogous to what <cit.> proposed. 

We can now proceed to generalize equation (<ref>). We take the perspective of firm f and seek to express its expected outcome from user i in target audience t as a function of its advertising policy and its competitors' experimentation policies. First, fix a partial treatment assignment D_-f,t. Given Assumptions <ref> and <ref>, we have that:

    𝔼 [ Y_ift| A_ft,D_-f,t ]    =∑_W ∈𝕎 (A_ft, D_-f,t )  p_Wt( A_ft,D_-f,t ) ×μ_fWt (W  ).


We need to account for all possible treatment and control combinations for user i across f's competitors given σ_-f,t. To this end, define 𝔻 ( σ_-f,t ) as the set of all such combinations, so that an element of this set is one possible D_-f,t.[For example, assume that f has three competitors, g, h and j. Further assume that g∈ E_t, h ∈ O_t and j ∉ E_t ∪ O_t. It then follows that D_-f,t = {D_0t,D_gt,D_ht,D_jt} and 𝔻 ( σ_-f,t ) = {{1,0,0,1 },   {1,1,0,1 }}.] We then have

    𝔼 [ Y_ift| A_ft,σ_-f,t ] = ∑_D_-f,t∈𝔻 ( σ_-f,t )∏_g ∈𝔽_0∖{f}σ_gt^D_gt (1-σ_gt)^1-D_gt𝔼 [ Y_ift| A_ft,D_-f,t ],

where we have directly leveraged Remark <ref>. The term ∏_g ∈𝔽_0∖{f}σ_gt^D_gt (1-σ_gt)^1-D_gt is a consequence of Assumption <ref>.

Finally, we can then generalize f's ATEs to be a function of firms f's competitors' experimentation policies. Denote such ATE for firm f and target audience t by ξ_ft (   σ_-f,t ). It follows that:

    ξ_ft(σ_-f,t)    = 𝔼 [ Y_ift| A_ft=1,σ_-f,t ] - 𝔼 [ Y_ift| A_ft=0, σ_-f,t ] 
       = ∑_D_-f,t∈𝔻 ( σ_-f,t )∏_g ∈𝔽_0∖{f}σ_gt^D_gt (1-σ_gt)^1-D_gtτ_ft(D_-f,t),

where τ_ft(·) is as in equation (<ref>). Once again, notice that setting all terms in σ_-f,t to either one or zero implies a unique D_-f,t, which, in turn, is equivalent to a unique A_-f,t, yielding τ_ft(A_-f,t). Thus, equation (<ref>) is a generalized version of equation (<ref>).

Furthermore, equation (<ref>) can also be used to assess different estimation approaches. First, ξ_ft(σ_-f,t) is a reduced-form object since it is a function of the parameters we seek to estimate, τ_ft(·). Second, ξ_ft(σ_-f,t) corresponds to firm f's ATE in target audience t during the experiment. Consequently, this is the object that a difference in means estimator will consistently recover. However, this object is not enough to recover τ_ft(·). If after the experiment the probabilities with which each scenario A_-f,t occurs do not correspond to the ones implied by σ_-f,t, then ξ_ft(σ_-f,t) will not be useful for firm f's decision-making. This motivates our approach of recovering τ_ft(·) instead. As we show below, these objects allow f to compute an ATE for any probability distribution over A_-f,t.



Identification

We now discuss nonparametric identification of the objects of interest enabled by the experimental design introduced above: the ATEs given in equations (<ref>) and (<ref>), and the objects that constitute them, namely the p_Wts and μ_fWts. These identification results are required so we can discuss if and under what circumstances the experimental findings can be transported to the post-experimentation environment, which we do in Section <ref>. 



 §.§ Identification of τ_fts


We begin by discussing the identification of the ATEs defined in equation (<ref>), which are our main objects of interest. Before doing so, we first state the following.

Minimum requirements to estimate ATEs  Firm f can only recover an ATE for target audience t if f ∈ E_t.  

Firm f can only assess how an ATE for target audience t varies with A_gt if g ∈ E_t. If g ∈ O_t, f cannot assess the impact of g advertising for t. If g∉ E_t∪ O_t, f cannot assess the impact of g not advertising for t. 


These statements are straightforward. Remark <ref> states that firm f can only estimate an expected effect of its advertising on target audience t if f is experimenting on this target audience. If so, there are users in t that are eligible to receive and users that are ineligible to receive f's ads, enabling f to contrast the outcomes between these two groups to recover an ATE. Otherwise, this comparison would not be possible and no ATE would be identified. By the same logic, Remark <ref> states that f can only measure how an ATE varies with firm g's advertising if g is experimenting on target audience t. In particular, f cannot assess the impact of g's advertising if g is not advertising. In turn, f cannot assess the impact of g's not advertising if g is advertising but not experimenting. Finally, notice that we explicitly write an ATE because, as can be seen in equation (<ref>), there are 2^F-1 possible ATEs for firm f on target audience t.

We now present a key result, which formalizes these statements. To do so, it leverages the experiment and assumptions given in Section <ref>.

Identification of τ_fts  Suppose Assumptions <ref> and <ref> hold. For each t, fix E_t and O_t. Further assume that D_it is known and utilized. Then for all f∈ E_t and all t, τ_ft (A_-f,t ), as defined in equation (<ref>), is identified for all A_-f,t such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t.


The proof is given in Appendix <ref>.


Theorem <ref> states that the expected effects of the advertising policy of all experimenting firms on a given target audience are identified from data gathered through the experiment outlined above if users' full treatment assignments are known and leveraged. However, these objects are only identified at points of no advertising for competing firms that are not advertising, that is, only at points such that A_gt=0 for all g∈ O_t, and at points of advertising for competing firms that are advertising but not experimenting, that is, only at points such that A_gt=1 for all g∉ E_t ∪ O_t. They are identified at both points for competing firms that are experimenting, that is, at A_gt=1 and at A_gt=0 for all g ∈ E_t. Furthermore, they are identified for all possible combinations of these three groups since users' full treatment assignments are used. Notice that if all firms are experimenting on all target audiences, then the experiment yields identification of all the F× T × 2^F-1 possible ATEs of advertising policies. 



 §.§ Identification of ξ_fts


We now discuss identification of ATEs as a function of a firm's competitors' experimentation policies as defined in equation (<ref>). The first result is a straightforward consequence of Theorem <ref>.

Identification of ξ_fts  Assume the conditions of Theorem <ref> hold. Then ξ_ft(σ_-f,t), as defined in equation (<ref>), is identified for all σ_-f,t associated with A_-f,t, that is, all σ_-f,t such that σ_gt=1 if g ∉ E_t ∪ O_t and σ_gt=0 if g ∈ O_t.


Given that Theorem <ref> ensures identification of τ_ft (A_-f,t ), it is possible to compute an associated ξ_ft(σ_-f,t) for any feasible σ_-f,t via equation (<ref>). We emphasize that this is only possible for a feasible σ_-f,t because if g ∉ E_t ∪ O_t, then it must hold that σ_gt=1, and if g ∈  O_t, then it must hold that σ_gt=0. This result can be further generalized, which we will do in Section <ref>.

We now consider identification of some ξ_fts under more restrictive conditions. In particular, we focus on the most restrictive condition possible, in which D_i,-f,t is unobserved or disregarded. This is an especially relevant situation because it corresponds to one in which a firm ignores or is unaware of parallel experimentation by its competitors. We establish the following result.

Identification of ξ_ft in the experiment  Suppose Assumptions <ref> and <ref> hold. For each t, fix E_t and O_t, but assume that D_i,-f,t is unobserved or disregarded. Then for all f∈ E_t and all t, ξ_ft (σ_-f,t ), as defined in equation (<ref>), is identified  only for the σ_-f,t that held during the experiment.


The proof is given in Appendix <ref>.


In a typical experimental design, in which only the focal firm's treatment assignment is accounted for, the sole ATE that is identified is the one that held during the experiment. In the presence of parallel experimentation, this ATE corresponds to a convex combination of the ATEs associated with each partial treatment assignment implied by the competitors' experiments, where the weight attributed to each ATE is determined by the probabilities with which users were allocated to the competitors' treatment groups, σ_-f,t. The resulting object is ξ_ft (σ_-f,t ) as defined in equation (<ref>).




 §.§ Identification of p_Wts and μ_fWts


We now discuss identification of the objects that constitute the ATEs defined in equation (<ref>). They are relevant to assess if and how the results obtained from the experiment can be transported to inform firms' decision-making after the experiment ends. We start by making the following assumption, followed by our first identification result.

Sequences of displayed ads are observed For all i∈ t and t∈𝕋, W_it is observed and utilized.


Identification of p_Wts  Suppose Assumptions <ref>, <ref> and <ref> hold. For each t, fix E_t and O_t. Further assume that D_it is known and utilized. Then for all t∈𝕋, p_Wt (A_ft,A_-f,t ) is identified for all W∈𝕎 ( A_ft, A_-,ft ) and all A_t such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t.



The proof is given in Appendix <ref>.


The proof of Theorem <ref> is straightforward: Assumptions <ref> and <ref> guarantee that the experiment generates all the data required to estimate the p_Wts, while Assumption <ref> ensures that the display probabilities observed in the experiment are equivalent to those from an environment without experimentation. We now establish our second result.

Identification of μ_fWts  Suppose Assumptions <ref>, <ref> and <ref> hold. For each t, fix E_t and O_t. Further assume that D_it is known and utilized. Then for all f ∈𝔽_0 and all t∈𝕋, μ_fWt (W  ) is identified for all W∈𝕎 ( A_ft, A_-,ft ) such that p_Wt (A_ft,A_-f,t ) > 0 and all A_t such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t.



The proof is given in Appendix <ref>.


The only additional condition required relative to Theorem <ref> is that p_Wt (A_ft,A_-f,t ) > 0. This is needed to ensure that observations exposed to sequence W are obtained in the data. Finally, notice that unlike Theorems <ref> and <ref>, Theorems <ref> and <ref> rely on Assumption <ref>. However, the objects identified by these theorems are not restricted to experimenting firms.



 §.§ Comments


The results presented above warrant further comments. Theorem <ref> is key because it demonstrates that the objects we want to recover, that is, the ATEs given in equation (<ref>), are identified.

The purpose of Theorem <ref> is to characterize the object the firm would recover if it proceeded as is typically done. The typical approach for an experimenter is to compare the average outcome for units in its treatment group with those in its control group, yielding the usual difference in means estimator. Under parallel experimentation, this estimator, which ignores that units belong to several treatment and control groups, recovers the object characterized in Theorem <ref> and given in equation (<ref>): a convex combination of the original ATEs as given in equation (<ref>). This object cannot be interpreted as a policy relevant treatment effect except for idiosyncratic situations which in all likelihood will not happen in practice.

Theorems <ref> and <ref> address identification of the objects that constitute the ATEs. We will use these objects below to describe the stability of the ATEs. 



Transportability

The ultimate goal of a firm is not only to estimate ATEs using data from an experiment, but to use these estimates to inform future policy in the post-experimental period. We now discuss under what circumstances the estimates obtained from the experiment described above can be transported to this new environment.

We consider a situation in which each firm only learns its own ATEs, but not its competitors', and then decides whether to continue to advertise or not. While there are many possible ways firms and the platform can act after the results are obtained and the experiment ends, we focus on this scenario due to its practical simplicity. First, it only requires the platform to have enough commitment to truthfully report to each firm its own ATEs, and not their rivals' ATEs, avoiding concerns about privacy and competitive leakage. Second, as will be outlined below, it only requires firms to have beliefs regarding their competitors' future policies, and these beliefs are left unconstrained.

We consider firms' unilateral responses under three different situations corresponding to the stability, or lack thereof, of the objects that constitute the ATEs as in equation (<ref>), namely the p_Wts and μ_fWts. As it will be made clear below, the firms' ability to transport the estimates they obtain from the experiment depends on whether these objects remain stable, affecting their decision-making in the post-experimental period. 



 §.§ When p_Wts and μ_fWts are fixed


We first consider the case where p_Wts and μ_fWts do not change after the experiment. The stability of p_Wts means that the display probability of every sequence W does not change. In turn, the stability of μ_fWts implies that users' behavioral responses to ads remain the same. By equations (<ref>) and (<ref>), the stability of p_Wts and μ_fWts implies that 𝔼 [Y_ift (A_ft, A_-f,t )  ] and τ_ft (A_-f,t ) do not change as well. 

Given knowledge of τ_ft (A_-f,t), firm f has to decide whether to advertise after the experiment ends. Because these objects depend on actions taken by firm f's competitors, f needs to have beliefs over these actions. Hence, the question is whether firm f can compute a relevant ATE as a function of its beliefs regarding its competitors' advertising policies. Under the conditions of Theorem <ref>, the answer is affirmative.

To see this more clearly, let π_ft ( A_-f,t ) be a probability distribution over A_-f,t that reflects firm f's beliefs regarding its competitors' advertising policies for target audience t. We impose no constraints on π_ft ( · ). Let 𝔸 be a set containing all possible combinations of f's competitors' advertising decisions. We can then define a more general version of the expected effect of f's advertising policy given in equation (<ref>) as a function of π_ft ( · ): 

    ξ_ft [ π_ft ( · )  ]     = ∑_A_-f,t∈𝔸π_ft ( A_-f,t ) ×τ_ft(A_-f,t).

The following result is immediate.

Identification of possible ξ_ft [ π_ft ( · )  ]s  Assume the conditions of Theorem <ref> hold and fix E_t and O_t for each t. Then ξ_ft [π_ft ( · )  ], as defined in equation (<ref>), is identified for all π_ft(· ) such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t.


Theorem <ref> implies that τ_ft(A_-f,t) is identified for all A_-ft such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t. Consequently, by equation (<ref>) it is then possible to calculate ξ_ft [ π_ft ( A_-f,t )  ] for every feasible π_ft(·). Notice that equation (<ref>) is a more general version of equation (<ref>). The difference is that beliefs are not constrained to the multiplicative form in equation (<ref>), which is a consequence of Assumption <ref>. Given that f is able to compute ξ_ft [ π_ft ( · )  ], it is then able to make a decision as to whether it should advertise after the experiment ends. Assuming f maximizes its expected payoff, the answer is affirmative if and only if ξ_ft [ π_ft ( · )  ]≥κ, where κ is the cost of advertising. Finally, notice that, as above, f's forecasts are constrained by the set of experimenting firms, E_t. To be precise, the only way all possible future advertising combinations can be assessed is if E_t=𝔽.



 §.§ When p_Wts are not fixed but μ_fWts are


We now consider the case in which p_Wts can change after the experiment ends while μ_fWts remain fixed. In other words, we now assume that the probability that each sequence of ads shown changes after the experiment, but the average effect of displaying a sequence to consumers does not.

To see why this could happen, remember that online ads are often sold via auctions. As discussed, for example, in <cit.>, a firm's optimal bidding strategy depends on the expected effect of showing its ad relative to a competitor's ad being displayed. Consequently, if firms acquire knowledge of μ_fWt (W  ), then it is possible that they will change their bids, which, in turn, will likely alter the probabilities p_Wt (A_ft,A_-f,t ).

We now assume that the platform truthfully reports μ_fWt ( W  ) to firm f. In this case, we can specify an even more general version of equation (<ref>). Now a firm's beliefs would not only be about which of its competitors would be advertising, but also over, conditional on that, which sequences of ads would be displayed. 

To be precise, let ζ_ft ( W| A_ft, A_-f,t ) be a probability distribution reflecting firm f's belief that ad sequence W is shown to a user in target audience t as a function of all firms' advertising policies, where W ∈𝕎 (A_ft, A_-f,t ). Once again, we impose no other constraints on these beliefs. We now define:

    ξ_ft [π_ft ( · ),ζ_ft ( · )  ]     = ∑_A_-f,t∈𝔸π_ft ( A_-f,t )  {∑_ W ∈𝕎 (1, A_-f,t ) ζ_ft (W | 1, A_-f,t ) μ_fWt (W  )  .  
        .                    -  ∑_ W ∈𝕎 (0, A_-f,t ) ζ_ft (W | 0, A_-f,t )μ_fWt (W   )  }.

The following result is immediate.

Identification of possible ξ_ft [ π_ft ( · ), ζ_ft ( · )  ]s  Assume the conditions of Theorems <ref> and <ref> hold, and fix E_t and O_t for each t. It then follows that ξ_ft [π_ft ( · ), ζ_ft ( · )  ], as defined in equation (<ref>), is identified for all π_ft(· ) such that A_gt=1 if g ∉ E_t ∪ O_t and A_gt=0 if g ∈ O_t, and all ζ_ft ( · ) such that if p_Wt(·)=0, then ζ_ft(W|·)=0.


Identification of the ATEs defined in equation (<ref>) follows a similar argument as the one given for Corollary <ref>. The inputs required to compute these ATEs for a given set of beliefs, π_ft(·) and ζ_ft(·), are the μ_fWts, whose conditions for identification are given in Theorem <ref>. However, notice that now the beliefs are not only restricted by E_t and O_t, but also by which sequences of ads could have been displayed with strictly positive probability, because the associated μ_fWt(W) would not be identified otherwise.



 §.§ When μ_fWts are not fixed


Finally, consider the case where μ_fWts change, which is arguably the most difficult case  to tackle since a change in μ_fWts means that the expected effects of displaying ads themselves change. This characterizes an inherent change in users' behavioral responses after being exposed to ads and therefore implies a significant change in the environment. 

Regardless of whether p_Wts are fixed, this situation would require the platform or the firms to estimate what the new μ_fWts would be. While they could have some knowledge or beliefs about the nature of this change (for example, μ_fWts could follow some sort of ARMA process) against which they could create expectations, typically a change of this nature would necessitate a new experiment to recover the new parameters. 



 §.§ Comments


The objects we are interested in measuring are the ATEs as given in equation (<ref>). However, this equation also demonstrates that these objects are functions of two others, the p_Wts and the μ_fWts. This section addressed if, how, and under what circumstances the estimated ATEs can be used successfully for decision-making after the experiment ends.

While it is indeed possible that the ATEs from equation (<ref>) can change after the experiment, we believe that, at least in the short-term, they are the policy relevant objects for firms for two reasons, one associated with each of the objects that constitute them. First, in the setting we consider for the p_Wts to change it would be necessary for firms to change their bidding policies when participating in advertising auctions. Because these algorithms are automated, it is unlikely that bidding policies will change quickly after the experiment ends, even in response to the findings provided by the experiment.

Second, for the μ_fWts to change it would be required that users' behavioral responses to the ads changed. Conditional on ad campaigns not being altered, this would necessitate an intrinsic change in user behavior, which can be difficult to justify in the short-term. 

For these reasons, we focus on the ATEs from equation (<ref>) as our objects of interest. The estimation strategy outlined below, as well as the framework given above, aim at recovering these objects.
 


Estimation

We now discuss estimators to recover the ATEs defined above. More specifically, our goal is to estimate the parameters τ_ft ( A_-f,t ) as defined in equation (<ref>), which are heterogeneous average treatment effects with respect to firm f's competitors' advertising policies. Before presenting these estimators, we state the following regarding the data requirements for estimation to be feasible. 

Data requirements for estimation  Fix t and σ_t. Let 𝔻 ( σ_t  ) be the set of all possible full treatment assignments, D_t, given σ_t. For estimation of all possible ATEs to be feasible, the sample must contain observations of all D_t ∈𝔻 ( σ_t  ).


This is a straightforward requirement. For the ATE associated with a given treatment assignment to be estimable, we must have observations that received this specific treatment assignment. Consequently, for all possible ATEs to be estimable we require observations with all possible treatment assignments.

We assume we have data that come from the experimental design described above. In particular, we assume that for firm f the observed data are {Y_if, D_if, D_i,-f,T_i }_i=1^n, where T_i∈𝕋 indicates to which target audience user i belonged and n corresponds to the total number of observations.



 §.§ Linear regression


The simplest estimator for τ_ft ( A_-f,t )  is given by the usual difference in means. Fix T_i=t and let D_i,-f=d be the partial treatment assignment that is equivalent to A_-f,t. Define

    τ̂_f ( d, t  )    = ∑_i=1^n 1{D_i,-f=d,T_i=t }D_ifY_if/∑_i=1^n 1{D_i,-f=d,T_i=t }D_if
               - ∑_i=1^n 1{D_i,-f=d,T_i=t }(1-D_if)Y_if/∑_i=1^n 1{D_i,-f=d,T_i=t }(1-D_if).

Under standard conditions τ̂_f ( d, t  ) is consistent for τ_ft (A_-f,t ) and asymptotically normal at the usual √(n) rate. 

The estimator in equation (<ref>) corresponds to the OLS estimator of the slope coefficient of a linear regression of Y_if on D_if using only observations such that D_i,-f=d and T_i=t. The corresponding estimators for every possible d and t can be obtained from the following regression

    Y_if   = ∑_t ∈𝕋1{ T_i = t } ( ∑_d ∈𝔻 (σ_-f,t )1{ D_i,-f = d } [  α_f (d, t  ) + τ_f (d, t  )D_if+ϵ_if(d,t)  ]  ),

where 𝔻 (σ_-f,t ) corresponds to the set of all possible partial treatment assignments given σ_-f,t and the ϵ_ifs are error terms.



 §.§ A kernel-based estimator


The approach described above is valid, but it potentially faces an important practical challenge: the number of observations for certain full treatment assignments might be limited and hinder the precision of the corresponding estimator. While Assumption <ref> and Remark <ref> guarantee that with a large enough population Remark <ref> will be satisfied, the sample obtained from the experiment can still be limited, especially if the number of firms is large. For example, in our application below we consider an experiment with one target audience and 16 different firms. Hence, for each firm there are 2^15=32,768 different ATEs to be recovered. For the estimator given in Section <ref> to be feasible with 100 observations per ATE, which is arguably a modest sample size, millions of observations would be required. This demonstrates why pooling observations and using an estimator that relies on smoothing is an attractive approach, which we follow below.

We use a method that pools observations with different treatment assignments and belonging to different target audiences through kernel smoothing. The intuition behind this approach is that to estimate τ_ft (A_-f,t ) it borrows information from observations with a different A_-f,t, say, A_-f,t'', while using a principled approach to determine how much information to borrow. It quantifies the discrepancy between A_-f,t and A_-f,t'' so that, if it is low, it borrows a lot of information; otherwise, little information is borrowed. In the limit, no information is used and the estimator collapses to the one presented in Section <ref>. These discrepancies, and therefore the amount of information that is borrowed, are captured by the bandwidth that controls the pooling of observations, which we now discuss in detail.

We use the estimator introduced by <cit.>, who use kernel functions designed to smooth over categorical variables. To our knowledge, this is the first study that implements this estimator in the context of a factorial design and for measuring advertising treatment effects.

We now give more details about this estimator. Consider the regression equation for a given D_i,-f and T_i:

    Y_if   = α_f  (D_i,-f, T_i  ) + τ_f  (D_i,-f, T_i  ) D_if+ϵ_if (D_i,-f, T_i  ) 
       = X_if'θ_f  (Z_if ) +ϵ_if (Z_if ),

where X_if=  [1, D_if ]', Z_if=  [D_i,-f', T_i  ]' and θ_f(·)=  [α_f(·), τ_f(·)  ]'.

Let Z_ifv be the v^th coordinate of the vector Z_if. The kernel function is:

    l  ( Z_ifv, z_v, λ_v  ) =  1,  if  Z_ifv=z_v 
    λ_v,  otherwise,

where z_v is one of the values Z_ifv can take and λ_v is the bandwidth associated with Z_ifv. The overall kernel is then given by:

    L  ( Z_if, z, λ ) = ∏_v l  ( Z_ifv, z_v, λ_v  ) = ∏_v λ_v^1{ Z_ifv≠ z_v },

where z is a vector collecting all the z_vs and λ is a vector containing all bandwidths. The estimator is:

    θ̂_f (d,t  ) = θ̂_f (z  ) =  ( 1/n∑_i=1^n X_if X_if ' L  ( Z_if, z, λ )   )^-1 ( 1/n∑_i=1^n X_if Y_if  L  ( Z_if, z, λ )   ),

where we used z =  [d',t  ]'.

<cit.> suggest computing λ via leave-one-out cross-validation. To be precise, λ is chosen to minimize

    CV(λ)=1/n∑_i=1^n  [Y_if-X_if'θ̂_-i,f (Z_if,λ )  ]^2,

where θ̂_-i,f (Z_if,λ ) is computed as in equation (<ref>) but without using observation i, that is,

    θ̂_-i,f (Z_if,λ ) =  ( 1/n∑_k≠ i X_kf X_kf ' L  ( Z_kf, Z_if, λ )   )^-1 ( 1/n∑_k ≠ i X_kf Y_kf  L  ( Z_kf, Z_if, λ )   ).

<cit.> further showed that the resulting estimator is consistent and asymptotically normal. More notably, the rate of convergence of this estimator is √(n), which is faster than typical kernel-based and machine learning estimators, which is an attractive property. 


When target audiences are defined by user-level characteristics, it can be the case that the practitioner prefers to smooth over these characteristics instead of over the target audiences themselves. In this setup, this can be achieved by following the approach proposed by <cit.>, which allows for smoothing over categorial and continuous variables. However, the rate of convergence of this estimator is affected by the bandwidths associated with the continuous variables, making it slower.



 §.§ Alternative machine learning approaches


While the kernel-based estimator above has several attractive properties, it also has shortcomings. As noted in page 555 of <cit.>, situations in which the number of variables that are smoothed increases with the sample size are not considered. This is especially relevant regarding target audiences: if they are defined based on observable characteristics of users and if they are numerous or can take many values, this estimator may not perform well. 

Thus, machine learning methods can be an attractive alternative given their ability to better handle high dimensional settings. Because of equation (<ref>), methods that rely on linear parametric specifications such as LASSO, ridge regression and elastic net lend themselves naturally to tackle this estimation problem.




Experiment

This section describes our implementation of the experiment presented in Section <ref>. We implemented the factorial design described in Section <ref> on 's Conversion Lift System <cit.>. Conversion Lift System is an on-demand product we designed that allows advertisers to run experiments to assess the performance of their campaigns on 's ad inventory. There are many particularities and details that need to be handled in translating the statistical framework we outlined to a practical functioning system. The ranking of ads given in equation (<ref>) has to accommodate the auction-driven allocation system used to serve ads. Furthermore, scalability requires reducing latency in serving ads while experimenting and improving statistical precision in the analysis of the data from the experiment. These particularities are common in large-scale ad auctions environments, so that these implementation details may be of more general interest in and of themselves. We describe each of them next.



 §.§ Auction-driven ads marketplace


Like most digital ad platforms, a large amount of ad inventory on  is allocation via real-time bidding (RTB) auctions. The process is as follows. Advertisers first set up campaigns on . Henceforth, we will refer to a specific slot  on the publisher's digital real estate where ads can be shown as “ad position.” Each campaign specifies a target audience, a set of ad positions where users belonging to these target audiences can be exposed to ads, a creative or set of creatives (i.e., images) associated with the campaign to be shown when an ad is served, and a set of rules specifying the bidding policy for the campaign.

When a user arrives at an ad position on , the system retrieves in real time a queue containing the list of advertisers that are eligible to show their ads to this user. The list is then sorted on the basis of a proprietary ad quality score that comprises several variables, including bids. These ad quality scores generate the rankings that determine which ad is served as in equation (<ref>). The use of this quality score in addition to just the bids reflects the desire of the publisher to ensure an experience that reduces user annoyance and to generate long-term value to all players in the ecosystem. Programmatic ads on most digital platforms are sold this way (e.g., ).



 §.§ Front-focused mobile ads


Within the system, the specific experiments reported in this paper pertain to “front-focused” ad positions on the  app or mobile page. These ads are served at the top of the app or mobile home page. Panel (a) of Figure <ref> shows a screenshot of the app. There are 8 front-focused ad position and each position can show a different ad in the upper rectangle. A user can only see one front-focused ad at a time. 



When a user arrives at 's app or mobile home page, they are served ad number one. After a few seconds, the banner automatically rotates to the next ad. The user can also manually rotate the ads by swiping. Once the user clicks on an ad, the app directs them to a “landing page,” which shows more information including larger images of the featured products, promotional coupons, and a collection of relevant products. Panel (b) of Figure <ref> shows an example. If a user clicks on a product on the landing page, they arrive at a product detail page featuring detailed information about the product including price, coupons and reviews. They can then add the product to their shopping cart and purchase it.



 §.§ Details behind ad serving mechanism


The front-focused ad positions are valued by advertisers because of their prominence and their large number of exposures, and are considered premium ad inventory. They are also key to drive visits to product detail pages, which, in turn, are key for conversions. Front-focused ad auctions feature specific characteristics that are relevant for engineering the experiment, which we discuss now.

First, when setting up campaigns for such inventory, advertisers are required to specify a base level bid that applies to all users, and then select specific target audiences to which premium bids may apply. This implies that no user is explicitly excluded; all exclusions and inclusions are induced by the bids the advertiser specifies. Advertisers also cannot specify a particular front-focused ad position to bid for. They can only specify they wish to show an ad at any of the available front-focused ad positions.

When a user arrives, advertisers targeting this user participate in an RTB auction to compete to show their ad at an available front-focused ad position. Some positions may not be available because of contractual agreements. During this process, auctions queues are generated independently for each ad position. These queues can be equal or different due to technical specifications. The queue rank is based on the aforementioned quality scores. A user experience control system further filters the ranked queues. This system tries to avoid repeating ads from the same advertiser across the various front-focused ad positions at a given impression opportunity. Additional user experience controls, such as pacing, also feed into the ad serving rules. Once the queue clears this system's filters, the top ranked ad at the corresponding ad position is served. This is the process through which the rankings R_ist from Section <ref> are generated. 

There is a distinction between the ad that is served and whether this ad is actually seen. Serving refers to the process by which ads are sent from the server to the user's app. To reduce latency, the ads for all eight positions are served to the user in one shot. A user may choose not to look or navigate away from the front-focused  space of the app before seeing the served ad at one of the ad positions. The “compliance” with the served ad is therefore a decision by the user, which affects the interpretation of the ATE as discussed below. Generally speaking, ad positions with lower index have higher chances of being seen by users since they are shown earlier. 

One implication of the complexity of this entire mechanism is that the relevant counterfactual to an ad is  not trivial to obtain. If an advertiser chooses not to show their ad to a user, the next ad in ranking queue would be served instead subject to this user experience control system filter. Since only the platform can retrieve this in real time, only the platform can run this experiment with a precisely constituted counterfactual as we described. This is a motivation for developing a product that delivers “experiments as a service.” The challenge is to engineer the experiment in the context of this complex system. 



 §.§ Engineering the experiment


Engineering this experiment requires working out a strategy for randomization, a way to adapt the ad serving mechanism to deliver the right factual and counterfactual ads for users in the experiment, and a way to log the data obtained through this experiment. We now discuss each of these components.



  §.§.§ Randomization


At the beginning of the experiment, the engineered system first retrieves all advertisers in the experiment and assigns a campaign index to each of them. It then uses the  quasi-randomization method <cit.> to assign each user i and advertiser f a common randomization seed:

    D_if=hash split(user id, campaign index, seed),

where D_if is user i's treatment assignment for advertiser f. This approach effectively stores a consistent randomization method instead of sorting the total assignment vector D_i for each user i and is helpful for reducing latency in the online system. Otherwise, it would  have to store tens of  billions of treatment assignments and impose a heavy cost to the online ad serving system. The hash method only takes user id, campaign index and a seed as independent, and, therefore, users are independently assigned to treatment or control groups across advertisers, so that Assumption <ref> holds.



  §.§.§ Serving ads


At ad serving time, the queue for each ad impression opportunity is generated as described above and passed through the auction and user experience control system. To induce the experiment into the system, we first remove all f such that D_if=0 from the auction queue. This ensures that user i is eligible to see f's ad only if D_if=1 and ineligible otherwise. The remaining ads in the ranked queue are passed through the user experience control system again and the top ad is served to the user. This way, we allocate the ad position to the top eligible ad, which is economically efficient to the platform while ensuring the user experience is not degraded while the experiment is ongoing. 

Figure <ref> shows an example to illustrate this system. We focus on one front-focused ad position and a situation with only three advertisers. Advertisers 1 and 2 are conducting a parallel experiment and advertiser 3 is not. User i is independently assigned to the treatment and control groups of 1 and 2, which determines their full treatment assignment, D_i. In an auction for this user, the possible ads are Ad 1, Ad 2 and Ad 3. Figure <ref> illustrates which ad would be served under each possible full treatment assignment. For example, looking at the first column the auction queue has Ad 1 at the top ad, followed by Ad 2 and finally Ad 3. If the user was in advertiser 1's treatment group, they would be served Ad 1 regardless of whether they were in advertiser 2's treatment or control group. If the user was in advertiser 1's control group but in 2's treatment group, they would be served Ad 2, which would be the auction-specific counterfactual ad. Finally, if the user was in 1's and 2's control groups, they would be served Ad 3.









  §.§.§ Multiple ad exposures


An experiment can last for multiple days and pertains to multiple front-focused ad positions. During the course of the experiment, users may visit 's app multiple times. Hence, the engineered system needs to handle users' participation in multiple auctions. What is critical for the design is that assignment of a user to treatment or control groups is consistent during the entire experiment, which is guaranteed by the hash split presented above. Figure <ref> shows how this works in the engineered system. User i participates in S auctions during the experiment, and in each auction they are eligible or ineligible to see ads based on their assignment status according to the system explained above. The outcome variable Y_ift is their cumulative behavior over the experiment.










  §.§.§ Counterfactual policy logging and Interpretation


The effects of digital ads may be small and may require data on hundreds of thousands of users to be estimated with precision. To improve precision, as part of the engineered system we log the auction queue before control ads are dropped, which represents the counterfactual ad queue if none of the firms were experimenting. In this counterfactual scenario, the top ad on the logged queue would be served to the user, so we refer to it as “auction-specific counterfactual ad.” For each auction, we also log which ad is actually served to the user. We call this “auction-specific factual ad.” In statistical analysis, we only consider users who factually or counterfactually are served the focal experimental ad in at least one auction. This ensures we implement this analysis on a set of users in the treatment group who are most likely to be factually served the focal ad, and on a set of equivalent users in the control group who are most likely to be counterfactually served the focal ad. By removing users with low propensity to be served the focal ad from the analysis, we improve precision.

The ATEs we measure capture the effect of being assigned to a treatment group compared to the control group. Assignment to treatment implies a user is eligible to be served ads of the focal advertiser, while assignment to the control group implies they are ineligible. Hence, the effects we measure should be interpreted as the intent-to-treat effect of seeing the focal advertiser's ads for the subpopulation of users the publisher would serve the advertiser's ad on its platform. Relating this to the setup given in Section <ref>, we focus on the subpopulation such that Remark <ref> holds with a strict inequality.




Data

We now describe the data we obtain from implementing the experiment. We start by presenting a series of randomization checks to confirm that the experiment is implemented successfully.

Our data originally corresponds to parallel experiments on front-focused ads for 27 campaigns which ran for a three-day period in September 2018.[As we noted in Section <ref>, no user is explicitly excluded from seeing front-focused ads, so that all campaigns face the same target audience; therefore, we disregard the issue of dealing with multiple target audiences in this analysis.] In this experiment, 70% of users are independently assigned to each campaign's treatment group, and the remaining 30% are assigned to each campaign's control group. Out of the 27 campaigns, we only keep the 16 campaigns that had at least 200,000 experimental users. We track user visits to the product detail page through searches, other ads, and organic recommendations as well. Advertisers value visits to the product detail page because it is an antecedent to actual conversion, and an explicit role of front-focused ads is to drive such visits. They are also a key outcome metric reported to advertisers as part of campaign performance reporting. Therefore, this is an important metric to analyze and forms our dependent variable in the analysis reported below. 



 §.§ Overlap across campaigns


Our sample comprises approximately 22 million users who are exposed to at least one of the 16 experimental ad campaigns. We say an individual is “exposed” to a campaign if they would have been served the ad if there was no experimentation. About 27% of users in our sample are exposed to more than one of the 16 campaigns. Table <ref> shows the distribution of exposure across users.

 

The percentage of a campaign's experimental users that are also exposed to other campaigns affects the potential for competitive interference between them. Table <ref> shows that 50% of users in the median campaign are also exposed to at least one other campaign. It further shows that substantial overlap can be due to a single competing campaign.

 

We will refer to the advertiser corresponding to the second column of Table <ref> as the focal campaign's “main competitor.” Part of the analysis below focuses solely on the focal campaign, which we denote by f, and its main competitor, which we denote by g.[Note that such overlap is only a crude measure of which other firm is the closest competitor to the focal firm f. For instance, a firm g may not be a relevant competitor to f for all overlapping users, and the extent to which g's presence affects f may depend on another firm, h.]



 §.§ Randomization check


Before displaying the empirical results we verify that the experiment is implemented correctly. We begin by assessing whether the fraction of users assigned to the treatment groups of each campaign corresponds to what was intended, 70%. To do so, we compute these fractions via sample averages and the p-values associated with a hypothesis test for whether the fraction is equal to 0.70. If this null hypothesis is correct, these p-values across campaigns should correspond to draws from a standard uniform distribution. We test whether their distribution corresponds to a standard uniform using a Kolmogorov-Smirnov (KS) test and display the results in Figure <ref>. The p-value associated with the KS test is 0.986, indicating that the fractions obtained in the experiment correspond to what was intended.



We now proceed to investigate possible differences in user behavior before the experiment as a function of whether they belong to the treatment versus the control group of each campaign. If randomization is implemented correctly, there should not be significant differences across groups. To verify whether this is the case, for each campaign, we run regressions of the number of adds to cart, of orders, of visits to the product detail page, and of gross merchandising value (GMV), a metric of money spent, during the three days prior to the start of the experiment on the treatment assignment indicator. The regression coefficients along with standard errors robust to heteroskedasticity are shown in Table <ref>. Out of the 64 coefficients, 54 are not statistically significant at the usual levels.

 

To account for multiple testing, we estimate a SUR model for each campaign, gather the p-values associated with a test for whether all coefficients were equal to zero, and test whether they correspond to draws from a standard uniform distribution using a KS test.[We only obtain 15 p-values because the variance matrix associated with the SUR estimates for campaign 4 was not invertible.] Results are shown in Figure <ref>. The p-value associated with the KS test is 0.173, so that we cannot reject the null hypothesis that the p-values follow a standard uniform distribution at the usual significance level.



Finally, we assess whether Assumption <ref> holds, that is, whether users are indeed allocated to treatment groups independently across campaigns. To do so, for each campaign, we run a regression of an indicator for whether the user is in its treatment group on an indicator for whether the user is in its main competitor's treatment group as determined in Table <ref>. We then collect all 16 p-values, computed based on standard errors robust to heteroskedasticity, and perform a KS test for whether they follow a standard uniform distribution, which should be the case if the treatment assignments are uncorrelated. Results are displayed in Figure <ref>. The p-value associated with this KS test is 0.428, indicating that Assumption <ref> is satisfied.







Results

This section presents the results from the experiment and data described above. We begin by displaying baseline results obtained from the typical approach in experiment studies, that is, a simple difference in means between treatment and control computed via a linear regression of outcomes on the treatment indicator. Henceforth, we will refer to this as the difference in means estimates of the unconditional ATE. These correspond to estimates of the terms ξ_ft(σ_-f,t) as given in equation (<ref>) for each of the 16 campaigns. We then assess the existence of competitive interference effects and find evidence of such effects. Following this, we leverage results from the kernel-based estimator given in Section <ref> to dissect the heterogeneity in ATEs that is induced by this interference, and outline implications for advertiser's decision-making.



 §.§ Baseline results


The difference in means estimates of the unconditional ATE for each of the 16 campaigns are shown in Table <ref>. We only use observations that, in the absence of the experiment, would have been exposed to the focal campaign, f, and to its main competitor, g. We choose this sample for the purposes of comparability to results we find when we investigate the existence of competitive interference effects.

 

Out of the 16 estimates, 7 ATEs are statistically significant at the usual levels. There is substantial variation across effects: among the significant effects, they range from 8.73% (campaign 15) to 98.95% (campaign 11), with an average of 36.40%. Finally, to account for multiple testing across the 16 campaigns, we estimate a SUR model and perform a joint test for whether all ATEs are equal to zero. We obtain a p-value lower than 0.00001, which indicates that we can reject this hypothesis. Overall, this establishes that front-focus advertising has a measurable treatment effect on visitation for several campaigns.




 §.§ Assessing the existence of competitive interference effects


We now assess the existence of competitive interference effects. For simplicity of interpretation, we begin with an analysis of how each campaign's effect varies with the presence of its main competitor as defined based on Table <ref>. Note that this is a first-cut only. To more comprehensively assess competitive interference, we would need to assess how each campaign's effect varies over all the 2^F-1=2^15=32,768 possible scenarios depending on each of its competitors' presence. As discussed in Section <ref>, several competitors may interfere with a given advertiser's campaign, so such a comprehensive analysis is needed to reveal the full picture regarding competition. 

To implement the first-cut analysis, we estimate the following regression:

    Y_if = α_f + β_f D_if + γ_f D_ig + δ_f D_ifD_ig + C_i'ψ_f + ϵ_if,

where g is firm f's main competitor as defined above. The variables contained in the vector C_i are the outcomes from three days prior to the experiment from Table <ref>. These variables are independent from treatment assignments and we include them to increase the precision of the estimator. We can detect whether competitive interference can alter the focal firm's ATE by assessing the significance of the parameter δ. Table <ref> displays the estimates for all campaigns. 

 



  
Statistical significanceOut of the 16 interaction coefficients, δ_f, six are statistically significant at one of the usual levels. To guard against multiple testing, we estimate a SUR model and perform a joint test of whether all the δ_fs are zero clustering at the user level (2,632,499 users). We obtain a p-value = 0.0192, which indicates the existence of competitive interference effects. 



  
Economic significanceTo assess the economic significance of the interference, we calculate the average of |δ_f|/|β_f| across campaigns. Across all 16 campaigns, this ratio is 0.6226, and across the campaigns that had a statistically significant β_f, the ratio is 0.6733. This demonstrates that interference equals a substantial fraction of a campaign's main effect. 

A takeaway from these results is that in this marketplace, competitive interference can substantively change the effect of a focal campaign's advertising and that changes to competition can substantively change the relevant treatment effect of advertising.



 §.§ Assessing the sources of competitive interference effects


The next step is to unpack the sources of such effects, which can help us understand the role of competition in driving the interactions we uncovered. In particular, we will look at the two effects we outlined above, namely allocation change and externalities. 

 


Figure <ref> is helpful in understanding how we unpack each effect in isolation.
 
 
 


 
To test for the presence of allocation change, we assess whether the presence of the focal advertiser's main competitor affects the probability with which the focal advertiser displays their ad. More precisely, we focus on auctions to display ads to users who belong to the focal advertiser's treatment group, so that the focal advertiser is always able to factually show their ad. We then separate this group into two, one in which users belong to the main competitor's treatment group, so that they can see the main competitor's ad, and one in which users belong to the main competitor's control group. The test then compares the fractions of auctions such that the focal advertiser's ad was shown between these two groups. If these fractions are different, we have evidence of the presence of allocation change. To implement this statistically, we perform a t-test to assess whether the fraction of auctions the focal advertiser wins varies with the presence of their main competitor. Results are displayed in Table <ref>.

 

We find strong evidence of allocation change. Across all 16 campaigns, we are able to reject the null hypothesis that the proportion of auctions the focal advertiser wins does not depend on the presence of their main competitor. This shows that: (a) competition interference effects exist, and (b) allocation change is an important channel by which they operate in this marketplace.

Combined with the results in Table <ref>, this also demonstrates the presence of externalities. This is because of the heterogeneity across the interaction parameters both qualitatively and quantitatively. Take, for example, the results for campaigns 2 and 3.  The presence of their main competitors, as indicated by Table <ref>, affects the probability with which their ads are shown, implying that their main competitors are able to show their ads to users in the focal firm's experiment. However, the interaction parameter in Table <ref> for campaign 2 is positive and significant, while that for campaign 3 is negative and significant. This implies that the focal advertiser's payoff is affected by the identity of the ad that is shown to users, which is what we refer to as externalities.

Therefore, we have evidence that the competitive interference effects we detected in Section <ref> are a result not only of allocation change but also of externalities, which demonstrates the complexity and richness of this competitive advertising environment. The effect of a firm's advertising is not only impacted by the number of competitors it has since this number directly impacts its ability to show its ad. In addition, the identities of the competitors that are able to show their ads is also relevant. Furthermore, it is noteworthy  that the interaction effects can be either positive or negative because this indicates that firms' advertising can be either complementary or substitutes.



 §.§ Recovering all possible ATEs

 
 We now unpack competitive interference effects in more detail by implementing the kernel-based estimator described in Section <ref>. We illustrate this approach by focusing on campaign number 3. We chose this campaign for two reasons. First, its estimated unconditional ATE is significant. Second, its relatively lower number of observations facilitates the use of the kernel-based estimator, whose implementation, given that the bandwidths are computed via leave-one-out cross-validation, is very computationally intensive.
 


  §.§.§ Cross-validated bandwidths


Before we present the main results, we first display in Figure <ref> the cross-validated bandwidths we compute for campaign number 3 with respect to each competitor. These bandwidths serve as a first indication not only of competitive interference effects, but also of which competitors are associated with stronger such effects.

We believe this is an important intuition. To see this, recall that setting the entire vector λ to zero in the kernel-based estimator yields the linear regression estimator shown in equation (<ref>). This can be more easily seen via equations (<ref>) and (<ref>). Equation (<ref>) shows that if λ=0, then L(Z_if,z,λ)=1 if Z_if=z and L(Z_if,z,λ)=0 otherwise. Plugging these results in equation (<ref>) shows that when λ=0, the only observations used to estimate θ(z) are those such that Z_if=z. Thus, the resulting estimator corresponds to the one from equation (<ref>). In contrast, setting λ=1 implies that L(Z_if,z,λ)=1 for all z, so that all observations are used regardless of z. 

A λ=0 thus implies no pooling of observations and corresponds to a situation in which the focal firm's ATE is different depending on each state of the competitors' advertising policies. In turn, a λ=1 implies complete pooling of observations and corresponds to a situation in which the focal firm's ATE does not vary at all depending on its competitors' advertising policies. Therefore, the smaller a focal firm's λ with respect to a competitor, the more that competitor induces interference on the focal firm.[To understand this more deeply, notice that a higher value of λ implies more pooling of observations and, consequently, borrowing more information from observations with z' to estimate θ_f(z), where z'≠ z. This “information loan” is traded off against worsening the model fit: a higher value of λ penalizes model performance by increasing mean squared error (MSE). If increasing λ causes a sizable increase in the MSE, this is evidence that θ_f(z)≠θ_f(z'), and therefore that there is relevant interference.] Note that because the bandwidths are computed based on the data, as shown in  equation (<ref>), we can interpret the computed bandwidths as empirical summaries from the data of the magnitude of  interference from each competitor. 

 

The cross-validated bandwidths computed for campaign 3 with respect to its 15 competitors are displayed in Figure <ref>. The bandwidths associated with competing firms 1, 4, 9 and 15 are very close to 1, indicating that firm 3 does not suffer significant competitive interference effects from these competitors relative to others. In turn, firms 2, 12,13 and 14 display the smallest bandwidths, suggesting that these are the competitors that cause the strongest  competitive interference effects on firm 3.[Interestingly, firm 10, which we previously assessed as firm 3's main competitor based on our simple “overlap” metric, has a relatively high bandwidth. This implies that it is not necessarily firm 3's closest competitor, suggesting that our previous test for interference based on the overlap metric was conservative.] Figure <ref> once again reiterates the presence of substantial competitive interference in the data.



  §.§.§ Estimates, competitive interference and comparison to naive estimate of ATE


We now present estimates of the 2^15=32,768 different ATEs of firm 3's advertising depending on the advertising policies of its rivals. The goal is to assess the extent to which the ATEs are heterogeneous, so that taking rivals' policies into account is crucial. Figure <ref> displays the estimated ATEs.

 

Figure <ref> distinguishes between statistically significant and non-significant estimates by displaying the former in blue and the latter in red. Significance is at the 5% level and is determined using a nonparametric bootstrap based on 50 replications. The properties of the estimator allow us to keep the bandwidths fixed when resampling, enabling us to follow a typical bootstrap procedure: we draw with replacement from the sample and recompute all ATEs for each bootstrap sample. Then, we compute standard errors for each ATE using the 50 bootstrap samples and use them to assess statistical significance.

Out of the 32,768 estimates, 11,616, or 35%, are statistically significant at the usual level, indicating that the variation we see in ATEs by competitor policies are not driven simply by sampling and estimation noise. It is interesting to note that most negative estimates, which are arguably more counter-intuitive, are not statistically significant. 

To visualize the spread in ATEs caused by competitive interference, Figure <ref> displays the estimated density of the estimated ATEs, computed using a Gaussian kernel and Silverman's rule-of-thumb bandwidth. At the top right we display relevant quantities of this distribution. It ranges from -0.2408 to 0.1853, with a mean of 0.0320. Compared to a mean of 0.0320, the distribution has a standard deviation of 0.0411, indicating the presence of meaningful heterogeneity across ATEs. Finally, with a median of 0.0348 the distribution is fairly symmetric.

The analysis of dispersion in the distribution of ATEs is relevant to assess the importance of what we refer as “environmental uncertainty”: it is a reflection of a firm's uncertainty over which of its competitors will be advertising in the market. Because ATEs are estimated to aid firms in their decision-making, high dispersion across them indicates that consideration of environmental uncertainty is important.

Nevertheless, environmental uncertainty is ignored by the typical approach firms follow, which focuses on the unconditional ATE, obtained typically by the difference in means estimator. If environmental uncertainty is relevant, following this typical approach can hurt decision-making, as we study in more detail below. As a precursor to this analysis, we now consider how much of the environmental uncertainty is captured by the statistical uncertainty around the estimate of the unconditional ATE obtained from the typical difference in means approach. If statistical uncertainty around the unconditional ATE captures very little of the environmental uncertainty, typical decision-making that incorporates the former, but ignores the latter, can be quite sub-optimal.

To assess this, Figure <ref> also displays the difference in means estimate of the unconditional ATE of firm 3, along with its respective 95% confidence interval, which we use as a measure of its statistical uncertainty. We compute the unconditional ATE using all users that are exposed either factually or counterfactually to firm 3's ad. This point estimate is 0.0300 with a 95% confidence interval ranging from 0.0147 to 0.0453. As Figure <ref> shows, the range of ATEs covered by this confidence interval is small relative to the full range of possible ATEs. The full range is from -0.2408 to 0.1853, so the confidence interval for the unconditional ATE covers only 7.18% of the full range. This suggests that even if decision-making accounts for statistical uncertainty around the unconditional ATE, it can still perform poorly by ignoring a substantial portion of the environmental uncertainty. We investigate this in more detail below.

 



  §.§.§ Implications for firm's decision-making


So far we have described the results we obtained when we recovered all possible ATEs and how they compare to the unconditional ATE recovered via simple differences in means. This section assesses how these different approaches can affect decisions made by firms after the experiment ends.



  
Decision scenario

We consider a simple scenario in which a firm needs to make the decision of whether to advertise on the platform. This decision is made to maximize its expected payoff from advertising, so that the optimal decision is to advertise if and only if the ATE exceeds the cost of advertising, which we denote by κ. In other words, firm f's optimal advertising policy as a function of its competitors' policies is given by:

    A_ft^* ( A_-f,t ) = 1{ ATE ( A_-f,t )≥κ},

where we assume for simplicity that κ does not vary with A_-f,t.[One could imagine that this is not the case in practice. For instance, if ads are bought through auctions, then the number of competitors should affect the price paid. We maintain this assumption to keep the analysis simple.] Going forward, we will refer to this decision as a “policy” and to the treatment effects required to implement it as policy relevant. For illustration, we will compute policies based on the point estimates of the ATEs.



  
Three simple policies for practical decision-making

If firm f could obtain all ATEs such as those shown in Figure <ref>, implementing the optimal advertising policy given in (<ref>) would be straightforward by using the appropriate ATE for each possible competitor state, so that no ex ante mistakes would be made. However, doing this may be difficult in practice because (a) the platform can face constraints on how much information it can disclose to firms, especially regarding their competitors' activities; and (b) leveraging all possible ATEs requires a large degree of analytical sophistication and computational capability on the part of firms, which may be lacking.  Hence, we consider three alternative policies that use more limited information than the entire distribution of ATEs. 

The first policy consists of the firm advertising if the unconditional ATE obtained from the typical difference in means estimator is above the advertising cost κ. In other words, the firm plugs this point estimate into equation (<ref>) and implements the resulting policy. Because experimenters typically compute only this ATE, we will refer to this policy as the “default-ATE policy.”

The second policy we consider is one in which the platform estimates all the τ_ft(A_-f,t) terms, but only reports to firm f the terms averaged across A_-f,t. The firm then plugs this “integrated” ATE into equation (<ref>) and implements the resulting estimated optimal policy. This policy is motivated by a Bayesian statistical decision theory problem faced by the firm. Assuming that the firm's uncertainty about the ATE is induced by uncertainty over A_-f,t, the firm's optimal decision would be to advertise if the posterior expected ATE exceeded the cost κ. This approach then takes the estimated distribution of ATEs as this posterior distribution of τ_ft(A_-f,t) to compute the posterior expected ATE. We will refer to this policy as the “integrated-ATE policy.”

Finally, the last policy we consider is one in which the platform reports, for each value of κ, the probability that the ATE exceeds it. The firm would then randomly advertise according to this probability, which corresponds to a simple form of probability matching. We will refer to this policy as the “randomized-ATE policy.”

The default-ATE and randomized-ATE policies are heuristics, while the integrated-ATE policy can be motivated by a Bayesian statistical decision theory problem. Nevertheless, their actual performance should be assessed empirically. 



  
Assessment across three scenarios

To compare these three policies, we consider three different scenarios. First, we assume that all A_-f,ts are equiprobable. Next, we only consider A_-f,t values such that only one competitor is present, and assume that the resulting 15 scenarios occur with equal probability. Finally, we consider a scenario in which advertisers run their experiments and follow the naive approach of only computing an unconditional ATE via a difference in means estimator that ignores parallel experimentation. Then, we only consider cases where the advertisers with a significant estimated ATE at the 5% significance level as shown in Table <ref> (i.e., advertisers 1, 5, 9, 11, and 15) always advertise. We assume these cases occur equiprobably. For each of the 3 scenarios, we compute the probability of firm 3 making an ex ante mistake when following each of the 3 possible policies, and repeat this exercise for different values of κ. The results are displayed in Figure <ref>. 





  
Results from following alternative policies

Overall, we find that all three policies have scope for inducing ex ante mistakes relative to using the full set of ATEs for decision-making. We can view this as reflecting inefficiencies induced by two reasons. One, by making decisions based on the false assumption of homogeneous treatment effects, when treatment effects (with respect to competition) are actually heterogeneous. Two, by using simple decision rules, when the optimal decision rule is in fact much more complex and nuanced. 

Furthermore, all policies display an inverted U-shape-like pattern of probability of ex ante mistakes. This has a straightforward explanation. When κ is very low, the majority of ATEs exceed it, so, by equation (<ref>), in most scenarios the optimal decision is to advertise; hence, regardless of what policy is used the ex ante probability of mistake is relatively low. An analogous result applies to high values of κ, which exceeds most ATEs so that in most scenarios the optimal decision is to not advertise, and this is captured by all policies. Consequently, the bigger discrepancies between the policies we consider happen when the value of κ is intermediate relative to the range of ATEs.

Empirically, however, we find that the integrated-ATE policy performs best across these three situations. It strictly dominates the default-ATE policy and the randomized-ATE policy when only competitor can be present at a time. When all scenarios are possible and when only scenarios where advertisers with a significant estimate of their ATE always advertiser are considered, the integrated-ATE policy dominates the randomized-ATE policy over the vast majority of values of κ (average probability of ex ante mistake of around 23% for the integrated-ATE policy versus an average probability of ex ante mistake of around 30% for the randomized-ATE policy). 

Overall, we have two takeaways from these results. First, relying on the naive unconditional ATE that ignores the competitive interactions can lead to many mistakes and poor-quality decision-making by experimenting firms. Second, the assessments of the simple policies are encouraging for practical decision-making. They indicate that the information obtained from the nonparametric estimator can be summarized in a single and easy-to-interpret quantity (i.e., the integrated ATE) which, when computed by platforms and reported to experimenting firms, can lead to much better decision-making. This may represent a practical way for platforms  to implement the theoretical intuition developed here into their experimentation reporting and decision-recommendation frameworks.





Conclusions

An assessment of experimentation on platforms when multiple firms are experimenting in parallel is presented. The core focus is on interactions that are induced by equilibrium platform competition between the firms and the impact of such interactions on the measurement of treatment effects under such parallel experimentation. The headline point of the paper is that when competition between firms is significant, such as when many advertisers compete for capacity constrained ad-slots on publishers, these interactions can non trivially affect the causal effects measured by platform experimentation. 

One channel of interaction, which we call allocation change, occurs when the chance a focal advertiser can show their ad to their targeted users is affected by the presence of competitors. Another channel of interaction, which we call externalities, occurs when the effect of a focal advertiser's ad on targeted users is affected by their exposure to competitors' ads. In competitive platforms, both these interactions become significant and affect what is measured during experimentation. When experimentation ends, the nature of these interactions are altered, leading to concerns about the transportability of the measurements obtained during the experimentation phase to the post-experiment phase where firms need to make decisions based on what they learned during the experimentation phase.


On this topic, this paper accomplishes the following:


  * it demonstrates that parallel experimentation can act as a double-edged sword. On the one hand, it helps to estimate a set of stable treatment effects, which are transportable to post-experimentation scenarios. On the other hand, it degrades the interpretability and empirical content of simple estimators, such as differences in means for a given advertiser's test and control groups, which ignore such parallel experimentation;

  * it outlines a framework to analyze these interactions formally, and presents a new set of causal estimands that have precise interpretability in the presence of competitive interactions and that are transportable;

  * it outlines a simple platform experimental design that enables the investigation of competitive interference, and proves that the proposed new causal estimands  are identified under this design;

  * it presents two estimators which facilitate inference under such a design. One of the estimators leverages recent advances in nonparametric econometrics to pool observations from different competitive states to handle the high-dimensional dependence of treatment effects of a firm on the presence or absence of its competitors, and thus scales to situations with a large number of parallel experiments while facilitating interpretability;

  * it demonstrates how these effects affect post-experimental decision-making by firms in the context of an ad-campaign's stay or exit problem. Some simple heuristics which can be used in practical settings with limited computational demands on firms and platforms are also presented and evaluated; 

  * it incorporates these aspects into an advertising experimentation platform  built for , an e-commerce platform and ad-publisher, showing how the experimental design can be engineered into an auction-driven marketplace, and how the relevant causal estimands  can be recovered by the platform; 

  * and it presents data from a large-scale field experiment  conducted on this platform, which documents that interactions matter and serving as a real-world empirical illustration of the proposed framework.


While the paper focuses on advertising, similar issues arise in other scenarios such as experimental testing of drugs in situations where tested drugs have interaction effects with other drugs; or in pricing experiments where the effect of a focal firm's price depends on the prices offered by other firms. Therefore, the ideas presented in this paper may be useful in developing experimental approaches to understanding situations with interactions across experiments more broadly. 













tablesubsection   

equationsubsection



§ APPENDIX




 §.§ Proofs




  §.§.§ Proof of Theorem <ref>


Fix t, E_t and O_t. Because of Assumption <ref>, the experiment generates data on all possible treatment and control combinations. That is, all combinations such that: if f ∉ E_t ∪ O_t, then D_ift=1 for all i ∈ t; if f∈ O_t, then D_ift=0 for all i ∈ t; and if f ∈ E_t, then D_ift=1 or D_ift=0. 

As described in Remark <ref>, if i belongs to g's treatment group, then i is eligible to see g's ad, so that D_igt=1 is equivalent to A_gt=1, and the opposite holds if i belongs to g's control group. Consequently, each one of the partial treatment assignments, D_i,-f,t, that are observed in the experiment corresponds to one A_-f,t subject to the constraints that if g∈ O_t, then A_gt=0, and if g∉ E_t∪ O_t, then A_gt=1. In addition, Assumption <ref> guarantees that the ad sequence exposure probabilities in the experiment correspond to those of the non-experimental environment. Putting these results together, it follows that

    𝔼 [Y_ift| D_ift,D_i,-f,t ] = 𝔼 [ Y_ift| A_ft,A_-f,t ].

The result then follows from equation (<ref>).



  §.§.§ Proof of Theorem <ref>


Fix t, E_t and O_t. We know that: (i) the experiment generates data on all possible treatment and control combinations given E_t and O_t; and (ii) for each i in t the partial treatment assignment D_i,-f,t corresponds to a different A_-f,t. 

However, we now do not know, or disregard, D_i,-f,t. Hence, the object that can be recovered is 𝔼 [Y_ift| D_ift, D_i,-f,t ], as given in equation (<ref>), but integrated with respect to D_i,-f,t given σ_-f,t. In other words, the identified object is the one in equation (<ref>). Fixing σ_-f,t, the set of all partial treatment assignments is given by 𝔻 ( σ_-f,t). Because of Assumption <ref>, the probability of each partial treatment assignment is 

    (D_i,-f,t|σ_-f,t )=∏_g∈𝔽_0∖{f}σ_gt^D_igt ( 1-σ_gt )^1-D_igt.

Hence, the experiment identifies

    𝔼 [Y_ift| D_ift, σ_-f,t ] =  (D_i,-f,t|σ_-f,t )×𝔼 [Y_ift| D_ift, D_i,-f,t ].

Finally, from Remark <ref> it follows that

    𝔼 [Y_ift| D_ift,σ_ -f,t ] = 𝔼 [ Y_ift| A_ft,σ_-f,t ].

The result then follows from equation (<ref>).



  §.§.§ Proof of Theorem <ref>


Consider a firm f ∈𝔽_0. Fix E_t and O_t. Assumption <ref> ensures that all feasible treatment assignments are observed. Consider one such treatment assignment, D_it. Given Assumption <ref>, p_Wt (D_ift, D_i,-f,t ) is identified from observations whose full treatment assignment is D_it. Furthermore, Assumption <ref> implies that the display probabilities during the experiment correspond to those that hold without experimentation. Hence, for the A_t that is equivalent to this D_it, it follows that

    p_Wt (D_ift,D_i,-f,t ) =  p_Wt (A_ft,A_-f,t ),

establishing the result.



  §.§.§ Proof of Theorem <ref>


Under the conditions stated, Theorem <ref> holds. Fix t, E_t, O_t and a D_it observed in the experiment, and let A_t be the advertising policy vector to which D_it is equivalent. Consider a firm f∈𝔽_0. If p_Wt (D_ift,D_i,-f,t ) =  p_Wt (A_ft,A_-f,t )>0, then Y_ift(W) is observed whenever W_it=W. Hence, since μ_fWt=𝔼 [Y_ift(W_it) | W_it=W   ], this object is identified.




