Learning control variables and instruments
for causal analysis in observational data

Nicolas Apfel*, Julia Hatamyar**, Martin Huber+, Jannis Kueck++

*University of Southampton, Department of Economics
**University of York, Centre for Health Economics
+University of Fribourg, Department of Economics
++Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition
Economics

###### Abstract

This study introduces a data-driven, machine learning-based method to detect
suitable control variables and instruments for assessing the causal effect of
a treatment on an outcome in observational data, if they exist. Our approach
tests the joint existence of instruments, which are associated with the
treatment but not directly with the outcome (at least conditional on
observables), and suitable control variables, conditional on which the
treatment is exogenous, and learns the partition of instruments and control
variables from the observed data. The detection of sets of instruments and
control variables relies on the condition that proper instruments are
conditionally independent of the outcome given the treatment and suitable
control variables. We establish the consistency of our method for detecting
control variables and instruments under certain regularity conditions,
investigate the finite sample performance through a simulation study, and
provide an empirical application to labor market data from the Job Corps
study.
JEL Classification: C12, C21, C26
Keywords: treatment effects, causality, conditional independence, instrument,
covariates, hypothesis test.

Addresses for correspondence: Nicolas Apfel, Murray Building (B58), Highfield
Campus, University of Southampton, SO17 1TR Southampton, United Kingdom;
<EMAIL_ADDRESS>Julia Hatamyar, Research Fellow Centre for Health
Economics, Heslington, York, YO10 5DD, United Kingdom;
<EMAIL_ADDRESS>Martin Huber, University of Fribourg, Bd. de
Pérolles 90, 1700 Fribourg, Switzerland<EMAIL_ADDRESS>Jannis Kueck,
Heinrich Heine University Düsseldorf, Universitätsstr. 1, 40225 Düsseldorf,
Germany<EMAIL_ADDRESS>
##  1 Introduction

Methods for policy evaluation and causal analysis, aimed at quantifying the
impact of a treatment or policy intervention on an outcome variable,
conventionally rely on identifying assumptions considered untestable. For
example, the well-known selection-on-observables, unconfoundedness,
conditional independence, or ignorability assumption requires that the
treatment is exogenous when conditioning on observed control variables,
hereafter referred to as covariates, see for instance the survey by Imbens and
Wooldridge (2009). The selection of covariates is typically justified based on
theoretical and/or empirical reasoning, intuition, domain expertise, or prior
empirical findings. Nonetheless, in many, if not most, empirical scenarios,
such selections are subject to debate, given that the optimal set of
covariates meeting the selection-on-observables assumption remains
fundamentally uncertain.

In this paper, we suggest a machine learning-based procedure to simultaneously
test the presence of (i) covariates satisfying the selection-on-observables
assumption and (ii) relevant and valid instruments in observational data, as
well as learning which variables in the data belong to either the set of
covariates or instruments. When we refer to relevant and valid instruments, we
mean variables that are associated with the treatment (relevance) but have no
direct association with the outcome other than through the treatment
(validity) conditional on covariates. We demonstrate that appropriate sets of
covariates satisfying the identification requirements for treatment effects
based on the selection-on-observables assumption, as well as relevant and
valid instruments, can be detected in a data-driven way rather than being
assumed by the researcher. For testing and learning covariates and
instruments, we exploit a conditional independence condition that must hold
when both relevant and valid instruments as well as appropriate covariates
(satisfying the selection-on-observables assumption) exist: The instruments
must be conditionally independent of the outcome, given the treatment and the
covariates, see for instance the discussions in de Luna and Johansson (2014),
Black et al. (2015), and Huber and Kueck (2022).

Our approach consists of the following steps. First, within the combined set
of potential covariates and instruments, we sequentially test which variable
is strongly associated with the treatment conditional on all remaining
variables in that set. Second, we consider each of these strong predictors of
the treatment as candidate instruments and sequentially test whether each of
them is conditionally independent of the outcome when controlling for the
treatment and all remaining variables in the combined set of potential
covariates and instruments. If the conditional independence assumption is
satisfied by (at least) one candidate instrument, then the instrument validity
and selection-on-observables assumptions hold. This implies that the treatment
is as good as random conditional on the remaining variables within the
combined set of potential covariates and instruments. Consequently, treatment
effects can be estimated utilizing methods that control for observed
covariates, such as matching, regression, inverse probability weighting, or
doubly robust techniques, as for instance discussed in Huber (2023).

Our discussion of the test focuses on conditional mean (rather than full)
independence of the instrument, which implies the identification of average
treatment effects (ATE). When assuming a limited set of observed variables
(relative to the sample size), we employ regression for both selecting the
candidate instruments in the first step and testing the conditional mean
independence of the instrument in the second step. More concisely, testing is
based on the mean squared difference in outcome prediction when regressing the
outcome (1) on the treatment, the control variables, and the candidate
instrument and (2) on the treatment and the control variables (but not the
candidate instrument). This approach is closely related to the mean squared
difference test based on a quadratic score function in Huber and Kueck (2022),
but applies is sequentially to all candidate instruments. We demonstrate that
our method is consistent for detecting covariates and instruments under
certain regularity conditions. In a simulation study, we find support for this
consistency result. However, as a word of caution for empirical applications,
the results also suggest that decent performance of the test might require a
relatively large dataset rather with more than just several 1000 observations.

As an empirical illustration, we apply our method to data from an experimental
study on Job Corps, a significant U.S. education program financed by the
Department of Labor, targeting disadvantaged individuals aged 16 to 24, which
has been previously analyzed in studies by Schochet et al. (2001) and Schochet
et al. (2008). In this dataset, assignment to the program was randomized,
making it a plausible candidate instrument for actual training participation
in the first year of Job Corps, the treatment under investigation, if any
direct effects of mere assignment on labor market outcomes like wage can be
ruled out. Indeed, our approach selects random assignment as valid instrument.
The method also suggests that training participation is exogenous conditional
on a set of 28 pre-assignment covariates, with the p-value of the test
statistic amounting to 76%.

Our paper contributes to a growing literature on testing identifying
assumptions for causal inference. For instance, de Luna and Johansson (2014)
and Black et al. (2015) make use of the same conditional independence of the
instrument as considered here to test the selection-on-observables assumption
for the treatment when assuming a valid instrument, based on matching or
regression estimators.111Angrist (2004), Brinch et al. (2017), and Huber
(2013) consider related tests when instrument validity is holds
unconditionally without controlling for covariates, in order to test the
unconditional independence of the treatment and potential outcomes. Bertanha
and Imbens (2015) consider the fuzzy regression discontinuity design (RDD),
where instrument validity holds at a cutoff of a running variable which
discontinuously affects treatment assignment. In contrast, Huber and Kueck
(2022) jointly test both instrument validity and selection-on-observables
assumptions with pre-defined instruments and covariates using machine learning
approaches which permit for high dimensional covariates. This is conceptually
closest to our approach, but one important difference is that the method of
Huber and Kueck (2022) requires specifying the sets of supposed instruments
and covariates to be used for testing, whereas in this paper, these sets of
covariates and instruments are learned from the data and may therefore be a
priori unknown.

Beyond that, we provide a new orthorgonalized quadratic score which improves
on the quadratic score function used in Huber and Kueck (2022). This new score
is of more general interest since it can be applied in many other contexts
than ours as well. One example is the study by Parikh et al. (2024), who
compare experimental and non-experimental treatment effect estimates and use
an indicator for distinguishing between experimental and non-experimental
evaluation designs, which has a comparable role as the instrument in our
paper. Imposing the external validity of the experiment permits assessing the
selection-on-observables assumption in the non-experimental design.
Conversely, imposing the selection-on-observables assumption in the non-
experimental design permits assessing the external validity of the experiment,
which is equivalent to testing IV validity in our context. Unlike our paper,
the authors do not consider testing both assumptions jointly. Also Angrist and
Rokkanen (2015) use the same conditional independence of the instrument as
considered here to test instrument validity within the framework of the sharp
regression discontinuity design (RDD), where the selection-on-observables for
the treatment holds by design, because it is a deterministic function of a
cutoff in a running variable. This allows for testing whether the running
variable is a valid instrument, i.e., whether it is not associated with the
outcome conditional on the treatment.222If the running variable is a valid
instrument, causal effects can also be identified away from the cutoff, which
is otherwise not feasible due to the lack of common support in the treatment
across different values of the running variable. Again, our approach differs
in that it tests the selection-on-observables and instrument validity
assumptions jointly (rather than the latter conditional on the former).

Our study is also related to several contributions in the computer science-
dominated literature on causal discovery, see see e.g. Kalisch and Bühlmann
(2014), Peters et al. (2017), Glymour et al. (2019) for reviews. For instance,
Peters et al. (2015) make use of pre-defined instruments and machine learning
techniques to learn, in a data-driven way, which of the observed variables are
treatments in the sense that they directly affect the outcome variable, under
the assumption that these treatments satisfy the selection-on-observables
assumption. The approach makes use of instruments in a way that may entail the
rejection of treatments which violate the selection-on-observables
assumptions, thereby providing power to detect identification failures. Our
approach is different in that it focuses on a single, pre-defined treatment of
interest, tests selection-on-observables and instrument validity jointly, and
learns the sets covariates and instruments from the data.

Soleymani et al. (2022) and Quinzan et al. (2023) provide algorithms that
select treatment variables in a data-driven way, while also controlling for
observed covariates in a data-driven manner based on the double machine
learning (DML) framework of Chernozhukov, Chetverikov, Demirer, Duflo, Hansen,
Newey and Robins (2018). The idea is to sequentially consider each of the
observed variables that may affect the outcome as treatment, while considering
all remaining observed variables as covariates to estimate the direct effect
of each candidate treatment on the outcome by DML. The algorithms retain only
those observed variables as treatment variables that exhibit statistically
significant effects on the outcome (where judging statistical significance
should account for issues related to testing multiple hypotheses based on
multiple candidate treatments). In contrast to our approach, these algorithms
do not exploit instruments to test identifying assumptions, but assume the
selection-on-observables assumption holds for all treatments. Nevertheless,
the algorithms are useful also in our context, namely for selecting potential
instruments with a sufficiently strong first stage effect on the pre-defined
treatment conditional on other observed variables.333Another domain of causal
discovery related to our study is so-called Y-learning, see e.g. Mani et al.
(2012) and Sevilla and Mayn (2021). Conditional on covariates, Y-learning
implies that if two variables are independent of each other when not
controlling for the treatment, statistically associated with each other when
controlling for the treatment, and both independent of the outcome when
controlling for the treatment, then these two variables are relevant and valid
instruments. A further implication is that the selection-on-observables
assumption holds. Our approach differs from Y-learning in that it imposes more
causal structure by assuming that potential instruments and covariates are not
affected by the treatment. For this reason, our method only requires a single
(and a priori unknown) instrument, while Y-learning hinges on the existence of
(at least) two instruments.

The computer science literature most closely related to our study is
representation learning of sets of variables that are confounders jointly
affecting a predefined treatment and outcome, instruments only affecting the
treatment, and outcome predictors only affecting the outcome. Hassanpour and
Greiner (2019) and Wu et al. (2021), for instance, propose deep learning
algorithms minimizing global loss functions to simultaneously decompose pre-
treatment variables into instruments, confounders, and outcome predictors.
Yet, there are several differences between their studies and ours: First,
their approaches impose selection-on-observables a priori to isolate
instruments (whose exclusion from treatment effect estimation can increase
efficiency) based on the same conditional independence condition as considered
in our paper. However, we exploit the conditional independence condition to
test selection-on-observables and instrument validity jointly. Second, their
algorithms, which minimize a global loss function, aim at learning the full
set of instruments, which is attractive for maximising efficiency in treatment
effect estimation but might be very ambitious to achieve in a finite sample.
In contrast, our algorithm of iteratively changing candidate instruments aims
at a more moderate goal of detecting at least one instrument, which is a
sufficient condition for identification. Third, we demonstrate the consistency
of our algorithm in selecting valid instruments under certain regularity
conditions, while the asymptotic behaviour of the deep learning methods in
Hassanpour and Greiner (2019) and Wu et al. (2021) has not been derived.

Finally, our contribution also relates to a growing literature in statistics
and econometrics employing machine learning techniques to separate valid from
invalid instruments (which are confounders to be controlled for), see for
instance Guo et al. (2018), Windmeijer et al. (2021), Windmeijer et al.
(2019), Apfel and Liang (2021), and Apfel et al. (2022). These approaches rely
on the assumption that a sufficiently large number of (a priori unknown)
instruments is valid, and in order to detect the set of valid instruments
Hansen-Sargan-type tests in combination with instrument-based estimators are
used. In contrast, our approach does not pre-impose the existence of valid
instruments, but tests the instrument validity (for a single instrument) and
selection-on-observables assumptions jointly, in order to apply estimation
based on the selection-on-observables assumption.

The remainder of this study is organized as follows. Section 2 discusses a set
of identifying assumptions, including instrument relevance and validity as
well as the selection-on-observables assumption, which imply the conditional
independence of instruments and the outcome conditional on the treatment and
covariates. Based on this conditional independence, Section 3 proposes machine
learning-based procedures for jointly testing the instrumental variable and
selection-on-observables assumptions in observational data. Section 4 suggests
an algorithm that applies these tests sequentially when rotating the role of
pre-treatment variables to detect relevant and valid instruments as well as
covariate sets satisfying the selection-on-observables assumption in a data-
driven manner. Section 5 provides a simulation study analyzing the finite
sample performance of our method. Section 6 presents an application to labor
market data from the Job Corps study. Section 7 concludes.

##  2 Identifying assumptions and testable conditional independence

This section reviews the findings in de Luna and Johansson (2014), Black et
al. (2015), and Huber and Kueck (2022), which imply that under instrument
validity and a selection-on-observables assumption concerning the treatment,
the instrument is conditionally independent of the outcome given the treatment
and observed covariates. To formalize the assumptions, let us denote by $D$ a
treatment variable whose causal effect on an outcome variable $Y$ is of
interest. Both $D$ and $Y$ might be discretely or continuously distributed.
Using the potential outcomes framework as for instance advocated in Neyman
(1923) and Rubin (1974), we denote by $Y(d)$ the potential outcome when
exogenously setting the treatment $D$ of a subject to value $d$ in the support
of the treatment. More generally, we will use capital and lower case letters
for referring to random variables and specific values thereof, respectively.
By representing the potential outcome $Y(d)$ as a function solely dependent on
a subject’s own treatment status $D=d$, we implicitly adhere to the assumption
that the potential outcomes of one subject are not influenced by the treatment
status of others. This is known as the ‘stable unit treatment value
assumption’ (SUTVA), see for instance the discussion in Rubin (1980) and Cox
(1958), and is invoked throughout. Furthermore, we denote by $X$ and $Z$ sets
of observed covariates and instruments, whose properties are yet to be
defined. Based on this notation, we consider the same identifying assumptions
as Huber and Kueck (2022).

The first assumption imposes some causal structure on our framework. First, it
rules out the existence of reverse causality, meaning that the outcome cannot
causally influence any other variables, and the treatment cannot causally
affect any variables other than the outcome. We formalise this causal
structure using the previously mentioned potential outcome notation, by
applying the latter also to other variables. To this end, let $A(b)$ and
$A(b,c)$ correspond to the potential value of variable $A$ when setting
variable $B$ to $b$, or variables $B$ and $C$ to $b$ and $c$, respectively.
Second, our first assumption enforces the principle of causal faithfulness,
which dictates that variables linked by causal pathways, possibly conditional
on other variables, are necessarily statistically dependent.

######  Assumption 1 (Causal structure).

$\displaystyle D(y)=D,\quad X(d,y)=X,\textrm{ and }Z(d,y)=Z\quad\forall
d\in\mathcal{D}\textrm{ and }y\in\mathcal{Y},$

only variables which are d-separated in some causal model are statistically
independent.

$\mathcal{D}$ and $\mathcal{Y}$ denote the support of $D$ and $Y$,
respectively. The first line of Assumption 1 rules out a causal effect of
outcome $Y$ on $D$, $X$, or $Z$ and of treatment $D$ on $X$ or $Z$. However,
it allows for the possibility of both $X$ and $Z$ affecting $D$, $Y$, or even
each other. This assumption aligns with the directed acyclic graph (DAG, see
e.g. Pearl (2000)) presented in Figure 1, where causal relationships between
variables are indicated by arrows: $Z$ and $X$ affect $D$, and $D$ and $X$
affect $Y$. Additionally, $X$ may influence $Z$ or vice versa, denoted by the
bidirectional causal arrow. This is in line with the conventional practice of
measuring covariates and instruments before treatment assignment, eliminating
the potential for reverse causality between $D$ and $Y$ and the pre-treatment
variables $X$ and $Z$. The DAG also features unobserved terms $U$ and $V$ that
affect $Y$ and $D$, respectively, with dashed arrows denoting the unobservable
nature of these effects. The second line of Assumption 1 enforces causal
faithfulness, ensuring that only variables which are d-separated in the sense
of Pearl (1988), i.e. not associated with each other via some causal paths
(possibly conditional on other variables) are statistically independent (or
conditionally independent).444 d-separation relies on blocking causal paths
between variables. Formally, a path between two (sets of) variables $A$ and
$B$ is blocked when conditioning on a (set of) control variable(s) $C$ if  1.
the path between $A$ and $B$ is a causal chain, implying that $A\rightarrow
M\rightarrow B$ or $A\leftarrow M\leftarrow B$, or a confounding association,
implying that $A\leftarrow M\rightarrow B$, and variable (set) $M$ is among
control variables $C$ (i.e. controlled for), 2. the path between $A$ and $B$
contains a collider, implying that $A\rightarrow S\leftarrow B$, and variable
(set) $S$ or any variable (set) causally affected by $S$ is not among control
variables $C$ (i.e. not controlled for). Based on this definition of blocking,
the d-separation criterion states that $A$ and $B$ are d-separated when
conditioning on control variable(s) $C$ if and only if $C$ blocks all paths
between $D$ and $Y$. While d-separation is generally a sufficient condition
for the (conditional) independence of two variables, it is a necessary
condition under causal faithfulness.

Figure 1:  Causal graph satisfying Assumption 1

The second assumption is a common support assumption concerning the treatment
and the instrument:

######  Assumption 2 (Common support).

$\displaystyle\mathbb{P}(D=d,Z=z|X)>0\quad\forall d\in\mathcal{D}\textrm{ and
}z\in\mathcal{Z},$

where $\mathcal{Z}$ denotes the support of $Z$. For discretely distributed
treatments and instruments, Assumption 2 states that the joint probabilities
of any $D=d$ and $Z=z$ conditional on $X$ are larger than zero. Under
continuous treatment and/or instrument variables, then the joint probabilities
are to be replaced by joint density functions conditional on $X$. Assumption 2
implies that both $\mathbb{P}(D=d|X)$, the so-called treatment propensity
score, and $\mathbb{P}(Z=z|D,X)$, the instrument propensity score, are larger
than zero.555Assumption 2 is strictly speaking not required for testing, if
testing only proceeds in a subgroup satisfying common support. However, this
may imply reduced testing power.

The third assumption imposes a statistical association between the instrument
and the treatment conditional on the covariates.

######  Assumption 3 (Conditional dependence between the treatment and
instrument).

$\displaystyle D\not\\!\perp\\!\\!\\!\perp Z|X,$

where $\not\\!\perp\\!\\!\\!\perp$ denotes statistical dependence. Together
with Assumption 1, which rules out effects of $D$ on $Z$, Assumption 3 either
implies that $Z$ causally affects $D$, which is known as first stage effect in
the IV literature, or that some (unobserved) characteristics jointly affect
$Z$ and $D$ given $X$. This assumption is satisfied in Figure 1, where $Z$
affects $D$.

The fourth assumption invokes selection on observables, i.e. quasi-random
treatment assignment conditional on covariates as e.g. considered in Imbens
(2004):

######  Assumption 4 (Conditional independence of the treatment).

$\displaystyle Y(d){\perp\\!\\!\\!\perp}D|X\quad\forall d\in\mathcal{D},$

where ${\perp\\!\\!\\!\perp}$ denotes statistical independence. Assumption 4
implies that conditional on covariates $X$, there exist no unobserved
confounders jointly affecting outcome $Y$ and treatment $D$.

The fifth assumption invokes instrument validity, requiring that the
instrument is conditionally independent of the potential outcomes given the
covariates $X$.

######  Assumption 5 (Conditional independence of the instrument).

$\displaystyle Y(d){\perp\\!\\!\\!\perp}Z|X\quad\forall d\in\mathcal{D}.$

Assumption 5 rules out unobserved confounders jointly affecting $Y$ and $Z$
when controlling for $X$. This is analogous to Assumption 4, but now
concerning $Z$ rather than $D$. Furthermore, by assuming that the potential
outcome is solely a function of $d$ (and not $z$), Assumption 5 also implies
that the instrument does not have a direct impact on the outcome, except
through its influence on the treatment, conditional on $X$. By this exclusion
restriction, it holds that conditional on $X$, $Y(d,z)=Y(d,z^{\prime})=Y(d)$
for any instrument values $z$ and $z^{\prime}$. Otherwise, Assumption 5 would
be violated, because it would follow that $Y(d)=Y(d,Z)$ and
$Y(d,Z)\not\\!\perp\\!\\!\\!\perp Z|X$.

The previous assumptions can be used to construct tests for the identification
of causal effects. Theorem 1 in Huber and Kueck (2022) demonstrates that
conditional on Assumptions 1 and 3, the testable conditional independence
$Y{\perp\\!\\!\\!\perp}Z|D=d,X$ is necessary and sufficient for the joint
satisfaction of Assumptions 4 and 5 when considering potential outcomes $Y(d)$
which match the factual treatment assignment $D=d$. Formally,

$\displaystyle Y(d){\perp\\!\\!\\!\perp}D|X,\quad
Y(d){\perp\\!\\!\\!\perp}Z|X\iff Y{\perp\\!\\!\\!\perp}Z|D=d,X\quad\forall
d\in\mathcal{D}.$ (2.1)

Instead of verifying $Y{\perp\\!\\!\\!\perp}Z|D,X$, we will focus on testing
whether conditional mean independence of the instrument holds:

$\displaystyle\mathbb{E}[Y|D,X]=\mathbb{E}[Y|D,X,Z].$ (2.2)

Condition (2.2) is sufficient when considering the identification of average
causal effects such as the conditional average treatment effect (CATE) given
$X$, $\mathbb{E}[Y(1)-Y(0)|X]$, or the average treatment effect (ATE),
$\mathbb{E}[Y(1)-Y(0)]$, rather than distributional parameters like quantile
treatment effects. Theorem 2 in Huber and Kueck (2022) shows that (2.2) holds
when replacing Assumptions 4 and 5 by the weaker conditional mean independence
assumptions $\mathbb{E}[Y(d)|D,X]=\mathbb{E}[Y(d)|X]$ and
$\mathbb{E}[Y(d)|Z,X]=\mathbb{E}[Y(d)|X]$ $\forall d\in\mathcal{D}$, as well
as Assumption 3 by the first stage condition
$\mathbb{E}[D|Z,X]\neq\mathbb{E}[D|X]$, implying that the conditional mean of
$D$ varies with $Z$. Formally, conditional on Assumption 1 and
$\mathbb{E}[D|X,Z]\neq\mathbb{E}[D|X]$, it holds that

$\displaystyle\mathbb{E}[Y(d)|D,X]=\mathbb{E}[Y(d)|X],\quad\mathbb{E}[Y(d)|X,Z]=\mathbb{E}[Y(d)|X]$
$\displaystyle\iff$
$\displaystyle\mathbb{E}[Y|D=d,X,Z]=\mathbb{E}[Y|D=d,X]\quad\forall
d\in\mathcal{D}.$

The testable implication $\mathbb{E}[Y|D=d,X,Z]=\mathbb{E}[Y|D=d,X]$ is
necessary and sufficient for the joint satisfaction of conditional mean
independence of the treatment and the instrument when considering potential
outcomes $Y(d)$ matching the (f)actual treatment assignment $D=d$.

Theorems 1 and 2 in Huber and Kueck (2022) imply that we are only able to test
the respective selection-on-observables and instrument validity assumptions
for factual outcomes, e.g. for the potential outcomes $Y(1)$ of individuals
with $D=1$ and $Y(0)$ of individuals with $D=0$. Conversely, we cannot perform
tests based on counterfactual outcomes, such as $Y(0)$ for individuals with
$D=1$ and $Y(1)$ for individuals with $D=0$. Strictly speaking, we can for
this reason only test a necessary, but not a sufficient condition for
identification. Nevertheless, in practical terms, it seems unlikely that such
violations would exclusively pertain to counterfactual outcomes, but never
affect factual outcomes, because this would require highly specific modelling
constraints. Therefore, testing the (mean) conditional independence of the
instrument is likely to have the power to detect violations of the selection-
on-observables and instrument validity assumptions in typical empirical
applications.

The testing approach of Huber and Kueck (2022) requires the prior
specification of the instrument $Z$ and covariates $X$. In contrast, our
testing approach, as introduced below, does not require the predefinition of
$Z$ and $X$ when testing (2.2). Instead, it learns them from the data by
iteratively considering observed variables as instrument $Z$. This feature
appears attractive in many practical contexts where obvious instruments are
not available.

##  3 Testing based on double machine learning

We henceforth suggest a testing approach based on double machine learning
based on the following null hypothesis $H_{0}$, which is equivalent to the
conditional mean independence of the instrument $Z$ provided in condition
(2.2):

$\displaystyle H_{0}:\mathbb{E}[Y|D=d,X=x,Z=z]-E[Y|D=d,X=x]=0\quad\forall
d\in\mathcal{D},x\in\mathcal{X},z\ \in\mathcal{Z}.$ (3.1)

Under the null hypothesis $H_{0}$, the mean conditional outcome is constant
across values of the instrument $Z$ given any value of $D$ and $X$, which may
be tested for any values of $D$, $X$, and $Z$ in their respective support.
However, if one or several variables are of rich support, this implies many
testable implications - even infinitely many if one or more variables are
continuous.

For this reason, one possible testing approach is to follow Huber and Kueck
(2022) and test violations of (3.1) based on the mean squared difference
between the conditional mean outcome when including versus excluding the
instrument in the conditioning set. Denoting the conditional means by
$\mu(d,x,z)=\mathbb{E}[Y|D=d,X=x,Z=z]$ and $m(D,X)=\mathbb{E}[Y|D=d,X=x]$, one
aims at testing the following implication of equation (3.1):

$\displaystyle\mathbb{E}[\left(\mu(D,X,Z)-m(D,X)\right)^{2}]=0,$ (3.2)

based on a moment condition which uses the following Neyman (1959)-orthogonal
score function:

$\displaystyle\phi(W,\theta,\eta)=(\eta_{1}(W)-\eta_{2}(W))^{2}-\theta+\zeta.$
(3.3)

$W=(Y,D,X,Z,\zeta)$ are random variables and $\eta=(\eta_{1},\eta_{2})$ are
the so-called nuisance parameters, whose true values correspond to
$\eta_{0,1}(W)=\mu(D,X,Z)$ and $\eta_{0,2}(W)=m(D,X)$. We note that the
independent mean-zero random variable $\zeta$ in (3.3) is added to avoid a
degenerate distribution of the estimator under $H_{0}$, a common problem in
specification tests, see e.g. Hong and White (1995) and Wooldridge (1992). A
disadvantage of testing based on the score function in equation (3.3) is the
requirement to choose a random term $\zeta$, as the optimal selection of
$\zeta$ in a given dataset is generally unknown. Further, while the estimator
based on equation (3.3) is asymptotically normally distributed under the null
hypothesis, as demonstrated in Huber and Kueck (2022), this is generally not
the case under the alternative hypothesis. For this reason, we subsequently
propose a new testing approach that is based on a refined score function that
does not require user-selected random terms and entails a test statistic that
is normally distributed under both the null and alternative hypotheses.666We
note that an alternative testing approach that could be applied in our context
is suggested by Kook and Lundborg (2024)

To formally discuss the test, let us for the moment assume that the instrument
$Z$ is binary and denote by $p(D,X)=\mathbb{P}(Z=1|D,X)$ the instrument
propensity score. The score function considered in this case is given below:

$\displaystyle\quad\tilde{\psi}(W,\theta,\eta)$ (3.4)
$\displaystyle=(\mu(D,X,1)-\mu(D,X,0))^{2}$
$\displaystyle+2(\mu(D,X,1)-\mu(D,X,0))\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right)$
$\displaystyle+\mu(D,X,1)-\mu(D,X,0)+\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right)$
$\displaystyle-\theta,$

with
$\theta_{0}=\mathbb{E}[\left(\mu(D,X,1)-\mu(D,X,0)\right)^{2}]+\mathbb{E}[\mu(D,X,1)-\mu(D,X,0)]$.
Testing based on (3.4) corresponds to an aggregate $L_{2}$-type measure that
can be used to test violations across values of $D$, $X$, and $Z$, which is
common in specification tests based on nonparametric regression, see e.g.
Racine (1997), Racine et al. (2006), Hong and White (1995) and Wooldridge
(1992). In addition to the squared difference in conditional mean outcomes
that we already encountered in the previous section, the score function
notably contains a term in which the difference in conditional mean outcomes
is multiplied with a difference in expressions obtained by inverse probability
weighting (IPW) with the instrument propensity score. In fact, our new score
above combines the orthogonalized squared score in (3.3) with the popular
doubly robust score, as also considered for testing in Huber and Kueck (2022).
Just as the squared difference in conditional mean outcomes, the score
function $\tilde{\psi}$ is zero in expectations,
$\mathbb{E}[\tilde{\psi}(W,\theta_{0},\eta_{0})]=0$, under the null hypothesis
that $\theta_{0}=0$, which follows from iterated expectations, and satisfies
the Neyman orthogonality property (see Appendix A.1).

In a next step, we adapt the score function in equation (3.4) to multivalued
instruments $Z$, which is also a key contribution of this paper. In the case
of a continuous $Z$, this requires discretizing its values in some parts of
the score function. To this end, let $l=1,\dots,L$ be a partition of its
support $\mathcal{Z}$ with $\cup_{l}Z_{l}=\mathcal{Z}$. For a discrete
instrument, $Z_{l}=z_{l}$, $l=1,\dots,L$, would be any value $Z$ can take with
probability $\mathbb{P}(Z=z_{l})>c$, with $c>0$. For a continuous instrument,
such a partition may be generated based on the quantile function (e.g.
percentiles) of $Z$. Let $1(Z\in Z_{l})$ denote the indicator function, which
is one if $Z$ falls into the partition $Z_{l}$ and zero otherwise, and
$p_{l}(D,X)=\mathbb{P}(Z\in Z_{l}|D,X)$ denote the corresponding instrument
propensity score. Then, testing with a multivalued $Z$ can be based on the
following score function:

$\displaystyle\quad\psi(W,\theta,\eta)$ (3.5)
$\displaystyle=\sum_{l=1}^{L}(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin Z_{l}))^{2}$
$\displaystyle+\sum_{l=1}^{L}2(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin Z_{l}))$
$\displaystyle\left(\frac{(Y-\mu(D,X,Z\in Z_{l}))1(Z\in
Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu(D,X,Z\notin Z_{l}))1(Z\notin
Z_{l})}{1-p_{l}(D,X)}\right)$ $\displaystyle+\sum_{l=1}^{L}(\mu(D,X,Z\in
Z_{l})-\mu(D,X,Z\notin Z_{l}))$
$\displaystyle+\sum_{l=1}^{L}\frac{(Y-\mu(D,X,Z\in Z_{l}))1(Z\in
Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu(D,X,Z\notin Z_{l}))1(Z\notin
Z_{l})}{1-p_{l}(D,X)}-\theta.$

This score has a variance that is bounded away from zero, is zero in
expectation under the null hypothesis and is Neyman-orthogonal, as formally
shown in Appendix A.2. We may construct cross-fitted estimators of
$\theta_{0}$ based on the score function (3.5). It is worth noting that the
corresponding target parameter in (3.5) is given by

$\displaystyle\mathbb{E}\left[\sum_{l=1}^{L}(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))^{2}+(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\right],$

which tests the null hypothesis given in equation (3.1) for binary and
discrete instruments, and approximates equation (3.1) in the case of
continuous instruments if the bins that define $Z_{l}$ become small.

When implementing the testing approach, we assume an i.i.d. sample of size
$n$, in which $i$ denotes the index of an observation and
$W_{i}=(Y_{i},D_{i},X_{i},Z_{i})$ are the variable values of observation $i$
in that sample, with $i$ $\in$ $\\{1,2,..,n\\}$. When testing, we apply cross-
fitting as for instance discussed in Chernozhukov, Chetverikov, Demirer,
Duflo, Hansen, Newey and Robins (2018) to avoid over-fitting due to a
correlation of the estimation of the nuisance parameters and $\theta_{0}$.
Therefore, we split the data into $K$ subsamples of size $N=n/K$. The cross-
fitted estimator is given by

$\displaystyle\hat{\theta}=\frac{1}{K}\sum_{k=1}^{K}\mathbb{E}_{N,k}[\psi_{k}(W_{i},0,\hat{\eta})],$
(3.6)

with $\hat{\eta}=(\hat{\mu},\hat{m},\hat{p}_{1},\dots,\hat{p}_{L})$. Under the
regularity conditions in Assumption 6, the estimator $\hat{\theta}$ is
asymptotically normal and $\sqrt{n}$-consistent, as stated in Theorem 1. The
proof is provided in Appendix A.3.

######  Assumption 6 (Asymptotic Normality).

The following assumption needs to hold for all $n\geq 3$,
$\mathbb{P}\in\mathcal{P}$ and $q>2$: (i) $\|Y\|_{\mathbb{P},q}<C$ and
$\mathbb{E}[\varepsilon^{2}1(Z\in Z_{l})]>c$ (ii) Given a random subset $I$ of
$[n]$ of size $N=n/K$, the nuisance parameter estimator
$\hat{\eta}_{0}=\hat{\eta}_{0}((W_{i})_{i\in I^{c}})$ obeys
$\|\hat{\eta}-\eta_{0}\|_{\mathbb{P},2q}\leq C$,
$\|\hat{\eta}-\eta_{0}\|_{\mathbb{P},4}\leq\delta_{N}$, and
$\|\hat{\eta}-\eta_{0}\|_{\mathbb{P},2}\leq\delta_{N}^{1/2}N^{-1/4}$ with
$\mathbb{P}$-probability not less than $1-o(1)$.

######  Theorem 1.

Conditional on Assumptions 6, the estimator in equation (3.6) satisfies

$\displaystyle\sqrt{n}\sigma^{-1}\hat{\theta}\rightsquigarrow
N(\theta_{0},1),$ (3.7)

uniformly over $P\in\mathcal{P}$, where
$\sigma^{2}=E[\psi(W,\theta_{0},\eta_{0})^{2}]$. Moreover, the result
continues to hold if $\sigma^{2}$ is replaced by
$\hat{\sigma}^{2}:=\mathbb{E}_{n}[(\psi(W_{i},\hat{\theta},\hat{\eta}))^{2}]$.
Consequently, a test that rejects the null hypothesis $H_{0}$, $\theta_{0}=0$,
if $|\sqrt{n}\hat{\sigma}^{-1}\hat{\theta}|>\Phi^{-1}(1-\alpha/2)$ has
asymptotic level $\alpha$.

Theorem 1 states that the test statistic is normally distributed both under
the null ($H_{0}:\theta_{0}=0$) and alternative hypothesis
($H_{1}:\theta_{0}\neq 0$). The following Corollary 1 shows that the proposed
statistical test is consistent, i.e., the power converges to one if the sample
size increases.

######  Corollary 1.

Let $c_{\alpha}:=\Phi^{-1}(1-\alpha/2)$ be the critical value of the test
proposed above. Under the alternative hypothesis ($\theta_{0}\neq 0$), it
holds

$\lim\limits_{n\rightarrow\infty}\,P(|\sqrt{n}\hat{\theta}/\hat{\sigma}|>c_{\alpha})=1.$

This holds true since

$P(|\sqrt{n}\hat{\theta}/\hat{\sigma}|>c_{\alpha})=P\left(\left|\widehat{\theta}_{j}\right|\geq
c_{\alpha}\frac{\widehat{\sigma}_{j}}{\sqrt{n}}\right)\geq
P\left(\left|\widehat{\theta}_{j}-\theta_{j}\right|\leq\left|\theta_{j}\right|-c_{\alpha}\frac{\widehat{\sigma}_{j}}{\sqrt{n}}\right)=1,$

as long as $c_{\alpha}=o(\sqrt{n})$.

##  4 Selection Method

The testing approaches outlined in the previous section 3 were conditional on
having already defined the instrument $Z$ and covariates $X$ under which the
selection-on-observables assumption with respect to the treatment supposedly
holds. However, a main contribution of this study is a data-driven approach
for learning partitions of observed pre-treatment variables into instruments
and covariates. To do so, we suggest applying the testing approach iteratively
when sequentially considering one variable from the set of all pre-treatment
variables, henceforth denoted by $Q$, as instrument $Z$ and the remaining
variables as covariates $X$. More specifically, our procedure consists of the
following steps:

  1. 1.

Select candidate instruments among observed variables $Q$ which are strongly
correlated with $D$ conditional on the remaining variables. Let $\mathcal{S}$
denote the set of candidate instruments with first stage effects on treatment
$D$ that are bounded away from zero conditional on the remaining variables in
$Q$, which is to be estimated in the data based on some statistical criterion
yet to be defined.

  2. 2.

Iteratively define each variable in the estimated set of strong instruments,
denoted by $\hat{S}$, as instrument $Z$ and all remaining variables in $Q$ as
covariates $X$ when testing the hypothesis (3.2), e.g. based on the score
functions in equations (3.4) or (3.5).

  3. 3.

If testing suggests that the hypothesis (3.2) is not violated in one or
several iterations according to a criterion of statistical significance yet to
be defined, then select that candidate instrument in $\hat{S}$ for which the
test statistic is closest to a non-violation as final instrument $Z$ and the
remaining variables in $Q$ as final covariates $X$. If testing suggests that
the hypothesis (3.2) is violated in all iterations, the testable implication
(2.2) is rejected.

We note that the first step is required to select candidate instruments which,
if ultimately selected as instrument $Z$, are statistically significant in
terms of satisfying the first stage condition $E[D|X,Z]\neq E[D|X]$. Let us
assume that $Q$ is low dimensional in the sense that the sample size $n$ is
larger than the number of variables in $Q$, denoted by $p$, such that $n>p$.
In this case, step one might be implemented based on a first stage regression
of $D$ on $Q$ and selecting all regressors with statistically significant
associations after controlling for multiple hypothesis testing issues into the
set of candidate instruments $\hat{S}$.

As statistical criterion for selecting candidate instruments, we suggest
first-stage hard thresholding, as performed in the IV selection literature for
instance by Guo et al. (2018) and Masten and Poirier (2021). To formalize the
discussion, let $F_{r}$ be the first stage F-statistic in the estimation of
$E[D|X,Z]$ when defining $Z=Q_{r}$ and $X=Q_{[r]}$, with $Q_{[r]}=Q\setminus
Q_{r}$ (and $Q=X\cup Z$). We obtain the estimated set of strong candidate
instruments, denoted by $\hat{S}$, as those whose F-statistic exceeds a
certain critical value $C_{n}$:

$\displaystyle\hat{\mathcal{S}}=\\{r:F_{r}\geq C_{n}\\}$ (4.1)

Following Proposition 3 in Masten and Poirier (2021), we have that if (a)
$F_{r}\overset{d}{\rightarrow}\chi^{2}$ under the null hypothesis that
$E[D|Q]=E[D|Q_{[r]}]$, (b) $\frac{F_{r}}{n}\overset{p}{\rightarrow}\kappa_{r}$
under the alternative hypothesis that $E[D|Q]\neq E[D|Q_{[r]}]$ with
$\kappa_{r}>0$ being a constant, and (c) $C_{n}\rightarrow\infty$ and
$C_{n}=o(n)$ as $n\rightarrow\infty$, first stage hard thresholding detects
strong candidate instruments with probability going to 1,
$\lim\limits_{n\rightarrow\infty}P(\hat{S}=\mathcal{S})=1$. A value for the
significance level conventionally used in the literature is $0.1/log(n)$.

An alternative strategy is to iteratively consider each variable in $Q$ as
instrument $Z$ and any remaining variables as covariates $X$ when estimating
the first stage association $E[D|X,Z]$ based on methods for treatment or
policy evaluation like doubly robust estimation, see e.g. Robins et al.
(1994), Robins et al. (1995) and Chernozhukov et al. (2018). Such doubly
robust approaches are also applicable in high-dimensional contexts with $n<<p$
when combining them with machine learning to control for important control
variables among high-dimensional covariates $X$. For instance, the double
machine learning algorithm suggested by Soleymani et al. (2022), selects
direct causal parents in a data-driven manner by iteratively changing the
definition of variables whose causal effect is of interest and the control
variables. It may be applied to select the set of candidate instruments $S$
which are those variables in $Q$ strongly associated with $D$ conditional on
the remaining elements in $Q$. Second, we proceed to test the null hypothesis
in equation (3.2) iteratively over all candidate instruments that pass the
first-stage threshold, $\hat{\mathcal{S}}$.

Our aim is to find a partition of variables for which the conditional
independence of the respective candidate instrument holds. To discuss this
more formally, we introduce the partition

$\mathcal{P}_{j}=\\{Z=Q_{j},\ \ X=Q_{[j]}\\},$

such that variable $j$ in set $Q$ is chosen to be the instrument, while the
remaining variables in set $Q$ are used as control variables. Furthermore, let
$\mathcal{V}$ denote the set of candidate instruments which are conditionally
mean independent of the outcome, satisfying

$\mathcal{V}=\\{j:E[Y|D,X]=E[Y|D,X,Z]\\}.$

Furthermore, we denote the set of partitions for which the instrument has a
first stage effect on the treatment and the conditional independence of the
instrument holds by $\mathcal{P}^{*}$:

$\mathcal{P}^{*}=\\{\mathcal{P}_{j}:j\in\left(\mathcal{S}\cap\mathcal{V}\right)\\}.$
(4.2)

The corresponding estimated set of partition(s) $\mathcal{P}^{*}$, denoted by
$\hat{\mathcal{P}}_{pass}$, is given by

$\hat{\mathcal{P}}_{pass}=\left\\{\mathcal{P}_{j}:j\in\left(\hat{\mathcal{S}}\cap\hat{\mathcal{V}}\right)\right\\},$
(4.3)

where
$\hat{\mathcal{V}}=\\{j:|\sqrt{n}\hat{\sigma}^{-1}_{j}\hat{\theta}_{j}|<c_{\alpha}\\}$
is the set of instruments for which conditional independence is not rejected
based on the test defined in Theorem 1 and Corollary 1 with significance level
$\alpha$ and the corresponding critical value $c_{\alpha}$.

If $|\hat{\mathcal{P}}_{pass}|>1$ such that there is more than one single
candidate instrument for which the first stage is sufficiently strong while
conditional independence is not rejected, we can proceed in two ways. For a
set of partitions
$\mathcal{P}_{pass}=\\{\mathcal{P}_{1},...,\mathcal{P}_{|\mathcal{P}_{pass}|},\\}$
where each partition is associated with a specific IV, we define the set of
IVs associated with that set of partitions as

$\mathcal{I}=\left\\{j:\mathcal{P}_{j}\in\mathcal{P}_{pass}\right\\},$ (4.4)

we can then select the final partition into IVs and controls as

$\mathcal{P}_{all}=\\{Z=Q_{\mathcal{I}},\ \ X=Q\setminus Q_{\mathcal{I}}\\}.$
(4.5)

Alternatively, we can select the final partition as the one which maximizes
the p-value when testing conditional independence:

$\hat{\mathcal{P}}_{pmax}=\underset{\mathcal{P}\in\hat{\mathcal{P}}_{pass}}{argmax}\
\ p(\hat{\mathcal{P}}_{pass}),$ (4.6)

where $p(\mathcal{P})$ is the $p$-value obtained in the conditional
independence test of the instrument and the outcome. Even more crucial than
the question how to choose the final partition if
$|\hat{\mathcal{P}}_{pass}|>1$ is the question whether
$\hat{\mathcal{P}}_{pass}=\emptyset$ or not.

The main contribution of this paper is the development of a new statistical
procedure which tests the identification of a causal effect in a data-driven
way. To this end, we consider the following hypotheses:

$\displaystyle H_{0}:\text{no identification}\quad v.s\quad
H_{1}:\text{identification}\quad(Y(d){\perp\\!\\!\\!\perp}D|X\quad\forall
d\in\mathcal{D}).$ (4.7)

We conclude that $H_{1}$ is true, i.e, our causal effect of interest is
identified if $\hat{\mathcal{P}}_{pass}\neq\emptyset$. The following Theorem 2
helps us to understand the type 1 error of our proposed test.

######  Theorem 2.

Under the assumptions of Theorem 1, assuming
$\lim\limits_{n\rightarrow\infty}P(\hat{S}=\mathcal{S})=1$, for a given
$\alpha$, it holds

$\lim\limits_{n\rightarrow\infty}P(\hat{\mathcal{P}}_{pass}\subseteq\mathcal{P}^{*})=1.$

###### Proof.

Since $\lim\limits_{n\rightarrow\infty}P(\hat{S}=\mathcal{S})=1$ we just have
to show that

$\lim\limits_{n\rightarrow\infty}P(\hat{V}\subseteq\mathcal{V})=1.$

If $\hat{\mathcal{P}}_{pass}=\emptyset$, our statement holds trivially. Hence,
consider a variable $j\in\hat{\mathcal{P}}_{pass}$, i.e,
$Z_{j}\in\hat{\mathcal{S}}$ and $Z_{j}\in\hat{\mathcal{V}}$. Therefore, it
holds

$|\sqrt{n}\hat{\sigma}^{-1}_{j}\hat{\theta}_{j}|<c_{\alpha}$

by definition of $\hat{\mathcal{V}}$. By Corollary 1, we have that

$\lim\limits_{n\rightarrow\infty}\,P(|\sqrt{n}\hat{\sigma}^{-1}_{j}\hat{\theta}_{j}|>c_{\alpha})=1$

if $j\notin\mathcal{V}$. This shows that $j\in\hat{\mathcal{V}}$ implies
$j\in{\mathcal{V}}$ which concludes the proof. ∎

Theorem 2 states that
$\lim\limits_{n\rightarrow\infty}P(\mathcal{V}\neq\emptyset)=1$ if
$\hat{\mathcal{V}}\neq\emptyset$. Hence, the type 1 error of our proposed
identification test in (4.7) goes to zero if sample size increases. This means
that if our test finds identification
($\hat{\mathcal{P}}_{pass}\neq\emptyset$), there is identification with
probability $1$ for large sample sizes. The next theorem provides us insights
about the type 2 error of our test, i.e., how likely it is that we can find
identification if there is identification. Theorem 3 states that the type 2
error is at least bounded by $\alpha$.

######  Theorem 3.

Assume that $\mathcal{P}^{*}\neq\emptyset$ (testable identification). Under
the assumptions of Theorem 1, assuming
$\lim\limits_{n\rightarrow\infty}P(\hat{S}=\mathcal{S})=1$, for a given
$\alpha$, it holds

$\lim\limits_{n\rightarrow\infty}P(\hat{\mathcal{P}}_{pass}\neq\emptyset)\geq
1-\alpha.$

###### Proof.

Consider any variable $Z_{j}\in\mathcal{P}^{*}$. We know that
$E[Y|D,X]=E[Y|D,X,Z_{j}]$ and hence by Theorem 1,

$P(\underbrace{|\sqrt{n}\hat{\sigma}_{j}^{-1}\hat{\theta}_{j}|>c_{\alpha}}_{:=A_{j}})=\alpha$

if $n\rightarrow\infty$. We can conclude that

$\displaystyle\lim\limits_{n\rightarrow\infty}P(\hat{\mathcal{P}}_{pass}\neq\emptyset)$
$\displaystyle=1-\lim\limits_{n\rightarrow\infty}P(\hat{\mathcal{P}}_{pass}=\emptyset)$
$\displaystyle=1-\lim\limits_{n\rightarrow\infty}P(\cap_{j=1,\dots,M}A_{j})\geq
1-\alpha$

with $M:=|\mathcal{P}^{*}|>0$. Understanding the stochastic dependence
structure of the events $A_{j}$, $j=1,\dots,M$, could obviously lead to
sharper bounds. ∎

Algorithm 1 describes the steps of our method by means of pseudo-code.

Algorithm 1 Testing procedure for selecting partition $\mathcal{P}^{*}$

1:procedure Testing

2: Inputs: $Y\leftarrow Outcome$ $D\leftarrow Treatment$,
$Q\leftarrow\textit{Potential controls and instruments}$

3:  _Select strong instruments $\hat{\mathcal{S}}$_:

4: Regress: D $\leftarrow$ Q

5: $\hat{F}_{j}\leftarrow\hat{t}_{j}^{2}$

6: if $\hat{F}_{j}>C_{0.1/log(n)}$ then $j\in\hat{\mathcal{S}}$

7:  _Select instruments passing conditional independence test
$\hat{\mathcal{V}}$_:

8: for $j\in\hat{\mathcal{S}}$ do

9: $\hat{\mu}(D,X,Z)\leftarrow$ and $\hat{m}(D,X)\leftarrow$ and
$\hat{p}(D,X)\leftarrow$ estimates from machine learning

10: $\hat{\theta}\leftarrow$ constructed via score in equation (3.4) (binary
case) or (3.5) (continuous case)

11: $\hat{t}_{ind,j}\leftarrow$ test $H_{0}:\hat{\theta}=0$

12: if $\hat{t}_{ind,j}<t_{crit}$ then $j\in\hat{\mathcal{V}}$

13:  _Select final partition $\hat{\mathcal{P}}$_:

14:
$\hat{\mathcal{P}}_{pass}\leftarrow\\{j\in\hat{\mathcal{S}},j\in\hat{\mathcal{V}}\\}$

15: if $\hat{\mathcal{P}}_{pass}=\emptyset$ then end and report that
identification is “rejected”

16: else if $|\hat{\mathcal{P}}_{pass}|=1$ then
$\hat{\mathcal{P}}\leftarrow\hat{\mathcal{P}}_{pass}$

17: else if $|\hat{\mathcal{P}}_{pass}|>1$ then

18: $p(\hat{\mathcal{P}}_{pass})\leftarrow\text{p-value}(\hat{t}_{ind,j})$ for
$j\in\hat{\mathcal{V}}$

19:
$\hat{\mathcal{P}}=\underset{\mathcal{P}\in\hat{\mathcal{P}}_{pass}}{argmax}\
\ p(\mathcal{P})$

20:  _In case identification is confirmed (
$\hat{\mathcal{P}}_{pass}\neq\emptyset$), estimate the treatment effect_:

21: Regress: $Y\leftarrow D$ and $X=Q_{[j]}\in\hat{\mathcal{P}}$

##  5 Simulation study

This section provides a simulation study to investigate the finite sample
behavior of our testing approach based on the following data generating
process:

$\displaystyle Y$ $\displaystyle=$ $\displaystyle D+X^{\prime}\beta+\gamma
Z+W+U,$ $\displaystyle D$ $\displaystyle=$ $\displaystyle
I\\{X^{\prime}\beta+Z+\delta W+V>0\\},$ $\displaystyle X,Z$
$\displaystyle\sim$ $\displaystyle binomial(\pi),$ $\displaystyle W$
$\displaystyle\sim$ $\displaystyle N(0,1),U\sim N(0,1),V\sim N(0,1),$

with $X,Z,W,U,V$ being independent of each other. Outcome $Y$ is a linear
function of $D$ (whose treatment effect is one), covariates $X$ (for
$\beta\neq 0$), the unobservables $W$ and $U$, and the supposed instrument $Z$
if the coefficient $\gamma\neq 0$. The binary treatment $D$ is a function of
$X$ and the unobservable $V$, as well as $W$ if coefficient $\delta\neq 0$.
While the supposed instrument $Z$ is binary, the unobserved terms $U,V,W$ are
random, standard normally distributed variables that are independent of each
other, of $Z$, and of $X$. Covariates $X$ are created as follows. We first
draw a matrix of multivariate normal variables $\tilde{X}\sim
N(\mathbf{0},\bm{\Sigma})$ where $\bm{\Sigma}$ is a covariance matrix with
$Cov(X_{j},X_{k})=0.5^{|j-k|}$. We then compute probabilities
$\pi_{i}=\frac{1}{1+exp(-\tilde{X}_{i})}$ and draw $X_{i}\sim
Bernoulli(\pi_{i})$. $\beta$ gauges the effects of the covariates on $Y$ and
$D$, respectively, and thus, the magnitude of confounding due to observables.
The $j$th element in the coefficient vector $\beta$ is set to $0.8/j$ for
$j=1,...,4$, implying a linear decay of covariate importance in terms of
confounding for the first 4 covariates. We use the final element, $j=20$ as
the instrument $Z$. This results in 20 total covariates, with the first 4 as
confounders, and the last as the instrument $Z$.

We investigate the performance of our testing approach in $100$ simulations
with sample sizes of $n=1000$, $4000$, and $16000$, setting the number of
covariates to $20$. Testing is based on the cross-fitted estimator
$\hat{\theta}$ of equation (3.6) with two folds ($K=2$) when using the doubly
robust score function in equation (3.4). For nuisance parameter estimation,
lasso regression with default parameters as implemented in the glmnet package
by Friedman et al. (2010) for the statistical software R is applied. We
consider several statistics from our simulations: the estimated violation,
$\hat{\theta}$, when using variable $Z$ as the instrument (est), the standard
deviation of these estimates (std), and the average of the standard errors
across all samples (mean se). These statistics are useful for judging the
performance of the test conditional on selecting $Z$ as the instrument, which
is the correct choice if $\gamma=0$, while there is no valid instrument if
$\gamma\neq 0$.

However, since the choice of the instrument is not fixed a priori but is part
of the estimation process, we also consider the following two performance
measures. The first one is the empirical detection rate of $Z$ (det.Z), which
indicates the frequency with which variable $Z$ is selected as the best
candidate instrument and has a p-value greater than 10% when testing
conditional independence based on the estimate $\hat{\theta}$. In other words,
this corresponds to the proportion of simulations in which our method
simultaneously selects $Z$ as the instrument and keeps the null hypothesis. In
scenarios where the null hypothesis holds ($\delta=\gamma=0$), this detection
rate should approach one as the sample size increases. The second measure is
an equivalent detection rate for the confounders $X_{1}$ to $X_{4}$ (det.X).
It indicates the frequency with which one of the confounders is selected as
the best candidate instrument and yields a p-value greater than 10% when
testing conditional independence using this confounder as the instrument. As
the sample size increases, this detection rate should always approach zero,
because confounders are never valid instruments.

Table 1: Simulations: Binary Instrument

N | est | std | mean se | det.Z | det.X
---|---|---|---|---|---
| Assumptions 4 and 5 hold ($\delta=0$, $\gamma=0$)
1000 | 0.002 | 0.039 | 0.018 | 31% | 48%
4000 | -0.001 | 0.010 | 0.006 | 46% | 27%
16000 | -0.000 | 0.003 | 0.002 | 61% | 0%
| Ass. 4 violated, Ass. 5 holds ($\delta=2$, $\gamma=0$)
1000 | -0.027 | 0.029 | 0.019 | 39% | 21%
4000 | -0.027 | 0.014 | 0.010 | 8% | 21%
16000 | -0.026 | 0.006 | 0.006 | 0% | 0%
| Ass. 4 holds, Ass. 5 violated ($\delta=0$, $\gamma=0.5$)
1000 | -0.353 | 0.185 | 0.146 | 6% | 69%
4000 | -0.457 | 0.111 | 0.091 | 0% | 72%
16000 | -0.578 | 0.072 | 0.053 | 0% | 2%

Notes: columns ‘est’, ‘std’, and ‘mean se’ provide the average estimate of
$\theta$ (the violation of conditional independence) conditional on using $Z$
as instrument, its standard deviation, and the average of the standard errors
across all samples, respectively. ‘det.Z’ gives the empirical detection rate
of selecting $Z$ as instrument and obtaining a p-value $>0.10$ (or 10%) when
estimating $\theta$. This corresponds to the frequency with which $Z$ is
selected as the best candidate instrument and also entails a p-value of more
than 10% when testing conditional independence. ‘det.X’ is an equivalent
measure for confounders $X_{1}$ to $X_{4}$. It corresponds to the frequency
with which one of the confounders is selected as the best candidate instrument
and also entails a p-value of more than 10% when testing conditional
independence.

Table 1 reports the simulation results for a binary $Z$ and binary $X_{j}$.
$Z$ is correlated with $X_{j}$ according to the covariance matrix
$\bm{\Sigma}$. The top rows focus on the case where $\delta=\gamma=0$, meaning
that both Assumptions 4 and 5 are satisfied. As the sample size increases, the
test is more likely to simultaneously select the true instrument $Z$ and keep
the null hypothesis that the assumptions are satisfied. Specifically, the
detection rate of $Z$ (det.Z) increases from 31% with the smaller sample size
of 1000 observations to 61% with 16000 observations. This improvement is
mirrored by a reduction in the rate at which a confounder variable
($X_{1}...X_{4}$) is selected as the instrument and passes the test (det.X).
Under the largest sample size of 16000 observations, it never occurs that any
confounder is selected as the best instrument and passes the conditional
independence test. Additionally, we observe that, conditional on having
selected the true instrument $Z$, the estimated violation $\hat{\theta}$ (est)
is close to zero for any sample size.

The intermediate rows present the results for a violation of Assumption 4,
selection on observables, when setting $\delta=2$, $\gamma=0$. As the sample
size increases, the detection rates of $Z$ (det.Z) and the confounder
variables (det.X) both go to zero, as expected. Furthermore, the estimated
violation $\hat{\theta}$ conditional on using $Z$ as an instrument (est) is
now bounded away from zero across all samples. The lower rows of Table 1
provide the performance measures under a violation of Assumption 5, instrument
validity, when considering $\delta=0$, $\gamma=0.5$. Again and as it should be
the case, both the detection rates of $Z$ (det.Z) and the confounder variables
(det.X) tend to zero as the sample size increases. At the same time, the
estimated violation conditional on using $Z$ as an instrument (est) is
substantially biased across all samples.

Our simulations point to the consistency of our procedure in terms of
selecting the correct instrument when Assumptions 4 and 5 are satisfied, and
the rejection of the null hypothesis when violations of these assumptions
occur. However and as a word of caution for empirical applications, our
simulations also show that the sample size might need to be quite substantial
for our testing approach to perform adequately, as the finite sample
performance in our simulations is not convincing in moderate samples of just
several thousand observations.

##  6 Empirical application

This section provides an empirical application to the Job Corps (JC) training
program, one of the largest and most comprehensive job training programs for
disadvantaged youth in the US that offers various types of academic and
vocational training. The data come from the National Job Corps Study, a
randomized experiment conducted in the mid-to-late 1990s in the United States
to evaluate the effectiveness of JC on various labour market outcomes.
Schochet et al. (2001) and Schochet et al. (2008) evaluate the impact of
random program assignment on a wide range of labor market outcomes and find
positive effects on education, employment, and earnings in the longer run.

We employ our testing methodology to analyze the experimental JC data as
available in the causalweight package for the statistical software R, provided
by Bodory and Huber (2018). This dataset named JC comprises 9240 observations
and 46 variables. Among these variables is a binary indicator denoting
participation in any form of academic or vocational training within the first
year following program assignment, which serves as our treatment variable of
interest $D$. It is noteworthy that while assignment to JC is randomized,
actual participation in training might be selective due to non-compliance with
the assignment. For instance, some individuals eligible for JC may choose not
to participate. Our outcome variable are weekly earnings in the fourth year
post-JC assignment. Alongside random program assignment, which stands as an
obvious candidate instrument, our vector $Q$ encompasses 28 pre-assignment
characteristics. These include gender, age, ethnicity, education level, mother
tongue, marital status, a binary indicator for having at least one child,
previous labor market participation and earnings, household size, parents’
education, welfare receipt during childhood, as well as variables related to
health and health behavior. Testing is based on the cross-fitted estimator
$\hat{\theta}$ of equation (3.6) with two folds ($K=2$) when using the doubly
robust score functions in equations (3.4) and (3.5) in the case of binary and
continuous candidate instruments, respectively. In the case of a continuous
candidate instrument, its support is partitioned based on the quartiles of its
distribution, such that $L=4$. As in the simulations, we apply lasso
regression for nuisance parameter estimation.

When running our algorithm, it selects random program assignment as the
instrument, yielding a very high p-value of 76% when running the conditional
independence test. On the one hand, this indicates the validity of the
instrumental variable (IV), implying not only the randomness of program
assignment inherent to the experimental design. It also suggests that the
assignment does not directly affect the earnings outcome other than through
the treatment (e.g. through motivation or disappointment when being or not
being eligible for the program). On the other hand, the testing result
suggests that the selection-on-observables assumption holds for the treatment
when controlling for the pre-assignment covariates available in the data.
Consequently, given the covariates, our result suggests that we may evaluate
the average treatment effect (ATE) on the total population. In contrast,
employing an IV-based approach utilizing random assignment as the instrument
for effect estimation would, under specific additional assumptions like
treatment monotonicity in the instrument, only permit assessing the local
average treatment effect (LATE) on the subpopulation of compliers, whose
participation in training aligns with the random assignment, as discussed in
Imbens and Angrist (1994) and Angrist et al. (1996).

##  7 Conclusion

In this paper, we introduced a machine learning-based algorithm based on a
novel doubly robust score function designed to detect and test, in a data-
adaptive manner, the presence of control variables sufficient for identifying
treatment effects in observational data, as well as variables satisfying
instrument validity. The method learns the partition of observed variables
into instruments and control variables from the data, relying on the condition
that valid instruments are conditionally independent of the outcome given the
treatment and control variables that imply the conditional treatment
exogeneity. This represents an important advancement over previously suggested
tests of identifying assumptions for causal inference in observational data,
which rely on a priori assumptions about whether a variable is a control
variable or an instrument.

We demonstrated the consistency of the method for detecting control variables
and instruments (if they exist) under certain regularity conditions and
investigated its finite sample performance through a simulation study, which
supports the consistency result. However, as a word of caution for empirical
applications, the results also suggest that decent performance of the test
might require a relatively large dataset rather with more than just several
1000 observations. Finally, we applied our algorithm to empirical labor market
data from the Job Corps study. Our approach (as expected) selected random
assignment into the Job Corps program as valid instrument, while also
suggesting that training participation in the first year after assignment, our
treatment of interest, is exogenous conditional on pre-assignment covariates.

##  Appendix A Appendix

###  A.1 Proof of moment condition and Neyman orthogonality of $\tilde{\psi}$

Equation (3.4) suggests the following score function for testing when $Z$ is
binary:

$\displaystyle\quad\tilde{\psi}(W,\theta,\eta)$ (A.1)
$\displaystyle=(\mu(D,X,1)-\mu(D,X,0))^{2}$
$\displaystyle+2(\mu(D,X,1)-\mu(D,X,0))\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right)$
$\displaystyle+\mu(D,X,1)-\mu(D,X,0)+\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right)$
$\displaystyle-\theta$ (A.2)
$\displaystyle:=\tilde{\psi}_{1}(W,\theta,\eta)+\tilde{\psi}_{2}(W,\theta,\eta)-\theta$

with

$\displaystyle\quad\tilde{\psi}_{1}(W,\theta,\eta)$
$\displaystyle=(\mu(D,X,1)-\mu(D,X,0))^{2}$
$\displaystyle+2(\mu(D,X,1)-\mu(D,X,0))\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right)$

and

$\tilde{\psi}_{2}(W,\theta,\eta)=\mu(D,X,1)-\mu(D,X,0)+\left(\frac{(Y-\mu(D,X,1))\cdot
Z}{p(D,X)}-\frac{(Y-\mu(D,X,0))\cdot(1-Z)}{1-p(D,X)}\right).$

The moment condition $\mathbb{E}[\tilde{\psi}(W,\theta_{0},\eta_{0})]=0$
holds, because

$\displaystyle\mathbb{E}\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))^{2}+(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\right]-\theta=0$

and

$\displaystyle\mathbb{E}\left[\left(\frac{(Y-\mu_{0}(D,X,1))\cdot
Z}{p_{0}(D,X)}-\frac{(Y-\mu_{0}(D,X,0))\cdot(1-Z)}{1-p_{0}(D,X)}\right)\right]=0,$

as

$\displaystyle\mathbb{E}\left[\frac{(Y-\mu_{0}(D,X,1))Z}{p_{0}(D,X)}\right]$
$\displaystyle=\mathbb{E}\left[\frac{Z}{p_{0}(D,X)}\mathbb{E}\left[Y-\mu_{0}(D,X,1)|D,X,Z\right]\right]$
$\displaystyle=P(Z=1)\mathbb{E}\left[\frac{1}{p_{0}(D,X)}\left(\mathbb{E}\left[Y|D,X,Z\right]-\mu_{0}(D,X,1)\right)\bigg{|}Z=1\right]$
$\displaystyle=P(Z=1)\mathbb{E}\left[\frac{1}{p_{0}(D,X)}\left(\mathbb{E}\left[Y|D,X,Z=1\right]-\mu_{0}(D,X,1)\right)\bigg{|}Z=1\right]=0$

and analogously
$\mathbb{E}\left[\frac{(Y-\mu_{0}(D,X,0))\cdot(1-Z)}{1-p_{0}(D,X)}\right]=0$.
Furthermore, by the same argument, we have

$\displaystyle\mathbb{E}\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\left(\frac{(Y-\mu_{0}(D,X,1))\cdot
Z}{p_{0}(D,X)}-\frac{(Y-\mu_{0}(D,X,0))\cdot(1-Z)}{1-p_{0}(D,X)}\right)\right]=0.$

Neyman orthogonality of $\tilde{\psi}$ can be shown by taking the Gateaux
derivates w.r.t. the nuisance parameters:

$\displaystyle\quad\partial_{r}E[\tilde{\psi}_{1}(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0})]\big{|}_{r=0}$
$\displaystyle=E\left[\partial_{r}\tilde{\psi}_{1},(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0}))\big{|}_{r=0}\right]$
$\displaystyle=2E\left[((\mu_{0}(D,X,1)-\mu_{0}(D,X,0))((\mu(D,X,1)-\mu_{0}(D,X,1))-(\mu(D,X,0)-\mu_{0}(D,X,0)))\\}\right]$
$\displaystyle\quad-2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{Z(\mu(D,X,1)-\mu_{0}(D,X,1))}{p_{0}(D,X)}\right]$
$\displaystyle\quad+2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{(1-Z)(\mu(D,X,0)-\mu_{0}(D,X,0))}{1-p_{0}(D,X)}\right]$
$\displaystyle\quad-2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{Z(Y-\mu_{0}(D,X,1))(p(D,X)-p_{0}(D,X))}{p_{0}(D,X)^{2}}\right]$
$\displaystyle\quad-2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{(1-Z)(Y-\mu_{0}(D,X,0))(p(D,X)-p_{0}(D,X))}{(1-p_{0}(D,X))^{2}}\right]$
$\displaystyle\quad+2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{(1-Z)(Y-\mu_{0}(D,X,0))(p(D,X)-p_{0}(D,X))}{(1-p_{0}(D,X))^{2}}\right]$
$\displaystyle\quad+2E\bigg{[}((\mu(D,X,1)-\mu_{0}(D,X,1))-(\mu(D,X,0)-\mu_{0}(D,X,0)))$
$\displaystyle\quad\left(\frac{(Y-\mu_{0}(D,X,1))\cdot
Z}{p_{0}(D,X)}-\frac{(Y-\mu_{0}(D,X,0))\cdot(1-Z)}{1-p_{0}(D,X)}\right)\bigg{]}$
$\displaystyle=2E\left[((\mu_{0}(D,X,1)-\mu_{0}(D,X,0))((\mu(D,X,1)-\mu_{0}(D,X,1))-(\mu(D,X,0)-\mu_{0}(D,X,0)))\\}\right]$
$\displaystyle\quad-2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{Z(\mu(D,X,1)-\mu_{0}(D,X,1))}{p_{0}(D,X)}\right]$
$\displaystyle\quad+2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))\frac{(1-Z)(\mu(D,X,0)-\mu_{0}(D,X,0))}{1-p_{0}(D,X)}\right]$
$\displaystyle=2E\left[((\mu_{0}(D,X,1)-\mu_{0}(D,X,0))((\mu(D,X,1)-\mu_{0}(D,X,1))-(\mu(D,X,0)-\mu_{0}(D,X,0)))\\}\right]$
$\displaystyle\quad-2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))(\mu(D,X,1)-\mu_{0}(D,X,1))\mathbb{E}\left[\frac{Z}{p_{0}(D,X)}\Bigg{|}D,X\right]\right]$
$\displaystyle\quad+2E\left[(\mu_{0}(D,X,1)-\mu_{0}(D,X,0))(\mu(D,X,0)-\mu_{0}(D,X,0))\mathbb{E}\left[\frac{(1-Z)}{1-p_{0}(D,X)}\Bigg{|}D,X\right]\right]$
$\displaystyle\quad=0$

and

$\displaystyle\quad\partial_{r}\mathbb{E}[\tilde{\psi}_{2}(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0})]\big{|}_{r=0}$
$\displaystyle=\mathbb{E}\left[\partial_{r}\tilde{\psi}_{2},(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0}))\big{|}_{r=0}\right]$
$\displaystyle=\mathbb{E}\left[(\mu(D,X,1)-\mu_{0}(D,X,1))\right]-\mathbb{E}\left[(\mu(D,X,0)-\mu_{0}(D,X,0))\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{Z(\mu(D,X,1)-\mu_{0}(D,X,1))}{p_{0}(D,X)}\right]$
$\displaystyle\quad+\mathbb{E}\left[\frac{(1-Z)(\mu(D,X,0)-\mu_{0}(D,X,0))}{1-p_{0}(D,X)}\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{Z(Y-\mu_{0}(1,D,X))(p(D,X)-p_{0}(D,X))}{p_{0}(D,X)^{2}}\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{(1-Z)(Y-\mu_{0}(0,D,X))(p(D,X)-p_{0}(D,X))}{(1-p_{0}(D,X))^{2}}\right]=0$

since $\mathbb{E}[Z|D,X]=p_{0}(D,X)$, $\mathbb{E}[Z(Y-\mu_{0}(D,X,1))|D,X]=0$
and $\mathbb{E}[(1-Z)(Y-\mu_{0}(D,X,0))|D,X]=0$.

###  A.2 Proof of moment condition and Neyman orthogonality of $\psi$

Equation (3.5) suggests the following score function for testing when $Z$ is
multivalued discrete or continuous:

$\displaystyle\quad\psi(W,\theta,\eta)$ (A.3)
$\displaystyle=\sum_{l=1}^{L}(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin Z_{l}))^{2}$
$\displaystyle+\sum_{l=1}^{L}2(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin Z_{l}))$
$\displaystyle\left(\frac{(Y-\mu(D,X,Z\in Z_{l}))1(Z\in
Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu(D,X,Z\notin Z_{l}))1(Z\notin
Z_{l})}{1-p_{l}(D,X)}\right)$ $\displaystyle+\sum_{l=1}^{L}(\mu(D,X,Z\in
Z_{l})-\mu(D,X,Z\notin Z_{l}))$
$\displaystyle+\sum_{l=1}^{L}\frac{(Y-\mu(D,X,Z\in Z_{l}))1(Z\in
Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu(D,X,Z\notin Z_{l}))1(Z\notin
Z_{l})}{1-p_{l}(D,X)}-\theta$
$\displaystyle:=\psi_{1}(W,\theta,\eta)+\psi_{2}(W,\theta,\eta)-\theta.$

First, we show that the moment condition holds. By definition, we have

$\displaystyle\sum_{l=1}^{L}(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin
Z_{l}))^{2}+\sum_{l=1}^{L}(\mu(D,X,Z\in Z_{l})-\mu(D,X,Z\notin
Z_{l}))-\theta=0.$

Analogous to the proof in Appendix A.1, we have

$\displaystyle\sum_{l=1}^{L}\mathbb{E}\left[\frac{(Y-\mu_{0}(D,X,Z\in
Z_{l}))1(Z\in Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu_{0}(D,X,Z\notin
Z_{l}))1(Z\notin Z_{l})}{1-p_{l}(D,X)}\right]=0.$

Hence, we have to show that

$\displaystyle\mathbb{E}\Bigg{[}\sum_{l=1}^{L}(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))$
$\displaystyle\quad\left(\frac{(Y-\mu_{0}(D,X,Z\in Z_{l}))1(Z\in
Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu_{0}(D,X,Z\notin Z_{l}))1(Z\notin
Z_{l})}{1-p_{l}(D,X)}\right)\Bigg{]}=0,$

which holds by the same argument. Next, we show that Neyman orthogonality
holds. First, note that

$\displaystyle\quad\partial_{r}\mathbb{E}[\psi_{2}(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0})]\big{|}_{r=0}$
$\displaystyle=\mathbb{E}\left[\partial_{r}\psi_{2},(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0}))\big{|}_{r=0}\right]$
$\displaystyle=\sum_{l=1}^{L}\Bigg{(}\mathbb{E}\left[(\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))\right]-\mathbb{E}\left[(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{1(Z\in Z_{l})(\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))}{p_{l}(D,X)}\right]$
$\displaystyle\quad+\mathbb{E}\left[\frac{1(Z\notin Z_{l})(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))}{1-p_{l}(D,X)}\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{1(Z\in Z_{l})(Y-\mu_{0}(D,X,Z\in
Z_{l}))(p(D,X)-p_{l}(D,X))}{p_{l}(D,X)^{2}}\right]$
$\displaystyle\quad-\mathbb{E}\left[\frac{1(Z\notin
Z_{l})(Y-\mu_{0}(D,X,Z\notin
Z_{l}))(p(D,X)-p_{l}(D,X))}{(1-p_{l}(D,X))^{2}}\right]\Bigg{)}=0$

since

$\displaystyle\quad\sum_{l=1}^{L}\mathbb{E}\left[\frac{1(Z\in
Z_{l})(Y-\mu_{0}(D,X,Z\in Z_{l}))(p(D,X)-p_{l}(D,X))}{p_{l}(D,X)^{2}}\right]$
$\displaystyle=\sum_{l=1}^{L}\mathbb{E}\left[\frac{1(Z\notin
Z_{l})(Y-\mu_{0}(D,X,Z\notin
Z_{l}))(p(D,X)-p_{l}(D,X))}{(1-p_{l}(D,X))^{2}}\right]=0,$

by the same arguments used before,

$\displaystyle\quad\sum_{l=1}^{L}\mathbb{E}\left[\frac{1(Z\notin
Z_{l})(\mu(D,X,Z\notin Z_{l})-\mu_{0}(D,X,Z\notin
Z_{l}))}{1-p_{l}(D,X)}\right]$
$\displaystyle=\sum_{l=1}^{L}\mathbb{E}\left[\mathbb{E}\left[\frac{1(Z\notin
Z_{l})(\mu(D,X,Z\notin Z_{l})-\mu_{0}(D,X,Z\notin
Z_{l}))}{1-p_{l}(D,X)}\Bigg{|}D,X\right]\right]$
$\displaystyle=\sum_{l=1}^{L}\mathbb{E}\left[(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\mathbb{E}\left[\frac{1(Z\notin
Z_{l})}{1-p_{l}(D,X)}\Bigg{|}D,X\right]\right]$
$\displaystyle=\sum_{l=1}^{L}\mathbb{E}\left[(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\right]$

by iterated expectation and

$\displaystyle\sum_{l=1}^{L}\mathbb{E}\left[\frac{1(Z\in Z_{l})(\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))}{p_{l}(D,X)}\right]$
$\displaystyle=\sum_{l=1}^{L}\mathbb{E}\left[(\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))\right].$
$\displaystyle\quad\partial_{r}\mathbb{E}[\psi_{1}(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0}))]\big{|}_{r=0}$
$\displaystyle=\mathbb{E}\left[\partial_{r}\psi_{1}(W,\theta_{0},\eta_{0}+r(\eta-\eta_{0}))\big{|}_{r=0}\right]$
$\displaystyle=2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\left((\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))-(\mu(D,X,Z\notin Z_{l})-\mu_{0}(D,X,Z\notin
Z_{l}))\right)\right]$
$\displaystyle\quad-2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\frac{1(Z\in Z_{l})(\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))}{p_{l}(D,X)}\right]$
$\displaystyle\quad+2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\frac{1(Z\notin Z_{l})(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))}{1-p_{l}(D,X)}\right]$
$\displaystyle\quad-2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\frac{1(Z\in Z_{l})(Y-\mu_{0}(D,X,Z\in
Z_{l}))(p(D,X)-p_{l}(D,X))}{p_{l}(D,X)^{2}}\right]$
$\displaystyle\quad-2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\frac{1(Z\notin
Z_{l})(Y-\mu_{0}(D,X,Z\notin
Z_{l}))(p(D,X)-p_{l}(D,X))}{(1-p_{l}(D,X))^{2}}\right]$
$\displaystyle\quad+2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\frac{1(Z\notin
Z_{l})(Y-\mu_{0}(D,X,Z\notin
Z_{l}))(p(D,X)-p_{l}(D,X))}{(1-p_{l}(D,X))^{2}}\right]$
$\displaystyle\quad+2\sum_{l=1}^{L}\mathbb{E}\bigg{[}((\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))-(\mu(D,X,Z\notin Z_{l})-\mu_{0}(D,X,Z\notin
Z_{l})))$ $\displaystyle\quad\left(\frac{(Y-\mu_{0}(D,X,Z\in Z_{l}))\cdot
1(Z\in Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu_{0}(D,X,Z\notin Z_{l}))\cdot 1(Z\notin
Z_{l})}{1-p_{l}(D,X)}\right)\bigg{]}$
$\displaystyle=2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot\left((\mu(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\in Z_{l}))-(\mu(D,X,Z\notin Z_{l})-\mu_{0}(D,X,Z\notin
Z_{l}))\right)\right]$
$\displaystyle\quad-2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot(\mu(D,X,Z\in Z_{l})-\mu_{0}(D,X,Z\in
Z_{l}))\right]$
$\displaystyle\quad+2\sum_{l=1}^{L}\mathbb{E}\left[(\mu_{0}(D,X,Z\in
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\cdot(\mu(D,X,Z\notin
Z_{l})-\mu_{0}(D,X,Z\notin Z_{l}))\right]$ $\displaystyle=0.$

###  A.3 Proof of Theorem 1

To prove Theorem 1, we apply Theorem 3.1 in Chernozhukov et al. (2018). Hence,
we only need to show Assumption 3.1 and 3.2 in Chernozhukov et al. (2018). All
bounds in the proof hold uniformly over $\mathbb{P}\in\mathcal{P}$ but we omit
this qualifier for brevity. We use $C$ to denote a strictly positive constant
that is independent of $n$ and $\mathbb{P}\in\mathcal{P}$. The value of $C$
may change at each appearance. In Appendix A.2 we have already shown that the
moment condition and Neyman orthogonality is satisfied. Next, we note that the
score in equation (3.5) is linear, i.e.

$\psi(W,\theta,\eta)=\psi^{a}(W,\eta)\theta+\psi^{b}(W,\eta),$

with $\psi^{a}(W,\eta)=-1,$ which is in line with Assumption 3.1 in
Chernozhukov et al. (2018). Next, we demonstrate the satisfaction of
Assumption 3.2 to complete the proof. We define the following nuisance
realization set $\mathcal{T}_{n}$ as the set of all P-square-integrable
functions $\eta$ such that

$\displaystyle\|\eta_{0}-\eta\|_{P,2q}$ $\displaystyle\leq C,$
$\displaystyle\|\eta_{0}-\eta\|_{P,4}$ $\displaystyle\leq\delta_{N},$
$\displaystyle\|\eta_{0}-\eta\|_{P,2}$
$\displaystyle\leq\delta_{N}^{1/2}N^{-1/4},$

for $\delta_{N}=o(1)$ and a constant $q>2$. Note that Assumption 3.2 (a)-(c)
hold by construction of the set $\mathcal{T}_{N}$ and Assumption 6 due to same
arguments as in Huber and Kueck (2022). Assumption 3.2 (d) in Chernozhukov et
al. (2018) holds since

$\displaystyle\quad\mathbb{E}[\psi(W,\theta_{0},\eta_{0})^{2}]$
$\displaystyle\geq\mathbb{E}\left[\left(\sum_{l=1}^{L}\frac{(Y-\mu_{0}(D,X,Z\in
Z_{l}))1(Z\in Z_{l})}{p_{l}(D,X)}-\frac{(Y-\mu_{0}(D,X,Z\notin
Z_{l}))1(Z\notin Z_{l})}{1-p_{l}(D,X)}\right)^{2}\right]$ $\displaystyle>0$

by Assumption 6 and since $p_{l}(D,X)>c>0$ by construction for all
$l=1,\dots,L$. This completes the proof.

## References

  * Angrist et al. (1996) Angrist, J., Imbens, G.W., Rubin, D., 1996.  Identification of causal effects using instrumental variables.  Journal of American Statistical Association 91, 444–472 (with discussion).
  * Angrist (2004) Angrist, J.D., 2004.  Treatment effect heterogeneity in theory and practice.  The Economic Journal 114, C52–C83.
  * Angrist and Rokkanen (2015) Angrist, J.D., Rokkanen, M., 2015\.  Wanna get away? regression discontinuity estimation of exam school effects away from the cutoff.  Journal of the American Statistical Association 110, 1331–1344.
  * Apfel et al. (2022) Apfel, N., Farbmacher, H., Groh, R., Huber, M., Langen, H., 2022.  Detecting grouped local average treatment effects and selecting true instruments.  arXiv preprint arXiv:2207.04481 .
  * Apfel and Liang (2021) Apfel, N., Liang, X., 2021.  Agglomerative Hierarchical Clustering for Selecting Valid Instrumental Variables.  arXiv preprint arXiv:2101.05774 .
  * Bertanha and Imbens (2015) Bertanha, M., Imbens, G., 2015\.  External validity in fuzzy regression discontinuity designs.  NBER working paper 20773 .
  * Black et al. (2015) Black, D.A., Joo, J., LaLonde, R.J., Smith, J.A., Taylor, E.J., 2015.  Simple tests for selection bias: Learning more from instrumental variables.  IZA Discussion Paper No 9346 .
  * Bodory and Huber (2018) Bodory, H., Huber, M., 2018\.  The causalweight package for causal inference in r.  SES Working Paper 493, University of Fribourg .
  * Brinch et al. (2017) Brinch, C.N., Mogstad, M., Wiswall, M., 2017.  Beyond late with a discrete instrument.  Journal of Political Economy 125, 985 – 1039.
  * Chernozhukov et al. (2018) Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Robins, J., 2018.  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal 21, C1–C68.  doi:10.1111/ectj.12097.
  * Cox (1958) Cox, D., 1958.  Planning of Experiments.  Wiley, New York.
  * Friedman et al. (2010) Friedman, J., Tibshirani, R., Hastie, T., 2010.  Regularization paths for generalized linear models via coordinate descent.  Journal of Statistical Software 33, 1–22.  doi:10.18637/jss.v033.i01.
  * Glymour et al. (2019) Glymour, C., Zhang, K., Spirtes, P., 2019.  Review of causal discovery methods based on graphical models.  Frontiers in Genetics 10, 1–15.
  * Guo et al. (2018) Guo, Z., Kang, H., Cai, T.T., Small, D.S., 2018.  Confidence Intervals for Causal Effects with Invalid Instruments by Using Two-Stage Hard Thresholding with Voting.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) .
  * Hassanpour and Greiner (2019) Hassanpour, N., Greiner, R., 2019\.  Learning disentangled representations for counterfactual regression, in: International Conference on Learning Representations.
  * Hong and White (1995) Hong, Y., White, H., 1995.  Consistent specification testing via nonparametric series regression.  Econometrica: Journal of the Econometric Society , 1133–1159.
  * Huber (2013) Huber, M., 2013.  A simple test for the ignorability of non-compliance in experiments.  Economics Letters 120, 389–391.
  * Huber (2023) Huber, M., 2023.  Causal analysis: Impact evaluation and Causal Machine Learning with applications in R.  MIT Press.
  * Huber and Kueck (2022) Huber, M., Kueck, J., 2022.  Testing the identification of causal effects in data.  arXiv preprint 2203.15890 .
  * Imbens (2004) Imbens, G.W., 2004.  Nonparametric estimation of average treatment effects under exogeneity: a review.  The Review of Economics and Statistics 86, 4–29.
  * Imbens and Angrist (1994) Imbens, G.W., Angrist, J., 1994\.  Identification and estimation of local average treatment effects.  Econometrica 62, 467–475.
  * Imbens and Wooldridge (2009) Imbens, G.W., Wooldridge, J.M., 2009\.  Recent developments in the econometrics of program evaluation.  Journal of Economic Literature 47, 5–86.
  * Kalisch and Bühlmann (2014) Kalisch, M., Bühlmann, P., 2014\.  Causal structure learning and inference: A selective review.  Quality Technology & Quantitative Management 11, 3–21.
  * Kook and Lundborg (2024) Kook, L., Lundborg, A.R., 2024\.  Algorithm-agnostic significance testing in supervised learning with multimodal data.  arXiv preprint 2402.14416 .
  * de Luna and Johansson (2014) de Luna, X., Johansson, P., 2014\.  Testing for the unconfoundedness assumption using an instrumental assumption.  Journal of Causal Inference 2, 187–199.
  * Mani et al. (2012) Mani, S., Spirtes, P.L., Cooper, G.F., 2012.  A theoretical study of y structures for causal discovery.  arXiv preprint 1206.6853 .
  * Masten and Poirier (2021) Masten, M.A., Poirier, A., 2021\.  Salvaging falsified instrumental variable models.  Econometrica 89, 1449–1469.
  * Neyman (1923) Neyman, J., 1923.  On the application of probability theory to agricultural experiments. essay on principles.  Statistical Science Reprint, 5, 463–480.
  * Neyman (1959) Neyman, J., 1959.  Optimal asymptotic tests of composite statistical hypotheses. Wiley.  pp. 416–444.
  * Parikh et al. (2024) Parikh, H., Morucci, M., Orlandi, V., Roy, S., Rudin, C., Volfovsky, A., 2024\.  A double machine learning approach to combining experimental and observational data.  arXiv preprint 2307.01449 .
  * Pearl (1988) Pearl, J., 1988.  Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.  Morgan Kaufmann.
  * Pearl (2000) Pearl, J., 2000.  Causality: Models, Reasoning, and Inference.  Cambridge University Press, Cambridge.
  * Peters et al. (2015) Peters, J., BÃ¼hlmann, P., Meinshausen, N., 2015.  Causal inference using invariant prediction: identification and confidence intervals.  arXiv preprint 1501.01332 .
  * Peters et al. (2017) Peters, J., Janzing, D., Schölkopf, B., 2017.  Elements of causal inference: foundations and learning algorithms.  The MIT Press.
  * Quinzan et al. (2023) Quinzan, F., Soleymani, A., Jaillet, P., Rojas, C.R., Bauer, S., 2023.  Drcfs: Doubly robust causal feature selection, in: International Conference on Machine Learning, PMLR. pp. 28468–28491.
  * Racine (1997) Racine, J., 1997.  Consistent significance testing for nonparametric regression.  Journal of Business & Economic Statistics 15, 369–378.
  * Racine et al. (2006) Racine, J.S., Hart, J., Li, Q., 2006.  Testing the significance of categorical predictor variables in nonparametric regression models.  Econometric Reviews 25, 523–544.
  * Robins et al. (1994) Robins, J.M., Rotnitzky, A., Zhao, L., 1994.  Estimation of regression coefficients when some regressors are not always observed.  Journal of the American Statistical Association 90, 846–866.
  * Robins et al. (1995) Robins, J.M., Rotnitzky, A., Zhao, L., 1995.  Analysis of semiparametric regression models for repeated outcomes in the presence of missing data.  Journal of the American Statistical Association 90, 106–121.
  * Rubin (1980) Rubin, D., 1980.  Comment on ’randomization analysis of experimental data: The fisher randomization test’ by d. basu.  Journal of American Statistical Association 75, 591–593.
  * Rubin (1974) Rubin, D.B., 1974.  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of Educational Psychology 66, 688–701.
  * Schochet et al. (2001) Schochet, P.Z., Burghardt, J., Glazerman, S., 2001.  National job corps study: The impacts of job corps on participants’ employment and related outcomes.  Report (Washington, DC: Mathematica Policy Research, Inc.) .
  * Schochet et al. (2008) Schochet, P.Z., Burghardt, J., McConnell, S., 2008.  Does job corps work? impact findings from the national job corps study.  The American Economic Review 98, 1864–1886.
  * Sevilla and Mayn (2021) Sevilla, J., Mayn, A., 2021\.  A conditional independence test for causality in econometrics.  arXiv preprint 2107.09765 .
  * Soleymani et al. (2022) Soleymani, A., Raj, A., Bauer, S., Schölkopf, B., Besserve, M., 2022.  Causal feature selection via orthogonal search.  Transactions on Machine Learning Research .
  * Windmeijer et al. (2019) Windmeijer, F., Farbmacher, H., Davies, N., Smith, G.D., 2019\.  On the Use of the Lasso for Instrumental Variables Estimation with Some Invalid Instruments.  Journal of the American Statistical Association 114, 1339–1350.
  * Windmeijer et al. (2021) Windmeijer, F., Liang, X., Hartwig, F., Bowden, J., 2021\.  The Confidence Interval Method for Selecting Valid Instrumental Variables.  Journal of the Royal Statistical Society: Series B Forthcoming.
  * Wooldridge (1992) Wooldridge, J.M., 1992.  A test for functional form against nonparametric alternatives.  Econometric Theory 8, 452–475.
  * Wu et al. (2021) Wu, A., Kuang, K., Yuan, J., Li, B., Wu, R., Zhu, Q., Zhuang, Y., Wu, F., 2021.  Learning decomposed representation for counterfactual inference.  arXiv preprint arXiv:2006.07040 .