# A Brief Introduction to
Causal Inference in Machine Learning

Kyunghyun Cho
New York University & Genentech

(2024)

PREFACE

This is a lecture note produced for DS-GA 3001.003 “Special Topics in DS -
Causal Inference in Machine Learning” at the Center for Data Science, New York
University in Spring, 2024. This course was created to target master’s and PhD
level students with basic background in machine learning but who were not
exposed to causal inference or causal reasoning in general previously. In
particular, this course focuses on introducing such students to expand their
view and knowledge of machine learning to incorporate causal reasoning, as
this aspect is at the core of so-called out-of-distribution generalization (or
lack thereof.)

This lecture note does not follow a traditional curriculum for teaching causal
inference. This lecture note does not subscribe solely to either the potential
outcome framework or the do-calculus framework, but is rather flexible in
taking concepts and ideas from these two camps (which after all do look more
or less the same) in order to build up the foundation of causal inference from
the first principles. In doing so, the first half of this note covers a
variety of basic topics, including probabilistic graphical models, structural
causal models, causal quantities of interest, conditional vs. interventional
probabilities, regression, randomized controlled trials, bandit algorithms,
inverse probability weighting, matching and instrumental variables. I do not
go too deep into each of these topics, although the emphasis is given to how
these topics are all connected with each other (and sometimes are equivalent.)
For this first half of the course, I read and consulted the following books
(lightly only, though) and recommend students go deeper into these books if
they are interested in learning more about causal inference:

  1. 1.

Pearl. Causality. 2nd eds. 2009. (Pearl, 2009)

  2. 2.

Imbens & Rubin. Causal inference in statistics, social, and biomedical
sciences. 2015. (Imbens and Rubin, 2015)

  3. 3.

Cunningham. Causal Inference: the Mixtape. 2021. (Cunningham, 2021)

Based on the foundation built in the first half (or more like two thirds) of
the course, the course takes a turn toward generalization in machine learning.
In particular, I try to argue that the probabilistic graphical model based
framework from causal inference can be an invaluable tool for specifying and
understanding so-called out-of-distribution generalization. I draw (coarse)
connections from causal inference to the following ideas in machine learning,
to demonstrate this point:

  1. 1.

Distributional shifts

  2. 2.

The principle of invariance

  3. 3.

Preference-based learning for language models

To be very honest, this is a very thin lecture note for a course with a very
thin content. This note should be considered as the very first sign post at
the entrance to a huge forest called causality, and nothing more. If you want
to expand slightly a bit more, see this short introductory material I have
written together with my PhD student, Jiwoong Daniel Im (Im and Cho, 2023).

Finally, I am infinitely grateful to Daniel Im, Divyam Madaan and Taro Makino
for helping me as amazing teaching assistants in preparing the lecture note as
well as giving the lab sessions in Spring 2024. The lab materials they have
prepared are all available at https://github.com/kyunghyuncho/2024-causal-
inference-machine-learning.

###### Contents

  1. 1 Probabilistic Graphical Models
    1. 1.1 Probababilistic Graphical Models
    2. 1.2 Structural Causal Models
    3. 1.3 Learning and a generative process
    4. 1.4 Latent variable models are not necessarily causal models
    5. 1.5 Summary
  2. 2 A Basic Setup
    1. 2.1 Correlation, Independence and Causation
    2. 2.2 Confounders, Colliders and Mediators
    3. 2.3 Dependence and Causation
    4. 2.4 Causal Effects
    5. 2.5 Case Studies
    6. 2.6 Summary
  3. 3 Active Causal Inference
    1. 3.1 Causal Quantities of Interest
    2. 3.2 Regression: Causal Inference can be Trivial
    3. 3.3 Randomized Controlled Trials
      1. 3.3.1 The Basic Foundation
      2. 3.3.2 Important Considerations
    4. 3.4 Causal Inference vs. Outcome Maximization
    5. 3.5 When Some Confounders are Observed
    6. 3.6 Summary
  4. 4 Passive Causal Inference
    1. 4.1 Challenges in Randomized Controlled Trials
    2. 4.2 When Confounders were also Collected
      1. 4.2.1 Inverse Probability Weighting
      2. 4.2.2 Matching.
    3. 4.3 Instrumental Variables: When Confounders were not Collected
    4. 4.4 Summary
  5. 5 Causality and Machine Learning
    1. 5.1 Out-of-Distribution Generalization
      1. 5.1.1 Setup: I.I.D.
      2. 5.1.2 Out-of-Distribution Generalization
      3. 5.1.3 Case Studies
    2. 5.2 Invariance: Stable Correlations are Causal Correlations
      1. 5.2.1 An Environment as a Collider
      2. 5.2.2 The Principle of Invariance
    3. 5.3 Prediction vs. Causal Inference
    4. 5.4 A Case Study: Language Modeling with Pairwise Preference
    5. 5.5 Summary
  6. 6 Remaining Topics
    1. 6.1 Other Techniques in Causal Inference
      1. 6.1.1 Difference-in-Difference
      2. 6.1.2 Regression Discontinuity
      3. 6.1.3 Double Machine Learning
    2. 6.2 Behaviour Cloning from Multiple Expert Policies Requires a World Model
    3. 6.3 Summary

##  Chapter 1 Probabilistic Graphical Models

The goal of causal inference is to figure out a causal relationship, or the
lack thereof, between two sets of variables; causes and outcomes. It is thus
only natural to think of how we determine whether any particular variable is a
cause or an outcome. It is often relatively more straightforward to determine
what an outcome variable is, as this determination is done based on our
subjective interest. For instance, a typical outcome variable in medicine is a
disease-free survival rate within some years since diagnosis or treatment.
This choice is natural, since this variable, or its quantity, is what we want
to maximize. It is however much less clear how to determine which variable
should be considered a cause. For instance, in the classical example of
‘smoking causes lung cancer’, what makes us choose ‘whether someone smokes
ciagarettes’ as a cause variable rather than ‘a mutation in a particular
gene’? It becomse even more mind-boggling once we realize that this choice of
‘smoking’ as a cause meant that we decided to ignore many variables, such as
‘whether a farmer decided to grow tobacco’.

It is thus an important, if not the most important, job for practitioners of
causal inference to convincingly argue why some variables are included and
others were omitted. They also must argue why some of the included variables
are considered potential causes and why they chose a particular variable as an
outcome. This process can be thought of as defining a small universe in which
causal inference must be performed. There are many different ways to define
and describe such a universe, and in this lecture note, we largely stick to
using a probabilistic graphical model, or a corresponding structural causal
model, as a way to describe each universe, which is the main topic of this
chapter.

###  1.1 Probababilistic Graphical Models

In this course, we rely on probabilistic graphical models quite extensively in
order to describe their statistical and causal relationships among random
variables. A probabilistic graphical model, which is also referred as a
Bayesian graphical model, is a directed graph $G=(V,E)$, where $V$ is a set of
vertices/nodes and $E$ is a set of directed edges. Each node $v\in V$
corresponds to a random variable, and each edge $e=(v_{s},v_{e})$ represents
the dependence of $v_{e}$ on $v_{s}$. Let us for now assume that this graph is
acyclic, that is, there is no cycle within this graph. In other words, $G$ is
a directed acyclic graph, throughout the rest of this course.

For each node $v\in V$, we define a probability distribution
$p_{v}(v|\mathrm{pa}(v))$ over this variable conditioned on all the parent
nodes

$\displaystyle\mathrm{pa}(v)=\left\\{v^{\prime}\in V|(v^{\prime},v)\in
E\right\\}.$ (1.1)

We can then write a joint distribution over all the variables as

$\displaystyle p_{V}(V)=\prod_{v\in V}p_{v}(v|\mathrm{pa}(v)),$ (1.2)

following the chain rule of probabilities. When $\mathrm{pa}(v)=\emptyset$,
i.e. $v$ does not have any incoming edges, we call the associated distribution
$p_{v}(v)$ a prior distribution.

With $P$ a set of all conditional probabilities $p_{v}$’s, we can denote any
probabilistic graphical model as a triplet $(V,E,P)$.

From this joint distribution, we can derive all kinds of conditional
distributions by marginalizing variables and applying the definition of a
conditional probability. If we are not interested in a particular node
$\bar{v}\in V$, we can marginalize out this variable by

$\displaystyle p(V\backslash\\{\bar{v}\\})=\sum_{\bar{v}}p_{V}(V).$ (1.3)

If $\bar{v}$ is a continuous random variable, we replace $\sum$ with $\int$.
We can always turn a joint probability into a conditional probability by

$\displaystyle
p(V\backslash\\{\tilde{v}\\}|\tilde{v})=\frac{p_{V}(V)}{p_{\tilde{v}}(\tilde{v})}.$
(1.4)

Using the definition of the conditional probability, we can write
marginalization in the following way:

$\displaystyle
p(V\backslash\\{\bar{v}\\})=\sum_{\bar{v}}\frac{p_{V}(V)}{p_{\tilde{v}}(\tilde{v})}p_{\tilde{v}}(\tilde{v})=\sum_{\bar{v}}p(V\backslash\\{\bar{v}\\})p_{\tilde{v}}(\tilde{v}).$
(1.5)

Marginalization corresponds to computing the weighted sum of the conditional
probability of the remaining variables according to the marginal probability
of the variable being marginalized.

Let’s assume we can sample readily from any conditional probability $p_{v}$
with $v\in V$. We can then draw a sample from the joint distribution readily
by breadth-first sweeping of all the variables. That is,

$\displaystyle\tilde{v}\sim p_{v}(v|\tilde{\mathrm{pa}}(v)),$ (1.6)

where $\tilde{\mathrm{pa}}(v)=\left\\{\tilde{v}^{\prime}\sim
p_{v^{\prime}}(v^{\prime}|\tilde{\mathrm{pa}}(v^{\prime}))|v^{\prime}\in\mathrm{pa}(v)\right\\}$.
This procedure is called ancetral sampling and is an exact, unbiased way to
sample from this joint distribution.

If we set aside efficiency, ancestral sampling is an extremely powerful tool,
as it allows us to sample from any marginal distribution as well as any
conditional distribution. In the former case, we simply discard the draws that
correspond to the uninteresting variables (that are being marginalized). In
the latter case, we only keep samples whose values, corresponding to the
conditioning variables (that are on the right hand side of $|$), are precisely
those that we want the distribution to be conditioned on. Both of these
approaches are not efficient, and it is often much better to use a more
sophisticated sampling algorithm, often based on the idea of Markov Chain
Monte Carlos (MCMC).

Any probability distribution can be expressed as a table (though, this table
may have infinitely many rows) that consists of two columns. The first column
takes the value of interest and the second column its probability (either
density or mass). The probability function $p_{v}$ above works by hashing $v$
into the row index in this table and retrieving the associated probability,
i.e. $p_{v}:\mathbb{V}\to\mathbb{R}_{+}$. This function satisfies the
following normalization property:

$\displaystyle 1=\begin{cases}\sum_{v\in\mathbb{V}}p_{v}(v),&\text{ if
}v\text{ is discrete.}\\\ \int_{v\in\mathbb{V}}p_{v}(v)\mathrm{d}v,&\text{ if
}v\text{ is continuous.}\\\ \end{cases}$ (1.7)

This view allows us to effectively turn a set of samples into the original
distribution from which these samples were drawn (of course, with some
variance.) Let
$S=\left(\tilde{v}_{1},\tilde{v}_{2},\ldots,\tilde{v}_{N}\right)$ be a multi-
set of samples drawn from an unknown distribution over a discrete variable
$v$, without loss of generality. We can then recover the original distribution
by constructing a table where each row is

$\displaystyle\left(v,\left.\sum_{n=1}^{N}\mathds{1}(\tilde{v}_{n}=v)\right/N\right),$
(1.8)

with $v\in\mathbb{V}$. This corresponds to maximum likelihood learning without
any regularization.

In this table, we can think of all these rows’ contents as the parameters of
this model we have built to approximate the underlying distribution from which
$S$ was drawn. This view will come handy later when we discuss using a deep
neural network instead of an explicit table to represent a probability
distribution.

A major downside of this explicit table based approach is that $q_{v}(v)=0$
for all $v\not\in S$. Such an extreme value (the probability of $0$) should
not be used when we estimate these probabilities from a small number of data
points, since we cannot rule out the fact that we simply did not see that
particular instance just because we did not draw enough samples. Furthermore,
this prevents us from properly defining a conditional probability
$p_{v^{\prime}}(v^{\prime}|v)$, since

$\displaystyle
p_{v^{\prime}}(v^{\prime}|v)=\frac{p_{v^{\prime},v}(v^{\prime},v)}{p_{v}(v)}.$
(1.9)

If $v$ is set such that $p(v)=0$, this conditional probability is not well
defined. We thus have to regularize maximum likelihood learning.

This probabilistic graphical model is a good way to abstract out some of the
details on how to implement individual probability distributions for studying
causal inference in machine learning, as it frees us from worrying about the
aspect of learning until it is absolutely necessary. Learning in this context
refers to inferring the underlying distribution from which data points were
drawn, and the table-based approach above is the most naive one that is not
really useful in practice. We can however for now assume that this table-based
approach works well and continue studying causal inference with already
inferred conditional distributions.

###  1.2 Structural Causal Models

Although a directed edge in the probabilistic graphical model looks like it
encodes the order in which the variables are generated, it is not necessarily
so from the perspective of probability. According to the Bayes’ rule,

$\displaystyle
p_{v}(v|v^{\prime})=\frac{p_{v^{\prime}}(v^{\prime}|v)p_{v}(v)}{p_{v^{\prime}}(v^{\prime})}.$
(1.10)

This implies that we can always flip the arrow of the edge between $v$ and
$v^{\prime}$ without altering the joint as well as conditional
probabilities.111 We will assume from here on that $p(x)>0$ for any $x$ and
$p$.

This lack of direct relevance of the direction of each edge in $G$ to the
joint probability raises a lot of confusion, when we try to use the
probabilistic graphical model in the context of causal inference and
reasoning. Instead, we may want to use a slightly different way to express the
same generative process underlying a given probability graphical model
$G=(V,E)$.

We do so by writing the process of sampling a value associated with each
variable $v\in V$ rather than its distribution, as the combination of a
deterministic function $f_{v}$ and external (exogenous) noise $\epsilon_{v}$:

$\displaystyle v\leftarrow f_{v}(\mathrm{pa}(v),\epsilon_{v}).$ (1.11)

This says that the value of $v$ is computed based on the values of its parent
nodes $\mathrm{pa}(v)$ and external noise $\epsilon_{v}$ by the deterministic
function $f_{v}$.

This way is much more explicit about the generating process than before, since
the use of the function $f$ clearly suggests that perturbing the output of the
function would not change the input to the function, although perturbing the
input to the function would change the output. For instance, you can imagine
that $f_{v}$ corresponds to pushing a book on your desk using your hand with
force of $v^{\prime}$ and that $v$ encodes the new position of the book.
$\epsilon_{v}$ can be an unknown level of friction cause by your earlier (but
forgotten) choice of your desk. Changing force $v^{\prime}$ of course affects
the new position of the book $v$ together with some changing level of
$\epsilon_{v}$, but changing the position of the book would not change the
force I have not applied yet to the book.

We call this representation of the generative process a structural causal
model. Similarly to the probabilistic graphical model above, we can represent
any structural causal model as a triplet $(V,F,U)$, where $V$ is a set of
variables, $F$ is a set of corresponding functions and $U$ is a set of
corresponding noise variables.

Any structural causal model can be turned into a probabilistic graphical model
by using the change of variables, i.e., $f_{v}(\mathrm{pa}(v),\cdot)$ and
assuming the known prior distribution over the external noise variable
$\epsilon_{v}$. Or, more simply, we can do so because we can find a
distribution over $v\sim f_{v}(\mathrm{pa}(v),\epsilon_{v})$.

We can draw samples from any given structural causal model, just like with the
probabilistic graphical models above, by ancestral sampling. Once we have
samples from the joint distribution, we can infer various conditional
distributions. Among these, the posterior distribution over the external noise
variables is of a particular interest for the purpose of counterfactual
reasoning. Let $q(U)$ be a distribution that corresponds to all samples of
$\epsilon_{v}$’s that led to a particular configuration $\hat{V}$. Then the
posterior distribution $q(v)$ can be thought of as the distribution over the
external (uncontrollable) factors that led to the particular outcome. We then
can answer the question what would have happened to a target variable $v$ had
some of the variables were set differently, by fixing the external factors to
follow $q(v)$ and the rest of the variables to the original values $\hat{V}$.
This corresponds to counterfactual reasoning (had I done something
differently, what would have happened?)

###  1.3 Learning and a generative process

Learning, in machine learning, refers to the process by which we configure a
black box predictive model to capture the predictive relationship between a
set of variables. In the simplest form, let us consider having two variables;
input $v$ and output $v^{\prime}$. $g_{\theta}$ is a predictive function that
models the relationship between $v$ and $v^{\prime}$ by mapping an instance of
$v$ to the corresponding $v^{\prime}$, i.e., $g_{\theta}(v^{\prime})$ is the
prediction of $v^{\prime}$ given $v$. $\theta$ is a collection of parameters
that a learning algorithm configures to make $g$ as predictive of $v$ given
$v^{\prime}$ as possible.

Learning starts from data which is a set of examples drawn from an unknown
data generating process $\mathcal{G}$. This data generating process can be
described using either a probabilistic graphical model or equivalently a
structural causal model. Let
$D=\left\\{(v_{1},v_{1}^{\prime}),\ldots,(v_{N},v_{N}^{\prime})\right\\}$ be
our training dataset. The goal is then to use this dataset to infer $\theta$
that would make $g$ highly predictive of $v$ given $v^{\prime}$.

There are many ways to measure how predictive $g$ is given a pair
$(v,v^{\prime})$, but we use here a log probability:

$\displaystyle r(\theta;v,v^{\prime})=\log
p(v|v^{\prime};g_{\theta}(v^{\prime})).$ (1.12)

This means that $g_{\theta}(v^{\prime})$ parameterized a conditional
distribution over $v$ given $v^{\prime}$, or equivalently,
$g_{\theta}(v^{\prime})$ outputs a conditional distribution over $v$. If this
quantity is large, it means that $v$ is highly probable under the predictive
distribution by $g$ given $v^{\prime}$.

Learning is then to solve the following optimization problem:

$\displaystyle\arg\max_{\theta}\frac{1}{N}\sum_{n=1}^{N}r(\theta;v_{n},v_{n}^{\prime}).$
(1.13)

Once learning is over, we can readily compute

$\displaystyle p(v|v^{\prime})\approx
p(v|v^{\prime};g_{\theta}(v^{\prime}))=p(v|v^{\prime};\theta).$ (1.14)

If we assume that $p(v|v^{\prime};\theta)$ is a great approximation of
$p(v|v^{\prime})$, we can use the former in place of the latter without too
much worry. In other words, learning corresponds to figuring out a conditional
distribution $p(v|v^{\prime})$ from data. This data was produced from the
underlying data generating process $\mathcal{G}$ which may have more variables
in addition to $v$ and $v^{\prime}$.

From this description of learning, it is immediately clear how this can be a
replacement of the table-based approach from earlier, and that the table-based
approach earlier was a special case of this learning-based approach, where
$\theta$ corresponded to all the entries within a table. Once we take this
learning-based approach, we can free ourselves from having to explicitly
construct a large table and can also use various regularization techniques to
avoid the issue of $0$ probability as well as benefit from generalization.

Let $\mathcal{G}=(V,E,P)$ with $v,v^{\prime}\in V$ and
$\overline{V}=V\backslash\left\\{v,v^{\prime}\right\\}$. Then,

$\displaystyle
p(v|v^{\prime})=\frac{\sum_{\overline{V}}p(\\{v,v^{\prime}\\}\cup\overline{V})}{\sum_{\\{v\\}\cup\overline{V}}p(\\{v^{\prime}\\}\cup
V^{\prime})}.$ (1.15)

That is, we marginalize out all variables in $\overline{V}$ and then divide it
by the marginal probability of $v^{\prime}$ to get the conditional probability
of $v$ given $v^{\prime}$.

On one hand, this tells us that learning allows us to recover any arbitrary
conditional distribution induced by an underlying data generating process as
long as we have data produced following the same data generating process. On
the other hand, this also tells us that there can be many different data
generating processes that result in the exactly same conditional distribution
given a pair of variables. In fact, as long as we do not use all variables
from the generating process, there will always be ambiguities that cannot be
resolved based on the learning procedure laid out above.

As an example, consider the following two structural causal models:

##### Causal Model 1.

$\displaystyle v^{\prime}\leftarrow\epsilon_{v^{\prime}}$ (1.16)
$\displaystyle v^{l}\leftarrow v^{\prime}+a+\epsilon_{v^{l}}$ (1.17)
$\displaystyle v^{r}\leftarrow v^{\prime}+b+\epsilon_{v^{r}}$ (1.18)
$\displaystyle v\leftarrow v^{l}+v^{r}+\epsilon_{v},$ (1.19)

where $\epsilon_{v^{l}}$ and $\epsilon_{v^{r}}$ both follow standard Normal
distributions ($\mathcal{N}(0,1^{2})$. $\epsilon_{v}$ also follows standard
Normal distribution.

##### Causal Model 2.

$\displaystyle v^{\prime}\leftarrow\epsilon_{v^{\prime}}$ (1.20)
$\displaystyle v^{c}\leftarrow v^{\prime}+a+b+\epsilon_{v^{c}}$ (1.21)
$\displaystyle v\leftarrow v^{c}+\epsilon_{v},$ (1.22)

where $\epsilon_{v^{c}}\sim\mathcal{N}(0,2)$.

Then, $p(v|v^{\prime})=\mathcal{N}(v;v^{\prime}+a+b,3)$ for both causal
models, although these two models are very different.

This ambiguity plays an important role in both causal inference and so-called
out-of-distribution generalization. We will study both more carefully later in
the course.

###  1.4 Latent variable models are not necessarily causal models

When we build a predictive model on a pair $(v,v^{\prime})$, there are
variables that are left out from the original data generating process. Those
unused variables may be the ones for which we simply threw away observations,
because they were not in our interest, or the ones we actually do not observe.
These unobserved variables contribute to the complexity of $p(v|v^{\prime})$
via the process of marginalization.

It is not trivial to build a predictive model, or equivalently in our context
a deep neural network, that outputs a highly complex predictive distribution,
as these complex distributions often do not have easily-parametrizable
analytical forms. In these cases, there are two commonly used approaches; (1)
autoregressive models and (2) latent variable models. The former relies on the
chain rule of probabilities and decomposes the conditional probability as a
product of coordinate-wise conditional probabilities:

$\displaystyle
p(v|v^{\prime};\theta)=\prod_{i=1}^{d}p(v_{i}|v^{\prime};\theta),$ (1.23)

where $v=[v_{1},\ldots,v_{d}]$. By assuming that each coordinate-wise
conditional probability is of a simpler form, we can still use a simple deep
neural network to approximate a highly complex joint conditional probability.
Because each conditional probability on the right hand side is parametrized
with the same set of parameters $\theta$, it is possible to build such an
autoregressive model for a variable-sized observation, that is,
$\mathrm{dim}(v)$ is not fixed a priori. This approach is behind the recently
successful and popular large-scale language models (Brown et al., 2020).

Unlike autoregressive modeling, latent variable models explicitly introduce an
unobserved variable $u$ that represents the missing portion of the underlying
data generating process. Just like the missing (unobserved) portion gave rise
to the highly complex predictive distribution via the process of
marginalization, the introduced (anonymous) latent variable $u$ does the same:

$\displaystyle
p(v|v^{\prime};\theta)=\sum_{u}p_{u}(u)p(v|v^{\prime},u;\theta).$ (1.24)

If $u$ is continuous, we replace $\sum$ with $\int$.

Because this marginalization is difficult almost always, it is natural to
resort to sampling-based approximation. Because we are often interested in
gradient-based learning, we consider sampling-based gradient approximation:

$\displaystyle\nabla\log p(v|v^{\prime};\theta)$
$\displaystyle=\sum_{u}\frac{p_{u}(u)p(v|v^{\prime},u;\theta)\nabla\log
p(v|v^{\prime},u;\theta)}{\sum_{u^{\prime}}p_{u}(u^{\prime})p(v|v^{\prime},u^{\prime};\theta)}$
(1.25) $\displaystyle=\sum_{u}p(u|v,v^{\prime};\theta)\nabla\log
p(v|v^{\prime},u;\theta)$ (1.26)
$\displaystyle\approx\frac{1}{M}\sum_{m=1}^{M}\nabla\log
p(v|v^{\prime},u^{m};\theta),$ (1.27)

where $u^{m}$ is the $m$-th posterior sample.

It is however as challenging to compute the posterior distribution
$p(u|v,v^{\prime};\theta)$ nor to sample from it. It is thus more common these
days to maximize the lower bound to $p(v|v^{\prime};\theta)$ while amortizing
approximate posterior inference into a separate neural network. This is called
a variational autoencoder (Kingma and Welling, 2013).

Despite the seemingly similarity, these latent variables are not closely
related to actual variables in the data generating process. They may or may
not. If they indeed correspond to actual variables in the data generating
process that we simply did not observe nor decided not to use data from, we
may be able to derive conditions under which we can identify these unobserved
variables and their associated distributions from partial data alone. It is
however more common and realistic to think of these latent variables as a
means to improving the expressive power of a deep neural network. In other
words, latent variables are useful even if there are truly two variables, $v$
and $v^{\prime}$, in the data generating process, since true
$p_{v}(v|v^{\prime})$ may be complicated on its own. Nevertheless, it is a
useful tool to model any complex distribution, and thus we have spent a little
bit of time discussing them.

###  1.5 Summary

In this chapter, we have established the very foundations on which we can
discuss causal inference and its relationships to machine learning in the rest
of the course:

  1. 1.

A brief discussion on the necessity of defining a universe over which causal
inference is performed;

  2. 2.

Two (equivalent) ways to define a universe, probabilistic graphical models and
structural causal models;

  3. 3.

What learning is, once the universe is defined.

Based on these, we begin our journey into causal inference in machine
learning.

##  Chapter 2 A Basic Setup

###  2.1 Correlation, Independence and Causation

Two random variables, $u$ and $v$, are independent if and only if

$\displaystyle p(u,v)=p(u)p(v).$ (2.1)

That is, the probability of $u$ taking a certain value is not affected by that
of $v$ taking another value. We say $u$ and $v$ are dependent upon each other
when the condition above does not hold.

In everyday life, we often confuse dependence with correlation, where two
variables are correlated if and only if

$\displaystyle\mathrm{cov}(u,v)=\mathbb{E}\left[(u-\mu_{u})(v-\mu_{v})\right]>0,$
(2.2)

where $\mu_{u}=\mathbb{E}[u]$ and $\mu_{v}=\mathbb{E}[v]$. When this
covariance is $0$, we say these two variables are uncorrelated.

Despite our everyday confusion, these two quantities are only related and not
equivalent. When two variables are independent, they are also uncorrelated,
but when two variables are uncorrelated, they may not be independent.
Furthermore, it turned out these two quantities are also only remotely related
to the existence/lack of causation between two variables.

You must have heard of the statement “correlation does not imply causation.”
There are two sides to this statement. First, the existence of correlation
between two variables does not imply that there exists a causal relationship
between these two variables. An extreme example of this is tautology; if the
relationship between $u$ and $v$ is identity, there is no causal relationship
but the correlation between these two is maximal.

Second, the lack of correlation does not imply the lack of causation. This is
the more important aspect of this statement. Even if there is no correlation
between two variables, there could be a causal mechanism behind these two
variables. Although it is a degenerate case, consider the following structural
causal model:

$\displaystyle a\leftarrow-u+\epsilon_{a}$ (2.3) $\displaystyle
b\leftarrow+u+\epsilon_{b}$ (2.4) $\displaystyle v\leftarrow
a+b+\epsilon_{v}.$ (2.5)

The value of $v$ is caused by $u$ via two paths; $u\to a\to v$ and $u\to b\to
v$, but these paths cancel each other. If we only observe $(u,v)$ pairs, it is
easy to see that they are uncorrelated, since $v$ is constant. We will have
more discussion later in the semester, but it is good time for you to pause
and think whether these two paths matter, since they cancel each other.

Consider as another example the following structural causal model:

$\displaystyle z\leftarrow\epsilon_{z}$ (2.6) $\displaystyle u\leftarrow
0.2z+\sqrt{1.04}\epsilon_{u}$ (2.7) $\displaystyle v\leftarrow
0.1u-0.5z+0.1\epsilon_{v},$ (2.8)

where $\epsilon_{z}$, $\epsilon_{u}$ and $\epsilon_{v}$ are all standard
Normal variables. Again, the structural causal model clearly indicates that
$u$ causally affects $v$ via $0.1u$, but the correlation between $u$ and $v$
is $0$ when we consider those two variables alone (that is, after
marginalizing out $z$).

This second observation applies equally to independence. That is, the
independence between two variables does not imply the lack of a causal
mechanism between two variables. The examples above apply here equally, as two
uncorrelated Normal variables are also independent.

This observation connects to an earlier observation that there are potentially
many data generating processes that give rise to the same conditional
distribution between two sets of variables. Here as well, the independence or
correlatedness of two variables may map to many different generating processes
that encode different causal mechanisms behind these variables. In other
words, we cannot determine the causal relationship between two variables
(without loss of generality) without predefining the underlying generating
process (in terms of either the probabilistic graphical model or equivalently
the structural causal model.)111 There are algorithms to discover an
underlying structural causal model from data, but these algorithms also
require some assumptions such as the definition of the goodness of a
structural causal model. This is necessary, since these algorithms all work by
effectively enumerating all structural causal models that can produce data
faithfully and choosing the best one among these.

In other words, we must consider both variables of interest and the associated
data generating model in order to determine whether there exists a causal
relationship between these variables and what that relationship is.

###  2.2 Confounders, Colliders and Mediators

Let us consider a simple scenario where there are only three variables; $u$,
$v$ and $w$. We are primarily interested in the relationship between the first
two variables; $u$ and $v$. We will consider various ways in which these three
variables are connected with each other and how such wiring affects the
relationship between $u$ and $v$.

##### Directly connected.

Consider the following probabilistic graphical model.

$u$$v$$w$

$w$ does not affect either $u$ nor $v$, while $u$ directly causes $v$. In this
case, the causal relationship between $u$ and $v$ is clear. If we perturb $u$,
it will affect $v$, according to the conditional distribution $p_{v}(v|u)$.
This tells us that we can ignore any node in a probabilistic graphical model
that is not connected to any variable of interest.

##### An observed confounder.

Consider the following probabilistic graphical model, where $w$ is shaded,
which indicates that $w$ is observed.

$u$$v$$w$

In this graph, the value/distribution of $u$ and that of $v$ are both
determined individually already because we have observed $w$. This corresponds
to the definition of conditional independence;

$\displaystyle p(u,v|w)=p(u|w)p(v|w).$ (2.9)

Because the edge is directed from $w$ to $u$, perturbing $u$ does not change
the observed value of $w$. The same applies to $v$ as well, since perturbing
$v$ does not affect $u$, since this path between $u$ and $v$ via $w$ is
blocked by observing $w$.

##### An unobserved confounder.

Consider the case where $w$ was not observed.

$u$$v$$w$

We first notice that in general

$\displaystyle p(u,v)=\int_{w}p(u|w)p(v|w)p(w)\mathrm{d}w\neq q(u)q(v),$
(2.10)

implying that $u$ and $v$ are not independent, unlike when $w$ was not
observed. In other words, $u$ and $v$ are not conditionally independent given
$w$. Perturbing $u$ however still does not affect $v$, since the value of $w$
is not determined by the value of $u$ according to the corresponding causal
structural model. That is, $u$ does not affect $v$ causally (and vice versa.)
This is the case where the independence and causality start to deviate from
each other; $u$ and $v$ are not independent but each is not the cause of the
other. Analogous to the former case of the observed $w$, where we say the path
$u\leftarrow w\to v$ was closed, we say that the same path is open in this
latter case.

Because of this effect $w$ has, we call it a confounder. The existence of a
confounder $w$ makes it difficult to tell whether the dependence between two
variables we see is due to a causal relationship between these variables. $w$
confounds this analysis.

##### An observed collider.

Consider the following graph where the arrows are flipped.

$u$$v$$w$

In general,

$\displaystyle p(u,v|w)=\frac{p(w|u,v)p(u)p(v)}{\int\int
p(w|u,v)p(u)p(v)\mathrm{d}u\mathrm{d}v}\neq q(u)q(v),$ (2.11)

which means that $u$ and $v$ are not independent conditioned on $w$. This is
sometimes called the explaining-away effect, because observing $w$ explains
away one of two potential causes behind $w$.

Although $u$ and $v$ are not independent in this case, there is no causal
relationship between $u$ and $v$. $w$ is where the causal effects of $u$ and
$v$ collide with each other (hence, $w$ is a collider) and does not pass along
the causal effect between $u$ and $v$. Similarly to the case of an unobserved
confounder above, this is one of those cases where independence does not imply
causation.

We say that the path $u\to w\leftarrow v$ is open.

##### An unobserved collider.

Consider the case where the collider $w$ is not observed.

$u$$v$$w$

By construction, $u$ and $v$ are independent, as

$\displaystyle p(u,v)=\int p(u)p(v)p(w|u,v)\mathrm{d}w=p(u)p(v)\int
p(w|u,v)\mathrm{d}w=p(u)p(v).$ (2.12)

Just like before, neither $u$ or $v$ is the cause of the other. This path is
closed.

##### An observed mediator.

Consider the case where there is an intermediate variable between $u$ and $v$:

$u$$w$$v$

Because

$\displaystyle p(u,v|w)=p(u)p(w|u)p(v|w)=q(u|w)q(v|w)$ (2.13)

$u$ and $v$ are independent conditioned on $w$. However, perturbing $u$ does
not affect $v$, since the value of $w$ is observed (that is, fixed.) We say
that $u\to w\to v$ is closed in this case, and independence implies the lack
of causality.

##### An unobserved mediator.

What if $w$ is not observed, as below?

$u$$w$$v$

It is then clear that $u$ and $v$ are not independent, since

$\displaystyle p(u,v)=\int p(u)p(w|u)p(v|w)\mathrm{d}w=p(u)\int
p(w|u)p(v|w)\mathrm{d}w=p(u)q(v|u).$ (2.14)

Perturbing $u$ will change the distribution/value of $w$ which will
consequently affect that of $v$, meaning that $u$ causally affects $v$. This
effect is mediated by $w$, and hence we call $w$ a mediator.

###  2.3 Dependence and Causation

We can chain the rules that were defined between three variables; $u$, $v$ and
$w$, in order to determine the dependence between two nodes, $u$ and $v$,
within any arbitrary probabilistic graphical model given a set $Z$ of observed
nodes. This procedure is called D-separation, and it tells us two things.
First, we can check whether

$\displaystyle u\perp\\!\\!\\!\perp_{Z}v.$ (2.15)

More importantly, however, we get all open paths between $u$ and $v$. These
open paths are conduits that carry statistical dependence between $u$ and $v$
regardless of whether there is a causal path between $u$ and $v$, where we
define a causal path as an open directed path between $u$ and $v$.222 Unlike a
usual path, in a direct path, the directions of all edges must agree with each
other, i.e., pointing to the same direction.

Dependencies arising from open, non-causal paths are often casually referred
to as ‘spurious correlation’ or ‘spurious dependency’. When we are performing
causal inference for the purpose of designing a causal intervention in the
future, it is imperative to dissect out these spurious correlations and
identify true causal relationship between $u$ and $v$. It is however unclear
whether we want to remove all spurious dependencies or whether we should only
remove spurious dependencies that are unstable, when it comes to prediction in
machine learning. We will discuss more about this contention later in the
course.

In this course, we do not go deeper into D-separation. We instead stick to a
simple setting where there are only three or four variables, so that we can
readily identify all open paths and determine which are causal and which
others are spurious.

###  2.4 Causal Effects

We have so far avoided defining more carefully what it means for a node $u$ to
effect another node $v$ causally. Instead, we simply said $u$ effects $v$
causally if there is a directed path from $u$ to $v$ under the underlying data
generating process. This is however unsatisfactory, as there are loopholes in
this approach. The most obvious one is that some of those directed edges may
correspond to a constant function. For instance, an extreme case is where the
structural causal model is

$\displaystyle a\leftarrow f_{a}(u,\epsilon_{a})$ (2.16) $\displaystyle
v\leftarrow f_{v}(a,\epsilon_{v}),$ (2.17)

where $f_{a}(\cdot)=0$. In this case, the edge from $u$ to $a$ is effectively
non-existent, although we wrote it as if it existed. Rather, we want to define
a causal effect of $u$ on $v$ by thinking of how perturbation on $u$
propagates over the data generating process and arrives at $v$.

More specifically, we consider forcefully setting the variable $u$ (the cause
variable) to an arbitrary value $\hat{u}$. This corresponds to replacing the
following line in the structural causal model $G=(V,F,U)$

$\displaystyle u\leftarrow f_{u}(\mathrm{pa}(u),\epsilon_{u})$ (2.18)

with

$\displaystyle u\leftarrow\hat{u}.$ (2.19)

$\hat{u}$ can be a constant or can also be a random variable as long as it is
not dependent on any other variables in the structural causal model.

Once this replacement is done, we run this modified structural causal model
$\overline{G}(\hat{u})=(\overline{V},\overline{F},U)$ in order to compute the
following expected outcome:

$\displaystyle\mathbb{E}_{U}[v_{\overline{\mathcal{G}}(\hat{u})}],$ (2.20)

where $U$ is a set of exogenous factors (e.g. noise.) If $u$ does not affect
$v$ causally, this expected outcome would not change (much) regardless of the
choice of $\hat{u}$.

As an example, assume $u$ can take either $0$ or $1$, as in treated or
placebo. We then check the expected treatment effect on the outcome $v$ by

$\displaystyle\mathbb{E}_{U}[v_{\overline{\mathcal{G}}(\hat{u}=1)}]-\mathbb{E}_{U}[v_{\overline{\mathcal{G}}(\hat{u}=0)}].$
(2.21)

We would want this quantity to be positive and large to know that the
treatment has a positive causal effect on the outcome.

This procedure of forcefully setting a variable to a particular value is
called a $\mathrm{do}$ operator. The impact of $\mathrm{do}$ is more starkly
demonstrated if we consider a probabilistic graphical model rather than a
structural causal model. Let $p_{u}(u|\mathrm{pa}(u))$ be the conditional
distribution over $u$ in a probabilistic graphical model $G=(V,E,P)$. We
construct a so-called interventional distribution as

$\displaystyle p(v|\mathrm{do}(u=\hat{u})),$ (2.22)

which states that we are now forcefully setting $u$ to $\hat{u}$ instead of
letting it be a sample drawn from the conditional distribution
$p_{u}(u|\mathrm{pa}(u))$. That is, instead of

$\displaystyle u\sim u|\mathrm{pa}(u),$ (2.23)

we do

$\displaystyle u\leftarrow\hat{u}.$ (2.24)

In other words, we replace the conditional probability
$p_{u}(u|\mathrm{pa}(u))$ with

$\displaystyle p_{u}(u|\mathrm{pa}(u))=\delta(u-\hat{u}),$ (2.25)

where $\delta$ is a Dirac measure, or replace all occurrences of $u$ in the
conditional probabilities of $\mathrm{child}(u)$ with a constant $\hat{u}$,
where

$\displaystyle\mathrm{child}(u)=\left\\{u^{\prime}\in V|(u,v^{\prime})\in
E\right\\}.$ (2.26)

As a consequence, in the new modified graph $\overline{G}$, there is no edge
coming into $u$ (or $\hat{u}$) anymore, i.e., $\mathrm{pa}(u)=\emptyset$. $u$
is now independent of all the other nodes a priori.

Because $\mathrm{do}$ modifies the underlying data generating process,
$p(v|u=\hat{u};G)$ and
$p(v|\mathrm{do}(u=\hat{u});G)=p(v|u=\hat{u};\overline{G})$ differ from each
other. This difference signifies the separation between statistical and causal
quantities. We consider this separation in some minimal cases.

###  2.5 Case Studies

##### An unobserved confounder and a direct connection.

Consider the case where $w$ was not observed.

$u$$v$$w$

Under this graph $G$,

$\displaystyle
p_{G}(v|u=\hat{u})=\frac{\sum_{w}p(w)p(\hat{u}|w)p(v|\hat{u},w)}{p(\hat{u})}=\mathbb{E}_{w}\left[\frac{p(\hat{u}|w)}{p(\hat{u})}p(v|\hat{u},w)\right],$
(2.27)

from which we see that there are two open paths between $v$ and $u$:

  1. 1.

$u\to v$: a direct path;

  2. 2.

$u\leftarrow w\to v$: an indirect path via the unobserved confounder.

The statistical dependence between333 We do not need to specify the direction
of statistical dependence, since the Bayes’ rule allows us to flip the
direction.  $u$ and $v$ flows through both of these paths, while the causal
effect of $u$ and $v$ only flows through the direct path $u\to v$. The
application of $\mathrm{do}(u=\hat{u})$ in this case would severe the edge
from $w$ to $u$, as in

$\hat{u}$$v$$w$

Under this modified graph $\overline{G}$,

$\displaystyle p_{G}(v|\mathrm{do}(u=\hat{u}))$
$\displaystyle=p_{\tilde{G}}(v|u=\hat{u})=\frac{1}{q(\hat{u})}\sum_{w}p(w)q(\hat{u})p(v|\hat{u},w)$
(2.28)
$\displaystyle=\sum_{w}p(w)p(v|\hat{u},w)=\mathbb{E}_{w}\left[p(v|\hat{u},w)\right],$
(2.29)

where we use $q(\hat{u})$ to signify that this is not the same as $p(\hat{u})$
above.

The first one $p_{G}(v|u=\hat{u})$ is a statistical quantity, and we call it a
conditional probability. The latter $p_{G}(v|\mathrm{do}(u=\hat{u}))$ is
instead a causal quantity, and we call it an interventional probability.
Comparing these two quantities, $p_{G}(v|u=\hat{u})$ and
$p_{G}(v|\mathrm{do}(u=\hat{u}))$, the main difference is the multiplicative
factor $\frac{p(\hat{u}|w)}{p(\hat{u})}$ inside the expectation. The numerator
$p(\hat{u}|w)$ tells us how likely this treatment $\hat{u}$ was given under
$w$, while the denominator $p(\hat{u})$ tells us how likely the treatment
$\hat{u}$ was given overall. In other words, we upweight the impact of
$\hat{u}$ and $w$ if $\hat{u}$ was more probable under $w$ than overall. This
observation will allow us to convert between these two quantities later.

##### An observed collider and a direct connection.

Let us flip the edges from $w$ so that those edges are directed toward $w$. We
further assume that we always observe $w$ to be a constant ($1$).

$u$$v$$w=1$

The $\mathrm{do}$ operator on $u$ does not alter the graph above. This means
that the conditional probability and interventional probability coincide with
each other in this case, conditioned on observing $w=1$.

This however does not mean that the conditional probability
$p(v|u=\hat{u},w=1)$, or equivalently the interventional probability
$p(v|\mathrm{do}(u=\hat{u}),w=1)$, measures the causal effect of $u$ on $v$
alone. As we saw before and also can see below, there are two open paths,
$u\to v$ and $u\leftarrow w\to v$, between $u$ and $v$ through which the
dependence between $u$ and $v$ flows:

$\displaystyle
p(v|u,w=1)=\frac{\cancel{p(u)}p(v|u)p(w=1|u,v)}{\sum_{v^{\prime}}\cancel{p(u)}p(v^{\prime}|u)p(w=1|u,v^{\prime})}.$
(2.30)

We must then wonder whether we can separate out these two paths from each
other. It turned out unfortunately that this is not possible in this scenario,
because we need the cases of $w\neq 1$ (e.g. $w=0$) for this separation. If
you recall how we can draw samples from a probabilistic graphical model while
conditioning some variables to take particular values, it was all about
selecting a subset of samples drawn from the same graph without any observed
variables. This selection effectively prevents us from figuring out the effect
of $u$ on $v$ via $w$. We will have more discussion on this topic later in the
semester in the context of invariant prediction.

Because of this inherent challenge, that may not even be addressable in many
cases, we will largely stick to the case of having a confounder in this
semester.

###  2.6 Summary

In this chapter, we have learned about the following topics:

  1. 1.

How to represent a data generating process: a probabilistic graphical model
vs. a structural causal model;

  2. 2.

How to read out various distributions from a data generating process:
ancestral sampling and Bayes’ rule;

  3. 3.

The effect of confounders, colliders and mediators on independence;

  4. 4.

Causal dependency vs. spurious dependency;

  5. 5.

The $\mathrm{do}$ operator.

##  Chapter 3 Active Causal Inference

In this chapter, we assume the following graph $G$. We use $a$, $y$ and $x$,
instead of $u$, $v$ and $w$, to denote the action/treatment, the outcome and
the covariate, respectively. The covariate $x$ is a confounder in this case,
and it may or may not observed, depending on the situation.

$a$$y$$x$

An example case corresponding to this graph is vaccination.

  * •

$a$: is the individual vaccinated?

  * •

$y$: has the individual been infected by the target infectious disease with
symptoms, within 6 months of vaccine administration?

  * •

$x$: the underlying health condition of the individual.

The edge $x\to a$ is understandable, since we often cannot vaccinate an
individual with an active underlying health condition. The edge $x\to y$ is
also understandable, since healthy individuals may contract the disease
without any symptoms, while immunocompromised individuals for instance may
show a greater degree of symptoms with a higher chance. The edge $a\to y$ is
also reasonable, as the vaccine must have been developed with the target
infectious disease as its goal. In other words, this graphs encodes our
structural prior about vaccination.

With this graph that encodes the reasonable data generating process, causal
inference then refers to figuring out the degree of the causal effect of $a$
on $y$.

###  3.1 Causal Quantities of Interest

In this particular case, we are interested in a number of causal quantities.
The most basic and perhaps most important one is whether the treament is
effective (i.e., results in a positive outcome) generally. This corresponds to
checking whether
$\mathbb{E}\left[y|\mathrm{do}(a=1)\right]>\mathbb{E}\left[y|\mathrm{do}(a=0)\right]$,
or equivalently computing

$\displaystyle\mathrm{ATE}=\mathbb{E}\left[y|\mathrm{do}(a=1)\right]-\mathbb{E}\left[y|\mathrm{do}(a=0)\right],$
(3.1)

where

$\displaystyle\mathbb{E}\left[y|\mathrm{do}(a=\hat{a})\right]$
$\displaystyle=\sum_{y}yp(y|\mathrm{do}(a=\hat{a}))$ (3.2)
$\displaystyle=\sum_{y}y\sum_{x}p(x)p(y|\hat{a},x)=\sum_{y}y\mathbb{E}_{x\sim
p(x)}\left[p(y|\hat{a},x)\right].$ (3.3)

In words, we average the effect of $\hat{a}$ on $y$ over the covariate
distribution but the choice of $\hat{a}$ should not depend on $x$. Then, we
use this interventional distribution $p(y|\mathrm{do}(a=\hat{a}))$ to compute
the average outcome. We then look at the difference in the average outcome
between the treatment and not (placebo), to which we refer as the average
treatment effect (ATE).

It is natural to extend ATE such that we do not marginalize the entire
covariate $x$, but fix some part to a particular value. For instance, we might
want to compute ATE but only among people in their twenties. Let us rewrite
the covariate $x$ as a concatenation of $x$ and $x^{\prime}$, where
$x^{\prime}$ is what we want to condition ATE on. That is, instead of
$p(y|\mathrm{do}(a=\hat{a}))$, we are interested in
$p(y|\mathrm{do}(a=\hat{a}),x^{\prime}=\hat{x}^{\prime})$. This corresponds to
first modifying $G$ into

$a$$y$$x$$\hat{x}^{\prime}$

and then into

$a$$y$$x$$\hat{x}^{\prime}$

We then get the following conditional average treatment effect (CATE):

$\displaystyle\mathrm{CATE}=\mathbb{E}\left[y|\mathrm{do}(a=1),x^{\prime}=\hat{x}^{\prime}\right]-\mathbb{E}\left[y|\mathrm{do}(a=0),x^{\prime}=\hat{x}^{\prime}\right],$
(3.4)

where

$\displaystyle\mathbb{E}\left[y|\mathrm{do}(a=\hat{a}),x^{\prime}=\hat{x}^{\prime}\right]$
$\displaystyle=\sum_{y}yp(y|\mathrm{do}(a=\hat{a}),x^{\prime}=\hat{x^{\prime}})$
(3.5)
$\displaystyle=\sum_{y}y\sum_{x}p(x|x^{\prime})p(y|\hat{a},x^{\prime}=\hat{x^{\prime}},x)$
(3.6) $\displaystyle=\sum_{y}y\mathbb{E}_{x\sim
p(x|x^{\prime})}\left[p(y|\hat{a},x^{\prime}=\hat{x^{\prime}},x)\right].$
(3.7)

You can see that this is really nothing but ATE conditioned on
$x^{\prime}=\hat{x}^{\prime}$.

From these two quantities of interest above, we see that the core question is
whether and how to compute the interventional probability of the outcome $y$
given the intervention on the action $a$ conditioned on the context
$x^{\prime}$. Once we can compute this quantity, we can computer various
target quantities under this distribution. We thus do not go deeper into other
widely used causal quantities in this course but largely stick to ATE/CATE.

###  3.2 Regression: Causal Inference can be Trivial

Assume for now that we are given a set of data points drawn from this graph
$G$:

$\displaystyle
D=\left\\{(a_{1},y_{1},x_{1}),\ldots,(a_{N},y_{N},x_{N})\right\\}.$ (3.8)

For every instance, we observe all of the action $a$, outcome $y$ and
covariate $x$. Furthermore, we assume all these data points were drawn from
the same fixed distribution

$\displaystyle p^{*}(a,y,x)=p^{*}(x)p^{*}(a|x)p^{*}(y|a,x)$ (3.9)

and that $N$ is large.

In this case, we can use a non-parametric estimator, such as tables, deep
neural networks and gradient boosted trees, to reverse-engineer each
individual conditional distribution from this large dataset $D$. This is just
like what we have discussed earlier in §1.3. Among three conditional
distributions above, we are only interested in learning $p^{*}(x)$ and
$p^{*}(y|a,x)$ from data, resulting $p(x;\theta)$ and $p(y|a,x;\theta)$, where
$\theta$ refers to the parameters of each deep neural network.111 Although
there is no reason to prefer deep neural networks over random forests or other
non-parametric learners, we will largely stick to deep neural networks, as I
like them more.

Once learning is over, we can use it to approximate ATE as

$\displaystyle\mathrm{ATE}$ $\displaystyle\approx\sum_{y}y\mathbb{E}_{x\sim
p(x;\theta)}\left[p(y|a=1,x;\theta)\right]-\sum_{y}y\mathbb{E}_{x\sim
p(x;\theta)}\left[p(y|a=0,x;\theta)\right]$ (3.10)
$\displaystyle=\sum_{y}y\mathbb{E}_{x\sim
p(x;\theta)}\left[p(y|a=1,x;\theta)-p(y|a=0,x;\theta)\right].$ (3.11)

There are two conditions that make this regression-based approach to causal
inference work:

  1. 1.

No unobserved confounder: we observe the covariate $x$ in $G$;

  2. 2.

Large $N$: we have enough examples to infer $p^{*}(y|a,x)$ with a low
variance.

If there is any dimension of $x$ that is not observed in the dataset, it is
impossible for any learner to infer neither $p^{*}(y|a,x)$ nor $p^{*}(x)$
correctly. “Correctly” here in particular refers to identifying the true
$p^{*}(y|a,x)$. This is not really important if the goal is to approximate the
conditional probability $p^{*}(y|a)$, since we can simply drop $x$ and use
$(a,y)$ pairs. It is however critical if the goal is to approximate the
interventional probaiblity $p^{*}(y|\mathrm{do}(a))$ because this necessitates
us to access $p^{*}(y|a,x)$ (approximately).

Large $N$ is necessary for two reasons. First, the problem may be ill-posed
when $N$ is small. Consider rewriting $p(y|a,x)$ as

$\displaystyle p(y|a,x)=\frac{p(y,a,x)}{p(a|x)p(x)}.$ (3.12)

This quantity has in the denominator both $p(a|x)$ and $p(x)$. If $N$ is small
to the point that we do not observe all possible combination of $(a,x)$ for
which $p(x)>0$, this conditional probability is not well-defined. This
connects to one of the major assumptions in causal inference, called
positivity, which we will discuss further later in the semester.

The second, perhaps less important, reason is that the variance of the
estimator is often inversely proportional to $N$. That is, with more $N$, we
can approximate $p^{*}(y|a,x)$ with less variance. The variance of this
estimate is critical, as it directly leads to that of ATE. If the variance of
ATE is high, we cannot draw a confident conclusion whether the treatment is
effective.

This section tells us that causal inference can be done trivially by
statistical regression when the following conditions are satisfied:

  1. 1.

There are no unobserved confounder: We observe every variable.

  2. 2.

Positivity: All possible combinations of $(a,y,x)$ are observed in data.

  3. 3.

We have enough data.

Unfortunately, it is rare that all these conditions are satisfied in real
life.

###  3.3 Randomized Controlled Trials

####  3.3.1 The Basic Foundation

In this section, we consider the case where there are unobserved confounders.
In such a case, we cannot rely on regression, as these unobserved confounders
prevent a learner from identifying $p^{*}(y|a,x)$ correctly. One may tempted
to simply fit a predictive model on $(a,y)$ pairs to get $p(y|a;\theta)$ to
approximate $p^{*}(y|a)$ and call it a day. We have however learned earlier
that this does not approximate the causal effect, that is,
$p^{*}(y|\mathrm{do}(a))$, due to the spurious path, $a\leftarrow x\to y$,
which is open because we did not observe $x$.

When there are unobserved confounders, we can actively collect data that
allows us to perform causal inference. This is a departure from a usual
practice in machine learning, where we often assume that data is provided to
us and the goal is for us to use a learning algorithm to build a predictive
model. This is often not enough in causal inference, and we are now presented
with the first such case, where the assumption of ‘no unobserved confounder’
has been violated.

In order to estimate the causal effect of the action $a$ on the outcome $y$,
we severed the edge from the confounder $x$ to the action $a$, as in

$a$$y$$x$

$G$

$\to$ $a$$y$$x$ $\overline{G}$

This suggests that if we collect data according to the latter graph
$\overline{G}$, we may be able to estimate the causal effect of $a$ on $y$
despite the unobserved confounder $x$. To do so, we need to decide on the
prior distribution over the action, $\overline{p}(a)$, such that $a$ is
independent of the covariate $x$. It is a common practice to choose a uniform
distribution over the action as $\overline{p}(a)$. For instance, if $a$ is
binary,

$\displaystyle\overline{p}(a=0)=\overline{p}(a=1)=0.5.$ (3.13)

The prior distribution over the covariate $x$, $p(x)$, is not what we choose,
but is what the environment is like. That is, instead of specifying $p(x)$ nor
sampling explicitly from it (which was by assumption impossible), we go out
there and recruit samples from this prior distribution. In the vaccination
example above, this would correspond to recruiting subjects from a general
population without any particular filtering or selection.222 Of course, we can
filter these subjects to satisfy a certain set of criteria (inclusion
criteria) in order to estimate a conditional average treatment effect.

For each recruited subject $x$, we assign the treatment $a$, drawn from
$\overline{p}(a)$ that is independent of $x$. This process is called
‘randomization’, because this process assigns an individual drawn from the
population, $x\sim p(x)$, randomly to either a treatment or placebo group
according to $\overline{p}(a)$, where ‘randomly’ refer to ‘without any
information’. This process is also ‘controlled’, because we control the
assignment of each individual to a group.

For the randomly assigned pair $(a,x)$, we observe the outcome $y$ by letting
the environment (nature) simulate and sample from $p^{*}(y|a,x)$. This process
is a ‘trial’, where we try the action $a$ on the subject $x$ by administering
$a$ to $x$. Putting all these together, we call this process of collecting
data from $\overline{G}$ a randomized controlled trial (RCT).

It is important to emphasize that $x$ is not recorded, fully known nor needed,
and we end up with

$\displaystyle D=\left\\{(a_{1},y_{1}),\ldots,(a_{N},y_{N})\right\\}.$ (3.14)

Using this dataset, we can now approximate the interventional probability as

$\displaystyle\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]$
$\displaystyle=\mathbb{E}_{\overline{G}}\left[y|a=\hat{a}\right]$ (3.15)
$\displaystyle=\sum_{y}y\frac{\sum_{x}p(x)\cancel{\overline{p}(\hat{a})}p(y|\hat{a},x)}{\cancel{\overline{p}(\hat{a})}}$
(3.16) $\displaystyle=\sum_{x}\sum_{y}yp(x)p(y|\hat{a},x)$ (3.17)
$\displaystyle\approx\frac{\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})y_{n}}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})},$
(3.18)

because $x_{n}\sim p(x)$, $a_{n}\sim\overline{p}(a)$ and $y_{n}\sim
p(y|\hat{a},x)$.

As evident from the final line, we do not need to know the confounder $x$,
which means that RCT avoids the issue of unobserved, or even unknown,
confounders. Furthermore, it does not involve $\overline{p}(a)$, implying that
we can use an arbitrary mechanism to randomly assign each subject to a
treatment option, as long as we do not condition it on the confounder $x$. If
we have strong confidence in the effectiveness of a newly developed vaccine,
for instance, we would choose $\overline{p}(a)$ to be skewed toward the
treatment ($a=1$).

####  3.3.2 Important Considerations

Perhaps the most important consideration that must be given when implementing
a randomized controlled trial is to ensure that the action $a$ is independent
of the covariate $x$. As soon as dependence forms between $a$ and $x$, the
estimated causal effect $\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]$
becomes biased. It is however very easy for such dependency to arise without a
careful design of an RCT, often due to subconscious biases formed by people
who implement the randomized assignment procedure. For instance, in the case
of the vaccination trial above, a doctor may subconsciously assign older
people less to the vaccination arm333 An ‘arm’ here refers to a ‘group’.
simply because she is subconsciously worried that vaccination may have more
severe side effects on an older population. Such a subconscious decision will
create a dependency between the action $a$ and the covariate $x$ (the age of a
subject in this case), which will lead to the bias in the eventual estimate of
the causal effect. In order to avoid such a subconscious bias in assignment,
it is a common practice to automate the assignment process so that the trial
administrator is not aware of to which action group each subject was assigned.

Second, we must ensure that the causal effect of the action $a$ on the outcome
$y$ must stay constant throughout the trial. More precisely, $p^{*}(y|a,x)$
must not change throughout the trial. This may sound obvious, as it is
difficult to imagine how for instance the effect of vaccination changes
rapidly over a single trial. There are however many ways in which this does
not hold true. A major way by which $p^{*}(y|a,x)$ drifts during a trial is
when a participant changes their behavior. Continuing the example of
vaccination above, let us assume that each participant knows that whether they
were vaccinated and also that the pandemic is ongoing. Once the participant
knows that they were given placebo instead of actual vaccine, they may become
more careful about their hygiene, as they are worried about potential
contracting the rampant infectious disease. Similarly, if they knew they were
given actual vaccine, they may become less careful and expose themselves to
more situations in which they could contract the disease. That is, the causal
effect of vaccination changes due to the alteration of participants’
behaviours. It is thus a common and important practice to blind participants
from knowing to which treatment groups they were assigned. For instance, in
the case of vaccination above, we would administer saline solution via
injection to control (untreated) participants so that they cannot tell whether
they are being injected actual vaccine or placebo.

Putting these two considerations together, we end up with a double blind trial
design. In a double blind trial design, neither the participant nor the trial
administrator is made aware of their action/treatment assignment. This helps
ensure that the underlying causal effect is stationary throughout the study
and that there is no bias creeping in due to the undesirable dependency of the
action/treatment assignment on the covariate (information about the
participant.)

The final consideration is not about designing an RCT but about interpreting
the conclusion from an RCT. As we saw above, the causal effect from the RCT
based on $\overline{G}$ is mathematically expressed as

$\displaystyle\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]=\sum_{x}\sum_{y}yp(x)p(y|\hat{a},x).$
(3.19)

The right hand side of this equation includes $p(x)$, meaning that the causal
effect is conditioned on the prior distribution over the covariate.

We do not have direct access to $p(x)$, but we have samples from this
distribution, in the form of participants arriving and being included into the
trial. That is, we have implicit access to $p(x)$. This implies that the
estimated causal effect would only be valid when it is used for a population
that closely follows $p(x)$. If there is any shift in the population
distribution itself or there was any filtering applied to participants in the
trial stage that does not apply after the trial, the estimated causal effect
would not be valid anymore. For instance, clinical trials, such as the
vaccination trial above, are often run by research-oriented and financially-
stable clinics which are often located in affluent neighbourhoods. The effect
of the treatment from such a trial is thus more precise for the population in
such affluent neighbourhoods. This is the reason why inclusion is important in
randomized controlled trials.

Overall, a successful RCT requires the following conditions to be met:

  1. 1.

Randomization: the action distribution must be independent of the covariate.

  2. 2.

Stationarity of the causal effect: the causal effect must be stable throughout
the trial.

  3. 3.

Stationarity of the popluation: the covariate distribution must not change
during and after the trial.

As long as these three conditions are met, RCT provides us with an opportunity
to cope with unobserved confounders.

###  3.4 Causal Inference vs. Outcome Maximization

Beside curiosity, the goal of causal inference is to use the inferred causal
relationship for better outcomes in the future. Once we estimate
$\mathbb{E}_{G}\left[y|\mathrm{do}(a)\right]$ using RCT, we would simply
choose the following action for all future subject:

$\displaystyle\hat{a}=\arg\max_{a\in\mathcal{A}}\mathbb{E}_{G}\left[y|\mathrm{do}(a)\right],$
(3.20)

where $\mathcal{A}$ is a set of all possible actions. This approach has
however one downside that we had to give an incorrect treatment (e.g. placebo)
to many trial participants who lost their opportunities to have a better
outcome (e.g. protection against the infectious disease.)

Consider an RCT where subjects arrive and are tested serially, that is, one at
a time. If $t$ subjects have participated in the RCT so far, we have

$\displaystyle D=\left\\{(a_{1},y_{1}),\ldots,(a_{t},y_{t})\right\\}.$ (3.21)

Based on $D$, we can estimate the outcome of each action by

$\displaystyle\hat{y}_{t}(a)=\frac{\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)y_{t^{\prime}}}{\sum_{t^{\prime\prime}=1}^{N}\mathds{1}(a_{t^{\prime\prime}}=a)}.$
(3.22)

This estimate would be unbiased (correct on average), if every
$a_{t^{\prime}}$ was drawn from an action distribution that is independent of
the covariate $x$ from an identical distribution $q(a)$.444 This is another
constraint on RCT, that every subject must be assigned according to the same
assignment policy $q(a)$.  More generally, the bias (the degree of
incorrectness) would be proportional to

$\displaystyle\epsilon_{\leq
t-1}=\frac{1}{t}\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}\text{ was
drawn independently of }x_{t^{\prime}}).$ (3.23)

If $\epsilon_{\leq t-1}=1$, the estimate is unbiased, corresponding to causal
inference. If $\epsilon_{\leq t-1}=0$, what we have is not interventional but
conditional.

Assuming $\epsilon_{t}\gg 0$ and $t\gg 1$, we have a reasonable causal
estimate of the outcome $y$ given each action $a$. Then, in order to maximize
the outcome of the next subject ($x_{t+1}$), we want to assign them to an
action sampled from the following Boltzmann distribution:

$\displaystyle
q_{t}(a)=\frac{\exp\left(\frac{1}{\beta_{t}}\hat{y}_{t}(a)\right)}{\sum_{a^{\prime}\in\mathcal{A}}\exp\left(\frac{1}{\beta_{t}}\hat{y}_{t}(a^{\prime})\right)},$
(3.24)

where $\beta_{t}\in[0,\infty)$ is a temperature parameter.

When $\beta_{t}\to\infty$ (a high temperature), this is equivalent to sampling
the action $a_{t+1}$ from a uniform distribution, which implies that we do not
trust the causal estimates of the outcomes, perhaps due to small $t$. On the
other hand, when $\beta_{t}\to 0$ (a low temperature), the best-outcome action
would be selected, as

$\displaystyle q_{t}(a)=_{\beta\to\infty}\begin{cases}1,&\text{ if
}\hat{y}(a)=\max_{a^{\prime}}\hat{y}_{t}(a^{\prime})\\\ 0,&\text{
otherwise}\end{cases}$ (3.25)

assuming there is a unique action that leads to the best outcome. In this
case, we are fully trusting our causal estimates of the outcomes and simply
choose the best action accordingly, which corresponds to outcome maximization.

We now combine these two in order to make a trade off between causal inference
and outcome maximization. At time $t$, we sample the action $a_{t}$ for a new
participant from

$\displaystyle
q_{t}(a)=\epsilon_{t}\frac{1}{|\mathcal{A}|}+(1-\epsilon_{t})\frac{\exp\left(\frac{1}{\beta_{t}}\hat{y}_{t}(a)\right)}{\sum_{a^{\prime}\in\mathcal{A}}\exp\left(\frac{1}{\beta_{t}}\hat{y}_{t}(a^{\prime})\right)},$
(3.26)

where $\epsilon_{t}\in[0,1]$ and $|\mathcal{A}|$ is the number of all possible
actions.

We can sample from this mixture distribution by

  1. 1.

Sample $e_{t}\in\left\\{0,1\right\\}$ from a Bernoulli distribution of mean
$\epsilon_{t}$.

  2. 2.

Check $e_{t}$

     * •

If $e_{t}=1$, we uniformly choose $a_{t}$ at random.

     * •

If $e_{t}=0$, we sample $a_{t}$ proportionally to $\hat{y}(a)$.

As we continue the RCT according to this assignment policy, we assign
participants increasingly more to actions with better outcomes, because our
causal estimate gets better over time. We however ensure that participants are
randomly assigned to actions at a reasonable rate of $\epsilon_{t}$, in order
to estimate the causal quantity rather than the statistical quantity. It is
common to start with a larger $\epsilon_{t}\approx 1$ and gradually anneal it
toward $0$, as we want to ensure we quickly estimate the correct causal effect
early on. It is also usual to start with a large $\beta_{t}\geq 1$ and anneal
it toward $0$, as the early estimate of the causal effect is often not
trustworthy.

When the first component (the uniform distribution) is selected, we say that
we are exploring, and otherwise, we say we are exploiting. $e_{t}$ is a
hyperparameter that allows us to compromise between exploration and
exploitation, while $\beta_{t}$ is how we express our belief in the current
estimate of the causal effect.

This whole procedure is a variant of EXP-3, which stands for the exponential-
weight algorithm for exploration and exploitation (Allesiardo et al., 2017),
that is used to solve the multi-armed bandit problem. However, with an
appropriate choice of $\epsilon_{t}$ and $\beta_{t}$, we obtain as a special
case RCT that can estimate the causal effect of the action on the outcome. For
instance, we can use the following schedules of these two hyperparameters:

$\displaystyle\epsilon_{t}=\beta_{t}=\begin{cases}1,&\text{ if }t<T\\\
0,&\text{ if }t\geq T\end{cases}$ (3.27)

with a larger $T\gg 1$. We can however choose smoother schedulers for
$\epsilon_{t}$ and $\beta_{t}$ in order to make a better compromise between
causal inference and outcome maximization, in order to avoid assigning too
many subjects to a placebo (that is, ineffective) group.

The choice of $\epsilon_{t}$ and $\beta_{t}$ also affects the bias-variance
trade-off. Although this is out of the scope of this course, it is easy to
guess that higher $\epsilon_{t}$ and higher $\beta_{t}$ lead to a higher
variance but a lower bias, and vice versa.

##### A never-ending trial.

A major assumption that must be satisfied for RCT is the stationarity. Both
the causal distribution $p^{*}(y|a,x)$ and the covariate distribution
$p^{*}(x)$ must be stationary in that they do not change throughout the trial
as well as after the trial. Especially when these distributions drift after
the trial, that is, after running the trial with $T$ participants, our causal
estimate as well as the decision based on it will become less accurate. We see
such instances often in the real world. For instance, as viruses mutate, the
effectiveness of vaccination wanes over time, although the very same vaccine
was found to be effective by an earlier RCT.

When the underlying conditional distributions are all stationary, we do not
need to keep the entire set of collected data points in order to compute the
approximate causal effect, because

$\displaystyle\hat{y}_{t}(a)=\frac{\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)y_{t^{\prime}}}{\sum_{t^{\prime\prime}=1}^{N}\mathds{1}(a_{t^{\prime\prime}}=a)}=\frac{\sum_{t^{\prime}=1}^{t-1}\mathds{1}(a_{t^{\prime}}=a)}{\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)}\hat{y}_{t-1}(a)+\frac{\mathds{1}(a_{t}=a)}{\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)}y_{t}.$
(3.28)

In other words, we can just keep a single scalar $\hat{y}_{t}(a)$ for each
action to maintain the causal effect over time.

We can tweak this recursive formula to cope with slow-drifting underlying
distributions by emphasizing recent data points much more so than older data
points. This can be implemented with exponential moving average,555
‘exponential-weight’ in EXP-3 comes from this choice.  as follows:

$\displaystyle\hat{y}_{t}(a)=\begin{cases}\eta\hat{y}_{t-1}(a)+(1-\eta)y_{t},&\text{
if }a_{t}=a\\\ \hat{y}_{t-1}(a),&\text{ if }a_{t}\neq a\end{cases}$ (3.29)

where $\eta\in[0,1)$. As $\eta\to 0$, we consider an increasingly smaller
window into the past and do not trust what we have seen happen given a
particular action. On the other hand, when $\eta\to 1$, we do not trust what
happens now but rather what we already know about the causal effect of an
action $a$ should be.

By keeping track of the causal effect with exponential moving average, we can
continuously run the trial. When doing so, we have to be careful in choosing
the schedules of $\epsilon_{t}$ and $\beta_{t}$. Unlike before, $\epsilon_{t}$
should not be monotonically annealed toward $0$, as earlier exploration may
not be useful later when the underlying distributions drift. $\beta_{t}$ also
should not be annealed toward $0$, as the estimate of the causal effect we
have at any moment cannot be fully trusted due to the unanticipated drift of
underlying distributions. It is thus reasonable to simply set both $\beta_{t}$
and $\epsilon_{t}$ to reasonably large constants.

##### Checking the bias.

At time $t$, we have accumulated

$\displaystyle
D_{t}=\left\\{(e_{1},a_{1},y_{1}),\ldots,(e_{t},a_{t},y_{t})\right\\},$ (3.30)

where $e_{t^{\prime}}$ indicates whether we explored ($1$) or exploited ($0$)
at time $t^{\prime}$.

We can then get the unbiased estimate of the causal effect of $a$ on $y$ by
only using the triplets $(e,a,y)$ for which $e=1$. That is,

$\displaystyle\tilde{y}_{t}(a)=\frac{\sum_{t^{\prime}=1}^{t}\mathds{1}(e_{t^{\prime}}=1)\mathds{1}(a_{t^{\prime}}=a)y_{t^{\prime}}}{\sum_{t^{\prime\prime}=1}^{t}\mathds{1}(e_{t^{\prime\prime}}=1)\mathds{1}(a_{t^{\prime\prime}}=a)}.$
(3.31)

This estimate is unbiased, unlike $\hat{y}(a)$ from EXP-3 above, since we only
used the action-outcome pairs when the action was selected randomly from the
same uniform distribution.

Assuming a large $t$ (so as to minimize the impact of a high variance,) we can
then compute the (noisy) bias of the causal effect estimated and used by EXP-3
above as

$\displaystyle b_{t}=(\hat{y}_{t}(a)-\tilde{y}_{t}(a))^{2}.$ (3.32)

Of course this estimate of the bias is noisy and especially so when $t$ is
small, since the effective number of data points used to estimate
$\tilde{y}_{t}$ is on average

$\displaystyle\sum_{t^{\prime}=1}^{t}\epsilon_{t^{\prime}}\leq t.$ (3.33)

###  3.5 When Some Confounders are Observed

The assignment of an individual $x$ to a particular treatment option $a$ is
called a policy. In the case of RCT, this policy was a uniform distribution
over all possible actions $a$, and in the case of EXP-3, it was a mixture of a
uniform policy and a effect-proportional policy from Eq. (3.26). In both
cases, the policy was not conditioned on the covariate $x$, meaning that no
information about the individual was used for assignment. This is how we
addressed the issue of unobserved confounders.

Such an approach is however overly restrictive in many cases, as some
treatments may only be effective for a subset of the population that share a
certain trait. For instance, consider the problem of inferring the effect of a
monoclonal antibody therapeutics called Trastuzumab (or Herceptin) for breast
cancer on the disease-free survival of a patient (Nahta and Esteva, 2007). If
we run RCT without taking into account any covariate information as above, we
very likely would not see any positive effect on the patient’s disease-free
survival, because Trastuzumab was specifically designed to work for
HER2-positive breast cancer. That is, only breast cancer patients with over-
expressed ERBB2 gene (which encodes the HER2 receptor) would benefit from
Trastuzumab. In these cases, we are interested in conditional average
treatment effect (CATE) from earlier, that is, to answer the question of what
causal effect Trastuzumab has on patients given their gene expression profile.
CATE given the overly-expressed ERBB2 gene of Trastuzumab would be positive,
while CATE without over-expression of ERBB2 gene would be essentially zero.

When we observe some confounders, such as the gene expression profile of a
subject in the example above, and do not observe all the other confounders, we
can mix RCT (§3.3) and regression (§3.2) to estimate the conditional causal
effect conditioned on the observed confounders.

The graph $G$ with the partially-observed confounder is depicted as below with
the unobserved $x$ and the observed $x^{\prime}$:

$a$$y$$x$$x^{\prime}$

Each subject in RCT then corresponds to the action-outcome-observed-covariate
triplet $(a_{t},y_{t},x^{\prime}_{t})$. Assume we have enrolled and
experimented with $t$ participants so far, resulting in

$\displaystyle
D_{t}=\left\\{(a_{1},y_{1},x^{\prime}_{1}),\ldots,(a_{t},y_{t},x^{\prime}_{t})\right\\}.$
(3.34)

Let $\hat{x}^{\prime}$ be the condition of interest, such as the
overexpression of ERBB2. we can then create a subset $D(\hat{x}^{\prime})$ as

$\displaystyle D_{t}(\hat{x}^{\prime})=\left\\{(a,y,x^{\prime})\in
D_{t}|x^{\prime}=\hat{x}^{\prime}\right\\}\subseteq D_{t}.$ (3.35)

We can then use this subset $D_{t}(\hat{x}^{\prime})$ as if it were the set
from the ordinary RCT, in order to estimate the conditional causal effect, as
follows.

$\displaystyle\hat{y}_{t}(a|x^{\prime})\approx\frac{\sum_{(a_{i},y_{i},x_{i}^{\prime})\in
D_{t}(x^{\prime})}\mathds{1}(a_{i}=a)y_{i}}{\sum_{(a_{j},y_{j},x_{j}^{\prime})\in
D_{t}(x^{\prime})}\mathds{1}(a_{j}=a)}.$ (3.36)

Just like what we did earlier, we can easily turn this into a recursive
version in order to save memory:

$\displaystyle\hat{y}_{t}(a|x^{\prime})=\begin{cases}\hat{y}_{t-1}(a|x^{\prime}),&\text{
if }x^{\prime}_{t}\neq x^{\prime}\\\
\frac{\hat{y}_{t-1}(a|x^{\prime})\sum_{(a_{j},y_{j},x_{j}^{\prime})\in
D_{t-1}(x^{\prime})}\mathds{1}(a_{j}=a)+y_{t}\mathds{1}(a_{t}=a)}{\sum_{(a_{k},y_{k},x_{k}^{\prime})\in
D_{t}(x^{\prime})}\mathds{1}(a_{k}=a)},&\text{ if
}x^{\prime}_{t}=x^{\prime}\end{cases}$ (3.37)

Similarly to earlier, we can instead of exponential moving average in order to
cope with the distribution drift over the sequential RCT, as

$\displaystyle\hat{y}_{t}(a|x^{\prime})=\begin{cases}\hat{y}_{t-1}(a|x^{\prime}),&\text{
if }x^{\prime}_{t}\neq x^{\prime}\\\
\eta_{t}\hat{y}_{t-1}(a|x^{\prime})+(1-\eta_{t})y_{t}\mathds{1}(a_{t}=a),&\text{
if }x^{\prime}_{t}=x^{\prime}\end{cases}$ (3.38)

We can then use this running estimates of the causal effects in order to build
an assignment policy $\pi_{t}(a|x^{\prime})$ that now depends on the observed
covariate $x^{\prime}$. If we go back to the earlier example of Trastuzumab,
this policy would increasingly more assign participants with over-expressed
ERBB2 to the treatment arm, while it would continue to be largely uniform for
the remaining population.

##### A Parametrized Causal Effect.

Up until this point, partially-observed confounders do not look like anything
special. It is effectively running multiple RCT’s in parallel by running an
individual RCT for each covariate configuration. There is no benefit of
running these RCT’s in parallel relative to running these RCT’s in sequence.
We could however imagine a scenario where the former is more beneficial than
the latter, and we consider one such case here.

Assume that $x^{\prime}$ is a multi-dimensional vector, i.e.,
$x^{\prime}\in\mathbb{R}^{d}$ and that the true causal effect of each action
$a$ is a linear function of the observed covariate $x^{\prime}$:

$\displaystyle\hat{y}^{*}(a|x^{\prime})=\theta^{*}(a)^{\top}x^{\prime}+b^{*}(a)=\sum_{i=1}^{d}\theta_{d}^{*}(a)x^{\prime}_{d}+b(a).$
(3.39)

This means that each dimension $x^{\prime}_{d}$ of the covariate has an
additive effect on the expected outcome $\hat{y}(a|x)$ weighted by the
associated coefficient $\theta_{d}^{*}(a)$, and that the effect
$\theta_{d}^{*}(a)x^{\prime}_{d}$ of each dimension on the expected outcome is
independent of the other dimensions’ effects.

As an example, consider estimating the effect of weight lifting on the overall
health. The action is whether to perform weight lifting each day, and the
outcome is the degree of the subject’s healthiness. Each dimension
$x^{\prime}_{d}$ refers to a habit of a person. For instance, it could be a
habit of smoking, and the corresponding dimension $x^{\prime}_{d}$ encodes the
number of cigarettes the subject smokes a day. Another habit could be jogging,
and the corresponding dimension would encode the number of minutes the subject
runs a day. Smoking is associated with a negative coefficient regardless of
$a$. On the other hand, jogging is associated with a negative coefficient when
$a=1$, because an excessive level of workout leads to frequent injuries, while
it is with a positive coefficient when $a=0$.

Of course, some of these habits may have nonlinear effects. Running just the
right duration each day in addition to weight lifting could lead to a better
health outcome. It is however reasonable to assume linearity as the first-
order approximation.

We can estimate the coefficients $\theta(a)$ by regression from §3.2 by
solving

$\displaystyle\min_{\theta(a)}\frac{1}{2}\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)\left(y_{t^{\prime}}-\theta(a)^{\top}x^{\prime}_{t^{\prime}}-b^{*}(a)\right)^{2}.$
(3.40)

Instead of keeping the count for each and every possible $x^{\prime}$, we now
keep only $\theta(a)$ for each action $a$. This has an obvious advantage of
requiring only $O(d)$ memory rather than $O(2^{d})$.

More importantly however is that the estimated causal effect generalizes
unseen covariate configuration. Let us continue from the example of having
smoking and jogging as two dimensions of $x^{\prime}$. During RCT, we may have
seen participants who either smoke or jog but never both. Because of the
linearity, the estimated causal effect predictor,

$\displaystyle\theta(a)_{\mathrm{smoke}}x^{\prime}_{\mathrm{smoke}}+\theta(a)_{\mathrm{run}}x^{\prime}_{\mathrm{run}},$
(3.41)

generalizes to participants who both smokes and jogs as well as who neither
smokes nor jogs.

This case of a linear causal effect suggests that we can rely on the power of
generalization in machine learning in order to bypass the strong assumption of
positivity. Even if we do not observe a covariate or an associated action, a
parametrized causal effect predictor can generalize to those unseen cases. We
will discuss this potential further later in the semester.

##### When there are many possible actions.

Assume we do not observe any confounder, that is, there is no $x^{\prime}$.
Then, at each time, RCT is nothing but estimating a single scalar for each
action. Let $\mathcal{A}$ be a set of all actions and $|\mathcal{A}|$ a
cardinality of this action set. Then, at any time $t$ of running an RCT, the
number of data points we can use to estimate the causal effect of a particular
action is

$\displaystyle N(a)=\sum_{t^{\prime}=1}^{t}\mathds{1}(a_{t^{\prime}}=a)\approx
tp_{a}(a),$

where $p_{a}(a)$ is the probability of selecting the action $a$ during
randomization. Just like the case above with partially observed confounders,
the variance of the estimate of the causal effect of an individual action
decreases dramatically as the number of possible actions increases.

We must have some extra information (context) about these actions in order to
break out of this issue. Let $c(a)\in\mathbb{R}^{d}$ be the context of the
action $a$. For instance, each dimension of $c(a)$ corresponds to the amount
of one ingredient for making the perfect steak seasoning, such as salt,
pepper, garlic and others. Then, each action $a$ corresponds to a unique
combination of these ingredients.

In this case, the causal effect of any particular action can be thought of
mapping $c(a)$ to the outcome $\hat{y}(a)$ associated with $a$. If we assume
this mapping was linear (Li et al., 2010), we can write it as

$\displaystyle\hat{y}(a)=c(a)^{\top}{\theta^{*}}+b^{*},$ (3.42)

where $\theta\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$.

Similarly to the case where there was an observed confounder above, with
linearity, we do not need to maintain the causal estimate for each and every
possible action, which amounts to $|\mathcal{A}|$ numbers, but the effect of
each dimension of the action context on the outcome, which amounts of $d$
numbers. When $d\ll|\mathcal{A}|$, we gain a significant improvement in the
variance of our estimates.

Furthermore, just like what we saw above, we benefit from the
compositionality, or compositional generalization. For instance, if the
effects of salt and pepper on the final quality of seasoning are independent
and additive, we can accurately estimate the effect of having both salt and
pepper even when all tested seasonings had either salt or pepper but never
both. Let $c(a)=[s_{\mathrm{salt}},s_{\mathrm{pepper}}]$, and assume
$s_{\mathrm{salt}},s_{\mathrm{pepper}}\in\\{0,1\\}$ and that all past trials
were such that $s_{\mathrm{salt}}=0$ or $s_{\mathrm{pepper}}=0$. We can
approximate $\theta^{*}_{\mathrm{salt}}$ and $\theta^{*}_{\mathrm{pepper}}$
from these past trials, and due to the linearity assumption, we can now
compute the causal effect of $c(a)=[1,1]$, as

$\displaystyle\hat{\theta}_{\mathrm{salt}}+\hat{\theta}_{\mathrm{pepper}}.$
(3.43)

This would not have been possible without the linearity, or more generally
compositionality, because this particular action of adding both salt and
pepper has never been seen before, i.e., it violates the positivity
assumption. This is yet another example of overcoming the violation of
positivity by generalization.

At this point, one sees a clear connection between having some confounders
observed and having many actions associated with their contexts. This is
because they are simply two sides of the same coin. I leave this to you to
think of why this is the case.

###  3.6 Summary

In this chapter, we have learned about the following topics:

  1. 1.

Average treatment effect;

  2. 2.

Regression for causal inference;

  3. 3.

Randomized controlled trials;

  4. 4.

Outcome maximization with a bandit algorithm;

  5. 5.

A contextual bandit.

##  Chapter 4 Passive Causal Inference

###  4.1 Challenges in Randomized Controlled Trials

A major issue with randomized controlled trials (RCT) is that we must
experiment with subjects. This raises many issues that are not necessarily
related to causal inference itself but are more broadly about ethics and
legality. For instance, the “Tuskegee Study of Untreated Syphilis in the Negro
Male” was the widely-known and widely-condemned study for investigating the
effect of untreated syphilis (for Disease Control et al., 2020). As RCT
requires careful, double blinding, the trial administrators did not reveal to
the study participants that they were diagnosed with (latent) syphilis. The
study was originally designed (and the participants were told) to run for six
months but lasted for 40 years until the details of the study were leaked to
the press. During these decades, the treatment for syphilis was made available
but none of the participants were treated properly, resulting in the death of
more than 100 participants due to syphilis, out of approximately 400
participants, the syphilis infection of the wives of fourty participants and
the congenital syphilis infection of 19 children. It took more than half a
century for the US government to formally issue apology.

A similar issue persists throughout medicine when it comes to RCT which is de
facto standard for establishing any causal effect of a treatment on the
outcome of a patient. Due to the necessity of randomization, some patient
participants will inevitably receive placebo rather than the actual treatment.
Even if the tested treatment ultimately turns out to be causally effective, by
then it may be already late for those patients who were put on the control arm
to receive and benefit from this new treatment. How ready are you to put
patients into suffering because we want to (and often need to) establish the
causal effect of a new treatment?

Sometimes, it is impossible to design a placebo that ensures double blindness
of a trial. Consider for instance an RCT on the effectiveness of masking on
preventing respiratory diseases. Participants will understandably alter their
behaviours based on their assignments; treatment (masking) or control (no
masking), as their perception of risk is altered, which violates the
stationarity of the causal effect $p^{*}(y|a,x)$. In order to avoid this, we
must ensure that participants cannot tell whether they are in the treatment or
control arm, but it is pretty much impossible to design a placebo mask that
looks and feels the same as an actual mask but does not filter any particle in
the air. In other words, RCT is only possible when placebos can be effectively
designed and deployed.

In this example, we run into yet another problem; how do we enforce the
treatment on subjects? In the case of vaccination, subjects come into clinics
and are for instance injected on the spot under the supervision of a
clinician, after which the subjects cannot get rid of injected vaccine. In the
case of masking, for instance, we cannot ensure that participants wear masks
as they are instructed, as this requires non-stop monitoring throughout the
trial period.

Finally, some actions take long to have measurable impact on the outcome. For
instance, consider a policy proposal of introducing a new course on
programming at elementary schools (1-6 grades) with the goal of improving
students’ job prospects and growing the information technology (IT) sector. It
will take anywhere between 12 to 20 years for these students to finish their
education and participate in society, and we will have to wait another 4 to 15
years to see any measurable economic impact on the IT sector. Such a long
duration between the action and the outcome further complicates RCT, as it is
often impossible to ensure the stationarity of underlying distributions over
that duration. RCT is thus not suitable for such actions that require a
significant amount of time to have any measurable impact.

In this chapter, we instead consider an alternative approach to RCT, where we
rely on existing data to infer the causal relationship between the action and
outcome. As we use already collected data, we can often avoid the issues
arising from actively experimentation, although we are now faced with another
set of challenges, such as the existence of spurious correlations arising from
various unobserved confounders that affected the choice of actions earlier. We
will discuss how we can avoid these issues in this chapter. It is however
important to emphasize that there is no silver bullet in causal inference.

###  4.2 When Confounders were also Collected

####  4.2.1 Inverse Probability Weighting

$a$$y$$x$

Let us come back to the original graph $G$ that consists of three variables;
$a$, $y$ and $x$, with the following joint probability:

$\displaystyle p^{*}(a,y,x)=p^{*}(x)p^{*}(a|x)p^{*}(y|a,x).$ (4.1)

We also assume that we have a set $D$ of triplets $(a_{n},y_{n},x_{n})$ drawn
from this underlying graph $G$. In other words, we assume that we observe the
confounder in this case.

If we have a large such set, i.e. $N=|D|\gg 1$, we can use regression, as in
§3.2, to approximate $p^{*}(y|a,x)$ in order to compute the causal effect as

$\displaystyle\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]$
$\displaystyle\approx\sum_{x}p^{*}(x)\sum_{y}y\hat{p}(y|\hat{a},x)$ (4.2)
$\displaystyle\approx\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}_{\hat{p}(y|\hat{a},x_{n})}[y].$
(4.3)

Although this regression-based approach is straightforward, this approach has
a disadvantage of having to fit a regressor on the concatenation of the action
$a$ and confounder $x$. The issue is exactly that of RCT with partially
observed confounders, that is, we must have enough data points for each
action-covariate combination in order for regression to have a low variance.
We can use regularization to reduce the variance, which may unfortunately
introduce a bias.

We can reduce the variance by using regression to estimate a simpler quantity.
In particular, we consider approximating $p^{*}(a|x)$. Because this is a map
from $\mathcal{X}$, we just need enough data points for each covariate
configuration rather than the action-covariate combination. Approximating
$p^{*}(a|x)$ allows us to estimate the causal effect using data points drawn
from the original graph $G$ rather than the modified graph $\overline{G}$ from
§3.3, because

$\displaystyle\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]$
$\displaystyle=\sum_{x}p^{*}(x)\sum_{a}\mathds{1}(a=\hat{a})\sum_{y}p^{*}(y|\hat{a},x)y$
(4.4)
$\displaystyle=\sum_{x}\sum_{a}\sum_{y}p^{*}(x)p^{*}(a|x)\mathds{1}(a=\hat{a})p^{*}(y|\hat{a},x)\frac{1}{p^{*}(a|x)}y$
(4.5)
$\displaystyle=\frac{1}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})}\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})\frac{y_{n}}{p^{*}(\hat{a}|x_{n})}.$
(4.6)

Instead of the true $p^{*}(\hat{a}|x_{n})$, we plug in the regression-based
approximation $\hat{p}(\hat{a}|x_{n})$ and arrive at

$\displaystyle\mathbb{E}_{G}\left[y|\mathrm{do}(a=\hat{a})\right]$
$\displaystyle\approx\frac{\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})\frac{y_{n}}{\hat{p}(\hat{a}|x_{n})}}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})}.$
(4.7)

In words, we look for all data points within the previously-collected set $D$
that are associated with the action $\hat{a}$ of interest. The simple average
of the associated outcomes would be a biased estimate of the casual effect,
since it combines the effects of $a$ on $y$ via two paths; (causal) $a\to y$
and (spurious) $a\leftarrow x\to y$. We correct this bias by weighting each
data point, or the associated outcome, by the inverse probability of the
action given the confounder, $\frac{1}{\hat{p}(\hat{a}|x_{n})}$. This approach
is thus called inverse probability weighting (IPW), and $p^{*}(a|x)$ is often
referred to as a propensity score.

##### It is fine to have missing outcomes.

One important advantage of the IPW-based approach is that we can approximate
$p^{*}(a|x)$ using all $(a,x)$ pairs even if they are not associated with the
outcome $y$, unlike the earlier regression-based approach which required
having all three variables observed $(a,y,x)$. Imagine using clinical notes
and measurements from the electronic health record (EHR) of a large hospital
in order to estimate the causal effect of a particular drug on a target
disease. Of course, the prescription of the drug $a$ is not made blindly but
based on the patient information which includes their underlying health
condition. Since the existing health conditions affect the outcome $y$ of
almost any kind of a disease, such patient information is a confounder $x$.
Some patients often do not return to the same hospital for follow-up checks,
meaning that the EHR does not record the outcome of these patients, leaving us
only with $(a,x)$. We can then use all the prescriptions to approximate the
propensity score $p(a|x)$ and then use a subset of these prescriptions for
which the patients’ outcomes are recorded (i.e. they came back for follow-up
visits) to compute the causal effect.

Mathematically, it means that we solve two separate optimization problems
using two different data sets (though, one is a superset of the other,) as
follows:

$\displaystyle\hat{p}(a|x)=\arg\max_{p}\sum_{(a^{\prime},x^{\prime})\in
D_{\pi}}\log p(a^{\prime}|x^{\prime}),$ (4.8)
$\displaystyle\hat{y}(\hat{a})=\frac{\sum_{(a^{\prime},x^{\prime},y^{\prime})\in
D_{\bar{y}}}\mathds{1}(a^{\prime}=\hat{a})\frac{y_{n}}{\hat{p}(\hat{a}|x_{n})}}{\sum_{(a^{\prime\prime},x^{\prime\prime},y^{\prime\prime})\in
D_{\bar{y}}}\mathds{1}(a^{\prime\prime}=\hat{a})},$ (4.9)

where $D_{\bar{y}}\subseteq D_{\pi}$.

##### A doubly robust estimator.

A major issue with the IPW-based approach is that the variance of the
estimator can be very large, even if the variance of estimating the propensity
score is low, because the propensity scores shows up in the denominator. On
the other hand, the regression-based approach has a low variance due to the
missing division by the propensity score. It is however likely a biased
estimator, as we cannot easily guarantee that the choice of a non-parametric
regressor can identify the correct conditional probability, due to a variety
of issues, such as the lack of realizability.

If the regression-based approach is correct for an instance $(a,y,x)$,
$\tilde{y}(a,x)$ would coincide with $y$, and we would just use
$\tilde{y}(a,x)$ as it is and prefer to avoid the IPW-based estimate due to
the potentially large variance arising from the denominator. Otherwise, we
want to rely on the IPW-based estimate to correct for the incorrectness
arising from the regression-based approach. This can be expressed as

$\displaystyle\hat{y}(\hat{a})=\sum_{x}p(x)\sum_{a}\left(\frac{1}{|\mathcal{A}|}\tilde{y}(a,x)+\underbrace{p^{*}(a|x)\frac{\tfrac{1}{|\mathcal{A}|}}{\hat{p}(a|x)}}_{(b)}\left(\underbrace{y^{*}(a|x)-\tilde{y}(a,x)}_{\mathrm{(a)}}\right)\right).$
(4.10)

If $\tilde{y}(a,x)$ is perfect, (a) disappears, as expected. If our estimate
of the propensity score is perfect, (b) is $1$, resulting in using the true
$y^{*}(a,x)$ while ignoring the regression-based approach.111 $y^{*}(a,x)$ is
the true expected outcome given the action $a$ and the covariate $x$. Since
expectation is linear, we can push the expectation all the way inside to
obtain this expression.

Since we are provided with data rather than the actual probability
distributions, we end up with

$\displaystyle\hat{y}(\hat{a})=\frac{1}{Z(\hat{a})}\sum_{(a^{\prime},x^{\prime},y^{\prime})\in
D}\mathds{1}(a^{\prime}=\hat{a})\left(\tilde{y}(\hat{a},x_{n})+\frac{1}{\hat{p}(\hat{a}|x_{n})}\left(y_{n}-\tilde{y}(\hat{a},x_{n})\right)\right),$
(4.11)

where
$Z(\hat{a})=\sum_{(a^{\prime\prime},x^{\prime\prime},y^{\prime\prime})\in
D}\mathds{1}(a^{\prime\prime}=\hat{a}).$ This estimator is called a doubly
robust estimator.

####  4.2.2 Matching.

Instead of estimating $p^{*}(a|x)$ and multiplying the observed outcome with
its inverse, we can achieve a similar outcome by manipulating data itself.
When $p^{*}(a|x)=p^{*}(a)$, that is, the action is independent of the
covariate, the IPW-based estimate coincides with simple averaging, just like
in RCT from §3.3. This happens when the ratio of actions associated with each
unique $x$ in the data is the same across the data set. To make it simpler,
let us express this ratio of actions by assigning the minimal number of each
action in relation to the other actions. For instance, if our desired ratio
between the treatment and placebo is $0.8:0.2$, we would express it for
instance as $4:1$.

Starting from the original data set
$D=\left\\{(a_{1},x_{1},y_{1}),\ldots,(a_{N},x_{N},y_{N})\right\\}$, we go
through each $x_{n}$ by collecting as many $(a,x_{n},y)\in D$ as $n_{a}$,
where $n_{a}$ is the target number of examples of action $a$, for instance $4$
above. This collection can be done randomly or by following some fixed
strategy, such as round-robin scheduling. Sometimes it is necessary to choose
the same triplet multiple times, which is not ideal but may be necessary. By
aggregating these collections, we arrive at a new dataset $\tilde{D}$.

Under this new dataset $\tilde{D}$, the propensity score is guaranteed to be

$\displaystyle\hat{p}(a|x)\propto n_{a},$ (4.12)

regardless of $x$. Furthermore, $\hat{p}(a|x)=\hat{p}(a)$ as well. Meanwhile,
$\hat{p}(x)$ stays the same as that under the original data set $D$. Then, the
expected causal outcome of any given action $a$ under this dataset is

$\displaystyle\hat{y}(\hat{a})=\frac{1}{N}\sum_{n=1}^{N}\mathds{1}(a^{\prime}_{n}=\hat{a})y^{\prime}_{n},$
(4.13)

where $(a^{\prime}_{n},y^{\prime}_{n},x^{\prime}_{n})\in\tilde{D}$. This
avoids the issue of the high variance arising from the IPW-based approach.
This however does not mean that this approach always works.

The most obvious issue with this approach is that the original data set may
not have enough triplets associated with each $x$ to ensure that $p^{*}(a|x)$
is identical for all $x$. Furthermore, even if we have enough associated
triplets for each $x$, we may end up with discarding many triples from the
original data set to form the new data set. We never want to discard data
points we have. This approach is thus appropriate when data is already well
balanced and the goal is to further ensure that the propensity score is
constant.

A relatively simple variant of this approach, called ‘matching’ because we
match triplets based on $x$, is to relax the constraint that we only consider
the exact match of the covariate $x$. The original formulation samples the
triplets according to the target counts with replacement from the following
multiset:222 It is a multiset, since there can be duplicates.

$\displaystyle D(x)=\left\\{(a^{\prime},y^{\prime},x^{\prime})\in
D|x^{\prime}=x\right\\}.$ (4.14)

This condition of $x^{\prime}=x$ may be too strict, leaving us with only a
very small $D(x)$. We can instead relax this condition using a predefined
notion of similarity such that

$\displaystyle\tilde{D}(x)=\left\\{(a^{\prime},y^{\prime},x^{\prime})\in
D|s(x^{\prime},x)<\epsilon\right\\},$ (4.15)

where $s(x^{\prime},x)$ is a predefined distance function and $\epsilon$ a
distance threshold.

###  4.3 Instrumental Variables: When Confounders were not Collected

So far in this section we have considered a case where the confounder $x$ was
available in the observational data. This allowed us to either fit the
regressor directly on $p(y|a,x)$, use inverse probability weighting or re-
balance the dataset using the matching scheme. It is however unlikely that we
are given full access to the confounders (or any kind of covariate) in the
real world. It is thus important to come up with an approach that works on
passively collected data without covariates.

##### An instrumental variable estimator.

Let us rewrite the following graph $G_{0}$ into a corresponding structural
causal model:

$a$$y$$x$

The structural causal model is then

$\displaystyle x\leftarrow\epsilon_{x}$ (4.16) $\displaystyle a\leftarrow
f_{a}(x,\epsilon_{a})$ (4.17) $\displaystyle y\leftarrow
f_{y}(a,x,\epsilon_{y}).$ (4.18)

From this structural causal model, we can read out two important points.
First, as we have learned earlier in §2.2, $x$ is a confounder, and when it is
not observed, the path $a\leftarrow x\to y$ is open, creating a spurious
effect of $a$ on $y$. Second, the choice of $a$ is not fully determined by
$x$. It is determined by the combination of $x$ and $\epsilon_{a}$, where the
latter is independent of $x$. We are particularly interested in the second
aspect here, since it gives us an opportunity to modify this graph by
introducing a new variable that may help us remove the effect of the
confounder $x$.

We now consider an alternative to the structural causal model above by
assuming that we found another variable $z$ that largely explains the
exogenous factor $\epsilon_{a}$. That is, instead of saying that the action
$a$ is determined by the combination of the covariate $x$ and an exogenous
factor $\epsilon_{a}$, we now say that it is determined by the combination of
the covariate $x$, this new variable $z$ and an exogenous factor
$\epsilon^{\prime}_{a}$. Because $z$ explains a part of the exogenous factor
rather than $x$, $z$ is independent of $x$ a priori. This introduction of $z$
alters the structural causal model to become

$\displaystyle x\leftarrow\epsilon_{x}$ (4.19) $\displaystyle
z\leftarrow\epsilon_{z}$ (4.20) $\displaystyle a\leftarrow
f^{\prime}_{a}(x,z,\epsilon^{\prime}_{a})$ (4.21) $\displaystyle y\leftarrow
f_{y}(a,x,\epsilon_{y}),$ (4.22)

which corresponds to the following graph $G_{1}$:

$z$$a$$y$$x$

This altered graph $G_{1}$ does not help us infer the causal effect of $a$ on
$y$ any more than the original graph $G_{0}$ did. It however provides us with
an opportunity to replace the original action $a$ with a proxy based purely on
the newly introduced variable $z$ independent of $x$.

We first notice that $x$ cannot be predicted from $z$, because $z$ and $x$ are
by construction independent. The best we can do is thus to predict the
associated action.333 We can predict the marginal distribution over the action
after marginalizing out $x$.  Let $p_{\tilde{a}}(a|z)$ be the conditional
distribution induced by $g_{\tilde{a}}(z,\epsilon_{a}^{\prime\prime})$. Then,
we want to find $g_{a}$ that minimizes

$\displaystyle\mathcal{E}_{a}=-\mathbb{E}_{\epsilon_{z}}\mathbb{E}_{\epsilon_{x}}\mathbb{E}_{\epsilon^{\prime}_{a}}\left[\log
g_{\tilde{a}}(f^{\prime}_{a}(\epsilon_{x},z,\epsilon^{\prime}_{a}),z)\right].$
(4.23)

We also notice that $z$ cannot be predicted from $a$ perfectly without $x$,
because $a$ is an observed collider, creating a dependency between $z$ and
$x$. The best we can do is to thus to predict the expected value of $z$. That
is, we look for $g_{z}$ that minimizes

$\displaystyle\mathcal{E}_{z}=-\mathbb{E}_{\epsilon_{z}}\mathbb{E}_{\epsilon_{x}}\left[\log
p_{\tilde{z}}(\epsilon_{z}|f^{\prime}_{a}(\epsilon_{x},\epsilon_{z},\epsilon^{\prime}_{a}))\right],$
(4.24)

where we have used $p_{\tilde{z}}(z|a)$ be the conditional distribution
induced by $g_{\tilde{z}}(a,\epsilon^{\prime}_{z})$, similarly to
$p_{\tilde{a}}(a|z)$ above.

Once we found reasonable solutions, $\hat{g}_{\tilde{a}}$ and
$\hat{g}_{\tilde{z}}$, to the minimization problems above, respectively, we
can further modify the structural causal model into

$\displaystyle a\leftarrow\hat{a}$ (4.25) $\displaystyle
x\leftarrow\epsilon_{x}$ (4.26)
$\displaystyle\tilde{z}\leftarrow\hat{g}_{\tilde{z}}(a,\epsilon^{\prime}_{z})$
(4.27)
$\displaystyle\tilde{a}\leftarrow\hat{g}_{\tilde{a}}(\tilde{z},\epsilon^{\prime\prime}_{a})$
(4.28) $\displaystyle y\leftarrow f_{y}(\tilde{a},x,\epsilon_{y}).$ (4.29)

Since $x$ is really nothing but an exogenous factor of $y$ without impacting
$\tilde{a}$ nor $z$ in this case, we can simplify this by merging $x$ and
$\epsilon_{y}$ into

$\displaystyle a\leftarrow\hat{a}$ (4.30)
$\displaystyle\tilde{z}\leftarrow\hat{g}_{z}(a,\epsilon^{\prime}_{z})$ (4.31)
$\displaystyle\tilde{a}\leftarrow\hat{g}_{\tilde{a}}(\tilde{z},\epsilon^{\prime\prime}_{a})$
(4.32) $\displaystyle y\leftarrow f_{y}(\tilde{a},\epsilon^{\prime}_{y}).$
(4.33)

Because we assume the action $a$ is always given, we simply set it to a
particular action $\hat{a}$.

This structural causal model can be depicted as the following graph $G_{2}$:

$\hat{a}$$\tilde{z}$$\tilde{a}$$y$

In other words, we start from the action, approximately infer the extra
variable $\tilde{z}$, approximately infer back the action and then predict the
outcome. During two stages of inference ($\tilde{z}|a$ and
$\tilde{a}|\tilde{z}$), we drop the dependence of $a$ on $z$. This happens,
because $z$ was chosen to be independent of $x$ a priori.

In this graph, we only have two mediators in sequence, $\tilde{z}$ and
$\tilde{a}$, from the action $a$ to the outcome $y$. We can then simply
marginalize out both of these mediators in order to compute the interventional
distribution over $y$ given $a$, as we learned in §2.2. That is,

$\displaystyle\hat{y}_{G_{0}}(a=\hat{a})=\hat{y}_{G_{1}}(a=\hat{a})\approx\hat{y}_{G_{2}}(a=\hat{a})=\mathbb{E}_{\tilde{z}|\hat{a}}\mathbb{E}_{\tilde{a}|\tilde{z}}\mathbb{E}_{y|\tilde{a}}\left[y\right].$
(4.34)

We call this estimator an instrumental variable estimator and call the extra
variable $z$ an instrument. Unlike the earlier approaches such as regression
and IPW, this approach is almost guaranteed to give you a biased estimate of
the causal effect.

Assume we are provided with
$D=\left\\{(a_{1},y_{1},z_{1}),\ldots,(a_{N},y_{N},z_{N})\right\\}$ after we
are done estimating those functions above. We can then get the approximate
causal effect of the action of interest $\hat{a}$ by

$\displaystyle\hat{y}_{\mathrm{IV}}(\hat{a})\approx\frac{\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})\mathbb{E}_{\epsilon^{\prime\prime}_{a},\epsilon^{\prime}_{y}}\hat{f}_{y}(\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a}),\epsilon^{\prime}_{y})}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})}.$
(4.35)

Additionally, if we are provided further with
$D_{\bar{z}}=\left\\{(a^{\prime}_{1},y^{\prime}_{1}),\ldots,(a^{\prime}_{N^{\prime}},y^{\prime}_{N^{\prime}})\right\\}$,
we can use $\hat{g}_{z}(a,\epsilon^{\prime}_{z})$ to approximate this
quantity, together with $D$. Let $(a_{n},y_{n},r_{n},z_{n})\in\bar{D}$ be

$\displaystyle(a_{n},y_{n},r_{n},z_{n})=\begin{cases}(a_{n},y_{n},1,z_{n}),&\text{
if }n\leq N\\\
(a^{\prime}_{n-N^{\prime}+1},y^{\prime}_{n-N^{\prime}+1},0,-1),&\text{ if
}N<n\leq N+N^{\prime}\end{cases}$ (4.36)

Then,

$\displaystyle\hat{y}_{\mathrm{IV}}(\hat{a})\approx\sum_{n=1}^{N+N^{\prime}}\frac{\mathds{1}(a_{n}=\hat{a})\mathbb{E}_{\epsilon^{\prime\prime}_{a},\epsilon^{\prime}_{y},\epsilon^{\prime}_{z}}\left[r_{n}\hat{f}_{y}(\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a}),\epsilon^{\prime}_{y})+(1-r_{n})\hat{f}_{y}(\hat{g}_{\tilde{a}}(\hat{g}_{z}(a_{n},\epsilon^{\prime}_{z}),\epsilon^{\prime\prime}_{a}),\epsilon^{\prime}_{y})\right]}{\sum_{n^{\prime}=1}^{N+N^{\prime}}\mathds{1}(a_{n^{\prime}}=\hat{a})}.$
(4.37)

In the former case, we must solve two regression problems, finding
$\hat{f}_{y}$ and $\hat{g}_{\tilde{a}}$. When we do so by solving a least
squares problem for each, we end up with two least squares problems that must
be solved sequentially. Such a case is often referred as two-stage least
squares. In the latter case, we benefit from extra data by solving three
regression problems. Though, in most cases, we choose an easy-to-obtain
instrument so that it is often enough to solve two regression problems.

There are two criteria that need to be considered when choosing an instrument
$z$. First, the instrument must be independent of the confounder $x$ a priori.
If this condition does not hold, we end up with the following graph:

$z$$a$$y$$x$

The undirected edge between $z$ and $x$ indicates that they are not
independent. In this case, even if we manage to remove the edge between $a$
and $x$, there is still a spurious path $a\leftarrow z\leftrightarrow x\to y$,
that prevents us from avoiding this bias. This first criterion is therefore
the most important consideration behind choosing an instrument.

##### Instrumental variables must be predictive of the action.

The second criterion is that $z$ be a cause of $a$ together with $x$. That is,
a part of whatever cannot be explained by $x$ in determining $a$ must be
captured by $z$. We can see why this is important by recalling the IPW-based
estimate against the instrumental variable based estimate. The IPW-based
estimate from Eq. (4.9) is reproduced here as

$\displaystyle\hat{y}_{\mathrm{IPW}}(\hat{a})\approx\frac{\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})\frac{y_{n}}{\hat{p}(\hat{a}|x_{n})}}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})}.$
(4.38)

If we contrast it with the instrument variable based estimate in Eq. (4.35),
we get

$\displaystyle\hat{y}_{\mathrm{IPW}}(\hat{a})-\hat{y}_{\mathrm{IV}}(\hat{a})=\frac{\sum_{n=1}^{N}\mathds{1}(a_{n}=\hat{a})\overbrace{\left(\frac{y_{n}}{\hat{p}(\hat{a}|x_{n})}-\mathbb{E}_{\epsilon_{y},\epsilon^{\prime\prime}_{a}}\hat{f}_{y}(\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a}),\epsilon_{y})\right)}^{\mathrm{(a)}}}{\sum_{n^{\prime}=1}^{N}\mathds{1}(a_{n^{\prime}}=\hat{a})},$
(4.39)

where we assume
$D=\left\\{(a_{1},y_{1},x_{1},z_{1}),\ldots,(a_{N},y_{N},x_{N},z_{N})\right\\}$.

There are two estimates within (a) above that result in a bias. Among these
two, $\hat{f}_{y}$ and $\hat{g}_{\tilde{a}}$, the former does not stand a
chance of being an unbiased estimate, because it is not given the unbiased
estimate of the action nor the covariate $x$. Contrast this with the
regression-based estimate above where $\hat{f}_{y}$ afforded to rely on the
true (sampled) action and the true (sampled) covariate. The latter is however
where we have a clear control over.

Assume that $f^{*}_{y}$ is linear. That is,

$\displaystyle
f^{*}_{y}(a,x,\epsilon_{y})=a^{\top}\alpha^{*}+x^{\top}\beta^{*}+\epsilon_{y},$
(4.40)

where $\alpha$ and $\beta$ are the coefficients. If $\mathbb{E}[x]=0$ and $a$
is selected independent of $x$,

$\displaystyle f^{*}_{y}(a,\epsilon_{y})=a^{\top}\alpha^{*}+\epsilon_{y}.$
(4.41)

The term (a), with the assumption that
$\frac{y_{n}}{\hat{p}(a_{n}|x)}=f^{*}_{y}(a_{n})$, can then be expressed as

$\displaystyle\left(a_{n}-\mathbb{E}_{\epsilon^{\prime\prime}_{a}}\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a})\right)^{\top}\alpha^{*}-\mathbb{E}_{\epsilon^{\prime\prime}_{a}}\left[\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a})\right]^{\top}r_{\alpha},$
(4.42)

where $r_{\alpha}$ is the error in estimating $\alpha^{*}$, i.e.,
$\hat{\alpha}=\alpha^{*}+r_{\alpha}$.

For brevity, let
$\hat{g}(z_{n})=\mathbb{E}_{\epsilon^{\prime\prime}_{a}}\hat{g}_{\tilde{a}}(z_{n},\epsilon^{\prime\prime}_{a})$,
which allows us to rewrite it as

$\displaystyle\left(a_{n}-\hat{g}(z_{n})\right)^{\top}\alpha^{*}-\hat{g}(z_{n})^{\top}r_{\alpha}.$
(4.43)

Let us now look at the overall squared error:

$\displaystyle\frac{1}{M}\sum_{m=1}^{M}\left(a_{m}^{\top}\alpha^{*}-\hat{g}(z_{m})^{\top}(\alpha^{*}-r_{\alpha})\right)^{2},$
(4.44)

where we use $m$ to refer to each example with $a_{m}=\hat{a}$. If we further
expand the squared term,

$\displaystyle\frac{1}{M}\sum_{m=1}^{M}{\alpha^{*}}^{\top}a_{m}a_{m}^{\top}\alpha^{*}+(\alpha^{*}-r_{\alpha})^{\top}\hat{g}(z_{m})\hat{g}^{\top}(z_{m})(\alpha^{*}-r_{\alpha})-2{\alpha^{*}}^{\top}a_{m}\hat{g}^{\top}(z_{m})(\alpha^{*}-r_{\alpha})$
(4.45)
$\displaystyle\approx{\alpha^{*}}^{\top}\mathbb{E}[aa^{\top}]\alpha^{*}+(\alpha^{*}-r_{\alpha})^{\top}\mathbb{E}[\hat{g}(z)\hat{g}^{\top}(z)](\alpha^{*}-r_{\alpha})-2{\alpha^{*}}^{\top}\mathbb{E}[a\hat{g}^{\top}(z)](\alpha^{*}-r_{\alpha}).$
(4.46)

The first term is constant. It simply tells us that the error would be greater
if the variance of the relevant dimensions of the action on the outcome, where
the relevancy is determined by $\alpha^{*}$, is great, the chance of mis-
approximating it would be simply great as well. The second term tells us that
the error would be proportional to the variance of the relevant dimensions of
the predicted action on the outcome, where the relevancy is determined by the
predicted coefficient, $\alpha^{*}-\hat{r}$. That is, if the variance of the
predicted outcome is great, the chance of a large error is also great. The
third term is where we consider the correlation between the true action and
the predicted action, again along the dimensions of relevance.

This derivation tells us that the instrument must be selected to be highly
predictive of the action (the third term) but also exhibit a low variance in
its prediction (the second term). Here comes the classical dilemma of bias-
variance trade-off in machine learning.

##### The linear case.

The instrument variable approach is quite confusing. Consider a 1-dimensional
fully linear case here in order to build up our intuition. Assume

$\displaystyle x\leftarrow\epsilon_{x}$ (4.47) $\displaystyle
a\leftarrow\gamma x+\epsilon_{a}$ (4.48) $\displaystyle y\leftarrow\alpha
a+\beta x+\epsilon_{y},$ (4.49)

where $\epsilon_{x}$ and $\epsilon_{y}$ are both zero-mean Normal variables.
If we intervene on $a$, we would find that the expected outcome equals

$\displaystyle\mathbb{E}[y|\mathrm{do}(a)]=\alpha a,$ (4.50)

and thereby the ATE is

$\displaystyle\mathrm{ATE}=\mathbb{E}[y|\mathrm{do}(a=1)]-\mathbb{E}[y|\mathrm{do}(a=0)]=\alpha.$
(4.51)

With a properly selected instrument $z$, that is, $z\perp\\!\\!\\!\perp x$, we
get

$\displaystyle z\leftarrow\epsilon_{z}$ (4.52) $\displaystyle
a\leftarrow\gamma x+\psi z+\epsilon^{\prime}_{a}.$ (4.53)

Because $z\perp\\!\\!\\!\perp x$, the best we can do is to estimate $\psi$ to
minimize

$\displaystyle\min_{\psi}\sum_{n=1}^{N}(a_{n}-\psi z_{n})^{2},$ (4.54)

given $N$ $(a,z)$ pairs. The minimum attainable loss is

$\displaystyle(\gamma x)^{2},$ (4.55)

assuming zero-mean $\epsilon^{\prime}_{a}$, because the contribution from $x$
cannot be explained by the instrument $z$.

With the estimated $\hat{\psi}$, we get

$\displaystyle\hat{a}\leftarrow\hat{\psi}z+\epsilon^{\prime\prime}_{a},$
(4.56)

and know that

$\displaystyle\hat{a}=a-\gamma x$ (4.57)

on expectation.

By plugging in $\hat{a}$ into the original structural causal model, we end up
with

$\displaystyle x\leftarrow\epsilon_{x}$ (4.58)
$\displaystyle\hat{a}\leftarrow\hat{\psi}z+\epsilon^{\prime\prime}_{a}$ (4.59)
$\displaystyle y\leftarrow\tilde{\alpha}\hat{a}+\beta
x+\epsilon_{y}^{\prime}.$ (4.60)

We can now estimate $\tilde{\alpha}$ by minimizing

$\displaystyle\sum_{n=1}^{N}\left(y_{n}-\tilde{\alpha}\hat{a}_{n}\right)^{2},$
(4.61)

assuming both $\epsilon_{x}$ and $\epsilon_{y}^{\prime}$ are centered.

If we assume we have $x$ as well, we get

$\displaystyle\sum_{n=1}^{N}\left(y_{n}-\tilde{\alpha}(a_{n}-\gamma
x_{n})\right)^{2}$ (4.62)
$\displaystyle=\sum_{n=1}^{N}\left(y_{n}-\alpha(a_{n}-\gamma
x_{n})+(\alpha-\tilde{\alpha})(a_{n}-\gamma x_{n})\right)^{2}$ (4.63)
$\displaystyle=\sum_{n=1}^{N}\left(\epsilon_{y,n}+(\alpha-\tilde{\alpha})\epsilon_{a,n}^{\prime}\right)^{2}$
(4.64)
$\displaystyle=\sum_{n=1}^{N}\left(\epsilon_{y,n}^{2}+(\alpha-\tilde{\alpha})^{2}(\epsilon_{a,n}^{\prime})^{2}+2(\alpha-\tilde{\alpha})\epsilon_{y,n}\epsilon_{a,n}^{\prime}\right)$
(4.65)
$\displaystyle=_{n\to\infty}\mathbb{V}[\epsilon_{y}]+(\alpha-\tilde{\alpha})^{2}\mathbb{V}[\epsilon_{a}^{\prime}].$
(4.66)

The first term is irreducible, and therefore we focus on the second term. The
second term is the product of two things. The first one,
$(\alpha-\tilde{\alpha})^{2}$, measures the difference between the correct
$\alpha$ and the estimated effect. It tells us that minimizing this loss
w.r.t. $\tilde{\alpha}$ is the right way to approximate the true causal effect
$\alpha$.

We can plug in $\hat{a}$ from Eq. (4.59) instead:

$\displaystyle\sum_{n=1}^{N}\left(y_{n}-\tilde{\alpha}(\hat{\psi}z_{n}+\epsilon^{\prime\prime}_{a,n})\right)^{2}$
(4.67)
$\displaystyle=\sum_{n=1}^{N}\left((y_{n}-\tilde{\alpha}\hat{\psi}z_{n})+\tilde{\alpha}\epsilon^{\prime\prime}_{a,n}\right)^{2}$
(4.68)
$\displaystyle=\sum_{n=1}^{N}\left((y_{n}-\tilde{\alpha}\hat{\psi}z_{n})^{2}+\tilde{\alpha}^{2}(\epsilon^{\prime\prime}_{a,n})^{2}+2(y_{n}-\tilde{\alpha}\hat{\psi}z_{n})\tilde{\alpha}\epsilon_{a,n}^{\prime\prime}\right)$
(4.69)
$\displaystyle=_{n\to\infty}\sum_{n=1}^{N}(y_{n}-\tilde{\alpha}\hat{\psi}z_{n})^{2}+\tilde{\alpha}^{2}\mathbb{V}[\epsilon^{\prime\prime}_{a}].$
(4.70)

The first term is about how predictive the instrument $z$ is of $y$, which is
a key consideration in choosing the instrument. If the instrument is not
predictive of $y$, the instrument variable approach fails dramatically. The
second term corresponds to the variance of the action not explained by the
instrument, implying that the instrument must also be highly correlated with
the action.

In this procedure, we have solved least squares twice, (4.54) and (4.61),
which is a widely used practice with instrument variables. We also saw the
importance of the choice of the instrument variable.

##### An example: taxation

One of the most typical example of an instrument is taxation. It is
particularly in the United States of America (USA), due to the existence of
different tax laws and rates across fifty states. For instance, imagine an
example where the action is cigarette smoking, the outcome is the contraction
of lung cancer and the confounder is an unknown genetic mutation that both
affects the affinity to nicotine addiction and the incidence of a lung cancer.
Because we do not know such a genetic mutation, we cannot easily draw a
conclusion about the causal effect of cigarette smoking on lung cancer. There
may be a spurious correlation arising from this unknown, and thereby
unobserved, genetic mutation. Furthermore, it is definitely unethical to
randomly force people to smoke cigarettes, which prevents us from running an
RCT.

We can instead use state-level taxation on tobacco as an instrument, assuming
that lower tax on tobacco products would lead to a higher chance and also rate
of smoking, and vice versa. First, we build a predictor of smoking from the
state (or even county, if applicable) tax rate. The predicted amount of
cigarettes smoked by a participant can now work as a proxy to the original
action, that is the actual amount of cigarettes smoked. We then build a
predictor of the incidence of lung cancer as well as the reverse predictor
(action-to-instrument prediction). We can then use one of the two instrument
variable estimators above to approximate the potential outcome of smoking on
lung cancer.

###  4.4 Summary

In this chapter, we have learned the following concepts:

  1. 1.

Challenges in active causal inference: practical, ethical and legal challenges

  2. 2.

When confounders were observed: Regression, inverse probability weighting and
matching

  3. 3.

When confounders were not observed: instrument variables

There are a few other widely used passive causal inference algorithms, but
they are left for the final section on §6 Remaining Topics, such as
difference-in-difference, regression discontinuity and double machine
learning.

##  Chapter 5 Causality and Machine Learning

In this chapter, we finally delve in to the ‘machine learning’ side of this
course, which is titled ‘Introduction to Causal Inference in Machine
Learning’. In order to do so, we need to start by establishing when we do not
need to think of causal inference, or more broadly causality, in machine
learning. After establishing it, we will move on to the other extreme, where
conventional machine learning cannot do anything on its own. We then try to
incorporate some of the concepts we have learned so far, in order to land
between these two extreme cases and solve some of the most challenging and
important problems in modern machine learning.

###  5.1 Out-of-Distribution Generalization

####  5.1.1 Setup: I.I.D.

We must start by defining what we mean by ‘prediction’. In this particular
course, we first assume that each and every input-output pair $(x,y)$, input
$x$ or output $y$ is sampled independently of each other. This is a pretty
strong assumption, since the world often changes based on what we have seen,
because those who saw a sample pair may and often do change their behaviors.
For instance, consider building a stock price forecasting model. Once you use
a predictor to predict whether the price of a particular stock goes up or down
and trade based on the outcome, the next input $x$, that is the stock of your
next interest, is not anymore independently selected but based on your own
success/failure from the previous trade.

This assumption is however also reasonable, because there are many phenomena
in which our behaviours do not matter much in a reasonably short horizon. For
instance, consider installing and using a bird classifier at a particular
forest. With a fixed camera, the input to this classifier will be largely
independent of which birds (or not) were seen earlier, although spotting of a
particular bird may attract poachers to this forest who would dramatically
affect the bird population in a longer time frame.

Next, we assume that all these pairs are drawn from the ‘identical’
distribution. This is similar if not identical to the stationarity assumption
from RCT. In RCT, we often rely on a double blind experiment design, in order
to ensure that the causal effect $p^{*}(y|a,x)$ does not change over the
trial. In this section as well as conventional statistical learning theory, we
assume all input-output pairs were drawn from the same distribution.

Combining these two assumptions, we arrive at a so-called training set $D$
which satisfies

$\displaystyle p(D)=\prod_{(x,y)\in D}p^{*}(x,y),$ (5.1)

according to the definition of independence. We do not have access to nor have
knowledge of $p^{*}$. We use this training set $D$ for both model fitting
(training) and selection (validation).

Once the predictive model $\hat{p}$ is ready, we deploy it to make a
prediction on a novel input $x^{\prime}$ drawn from a distribution $q^{*}$.
That is,

$\displaystyle\hat{y}\sim\hat{p}(y|x^{\prime}),$ (5.2)

where $(x^{\prime},y^{\prime})\sim q^{*}$. We are often not given
$y^{\prime}$. After all, $y^{\prime}$ is what we want to use our predictive
model to infer.

We say that the predictive model is accurate, if the following quantity is
low:

$\displaystyle R(\hat{p})=\mathbb{E}_{(x^{\prime},y^{\prime})\sim
q^{*}}\left[l(y^{\prime},\hat{p}(y|x^{\prime}))\right],$ (5.3)

where $l(\cdot,\cdot)\geq 0$ is the loss (misclassification rate).

In traditional statistical learning theory, $q^{*}$ is assumed to be $p^{*}$,
and under this assumption, the goal of designing a learning algorithm is to
minimize a so-called excess risk:

$\displaystyle R_{\mathrm{excess}}(\hat{p})=R(\hat{p})-R(p^{*})$ (5.4)

with respect to $\hat{p}$. Since we do not have access to $p^{*}$, we often
use Monte Carlo approximation to compute $R(\hat{p})$, as follows

$\displaystyle
R(\hat{p})\approx\hat{R}_{N}(\hat{p})=\frac{1}{N}\sum_{n=1}^{N}l(y_{n},p(y|x_{n})),$
(5.5)

where $(x_{n},y_{n})\sim p^{*}$.

With a (strong) assumption of uniform convergence, which is defined as

$\displaystyle\sup_{\hat{p}}\left|R(\hat{p})-\hat{R}_{N}(\hat{p})\right|\to_{p}0,$
(5.6)

we can minimize $R$ using $\hat{R}$ with a large enough data set, i.e.,
$N\to\infty$, and find a good predictive model $\hat{p}$. Of course, since $N$
is always finite in reality, there is almost always non-zero generalization
error.

Since we never have access to $R(\hat{p})$ even after learning, it is a usual
practice to use a separate (held-out) set of examples again drawn from the
same distribution $p^{*}=q^{*}$ as the test set to approximate the
generalization error of a trained model $\hat{p}$. Let
$D^{\prime}=\left\\{(x^{\prime}_{1},y^{\prime}_{1}),\ldots,(x^{\prime}_{K},y^{\prime}_{K})\right\\}$.
Then,

$\displaystyle
R(\hat{p})\approx\frac{1}{K}\sum_{k=1}^{K}l(y^{\prime}_{k},\hat{p}(y|x^{\prime}_{k})).$
(5.7)

Such a test-set accuracy, or more simply a test accuracy, has been a workhorse
behind rapid advances in machine learning over the past several decades.

With this whole paradigm in your mind, it is important to notice that the key
assumption here is $q^{*}(x,y)=p^{*}(x,y)$. In other words, we assume that an
instance a predictive model would be tested in the deployment would follow the
same distribution as that from which the training examples were drawn, i.e.,
$q^{*}(x)=p^{*}(x)$. Furthermore, the conditional distribution over the
outcome does not change either, i.e., $q^{*}(y|x)=p^{*}(y|x)$. In this case,
there is no reason for us to consider the underlying generating process behind
$p^{*}$ nor $q^{*}$ separately.

####  5.1.2 Out-of-Distribution Generalization

##### Impossibility of Out-of-Distribution (ood) generalization.