# A Survey of Methods, Challenges and Perspectives in Causality

Gaël Gendron  0000-0002-2457-934X University of AucklandAucklandNew Zealand
<EMAIL_ADDRESS>,  Michael Witbrock  0000-0002-7554-0971 University
of AucklandAucklandNew Zealand<EMAIL_ADDRESS>and  Gillian Dobbie
0000-0001-7245-0367 University of AucklandAucklandNew Zealand
<EMAIL_ADDRESS>
###### Abstract.

Deep Learning models have shown success in a large variety of tasks by
extracting correlation patterns from high-dimensional data but still struggle
when generalizing out of their initial distribution. As causal engines aim to
learn mechanisms independent from a data distribution, combining Deep Learning
with Causality can have a great impact on the two fields. In this paper, we
further motivate this assumption. We perform an extensive overview of the
theories and methods for Causality from different perspectives, with an
emphasis on Deep Learning and the challenges met by the two domains. We show
early attempts to bring the fields together and the possible perspectives for
the future. We finish by providing a large variety of applications for
techniques from Causality.

††copyright: none

##  1\. Introduction

One motivation for studying Causality comes from the sentence ”correlation
does not imply causation”, meaning that it is not sufficient to have two
statistically correlated variables X and Y to deduce that one is causing the
other. The Common Cause Principle (Reichenbach, 1956) brings restriction to
this statement by saying that either X causes Y, Y causes X, or a third
variable Z is the cause of both. However, this is not enough to fully
characterize a causal relationship, and different theories have been developed
to this end. Many fields of science aim to find causal relationships, e.g.
social sciences (Duncan, 2014; Dündar-Coecke et al., 2022), economics
(Haavelmo, 1943), neuroscience (French and DeAngelis, 2020). This explains the
great interest in the topic of Causality and the variety of approaches
tackling it.

Deep Learning (DL) has been very successful in a large variety of tasks by
extracting correlation patterns from high-dimensional data. However, current
methods still struggle at generalizing in a distribution different from their
training data. This can be explained by the presence of confounding effects in
the data, leading to spurious correlations. The field of Causality studies the
effects of such confounders and aims to find systematic ways to learn general
causal effects invariant of the data distribution. Therefore, it can be
hypothesised that adding Causality theory to DL can improve performance and
generalization capabilities. However, the fields of Deep Learning and
Causality developed separately, until recently when a growing interest in
merging the two fields has arisen (Schölkopf and von Kügelgen, 2022).

These observations motivate this survey of Causality theories and techniques.
We look at it from the perspective of Machine Learning but attempt to cover a
large spectrum of methods that are applied to various domains that do not
typically intersect. We cover views of Causality for static data and time
series and present the domains in which they are applied. We also describe the
challenges encountered by Causality and Deep Learning and how techniques
combining the two fields have been applied to solve them. Many open questions
remain to be solved, we introduce them and describe the current perspectives
in the research literature on these topics to answer them. We summarize our
contributions here:

  * •

We conduct an extensive survey of the field of Causality within the Structural
Causal Model, broadened with the Potential Outcome Framework and Granger
Causality, domains usually considered separately.

  * •

We motivate the use of Causality for Deep Learning and the possible future
perspectives in the field.

  * •

We find and describe several connections between Causality and Deep Learning
including Graph Neural Networks, sparse architectures, data augmentation, and
disentanglement.

  * •

We provide a detailed list of works and domains of application for Causality.

To the best of our knowledge, these contributions are the first to gather the
various approaches of Causality and bring together the applications using
them. We also make original connections between Causality and with Deep
Learning. In contrast, the survey by Zanga et al. (2022) focuses on causal
structure discovery and Yao et al. (2021) discuss only causal inference
approaches. Surveys with a broader range (Glymour et al., 2019; Guo et al.,
2021; Nogueira et al., 2022; Vowels et al., 2023) include both tasks but do
not cover the most recent Deep Learning techniques. Additionally, Moraffah et
al. (2021) discuss methods for time-series and Kaddour et al. (2022) provide
another extensive survey of Machine Learning methods for Causality.

The rest of the paper is structured as follows: In Section 2, we introduce the
concepts and terminologies of the main causality theories. In Section 3, we
motivate and introduce Neural Causal Models, a novel framework for causal
reasoning based on structural causal models and taking advantage of Deep
Learning techniques. In Sections 4 and 5, we perform an extensive survey of
causality frameworks and methods for causal structure discovery and causal
inference from a computer science perspective. Section 6 introduces another
causal task that has arisen with the advent of Deep Learning, called causal
representation learning, and introduces the recent techniques developed in the
domain. We present applications of Causality in various fields of science in
Section 7. We provide a conclusion to our work in Section 8.

Categories and subcategories |  Papers
---|---
Causal Structure Discovery |  Stationary |  Graphical |  Constraints (4.1) |  (Verma and Pearl, 1990; Spirtes et al., 1989, 2000; Colombo and Maathuis, 2014; Ramsey et al., 2006; Bühlmann et al., 2010; Tsamardinos et al., 2006; Spirtes et al., 2000; Colombo et al., 2012; Rohekar et al., 2021; Ogarrio et al., 2016)
|  |  Scores (4.2) |  (Cooper and Herskovits, 1992; Meek, 1997; Chickering, 2002; Hauser and Bühlmann, 2011; Ramsey et al., 2017; Ogarrio et al., 2016; Zheng et al., 2018)
|  |  Non-Gaussian (4.3) |  (Shimizu et al., 2006; Zhang and Hyvärinen, 2010, 2009; Wang and Boddeti, 2022)
|  |  Variational (4.4) |  (Annadani et al., 2021)
|  |  LLMs (4.5) |  (Zecevic et al., 2023; Jin et al., 2023)
Time-Series |  Graphical |  Constraints (4.1) |  (Entner and Hoyer, 2010; Runge et al., 2019)
Causal Inference |  Stationary |  Do-Calculus |  (5.1) |  (Pearl, 2009; Huang and Valtorta, 2006)
|  Learning |  Re-weighting (5.2) |  (Rosenbaum and Rubin, 1983; Robins et al., 1994; Imai and Ratkovic, 2014)
|  |  Matching (5.3) |  (Austin, 2011; Rosenbaum and Rubin, 1983; Stuart, 2010; King and Nielsen, 2019; Li et al., 2016; Iacus et al., 2012; Yao et al., 2021)
|  |  Decision Trees (5.4) |  (Loh, 2011; Chipman et al., 2010)
|  |  NCMs (5.5) |  (Zecevic et al., 2021a, b)
|  |  Variational (5.6) |  (Kipf and Welling, 2016; Zecevic et al., 2021b; Sánchez-Martin et al., 2022; Louizos et al., 2017; Wu and Fukumizu, 2021; Geffner et al., 2022)
Time-Series |  Learning |  Transformers (5.7) |  (Melnychuk et al., 2022)
|  |  VARs (5.8) |  (Nogueira et al., 2022)
Causal Representation Learning |  Stationary |  Disent-anglement |  Features (6.1) |  (Higgins et al., 2018; Kim and Mnih, 2018; Locatello et al., 2019; Mathieu et al., 2019; Chen et al., 2018; Higgins et al., 2017; Whitney, 2016; Achille and Soatto, 2018; Esmaeili et al., 2019; Kumar et al., 2018; Locatello et al., 2020a; Khemakhem et al., 2020; Lachapelle et al., 2022; Gresele et al., 2021)
|  |  Domains (6.1) |  (Hadad et al., 2018; Bau et al., 2019; Peng et al., 2019; Gabbay et al., 2021; Zhang et al., 2021b)
|  Augmentation |  (6.2) |  (Geiger et al., 2022; Zhang et al., 2018; Verma et al., 2019; Mangla et al., 2020; Balestriero et al., 2022)
Table 1. Causality taxonomy for the listed papers.

##  2\. Causality Theory

Causality is the field of research aiming to find systematic methods for
uncovering cause-effect relationships. It aims to answer the following
questions: what evidence is needed to infer a causal link, and what can we
infer from it? (Pearl, 2009). Causality has been studied in several domains of
science and philosophy (Woodward, 2005), but we will restrict ourselves to the
approaches from computer science.

The recovery of causal relationships can be divided into two tasks: causal
structure recovery and causal inference. Causal structure recovery attempts to
extract and represent the causal structure of the information from an
unstructured source (e.g. tabular data). This task is so far exclusively
tackled by graphical models like the Structural Causal Model (SCM) (Pearl,
2009). In contrast, causal inference aims to answer causal queries, i.e.
estimate the values of causal variables in given situations. Causal inference
methods either attempt to solve specific cases (e.g. answer a single query) or
generate functions that can solve all queries (e.g. estimate the mapping
functions of an SCM as explained in the next section). The term causal
inference has often been used to encompass both tasks (Pearl, 2009) but we
will distinguish them throughout this survey to provide more clarity on the
tasks each presented method aims to solve. The majority of the literature on
Causality tackles causal inference.

In this survey, we will focus on three paradigms for representing cause-
effects relationships: Pearl’s Structural Causal Model (Pearl, 2009), Rubin
Causal Model (Rubin, 1974), and Granger Causality (Granger, 2001). They are
discussed in Sections 2.1, 2.2, and 2.3 respectively.

###  2.1. Structural Causal Model

The Structural Causal Model (Pearl, 2009, 2014; Xia et al., 2021) is a
graphical model for representing cause-effect relationships. A graph
$\mathcal{G}=(\mathbf{V},\mathbf{E})$ is a set of nodes (or vertices)
$\mathbf{V}$ and edges $\mathbf{E}\subset\mathbf{V}\times\mathbf{V}$
connecting the nodes, with $N=|\mathbf{V}|$ the size of the set of nodes and
$M=|\mathbf{E}|$ the size of the set of edges. Edges can either be directed or
undirected. In an SCM, the edges are directed to represent causation and
written with an arrow $\rightarrow$. A directed path in a graph is a sequence
of nodes linked together by directed edges (following the direction of the
arrow). A cycle in a graph is a directed (non-empty) path where the first and
last nodes are identical. A graph containing only directed edges and without
cycles is called a Directed Acyclic Graph (DAG). Parent nodes are linked to
children nodes via an edge: $parent\rightarrow children$. The ancestors of a
node $V$ are the nodes that are on a directed path to $V$ and the descendants
of $V$ are on a directed path from $V$. The respective parent, children,
ancestor and descendent sets of a node $V$ are noted $\mathbf{pa}$,
$\mathbf{ch}$, $\mathbf{anc}$ and $\mathbf{desc}$. An SCM is a model based on
a DAG extended with a set of functions and a probability distribution, and
where the nodes are causal variables and the edges are causal dependencies
between the variables. Each node is associated with a function taking its
parents (causes) as inputs and computing an (effect) value for the node.

####  2.1.1. Formal definition

An SCM is defined as a tuple
$\mathcal{M}=\langle\mathbf{U},\mathbf{V},\mathcal{F},P(\mathbf{U})\rangle$
where:

  * •

$\mathbf{U}$ is the set of unobserved or exogenous variables

  * •

$\mathbf{V}$ is the set of observed or endogenous variables

  * •

$\mathcal{F}$ is a set of mapping functions
$\\{f_{V_{i}}\in\mathcal{F}:V_{i}\leftarrow
f_{V_{i}}(\mathbf{pa}_{V_{i}},\mathbf{U}_{V_{i}})\\}_{i\in 1\dots|V|}$,
$f_{V_{i}}$ computes the instance value of the endogenous variable $V_{i}$
given its causes: the instance values of the parent endogenous variables
$\mathbf{pa}_{V_{i}}\subset\mathbf{V}\setminus(V_{i}\cup\mathbf{desc}(V_{i}))$
and exogenous variables $\mathbf{U}_{V_{i}}\subset\mathbf{U}$

  * •

$P(\mathbf{U})$ is a probability function over the exogenous variables
$U_{i}\in\mathbf{U}$

The structure of an SCM, (i.e. the dependencies of each observed variable) can
be represented with a DAG. Figure 1 provides an example of an SCM. An SCM aims
to model the causal mechanisms underlying the observed variables $\mathbf{V}$.
The mechanisms of the unobserved variables $\mathbf{U}$ cannot be determined.
The consequences on the SCM are that variables of $\mathbf{U}$ have no parents
and are modelled by a probability distribution to represent uncertainty
instead of a deterministic function. The acyclicity assumption of the causal
graph is useful to disambiguate the causes from the effects, as we detail in
Section 4. A complete SCM can fully represent a causal system. However,
building the SCM of a set of variables is a challenging task requiring the
identification and estimation of the causal links between the variables
(structure discovery and inference). Methods for generating an SCM are
introduced in Sections 4 and 5.

[Structural Causal Model example]Example of Structural Causal Model (SCM).
Endogenous node V1 is connected to V2 by an edge. Exogenous node U1 is
connected to V1 and, similarly, U2 is connected to V2.

$\mathbf{U}=\\{U_{1},U_{2}\\}$
$\mathbf{V}=\\{V_{1},V_{2}\\}$

$\mathcal{F}=\begin{cases}V_{1}\leftarrow f_{V_{1}}(U_{1})\\\ V_{2}\leftarrow
f_{V_{2}}(V_{1},U_{2})\end{cases}$

$P(U_{i})\sim\mathcal{N}(0,1)~{}\forall U_{i}\in\mathbf{U}$

Figure 1. Example of Structural Causal Model (SCM) with two endogenous
variables $V_{1}$ and $V_{2}$ and two exogenous variables $U_{1}$ and $U_{2}$.
$\mathcal{F}$ is the set of mapping functions linking $V_{1}$, $V_{2}$ and
$U_{1}$, $U_{2}$ together. $P(\mathbf{U})$ is the probability function defined
over $U_{1}$ and $U_{2}$.

####  2.1.2. Hidden confounders and Markov-equivalence

Recovering the causal structure of a system of variables is challenging and
not always possible. First, the main challenge in causal tasks is dealing with
confounding effects. A hidden confounder is an exogenous variable that is a
shared cause of several endogenous variables. In Markovian graphs, the value
of an observed variable does not depend on other observed variables than its
parents (i.e. $V\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}\mathbf{anc}_{V}\setminus\mathbf{pa}_{V}|\mathbf{pa}_{V}$).
Graphs with hidden confounders are non-Markovian (also called semi-Markovian
if they are acyclic) because the exogenous factors $\mathbf{U}$ are not
independent and can affect multiple variables $\mathbf{V}$ distant in the
graph, creating non-Markovian dependencies (Xia et al., 2021). Second, even
without hidden confounders, the true causal structure may only be recoverable
up to a Markov-equivalent class. Markov-equivalent SCMs have a different
causal structure that yields the same observations.

[Markov relative DAG example]Example of Markov-relative DAG. S is connected to
T and T is connected to C. One exogenous node U1 is connected to S.

(a) Markov-relative DAG.

[Markov equivalent DAG example]Example of DAG Markov-equivalent to the
previous example. C is connected to T and T is connected to S. One exogenous
node U1 is connected to C.

(b) Markov-equivalent DAG with another cause.

[Markov relative DAG example with exogenous variables]Example of Markov-
relative DAG with exogenous variables. S is connected to T and T is connected
to C. Two exogenous nodes U1 and U3 are connected to S and C, respectively.

(c) Markov-relative DAG with exogenous variables.

[Semi Markov DAG example with single confounder]Example of Semi Markov DAG
with a single confounder. The graph is the same as the previous one DAG but
exogenous nodes U1 and U3 have been merged to a single node G, connected to S
and C, which acts as a hidden confounder if unobserved.

(d) Semi-Markovian DAG with exogenous variables.

Figure 2. Examples of Markov-equivalent and semi-Markovian (acyclic) DAG
causal structures. (2(a)) a Markov-relative DAG modeling the causal effects of
smoke (S) on the risk of cancer (C) with an intermediate variable
corresponding to the presence of tar (T) in the lungs. (2(b)) a Markov-
equivalent model that can be obtained with the same observational data
interpreted differently. (2(c)) another Markov-relative DAG where exogenous
variables forbid the previous interpretation. (2(d)) the same model where the
unobserved variables have been merged to a single latent confounder: a genetic
factor (G) causing both the tendency to smoke and the high risk of cancer. The
last graph is semi-Markovian.

Figure 2 illustrates these two notions: we take the example of smoke and
cancer. Smoking (S) is correlated with the presence of tar in the lungs (T)
and tar is correlated with a high risk of cancer (C). Figures 2(a) and 2(b)
show two Markov-equivalent graphs that can be obtained from the same
observations. In these examples, variables are systematically correlated
because they are all deterministic consequences of the same exogenous process.
If another source of stochasticity is involved as in Figure 2(c) (yielding a
different data distribution), the causal direction may be recoverable (e.g. T
always follows S but C is only sometimes observed). Another possible causal
structure may involve an unobserved common cause (e.g. genetics), as seen in
Figure 2(d) ($C\leftarrow G\rightarrow S\rightarrow T$ is another possible
graph). If we can find a single causal structure that matches the data, the
graph is identifiable. If two causal models match the observations, they may
be separated by their abilities to answer interventional and counterfactual
queries, as discussed in Section 2.1.3.

####  2.1.3. Pearl’s Causal Hierarchy

The purpose of the SCM is to provide a framework for answering causal queries.
These queries can be divided into three types and ordered into a hierarchy of
increasing difficulty. This hierarchy is called Ladder of Causation or Pearl’s
Causal Hierarchy (PCH) (Bareinboim et al., 2022) and is shown in Table 2.

Table 2. Pearl’s Causal Hierarchy. Each layer corresponds to a type of causal query with an increasing level of required data. Each layer is associated with a corresponding intuitive activity and a probabilistic formula. The formulae illustrate the simplest case for each layer, assuming the minimum number of variables required.  |  Layer |  Activity |  Formula
---|---|---|---
$\mathcal{L}_{1}$ |  Associational |  Seeing |  $P(y|x)$
$\mathcal{L}_{2}$ |  Interventional |  Doing |  $P(y|do(x))$
$\mathcal{L}_{3}$ |  Counterfactual |  Imagining |  $P(y|do(x),x^{\prime},y^{\prime})$

The first layer $\mathcal{L}_{1}$ in the hierarchy corresponds to
observational queries, i.e. queries based on data collected passively. They
are of the form: ”What does X tell us about Y?”, modeled by the conditional
probability $P(y|x)$, which can be solved using Bayesian statistics. Machine
Learning based on supervised and unsupervised methods can only answer this
type of query (Bareinboim et al., 2022).

The second layer $\mathcal{L}_{2}$ in the hierarchy corresponds to
interventional queries, i.e. queries about the effect of a change/intervention
on an environment. They are of the form: ”What will happen to Y if we force
X?”. A new operator is used to model this query mathematically, the do-
operator (Pearl, 1995). $\mathbf{do(X=x)}$ corresponds to the forced
assignment of the variable $\mathbf{X}$ to the value $\mathbf{x}$ (even if the
case is never observed). The notation $\mathbf{Y_{x}}$ is also often used for
$\mathbf{Y}|\mathbf{do(X=x)}$. This operator can be reduced into a conditional
probability using the rules of do-calculus, introduced in Section 5.
Reinforcement Learning can answer queries at this layer by interacting with
its environment (Bareinboim et al., 2022).

The third and last layer $\mathcal{L}_{3}$ represents counterfactual queries,
i.e. questions about an imagined situation. they are of the form: ”What would
have happened to Y if X had happened instead of X’ (which led to Y’)?”. This
type of query is the hardest to solve in the causal hierarchy as they require
reasoning on things that did not happen (”What if?” questions). However, by
marginalizing over the exogenous variables, the query can be reduced into the
following expression (for the discrete case, assuming the simplest case with a
single exogenous variable $U$) (Pearl, 2009; Bareinboim et al., 2022):

(1) $P(y|do(x),x^{\prime},y^{\prime})=\sum\limits_{u\in
U}P(y|do(x),u)P(u|x^{\prime},y^{\prime})$

The main idea consists in connecting the factual and counterfactual worlds
using the exogenous variables as shared evidence. The computation process is
divided into three steps: (1) abduction computes $P(u|x^{\prime},y^{\prime})$;
(2) action performs the intervention $do(x)$ and; (3) prediction computes the
final result using the operands.

Figure 3 illustrates each type of query with an example. A disease propagates
in a population with a mechanism not covered in our causal model and
represented by an exogenous variable $U_{1}$. When infected with the disease,
a patient will develop symptoms. The symptoms can be mild or severe depending
on environmental factors (e.g. healthcare access may reduce the severity of
the symptoms), not observed. The patient can then take a treatment to be
cured. The access to the treatment may depend on unobserved factors correlated
with the symptoms (e.g. the treatment may be accessible with the same
healthcare system). The environmental factor here acts as a hidden confounder.
The problem in this situation is assessing the effect of the treatment on the
probability of getting cured. In our model, the answer depends (or not) on the
treatment and the severity of the symptoms, but the two variables are not
independent as they share a common hidden cause.

(a) $\mathcal{L}_{1}$ query: What do the symptoms tell us about the disease?

[Causal Graph Example illustrating L1 query]Causal graph illustrating queries
of type L1. It is described in the text, each variable corresponds to a node
in the graph.

(b) $\mathcal{L}_{2}$ query: Will the patient be cured if he takes the
treatment?

[Causal Graph Example illustrating L2 query]Causal graph illustrating queries
of type L2. It is the same graph as for the L1 query except the links to the
treatment variable have been removed.

(c) $\mathcal{L}_{3}$ query: Would the patient had been cured if he had not
taken the treatment?

[Causal Graph Example illustrating L3 query]Causal graph illustrating queries
of type L3. The graph from L1 and L2 queries are represented next to each
other. All nodes from L1 have their L2 counterparts representing the
counterfactual scenario. Only the exogenous variables U1 and environment do
not change and are connected to both graph.

Figure 3. Causal graphs involved in the computation of layer $\mathcal{L}_{i}$
queries. In an $\mathcal{L}_{2}$ query, the intervened variable (represented
with a double edge) has its value forced into the graph and does not depend on
its parent variables. For the $\mathcal{L}_{3}$ query, the factual world and
the counterfactual world (variables with a *) are linked by the same exogenous
factors. The observations from the factual worlds are used to estimate the
$U_{1}$ and $env$ variables (abduction), then the counterfactual graph is
intervened on (action) and the final estimation is computed (prediction).

An SCM can accurately represent causal mechanisms. All query layers in the PCH
can be computed with a fully specified SCM. Under the SCM framework, causal
inference aims to estimate the true causal model as closely as possible.
However, recovering the true SCM is hard to achieve, and the Causal Hierarchy
Theorem (CHT) (Bareinboim et al., 2022) furthermore states that the amount of
information needed to solve a query at a layer $\mathcal{L}_{i}$ is not
sufficient in general to solve queries at layers $\mathcal{L}_{i+1}$. In
particular, the CHT states that for a causal model able to solve
$\mathcal{L}_{1}$ queries, it is always possible to find a second model that
yield the same responses to $\mathcal{L}_{1}$ queries but different responses
to $\mathcal{L}_{2}$ queries. Collapses can be observed between
$\mathcal{L}_{2}$ and $\mathcal{L}_{3}$ for exceptional SCMs but it is in
general possible to find two causal models with the same responses to
$\mathcal{L}_{2}$ but different responses to $\mathcal{L}_{3}$ queries. This
information acquisition problem is the main challenge in Causality as once
achieved, the problem can be reduced to probabilistic reasoning (Mossé et al.,
2021).

###  2.2. Rubin Causal Model

The Rubin Model, or Neyman-Rubin Causal Model, or Potential Outcome Framework
(POF) (Splawa-Neyman et al., 1990; Rubin, 1974; Yao et al., 2021), is a causal
model based on the idea of observed and counterfactual outcomes. As opposed to
the SCM, this model does not aim to recover the causal structure of the data
but to estimate the distribution of an outcome given the application of a
treatment variable. This situation is often encountered in medical
applications where researchers would like to estimate the effect of a
treatment on a disease. A standard method for achieving it is to perform a
Randomized Control trial where the treatment is randomly assigned to a group
of people to mitigate confounding effects, e.g. age, ethnicity, and health
status of the patients. However, this is a cost-intensive method and may not
be achievable in some situations for ethical, feasibility or physical reasons.
For instance, injecting medication with a high death rate into a treatment
group is unethical. Alternatively, when measuring the effects of an economic
decision on a country, it is infeasible to have a treated and a control group.
It is even impossible when working on a unique subject: after deciding whether
to take an action against climate change or not, only one of the two outcomes
is observed. The other, called counterfactual outcome can never be observed.
The latter example is an illustration of the fundamental problem of causal
inference: we would like to estimate the outcome of a situation that is not
observed.

####  2.2.1. Formal definition

In the Rubin model, a treatment is applied to a subject of interest, called
unit. The treatment variable is denoted $X\in\\{0,1,\dots,N_{X}\\}$ with
$N_{X}$ the number of possible treatments and $0$ representing the absence of
treatment. We note the random variable corresponding to the potential outcome
as $Y(X=x)$, $Y_{x}$ or $Y|do(x)$ to be consistent with the notation from the
previous section. $Y_{0}$ represents the outcome for the control group and
$Y_{1}$ the outcome for the group with treatment 1 (or the treated group if
only one treatment is studied). The background random variables, or
covariates, corresponding to the attributes of the patients are denoted $Z$.
We note that alternative notations can be used in the literature, we choose
this one to maintain a consistent theoretical notation with the one used by
the SCM. Within the POF, the goal of causal inference is to determine the
treatment effect at different levels of granularity: population level,
subgroup level, or individual level.

##### Average Treatment Effect (ATE)

The ATE, defined in Equation 2, is the expected average difference of effect
on the outcome between units taking the treatment and units not taking it.

(2) $\text{ATE}=\mathbb{E}[Y_{1}-Y_{0}]$

##### Average Treatment Effect on the treated group (ATT)

The ATT, defined in Equation 3, is the difference between the expected outcome
for the treated group and the counterfactual outcome if the same treated group
did not take the treatment.

(3) $\text{ATT}=\mathbb{E}[Y_{1}|X=1]-\mathbb{E}[Y_{0}|X=1]$

##### Conditional Average Treatment Effect (CATE)

The CATE, defined in Equation 4, is the difference of effect on the outcome
between units taking the treatment and units not taking it, given a subgroup
conditioned on a background variable $Z$.

(4) $\text{CATE}=\mathbb{E}[Y_{1}|Z=z]-\mathbb{E}[Y_{0}|Z=z]$

##### Individual Treatment Effect (ITE)

The ITE, defined in Equation 5, for a particular individual or unit i, is the
difference of effect on the outcome between a situation where the treatment is
applied and one where the treatment is not. In this last situation, it appears
that one of the situations cannot be observed as a single individual cannot
take the treatment and not take it at the same time. This counterfactual
outcome (which could be either $Y_{1}$ or $Y_{0}$) must be determined with
other methods.

(5) $\text{ITE}^{(i)}=Y_{1}^{(i)}-Y_{0}^{(i)}$

The goal of causal inference with the Rubin model is to compute the above
formulae, the final objective being the computation of the ITE for any given
individual. In order to achieve this goal, it is necessary to compute
counterfactual outcomes despite the lack of observations. Techniques have been
developed to this end, detailed in Section 5.

###  2.3. Granger Causality

The reader may have noticed that the notion of Causality we have mentioned so
far does not include any notion of time. It may seem surprising as it is a
fundamental notion in our intuitive understanding of causality and a solid
inductive bias when causal inference is conducted by the brain (Bramley et
al., 2018). However, as we have seen in the previous sections, time is not
central and not even needed in most popular causal frameworks. We are now
interested in the other end of the spectrum, causal models that rely on time-
sensitive information and apply causal learning to time series.

A time series is a set of variables with values evolving through time
$\mathbf{X}(t)=(x_{1}(t),x_{2}(t),\dots,x_{N}(t))$ where $t$ corresponds to
the current timestep, $\\{X_{i}\in\mathbf{X}\\}_{i\in 1\dots N}$ is the set of
$N$ time-sensitive variables, and $\mathbf{X}(t)$ is a snapshot of the values
for each variable of $\mathbf{X}$ at timestep $t$. Granger Causality (Granger,
2001) is a theory of Causality developed for time series. In this model, a
series $\mathbf{X}$ causes (or ”Granger-causes” (Moraffah et al., 2021))
$\mathbf{Y}$ if we can better predict $\mathbf{Y}(t)$ using all the
information available, denoted $\mathbf{U}$, than with all the information
deprived of the past values of $\mathbf{X}$:
$\mathbf{U}\setminus\\{\mathbf{X}(0),\mathbf{X}(1),\dots,\mathbf{X}(t-1)\\}$.
More formally: if
$\sigma^{2}_{t}(\mathbf{Y}|\mathbf{U})<\sigma^{2}_{t}(\mathbf{Y}|\mathbf{U}\setminus\\{\mathbf{X}(0),\mathbf{X}(1),\dots,\mathbf{X}(t-1)\\})$,
where $\sigma^{2}_{t}(\mathbf{A}|\mathbf{B})$ is the variance of the
predictive error
$\epsilon_{t}(\mathbf{A}|\mathbf{B})=\mathbf{A}(t)-P(\mathbf{A}(t)|\mathbf{B})$.
It is important to highlight that Granger Causality does not recover true
causes (Maziarz, 2015). In the above definition, if $\mathbf{X}$ and
$\mathbf{Y}$ are correlated but caused by a third hidden series $\mathbf{Z}$
($\mathbf{X}\leftarrow\mathbf{Z}\rightarrow\mathbf{Y}$), the Granger Causality
test will not differenciate it from the causal case
$\mathbf{X}\rightarrow\mathbf{Y}$.

Another line of methods for inferring causality from time series is based on
Information Theory and uses entropy measures to infer causal relationships.
The Shannon entropy (Shannon, 1948) of a variable $X$ can defined as shown in
Equation 6. similarly, we can define the joint entropy between two variables
$X$ and $Y$, shown in Equation 7, and the conditional entropy of $X$ given
$Z$, shown in Equation 8.

(6) $\displaystyle H(X)$ $\displaystyle=-\sum\limits_{x}P(x)\log P(x)$ (7)
$\displaystyle H(X,Y)$
$\displaystyle=-\sum\limits_{x}\sum\limits_{y}P(x,y)\log P(x,y)$ (8)
$\displaystyle H(X|Z)$
$\displaystyle=-\sum\limits_{x}\sum\limits_{z}P(x,z)\log P(x|z)$

Entropy measures the amount of uncertainty from a variable. The lower the
entropy, the easier the value of a random variable can be predicted. From
these definitions, we can define the Mutual Information between two variables
$X$ and $Y$, given $Z$:

(9) $I(X;Y|Z)=H(X|Z)+H(Y[Z)-H(X,Y|Z)$

The Mutual Information between two variables represents the amount of
information shared between the two. Methods based on Mutual Information
attempt to detect the flow of information in the causal system over time
(Palus et al., 2001). The above formulae are symmetric and therefore do not
account for the direction taken by the information. In order to alleviate this
problem, the concept of Transfer Entropy was introduced by Schreiber (2000) to
detect asymmetries between variables from the data. Transfer Entropy is
defined in Equation 10 (timesteps are subscript for simplicity).

(10) $T_{X\rightarrow
Y}=H(Y_{t}|Y_{t-1},Y_{t-2},\dots,Y_{t-K})-H(Y_{t}|Y_{t-1},\dots,Y_{t-K},X_{t-1},\dots,X_{t-K})$

Transfer Entropy is an asymmetric measure of the information flow from $X$ to
$Y$. It compares the entropy of process $Y$ at timestep $t$, given its past
values up to step $t-K$, and the entropy of $Y$ given its past values and the
ones of a second variable $X$. It can be used to recover the causal structure
between time series. Granger Causality and Transfer Entropy are two
interlinked concepts, particularly in the case of Gaussian variables where the
two measures are equivalent (Barnett et al., 2009). Transfer Entropy works
well with two processes but has trouble when multiple variables can influence
the system (Moraffah et al., 2021; Sun et al., 2015). Further work attempt to
overcome this issue by proposing new measures for causality in time series
(Vakorin et al., 2009; Vlachos and Kugiumtzis, 2010; Sun et al., 2015).

##  3\. Neural Causal Model

Causality offers a large set of tools for reasoning over causes and effects,
but despite theoretical correctness, these traditional approaches struggle
when dealing with high-dimensional variables. Conversely, Deep Learning has
been proven very successful at extracting patterns in large datasets and
applying them to various tasks. However, they do not generalize well out of
their initial distribution, show poor causal reasoning capabilities, and can
easily be fooled by confounding effects (Goodfellow et al., 2015; Koh et al.,
2021; Goyal and Bengio, 2020; Schölkopf and von Kügelgen, 2022). It seems
natural to combine the best of both worlds, but interestingly, the two fields
have arisen in separate ways until very recently. Attempts have been made to
generate neural models capable of causal reasoning, but these approaches are
still stammering and how to develop causal reasoning engines at scale remains
an open problem. This section focuses on the potential benefits of causal
reasoning engines and the theory that has been developed. Sections 4 and 5
will introduce causal structure discovery and inference methods based on the
framework along methods based on the farameworks from Section 2. Section 6
will address the problem of representation learning under a causal
perspective, i.e. the problem of learning causal variables from low-level
data.

###  3.1. Motivations

As mentioned in Section 2.1.3, traditional Machine Learning methods based on
supervised learning rely only on observations. The CHT (Bareinboim et al.,
2022) states that such models cannot answer causal queries in general (queries
in the second and third layers in the hierarchy of Table 2). Adding Causality
theory into Machine Learning techniques, and Deep Learning (DL) in particular,
could help leverage this issue and bring additional improvements to the field.
We summarize the potential impact causal models can bring to Deep Learning.

####  3.1.1. Generalization

First, modern supervised learning techniques rely on the assumption that the
data are independent and identically distributed (i.i.d). This assumption can
lead to brittle models (Goodfellow et al., 2015; Koh et al., 2021; Goyal and
Bengio, 2020; Schölkopf and von Kügelgen, 2022) in environments where it does
not hold. The i.i.d assumption ensures that the learned patterns can be
applied in general cases. However, datasets can be biased or not
representative enough of real-world distributions. One model may perform well
on one image dataset but poorly on another one where the distribution is
slightly different (e.g. a model trained on photographs may not perform well
on sketches). Models are not only prone to bias, but can also be fooled by
spurious examples, adversarial attacks and distribution shifts (Goyal and
Bengio, 2020; Schölkopf et al., 2021; Schölkopf and von Kügelgen, 2022). To
counter this effect, many techniques have been attempted, the most successful
consisting in increasing as much as possible the size of the model and the
data, leading to the data-centered approach to DL, dominated by the Foundation
models (Bommasani et al., 2021). Causal models, on the other hand, can be
considered model-centered approaches as they do not rely on the absence of
spurious effects in the data. Causal models aim to represent the causal
structure of data-generating mechanisms, invariant to the way the data is
distributed. Moreover, they acknowledge the spurious correlations and the
confounding effects and can take them into account, as seen in Section 2,
without needing much data. Causal models have recently proven to less overfit
and yield better, faster, and more robust results in low-data regimes
(Scherrer et al., 2022). Incorporating causal engines into DL tasks can lead
to more robust models, less prone to bias, and with better generalization.

####  3.1.2. Modularity

Second, DL engines are usually black boxes made of dense and entangled
computations. It makes them hardly interpretable and very computation-
intensive. A great line of work argues that modular architectures would help
tackle those two downsides and learn transferable mechanisms that could be
reused from one task to the next (Goyal et al., 2020; Rahaman et al., 2021;
Bommasani et al., 2021). In particular, the Independent Causal Mechanism (ICM)
principle (Peters et al., 2017; Schölkopf et al., 2021) is a postulate stating
that causal mechanisms are autonomous and do not interact with each other. The
Sparse Mechanism Shift (SMS) hypothesis (Schölkopf et al., 2021) furthermore
adds that small changes in the distribution generate only local changes in a
causal model, not affecting all factors. Dense computations should not be
necessary and only yield additional spurious correlations. Recent work showed
that modular neural networks have a much lower gradient magnitude and work
better on tasks with sparse causal structure (Scherrer et al., 2022). Within
the DL field, several attempts have been made to generate such sparse and
modular architectures (Goyal et al., 2020; Fedus et al., 2021; Du et al.,
2021; Rahaman et al., 2021). They mostly rely on Mixtures-of-Experts and use
routing mechanisms for selecting the right experts. These architectures are
composed of stacks of experts and do not have a graph structure. Their
modularity is still quite limited. Graphical causal models, on the other hand,
are inherently based on a graph structure, usually sparse, that would be very
appropriate to this end. Theories from neuroscience also posit that the brain
works in a modular way. Each module is an expert in a domain and communicates
sparsely with others experts, transmitting only few information (Baars, 1997;
Essington, 2022). Causal models composed of a DAG with nodes sparsely
exchanging information would be closer to the actual behavior of the brain
(Goyal and Bengio, 2020).

####  3.1.3. Interventions and Counterfactuals

Third, as mentioned at the beginning of this section, DL models currently do
not have the ability to answer queries of the type ”what would happen if we do
X? What if Y had happened instead?” which correspond to causal queries of
layer $\mathcal{L}_{2}$ and $\mathcal{L}_{3}$, as specified by the CHT in
Section 2.1.3. Online Reinforcement Learning (RL) techniques can interact with
the environment and by this mean directly perform interventions. Therefore,
they can answer some layer $\mathcal{L}_{2}$ queries but causal reasoning may
help improve the capabilities of RL models at reasoning in an imagined space
and answer counterfactuals queries (Dasgupta et al., 2019; Schölkopf et al.,
2021). Going further, we can state that counterfactual reasoning is a critical
key towards more human-like AI agents. An essential step towards this goal is
believed to be the generation of World Models, i.e. models able to understand
the inner mechanisms of their surroundings and perform reasoning and planning
in their environment (Schmidhuber, 2015; Ha and Schmidhuber, 2018; Matsuo et
al., 2022).

###  3.2. The Neural Causal Model

The Neural Causal Model (NCM) (Xia et al., 2021) is a theoretical framework
providing a link between SCMs and neural networks. An NCM is defined,
similarly as the SCM defined in Section 2.1.1, as a triple
$\mathcal{M}(\boldsymbol{\theta})=\langle\mathbf{U},\mathbf{V},\mathcal{F},P(\mathbf{U})\rangle$,
where:

  * •

as for the SCM, $\mathbf{U}$ is the set of exogenous variables

  * •

as for the SCM, $\mathbf{V}$ is the set of endogenous variables

  * •

as for the SCM, $\mathcal{F}$ is a set of mapping functions computing the
effect value of a variable given its causes (instance values of its parent
endogenous variables
$\mathbf{pa}_{V_{i}}\subset\mathbf{V}\setminus(V_{i}\cup\mathbf{desc}(V_{i}))$
and exogenous variables $\mathbf{U}_{V_{i}}\subset\mathbf{U}$), the NCM
restricts the class of functions to feedforward neural networks:
$\\{f_{i}\in\mathcal{F}:v_{i}\leftarrow
f_{V_{i}}(\theta_{V_{i}};\mathbf{pa}_{V_{i}},\mathbf{U}_{V_{i}})\\}_{i\in
1\dots|V|}$, $f_{V_{i}}$

  * •

$\boldsymbol{\theta}=\\{\theta_{V_{i}}:V_{i}\in\mathbf{V}\\}$ is the set of
parameters for the mapping functions

  * •

as for the SCM $P(\mathbf{U})$ is the probability function defined over the
noise variables in $U_{i}\in\mathbf{U}$, the NCM furthermore constraints the
probability law to $\mathbf{u_{i}}\sim\text{Unif}(0,1)~{}\forall
U_{i}\in\mathbf{U}$

In a nutshell, NCMs are a subclass of SCMs where the mapping functions
representing the causal relationships between the variables are limited to the
class of parametrized neural networks. The initial paper (Xia et al., 2021)
furthermore restrict the distribution of exogenous variables to a uniform law,
and the mapping functions to Multi-Layer Perceptrons (MLPs). As MLPs are
universal function approximators and all distributions can be estimated from
the uniform law (Angus, 1994), this restriction guarantees that the NCM class
has the same expressivity as the broader SCM class. NCMs are able to represent
queries at layers $\mathcal{L}_{1}$, $\mathcal{L}_{2}$ and layers
$\mathcal{L}_{3}$ (observations, interventions and counterfactual) (Xia et
al., 2021). In practice, however, causal inference is intractable with MLPs
(Zecevic et al., 2021a). Subsequent works relax these two assumptions. This
choice can leads to a loss in expressivity but yields learnable models. We
also emphasize that works combining neural models with causal engines do not
necessarily follow the NCM principles.

##  4\. Causal Structure Discovery

Causal structure discovery is the task of learning the causal relationships in
a set of variables, whether or not two variables are causally linked, and the
direction of the causation when this is possible. The structure discovery
methods can be divided into multiple categories: Constraint-based, score-
based, and non-Gaussian methods are the traditional approaches to causal
structure discovery. Some non-Gaussian methods also take advantage of neural
models. In constrast, Variational and Transformer approaches rely on recent
Deep Learning techniques.

###  4.1. Constraint-based methods

The Constraint-based causal structure discovery methods rely on independence
tests to create causal connections between variables (Nogueira et al., 2022)
and build an SCM. They are based on the Inductive Causation (IC) algorithm
(Verma and Pearl, 1990). In the original article, this algorithm can be used
in the presence of hidden confounders. However, following the separation made
in Pearl (2009) and in order to help comprehension, we first introduce a
version of IC in an unconfoundedness regime, i.e. we assume that there are no
hidden confounders in the model.

The IC algorithm has three steps. The first step builds undirected causal
dependencies. An undirected graph is constructed from the set of (observed)
variables of interest V. Edges between two variables $A$ and $B$ are added to
the graph if it is not possible to find a set $S_{AB}\subset V\times
V\setminus{A,B}$ s.t. $A\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}B|S_{AB}$. In other words, $A$ and $B$ share a
dependency that does not depend on other variables. This condition is
considered sufficient to state that $A$ and $B$ are causally linked. The
second and third steps find the direction of causality. The second step
creates $\nu\text{-}structures$, i.e. structures with the shape $A\rightarrow
C\leftarrow B$ (also called colliders). For each pair of nonadjacent variables
$(A,B)$ which have a neighbor $C$ in common ($A-C-B$), if $C\notin S_{AB}$, it
can be deduced that $C$ does not have a causal effect on $A$ and $B$ and
cannot be a cause for one of them. The three variables can, therefore, only
have a $A\rightarrow C\leftarrow B$ structure. The third step orients the
remaining possible edges. If orienting an edge in one direction creates a
cycle, the edge has to be oriented in the other direction. The process is also
applied if it creates a $\nu\text{-}structure$, as all structures should be
discovered in the second step.

The steps of the IC algorithm are abstract and do not offer a direct
implementation. Many algorithms will build upon the ideas developed in IC and
provide practical implementation of the method. A first application is the SGS
algorithm, which has exponential complexity in the worst case (Spirtes et al.,
1989). One of the most popular is the Peters and Clark (PC) algorithm (Spirtes
et al., 2000), which runs in polynomial time. Improvements attempt to reduce
the complexity and increase the stability of PC by reducing the number of
conditional independence tests and relaxing assumptions on the order of nodes,
like PC-stable (Colombo and Maathuis, 2014), conservative-PC (Ramsey et al.,
2006), PC-select (called PC-simple in the original paper) (Bühlmann et al.,
2010), or MMHC (Tsamardinos et al., 2006), a PC-based method guided with a
min-max heuristic. All the above methods assume that no hidden confounders are
hidden in the model. This assumption helps simplify the problem but is rarely
true in practice.

We now introduce methods for recovering the graph structure in the presence of
hidden confounders. If the performed tests are statistically significant,
algorithms in the unconfoundedness regime can recover the causal DAG structure
up to a Markov-equivalence class (Meek, 1995; Verma and Pearl, 1990; Flesch
and Lucas, 2007). This is no longer the case in the confoundness regime as
there may not be a DAG structure that can account for the observed data. The
IC algorithm with hidden confounders (called IC* in Pearl (2009)) has its
third step modified to account for confounding effects: edges are left
undirected when the causal relationship $A-B$ cannot be disambiguated between
$A\rightarrow B$, $B\rightarrow A$ and $A\leftarrow Z\rightarrow B$, with $Z$
a hidden confounder.

The Fast Causal Inference (FCI) algorithm (Spirtes et al., 2000) provides a
way to implement IC in the presence of hidden confounders, and the Real Fast
Causal Inference (RFCI) algorithm (Colombo et al., 2012) reduces its running
time. More recently, the Iterative Causal Discovery (ICD) method (Rohekar et
al., 2021) reduces the number of conditional tests needed by working
iteratively. The algorithm starts with a full graph and progressively removes
wrong edges. Independence tests are conditioned on a local neighborhood, with
the size of this neighborhood increasing at each step, but as the graph
becomes sparser at each step with the removal of the edges, the causal
inference remains fast even with an extensive set of variables in the
neighborhood.

Finally, these approaches have also been adapted to time series. The Time
series Fast Causal Inference (TsFCI) algorithm (Entner and Hoyer, 2010) is an
adaptation of the FCI algorithm to time series. The method uses a sliding
window to divide the series into several data frames, each with its own set of
variables, where the variables are treated without their time component. Steps
from the same time series are separate variables within a frame, so additional
rules are added to consider this background knowledge. More recently, PCMCI
(Runge et al., 2019) combines the PC algorithm with Momentary Conditional
Independence testing. The framework is divided into two stages: a first stage
removes parent candidates for each variable until a minimal set is reached,
and a second stage creates the causal links from this small amount of
variables.

###  4.2. Score-based methods

The score-based methods compute a likelihood score for each potential causal
graph given a set of variables. As the size of the search space of possible
graphs increases exponentially with the number of nodes, the main challenge of
this class of algorithms is to prune the search space (Nogueira et al., 2022).
The joint probability distribution over the variables V with a causal
structure given by a graph $\mathcal{G}$ can be given by a Bayesian network
$\mathcal{B}$, as shown in Equation 11. The methods described in this section
attempt to recover the true DAG $\mathcal{G}$ matching the distribution.

(11)
$P_{\mathcal{B}}(V_{1}=v_{1},V_{2}=v_{2},\dots,V_{N}=v_{N})=\prod\limits_{i=1}^{N}P(V_{i}=v_{i}|\mathbf{pa}_{V_{i}}^{\mathcal{G}})$

An early search method is the K2 procedure (Cooper and Herskovits, 1992). It
is based on a greedy search on parent nodes. Parents are added to a node if
adding them increases the probability of the structure. However, the scoring
done by K2 is not strictly equivalent to conditional independence as they
assume priors on the data distribution (Chickering, 2002). The popular Greedy
Equivalence Search (GES) method (Meek, 1997; Chickering, 2002) iteratively
constructs the causal graph from an empty DAG in two stages: in the first
stage, an edge is added at each step if it increases the score of the DAG. The
second stage starts when the score no longer increases. An edge is removed at
each step if it increases the score of the DAG. Like the constraint-based
methods, this method is exact and returns the optimal DAG (up to an equivalent
class, assuming no confounders). The Greedy Interventional Equivalence Search
(GIES) (Hauser and Bühlmann, 2011) improves this method by performing
interventions, restricting the equivalent solution class to a smaller subset
that remains valid when performing interventions. The Fast Greedy Equivalence
Search (fGES) (Ramsey et al., 2017) modifies the GES algorithm to allow
parallelization and scaling of the algorithm to a high number of variables.
These methods are an efficient way to compute a causal model, but they all
assume no hidden confounders, the Greedy Fast Causal Inference (GCFI) removes
this restriction by combining GES with FCI (Ogarrio et al., 2016). In
constrast with the previous works which evolve in the discrete domain, the
Non-combinatorial Optimization via Trace Exponential and Augmented lagRangian
for Structure learning (NOTEARS) model (Zheng et al., 2018) considers DAG
learning as a continuous optimization problem. The node values are represented
using a feature matrix $\mathbf{X}$ and the edges of the graph are represented
with a continuous weighted adjacency matrix $\mathbf{W}$. The method further
assumes that the relationships between the causal variables are linear.
Following this assumption, the loss function to minimize can be simply defined
as $\mathcal{L}(\mathbf{W})=\lVert\mathbf{X}-\mathbf{X}\mathbf{W}\rVert_{2}$.
A regularization term enhances the sparsity of the graph and a constraint
ensures the graph is acyclic (and nontrivial). the discrete adjacency matrix
can be obtained by considering an edge exists if its corresponding weight in
$\mathbf{W}$ is non-zero. This approach alleviates the issue of the
combinatorial search space of DAGs. However, Kaiser and Sipos (2022) pointed
out that NOTEARS leads to biased graphs when the causal variables have
different variances and argue that the method is not suited for causal
discovery.

###  4.3. Non-Gaussian methods

Constraint-based and score-based methods are the fundamental approaches to
causal discovery. With the exception of NOTEARS, all rely on statistical
tests. However, more recently, new directions have been studied to tackle the
problem without the need for statistical tests, which are computation-
intensive and require a lot of data to be statistically significant (Glymour
et al., 2019). The proposed methods take inspiration from the field of
Independent Component Analysis (ICA) (Hyvärinen and Oja, 2000) and rely on the
assumption that the data follows a non-Gaussian distribution. This assumption
allows extracting asymmetry in the distribution of causes and effects in the
data and determining the causal structure. These methods furthermore assume
that no confounders are present in the model.

A first work, the Linear Non-Gaussian Acyclic Model (LiNGAM) (Shimizu et al.,
2006) represents the causal links between variables as linear functions with
the form:

(12)
$v_{i}=\sum\limits_{v_{j}\in\mathbf{pa}_{v_{i}}}b_{ij}v_{j}+\epsilon_{i}+c_{i}$

where any variable in the model is a linear function of its parents.
$\epsilon_{i}$ is a disturbance term, and $c_{i}$ is an additional constant.
The disturbance term accounts for noise (or exogenous variables in the SCM),
and all $\epsilon_{i}$ are assumed independent (no confounders assumption).
The second and most crucial assumption in LiNGAM is that the distribution of
the $\epsilon_{i}$ is non-Gaussian. This restriction comes from the field of
Independent Component Analysis (ICA) where this assumption is used to separate
linear combinations of independent variables. If the mixture follows a
Gaussian distribution, it is not possible to recover the variables that
compose them (a consequence of the central limit theorem). If the mixture and
all the underlying variables are non-Gaussian, it becomes possible to use
their asymmetries to tear them apart. Going back to our problem, in matrix
form, the above equation can be written as:

(13) $\displaystyle\mathbf{v}$ $\displaystyle=\mathbf{Bv}+\mathbf{\epsilon}$
(14) $\displaystyle\mathbf{v}$
$\displaystyle=(1-\mathbf{B})^{-1}\mathbf{\epsilon}$ (15)
$\displaystyle\mathbf{v}$ $\displaystyle=\mathbf{A\epsilon}$

The problem to solve can then be reduced to computing the values of the matrix
$\mathbf{A}$, which is achieved in LiNGAM using ICA: all mixtures
$v\in\mathbf{v}$ are decomposed into their independent (cause) variables from
$\mathbf{v}\setminus v$. A direct extension of LiNGAM is the Post-Nonlinear
model (PNL) (Zhang and Hyvärinen, 2010, 2009) which replaces the linear
mapping with a composition of nonlinear functions:
$v_{i}=f_{2}(f_{1}(v_{j})+\epsilon_{i})$.

Traditional methods for causal structure discovery and causal inference work
well when the number of dimensions is low and sufficient data is provided to
apply statistical tests. However, despite the improvements in causal structure
discovery, it remains hard to recover a causal structure at scale (Schölkopf
et al., 2021). To alleviate this issue, we investigate neural methods for
structure discovery. Despite its name, the Neural Causal Inference Network
(NCINet) (Wang and Boddeti, 2022) is a causal structure discovery framework
for two variables settings based on neural networks. The resulting graph can
therefore be causal ($X\rightarrow Y$), anticausal ($X\leftarrow Y$), or
unassociated ($X~{}Y$). The core idea is that the mean squared error of a
regression in the causal direction should be lower than a regression in the
anticausal direction, i.e. if $X$ causes $Y$ then $y=f_{1}(x)$ should have a
better predictive power than $x=f_{2}(y)$. A neural encoder generates hidden
representations $\mathbf{H}_{X}$ and $\mathbf{H}_{Y}$ for variables $X$ and
$Y$ and feeds them to two regressors, attempting to learn functions $f_{1}$
and $f_{2}$ respectively. The result is given to a classifier that outputs the
type of relation between the two variables (causal, anticausal or
unassociated).

###  4.4. Variational Models

The Variational Causal Network (VCN) (Annadani et al., 2021) is a method for
learning the adjacency matrix of a causal graph iteratively from data. The
graph is constructed autoregressively by recurrently feeding it to a Long-
Short Term Memory (LSTM) network $q_{\theta}$, parametrized by $\theta$, that
generates the graph $\mathcal{G}$. The model maximizes the following Evidence
Lower Bound (ELBO) loss:

(16) $\mathcal{L}=\mathbb{E}_{q_{\theta}(\mathcal{G})}[\log
P(\mathcal{D}|\mathcal{G})]-\text{KL}(q_{\theta}(\mathcal{G})||P(\mathcal{G}))$

where $\mathcal{D}$ is the data distribution, and $P(\mathcal{D}|\mathcal{G})$
represents the probability of obtaining the data given the causal graph. The
first term of the loss verifies that $\mathcal{G}$ can yield the data
distribution $\mathcal{D}$. $P(\mathcal{G})$ is the prior distribution of the
generated causal graph, i.e. the prior knowledge about the causal graph (here,
acyclic graph). The second term uses Kullback-Leibler divergence to ensure
that the generated graph follows the prior distribution. The VCN uses a
modified version of the ELBO loss used by variational models (Kingma and
Welling, 2014). The latter are also used for causal inference (Section 5.6)
and causal representation learning (Section 6.1).

###  4.5. Large Language Models

A recent body of work studied the capabilities of large Language Models (LLMs)
at performing causal structure discovery. The studies focus on in-context
learning (Xie et al., 2022), i.e. they prompt the models to answer questions
from a short description of the problem and a few example. No explicit
training is performed on the given task. Zecevic et al. (2023) performed
experiments on multiple language models, including the popular GPT-3 (Brown et
al., 2020). When prompted to tell if a causal link exists between two
concepts, the LLM responses were mostly accurate. However, they can change
significantly depending on the wording used in the prompt, with knowledge
being inconsistent. While these experiments showed that current LLMs do not
exhibit robust causal reasoning capabilities, they also highlight the
difficulty to assess if a model understand the causal links or recite them. In
particular, the authors argue that LLMs can exhibit good performance on
$\mathcal{L}_{2}$ and $\mathcal{L}_{3}$ queries of the CHT by learning the
structure of the true causal graph in their training data. To assess if LLMs
can discover new causal structures (not leaked in the training data), Jin et
al. (2023) proposes an abstract causal discovery evaluation benchmark. The
authors show that LLMs, including GPT-3, perform badly. If fine-tuned on a
training set, their performance improves greatly but this increase does not
hold if the distribution of the test set differs from the one of the training
set, highlighting that LLMs rely mostly on non-causal clues.

###  4.6. Connex Problems

Although, it is not the main focus of this section, we take a brief look at
the problem of retrieving causal structures by merging several sources of data
(Claassen and Heskes, 2010; Tillman and Spirtes, 2011; Triantafillou and
Tsamardinos, 2015; Mooij et al., 2020). This is known as the structural causal
marginal problem (Gresele et al., 2022) (i.e. merging several SCMs sharing
some unknown nodes in common). Some works also attempt to generate causal
graphs with cycles (Rothenhäusler et al., 2015; Forré and Mooij, 2018; Bongers
et al., 2021; Ahsan et al., 2022) or identify the invariant mechanisms in
nonstationary data (Huang et al., 2020; Schölkopf and von Kügelgen, 2022).

##  5\. Causal Inference

Causal structure discovery methods can recover the links between variables but
does not provide a way to estimate the causal quantities and explain how
variables of interest causally affect each others. We describe causal
inference as the problem of computing causal effects (for observational,
interventional or counterfactual events) or building the functions that can
compute them. In this section, we provide extensive overview and explanations
of causal inference methods.

###  5.1. Do-calculus

The do-calculus (Pearl, 2009) establishes inference rules for reducing
interventional queries to a probabilistic formulation solvable with
observations only. Do-calculus does not directly provide a computation model
for causal inference but is a tool that can be used in conjunction with such a
model to estimate causal effects. Do-calculus can be applied once the causal
structure of the problem at hand has been found. The three rules of do-
calculus are described in Equations 17, 18, and 19.

  * •

Rule 1 Deletion of observation

(17) $P(y|do(x),z,w)=P(y|do(x),w)\text{~{}if~{}}(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X}}}$

  * •

Rule 2 Reduction of intervention to observation

(18)
$P(y|do(x),do(z),w)=P(y|do(x),z,w)\text{~{}if~{}}(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X}\underline{Z}}}$

  * •

Rule 3 Deletion of intervention

(19)
$P(y|do(x),do(z),w)=P(y|do(x),w)\text{~{}if~{}}(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X},\overline{Z(W)}}}$

where $\mathcal{G}$ is the causal DAG describing the structure of the problem.
The reduction rules are based on the structure of $\mathcal{G}$ and
d-separation (Pearl, 1989). A node $Z$ (or a set of nodes, representing causal
variables) d-separates (or blocks the path between) two nodes $X$ and $Y$ if
the latter are independent conditioned on $Z$ (for all distributions
represented by the graph $\mathcal{G}$).

Rule 1 (17) states that the observation of $Z$ can be ignored if it is
independent from $Y$ given some background interventions (on $X$) and
observations ($W$). $\mathcal{G}_{\overline{X}}$ is the graph $\mathcal{G}$
under intervention on $X$, i.e. with the incoming links of $X$ removed. We
note that $X$ and $W$ can also be (potentially empty) sets of variables. Rule
2 (18) states that intervening on $Z$ is the same as observing $Z$ if they do
not share a common cause (or share a cause already d-separated). If removing
directed paths from $Z$ to $Y$ ($\mathcal{G}_{\underline{Z}}$ removes the
outgoing links of $Z$) makes them independent, it means that the only path
from $Z$ to $Y$ is directed. As the $\mathbf{do}(Z)$ operation forces the
value of $Z$ and disconnects it from its causes, the intervention has no
impact. Rule 3 (19) states that an intervention on $Z$ can be ignored if there
is not directed path from $Z$ to $Y$. $\mathcal{G}_{\overline{Z(W)}}$ removes
the incoming links of $Z$ if $Z$ is not an ancestor of $W$. This is to avoid
catching $Y\rightarrow Z$ paths. These rules allow us to estimate the values
of some variables under interventional settings using observations only.
Whether or not the query can be answered depends on the local shape of the
graph. A query is identifiable if and only if it can be reduced to
observations using the three rules of do-calculus (Huang and Valtorta, 2006).
Many identifiability and unidentifiability patterns have been discussed in
Pearl (2009). The best known are referred to as the front-door and back-door
paths, illustrated in Figure 4.

(a) Front-door path

[Front door path example]Example of front door path. A node X is connected to
Z, itself connected to Y. A confounder C is connected to both X and Y.

(b) Back-door path

[Back door path example]Example of back door path. A node X is connected to Y
and a node Z is connected to X and Y. A confounder C is connected to both X
and Z.

Figure 4. Examples of front-door and back-door paths. The hidden confounder
$C$ is not known but the query $P(y|do(x))$ can be answered nonetheless using
do-calculus under these graph structures.

Under the front-door criterion, the do-calculus rules can be derived into the
following formula:

(20) $P(y|do(x))=\sum_{z\in Z}P(z|x)\sum_{x^{\prime}\in
X}P(y|x^{\prime},z)P(x^{\prime})$

And under the back-door criterion, the do-calculus rules can be derived into
the following formula:

(21) $P(y|do(x))=\sum_{z\in Z}P(y|x,z)P(z)$

We continue the example shown on Figure 2 with now the aim to estimate the
values of the variables of interest. Having a potential causal graph linking
smoke (S) to cancer (C), can we estimate the effect of smoke on cancer? Or do
they have an unobserved genetic common cause (G)? If no additional information
is available (Figure 5(a)), it is not possible to conclude from observations
only. Performing an interventional study (via a Randomized Control Trial) is
necessary. However, if we possess additional information, like the presence of
tar (T) in the lungs as shown on Figure 5(b), we are in the condition of a
front-door path. We can use the formula of Equation 20 to solve the query:
$P(c|do(s))=\sum\limits_{t}P(t|s)\sum\limits_{s^{\prime}}P(c|s^{\prime},t)P(s^{\prime})$,
corresponding to the question ”what is the probability a patient gets a cancer
if he starts smoking?”. It can be noted that the ATE or ITE (from the Rubin
model) corresponds exactly to the answer we are looking for as it compares
$P(c|do(s))$ with $P(c|do(\neg s))$. The do-calculus can fully answer causal
queries, but it requires prior knowledge about the causal graph structure and
the data distribution, limiting its application in practice.

(a) Unidentifiable model

[Unidentifiable causal graph example]Example of unidentifiable causal graph.
Node S is connected to C but a confounder G is connected to both S and C.

(b) Identifiable (front-door path)

[Identifiable causal graph example]Example of identifiable causal graph. Node
S is connected T, itself is connected to C. Confounder G is connected to both
S and C but not to T.

Figure 5. Example of causal graph. Smoke (S) is a parent cause of cancer (C),
a hidden variable (G), corresponding to genetic factors, acts as a confounding
effect. The model becomes identifiable if a causal link can be made with an
intermediate variable (T), tar in the lungs.

###  5.2. Re-weighting

We now discuss techniques developed to perform causal inference without a
graphical model. Instead, we aim to train a model (e.g. with Machine Learning)
to compute interventional queries $P(y|do(x))$ from data. Re-weighting methods
assign weights to samples in the data to counter selection bias and get closer
to an i.i.d distribution. Cases that happen rarely will be attributed a high
weight while cases often occurring will have a low weight. The implicit
assumption behind re-weighting methods is that there is a back-door path
(Figure 4(b)) where the $Z$ variable corresponds to the distribution biases.
This category of techniques approximates the back-door formula of Equation 21
where $P(z)$ is the weight associated with the current sample ($x$, $y$). Re-
weighting approaches can be understood from the perspective of the Rubin
model. In this context, the goal is to distribute background variables
identically with respect to treatment. For instance, in a medical application,
if the data is mostly composed of male patients with few female patients, then
the re-weighting will assign a high weight to females. If other background
variables are imbalanced, they should be corrected as well. This is typically
achieved using a balancing score (Yao et al., 2021). A widely used scoring
metric is the propensity score (Rosenbaum and Rubin, 1983). It corresponds to
the probability of receiving the treatment $X=1$, given the covariates $z$,
and is defined by:

(22) $e(z)=P(X=1|Z=z)$

A common way to assign a weight to each sample is to use an Inverse Propensity
Weighting (IPW) estimator (Rosenbaum and Rubin, 1983). The weight $w$ of a
given sample corresponds to a division by the propensity score if the patient
is treated ($X=1$) and a division by the inverse probability of the propensity
score if the patient is not treated ($X=0$). It is defined by:

(23) $w=\dfrac{X}{e(z)}+\dfrac{1-X}{1-e(z)}$

Other estimators work on increasing the robustness of the model if the
propensity score cannot be easily estimated, like the Doubly Robust (DR)
estimator (Robins et al., 1994), or better estimating the propensity score
like the Covariate Balancing Propensity Score (CBPS) (Imai and Ratkovic,
2014). However, the problem remains hard to solve in the presence of hidden
confounders.

###  5.3. Matching

Matching methods attempt to reproduce the conditions of a randomized trial by
dividing the samples into subgroups (or units) of identical covariates.
Inference is then performed in a singele subgroup. Each unit in the treated
group is matched with another unit in the control group with the same
background variables. The intuitive idea behind matching is that, since
samples within the same subgroup have the same covariates, only treatment
application should differentiate treated and control units (Austin, 2011; Yao
et al., 2021). A popular technique for assigning a sample to a subgroup is the
Propensity Score Matching (Rosenbaum and Rubin, 1983). The propensity score is
used as a distance metric for group assignment:
$D(z_{i},z_{j})=|e(z_{i})-e(z_{j})|$. This assignment can be performed using
nearest-neighbor search, subclassification, or weighting adjustment (Stuart,
2010). However, recent work showed that PSM could create imbalance, model
dependence, and bias (King and Nielsen, 2019). Other matching methods do not
use the propensity score and can alleviate this problem (Li et al., 2016;
Iacus et al., 2012; Yao et al., 2021).

###  5.4. Decision Trees

Decision trees are another popular structure for causal inference. We follow
the Potential Outcome Framework settings and aim to predict the outcome $Y$ of
a treatment given background variables $Z=z$. As for matching methods, the
principle relies on learning functions operating within a single group of
background covariates $z$. A famous example of such models is the
Classification and Regression Trees model (CART) (Loh, 2011), described as
$Y=g(z;\mathcal{T},\mathbf{M})+\epsilon$, where
$\mathbf{M}=\\{\theta_{1},\dots,\theta_{B}\\}$ is a set of parameters, each
associated with one of $B$ terminal nodes of the decision tree $\mathcal{T}$.
Regression is performed between $Z$ and $Y$, represented by the linear
function $g(\cdot)$. The function is parameterized by the $\theta_{i}$ values
corresponding to the terminal node reached (depending on the $z$ values).
$\epsilon\sim N(0,\sigma^{2})$ is a noise term. The trees and parameters are
learned using Markov Chain Monte-Carlo (MCMC) algorithm. The Bayesian Additive
Regression Tree (BART) (Chipman et al., 2010) model is an extension of CART
based on Random Forests. Instead of relying on a single tree, BART is composed
of $T$ trees and sums the output of each one:

(24) $Y=\sum\limits_{j=1}^{T}g(x;\mathcal{T}_{j},\mathbf{M}_{j})+\epsilon$

Relying on the combination of several models reduces the tree sizes. Each tree
is much less complex than the original tree in CART methods and behaves as a
weak classifier. As several trees are used, BART can incorporate many
covariate effects, thus being more robust than single-tree models and yielding
great accuracy (Yao et al., 2021).

###  5.5. Neural Causal Models

An improvement of the NCM described in Section 3, called Tractable NCM (TNCM)
(Zecevic et al., 2021a), replaces the MLPs with Sum-Product Networks (SPNs)
(Poon and Domingos, 2011; Sánchez-Cauce et al., 2022). An SPN is a DAG with a
single root and with leaf nodes representing a probability distribution over
causal variables $\mathbf{X}$. In an SPN, the root node represents the output
node. In the context of causal inference, it contains the value $y$ of the
query $P(y|x)$. Intermediate nodes represent either sum or product
parameterized operations, alternatively.

The GNN-SCM (Zecevic et al., 2021b) is another NCM replacing the parametric
functions for the causal inference (i.e. the MLPs) with Graph Neural Networks.
A Graph Neural Network (GNN) (Scarselli et al., 2009; Kipf and Welling, 2017)
is a particular type of neural network operating on a graph structure. A layer
of a GNN takes a graph as input, composed of nodes with feature vectors and an
edge list, and returns an aggregation of the node features according to a
parametric function and the graph structure. The general form of the GNN
equation is described in Equation 25:

(25)
$\mathbf{h}_{V}^{(l+1)}=\phi_{\mathbf{\theta_{2}}}\left(\mathbf{h}_{V}^{(l)},\bigoplus_{U\in\mathcal{N}_{V}}\gamma_{\mathbf{\theta_{1}}}(\mathbf{h}_{V}^{(l)},\mathbf{h}_{U}^{(l)})\right)$

where the vector $\mathbf{h}_{V}^{(l)}$ represents the node features of node
$V$ at layer $(l)$, $\mathcal{N}_{V}$ is the set of neighbor nodes of $V$
(nodes $U$ with an outgoing edge $U\rightarrow V$), and $\bigoplus$ is the
generic aggregation operator. For example, it can represent a sum $\sum$ or
product $\prod$. The GNN layer is composed of the functions $\gamma$ and
$\phi$, parametrized by $\theta=\\{\theta_{1},\theta_{2}\\}$.

The graph structure of the GNN can naturally be used in an SCM: the causal
graph corresponds to the input graph of the GNN, and each node is a causal
variable. The neighbors of a node $V$ correspond to its parents
$\mathbf{pa}_{V}$ in the DAG and the output of the GNN is
$P(V|\mathbf{pa}_{V})$. An intervention on a node $V$ in a GNN can naturally
be defined following the GNN formula described above and depriving the set
$\mathcal{N}_{V}$ of the parents of $V$. The new set of neighbors is defined
as $\mathcal{M}_{V}=\\{j|j\in\mathcal{N}_{V}\setminus\mathbf{pa}_{V}\\}$. As
opposed to the above models, the GNN-SCM uses the same GNN mapping function
for all node inferences. Only the input graph is modified to match the parents
$\mathbf{pa}_{V}$ of the node of interest $V$. However, using a single global
function to represent many local mechanisms makes optimization difficult. For
this reason, the authors do not consider counterfactual $\mathcal{L}_{3}$
queries in their work.

###  5.6. Variational Graph Auto-Encoders

The NCM provides a theoretical framework for building neural-based causal
models but is not the only way to represent causal relationships with deep
neural networks. We present another line of work in this direction, based on
Variational Auto-Encoders (VAEs) (Kingma and Welling, 2014) and GNNs. VAEs are
also widely used for representation learning, as will be explained in Section
6.

The Variational Autoencoder (VAE) is an encoder-decoder architecture trained
by optimizing the Evidence Lower Bound (ELBO, a similar loss was briefly
described in Section 4.4). The core idea of a VAE is to encode data into a
latent distribution. A decoder samples from this distribution and decodes back
the data. The latent distribution is a multivariate normal distribution where
the parameters are the output of the encoder. The encoder and decoder are
neural networks. Their parameters are learned by optimizing two terms: (i) a
data reconstruction term ensures that the output of the decoder is close to
the input data and (ii) a regularization term forces the latent distribution
to be normal (using Kullback-Leibler divergence). VAEs are usually applied on
grid data like images or tabular data, but Kipf and Welling (2016) applied the
VAE model onto graph structures. This model is called the Variational Graph
Auto-Encoder (VGAE) and is described in Figure 6. It is used to generate an
embedded dense vector $\mathbf{z}$ of a sparser input feature matrix
$\mathbf{X}$. A GNN encoder $q$ learns to generate the distribution of
$\mathbf{z}\sim\mathcal{N}(\boldsymbol{\nu},\boldsymbol{\sigma})$ and a GNN
decoder $p$ learns to reconstruct $\mathbf{X}$. The encoder is divided into
two terms $\boldsymbol{\nu}=\text{GNN}_{\nu}(\mathbf{X},\mathbf{A})$ and
$\boldsymbol{\sigma}=\text{GNN}_{\sigma}(\mathbf{X},\mathbf{A})$. A VGAE
operates similarly as a VAE but with an additional adjacency matrix
$\mathbf{A}$ given to the encoders and decoders. Therefore, the encoded
$\mathbf{z}$ has the same number $N$ of variables as $\mathbf{X}$ but with
fewer dimensions.

Figure 6. VGAE pipeline. A feature matrix $\mathbf{X}$ and an adjacency matrix
$\mathbf{A}$ are fed in input to a GNN encoder to generate the parameters of a
multivariate normal distribution, from which the encoded vector $\mathbf{z}$
is sampled. A GNN decoder, fed with $\mathbf{z}$ and $\mathbf{A}$, generates
an approximation $\mathbf{\hat{X}}$ reconstructing the original features
$\mathbf{X}$.

[Illustration of the VGAE process.]Illustration of the VGAE process. The
figure is described in the text.

The interventional VGAE (iVGAE) (Zecevic et al., 2021b) is a causal model able
to perform $\mathcal{L}_{2}$ causal inference, based on the VGAE architecture.
Assuming a causal graph $\mathcal{G}$ (as on Figure 7(a)) and a matrix
$\mathbf{X}$ representing the values of the causal variables in $\mathcal{G}$,
the reconstruction term
$\mathbf{\hat{X}}=p(q(\mathbf{X},\mathbf{A}_{\mathcal{G}}),\mathbf{A}_{\mathcal{G}})$
computes $\mathcal{L}_{1}$ queries for every variable $X\in\mathbf{X}$ based
on its parents $\mathbf{pa}_{X}$. This is allowed by the GNN nature of the
encoder-decoder, which follows the graph structure when doing inference. Under
the assumption that the causal graph is correct, complete, and has no hidden
confounders, the iVGAE computes queries $P(y|x)~{}\forall
y\in\mathbf{X},x\in\mathbf{pa}_{Y}\subset\mathbf{X}$. Now, if we amputate the
graph $\mathcal{G}$ of precise edges, we are able to generate a do
intervention on $\mathcal{G}$. As an example, the graph $\mathcal{G}^{\prime}$
on Figure 7(b), corresponds do an intervention on $X_{3}$. Under graph
$\mathcal{G}^{\prime}$,
$\mathbf{\hat{X}}=p(q(\mathbf{X},\mathbf{A_{\mathcal{G}^{\prime}}}),\mathbf{A_{\mathcal{G}^{\prime}}})\approx
p(\mathbf{X}|\text{do}(X_{3}=x_{3}))$. The iVGAE is a VGAE architecture used
as a causal model. Amputating the input graph and forcing the value of the
intervened node allows performing $\mathcal{L}_{2}$ causal inference. However,
the iVGAE cannot marginalize over exogenous variables and cannot compute
$\mathcal{L}_{3}$ queries.

(a) Original graph $\mathcal{G}$.

[Illustration of an intervention on a causal graph, original
graph]Illustration of an intervention on a causal graph. The original graph
has 4 nodes X1 to X4. X1 is connected to X3 and X4. X2 is connected to X3. X3
is connected to X4.

(b) Amputated graph $\mathcal{G}^{\prime}$.

[Illustration of an intervention on a causal graph, amputated
graph]Illustration of an intervention on a causal graph. The amputated graph
is similar to the original graph but the links to X3 are removed.

Figure 7. Intervention on the node $X_{3}$ of graph $\mathcal{G}$.

The Variational Causal Graph Encoder (VACA) (Sánchez-Martin et al., 2022) is
another model based on a VGAE architecture for causal inference. VACA is
defined similarly to the iVGAE but with additional constraints allowing it to
perform counterfactual inference, assuming that there are no hidden
confounders in the model: The first constraint is put on the decoder, which
should have at least l-1 hidden layers, with $l$ being the longest directed
path in the causal graph $\mathcal{G}$. It ensures that message propagation in
the GNN can transmit the causal effect to every node. The second constraint is
put on the encoder, which should have no hidden layers. It ensures that the
encoded distribution for each causal variable depends only on its parents.
VACA performs interventions similarly to iVGAE by modifying the causal graph
$\mathcal{G}$ into a graph $\mathcal{G}^{\prime}$. Counterfactual inference is
performed by combining two inputs. As shown in Section 2.1.3, counterfactual
requires abduction, action and prediction. A first set of latent variables
$\mathbf{Z}_{fact}$ is encoded using the values of $\mathbf{X}$ and the graph
$\mathcal{G}$ (abduction), a second set $\mathbf{Z}_{inter}$ is encoded using
$\mathbf{X}_{X_{i}=x_{i}}$, i.e. $\mathbf{X}$ under the intervention
$\text{do}(X_{i}=x_{i})$ on the variable of interest $X_{i}$ and the modified
graph $\mathcal{G}^{\prime}$. The vector $\mathbf{z}_{i}$, corresponding to
the encoded value of the intervened variable $X_{i}$, is merged into the set
$\mathbf{Z}_{fact}$ to form $\mathbf{Z}$. This new set is then given to the
decoder, which returns the expected value for each causal variable
(prediction). If the constraints described above and the unconfoundedness
assumption are respected, then VACA is a Neural Causal Model able to represent
interventional and counterfactual distributions. In the presence of
confounders, some queries can still be approximated if the GNNs are expressive
enough (Sánchez-Martin et al., 2022). The Causal Effect Variational
Autoencoder (CEVAE) (Louizos et al., 2017) and the Identifiable Treatment
Conditional VAE (Intact-VAE) (Wu and Fukumizu, 2021) are VAE-based models
performing causal-effect inference without using graph structure. These
methods aim to identify the individual treatment effect (ITE). The causal
graph has a simple and known shape $X\rightarrow Y\leftarrow T$ (outcome $Y$
depends on covariates $X$ and treatment $T$) with possible hidden confounders.
The Deep End-to-End Causal Inference (DECI) (Geffner et al., 2022) model is a
pipeline performing both causal structure discovery and causal inference. It
prevents misalignment issues between the learned causal structure and the
estimation task. The model optimizes an ELBO loss based on a prior over the
structure of the causal graph $p(\mathcal{G})$ and a generic neural network
for encoding the posterior distribution
$q_{\theta}(\mathbf{A}_{\mathcal{G}})$. Causal inference is computed using a
GNN architecture. DECI assumes that the graph is identifiable and does not
contain hidden confounders. DECI combines ideas from the VCN, described in
Section 4.4, and VGAE.

###  5.7. Transformers

The Causal Transformer (CT) (Melnychuk et al., 2022) is a Transformer-based
architecture (Vaswani et al., 2017) for causal inference in time series. It
attempts to recover the counterfactual outcome or individual treatment effect
(ITE) $\hat{Y_{t}}$ based on three separate inputs: the past covariates
$\mathbf{Z}_{t}$, the past outcomes $\mathbf{Y}_{t-1}$, and the past
interventions or treatments $\mathbf{X}_{t-1}$ with the one to be performed
next $X_{t}$. Each input is fed to a separate self-attention sub-network. All
the outputs are then merged into a joint attention network with cross-
attention, fed in return to two classifiers trained in an adversarial fashion:
the first is trained to predict the outcome $\hat{Y_{t}}$ and the second is
trained to predict the current intervention $X_{t}$ ($X_{t}$ is fed only to
the first classifier). This adversarial training aims to learn a
representation predictive of the outcome but not predictive of the
intervention as background variables should not contain any information
regarding the intervention. Information about the current treatment from
covariates or past treatments necessarily comes from a spurious correlation.
Adversarial training is used here to mitigate confounding effects.

###  5.8. Vector Autoregressive Models

Different approaches have been used to take advantage of time information for
causal inference, the most popular being the Vector Autoregressive models
(VARs). Granger Causality is usually used with a VAR model (Nogueira et al.,
2022; Moraffah et al., 2021), where the variables are linked following a
linear relation. A widely used model is the AutoRegressive Integrated Moving
Average (ARIMA). ARIMA is made of three components: the autoregressive (AR)
part predicts the next value of a series $X(t)$ based on its previous values
and an error term, the moving average (MA) part performs the prediction based
on the past error terms of AR to take into account correlation between the
noise terms, and the integration (I) part makes a series stationary if its
trend was not stable. The components are represented in Equations 26, 27, and
28, respectively.

(26) $\displaystyle X_{AR}(t)$
$\displaystyle=c_{1}+\epsilon_{t}+\sum\limits_{i=1}^{p}\alpha_{i}X(t-i)$ (27)
$\displaystyle X_{MA}(t)$
$\displaystyle=c_{2}+\epsilon_{t}+\sum\limits_{j=1}^{q}\beta_{j}\epsilon_{t-j}$
(28) $\displaystyle X_{I}(t)$ $\displaystyle=X(t)-X(t-d)$

where $\alpha_{i}$, $\beta_{j}$, $\epsilon_{t}$, $c_{1}$, and $c_{2}$ are
trainable parameters and $p$, $q$, and $d$ are hyperparameters. The three
equations can be combined together to create the ARIMA model: the series
$X(t)$ is shifted with the I component and the resulting is given to a merged
ARMA equation. This decomposition allows representing many architectures by
changing (or setting to zero) the coefficients of the AR,I,and MA components.
Other approaches for modelling time-series for causal inference include
Dynamic Bayesian Networks, an extension of Bayesian networks for discrete
time, Hidden Markov Models or Gaussian processes, as surveyed in Moraffah et
al. (2021).

##  6\. Causal Representation Learning

Section 3 introduced neural causal models, i.e. causal reasoning engines
taking advantage of Deep Learning techniques, and the benefits of such models.
However, while DL is applied to low-level data in high-dimension, causal
reasoning is performed on high-level variables, usually with semantic meaning
and in small dimensions (Goyal and Bengio, 2020). Going from the former to the
latter is a challenging task that uses Representation Learning, and generating
variables that can be used for causal structure discovery and inference (i.e.
causal variables) is called Causal Representation Learning (Schölkopf et al.,
2021). This section investigates the current representation learning
techniques for causal variables.

###  6.1. Disentanglement

As discussed in the previous sections, causal variables represent abstract
concepts sparsely connected to other variables. In particular, the ICM
principle states that causal mechanisms are independent, and modifying one
should not affect the others (Schölkopf et al., 2021). Disentangled features
respect these properties. They also require fewer samples to achieve similar
performance, are less sensitive to noise, and are more robust (van Steenkiste
et al., 2019). Disentangling the semantic features in the data could therefore
be a way to generate high-level causal variables suitable for causal discovery
and inference. This section reviews the fundamental properties of disentangled
features, the current disentanglement methods, and how they could help causal
representation learning.

Despite being a widely studied topic in Deep Learning research, there is no
clear definition of disentanglement. Many definitions have been proposed, but
none have been widely accepted. For instance, Higgins et al. (2018) defines a
vector representation as disentangled if it can be decomposed into orthogonal
subspaces where a symmetric modification in one subspace will not affect the
others. Even if no rigorous definition is available, most of the works on
disentanglement share common principles. The key idea of disentanglement
shared in the research community is that a disentangled representation should
separate the factors of variation affecting the data (Kim and Mnih, 2018;
Locatello et al., 2019). They represent independent semantic interpretable
information describing the data (Mathieu et al., 2019; Chen et al., 2018). The
notion of local modification is essential in the disentanglement literature
and can be reconciled with the notion of intervention in Causality (Schölkopf
and von Kügelgen, 2022). Modifying (i.e. intervening on) a disentangled
variable should not have any effect on the other variables not that do not
causally depend on the former. For instance, changing the lighting of a scene
in an image should not change the pose, the orientation, or the labels of the
objects. This example is common in the literature as the core domain in which
disentanglement is studied and applied is controlled image generation (Higgins
et al., 2017; Chen et al., 2018). As the effect of an intervention on a
disentangled latent space can be easily seen on the output image, this domain
is well suited for developing disentanglement models.

The majority of articles in the domain rely on Variational Autoencoders and
propose techniques for constraining the dimensions of the encoded space to be
disentangled. The Deep Convolutional Inverse Graphics Network (DC-IGN)
(Whitney, 2016) is an early work in this direction. In order to force the
disentanglement of the latent space, the training samples are divided into
mini-batches of nearly identical images with a single difference between them,
corresponding to the modification of a single factor of variation (e.g.
orientation, color, lighting). For each mini-sample, the parameters of only
one dimension are updated, forcing it to represent the only desired effect.
This method requires many samples where only the measured feature changes.
This requirement is hardly achievable in practice when dealing with real-world
datasets. Achille and Soatto (2018) uses the Information-Bottleneck Lagrangian
(Tishby et al., 2000) to reduce the amount of data information in the weights
of neural networks and increase the level of invariance and disentanglement of
the latent representations. The IB Lagrangian is defined in Equation 29, using
the measure of Entropy $H$ and Mutual Shared Information $I$ as defined in a
previous section in Equations 8 and 9:

(29) $\mathcal{L}=H(\mathbf{Y}|\mathbf{Z})+\alpha I(\mathbf{Z};\mathbf{X})$

where $\mathbf{X}$ and $\mathbf{Y}$ represent the sets of random variables for
the input and output images, respectively. $\mathbf{Z}$ is the latent space we
are trying to disentangle. We can note that $\mathbf{Z}$ serves the same
purpose as hidden confounders as it represents the mechanisms underlying the
generation of the data $\mathbf{X}$. The first term controls sufficiency, i.e.
if $H$ is minimal, then $\mathbf{Y}$ can be fully predicted from $\mathbf{Z}$,
and $\mathbf{Z}$ is sufficient to represent $\mathbf{Y}$. The second term
controls minimality, i.e. the smaller $I$, the lower amount of information is
shared from $\mathbf{X}$ to $\mathbf{Z}$. The main idea of the paper is that
minimal and sufficient representations $\mathbf{Z}$ should also be invariant
and disentangled. However, the information measure is not a measure of the
quality of the representation, and the representation may not be more
informative than the input $\mathbf{X}$.

More recent approaches build upon the $\beta$-VAE model (Higgins et al.,
2017). This model provides a modification to the loss function that a
Variational Autoencoder (VAE) learns to maximize, i.e. the Evidence Lower
Bound (ELBO) (Kingma and Welling, 2014), given in Equation 30:

(30) $\mathcal{L}_{\beta}=\mathbb{E}_{q_{\phi}}[\log
p_{\theta}(\mathbf{X}|\mathbf{Z})]-\beta\text{
KL}(q_{\phi}(\mathbf{Z}|\mathbf{X})||P(\mathbf{Z}))$

where $p_{\theta}$ is the probability of observing the input data distribution
of $\mathbf{X}$ from the decoder output conditioned by the distribution of
$\mathbf{Z}$, $q_{\phi}$ is the distribution of $\mathbf{Z}$ obtained with the
encoder from $\mathbf{X}$, and KL is the Kullback-Leibler divergence. $\phi$
and $\theta$ are the learned parameters of the encoder and decoder,
respectively. $\beta>0$ is a hyperparameter. The $\beta$-VAE uses values
$\beta>1$ whereas the case $\beta=1$ corresponds to the classic ELBO loss. The
paper does not contain a full explanation regarding why this method should
yield a disentangled representation, but subsequent works (Chen et al., 2018;
Mathieu et al., 2019; Esmaeili et al., 2019) explain it by decomposing the
Kullback–Leibler divergence term. For instance, the $\beta$-TCVAE (Chen et
al., 2018) decomposes the KL term into a sum of three components:

(31)
$\text{KL}(q_{\phi}(\mathbf{Z}|\mathbf{X})||p(\mathbf{Z}))=I(\mathbf{Z};\mathbf{X})+\text{KL}(q_{\phi}(\mathbf{Z})||\prod\limits_{j}q_{\phi}(Z_{j}))+\sum\limits_{j}\text{KL}(q_{\phi}(Z_{j})||P(Z_{j}))$

where the first term is called the Mutual Information (I) component. It
represents the information shared between $\mathbf{X}$ and $\mathbf{Z}$. The
second term is the Total Correlation (TC) component. It is a popular measure
of the degree of independence of the dimensions of $\mathbf{Z}$. The third
term is the dimension wise KL divergence. It measures if each individual
dimension of $\mathbf{Z}$ is different from its prior. The intuition behind
$\beta$-TCVAE is that minimizing the Total Correlation
$\text{TC}(z)=\text{KL}(q(z)||\prod_{j}q(z_{j}))$ increases the level of
disentanglement of $\mathbf{Z}$. The distribution of a disentangled
representation $\mathbf{Z}$ should be the same as the product of the
distribution of its components $Z_{j}$. The authors show experimentally that
granting a higher weight to TC enhances disentanglement. The FactorVAE (Kim
and Mnih, 2018) uses a similar decomposition to increase independence across
dimensions of $\mathbf{Z}$. Instead of minimizing the TC term, the
Disentangled Inferred prior VAE (DIP-VAE) (Kumar et al., 2018) introduces a
regularizer based on the covariance of the two distributions $\mathbf{X}$ and
$\mathbf{Z}$.

Another line of work, based on Generative Adversarial Networks, aims to
separate variables linked to a label, sometimes called factors or attributes
of interest, and the background variables or residual attributes (Hadad et
al., 2018; Bau et al., 2019; Peng et al., 2019; Gabbay et al., 2021). These
works do not attempt to identify each variable individually but try to
separate them into two separate latent spaces: invariant variables
$\mathbf{Z}_{I}$ (e.g. the object represented) and domain-specific ones
$\mathbf{Z}_{D}$ (e.g. the environment, style, orientation).

Recently, a set of experiments and theoretical analysis (Locatello et al.,
2019) showed that learning a disentangled representation in an unsupervised
way was impossible without additional assumptions or inductive biases on the
nature of the task. The authors prove the existence of an infinite family of
functions yielding an entangled $\mathbf{Z}$ while minimizing TC.

This finding induced a shift from unsupervised disentanglement to semi-
supervised methods. The Adaptive-Group VAE (ada-GVAE) (Locatello et al.,
2020a) is a weakly-supervised method for disentanglement. The authors show
that weakly-supervised disentanglement is possible with few assumptions. If
the number of modified factors of variations between two samples if small,
then it is possible to retrieve the marginal distributions of each factor and
learn disentangled representations. Whether or not unsupervised
disentanglement is achievable remains, however, debated (Locatello et al.,
2020b; Horan et al., 2021). Nonetheless, disentanglement remains actively
researched in several domains: vision (for controlled image generation, image-
to-image translation or object detection) (Li et al., 2021; Gabbay et al.,
2021; Wu et al., 2021; Gendron et al., 2023b) , Independent Component Analysis
(ICA) (Khemakhem et al., 2020; Lachapelle et al., 2022), medical imaging (Liu
et al., 2021), or Graph representations (Ma et al., 2019). Moreover, the
recent success of text-to-image diffusion models (Ramesh et al., 2022; Saharia
et al., 2022) at generating images could help drive forward the field of
disentanglement, mainly based on VAE architectures or adversarial networks.
Recent works on disentanglement take their inspiration from other domains like
ICA and Causality. The Identifiable VAE (iVAE, not to be confused with the
interventional VGAE introduced earlier) (Khemakhem et al., 2020) and the
regularized VAE (Lachapelle et al., 2022) propose methods for retrieving both
the disentangled latent variables and the sparse causal graph linking them
together. They can be linked back to the CEVAE (Louizos et al., 2017) and the
Intact-VAE (Wu and Fukumizu, 2021), described in Section 5.6, which build
similar ideas to perform causal inference. The DDG network (Disentanglement-
constrained Domain Generalization) (Zhang et al., 2021b) is a method for
disentangling domain attributes and the ones of interest by imposing
constraints in a semantic and a variation space. The authors use it to learn
true disentangled causal variables and perform interventions on either the
semantic or the variation space, which can be used in return for data
augmentation and better generalization. A recent approach named Independent
Mechanism Analysis (IMA) combines ICA with the ICM principle (Gresele et al.,
2021). IMA-based regularization aims to recover and distinguish the true
sources responsible for the generation of a mixture of signals, assuming that
each source influences the data independently. The method has been shown to
work when the aforementioned assumption is violated and recovers the true
causes in the presence of confounders.

Schölkopf et al. (2021) postulates that automatic disentanglement is a
necessary step towards learning causal mechanisms from data at scale. This
claim is debated (Goyal and Bengio, 2020). However, even if it reveals
founded, it is not sufficient, as once high-level disentangled features have
been extracted, causal representation learning still requires to recover the
causal links between the features. Finding the appropriate features at the
right level of granularity (trade-off between precision and abstraction)
remains a challenging problem.

###  6.2. Data Augmentation

Generating a good representation of reality relies heavily on the observations
available to the model. A fundamental hypothesis in Deep Learning is the
assumption that the training data is independent and identically distributed
(i.i.d). When true, a model can only learn the desired distinguishing features
to improve its performance. However, if this assumption is not respected, the
model may learn undesired behaviors caused by confounding effects. The
techniques discussed so far are model-driven. They rely mainly on improvements
of the architecture used, allowing relaxations of the i.i.d assumption. On the
other hand, data-driven approaches aim to provide better data to the model. In
particular data augmentation can ensure that the data remain i.i.d. and has
been shown to improve the performance of Deep Learning models (Balestriero et
al., 2022). In this section, we briefly introduce methods aiming to provide a
greater quantity or quality of data for training the models to approximate a
causal behavior.

The Interchange Intervention Training (IIT) (Geiger et al., 2022) is a method
for training a neural network to match a causal model by feeding it with
interventional data. The process consists of two steps: aligning the hidden
representations of a neural network with a ground-truth causal model and
training the network to match the result of the causal model in a
counterfactual setting. IIT mixes two causal situations together to create two
counterfactual scenarii running in parallel, as illustrated on Figure 8. Two
inputs are given to the network, and two hidden representations are exchanged
while performing inference, corresponding to performing an intervention in
each model. The outputs are compared against the counterfactual outcomes of
the ground-truch causal model. The IIT method can be considered a data
augmentation method where the augmented data is directly inserted into
subparts of the network. However, this training procedure requires knowing the
true SCM for the task and decomposing the latent space into several components
that can be mapped to causal variables. In practice, the true SCM of a given
task is unknown, greatly reducing the applicability of this method.

(a) Standard forward pipeline.

[Illustration of the forward pipeline in a neural network.]Illustration of the
forward pipeline in a neural network, inputs x1 and x2 are multiplied by both
theta1 and theta2. The output of theta1 and theta2 multiplications, called
hidden representations H1 and H2, are given to theta3. The output of the
theta3 multiplication is the final output Y. The same process is applied to
inputs x1’ and x2’, which generate the hidden representations H1’, H2’ and the
output y’.

(b) IIT forward pipeline.

[Illustration of the IIT pipeline in a neural network.]Illustration of the
Interchange Intervention Training (IIT) pipeline in a neural network, inputs
x1 and x2 are multiplied by both theta1 and theta2 as in the standard
pipeline. The hidden representation H1 is exchanged with H1’. Therefore, H1’
and H2 are given to theta3. The output of the theta3 multiplication is the
final output Y tilde. Similarly, H1 and H2’ are given to theta3, which
generate the output y’ tilde.

Figure 8. Illustration of Interchange Intervention Training (IIT) in a simple
neural network. Two inputs $\mathbf{X}=\\{\mathbf{X_{1}},\mathbf{X_{2}}\\}$
and
$\mathbf{X^{\prime}}=\\{\mathbf{X_{1}^{\prime}},\mathbf{X_{2}^{\prime}}\\}$
are fed to the same network with parameters
$\mathbf{\theta}=\\{\mathbf{\theta_{1}},\mathbf{\theta_{2}},\mathbf{\theta_{3}}\\}$.
When performing IIT, a subset of the hidden representations is interchanged
before being given to the last layer. Here, $H_{1}$ and $H_{1}^{\prime}$. The
outputs after intervention $\mathbf{\tilde{Y}}$ and $\mathbf{\tilde{Y}}$’ are
compared against the values obtained with a true SCM aligned with the network,
fed with the same observational and interventional data.

We make a connection between this work and a data augmentation technique
called ManifoldMixup (Zhang et al., 2018; Verma et al., 2019; Mangla et al.,
2020). The latter consist of interpolating the inputs of one neural network
layer and the output labels as shown below:

(32)
$\tilde{\mathbf{H_{i}}}=\alpha\cdot\mathbf{H_{i}}+(1-\alpha)\cdot\mathbf{H_{i}}^{\prime}$

(33)
$\tilde{\mathbf{Y}}=\alpha\cdot\mathbf{Y}+(1-\alpha)\cdot\mathbf{Y}^{\prime}$

IIT can be seen as a particular case of the ManifoldMixup technique, where
$\alpha=0$ in Equation 32 and the label is determined in a supervised manner
using a causal model. The goal of the above techniques is to train a neural
model on a more extensive diversity of samples, different than what can be
achieved from the training set, to increase model robustness and
generalization. It has been shown that ManifoldMixup also increases
performance for few-shot tasks (Mangla et al., 2020). We can argue that the
representations generated by the samples correspond to interventions in the
neural network. The interchange or mixup of inputs, especially in the hidden
layers of the network, yields representations that cannot be reached by simply
feeding the network with data (Verma et al., 2019; Geiger et al., 2022). This
could increase the expressivity of current data augmentation techniques or
reduce the amount of samples needed (Balestriero et al., 2022).

##  7\. Applications

The evolution of the Causality field is highly tied to the evolution of
science. One of the first occurrences of Causality can be traced back to the
second book of Physics by Aristotle (Falcon, 2006). Causality, therefore, has
applications in numerous and broad domains of science, from biology to
economics. In this section, we briefly overview the fields using Causality
methods. A more extensive review is given in the appendix.

As already discussed in the previous sections, Causality can be applied to
improve the performance of Deep Learning models. Some articles have applied
Causality theory to improve performance over vision tasks (Qi et al., 2020),
complex reasoning tasks (Sui et al., 2022), autonomous driving (de Haan et
al., 2019) and causal commonsense (Gordon et al., 2012). Causality also has
applications in biology for modeling dynamic systems such as ecosystems
(Wright, 1921; Sugihara et al., 2012) or attributing gene expression to
phenotypical attributes in an individual (Yu et al., 2004; Delaplace et al.,
2010; Schadt et al., 2005). Early Causality theories for time series have been
derived for economic applications (Granger, 1969; Sims, 1972), particularly
econometrics. Those works attempt to use Causality to extract relationships
and provide a theoretical economic model. Another prominent application of
Causality is in finance and the prediction of trends and crises in the stock
market (Billio et al., 2012; Papana et al., 2017). As the Granger framework
was initially developed for the economy, the Rubin Causal Model was created
for medical applications (Rubin, 1974), particularly for studying the impact
of randomized trials. Causal reasoning reduces bias and confounding effects in
such studies, and researchers argue that causal methods should be more widely
used to reduce spurious correlations that lead to wrong conclusions (Hernán et
al., 2019). Causal models also exist in epidemiology to identify the
covariates and confounders propagating a disease and to model an epidemic in a
population (Greenland et al., 1999; Hernán et al., 2002). The primary use of
Causality in climatology is for modelling climate events as networks. For
instance, Malik et al. (2012) uses network representations for studying
rainfall in India. Climatology tasks are challenging as many factors are
involved, with complex non-linear interactions. Some works pointed out the
limitations of correlation-based methods for these tasks (Ebert-Uphoff and
Deng, 2012). Causality applications can also be found in ethology (Timberlake,
1997; Lord et al., 2016; Chen et al., 2020; Gendron et al., 2023a),
neurosciences (Körding et al., 2007; Rohe and Noppeney, 2015; Fornito et al.,
2013), and social sciences (Bramley et al., 2018; Waismeyer et al., 2015).

##  8\. Conclusion

Causality has a branching in many fields of science. Recently, a growing
interest arose from Deep Learning as it can help tackle critical DL
challenges, including strong generalization and complex causal reasoning. This
survey provides an in-depth introduction to Causality and the attempts made to
combine it with DL to guide researchers that aim to use Causality with Deep
Learning. We describe Causality theories for static and time series data, and
survey the various methods used for causal tasks. We provide a deeper look at
the most recent approaches that combine Deep Learning with Causality. We argue
that bridging the gap between the two fields is a way to solve the challenges
of generalization, modularity, and interventional and counterfactual reasoning
encountered by Deep Learning models. All the methods described in this paper
pave the way to building better causal AI models that could solve these
challenges, although the perspectives of causality are not limited by the set
described here. Causality can for instance provide insight and better
understanding of the causal reasoning abilities of existing systems and help
create more interpretable models.

## References

  * (1)
  * Achille and Soatto (2018) Alessandro Achille and Stefano Soatto. 2018.  Emergence of Invariance and Disentanglement in Deep Representations.  _J. Mach. Learn. Res._ 19 (2018), 50:1–50:34.  http://jmlr.org/papers/v19/17-646.html
  * Ahsan et al. (2022) Ragib Ahsan, David Arbour, and Elena Zheleva. 2022\.  Relational Causal Models with Cycles: Representation and Reasoning. In _1st Conference on Causal Learning and Reasoning, CLeaR 2022, Sequoia Conference Center, Eureka, CA, USA, 11-13 April, 2022_ _(Proceedings of Machine Learning Research, Vol. 177)_ , Bernhard Schölkopf, Caroline Uhler, and Kun Zhang (Eds.). PMLR, 1–18.  https://proceedings.mlr.press/v177/ahsan22a.html
  * Angus (1994) John E Angus. 1994\.  The probability integral transform and related results.  _SIAM review_ 36, 4 (1994), 652–654.
  * Annadani et al. (2021) Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste, Nino Scherrer, Anirudh Goyal, Yoshua Bengio, and Stefan Bauer. 2021.  Variational Causal Networks: Approximate Bayesian Inference over Causal Structures.  _CoRR_ abs/2106.07635 (2021).  arXiv:2106.07635 https://arxiv.org/abs/2106.07635
  * Austin (2007) Mike Austin. 2007\.  Species distribution models and ecological theory: a critical assessment and some possible new approaches.  _Ecological modelling_ 200, 1-2 (2007), 1–19.
  * Austin (2011) Peter C. Austin. 2011\.  An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies.  _Multivariate Behavioral Research_ 46, 3 (2011), 399–424.  https://doi.org/10.1080/00273171.2011.568786 arXiv:https://doi.org/10.1080/00273171.2011.568786  PMID: 21818162.
  * Baars (1997) Bernard J. Baars. 1997\.  In the Theatre of Consciousness. Global Workspace Theory, A Rigorous Scientific Theory of Consciousness.
  * Balestriero et al. (2022) Randall Balestriero, Ishan Misra, and Yann LeCun. 2022.  A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments.  _CoRR_ abs/2202.08325 (2022).  arXiv:2202.08325 https://arxiv.org/abs/2202.08325
  * Baradel et al. (2020) Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. 2020.  CoPhy: Counterfactual Learning of Physical Dynamics. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.  https://openreview.net/forum?id=SkeyppEFvS
  * Bareinboim et al. (2022) Elias Bareinboim, Juan David Correa, Duligur Ibeling, and Thomas F. Icard. 2022.  On Pearl’s Hierarchy and the Foundations of Causal Inference.  _Probabilistic and Causal Inference_ (2022).
  * Barnett et al. (2009) Lionel Barnett, Adam B. Barrett, and Anil K. Seth. 2009\.  Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables.  _Phys. Rev. Lett._ 103 (Dec 2009), 238701.  Issue 23. https://doi.org/10.1103/PhysRevLett.103.238701
  * Bau et al. (2019) David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. 2019.  GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.  https://openreview.net/forum?id=Hyg_X2C5FX
  * Billio et al. (2012) Monica Billio, Mila Getmansky, Andrew W Lo, and Loriana Pelizzon. 2012.  Econometric measures of connectedness and systemic risk in the finance and insurance sectors.  _Journal of financial economics_ 104, 3 (2012), 535–559.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. 2021.  On the Opportunities and Risks of Foundation Models.  _CoRR_ abs/2108.07258 (2021).  arXiv:2108.07258 https://arxiv.org/abs/2108.07258
  * Bongers et al. (2021) Stephan Bongers, Patrick Forré, Jonas Peters, and Joris M Mooij. 2021.  Foundations of structural causal models with cycles and latent variables.  _The Annals of Statistics_ 49, 5 (2021), 2885–2915.
  * Bramley et al. (2018) Neil R. Bramley, Tobias Gerstenberg, Ralf Mayrhofer, and David A. Lagnado. 2018.  Time in causal structure learning.  44, 12 (2018), 1880–1910.  https://doi.org/10.1037/xlm0000548 Place: US Publisher: American Psychological Association.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Bühlmann et al. (2010) Peter Bühlmann, Markus Kalisch, and Marloes H Maathuis. 2010\.  Variable selection in high-dimensional linear models: partially faithful distributions and the PC-simple algorithm.  _Biometrika_ 97, 2 (2010), 261–278.
  * Caselli and Vossen (2017) Tommaso Caselli and Piek Vossen. 2017.  The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In _Proceedings of the Events and Stories in the News Workshop@ACL 2017, Vancouver, Canada, August 4, 2017_ , Tommaso Caselli, Ben Miller, Marieke van Erp, Piek Vossen, Martha Palmer, Eduard H. Hovy, Teruko Mitamura, and David Caswell (Eds.). Association for Computational Linguistics, 77–86.  https://doi.org/10.18653/v1/w17-2711
  * Chen et al. (2020) Duxin Chen, Mingyu Kang, and Wenwu Yu. 2020.  Probabilistic causal inference for coordinated movement of pigeon flocks.  _EPL (Europhysics Letters)_ 130, 2 (5 2020), 28004.  https://doi.org/10.1209/0295-5075/130/28004
  * Chen et al. (2018) Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. 2018\.  Isolating Sources of Disentanglement in Variational Autoencoders. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 2615–2625.  https://proceedings.neurips.cc/paper/2018/hash/1ee3dfcd8a0645a25a35977997223d22-Abstract.html
  * Chen et al. (2022) Ziyue Chen, Miaoqing Xu, Bingbo Gao, George Sugihara, Feixue Shen, Yanyan Cai, Anqi Li, Qi Wu, Lin Yang, Qi Yao, et al. 2022\.  Causation inference in complicated atmospheric environment.  _Environmental Pollution_ 303 (2022), 119057.
  * Chickering (2002) David Maxwell Chickering. 2002\.  Optimal Structure Identification With Greedy Search.  _J. Mach. Learn. Res._ 3 (2002), 507–554.  http://jmlr.org/papers/v3/chickering02b.html
  * Chipman et al. (2010) Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. 2010\.  BART: Bayesian additive regression trees.  _The Annals of Applied Statistics_ 4, 1 (2010), 266 – 298.  https://doi.org/10.1214/09-AOAS285
  * Claassen and Heskes (2010) Tom Claassen and Tom Heskes. 2010.  Causal discovery in multiple models from different experiments. In _Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada_ , John D. Lafferty, Christopher K. I. Williams, John Shawe-Taylor, Richard S. Zemel, and Aron Culotta (Eds.). Curran Associates, Inc., 415–423.  https://proceedings.neurips.cc/paper/2010/hash/5807a685d1a9ab3b599035bc566ce2b9-Abstract.html
  * Colombo and Maathuis (2014) Diego Colombo and Marloes H. Maathuis. 2014.  Order-independent constraint-based causal structure learning.  _J. Mach. Learn. Res._ 15, 1 (2014), 3741–3782.  https://doi.org/10.5555/2627435.2750365
  * Colombo et al. (2012) Diego Colombo, Marloes H. Maathuis, Markus Kalisch, and Thomas S. Richardson. 2012.  Learning high-dimensional directed acyclic graphs with latent and selection variables.  40, 1 (2012).  https://doi.org/10.1214/11-AOS940
  * Cooper and Herskovits (1992) Gregory F. Cooper and Edward Herskovits. 1992.  A Bayesian Method for the Induction of Probabilistic Networks from Data.  _Mach. Learn._ 9 (1992), 309–347.  https://doi.org/10.1007/BF00994110
  * Dasgupta et al. (2019) Ishita Dasgupta, Jane X. Wang, Silvia Chiappa, Jovana Mitrovic, Pedro A. Ortega, David Raposo, Edward Hughes, Peter W. Battaglia, Matthew M. Botvinick, and Zeb Kurth-Nelson. 2019.  Causal Reasoning from Meta-reinforcement Learning.  _CoRR_ abs/1901.08162 (2019).  arXiv:1901.08162 http://arxiv.org/abs/1901.08162
  * de Haan et al. (2019) Pim de Haan, Dinesh Jayaraman, and Sergey Levine. 2019\.  Causal Confusion in Imitation Learning. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 11693–11704.  https://proceedings.neurips.cc/paper/2019/hash/947018640bf36a2bb609d3557a285329-Abstract.html
  * De Winkel et al. (2018) Ksander N De Winkel, Mikhail Katliar, Daniel Diers, and Heinrich H Bülthoff. 2018.  Causal inference in the perception of verticality.  _Scientific reports_ 8, 1 (2018), 1–12.
  * Delaplace et al. (2010) Franck Delaplace, Hanna Klaudel, and Amandine Cartier-Michaud. 2010\.  Discrete causal model view of biological networks. In _Proceedings of the 8th International Conference on Computational Methods in Systems Biology_. 4–13.
  * Du et al. (2022) Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022.  e-CARE: a New Dataset for Exploring Explainable Causal Reasoning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 432–446.  https://doi.org/10.18653/v1/2022.acl-long.33
  * Du et al. (2021) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021.  GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.  (2021).  arXiv:2112.06905 http://arxiv.org/abs/2112.06905
  * Duan et al. (2022) Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. 2022.  A Survey on Machine Learning Approaches for Modelling Intuitive Physics. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_ , Luc De Raedt (Ed.). ijcai.org, 5444–5452.  https://doi.org/10.24963/ijcai.2022/763
  * Duncan (2014) Otis Dudley Duncan. 2014\.  _Introduction to structural equation models_.  Elsevier.
  * Dündar-Coecke et al. (2022) Selma Dündar-Coecke, Gideon Goldin, and Steven A Sloman. 2022\.  Causal reasoning without mechanism.  _PloS one_ 17, 5 (2022), e0268219.
  * Ebert-Uphoff and Deng (2012) Imme Ebert-Uphoff and Yi Deng. 2012.  Causal discovery for climate research using graphical models.  _Journal of Climate_ 25, 17 (2012), 5648–5665.
  * Entner and Hoyer (2010) Doris Entner and Patrik Hoyer. 2010.  On Causal Discovery from Time Series Data using FCI.  _Proceedings of the 5th European Workshop on Probabilistic Graphical Models, PGM 2010_ (09 2010).
  * Esmaeili et al. (2019) Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N. Siddharth, Brooks Paige, Dana H. Brooks, Jennifer G. Dy, and Jan-Willem van de Meent. 2019\.  Structured Disentangled Representations. In _The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan_ _(Proceedings of Machine Learning Research, Vol. 89)_ , Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 2525–2534.  http://proceedings.mlr.press/v89/esmaeili19a.html
  * Essington (2022) John Essington. 2022\.  How we learn: why brains learn better than any machine … for now.  _Educational Review_ 74, 4 (2022), 899–899.  https://doi.org/10.1080/00131911.2021.1930914 arXiv:https://doi.org/10.1080/00131911.2021.1930914
  * Falcon (2006) Andrea Falcon. 2006\.  Aristotle on causality.  (2006).
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer. 2021\.  Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.  (2021).  arXiv:2101.03961 http://arxiv.org/abs/2101.03961
  * Flesch and Lucas (2007) Ildikó Flesch and Peter JF Lucas. 2007.  _Markov Equivalence in Bayesian Networks_.  Springer Berlin Heidelberg, Berlin, Heidelberg, 3–38.  https://doi.org/10.1007/978-3-540-68996-6_1
  * Fornito et al. (2013) Alex Fornito, Andrew Zalesky, and Michael Breakspear. 2013\.  Graph analysis of the human connectome: promise, progress, and pitfalls.  _Neuroimage_ 80 (2013), 426–444.
  * Forré and Mooij (2018) Patrick Forré and Joris M. Mooij. 2018.  Constraint-based Causal Discovery for Non-Linear Structural Causal Models with Cycles and Latent Confounders. In _Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018_ , Amir Globerson and Ricardo Silva (Eds.). AUAI Press, 269–278.  http://auai.org/uai2018/proceedings/papers/117.pdf
  * French and DeAngelis (2020) Ranran L French and Gregory C DeAngelis. 2020.  Multisensory neural processing: from cue integration to causal inference.  _Current Opinion in Physiology_ 16 (2020), 8–13.  https://doi.org/10.1016/j.cophys.2020.04.004 Vision Physiology.
  * Gabbay et al. (2021) Aviv Gabbay, Niv Cohen, and Yedid Hoshen. 2021.  An Image is Worth More Than a Thousand Words: Towards Disentanglement in The Wild. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 9216–9228.  https://proceedings.neurips.cc/paper/2021/hash/4cef5b5e6ff1b3445db4c013f1d452e0-Abstract.html
  * Geffner et al. (2022) Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, Miltiadis Allamanis, and Cheng Zhang. 2022\.  Deep End-to-end Causal Inference.  _CoRR_ abs/2202.02195 (2022).  arXiv:2202.02195 https://arxiv.org/abs/2202.02195
  * Geiger et al. (2022) Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. 2022.  Inducing Causal Structure for Interpretable Neural Networks. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 7324–7338.  https://proceedings.mlr.press/v162/geiger22a.html
  * Gendron et al. (2023a) Gaël Gendron, Yang Chen, Mitchell Rogers, Yiping Liu, Mihailo Azhar, Shahrokh Heidari, David Arturo Soriano Valdez, Kobe Knowles, Padriac O’Leary, Simon Eyre, Michael Witbrock, Gillian Dobbie, Jiamou Liu, and Patrice Delmas. 2023a.  Behaviour Modelling of Social Animals via Causal Structure Discovery and Graph Neural Networks.  _CoRR_ abs/2312.14333 (2023).  arXiv:2312.14333 http://arxiv.org/abs/2312.14333
  * Gendron et al. (2023b) Gaël Gendron, Michael Witbrock, and Gillian Dobbie. 2023b.  Disentanglement of Latent Representations via Causal Interventions. In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China_. ijcai.org, 3239–3247.  https://doi.org/10.24963/IJCAI.2023/361
  * Glymour et al. (2019) Clark Glymour, Kun Zhang, and Peter Spirtes. 2019.  Review of Causal Discovery Methods Based on Graphical Models.  _Frontiers in Genetics_ 10 (2019).  https://doi.org/10.3389/fgene.2019.00524
  * Goddu et al. (2020) Mariel K Goddu, Tania Lombrozo, and Alison Gopnik. 2020\.  Transformations and transfer: Preschool children understand abstract relations and reason analogically in a causal task.  _Child development_ 91, 6 (2020), 1898–1915.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and Harnessing Adversarial Examples. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1412.6572
  * Gordon et al. (2012) Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012\.  SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In _Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2012, Montréal, Canada, June 7-8, 2012_ , Eneko Agirre, Johan Bos, and Mona T. Diab (Eds.). The Association for Computer Linguistics, 394–398.  https://aclanthology.org/S12-1052/