# Counterfactual Generation Under
Confounding

Abbavaram Gowtham Reddy
IIT Hyderabad, India
<EMAIL_ADDRESS>
&Saloni Dash*
Microsoft Research Bengaluru, India
<EMAIL_ADDRESS>
&Amit Sharma
Microsoft Research Bengaluru, India
<EMAIL_ADDRESS>
& Vineeth N Balasubramanian
IIT Hyderabad, India
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

A machine learning model, under the influence of observed or unobserved
confounders in the training data, can learn spurious correlations and fail to
generalize when deployed. For image classifiers, augmenting a training dataset
using counterfactual examples has been empirically shown to break spurious
correlations. However, the counterfactual generation task itself becomes more
difficult as the level of confounding increases. Existing methods for
counterfactual generation under confounding consider a fixed set of
interventions (e.g., texture, rotation) and are not flexible enough to capture
diverse data-generating processes. Given a causal generative process, we
formally characterize the adverse effects of confounding on any downstream
tasks and show that the correlation between generative factors (attributes)
can be used to quantitatively measure confounding between generative factors.
To minimize such correlation, we propose a counterfactual generation method
that learns to modify the value of any attribute in an image and generate new
images given a set of observed attributes, even when the dataset is highly
confounded. These counterfactual images are then used to regularize the
downstream classifier such that the learned representations are the same
across various generative factors conditioned on the class label. Our method
is computationally efficient, simple to implement, and works well for any
number of generative factors and confounding variables. Our experimental
results on both synthetic (MNIST variants) and real-world (CelebA) datasets
show the usefulness of our approach.

##  1 Introduction

A confounder is a variable that causally influences two or more variables that
are not necessarily directly causally dependent (Pearl, 2001). Often, the
presence of confounders in a data-generating process is the reason for
spurious correlations among variables in the observational data. The bias
caused by such confounders is inevitable in observational data, making it
challenging to identify invariant features representative of a target variable
(Rothenhäusler et al., 2021; Meinshausen & Bühlmann, 2015; Wang et al., 2022).
For example, the demographic area an individual resides in often confounds the
race and perhaps the level of education that individual receives. Using such
observational data, if the goal is to predict an individual’s salary, a
machine learning model may exploit the spurious correlation between education
and race even though those two variables should ideally be treated as
independent variables. Removing the effects of confounding in trained machine
learning models has shown to be helpful in various applications such as zero
or few-shot learning, disentanglement, domain generalization, counterfactual
generation, algorithmic fairness, healthcare, etc. (Suter et al., 2019;
Kilbertus et al., 2020; Atzmon et al., 2020; Zhao et al., 2020; Yue et al.,
2021; Sauer & Geiger, 2021; Goel et al., 2021; Dash et al., 2022; Reddy et
al., 2022; Dinga et al., 2020).

In observational data, confounding may be observed or unobserved and can pose
various challenges in learning models depending on the task. For example,
disentangling spuriously correlated features using generative modeling when
there are confounders is challenging (Sauer & Geiger, 2021; Reddy et al.,
2022; Funke et al., 2022). As stated earlier, a classifier may rely on non-
causal features to make predictions in the presence of confounders (Schölkopf
et al., 2021). Recent years have seen a few efforts to handle the spurious
correlations caused by confounding effects in observational data (Träuble et
al., 2021; Sauer & Geiger, 2021; Goel et al., 2021; Reddy et al., 2022).
However, these methods either make strong assumptions on the underlying causal
generative process or require strong supervision. In this paper, we study the
adversarial effect of confounding in observational data on a classifier’s
performance and propose a mechanism to marginalize such effects when
performing data augmentation using counterfactual data. Counterfactual data
generation provides a mechanism to address such issues arising from
confounding and building robust learning models without the additional task of
building complex generative models.

The causal generative processes considered throughout this paper are shown in
Figure 1(a). We assume that a set of generative factors (attributes)
$Z_{1},Z_{2},\dots,Z_{n}$ (e.g., background, shape, texture) and a label $Y$
(e.g., cow) cause a real-world observation $X$ (e.g., an image of a cow in a
particular background) through an unknown causal mechanism $g$ (Peters et al.,
2017b). To study the effects of confounding, we consider
$Y,Z_{1},Z_{2},\dots,Z_{n}$ to be confounded by a set of confounding variables
$C_{1},\dots,C_{m}$ (e.g., certain breeds of cows appear only in certain
shapes or colors and appear only in certain countries). Such causal generative
processes have been considered earlier for other kinds of tasks such as
disentanglement Suter et al. (2019); Von Kügelgen et al. (2021); Reddy et al.
(2022). The presence of confounding variables results in spurious correlations
among generative factors in the observed data, whose effect we aim to remove
using counterfactual data augmentation.

Figure 1: (a) causal data generating process considered in this paper (CONIC =
Ours); (b) causal data generating process considered in CGN (Sauer & Geiger,
2021).

A related recent effort by (Sauer & Geiger, 2021) proposes Counterfactual
Generative Networks (CGN) to address this problem using a data augmentation
approach. This work assumes each image to be composed of three Independent
Causal Mechanisms (ICMs) (Peters et al., 2017a) responsible for three fixed
factors of variations: shape, texture, and background (as represented by
$Z_{1},Z_{2}$, and $Z_{3}$ in Figure 1(b). This work then trains a generative
model that learns three ICMs for shape, texture, and background separately,
and combines them in a deterministic fashion to generate observations. Once
the ICMs are learned, sampling images by making interventions to these
mechanisms give counterfactual data that can be used along with training data
to improve classification results. However, fixing the architecture to
specific number and types of mechanisms (shape, texture, background) is not
generalizable, and may not directly be applicable to settings where the number
of underlying generative factors is unknown. It is also computationally
expensive to train different generative models for each aspect of an image
such as texture, shape or background.

In this work, we begin with quantifying confounding in observational data that
is generated by an underlying causal graph (more general than considered by
CGN) of the form shown in Figure 1(a). We then provide a counterfactual data
augmentation methodology called CONIC (COunterfactual geNeratIon under
Confounding). We hypothesize that the counterfactual images generated using
the proposed CONIC method provide a mechanism to marginalize the causal
mechanisms responsible for spurious correlations (i.e., causal arrows from
$C_{i}$ to $Z_{j}$ for some $i,j$). We take a generative modeling approach and
propose a neural network architecture based on conditional CycleGAN (Zhu et
al., 2017) to generate counterfactual images. The proposed architecture
improves CycleGAN’s ability to generate quality counterfactual images under
confounded data by adding additional contrastive losses to distinguish between
fixed and modified features, while learning the cross domain translations. To
demonstrate the usefulness of such counterfactual images, we consider
classification as a downstream task and study the performance of various
models on unconfounded test set. Our key contributions include:

  * •

We formally quantify confounding in causal generative processes of the form in
Fig 1(a), and study the relationship between correlation and confounding
between any pair of generative factors.

  * •

We present a counterfactual data augmentation methodology to generate
counterfactual instances of observed data, that can work even under highly
confounded data ($\sim 95\%$ confounding) and provides a mechanism to
marginalize the causal mechanisms responsible for confounding.

  * •

We modify conditional CycleGAN to improve the quality of generated
counterfactuals. Our method is computationally efficient and easy to
implement.

  * •

Following previous work, we perform extensive experiments on well-known
benchmarks – three MNIST variants and CelebA datasets – to showcase the
usefulness of our proposed methodology in improving the accuracy of a
downstream classifier.

##  2 Related Work

Counterfactual Inference: (Pearl, 2009), in his seminal text on causality,
provided a three-step procedure for generation of a counterfactual data
instance, given an observed instance: (i) Abduction: abduct/recover the values
of exogenous noise variables; (ii) Action: perform the required intervention;
and (iii) Prediction: generate the counterfactual instance. One however needs
access to the underlying structural causal model (SCM) to perform the above
steps for counterfactual generation. Since real-world data do not come with an
underlying SCM, many recent efforts have focused on modeling the underlying
causal mechanisms generating data under various assumptions. These methods
then perform the required intervention on specific variables in the learned
model to generate counterfactual instances that can be used for various
downstream tasks such as classification, fairness, explanations etc. (Kusner
et al., 2017; Joo & Kärkkäinen, 2020; Denton et al., 2019; Zmigrod et al.,
2019; Pitis et al., 2020; Yoon et al., 2018; Bica et al., 2020; Pawlowski et
al., 2020).

Generating Counterfactuals by Learning ICMs: In a more recent effort, assuming
any real-world image is generated with three independent causal mechanisms for
shape, texture, background, and a composition mechanism of the first three,
(Sauer & Geiger, 2021) developed Counterfactual Generative Networks (CGN) that
generate counterfactual images of a given image. CGN trains three Generative
Adversarial Networks (GANs) (Goodfellow et al., 2014b) to learn shape,
texture, background mechanisms and combine these three mechanisms using a
composition mechanism $g$ as $g(shape,texture,background)=shape\odot
texture+(1-shape)\odot background$ where $\odot$ is the Hadamard product. Each
of these independent mechanisms is given an input of noise vector $u$ and a
label $y$ specific to that independent mechanism while training. Once the
independent mechanisms are trained, counterfactual images are generated by
sampling a label and a noise vector corresponding to each mechanism and then
feeding the input to CGN. Finally, a classifier is trained with both original
and counterfactual images to achieve better test time accuracy, showing the
usefulness of CGN. However, such deterministic nature of the architecture is
not generalizable to the case where the number of underlying generative
factors are unknown and it is computationally infeasible to train generative
models for specific aspect of an image such as texture/background.

Disentanglement and Data Augmentation: The spurious correlations among
generative factors have been considered in disentanglement (Funke et al.,
2022; von Kügelgen et al., 2021). The general idea in these efforts is to
separate the causal predictive features from non-causal/spurious predictive
features to predict an outcome. Our goal is different from disentanglement,
and we focus on the performance of a downstream classifier instead of
separating the sources of generative factors. Traditional data augmentation
methods such as rotation, scaling, corruption, etc. (Hendrycks et al., 2020;
Devries & Taylor, 2017; Zhang et al., 2018; Yun et al., 2019) do not consider
the causal generative process and hence they can not remove the confounding in
the images via data augmentation (e.g., color and shape of an object can not
be separated using simple augmentations). We hence focus on counterfactual
data augmentations that is focused on marginalizing the confounding effect
caused by confounders.

A similar effort to our paper is by (Goel et al., 2021) who use CycleGAN to
generate counterfactual data points. However, they focus on the performance of
a subgroup (a subset of data with specific properties) which is different from
our goal of controlling confounding in the entire dataset. Another recent work
by (Wang et al., 2022) considers spurious correlations among generative
factors and uses CycleGAN to generate counterfactual images. Compared to these
efforts, rather than using CycleGAN directly, we propose a CycleGAN-based
architecture that is optimized for controlled generation using contrastive
losses.

Applications of Counterfactuals: Augmenting the training data with appropriate
counterfactual data has shown to be helpful in many applications ranging from
vision to natural language tasks (Joo & Kärkkäinen, 2020; Lample et al., 2017;
Kusner et al., 2017; Kaushik et al., 2019; Dash et al., 2022). (Joo &
Kärkkäinen, 2020) identified existing biases in computer vision APIs deployed
in the real world by Amazon, Google, IBM, and Clarifai by looking at the
differences made by those APIs on counterfactual images that differ by
protected/sensitive attributes (e.g., race and gender). Using locally
independent causal mechanisms, (Pitis et al., 2020) augmented training data
with counterfactual data points in a model-free reinforcement learning
setting. Here, the idea is to use any two factual trajectories of an episode
and combine the two trajectories at a particular point in time to generate the
counterfactual data point, which will then be added to the replay buffer.
Independently factored samples are essential to get plausible and realistic
counterfactual instances.

##  3 Information Theoretic Measure of Confounding

Background and Problem Formulation: Let $\\{Z_{1},Z_{2},\dots,Z_{n}\\}$ be a
set of random variables denoting the generative factors of an observed data
point $X$, and $Y$ be the label of the observation $X$. Each generative factor
$Z_{i}$ (e.g., color) can take on a value form a discrete set of values
$\\{z_{i}^{1},\dots,z_{i}^{d}\\}$ (e.g., red, green etc.). Let the set
$S=\\{Y,Z_{1},\dots,Z_{n}\\}$ generates $N$ real-world observations
$\\{X_{i}\\}_{i=1}^{N}$ through an unknown causal mechanism $g$. Each $X_{i}$
can be thought of as an observation generated using the causal mechanism $g$
with certain intervention on the variables in the set $S$. Variables in $S$
may potentially be confounded by a set of confounders
$C=\\{C_{1},\dots,C_{m}\\}$ that denote real-world confounding such as
selection bias. Let $\mathcal{D}$ be the dataset of real-world observations
along with corresponding values taken by $\\{Y,Z_{1},\dots,Z_{n}\\}$. Causal
graph in Figure 1(a) shows the general form of this setting. From a causal
effect perspective, each variable in $S$ has a direct causal influence on the
observation $X$ (e.g., the causal edge $Z_{i}\rightarrow X$) and also has non-
causal influence on $X$ via the confounding variables $C_{1},\dots,C_{m}$
(e.g., $Z_{i}\leftarrow C_{j}\rightarrow Z_{k}\rightarrow X$ for some $C_{j}$
and $Z_{k}$). These paths via the confounding variables, in which there is an
incoming arrow to the variables in $S$, are also referred to as backdoor paths
(Pearl, 2001). Due to the presence of backdoor paths, we may observe spurious
correlations among the variables in $S$ in the observational data
$\mathcal{D}$.

In any downstream application where $\mathcal{D}$ is used to train a model
(e.g., classification, disentanglement etc.), it is desirable to minimize or
remove the effect of confounding variable to ensure that a model is not
exploiting the spurious correlations in the data to arrive at a decision. In
this paper, we present a method to remove the effect of such confounding
variables using counterfactual data augmentation. We start by studying the
relationship between the amount of confounding and the correlation between any
pair of generative factors in causal processes of the form shown in Figure
1(a).

######  Definition 3.1.

No Confounding (Pearl, 2009). In a causal directed acyclic graph (DAG)
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ denotes the set
of variables and $\mathcal{E}$ denotes the set of directed edges denoting the
direction of causal influence among the variables in $\mathcal{V}$, an ordered
pair $(Z_{i},Z_{j});Z_{i},Z_{j}\in\mathcal{V}$ is unconfounded if and only if
$p(Z_{i}=z_{i}|do(Z_{j}=z_{j}))=p(Z_{i}=z_{i}|Z_{j}=z_{j}),\forall
z_{i},z_{j}$. Where $do(Z_{i}=z_{i})$ denotes an intervention to the variable
$Z_{i}$ with the value $z_{i}$. This definition can also be extended to
disjoint sets of random variables.

Definition 3.1 provides the notion of no confounding, however, to quantify the
notion of confounding between a pair of variables, we consider the following
definition that relates the interventional distribution $p(Z_{i}|do(Z_{j}))$
and the conditional distribution $p(Z_{i}|Z_{j})$.

######  Definition 3.2.

(Directed Information (Raginsky, 2011; Wieczorek & Roth, 2019)). In a causal
directed acyclic graph (DAG) $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}$ denotes the set of variables and $\mathcal{E}$ denotes the set
of directed edges denoting the direction of causal influence among the
variables in $\mathcal{V}$, the directed information from a variable
$Z_{i}\in\mathcal{V}$ to another variable $Z_{j}\in\mathcal{V}$ is denoted by
$I(Z_{i}\rightarrow Z_{j})$. It is defined as follows.

$\displaystyle I(Z_{i}\rightarrow Z_{j})$ $\displaystyle\coloneqq
D_{KL}(p(Z_{i}|Z_{j})||p(Z_{i}|do(Z_{j}))|p(Z_{j}))\coloneqq\mathbb{E}_{p(Z_{i},Z_{j})}\log\frac{p(Z_{i}|Z_{j})}{p(Z_{i}|do(Z_{j}))}$
(1)

Using Definitions 3.1 and 3.2, it is easy to see that the variables $Z_{i}$
and $Z_{j}$ are unconfounded if and only if $I(Z_{j}\rightarrow Z_{i})=0$. Non
zero directed information $I(Z_{j}\rightarrow Z_{i})$ entails that,
$p(Z_{i}|Z_{j})\neq p(Z_{i}|do(Z_{j}))$ and hence the presence of confounding
(if there is no confounder, $p(Z_{i}|Z_{j})$ should be equal to
$p(Z_{i}|do(Z_{j}))$). Also, it is important to note that the directed
information is not symmetric (i.e., $I(Z_{i}\rightarrow Z_{j})\neq
I(Z_{j}\rightarrow Z_{i})$) (Jiao et al., 2013). We use this fact in defining
the measure of confounding below. Since we need to quantify the notion of
confounding (as opposed to no confounding), we use directed information to
quantify confounding as defined below.

######  Definition 3.3.

(An Information Theoretic Measure of Confounding.) In a causal directed
acyclic graph (DAG) $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}$ denotes the set of variables and $\mathcal{E}$ denotes the set
of directed edges denoting the direction of causal influence among the
variables in $\mathcal{V}$, the amount of confounding between a pair of
variables $Z_{i}\in\mathcal{V}$ and $Z_{j}\in\mathcal{V}$ is equal to
$I(Z_{i}\rightarrow Z_{j})+I(Z_{j}\rightarrow Z_{i})$.

Since directed information is not symmetric, we define the measure of
confounding to include the directed information from one variable to the other
for a given pair of variables $Z_{i},Z_{j}$. We now relate the quantity
$I(Z_{i}\rightarrow Z_{j})+I(Z_{j}\rightarrow Z_{i})$ with the correlation
between generative factors so that it is easy to quantify the amount of
confounding in observational data. Before that, we present the following
proposition which will be used in the proof of the subsequent proposition.

######  Proposition 3.1.

In causal processes of the form 1(a), the interventional distribution
$p(Z_{i}|do(Z_{j}))$ is same as the marginal distribution $p(Z_{i})$.

###### Proof.

In causal processes of the form 1(a), let $C^{\prime}$ denote the set of all
confounding variables that are part of some backdoor path from $Z_{i}$ to
$Z_{j}$. That is $C^{\prime}=\\{C|Z_{i}\leftarrow C\rightarrow Z_{j}\\}$ for
some $i,j$. Then we can evaluate the quantity $p(Z_{i}|do(Z_{j}))$ as

$\displaystyle p(Z_{i}|do(Z_{j}))$
$\displaystyle=\sum_{C^{\prime}}p(Z_{i}|Z_{j},C^{\prime})p(C^{\prime})=\sum_{C^{\prime}}p(Z_{i}|C^{\prime})p(C^{\prime})=\sum_{C^{\prime}}p(Z_{i},C^{\prime})=p(Z_{i})$

Where the first equality is because of the adjustment formula (Pearl, 2001)
and the second equality is because of the fact that $Y$ is a collider in
causal graph 1(a) and hence conditioned on $C^{\prime}$, $Z_{i}$ is
independent of $Z_{j}$. ∎

######  Proposition 3.2.

For causal generative processes of the form 1(a), the correlation between a
pair of generative factors $(Z_{i},Z_{j})$ is proportional to the amount of
confounding between $Z_{i}$ and $Z_{j}$.

###### Proof.

Expanding the quantity $I(Z_{i}\rightarrow Z_{j})+I(Z_{j}\rightarrow Z_{i})$,
we get the following,

$\displaystyle I(Z_{i}\rightarrow Z_{j})+I(Z_{j}\rightarrow
Z_{i})=\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i}|Z_{j})}{p(Z_{i}|do(Z_{j}))})\right]+\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{j}|Z_{i})}{p(Z_{j}|do(Z_{i}))})\right]$
(2)
$\displaystyle=\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i}|Z_{j})p(Z_{j}|Z_{i})}{p(Z_{i}|do(Z_{j}))p(Z_{j}|do(Z_{i}))})\right]=\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i}|Z_{j})p(Z_{j}|Z_{i})}{p(Z_{i})p(Z_{j})})\right]$
$\displaystyle=\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i}|Z_{j})p(Z_{j})p(Z_{j}|Z_{i})p(Z_{i})}{p(Z_{i})p(Z_{j})p(Z_{i})p(Z_{j})})\right]=\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i},Z_{j})p(Z_{j},Z_{i})}{p(Z_{i})^{2}p(Z_{j})^{2}})\right]$
$\displaystyle=2\times\mathbb{E}_{Z_{i},Z_{j}}\left[\log(\frac{p(Z_{i},Z_{j})}{p(Z_{i})p(Z_{j})})\right]=2\times
I(Z_{i};Z_{j})$

Where $I(Z_{i};Z_{j})$ is the mutual information between $Z_{i}$ and $Z_{j}$.
The third equality is due to Proposition 3.1. Since non-zero mutual
information implies positive correlation, we see that the amount of
confounding between $Z_{i}$ and $Z_{j}$ is directly proportional to the
correlation between $Z_{i}$ and $Z_{j}$. Hence, we use the correlation as a
measure of confounding between generative factors in the causal processes of
the form 1(a). ∎

Using the connection between the confounding and correlation in causal graph
1(a), our objective is to generate counterfactual data such that the resultant
dataset after augmentation looks similar to the data obtained from a causal
process where there is no confounding between generative factors (i.e., no
paths of the from $Z_{i}\leftarrow C_{j}\rightarrow Z_{k};\forall i,j,k$).
Equivalently, our counterfactual data generation algorithm removes the
spurious correlations between generative factors by marginalizing the causal
arrows $C_{i}\rightarrow Z_{j}$ for some $i,j$. To understand how
counterfactual instances break the correlations, consider the following
definition.

######  Definition 3.4.

(Counterfactual (Pearl, 2009)). Given an observed instance $X$ whose
generative factors $Z_{1},\dots,Z_{i},\dots,Z_{n}$ take on the values
$z_{1},\dots,z_{i},\dots,z_{n}$, the counterfactual instance $X^{\prime}$ of
$X$ (generated using the 3-step counterfactual inference procedure) differed
from $X$ w.r.t. the generative factor $Z_{i}$, is an instance whose generative
factors $Z_{1},\dots,Z_{i},\dots,Z_{n}$ take on the values
$z_{1},\dots,z_{i}^{\prime},\dots,z_{n}$. Here $Z_{i}$’s value is changed from
$z_{i}$ to $z_{i}^{\prime}$ through an external intervention
$do(Z_{i}=z_{i}^{\prime})$.

If we observe spurious correlation between two generative factors
$Z_{i},Z_{j}$ when they take on the values $z_{i}$ and $z_{j}$ respectively,
generating counterfactual instances w.r.t. $Z_{j}$ with the intervention
$do(Z_{j}=z_{j}^{\prime})$ and adding the counterfactual instances to original
data breaks the correlation between $Z_{i},Z_{j}$. With this idea, we now
present our algorithm to generate counterfactual images in a systematic manner
remove confounding from observational data.

##  4 CONIC: Methodology

Our goal is to remove the effect of confounding in the observational data on a
downstream task such as classification. To this end, we propose a way to
systematically generate counterfactual data that can marginalize the effect of
any confounding edge $C_{i}\rightarrow Z_{j}$ in Fig. 1 (a) as explained
below.

Removing The Confounding Effect of $C_{i}\rightarrow Z_{j}$: In the causal
graphs of the form 1(a), for paths of the form $Z_{j}\leftarrow
C_{i}\rightarrow Z_{l}$, we call the edges $C_{i}\rightarrow Z_{j}$ and
$C_{i}\rightarrow Z_{l}$ as confounding edges since together, their existence
is the reason for confounding in the data. Also, let $(z_{j}^{p},z_{l}^{q})$
is one pair of attribute values taken by the variable pair $(Z_{j},Z_{l})$
under extreme confounding (e.g., in the training set of colored MNIST dataset,
correlation coefficient of $0.99$ between color and digit is observed such
that whenever color is red, digit is $7$ etc.). To remove the effect of the
confounding edge $C_{i}\rightarrow Z_{j}$ w.r.t. the another confounding edge
$C_{i}\rightarrow Z_{l}$ (recall that confounding between $Z_{j},Z_{l}$ is
present if and only if there exists a pair of causal arrows $C_{i}\rightarrow
Z_{j}$ and $C_{i}\rightarrow Z_{l}$ for some $i$; due to this reason we
consider the confounding effect of the confounding edge $C_{i}\rightarrow
Z_{j}$ w.r.t. another confounding edge $C_{i}\rightarrow Z_{l}$), we consider
two subsets $T_{1},T_{2}$ of the observational data $\mathcal{D}$ which are
constructed as follows. $T_{1}$ consists of the set of instances for which
$Z_{j}\neq z_{j}^{p}$ and $Z_{l}=z_{l}^{q}$, $T_{2}$ consists of the set of
instances for which $Z_{j}=z_{j}^{p}$ and $Z_{l}=z_{l}^{q}$. The size of
$T_{1}$ is usually much smaller than the size of $T_{2}$ because of high
correlation between $Z_{j}$ and $Z_{l}$ (e.g., there are more red $7$’s than
non-red $7$’s).

Figure 2: Architecture of the proposed modified conditional CycleGAN to
generate counterfactual images. Pre-trained modules are shown in green color
and target attribute is shown in blue color. Note that, for simplicity, we
only show one pass of conditional CycleGAN (translation from $T_{1}$ to
$T_{2}$) in this figure.

Now, we learn a mapping $\mathcal{M}$ from the set $T_{1}$ to the set $T_{2}$
that changes the attribute $Z_{j}$ while fixing the value of $Z_{l}$ at
$z_{l}^{q}$. That is, for any given instance $X\in T_{1}$, for which
$Z_{j}\neq z_{j}^{p}$, $\mathcal{M}$ maps $X$ to a different instance
$X^{\prime}$ in which the value of the generative factor $Z_{j}$ is changed to
$z_{j}^{p}$ (e.g., $\mathcal{M}$ takes red $9$ as input and returns red $7$ as
output). This mapping $\mathcal{M}$ can be thought of as a function performing
the 3-step counterfactual inference: learning the underlying generative
factors, performing the intervention $do(Z_{j}=z_{j}^{p})$ and then generating
the counterfactual instance $X^{\prime}$. Now, given an instance $X$ for which
$Z_{j}\neq z_{j}^{p}$ and $Z_{l}\neq z_{l}^{q}$, using $\mathcal{M}$, we can
generate counterfactual instance $X^{\prime}$ in which $Z_{j}=z_{j}^{p}$ and
$Z_{l}\neq z_{l}^{q}$. These counterfactual instances, when augmented with the
original observed dataset $\mathcal{D}$, removes the effect of the confounding
edge $C_{i}\rightarrow Z_{j}$ w.r.t. the edge $C_{i}\rightarrow Z_{l}$. That
is, the counterfactual instances, when augmented with original data, breaks
the correlation between $Z_{j}$ and $Z_{l}$. This process can now be repeated
systematically for each confounding edge to generate counterfactual instances
that remove the spurious correlations. Such augmented data points which differ
from original data points w.r.t. only one feature (e.g., if original image is
a male with blond hair color, augmented image is same male with black hair
color) are referred as coupled sets by (Goel et al., 2021), images generated
by causal essential transformations by (Wang et al., 2022). The overall
procedure to generate counterfactual instances is summarized in Algorithm 1.

Earlier works use CycleGAN to generate counterfactual images that differ from
original image by a single attribute/feature (Wang et al., 2022; Goel et al.,
2021). Given two domains/sets of images that differ w.r.t. only one generative
factor $Z_{j}$, a CycleGAN can learn to translate between the two domains by
changing the attribute value of $Z_{j}$. In this case, one can think of
CycleGAN as a function performing the required intervention $Z_{j}$ and
generating counterfactual instance without modeling the underlying causal
process. Concretely, CycleGAN is an architecture used to perform unsupervised
domain translation using unpaired images. In a CycleGAN, a generator $G_{1}$
first transforms a given image $X$ from a domain/set $T_{1}$ into $X^{\prime}$
so that $X^{\prime}$ appears to come from another domain/set $T_{2}$ such that
certain features from input $X$ are preserved in the output $X^{\prime}$. A
discriminator $D_{T_{2}}$ then classifies whether the translated image
$X^{\prime}$ is original (i.e., sampled from $T_{2}$) or fake (i.e., generated
by $G_{1}$). A second generator $G_{2}$ transforms the image $X^{\prime}$ back
to original image $X$ to ensure that $G_{1}$ is using the contents of $X$ to
generate $X^{\prime}$. The same procedure is repeated to translate images from
domain $T_{2}$ into domain $T_{1}$. The loss function of CycleGAN can be
written as follows.

$\mathcal{L}_{CycleGAN}=\mathcal{L}_{GAN}(G_{1},D_{T_{2}},X,X^{\prime})+\mathcal{L}_{GAN}(G_{2},D_{T_{1}},X^{\prime},X)+\mathcal{L}_{cycle}(G_{1},G_{2})$
(3)

Where $\mathcal{L}_{GAN}$ is simple Generative Adversarial Network (GAN)
(Goodfellow et al., 2014a) loss and $\mathcal{L}_{cycle}$ is cycle consistency
loss measuring how well the output of $G_{2}$ is matching with the original
input $X$. For example,
$\mathcal{L}_{cycle}(G_{1},G_{2})=\mathbb{E}_{X\sim\mathcal{D}}[||G_{2}(G_{1}(X))-X||_{1}]$
can ensure that $G_{2}(G_{1}(X))=X$. In this work, to learn the mapping
function $\mathcal{M}$, we use conditional variant of CycleGAN to leverage the
supervision in terms of attribute values. For each generator, along with
input, we also feed a desired target attribute as shown in the Figure 2.

To improve the quality of counterfactual images generated by conditional
CycleGAN under extreme confounding, we propose a modification to conditional
CycleGAN as detailed below. As discussed earlier, $X^{\prime}$, the output of
$G_{1}$, can be thought of as a counterfactual image of $X$. When changing the
feature $Z_{j}$ of $X$, we keep the feature $Z_{l}$ fixed. That is, the
representation for $Z_{j}$ in both $X$ and $X^{\prime}$ should be different
and the representation for $Z_{l}$ in both $X$ and $X^{\prime}$ should be
same. To ensure this, as shown in Figure 2, along with two generators
$G_{1},G_{2}$ and a discriminator $D_{T_{2}}$ that are part of conditional
CycleGAN, we add two pre-trained discriminators $L_{1},L_{2}$ (shown in green
color in Fig. 2). $L_{1}$ takes two images $X,X^{\prime}$ as input and returns
high penalty if the representation of $Z_{j}$ is similar in $X,X^{\prime}$ and
small penalty otherwise. $L_{2}$ takes two images $X,X^{\prime}$ as input and
returns high penalty if the representation of $Z_{l}$ is different and small
penalty otherwise. Thus, our overall objective to generate good quality
counterfactual images is to train the modified conditional CycleGAN by
minimizing the following objective.

$\displaystyle\mathcal{L}_{conic}$
$\displaystyle=\mathcal{L}_{CycleGAN}+\alpha(-\mathcal{L}_{contrastive}(L_{1}(X),L_{1}(G_{1}(X)))+\mathcal{L}_{contrastive}(L_{2}(X),L_{2}(G_{1}(X)))$
(4)
$\displaystyle-\mathcal{L}_{contrastive}(L_{1}(X^{\prime}),L_{1}(G_{2}(X^{\prime})))+\mathcal{L}_{contrastive}(L_{2}(X^{\prime}),L_{2}(G_{2}(X^{\prime}))))$

Where $\alpha$ is a hyperparameter and $\mathcal{L}_{contrastive}$ is the
contrastive loss (Hadsell et al., 2006). For a pair of images
$(X,X^{\prime})$, $\mathcal{L}_{contrastive}$ defined as follows.

$\mathcal{L}_{contrastive}(X,X^{\prime})=AD^{2}+(1-A)\max(\epsilon-D,0)^{2}$
(5)

Where $A=1$ if $X,X^{\prime}$ belong to same class (or have same attribute
values), $A=0$ if $X,X^{\prime}$ belong to different classes (or have
different attribute values). $D$ is the distance between the representations
of $X,X^{\prime}$ (e.g., Euclidean distance). $\epsilon$ is the margin of
error allowed between two representations of the images of different classes.
$L_{1}$ and $L_{2}$ are pre-trained models and the parameters of $L_{1}$ and
$L_{2}$ are fixed. That is, the loss values returned by
$\mathcal{L}_{contrastive}$ are only used to update the trainable parameters
of conditional CycleGAN.

Algorithm 1 Counterfactual Generation to Remove the Effect of Confounding Edge
$C_{i}\rightarrow Z_{j}$

Result: Counterfactual images that remove the confounding effect caused by the
edge $C_{i}\rightarrow Z_{j}$

Input: $\mathcal{D}=\\{X_{i}\\}_{i=1}^{N}$,
$\texttt{Nodes}=\\{Z_{l}|C_{i}\rightarrow Z_{j}\&C_{i}\rightarrow Z_{l}\\}$

Initialize: $\texttt{cf\\_images}=[]$

for each $Z_{l}\in\texttt{Nodes}$ do

$T_{1}=\\{X\in\mathcal{D}|Z_{j}\neq z_{j}^{p}\&Z_{l}=z_{l}^{q}\\}$ \\* divide
data into two domains using attribute values * \

$T_{2}=\\{X\in\mathcal{D}|Z_{j}=z_{j}^{p}\&Z_{l}=z_{l}^{q}\\}$

$\mathcal{M}=\texttt{conditional CycleGAN}(T_{1},T_{2})$ \\* Learn
$\mathcal{M}$ to translate $T_{1}$ to $T_{2}$ * \

$\texttt{Factual\\_Imgs}=\\{X\in\mathcal{D}|Z_{j}\neq z_{j}^{p}\&Z_{l}\neq
z_{l}^{q}\\}$ \\* Pick factual images from train set * \

CFs = $\mathcal{M}(\texttt{Factual\\_Imgs})$ \\* Generate counterfactuals
using $\mathcal{M}$ * \

Append CFs to cf_images

end for

return cf_images

A Downstream Task - Image Classification: To measure the goodness of
counterfactual generation under confounding using Algorithm 1, we consider the
classification task on the unconfounded test set as a downstream task. Let
$\mathcal{D}^{aug}=\\{(X_{i},Y_{i})\\}_{i=1}^{M}$ be the dataset consisting of
original data points from $\mathcal{D}$ and corresponding counterfactual data
points. Usual empirical risk minimizer minimizes the following loss over
$\mathcal{D}$.

$\mathcal{L}_{erm}\coloneqq\mathbb{E}_{(X,y)\sim\mathcal{D}}[l(f_{\theta}(X),y)]$
(6)

Where $l$ is cross entropy loss. Using $\mathcal{D}^{aug}$, we minimize the
following loss $\mathcal{L}_{aug}$:

$\mathcal{L}_{aug}\coloneqq\mathbb{E}_{(X,y)\sim\mathcal{D}^{aug}}[l(f_{\theta}(X),y)]$
(7)

To further improve the performance of a classifier using $\mathcal{D}^{aug}$,
for each pair of images $X_{i},X_{j}$ we minimize the contrastive loss
$\mathcal{L}_{contrastive}(X_{i},X_{j})$ on the logits in the final layer.
Now, the final objective to optimize for classification task is to minimize
the following loss.

$\mathcal{L}=\mathcal{L}_{aug}+\lambda\mathbb{E}_{(X_{i},X_{j})\sim(\mathcal{D}^{aug}\times\mathcal{D}^{aug})}[\mathcal{L}_{contrastive}(X_{i},X_{j})]$
(8)

Where $\lambda>0$ is a regularization hyperparameter.

##  5 Experiments and Results

In this section, we present the experimental results on both synthetic (MNIST
variants) and real world (CelebA) datasets. Having access to the ground truth
generative factors (i.e., $Z_{1},\dots,Z_{n}$) of images,we artificially
create confounding in the training data and we leave test data to be
unconfounded (i.e., no correlation among generative factors). We compare CONIC
with various baselines including traditional Empirical Risk Minimizer (ERM),
Conditional GAN (CGAN) (Goodfellow et al., 2014a), Conditional VAE (CVAE)
(Kingma & Welling, 2013), Conditional-$\beta$-VAE (C-$\beta$-VAE) (Higgins et
al., 2017), AugMix (Hendrycks et al., 2020), CutMix (Yun et al., 2019),
Invariant Risk Minimization (IRM) (Arjovsky et al., 2019), and Counterfactual
Generative Networks (CGN) (Sauer & Geiger, 2021). To check the goodness of
each of these methods, we check how well the performance of the downstream
classifier on the test set is improved using the augmented images.

MNIST Variants: We construct the following three synthetic datasets based on
MNIST dataset (Lecun et al., 1998) and its colored, texture, and morpho (where
the digit thickness is controlled; Fig. 3) variants (Arjovsky et al., 2019;
Castro et al., 2019; Sauer & Geiger, 2021): (i) colored morpho MNIST (CM-
MNIST), (ii) double colored morpho MNIST (DCM-MNIST), and (iii) wildlife
morpho MNIST (WLM-MNIST). We consider extreme confounding among generative
factors as explained below.

Figure 3: Left: sample thin morpho MNIST images and corresponding labels.
Right: Sample thick morpho MNIST images and corresponding labels.

For the experimental results shown in Table 1, in the training set of CM-MNIST
dataset, the correlation coefficient between digit label and digit color
$r(label,color)$ is $0.95$ and the digits from $0$ to $4$ are thin and digits
from $5$ to $9$ are thick (see Figure 3). That is, $r(label,thin)=1$ if the
digit is in [0,1,2,3,4] else $r(label,thick)=1$. In the training set of DCM-
MNIST dataset, digit label, digit color, and background color jointly take a
fixed set of values 95% of the time. That is,
$r(label,color)=r(color,background)=r(label,background)=0.95$ and the digits
from $0$ to $4$ are thin and digits from $5$ to $9$ are thick. In the training
set of WLM-MNIST dataset digit shape, digit texture, and background texture
jointly take a fixed set of attribute values 95% of the time and the digits
from $0$ to $4$ are thin and digits from $5$ to $9$ are thick.

Model | CM-MNIST | DCM-MNIST | WLM-MNIST | CelebA
---|---|---|---|---
ERM | 46.41$\pm$ 0.81% | 43.31 $\pm$ 2.30% | 28.28 $\pm$ 0.70% | 70.64 $\pm$ 6.93%
CGAN | 41.86 $\pm$ 1.79% | 30.66 $\pm$ 3.86% | 17.50 $\pm$ 0.85% | 70.99 $\pm$ 2.35%
CVAE | 49.58 $\pm$ 1.50% | 41.99 $\pm$ 1.10% | 34.19 $\pm$ 1.58% | 71.50 $\pm$ 1.82%
C-$\beta$-VAE | 51.22 $\pm$ 1.00% | 51.58 $\pm$ 2.36% | 33.90 $\pm$ 1.87% | 74.29 $\pm$ 0.65%
AugMix | 47.36 $\pm$ 0.01% | 44.85 $\pm$ 0.02% | 26.30 $\pm$ 1.30% | 71.93 $\pm$ 4.64%
CutMix | 20.44 $\pm$ 1.22% | 23.10 $\pm$ 2.98% | 12.08 $\pm$ 1.59% | 73.66 $\pm$ 0.76%
IRM | 55.25 $\pm$ 0.89% | 49.71 $\pm$ 0.71% | 50.26 $\pm$ 0.48% | 72.30 $\pm$ 2.71%
CGN | 42.15 $\pm$ 3.89% | 47.50 $\pm$ 2.18% | 43.84 $\pm$ 0.25% | 69.25 $\pm$ 0.29%
CONIC | 65.57 $\pm$ 0.34% | 92.41 $\pm$ 0.26% | 77.72$\pm$ 1.00% | 79.56 $\pm$ 1.28%

Table 1: Test set accuracy results on MNIST variants and CelebA

In all of these MNIST variants, test set images are unconfounded (e.g., in the
test set of DCM-MNIST, any digit can be thin or think, can be in any
background color, can be in any foreground color). In these experiments, under
extreme confounding, our goal is to generate counterfactual images that break
the confounding among generative factors. We evaluate models on this grounds
by training a classifier using the augmented data and testing it on the
unconfounded test data. Table 1 shows the results in which CONIC outperforms
all the baselines. See Appendix for comparison of augmented images by various
baselines. Coninc uses only $10000$, $15000$, $15000$ counterfactual images in
CM-MNIST, DCM-MNIST, and WLM-MNIST experiments as augmented images
respectively to get improved performance. The regularization hyperparameter
$\lambda$ in Equation 8 set to $0.5$ for all MNIST experiments.

CelebA: Unlike MNIST variants, CelebA (Liu et al., 2015) dataset implicitly
contains confounding (e.g., the percentage of males with blond hair is
different from the percentage of females with blond hair, in addition to the
difference in the total number of males and females in the dataset). In this
experiment, we consider the performance of a classifier trained on the
augmented data that predicts hair color given an image. Our test set is the
set of males with blond hair.

Figure 4: Top: CelebA original images of males with non-blond hair color.
Bottom: Counterfactual images of males with blond hair generated using
Algorithm 1

We train models on the train set and test the performance on the set of males
with blond hair. Since the number of males with blond hair is very low in the
dataset (approximately 4% of males have blond hair), we show that the
augmenting the train set with only 10000 images of males with blond hair
improves the performance over baselines (see Table 1) whereas other baselines
require more than 50000 augmented images to get minor improvement over ERM.
Given a male image with non-blond hair, CONIC generates the counterfactual
image with blond hair without changing the male attribute (see Figure 4 for
sample counterfactual images). We also note that the deterministic models such
as CGN fail when they are applied to a different task where the number and
type of generative factors are not fixed and are difficult to separate (e.g.,
CelebA). CGN results in table 1 are obtained with only 1000 counterfactual
images as augmented data points. When we increase the number of counterfactual
instances, performance of CGN reduces further.

Time Complexity Analysis: Apart from its simple methodology, CONIC brings

Dataset | CONIC | CGN
---|---|---
CM-MNIST | 2.76 $\pm$ 0.19 | 103 $\pm$ 1.50
DCM-MNIST | 2.22 $\pm$ 0.01 | 103 $\pm$ 2.04
WLM-MNIST | 1.22 $\pm$ 0.01 | 111 $\pm$ 2.50
Table 2: Run time (in minutes) of CONIC compared to CGN on MNIST variants

additional advantages in terms of computing time required to train the model
that generates counterfactual images. As shown in Table 2, the time required
to run our method to generate counterfactual images w.r.t. a generative factor
$Z_{j}$ is significantly less than CGN that learns deterministic causal
mechanisms as discussed in Section 2. Even though we used CycleGAN in this
work, for the cases where the number of generative factors are more, StarGAN
(Choi et al., 2018) can be used to minimize the time required to learn the
mappings from one domain to another domain (Wang et al., 2022; Goel et al.,
2021).

##  6 Conclusions

We studied the adverse effects of confounding in observational data on the
performance of a classifier. We showed the relationship between confounding
and correlation in the causal processes considered, and we proposed a
methodology to remove the correlation between the target variable and
generative factors that works even when the dataset is highly confounded.
Specifically, we proposed a counterfactual data augmentation method that
systematically removes the confounding effect rather than addressing the
confounding problem through random augmentations. Using the generated
counterfactuals leads to substantial increase in a downstream classifier’s
accuracy. That said, we observed that the counterfactual quality can still be
improved, which will be interesting future work.

## References

  * dif (2022) Diffusion causal models for counterfactual estimation, 2022.
  * Arjovsky et al. (2019) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization, 2019.
  * Atzmon et al. (2020) Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik.  A causal view of compositional zero-shot recognition.  In _NeurIPS_ , 2020.
  * Bica et al. (2020) Ioana Bica, James Jordon, and Mihaela van der Schaar.  Estimating the effects of continuous-valued interventions using generative adversarial networks.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), _NeurIPS_ , volume 33, pp. 16434–16445. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/bea5955b308361a1b07bc55042e25e54-Paper.pdf.
  * Castro et al. (2019) Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker.  Morpho-MNIST: Quantitative assessment and diagnostics for representation learning.  _JMLR_ , 20(178), 2019.
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _CVPR_ , 2018.
  * Dash et al. (2022) Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma.  Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals.  In _WACV_ , 2022.
  * Denton et al. (2019) Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru.  Detecting bias with generative counterfactual face attribute augmentation, 2019.
  * Devries & Taylor (2017) Terrance Devries and Graham W. Taylor.  Improved regularization of convolutional neural networks with cutout.  _ArXiv_ , abs/1708.04552, 2017.
  * Dinga et al. (2020) Richard Dinga, Lianne Schmaal, Brenda W.J.H. Penninx, Dick J. Veltman, and Andre F. Marquand.  Controlling for effects of confounding variables on machine learning predictions.  _bioRxiv_ , 2020.
  * Funke et al. (2022) Christina M Funke, Paul Vicol, Kuan-Chieh Wang, Matthias Kuemmerer, Richard Zemel, and Matthias Bethge.  Disentanglement and generalization under correlation shifts.  In _ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality_ , 2022.  URL https://openreview.net/forum?id=H8QVUSuIqgq.
  * Goel et al. (2021) Karan Goel, Albert Gu, Yixuan Li, and Christopher Re.  Model patching: Closing the subgroup performance gap with data augmentation.  In _ICLR_ , 2021.
  * Goodfellow et al. (2014a) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014a.
  * Goodfellow et al. (2014b) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NIPS_ , pp. 2672–2680, 2014b.  URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun.  Dimensionality reduction by learning an invariant mapping.  In _CVPR_ , 2006.
  * Hendrycks et al. (2020) Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple method to improve robustness and uncertainty under data shift.  In _ICLR_ , 2020.  URL https://openreview.net/forum?id=S1gmrxHFvB.
  * Higgins et al. (2017) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning basic visual concepts with a constrained variational framework.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=Sy2fzU9gl.
  * Hu & Li (2021) Zhiting Hu and Li Erran Li.  A causal lens for controllable text generation.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=kAm9By0R5ME.
  * Idrissi et al. (2022) Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz.  Simple data balancing achieves competitive worst-group-accuracy.  In _Proceedings of the First Conference on Causal Learning and Reasoning_ , volume 177 of _Proceedings of Machine Learning Research_ , 2022\.
  * Jiao et al. (2013) Jiantao Jiao, Haim H Permuter, Lei Zhao, Young-Han Kim, and Tsachy Weissman.  Universal estimation of directed information.  _IEEE Transactions on Information Theory_ , 59(10):6220–6242, 2013.
  * Joo & Kärkkäinen (2020) Jungseock Joo and Kimmo Kärkkäinen.  Gender slopes: Counterfactual fairness for computer vision models by attribute manipulation.  In _Proceedings of the 2nd International Workshop on Fairness, Accountability, Transparency and Ethics in Multimedia_ , FATE/MM ’20, pp. 1–5, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450381482.  doi: 10.1145/3422841.3423533.  URL https://doi.org/10.1145/3422841.3423533.
  * Joshi & He (2022) Nitish Joshi and He He.  An investigation of the (in)effectiveness of counterfactually augmented data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3668–3681, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton.  Learning the difference that makes a difference with counterfactually-augmented data.  _arXiv preprint arXiv:1909.12434_ , 2019.
  * Kilbertus et al. (2020) Niki Kilbertus, Philip J Ball, Matt J Kusner, Adrian Weller, and Ricardo Silva.  The sensitivity of counterfactual fairness to unmeasured confounding.  In _UAI_ , 2020.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * Kusner et al. (2017) Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _NeurIPS_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf.
  * Lample et al. (2017) Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, and Marc' Aurelio Ranzato.  Fader networks:manipulating images by sliding attributes.  In _NeurIPS_ , 2017.
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.  doi: 10.1109/5.726791.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _ICCV_ , 2015.
  * Meinshausen & Bühlmann (2015) Nicolai Meinshausen and Peter Bühlmann.  Maximin effects in inhomogeneous large-scale data.  _The Annals of Statistics_ , 43(4):1801 – 1830, 2015.
  * Pawlowski et al. (2020) Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker.  Deep structural causal models for tractable counterfactual inference.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), _NeurIPS_ , volume 33, pp. 857–869. Curran Associates, Inc., 2020\.  URL https://proceedings.neurips.cc/paper/2020/file/0987b8b338d6c90bbedd8631bc499221-Paper.pdf.
  * Pearl (2001) Judea Pearl.  Direct and indirect effects.  In _UAI_ , pp. 411–420, 2001.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Peters et al. (2017a) J. Peters, D. Janzing, and B. Schölkopf.  _Elements of Causal Inference: Foundations and Learning Algorithms_.  MIT Press, Cambridge, MA, USA, 2017a.
  * Peters et al. (2017b) J. Peters, D. Janzing, and B. Scholkopf.  _Elements of Causal Inference: Foundations and Learning Algorithms_.  Adaptive Computation and Machine Learning series. MIT Press, 2017b.
  * Pitis et al. (2020) Silviu Pitis, Elliot Creager, and Animesh Garg.  Counterfactual data augmentation using locally factored dynamics.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), _NeurIPS_ , volume 33, pp. 3976–3990. Curran Associates, Inc., 2020\.  URL https://proceedings.neurips.cc/paper/2020/file/294e09f267683c7ddc6cc5134a7e68a8-Paper.pdf.
  * Raginsky (2011) Maxim Raginsky.  Directed information and pearl’s causal calculus.  In _2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_ , pp. 958–965, 2011.  doi: 10.1109/Allerton.2011.6120270.
  * Reddy et al. (2022) Abbavaram Gowtham Reddy, Benin L Godfrey, and Vineeth N Balasubramanian.  On causally disentangled representations.  In _AAAI_ , 2022.
  * Ronneberger et al. (2015) O. Ronneberger, P.Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , volume 9351 of _LNCS_ , pp. 234–241. Springer, 2015.
  * Rothenhäusler et al. (2021) Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, Jonas Peters, et al.  Anchor regression: Heterogeneous data meet causality.  _Journal of the Royal Statistical Society Series B_ , 83(2):215–246, 2021.
  * Sauer & Geiger (2021) Axel Sauer and Andreas Geiger.  Counterfactual generative networks.  In _ICLR_ , 2021.
  * Schölkopf et al. (2021) Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio.  Towards causal representation learning.  _CoRR_ , abs/2102.11107, 2021.  URL https://arxiv.org/abs/2102.11107.
  * Suter et al. (2019) Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer.  Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.  In _ICML_ , 2019.
  * Träuble et al. (2021) Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer.  On disentangled representations learned from correlated data.  In _ICML_ , 2021.
  * von Kügelgen et al. (2021) Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello.  Self-supervised learning with data augmentations provably isolates content from style.  In _NeurIPS_ , 2021.
  * Von Kügelgen et al. (2021) Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello.  Self-supervised learning with data augmentations provably isolates content from style.  _Advances in neural information processing systems_ , 34:16451–16467, 2021.
  * Wang et al. (2022) Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu.  Out-of-distribution generalization with causal invariant transformations.  In _CVPR_ , 2022.
  * Wieczorek & Roth (2019) Aleksander Wieczorek and Volker Roth.  Information theoretic causal effect quantification.  _Entropy_ , 21(10), 2019.
  * Yoon et al. (2018) Jinsung Yoon, James Jordon, and Mihaela van der Schaar.  GANITE: estimation of individualized treatment effects using generative adversarial nets.  In _6th ICLR, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=ByKWUeWA-.
  * Yue et al. (2021) Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang.  Counterfactual zero-shot and open-set visual recognition.  In _CVPR_ , 2021.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _ICCV_ , pp. 6022–6031, 2019.  doi: 10.1109/ICCV.2019.00612.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _ICLR_ , 2018.  URL https://openreview.net/forum?id=r1Ddp1-Rb.
  * Zhao et al. (2020) Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl.  Training confounder-free deep learning models for medical applications.  _Nature communications_ , 11(1):1–9, 2020.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _ICCV_ , 2017.
  * Zmigrod et al. (2019) Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology, 2019.

## Appendix

In this appendix, we include the following additional information, which we
could not fit in the main paper due to space constraints:

  * •

Additional implementation details

  * •

More details on related work

  * •

Empirical evidence on the relationship between confounding and correlation

  * •

Ablation studies

  * •

Counterfactual images of MNIST variants for various methods

##  Appendix A Additional Implementation Details

The generators used in CycleGAN are U-Networks Ronneberger et al. (2015) made
of a 4-layer encoder and a 4-layer decoder. The discriminators are 4-layer
encoders that output the probability of an image being a particular class.
Since all the contrastive loss terms in Equation 4 work together towards
removing the confounding effect of a confounding edge (unlike loss functions
where each term has a different purpose). Hence, we use a common weighting
term $\alpha$ in Equation 4. For all the experiments, batch size of 256 is
used to train classifier (Equation 8). In all of the architectures, leaky-relu
activation is used as an activation and sigmoid activation is used in the
final layer to get probabilities. Adam optimizer is used in all the
experiments.

##  Appendix B More on Related Work

In this section, we continue with the related work presented in the Section 2
of the main paper. Joshi & He (2022) discusses a potential issue for a
counterfactual data augmentation method, viz.: if counterfactual data
augmentation does not consider/augment counterfactuals w.r.t. all robust
features that are spuriously correlated with non-robust features, then the
performance of a model may drop in unseen distributions. To contrast this with
our work, since we are able to quantify the confounding and hence correlation
between any pair of generative factors, CONIC can generate all possible
counterfactuals, which may in fact help in generate counterfactual images
w.r.t. all robust/causal features. Hence, models trained on the counterfactual
images generated using CONIC are more robust.

Idrissi et al. (2022) is similar to our work in performing data augmentation
(for e.g., results on CelebA) with a difference that our method can be
extended to the performance on the entire test set instead of on the worst
group (e.g., MNIST results). Also, Hu & Li (2021) is similar in a sense to our
work, but aims at controllable generation and counterfactual generation in the
natural language setting. dif (2022) also generates counterfactual images but
it assumes the availability of the data generation process and does not
explicitly tackle confounding. Our work only assumes access to the attribute
information but not the data generating process.

##  Appendix C Relationship between correlation and confounding

Table 3 shows the empirical evidence that confounding is directly proportional
to correlation between generative factors in CM-MNIST dataset.

Correlation Coefficient (color, digit) | Confounding (color, digit) (Defn 3)
---|---
0.10 | 0.072
0.20 | 0.249
0.50 | 1.244
0.90 | 3.585
0.95 | 4.041
Table 3: Relationship between correlation coefficient and confounding between
color and digit in CM-MNIST dataset. Correlation is directly proportional to
confounding.

##  Appendix D Ablation Studies

In this section, we present the results on some ablation studies to understand
the usefulness of the proposed regularizers. Without the additional
regularizers in Eqn 4, the accuracy on the downstream classifier on CelebA is
$73.69\pm 1.10$. However, using the additional regularizer, the accuracy
improves upto $\mathbf{79.56\pm 1.28}$. The additional contrastive loss in Eqn
8 brings a slight improvement over Eqn 7. In CelebA experiments, while
accuracy obtained when using Eqn 7 is around $78.73\pm 1.22\%$ but using Eqn
8, the accuracy improved to $\mathbf{79.56\pm 1.28}$.

Also, to understand the performance of ERM model using only the 5%
unconfounded data, we experimented on MNIST variants and observed that the
accuracy on CM-MNIST, DCM-MNIST, and WLM-MNIST are $34.39\pm 0.02$, $17.22\pm
0.02$, $17.72\pm 0.05$, respectively. These results are worse than ERM model
trained on entire training dataset (Table 1). We believe that the reason for
these poor results by ERM model is due to the presence of multiple
confounders. In MNIST variants, along with the confounding between color and
digit, there is another feature called thickness that is challenging to learn,
especially when the digits are thin (Figure 3 left). When we take only 5%
unconfounded data, the train set size is very small, with many thin digits,
making it difficult for ERM to learn the features.

##  Appendix E Counterfactual Images by Various Methods

Figure 5 and 6 shows the counterfactual images generated by various methods on
MNIST variants.

Figure 5: Conditional GAN generations and their conditioned value on CM-MNIST
dataset. Because of extreme confounding, digit and shape are not de-confounded
by Conditional GAN model.

CM-MNIST Samples DCM-MNIST Samples WLM-MNIST Samples CM-MNIST CONIC DCM-MNIST
CONIC WLM-MNIST CONIC CM-MNIST CUTMix DCM-MNIST CUTMix WLM-MNIST CUTMix CM-
MNIST AUGMix DCM-MNIST AUGMix WLM-MNIST AUGMix CM-MNIST CGN DCM-MNIST CGN WLM-
MNIST CGN

Figure 6: Sample images from MNIST variants and augmented images by various
methods.


Joseph<EMAIL_ADDRESS>Andrew
Gilberthttps://www.surrey.ac.uk/people/andrew-gilbert1 Ranko
<EMAIL_ADDRESS>Adithya
<EMAIL_ADDRESS>Frank
Guerinhttps://www.surrey.ac.uk/people/frank-guerin1 University of Surrey
Guildford, UK National Institute of Technology Karnataka
Mangalore City,
Karnataka State, India, BMVC Author Guidelines

# Human-like Relational Models for Action Recognition in Video

###### Abstract

Video activity recognition by deep neural networks is impressive for many
classes. However, it falls short of human performance, especially for
challenging to discriminate activities. Humans differentiate these complex
activities by recognising critical spatio-temporal relations among explicitly
recognised objects and parts, for example, an object entering the aperture of
a container. Deep neural networks can struggle to learn such critical
relationships effectively. Therefore we propose a more human-like approach to
activity recognition, which interprets a video in sequential temporal phases
and extracts specific relationships among objects and hands in those phases.
Random forest classifiers are learnt from these extracted relationships. We
apply the method to a challenging subset of the something-something dataset
[Goyal et al. (2017)] and achieve a more robust performance against neural
network baselines on challenging activities.

Figure 1: In the something-something dataset [Goyal et al. (2017)] some
actions have very similar visual and motion features: Top: action “putting
something next to”; Bottom: action “pretending to put something next to”.
These actions are easily confused by deep learning approaches (See confusion
matrix Fig. 5). We can overcome this if we can identify the temporal segments
where key discriminating events happen and then look at the bounding boxes of
objects and hand and their motion and relations (e.g. contact) in these
segments. These critical features can discriminate these easily confused
categories.

##  1 Introduction

Current approaches to video activity recognition are generally based on deep
learning networks that internally learn local and global features. However,
superficially similar activities will often confuse these aproaches. For
example, in the AVA-Kinetics Challenge 2020, the winning mean average
precision was 39.6%, and for the most challenging classes, average precision
ranged from 3% to 14% 111https://research.google.com/ava/challenge.html.
Discriminating between such activities is of great importance in many
applications of significance to society, for example, monitoring older adults
in their residences and alerting them if an event requiring intervention has
occurred. Or to discriminate taking medicine from drinking water, or
differentiate pouring in a container from spilling, possibly causing an
electric or falling hazard. Also, in urban surveillance, to discriminate an
assault from a friendly hug or a theft from a legitimate object transfer.

Research has shown [Ben-Yosef and Ullman (2018)] that humans discriminate
similar activities by differences in relationships among critical parts of
objects, such as the position of the hand relative to another person’s back,
differentiating between fighting and hugging. While such relationships could,
in principle, be learnt by a deep learning approach, in practice, deep
learning tends not to make use of features that are easily interpretable by
humans, as the training process would need to be conditioned to take human
characteristics into account.

Therefore in this work, we take an opposite approach, a more top-down based
approach, using only a minimal number of higher level elements extracted from
the video that are more meaningful to humans. Specifically, we use only the
bounding boxes of critical objects and hands in the video frames, without any
other visual features or optical flow. From this, we compute relationships
among these essential objects and learn these patterns via machine learning.
Our goal is to investigate the performance that we could achieve by only using
this high-level interpretable information. Ultimately we can fuse this
complementary information with deep learning features for the most robust
performance.

##  2 Related Work

Current leading approaches to video activity recognition include Temporal
Shift Module (Lin et al., 2019), SpatioTemporal and Motion Encoding (Jiang et
al., 2019), R(2+1)D et al. (2018), D3D (et al., 2020), TRN B. Zhou et al.
(2018), and Temporal Reasoning Graphs (et al, 2020). These techniques use
bottom-up features derived from networks for object recognition, short-range
temporal information, and motion features, but no high level models explicitly
capturing relationships among parts. Human visual processing, in contrast,
seems to use models that explicitly capture relationships among significant
parts Ben-Yosef et al. (2018); Ben-Yosef and Ullman (2018). We also see
significant differences in the accuracy with which humans recognise
activities, especially for low quality or similar action videos Ben-Yosef et
al. (2020). For this reason, there is increasing interest in augmenting the
deep learning architectures with the kinds of features that humans seem to
use. For example, Wu et al. (2019) add long term features from regions of
interest. Recent work by Materzynska et al. (2020) augments pure bottom-up
deep learning with the explicit inclusion of detected bounding boxes for the
critical objects in the scene and object features for those objects. These are
concatenated with standard video features from pre-existing architectures such
as I3D and STRG Wang and Gupta (2018). However, how these meaningful bounding
boxes are used is determined by the black box neural network; we do not have
an interpretable explanation of why the video is determined to be in a
particular class. Such interpretability can be essential to debugging
erroneous classifications and making the system learn more human-like models
with higher accuracy.

For still images Ben-Yosef et al. (2018) presented a human-like approach that
learns models for specific classes; the models can then be applied top-down to
produce an interpretation of that image. The model contains parts and
relationships among them; an interpretation maps model parts to regions of the
image. Unfortunately, this same approach has not been applied to video to the
best of our knowledge. In a video, it would be necessary to map significant
parts in frames and map temporal segments of the video to temporal phases of
the activity.

##  3 Methodology

###  3.1 Methodology Overview

Figure 2: Overview of the proposed top-down recognition model. Sequential
frames are processed by a set of hand-crafted action models that consist of
object relations and scoring for each phase of an action. Feature vectors are
formed from the object relations of phase-assigned frames. These are used to
train a set of random forests

The high-level overview of our system222we will provide code repository link
in the final paper version. is depicted in Fig. 2. For our top-down
interpretable mode, we use as an input only the bounding boxes of the
principal objects and hands from the video frames (there are no lower level
visual or optical flow features from the videos). We interpret the video as a
possible member of each of the 17 action categories using 17 action specific
models (described below). We temporally segment the video into five phases for
each action category, representing the sequential stages involved in that
action. We then compute feature vectors characterising each phase and train a
random forest classifier for each category (with the positive examples being
that category and other categories being negative examples). When doing multi-
class classification, the highest probability random forest prediction is
returned as the class.

###  3.2 Assigning Action Phases and Computing Phase Features

In a traditional deep learning approach, a model architecture would be
handcrafted and trained. Instead, for each activity, we propose to handcraft a
‘model’ which interprets the video. We identified that all actions could be
segmented (temporally) into five broad phases, with a characteristic
difference in each phase. The phases are labelled $a,b,c,d$ or $e$ and a
typical segmentation is illustrated in Fig. 3 (top).

  * •

Phase $a$: The object(s) is present in the scene, the manipulation has not
happened;

  * •

Phase $b$: The hand enters (possibly carrying an object);

  * •

Phase $c$: The critical manipulation happens (e.g. object placed or picked);

  * •

Phase $d$: The hand departs (possibly carrying an object);

  * •

Phase $e$: The objects are present, with the result of the manipulation
evident.

Most videos depict all five phases, but some do not; e.g. some videos start at
phase $c$, with the hand already contacting the moved object, while other
videos finish at $c$ without sufficient frames to make a clear $d$.

An activity model consists of a scoring function for each of the five phases.
A scoring function for a phase $p$, for activity $\mathit{act}$, maps each
frame of a video to a real-valued score which is a heuristic to rate how well
that frame matches the typical features expected of a frame in phase $p$ of
$\mathit{act}$. Features used for scoring (and also eventually used to
characterise the phase) include binary values such as the presence of objects
and hand, and whether they are moving, and real-valued features such as the
relative motion between objects or object and hand (see Table 1).

Size($O_{1}$) = width $\times$ height
---
OverlapNormalised($O_{1}$,$O_{2}$) = (x overlap $\times$ y overlap) $/$
(0.1$\times$size of smaller box)
Offset($O_{1}$) = centre in current frame - centre in previous frame
OffsetDist($O_{1}$,$O_{2}$) = difference in offsets of $O_{1}$, $O_{2}$
OffsetAngle($O_{1}$,$O_{2}$) = difference in offset angles
CentreDist($O_{1}$,$O_{2}$) = difference in centres
Binary relations: Present($O_{1}$), Touching($O_{1}$,$O_{2}$),
Contained($O_{1}$,$O_{2}$),
CentreOnTop($O_{1}$,$O_{2}$), CentreUnderneath($O_{1}$,$O_{2}$),
MoveWithHand(..), HandMoveRelative(..), ObjectMoveRelative(..)
Table 1: The functions used to compute relational features between bounding
boxes of objects and hands within each frame of video. $O_{1}$ and $O_{2}$ are
two objects annotated with bounding boxes. As in Fig. 3 (top), $O_{1}$ is the
stationary object while $O_{2}$ is the moving one, but the system tries each
video with the order swapped as well, picking the best match. ‘Offset’ and
‘Move’ features are computed across two consecutive frames. The Move features
are boolean, true if movement exceeds a threshold.

Given a video we want to interpret, the scoring functions compute a ‘phase
score’ for each video frame. Thus, the scoring function describes how well
each frame matches the expected features for each phase. The phase scores
across frames of a sample video are depicted in Fig. 3 (bottom). The score for
phase $b$ peaks around a quarter of the way through the sequences when the
grapes move into the bowl. While phase $c$ peaks during the stationary portion
when the hand holds the grapes in the bowl before withdrawing in $d$, phase
$e$ is maximal in the later frames when the action is complete, and the result
is evident.

Figure 3: Top: Activity frames illustrating the 5 phases for the action
’Putting something into something’. (red and blue bounding boxes indicate
objects and light blue the hand)
Bottom: Phase scores for all frames in an example video. For example, the
second bar chart depicts applying phase $b$ scoring across all frames; we can
see that the beginning and end regions give low scores for phase $b$, while
the portion indicated by the top bracket gives the maximal response. Note: due
to different scales between phases, any comparisons in the phase assignment
algorithm are made within the same phase.

Given the frame-wise phase scores, a Gaussian kernel is convolved with the
sequence of scores for each phase to identify the maximal response. This
procedure helps to overcome noise in the phase scores. For example, the
bounding boxes can be noisy when the annotation (by a crowd-worker) is copied
from a previous frame but eventually is adjusted to the new appropriate size,
causing an extreme and noisy increase in the vector of the box’s motion. Also,
sometimes the hand may briefly take an object away from the centre of the
central object during phase $b$, especially if the main object is large and
the hand needs to move up to put an item in or on, see Fig. 4.

Figure 4: In this example Object 1’s trajectory is away from the centre of
Object 2, during phase $b$. In general a short segment of video cannot
unambiguously be assigned a phase; instead we need to find a best global
assignment for the video, meaning that the segment pictured is effectively
having phase $b$ imposed on it top-down, because that makes a good global
assignment.

There is a strict order $a,b,c,d,e$ to phases in an activity. Assigning frames
to phases is not simple; e.g. if we were to first assign phase $a$ in Fig. 3
we would assign it near the end of the video because the gaussian has the
highest response there; this is a wrong assignment. In general, the scores for
any particular phase could be misleading, but most of the phase scores are
likely accurate. Therefore we want to do an assignment that respects the
strict order while maximising the total matched phase scores across the whole
video. We approached this in a simple way that is not guaranteed to be
optimal.

We identify the centre frame $f$ for each phase based on the highest score
responses. However we perform this in a greedy fashion in the phase order:
$b,a,d,c,e$ i.e. $f_{b}$ the centre frame of the $b$ phase is assigned first,
then $f_{a}$ is assigned to the highest response region in frames preceding
$f_{b}$, $f_{d}$ is the highest response in frames succeeding $f_{b}$, etc.
This order is used as we heuristically found that phase $b$ is a key anchor to
the sequence. Each phase consists of the central frame and three frames on
each side, unless that would overlap another phase, in which case the phase
stops at the midpoint.

We also did an alternative assignment picking the second best response for
$b$, because sometimes the highest response $b$ is not the correct $b$, and if
so, assigning that to $b$ and restricting $c,d,e$ to subsequent frames will
produce a poor assignment (with low total score). To find the second best $b$,
we remove the best $b$ from the sequence and three frames on each side. The
complete assignment using the best $b$ and the complete assignment using the
second best $b$ are scored to find a winner. It could be the third or fourth
best $b$ is the correct one, But we found going as far as the second was
adequate.

Finally, this entire procedure was repeated for swapped assignments of object
1 and object 2 (because we cannot assume that the annotated object 1 is the
primary object doing the movement), creating four alternative assignments,
each of which is scored the winner chosen.

###  3.3 Phase feature vector

To construct a feature vector to represent each phase. The detected central
frames, and a window of +/- $n$ frames (generally $n=3$) are selected to
represent each phase $p$. The phase feature vector $F_{p}$ is defined as the
concatenation of four data statistics over the phase scores $s$.

$F_{p}=\\{s_{mean},s_{med},s_{max},s_{min}\\}$ (1)

Where $s_{mean}$ is the mean of the $2n+1$ frame scores, $s_{med}$ is the
median and $s_{max}$ and $s_{min}$ and the maximum and minimum scores over the
selected frames. This is repeated for all the phases $a,b,c,d,e$ to create a
feature vector embedding $V$ for a video sequence of
$V=\\{F_{a},F_{b},F_{c},F_{d},F_{e}\\}$ which is the input to a random forest
classifier.

###  3.4 Activity Recognition with Random Forests

To classify the video sequences correctly, given the feature embedding created
by the phase ordered object relations $V$, we trained a random forest
classifier Breiman (2001). We also tried Gradient Boosting methods and Neural
Networks (MLP), but Random Forests performed best. We identified that due to
the responses in $V$ varying dramatically in scale and sparsity that even
scaling each feature by its standard deviation, MLP based neural networks were
out-performed by an ensemble-based random decision forest (RDF). An RDF is an
ensemble of random decision trees (RDTs), which considerably improves the
stability and accuracy of individual RDT. Ideally suited to diverse features
as the decision tree nodes define splits on each feature without combining
them.

A Decision Tree is a tree (and a type of directed, acyclic graph) in which the
nodes represent decisions or terminal nodes. The edges or branches are binary
(yes/no, true/false), representing possible paths from one node to another.
The root or topmost node of the tree (and there is only one root) is the
decision node that splits the dataset using a variable or feature that results
in the best splitting metric evaluated for each subset or class in the dataset
that results from the split. According to the splitting metric at each
decision node, the decision tree learns by recursively splitting the dataset
from the root onwards (in a greedy, node by node manner). The terminal nodes
are reached when the splitting metric is at a global extremum.

The Feature vectors are defined per action and so vary in length between
different activities. Because of this, rather than have one multi-class model,
we fit a binary-class random forest on each activity that detects its
presence. Then at test time, each video is run on all binary classifiers, each
producing a probability of the action. Thus, the multi-class problem is
resolved simply by taking random forest with the highest probability on the
video.

##  4 Results

The 20bn project "something-something" data set is an extensive, crowd-sourced
collection of over 220K videos, each depicting one of 174 actions being done
on one or more objects. The data set is introduced in Goyal et al. (2017)
which also provides a CNN model to recognise the actions that we take as a
baseline CNN. The dataset is challenging as the action classes can be very
similar and provide extensive confusion to algorithms. The "something-else"
project (Materzynska et al., 2020) extends upon the baseline by introducing
annotations to the data set in the form of defined and detected object and
hand bounding boxes. As our model involves human-defined abstract features, we
directly make use of these annotations. There are up to five objects, which we
call Object 1 ($O_{1}$), Object 2 ($O_{2}$), etc., but for the action
categories we analysed, only two were needed. We also run comparisons with the
"something-else" model that combines CNN features with the bounding boxes in a
more general way (we refer to as CNN+bbox).

We focus on a subset of the dataset, focusing on the set of related ’Putting’
and ’Taking’ actions such as _Pretending to put something next to something_
and _Putting something in front of something_. These are particularly hard to
classify for the existing works, and therefore we wish to focus our novel
approach around this area. In total there are a 17 actions(class ids: 14, 72,
73, 74, 76, 77, 83, 105, 106, 107, 108, 111, 113, 114, 119, 122 and 149) the
description of the actions are listed in the supplementary material. This
subset contains over 16K examples sequences ( 800,000 frames) for training and
1.6k for validation. so as to compare like-for-like we re-train baseline (VGG-
style 3D CNN with 11 layers) Goyal et al. (2017) and CNN+bbox Materzynska et
al. (2020) with this subset.

The object bounding box annotations are taken from Materzynska et al. (2020)
and use the weighted average precision as the metric for success as in common
when reporting results for this dataset. In Tab 2 the average precision across
the _something-something_ subset dataset for the specific actions are shown
together with the retrained baseline CNN and Temporal Relational Networks
(TRM) \- however TRM was trained on the complete dataset. We also show the
performance of the fusion of our work with the baseline CNN approach of Goyal
et al. (2017),  Fused:Proposed+ Goyal et al. (2017), and fusion with the work
of CNN+bbox Materzynska et al. (2020), Fused:Proposed+ Materzynska et al.
(2020).

Smth Smth Class 14 72 73 74 76 77 83 105 106 107 108 111 113 114 119 122 149
mAP Baseline CNN Goyal et al. (2017) 0.25 0.06 0.20 0.31 0.11 0.17 0.12 0.31
0.72 0.44 0.47 0.44 0.25 0.31 0.28 0.60 0.61 0.42 TRM B. Zhou et al. (2018)
0.21 0.06 0.30 0.45 0.23 0.10 0.11 0.54 0.67 0.40 0.61 0.55 0.27 0.33 0.28
0.53 0.57 0.37 Proposed 0.11 0.07 0.30 0.27 0.20 0.02 0.41 0.32 0.70 0.56 0.64
0.25 0.38 0.32 0.27 0.69 0.74 0.50 Fused:CNN+bbox Materzynska et al. (2020)
0.39 0.02 0.23 0.24 0.31 0.20 0.45 0.48 0.68 0.55 0.60 0.62 0.47 0.22 0.36
0.69 0.74 0.42 Fused:Proposed+Goyal et al. (2017) 0.30 0.15 0.38 0.43 0.33
0.11 0.46 0.40 0.77 0.49 0.61 0.43 0.37 0.39 0.34 0.72 0.70 0.52
Fused:Proposed+Materzynska et al. (2020) 0.39 0.06 0.42 0.39 0.39 0.11 0.56
0.47 0.68 0.56 0.70 0.45 0.46 0.39 0.36 0.67 0.78 0.57

Table 2: Precision rates for each method tested on the validation set the
Something Something ’hard’ subset of video sequences.

Our proposed relational model’s approach outperforms the Baseline CNN, but
there are significant differences in the samples they classify correctly, as
shown in Fig. 5.

Figure 5: Confusion matrices comparing our top-down model against the CNN
baseline. The numbered classes 0-16 match in sequence the something-something
classes in Table 2

This performance difference suggested the possibility of a combined, fused
classifier of our proposed relational classifier and the more common CNN based
low-level feature work of Goyal et al. (2017). We combine our model with the
predictions from the baseline CNN, the pre-threshold probability of the
baseline model is summed with ours before selecting the overall highest. The
Combined approach improves on the current state the art Materzynska et al.
(2020), by 2% across the 1600 videos. Similarly, when comparing the
performance of the fused model and the current sota Materzynska et al. (2020),
there are significant differences across categories, for example, for actions
14, 72 and 74. This demonstrates a substantial difference in the type of
relationship that the approach can capture with the human-like approach.

Figure 6: Phase assignment to “pretending to take something out of something”.
Figure 7: Classification rate (overall, not mAP) for our top-down model versus
the CNN baseline. Commonly and exclusively correct samples show the extent of
diversity between the classifiers. Figure 8: Selected images of highly
confused categories, see text for details.

Comparing the confusion matrices for our approach and the CNN+bbox Materzynska
et al. (2020), we see that the latter suffers significant confusion between
“putting something next to” and “pretending to put something next to”. At the
same time, ours is most decisive in these categories. The phase assignment of
our approach on samples from these categories is shown in Fig. 1. Even though
CNN+bbox uses the same bounding boxes as ours, how these are used within its
deep learning network somehow fails to capture the critical relationship. A
further example where ours is less confused than CNN+bbox is “pretending to
take something out of something”, depicted in Fig. 6. This is a difficult
category because of the variety of ways in which crowd workers have chosen to
interpret ‘pretending’ for this action. Some take the object and return it,
some merely pass over it, and some make several passes. However, there is at
least one simple and clear commonality: the object is always left at the end.
Our approach with sparse features makes it easier for the learner to focus on
such things.

###  4.1 Error Analysis

The interpretable approach makes it easier for us to diagnose errors. Our
worst performing category is “Pretending to put something underneath
something”. Inspection of the phase assignments shows difficulties with
assigning phase $b$ due to the prevalence of substantial $O_{2}$ objects in
this category (See red bounding box in Fig 8, left; i.e. the object that
something is put under); often all of the action happens when the hand and
$O_{1}$ are wholly contained in $O_{2}$. The heuristics that worked well to
identify $b$ in other categories included bounding boxes coming into contact
and overlapping boxes increasing; these fail here. The same issue causes
significant confusion in Fig. 5 where “Putting something underneath something”
(see Fig 8 second from left) is being mistaken for “Putting something into
something”.

A significant limitation of our approach is a lack of 3D information. For
example, to discriminate ‘into’ from ‘onto’ would be difficult for humans if
looking only at bounding boxes, especially when the camera views from a high
angle (see Fig 8, two images to the right). The only discriminating feature is
that the stationary object $O_{2}$ has higher walls (is a container).
Sometimes objects are moving closer together in 3D space. Still, in 2D
bounding boxes, they move away because the video starts with one object held
near the camera, and as it moves away from the camera, its bounding box gets
much smaller, and the centre of the box moves away from the target (in 2D).
Sometimes as an object $O_{1}$ is withdrawn, it is taken across the front of
$O_{2}$, so in 2D it looks like it is getting closer during the withdrawal
phase $d$. Furthermore, videos in some classes can only be discriminated by
understanding that the camera looks down or from the side, e.g. for ‘putting
something underneath’ versus ‘putting something behind’. A human clearly
infers 3D from the 2D videos, and uses this to interpret what is happening.
Inferring information such as whether surfaces are vertical or horizontal in
3D space would be very challenging in the types of close up videos common in
this dataset.

##  5 Conclusion

We have presented an alternative novel method to perform activity recognition
on challenging similar videos. We took inspiration from a more human-like
approach utilising object bounding boxes and created handcrafted rules to
identify the phases of the activity. These features extracted from the
identified phases are then trained through a standard random forest classifier
to provide state of the art performance on the challenging subset of the
Something Something dataset. In future, we aim to improve this performance
further through incorporating 3D depth detail and adding additional high level
human inspired features such as object recognition.

## References

  * B. Zhou et al. (2018) B. Zhou et al.  Temporal relational reasoning in videos.  In _ECCV_ , 2018.
  * Ben-Yosef and Ullman (2018) Guy Ben-Yosef and Shimon Ullman.  Image interpretation above and below the object level.  _Interface focus_ , 8 4, 2018.
  * Ben-Yosef et al. (2018) Guy Ben-Yosef, Liav Assif, and Shimon Ullman.  Full interpretation of minimal images.  _Cognition_ , 171:65 – 84, 2018.  ISSN 0010-0277.  https://doi.org/10.1016/j.cognition.2017.10.006.
  * Ben-Yosef et al. (2020) Guy Ben-Yosef, Gabriel Kreiman, and Shimon Ullman.  Minimal videos: Trade-off between spatial and temporal information in human and machine vision.  _Cognition_ , 201:104263, 2020.  ISSN 0010-0277.  https://doi.org/10.1016/j.cognition.2020.104263.
  * Breiman (2001) L. Breiman.  Machine learning.  In _Random forests_ , 2001.
  * et al. (2018) D. Tran et al.  A closer look at spatiotemporal convolutions for action recognition.  In _CVPR_ , 2018.  10.1109/CVPR.2018.00675.
  * et al. (2020) J. Stroud et al.  D3d: Distilled 3d networks for video action recognition.  In _IEEE WACV_ , 2020.
  * et al (2020) J. Zhang et al.  Temporal reasoning graph for activity recog.  _IEEE Tran. Image Proc._ , 29:5491–5506, 2020.  10.1109/TIP.2020.2985219.
  * Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The "something something" video database for learning and evaluating visual common sense.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5842–5850, 2017.
  * Jiang et al. (2019) B. Jiang, M. Wang, W. Gan, W. Wu, and J. Yan.  Stm: Spatiotemporal and motion encoding for action recognition.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2000–2009, 2019.  10.1109/ICCV.2019.00209.
  * Lin et al. (2019) J. Lin, C. Gan, and S. Han.  Tsm: Temporal shift module for efficient video understanding.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7082–7092, 2019.  10.1109/ICCV.2019.00718.
  * Materzynska et al. (2020) Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell.  Something-else: Compositional action recognition with spatial-temporal interaction networks.  In _CVPR_ , 2020.
  * Wang and Gupta (2018) Xiaolong Wang and Abhinav Gupta.  Videos as space-time region graphs.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 399–417, 2018.
  * Wu et al. (2019) Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.  Long-term feature banks for detailed video understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.