# Generative Counterfactuals for Neural Networks via Attribute-Informed
Perturbation

Fan Yang, Ninghao Liu, Mengnan Du, Xia Hu  Department of Computer Science and
Engineering, Texas A&M UniversityCollege Station, TX, USA, 77843 nacoyang,
nhliu43, dumengnan<EMAIL_ADDRESS>
(2020)

###### Abstract.

With the wide use of deep neural networks (DNN), model interpretability has
become a critical concern, since explainable decisions are preferred in high-
stake scenarios. Current interpretation techniques mainly focus on the feature
attribution perspective, which are limited in indicating _why_ and _how_
particular explanations are related to the prediction. To this end, an
intriguing class of explanations, named _counterfactuals_ , has been developed
to further explore the “what-if” circumstances for interpretation, and enables
the reasoning capability on black-box models. However, generating
counterfactuals for raw data instances (i.e., text and image) is still in the
early stage due to its challenges on high data dimensionality and unsemantic
raw features. In this paper, we design a framework to generate counterfactuals
specifically for raw data instances with the proposed Attribute-Informed
Perturbation (AIP). By utilizing generative models conditioned with different
attributes, counterfactuals with desired labels can be obtained effectively
and efficiently. Instead of directly modifying instances in the data space, we
iteratively optimize the constructed attribute-informed latent space, where
features are more robust and semantic. Experimental results on real-world
texts and images demonstrate the effectiveness, sample quality as well as
efficiency of our designed framework, and show the superiority over other
alternatives. Besides, we also introduce some practical applications based on
our framework, indicating its potential beyond the model interpretability
aspect.

††copyright: acmcopyright††journalyear: 2020††doi: 10.1145/1122445.1122456

##  1\. Introduction

The past decade has witnessed the success of deep neural networks (DNN) in a
wide range of application domains (Pouyanfar et al., 2018). Despite the
superior performance, DNN models have been increasingly criticized due to its
black-box nature (Doshi-Velez and Kim, 2017). Interpretable machine learning
techniques (Du et al., 2020) are thus becoming significantly vital, especially
in those high-stake scenarios, such as medical diagnosis. To effectively
interpret black-box DNNs, most approaches investigate the feature attributions
between input instances and output predictions through correlation analysis,
so that humans can have a sense of which part of the instance contributes most
to the model decision. A typical example is the heatmaps employed for image
classification (Selvaraju et al., 2017), where related saliency scores are
capable of indicating the feature importance for one particular prediction
label.

However, existing correlation-based explanations are neither discriminative
nor counterfactual (Pearl et al., 2009), since they are not able to help
understand why and how particular explanations are relevant to model
decisions. Thus, to further explore the decision boundaries of black-box DNN,
_counterfactuals_ have gradually come to the attention of researchers, as an
emerging technique for model interpretability. Counterfactuals are essentially
some synthetic samples within data distribution, which can flip the model
prediction. With counterfactuals, humans can understand how input changes
affect the model and conduct reasoning under the “what-if” circumstances. Take
a loan applicant who got rejection for instance. Correlation-based
explanations may simply indicate those most contributed features (e.g., income
and credit) for rejection, while counterfactuals are capable of showing how
the application could be accepted with certain changes (e.g., increase the
monthly income from $\$5,000$ to $\$7,000$).

Recent work have already made some initial attempts on conducting such
counterfactual analysis. The first line of research (Kim et al., 2016; Chen et
al., 2019) employed the prototype and criticism samples in the training set as
the raw ingredients for counterfactual analysis, even though those selected
samples are not counterfactuals in nature. Prototypes indicate the set of data
samples that best represent the original prediction label, while criticisms
are the samples with the desired prediction label which are close to the
decision boundary. Some other work (Goyal et al., 2019; Agarwal et al., 2019)
further utilized feature replacement techniques to create hypothetical
instances as counterfactuals, where a query instance and a distractor instance
are typically needed for counterfactual generation. The key of this kind of
methodologies lies in the effective feature extraction and efficient
replacement algorithm. Besides, contrastive intervention (Dhurandhar et al.,
2018; White and Garcez, 2019) on the query instance is another common way to
generate counterfactuals regarding to the desired label. By reasonably
perturbing input features, counterfactuals can be obtained in the form of
modified data samples.

Despite the existing efforts, generating valid counterfactuals for raw data
instances is still challenging due to the following reasons. First, effective
counterfactuals for certain label are not guaranteed to be existed in training
set, so the selected prototypes and criticisms are not always sufficient for
counterfactual analysis. The related sample selection algorithms are highly
possible to select some “unexpected” instances due to data constraints (Kim et
al., 2016), which would largely limit the reasoning on model behaviors.
Second, efficient feature replacement for raw data instances could be very
hard and time-consuming (Goyal et al., 2019). Also, relevant distractor
instances for replacement may not be available in particular scenarios
considering privacy and security issues, such as loan applications. Third,
modifying query samples with intervention can simply work on a limited types
of data, such as tabular data (White and Garcez, 2019) and naive image data
(Dhurandhar et al., 2018). For general raw data like real-world texts or
images, intervention operation in data space can be extremely complicated and
intractable, which makes it difficult to be used in practice.

To handle the aforementioned challenges of counterfactual generation for raw
instances, the high-dimension data space and unsemantic raw features are the
two obstacles ahead. To this end, in this paper, we design a framework to
generate counterfactuals specifically for raw data instances with the proposed
Attribute-Informed Perturbation (AIP) method. By utilizing the power of
generative models, we can obtain useful hypothetical instances within the data
distribution for counterfactual analysis. Essentially, our proposed AIP method
can guide a well-trained generative model to generate valid counterfactuals by
updating its parameters in the attribute-informed latent space, which is a
joint embedding space for both raw features and data attributes. Compared with
the original input space, attribute-informed latent space has two significant
merits for counterfactual generation: (1) raw features are embedded as low-
dimension ones which are more robust and efficient for generation; (2) data
attributes are modeled as joint latent features which are more semantic for
conditional generation. As for the construction of attribute-informed latent
space, we typically employ two types of losses to conduct the training of
generative models, where the reconstruction loss is used to guarantee the
quality of raw feature embedding and the discrimination loss is used to ensure
the correct attribute embedding. Through the gradient-based optimization, the
proposed AIP method can iteratively derive the valid generative
counterfactuals which are able to flip the prediction of target model. In the
experiments, although we simply consider the DNN as the target prediction
model, due to its general good performance for raw data instances, our
proposed framework can also be easily applied with other different prediction
models. The main contributions of this paper are summarized as follows:

  * •

We design a general framework to derive counterfactuals for raw data instances
by employing generative models, aiming to facilitate the reasoning on model
behaviors of black-box DNN;

  * •

We develop AIP to iteratively update the parameters of generative models in
the attribute-informed latent space, according to the counterfactual loss with
regards to the desired prediction label;

  * •

We evaluate the designed framework with AIP on several real-world datasets
including raw texts and images, and demonstrate the superiority both
quantitatively and qualitatively.

##  2\. Preliminaries

In this section, we briefly introduce some related contexts to our problem, as
well as some basics of the employed techniques.

###  2.1. Counterfactual Explanation

Counterfactual explanation is essentially a natural extension under the
framework of example-based reasoning (Rissland, 1991), where particular data
samples are provided to promote the understandings on model behaviors.
Nevertheless, counterfactuals are not common examples for model
interpretation, since they are typically generated under the “what-if”
circumstances which may not necessarily exist. According to the theory
proposed by J. Pearl (Pearl and Mackenzie, 2018), three distinct levels of
cognitive ability are needed to fully master the behaviors of a particular
model, i.e., _seeing_ , _doing_ and _imagining_ from the easiest to the
hardest. In fact, counterfactual explanation is just raised to meet the
imagining-level cognition for model interpretation.

Within the contexts of this paper, we only discuss counterfactuals under the
assumption of “closest possible world” (Wachter et al., 2017), where desired
outcomes can be obtained through the smallest changes to the world. To be
specific and simple without loss of generality, consider a binary
classification model $f_{\bm{\theta}}:~{}\mathbb{R}^{d}\rightarrow\\{0,1\\}$,
where $0$ and $1$ respectively indicate the undesired and desired output. The
model input $\mathbf{x}\in\mathbb{R}^{d}$ is further assumed to be sampled
from data distribution $\mathcal{P}(\mathbf{x})$. Then, given a query instance
$\mathbf{x}_{0}$ with the undesired model output (i.e.,
$f_{\bm{\theta}}(\mathbf{x}_{0})=0$), the corresponding counterfactual
$\mathbf{x}^{*}$ can be mathematically represented as:

(1)
$\mathbf{x}^{*}=\operatorname*{arg\,min}_{\mathbf{x}|\mathcal{P}(\mathbf{x})>\eta}l(\mathbf{x},\mathbf{x}_{0})\quad\mathrm{s.t.}\
f_{\bm{\theta}}(\mathbf{x}^{*})=1,$

where $l:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{+}$
indicates a distance measure defined in the input space, and $\eta>0$ denotes
the threshold which quantifies how likely the sample $\mathbf{x}$ is under the
distribution $\mathcal{P}(\mathbf{x})$. The obtained counterfactual
$\mathbf{x}^{*}$ is regarded to be valid if it can effectively flip the target
classifier $f_{\bm{\theta}}$ to the desired prediction.

Although finding counterfactuals is somewhat similar to generating adversarial
examples (in terms that both tasks aim to flip the model decision by minimally
perturbing the input instance), they are essentially different in nature.
Following the previous settings, the adversarial sample $\mathbf{x}^{adv}$ for
model $f_{\bm{\theta}}$, with query instance $\mathbf{x}_{0}$, can be
generally indicated by:

(2)
$\mathbf{x}^{adv}=\operatorname*{arg\,min}_{\mathbf{x}=\mathbf{x}_{0}+\bm{\delta}}\|\bm{\delta}\|_{p}\quad\mathrm{s.t.}\
f_{\bm{\theta}}(\mathbf{x}^{adv})\neq f_{\bm{\theta}}(\mathbf{x}_{0}),$

where $\bm{\delta}$ denotes the adversarial perturbation on the query,
$\|\cdot\|$ represents the norm operation and $p\in\\{\infty,1,2,\cdots\\}$.
Comparing with Eq. 1, we note that counterfactual example has two significant
differences from adversarial sample. First, counterfactual generation process
is subject to the original data distribution, while adversarial samples are
not constrained by the distribution. This difference brings about the fact
that counterfactuals are all in-distribution samples, but adversarial examples
are mostly out-of-distribution (OOD) samples. Second, counterfactual changes
on the query need to be human-perceptible, while adversarial perturbations are
usually inconspicuous (Sen et al., 2019). Therefore, the key problem of
counterfactual explanation actually lies in how to generate such in-
distribution sample, with human-perceptible changes on the query, to flip the
model decision as desired.

###  2.2. Generative Modeling

Generative modeling is a typical task under the paradigm of unsupervised
learning. Different from discriminative ones, which involves discriminating
input samples across classes, generative modeling aims to summarize the data
distribution of input variables and further create new samples that plausibly
fit into that distribution (Murphy, 2012). In practice, a well-trained
generative model is capable of generating new examples that are not only
reasonable, but also indistinguishable from real examples in the problem
domain. Conventional examples of generative modeling include Latent Dirichlet
Allocation (LDA) and Gaussian Mixture Model (GMM).

As emerging families of generative modeling, Generative Adversarial Network
(GAN) (Goodfellow et al., 2014) and Variational Auto-Encoder (VAE) (Kingma and
Welling, 2013) have been attracting lots of attentions due to their
exceptional performance in a myriad of applications, especially for the task
on image and text generation (Van den Oord et al., 2016; Hu et al., 2017). By
taking full advantage of their power on raw data with high dimensionality, we
are able to better investigate how those data samples were created in the
first place, which potentially benefits the generation of certain hypothetical
example. To this end, we specifically employ some advanced generative models
(i.e., GAN and VAE) to study the counterfactual explanation for black-box DNN
on raw data instances, providing effective generative counterfactuals for
better model understanding.

##  3\. Counterfactual Generation

In this section, we first introduce the designed generative counterfactual
framework for raw data instances. Then, we present how to specifically
construct the attribute-informed latent space with generative models. Finally,
we show the details of our proposed AIP method on how to effectively obtain
such counterfactuals.

###  3.1. Generative Counterfactual Framework

Figure 1. Designed framework for counterfactual sample.

We design a framework to create counterfactual samples for raw data instances,
as illustrated by Fig. 1. To effectively handle the high dimensionality and
unsemantic features, we utilize the generative modeling techniques to aid the
counterfactual generation process. Consider a target DNN
$F_{\bm{\phi}}:\mathbb{R}^{d}\rightarrow\\{1,\cdots,C\\}$, which is the black-
box model for counterfactual analysis, where $\mathbb{R}^{d}$ is the input
data space and $\\{1,\cdots,C\\}$ denotes the model prediction space with $C$
different outputs. Given a query instance $\mathbf{x}_{0}$,
$F_{\bm{\phi}}(\mathbf{x}_{0})=\mathbf{y}_{0}$ outputs a one-hot vector. To
effectively generate a valid counterfactual sample
$\mathbf{x}^{*}\in\mathbb{R}^{d}$ that can flip the $F_{\bm{\phi}}$ decision
to $\mathbf{y}^{*}\in\\{1,\cdots,C\\}$ as desired, a generative model is
trained to achieve this in the framework. The applied generative modeling
actually plays two important roles in the counterfactual generation process:
(1) generative modeling guarantees that all created instances are in-
distribution samples, since it can be regarded as a stochastic procedure that
generates samples $\mathbf{x}\in\mathbb{R}^{d}$ under the particular data
distribution $\mathcal{P}(\mathbf{x})$; (2) generative modeling generally
assumes that underlying latent variables can be mapped to the data space under
certain circumstances, which ensures the sufficient feasibility for
hypothetical examples. Thus, a well-trained generative model is the basis for
high-quality counterfactuals within the designed framework.

The employed generative model specifically serves two sub-tasks for
counterfactual generation, i.e., data _encoding_ and _decoding_. For raw data
instances like images, the input space $\mathbb{R}^{d}$ could be extremely
large, which makes it difficult and inefficient to directly create
counterfactuals for the query. In our designed framework, data encoding is
conducted to map the input data space to a low-dimension attribute-informed
latent space, which is formulated as a joint embedding space for both raw
features and data attributes. In this way, each data sample $\mathbf{x}$ can
be effectively encoded through the function
$G^{enc}_{\bm{\psi}}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}\oplus\mathbb{R}^{t}$,
where $\mathbb{R}^{k}$ is the latent space for raw feature embeddings,
$\mathbb{R}^{t}$ indicates the data attribute space, and $\oplus$ represents a
concatenation operator. Reversely, the mapping for decoding is from the
attribute-informed latent space to the original data space. The decoder
function can be similarly indicated by
$G^{dec}_{\bm{\omega}}:\mathbb{R}^{k}\times\mathbb{R}^{t}\rightarrow\mathbb{R}^{d}$.
Although $G^{enc}_{\bm{\psi}}$ and $G^{dec}_{\bm{\omega}}$ typically have two
different focuses, they are jointly trained as a whole generative model in an
end-to-end manner. The issues about how to derive $G^{enc}_{\bm{\psi}}$ and
$G^{dec}_{\bm{\omega}}$ will be particularly discussed in Sec. 3.2.

To finally obtain the counterfactual sample $\mathbf{x}^{*}$ for model
$F_{\bm{\phi}}$ with query $\mathbf{x}_{0}$, we further need to modify the
attribute-informed latent space of the deployed generative model.
Specifically, we use the proposed AIP method to update the attribute-informed
latent vector of $\mathbf{x}_{0}$, according to the counterfactual loss
calculated. Assuming
$G^{enc}_{\bm{\psi}}(\mathbf{x}_{0})=\mathbf{z}_{0}\oplus\mathbf{a}_{0}$
($\mathbf{z}_{0}\in\mathbb{R}^{k},\mathbf{a}_{0}\in\mathbb{R}^{t}$), AIP
method can jointly update $\mathbf{z}_{0}$ and $\mathbf{a}_{0}$, so as to
minimize the corresponding loss counter-factually. The overall counterfactual
loss consists of two parts, i.e., _prediction loss_ and _perturbation loss_.
Prediction loss is set to ensure the flip of model decisions, and perturbation
loss is involved to guarantee the “closest possible” changes on the query,
which are both indispensable for counterfactual generation. For the prediction
loss, we simply follow the common cross-entropy term, expressed as
$L_{d}(F_{\bm{\phi}}(\mathbf{x}),\mathbf{y}^{*})=-\mathbf{y}^{*}\log(F_{\bm{\phi}}(\mathbf{x}))-(1-\mathbf{y}^{*})\log(1-F_{\bm{\phi}}(\mathbf{x}))$.
For the perturbation loss, we employ two $l_{2}$ norms respectively on
$\mathbb{R}^{k}$ and $\mathbb{R}^{t}$, indicated by
$L_{b}(\mathbf{z},\mathbf{a},\mathbf{z}_{0},\mathbf{a}_{0})=\|\mathbf{z}-\mathbf{z}_{0}\|_{2}+\|\mathbf{a}-\mathbf{a}_{0}\|_{2}$
($\mathbf{z}\in\mathbb{R}^{k},\mathbf{a}\in\mathbb{R}^{t}$), to restrain the
query changes, which can also be regarded as a regularization term as well.
Further, the overall counterfactual loss can thus be represented as follows:

(3)
$L_{c}(\mathbf{z},\mathbf{a},\mathbf{z}_{0},\mathbf{a}_{0},\mathbf{y}^{*})=L_{d}\left(F_{\bm{\phi}}(G^{dec}_{\bm{\omega}}(\mathbf{z},\mathbf{a})),\mathbf{y}^{*}\right)+\alpha
L_{b}(\mathbf{z},\mathbf{a},\mathbf{z}_{0},\mathbf{a}_{0}),$

where $\alpha$ is a balance coefficient between the two loss terms. With the
proposed AIP method, the designed framework can generate the valid
counterfactual example $\mathbf{x}^{*}$ with the aid of optimized
$\mathbf{z}^{*},\mathbf{a}^{*}$ through the decoder function (i.e.,
$\mathbf{x}^{*}=G^{dec}_{\bm{\omega}}(\mathbf{z}^{*},\mathbf{a}^{*})$). The
details of the proposed AIP method will be introduced in Sec. 3.3.

###  3.2. Attribute-Informed Latent Space

Figure 2. General illustration of the attribute-informed latent space in
generative models. Particularly, blue arrows indicate the forward flow of
computations, while orange arrows indicate the back-propagation flow of
gradients. The dash lines denote the losses for generative model training.

Constructing an appropriate attribute-informed latent space is the key part
for generative modeling in our designed framework, which has direct influences
on the quality of generated counterfactuals. To achieve this, we need to well
train a generative model, better capturing the raw data features as well as
relevant data attributes, where embedded features can bring about more robust
bases for counterfactual analysis, and incorporated attributes are able to
provide more semantics for conditional generation. Here, the data attributes
mainly indicate those extra information from humans along with raw instances,
such as annotations or labels, which can usually be represented as one-hot
vectors.

In practice, it is common that different generative models are employed for
different tasks or data. Since different models typically involve disparate
architectures, their training schemes can totally differ from each other. Take
the GAN and VAE for example, where GAN is usually trained to obtain an
equilibrium between a generator and a discriminator function, while VAE is
typically trained to maximize a variational lower bound of the data log-
likelihood. Therefore, to better introduce how to specifically construct the
attribute-informed latent space with generative models, we present a general
illustration shown by Fig. 2, although it may not be fully representative for
all kinds of models.

We generally introduce the modeling process with an encoder-decoder structure,
which corresponds to the data encoding and decoding in our designed framework.
Essentially, the attribute-informed latent space can be regarded as an
extended code space of auto-encoders. By concatenating attribute vector
$\mathbf{a}$ to raw feature embedding $\mathbf{z}$, the decoder function aims
to achieve the conditional generation based on $\mathbf{a}$. To ensure the
attribute consistency between original sample $\mathbf{x}$ and generated
sample $\hat{\mathbf{x}}$, discriminator $D_{\xi}$ is particularly employed,
which is trained separately and used to classify the attributes of
$\hat{\mathbf{x}}$. To effectively train such generative model, two basic loss
terms are required, which are the reconstruction loss and discrimination loss.
The overall training can be indicated by:

(4)
$\min_{\psi,\omega}\quad\underbrace{\underset{\begin{subarray}{c}\mathbf{x}\sim\mathcal{P}(\mathbf{x})\\\
\mathbf{a}\sim\mathcal{P}(\mathbf{a})\end{subarray}}{\mathbb{E}}\
\sum_{i=1}^{t}-a_{i}\log\left(D_{\xi}^{i}(\hat{\mathbf{x}})\right)-(1-a_{i})\log\left(1-D_{\xi}^{i}(\hat{\mathbf{x}})\right)}_{\text{Discrimination
Loss}}\\\
+\underbrace{\underset{\mathbf{x}\sim\mathcal{P}(\mathbf{x})}{\mathbb{E}}\|\mathbf{x}-\hat{\mathbf{x}}\|_{2}}_{\text{Reconstruction
Loss}},$

where $a_{i}$ denotes the $i$-th attribute in $\mathbf{a}$, and $D_{\xi}^{i}$
indicates the prediction of $D_{\xi}$ on the $i$-th attribute. After
sufficient training, $G^{enc}_{\bm{\psi}}$ and $G^{dec}_{\bm{\omega}}$ can be
effectively obtained, and the attribute-informed latent space can be further
constructed with the aid of $G^{enc}_{\bm{\psi}}$. For specific tasks and
architectures, the generative modeling process could be further enhanced with
more specific losses or other advanced tricks.

###  3.3. Attribute-Informed Perturbation

With the obtained $G^{enc}_{\bm{\psi}}$ and $G^{dec}_{\bm{\omega}}$ for
generative modeling, we then introduce the proposed AIP method to finally
derive the counterfactual for target DNN $F_{\bm{\phi}}$ with the query
$\mathbf{x}_{0}$. To guarantee the quality of the generated counterfactuals,
AIP needs to find the sample that can minimize the counterfactual loss
indicated by Eq. 3. Under the “closest possible world” assumption, the
corresponding counterfactual sample can be denoted as:

(5)
$\mathbf{x}^{*}=G^{dec}_{\bm{\omega}}\left(\operatorname*{arg\,min}_{\mathbf{z}\in\mathbb{R}^{k},\
\mathbf{a}\in\mathbb{R}^{t}}L_{c}(\mathbf{z},\mathbf{a},\mathbf{z}_{0},\mathbf{a}_{0},\mathbf{y}^{*})\right).$

To effectively solve Eq. 5, the proposed AIP method utilizes an iterative
gradient-based optimization algorithm with dynamic step sizes (controlled by a
decaying factor $\beta$), which helps the iteration process converge faster.
In each iteration, the updated $\mathbf{z}$ and $\mathbf{a}$ can be derived as
follows:

(6) $\left\\{\begin{aligned}
\mathbf{z}^{(n+1)}=\mathbf{z}^{(n)}-\mu^{(n)}\nabla_{\mathbf{z}}L_{c}\left(\mathbf{z}^{(n)},\mathbf{a}^{(n)},\
\mathbf{z}_{0},\ \mathbf{a}_{0},\ \mathbf{y}^{*}\right)\\\
\mathbf{a}^{(n+1)}=\mathbf{a}^{(n)}-\gamma^{(n)}\nabla_{\mathbf{a}}L_{c}\left(\mathbf{z}^{(n)},\mathbf{a}^{(n)},\
\mathbf{z}_{0},\ \mathbf{a}_{0},\ \mathbf{y}^{*}\right)\end{aligned}\right.,$

where $n$ indicates the iteration index, $\mu$ and $\gamma$ respectively
denotes the step sizes of updates on $\mathbf{z}$ and $\mathbf{a}$.
Specifically, the proposed AIP method can be summarized in Algorithm 1. It is
important to note that AIP only works on the optimization of
$\mathbf{z},\mathbf{a}$, and does not involve the parameter update on
$F_{\bm{\phi}}$, $G^{enc}_{\bm{\psi}}$, $G^{dec}_{\bm{\omega}}$. Thus, the
proposed AIP method should be less time-consuming and easily deployed for
counterfactual generation task, compared with those generative frameworks
which need extra model training (Samangouei et al., 2018; Singla et al.,
2020).

Input: $F_{\bm{\phi}}$, $G^{enc}_{\bm{\psi}}$, $G^{dec}_{\bm{\omega}}$,
$\mathbf{x}_{0}$, $\mathbf{y}^{*}$, $\mu$, $\gamma$, $\alpha$, $\beta$,
$n_{\max}$

1

Output: Counterfactual sample $\mathbf{x}^{*}$

2

3Initialize $\mu$, $\gamma$, $\alpha$, $\beta$ ;

4

5Initialize $n=0,\ \mathbf{x}=\mathbf{x}_{0}$ ;

6

7Construct the latent space with $\mathbf{z}\oplus\mathbf{a}\leftarrow
G^{enc}_{\bm{\psi}}(\mathbf{x})$ ;

8

9while
_$F_{\bm{\phi}}(G^{dec}_{\bm{\omega}}(\mathbf{z},\mathbf{a}))\neq\mathbf{y^{*}}\
\mathrm{or}\ n\leq n_{\max}$_ do

10

11 Update $\mathbf{z}$ and $\mathbf{a}$ according to Eq. 6 ;

12

13 Update step sizes with $\mu\leftarrow\beta\mu$ and
$\gamma\leftarrow\beta\gamma$ ;

14

15 $n\leftarrow n+1$

16

17Reconstruct the optimized sample with $\mathbf{x}^{*}\leftarrow
G^{dec}_{\bm{\omega}}(\mathbf{z},\mathbf{a})$ ;

18

19if  _$F_{\bm{\phi}}(\mathbf{x}^{*})==\mathbf{y}^{*}$_ then

20

21 Return $\mathbf{x}^{*}$ as the counterfactual for $F_{\bm{\phi}}$ with
query $\mathbf{x}_{0}$;

22else

23 Return None – No valid counterfactual exists;

24

Algorithm 1 Attribute-Informed Perturbation (AIP)

##  4\. Experiments

In this section, we evaluate the designed counterfactual generation framework
with the proposed AIP method on several real-world datasets, both
quantitatively and qualitatively. Overall, we mainly conduct two sets of
experiments respectively on _text_ and _image_ counterfactual generation, by
utilizing different data modeling techniques. With all conducted experiments,
we aim to answer the following four key research questions:

  * •

How _effective_ is the designed framework in generating counterfactuals with
AIP, regarding to different types of raw data?

  * •

How is the _quality_ of created counterfactuals from our designed framework
aided by AIP, comparing with other methods?

  * •

How _efficient_ is the counterfactual generation under the designed framework
with AIP, comparing with other potential ways?

  * •

Can we further _benefit_ other practical tasks with the counterfactuals
generated from our design framework with AIP method?

###  4.1. Experimental Settings

####  4.1.1. Real-World Datasets.

Throughout the whole experiments, we employ three real-world datasets to
evaluate the performance of the designed framework with AIP method, including
both raw texts and images. The relevant data attributes depend on the
particular tasks, which are collected either from labels or annotations. The
statistics of the involved datasets are shown in Table 1.

Table 1. Dataset statistics in experiments. Datasets | #Instance | #Attribute | Type | Domain
---|---|---|---|---
Yelp | $455,000$ | 1 | Raw Texts | Sentiment
Amazon | $558,000$ | 1 | Raw Texts | Sentiment
CelebA | $202,599$ | 13 | Raw Images | Face Feature

  * •

Yelp User Review Dataset111https://www.yelp.com/dataset (Asghar, 2016): This
dataset consists of user reviews from the Yelp associated with relevant rating
scores. We involve a tailored and modified version of this data for our
experiments on text counterfactual generation. Specifically, we consider the
reviews with ratings higher than three as _positive_ samples and regard the
others as _negative_ ones, and we further use these sentiment labels as the
relevant attribute for data modeling. The vocabulary of our involved Yelp data
contains more than $9,000$ words, and the average review length is around $9$
words.

  * •

Amazon Product Review
Dataset222http://jmcauley.ucsd.edu/data/amazon/index_2014.html (He and
McAuley, 2016): This dataset is also involved as a raw textual dataset for our
experiments. Similar to the Yelp data, we modify the original rating
information of reviews into the sentiment categories (i.e., _positive_ and
_negative_), and further model these labels as an sentiment attribute of the
raw textual reviews. Amazon dataset has more than $50,000$ words in
vocabulary, and the average length is around $15$.

  * •

CelebFaces Attributes (CelebA)
Dataset333http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html (Liu et al., 2015):
This is a large-scale face attributes dataset, containing tons of raw face
images with human annotations. We employ this dataset for our experiments on
image counterfactual generation, and select out $13$ representative face
attributes (out of 40) for data modeling along with raw face images. The
involved thirteen attributes include: _Male_ , _Young_ , _Blond_Hair_ ,
_Pale_Skin_ , _Bangs_ , _Mustache_ and etc.

####  4.1.2. Target Model for Interpretation.

Since we mainly discuss the counterfactuals of raw data instances, DNN is a
better choice as our target model. For the target DNN $F_{\bm{\phi}}$ in our
experiments, we employ some regular structures for the corresponding tasks.
Particularly, for the text sentiment classification, we use a common
convolutional architecture in (Kim, 2014) to pre-train a DNN classifier for
further counterfactual analysis. For the image attribute classification task,
similarly, we utilize a simple convolutional network (Guo et al., 2017) to
prepare a target classifier, where the model is trained with one of those
attributes as the label. During the evaluations on counterfactual generation,
target DNNs are fixed without further training.

####  4.1.3. Employed Generative Modeling Techniques.

In our experiments, different data modeling techniques are employed for
different types of data. In particular, we use different generative models to
construct the corresponding attribute-informed latent space, regarding to text
and image data. For textual reviews (Yelp, Amazon), we utilize the modeling
techniques introduced in (Hu et al., 2017), and build a transformer-based VAE
to effectively formulate the relevant attribute-informed latent space. For
face images (CelebA), we mainly follow the modeling method of AttGAN (He et
al., 2019), where more complicated training schemes are employed, compared
with the general one shown in Sec. 3.2, for better visual quality of the
generated images. Both of the employed generative models should be well-
trained on the corresponding datasets before the counterfactual generation
process, so as to guarantee the high quality of generated counterfactuals.

####  4.1.4. Alternative Methods and Baselines.

To effectively evaluate the performance of the designed framework with AIP, we
incorporate following alternative methods and baselines for comparison.

  * •

TextBugger (Li et al., 2018): This is a general method for adversarial text
generation, which is built based on the word attribution and bug selection.
The created text samples can effectively flip the prediction of the target
classifier. We employ this method as a baseline to specifically compare with
our generated text counterfactuals.

  * •

DeepWordBug (Gao et al., 2018): This is another method focusing on the
adversarial text generation, where a token scoring strategy is utilized to
guide the character-level adversarial perturbation. This method is employed as
a baseline for text counterfactuals as well.

  * •

FGSM (Goodfellow et al., 2015): Fast gradient sign method is a common way to
generate image adversarial samples, by using the gradients of the loss with
respect to the input. The sample created by this method can effectively
maximize the loss, so as to flip the original prediction. We employ this
method as a baseline specifically for our generated image counterfactuals.

  * •

Counter_Vis (Goyal et al., 2019): This is a recent method in generating image
counterfactuals, where particular image regions are replaced to flip the model
decision. We employ this method as an alternative method for image
counterfactual generation.

  * •

CADEX (Moore et al., 2019): This is a state-of-the-art method for
counterfactual generation, where the gradient-based method is directly applied
to modify the input space of query. This method is originally proposed for
tabular data, and we modify it simply as an alternative for image
counterfactuals, due to the particularity of texts.

  * •

xGEMs (Joshi et al., 2018): This is a state-of-the-art method for generating
counterfactuals, which also employs the generative modeling technique for
sample generation. This method only involves the latent space modeling and
cannot achieve the conditional generation with semantic attributes. We employ
this method as an important alternative for both text and image
counterfactuals.

  * •

AIP_R: This is the random version of our proposed AIP method, which updates
all parameters in a random way.

###  4.2. Implementations

In this part, we introduce the implementation details for our experiments on
text and image counterfactual generation, corresponding to the following Sec.
4.3 and Sec. 4.4.

####  4.2.1. Details for Text Counterfactuals

##### Hyper-parameters.

We implement all related algorithms and models by the PyTorch
framework444https://pytorch.org/. Specifically for text counterfactuals, we
set the balance coefficient $\alpha=0.8$ in Eq. 3 during the main evaluations.
The decaying factor is set as $\beta=0.95$, and the maximum iteration is set
as $n_{\max}=300$. As for the initial step sizes for optimization, we set
$\mu=1$ and $\gamma=2$ in the related experiments.

##### Target DNN model $F_{\mathbf{\phi}}$ for texts.

We train two CNN classifiers respectively on Yelp and Amazon datasets as our
target models, following the architectures introduced in work (Kim, 2014). In
particular, both CNN employ the _CNN-non-static_ version of training, as
illustrated in the paper. The deployed target CNN for Yelp data has $89.58\%$
testing accuracy, and the deployed CNN for Amazon has $88.16\%$ testing
accuracy.

##### Generative model $G^{enc}_{\psi},G^{dec}_{\omega}$ for texts.

We employ a transformer based VAE to conduct the relevant data modeling.
Specifically, the overall structure of the employed _Encoder_ and _Decoder_
can be illustrated by the following Tab. 2. The size of transformer and latent
space are both set as $256$. As for the training phase, we use another
classifier, trained separately with $F_{\mathbf{\phi}}$, as the discriminator
$D_{\xi}$. The relevant batch size is $128$, embedding dropout rate is $0.5$,
and learning rate is $10^{-3}$.

Table 2. Structure for the employed transformer-based VAE. Encoder (_from top to down_) |  Decoder (_from top to down_)
---|---
Embedding Layer | Multi-Head Attention Layers
Multi-Head Attention Layers | Addition & Normalization
Addition & Normalization | Dense Layer
Dense Layer | Addition & Normalization
Addition & Normalization | Multi-Head Attention Layers
Multi-Head Attention Layers | Fully-Connected Layer
GRU Layer | Softmax
Summation | /

####  4.2.2. Details for Image Counterfactuals

##### Hyper-parameters.

Similar as text counterfactuals, we also use PyTorch to implement relevant
models and algorithms. In particular, we set $\alpha=1.5$ in Eq. 3. The
decaying factor and maximum iteration in Algorithm 1 is respectively set as
$\beta=0.9$ and $n_{\max}=500$. Besides, the initial optimization step sizes
is set as $\mu=2$ and $\gamma=3$.

##### Target DNN model $F_{\mathbf{\phi}}$ for images.

We train a basic CNN on the CelebA dataset as the target according to the
architecture in (Guo et al., 2017), specifically using the “Male” annotation
as the label for training. Essentially, the target CNN here is a binary gender
classifier which predicts an input image as either “Male” or “Female”. The
deployed CNN has the testing accuracy with $95.67\%$.

##### Generative model $G^{enc}_{\psi},G^{dec}_{\omega}$ for images.

The generative modeling for CelebA data largely follows the training schemes
in (He et al., 2019). Our modeling attributes include: ”Pale_Skin”, ”Bangs”,
”Black_Hair”, ”Blond_Hair”, ”No_Beard”, ”Brown_Hair”, ”Bushy_Eyebrows”,
”Male”, ”Eyeglasses”, ”Young”, ”Mustache”, ”Bald”, ”Mouth_Slightly_Open”.
Particularly, we present our employed structure of _Encoder_ and _Decoder_ in
the following Tab. 3. Here, DeConvolution indicates the transposed operation
on convolution, and (64,4,2) respectively denotes the dimension, kernel size
and stride, for example. From the structure, we note that the latent size is
$1,024$. For the training phase, we specifically set the batch size as $32$,
and learning rate as $2\times 10^{-4}$. The discriminator $D_{\xi}$ is a
multi-class classifier, trained separately from $F_{\mathbf{\phi}}$.

Table 3. Structure for the employed AttGAN. Encoder (_from top to down_) |  Decoder (_from top to down_)
---|---
Convolution Layer (64,4,2) | DeConvolution Layer (1024,4,2)
Normalization | Normalization
Convolution Layer (128,4,2) | DeConvolution Layer(512,4,2)
Normalization | Normalization
Convolution Layer (256,4,2) | DeConvolution Layer(256,4,2)
Normalization | Normalization
Convolution Layer (512,4,2) | DeConvolution Layer(128,4,2)
Normalization | Normalization
Convolution Layer (1024,4,2) | DeConvolution Layer(3,4,2)
Normalization | /

###  4.3. Text Counterfactual Evaluations

In this part, we evaluate the experimental results of the designed framework
with AIP in generating text counterfactuals, regrading to a convolutional
neural network (CNN) built for sentiment classification. The involved raw
texts for target DNN come from the user/product reviews in Yelp and Amazon
datasets, where $90\%$ are used for training, $5\%$ for development and $5\%$
for testing.

####  4.3.1. Effectiveness Evaluation.

Figure 3. Effectiveness evaluation for text counterfactuals.

In order to evaluate the effectiveness for text counterfactuals, we employ the
metric _Flipping Ratio_ (FR) to measure the relevant performance, which
reflects how likely the generated text samples would flip the model decision
to $\mathbf{y}^{*}$. Specifically, FR can be calculated as follows:

(7)
$\mathrm{FR}=|\mathcal{X}_{f}|\Big{/}|\mathcal{X}_{q}|\quad\left(\mathbf{x}_{0}\in\mathcal{X}_{f}\
\mathrm{if}\ F_{\bm{\phi}}(\mathbf{x}_{0})=\mathbf{y}^{*}\right),$

where $\mathcal{X}_{f}$ indicates the set of query samples with which new
flipping instances can be generated by particular methods, and
$\mathcal{X}_{q}$ denotes the set of all testing queries. In our experiments,
there are $500$ testing queries in total (i.e., $|\mathcal{X}_{q}|=500$),
which are randomly selected from the test set. Fig. 3 illustrates our
experimental results on both Yelp and Amazon datasets. According to the
numerical results, we note that our designed framework with AIP can work well
on both datasets, and has competitive performance among all other alternatives
as well as baselines, although TextBugger achieves the highest FR score with
better robustness. Besides, we also observe that AIP_R does not effectively
work for generating flipping samples, which indicates that random optimization
in attribute-informed latent space cannot help for counterfactual sample
generation.

####  4.3.2. Quality Evaluation.

Figure 4. Quality evaluation for text counterfactuals.

As for the quality assessment of counterfactual samples, we employ the _Latent
Perturbation Ratio_ (LPR) metric to measure the latent closeness between the
generated sample $\mathbf{x}^{*}$ and original query instance
$\mathbf{x}_{0}$. Since high-quality counterfactual samples typically need to
ensure sparse changes in the robust feature space, thus the smaller the LPR
is, the better the counterfactual we have. To be specific, the LPR can be
calculated by:

(8) $\mathrm{LPR}=\big{\|}\mathbf{z}^{*}-\mathbf{z}_{0}\big{\|}_{0}\Big{/}k,$

where $\|\cdot\|_{0}$ indicates the $l_{0}$ norm operation, $\mathbf{z}^{*}$
and $\mathbf{z}_{0}$ are the raw feature embeddings respectively for
$\mathbf{x}^{*}$ and $\mathbf{x}_{0}$. To make a fair comparison, we use the
same encoder function $G^{enc}_{\bm{\psi}}$ for all generated samples to
obtain the corresponding latent representation vectors. In this set of
experiments, the latent dimension is $256$ (i.e., $k=256$), and the final LPR
value for particular method is recorded with the average over $500$ testing
queries. The relevant numerical results are presented in Fig. 4. From the
experiments, it is noted that xGEMs and the proposed AIP method significantly
outperform other baselines, indicating that the corresponding generated
samples actually maintain more robust features regarding to the query.
Furthermore, the proposed AIP is noted to be slightly better than xGEMs, which
may partially result from the conditional generation brought by attribute
vector $\mathbf{a}$. This set of results also validate a fact that adversarial
samples typically utilize some artifacts to flip the model decisions, instead
of using some robust features.

####  4.3.3. Efficiency Evaluation.

Figure 5. Efficiency evaluation for text counterfactuals.

To compare the efficiency, we record the time consumption for each method over
$500$ testing queries in the generation phase on the same machine.
Specifically, we calculate the average time cost for one query, and further
employ this as the metric to access the efficiency for particular methods.
Fig. 5 shows the relevant experimental results. Based on the statistics, it is
observed that adversarial related methods (i.e., TextBugger and DeepWordBug)
typically consume less time per query in average, compared with the
counterfactual generation methods, which is mainly due to the fact that
adversarial methods do not need to conduct encoding computations before sample
generation. As for our proposed AIP method, the time efficiency is roughly the
same as the alternative xGEMs, but it is significantly better than its random
version AIP_R which needs more iterations to converge.

####  4.3.4. Qualitative Case Studies.

Table 4. Case studies on generated text samples.

Counterfactual on _Negative_ sentiment (Yelp)
---
Query: this is the worst walmart neighborhood market out of any of them
TextBugger: this is the worst wa1mart neighborho0d market out of a ny of them
DeepWordBug: this id the wosrt walmart neighobrhood market out of any of htem
xGEMs: that is good walmart market out of any neighborhood
AIP: this is the best walmart neighborhood market for all of them
Counterfactual on _Positive_ sentiment (Amazon)
Query: this item works just as i thought it would
TextBugger: this item w0rks just as i tho ught it wou1d
DeepWordBug: this item wroks just ae i thought it wolud
xGEMs: this item works out poorly just as i thought disappointed
AIP: this item works bad just as i thought it would not play

Figure 6. Evaluations for image counterfactual generation. Figure 7.
Qualitative case studies on generated image samples.

Here, we present several representative case studies from different methods,
shown in Tab. 4, aiming to provide a qualitative comparison for generated text
samples. Based on the Tab. 4, we can see that adversarial texts typically
provide limited insights for humans on counterfactual analysis, since they
mainly make use of the model artifacts to flip the prediction. Nevertheless,
with the samples generated by xGEMs and AIP, we can easily observe some
sentiment variation regarding to the query instance, which sheds light on
model behaviors and facilitates further human reasoning on black-box models.
Besides, compared with xGEMs, the proposed AIP method usually can generate
more sensible counterfactuals with the aid of attribute conditions.

###  4.4. Image Counterfactual Evaluations

In this part, we specifically evaluate the designed framework with AIP on
image counterfactual generation. Instead of simply considering one attribute
for conditional generation in texts, we take multiple attributes into account
for image counterfactuals. In this set of experiments, our target DNN follows
the common CNN architecture and is trained as a gender classifier, which can
classify an input image as _Male_ or _Female_. All involved raw images for
target DNN come from the CelebA dataset, and we use $90\%$ data for training,
$5\%$ for development, $5\%$ for testing. The relevant quantitative results
are all illustrated by Fig. 6.

####  4.4.1. Effectiveness Evaluation.

For the effectiveness assessment, we still use the FR metric indicated by Eq.
7. In the experiments, we set $|\mathcal{X}_{q}|=500$, and aim to test how
many of them can be effectively flipped with particular methods. Fig. 6(a)
illustrates the relevant numerical results, where adversarial method FGSM
performs the best on FR and can flip nearly every testing query. We note that
the proposed AIP method ranks the second, and outperforms other counterfactual
generation methods. Besides, it is also observed that CADEX and AIP_R performs
relatively bad for the image counterfactual task within certain iterations,
even though CADEX is proved to work well for tabular instances (Moore et al.,
2019).

####  4.4.2. Quality Evaluation.

Similar to text counterfactual scenario, we employ the LPR metric, shown as
Eq. 8, to measure the quality of the generated image counterfactuals. In
experiments, the latent dimension $k$ constructed by $G^{enc}_{\bm{\psi}}$ is
$1,024$ (i.e., $k=1024$), and the corresponding LPR for particular method is
recorded by calculating the average over $500$ testing queries. Relevant
experimental results are shown by Fig. 6(b). Based on the LPR comparison, we
note that the samples generated by FGSM and CADEX change a lot in the latent
feature space, because both methods directly rely on the input perturbation
for sample generation. As for the proposed AIP, it achieves the lowest LPR
among all the alternatives and baselines, and it is significantly better than
its random version AIP_R.

####  4.4.3. Efficiency Evaluation.

We similarly employ the average time consumption per query to evaluate the
efficiency aspect for image counterfactual generation. Specifically, the
average time is obtained over the $500$ testing queries randomly selected from
the test set. Fig. 6(c) shows the relevant experimental results. According to
the statistics and comparison, we note that FGSM is the most efficient one,
and xGEMs consumes the least time in average among all other counterfactual-
based methods. As for the proposed AIP, a competitive efficiency performance
is observed, and is remarkably superior compared with that of Counter_Vis,
CADEX and AIP_R.

####  4.4.4. Qualitative Case Studies.

To facilitate a qualitative comparison among different methods, we
specifically show some case studies, illustrated by Fig. 7. We select several
query instances whose model predictions are female, and then employ different
methods to generate the corresponding image samples which flip the model
decisions for counterfactual purpose. According to the results, we note that
the samples generated by FGSM and CADEX do not have salient visual changes
regarding to the query instances, which largely limits the human reasoning on
model behaviors. Among other alternative methods, it is observed that the
proposed AIP is capable of generating counterfactuals with better visual
quality, which present much smoother transitions from female to male.

###  4.5. Influence of Hyper-parameter $\alpha$

In this part, we show some additional results on the influence of hyper-
parameter $\alpha$ in Eq. 3. Other experimental settings keep unchanged. The
relevant results are shown by Fig. 8.

Figure 8. Influence of $\alpha$ on FR and LPR metrics.

Based on the results, we observe that $\alpha$ serves as a knob to control the
effectiveness and sample quality of the designed framework. To select an
appropriate $\alpha$, we actually need to strike a balance between FR and LPR,
where the larger the $\alpha$ is, the lower the effectiveness is and the
higher the sample quality is. Different data types may also have different
trade-off curves.

###  4.6. Applications

In this part, we focus on some practical scenarios which may benefit from the
counterfactual samples generated by our designed framework. In particular, we
show the applications of the framework respectively on _feature interaction_
and _data augmentation_.

####  4.6.1. Feature Interaction

Figure 9. Feature interactions for the decision change.

Understanding the feature interaction could be very important in lots of real-
world domains. A typical example is the bias detection task, where humans aim
to find out a related set of features which can significantly influence the
correctness or fairness of model decision. Utilizing our designed framework
for counterfactual analysis can partially help this practical task. By
observing the perturbation scale on attribute vector $\mathbf{a}$ of the
generated counterfactual, humans can have a sense on which semantic features
contribute significantly to the flipping of model decision. To illustrate the
point, we show another case result from the designed framework with AIP in
Fig. 9. Here, we train an age classifier on the CelebA dataset as our target
DNN, and aim to analyze the feature interaction of a query prediction as
“Old”. Based on the attribute perturbations of the generated sample, we note
that the top semantic attributes are “Male”, “Bushy_Eyebrows”, “Black_Hair”
and “Bangs”, besides the target attribute. This result directly demonstrates
the fact that the “Male” attribute has a strong interaction with the predicted
attribute for this particular query, and the target DNN exists potential
gender bias for its age predictions.

####  4.6.2. Data Augmentation

Table 5. Model performance with data augmentation.

Dataset |  CNN (Kim, 2014) |  VDCNN (Conneau et al., 2017)
---|---|---
Yelp | Initial | 82.33% ($\pm$ 0.61%) | 88.79% ($\pm$ 0.53%)
Augmented | 83.16% ($\pm$ 0.57%) | 89.95% ($\pm$ 0.46%)
Amazon | Initial | 81.96% ($\pm$ 0.52%) | 88.55% ($\pm$ 0.63%)
Augmented | 82.41% ($\pm$ 0.49%) | 88.76% ($\pm$ 0.55%)

Dataset |  CNN (Guo et al., 2017) |  ResNet (He et al., 2016)
---|---|---
CelebA | Initial | 87.32% ($\pm$ 0.22%) | 90.96% ($\pm$ 0.27%)
Augmented | 88.85% ($\pm$ 0.21%) | 91.35% ($\pm$ 0.25%)

Another application of the designed framework is the data augmentation for
model training. By taking full advantage of the generated counterfactual
samples as new training instances, we aim to obtain better DNN models with
higher performance and robustness. Specifically, to test the improvement, we
train several DNN models on relatively smaller training sets, which are
essentially the subsets of original data. For the sentiment classifiers on
Yelp and Amazon, our initial training size is $20,000$, containing $10,000$
positive and $10,000$ negative reviews. The extra counterfactual training size
is $2,000$ whose queries are randomly selected from the initial training set.
For the binary age classifier on CelebA, we employ a similar setting for
training, where each class includes $10,000$ initial samples, and $2,000$
generated counterfactual samples are further incorporated for augmentation.
Relevant experimental results are shown in Tab. 5. Based on the statistics, we
note that the augmented training with counterfactual samples typically
achieves higher classification accuracies with smaller variances, which can
also be observed under some advanced DNN structures.

##  5\. Related Work

Generating counterfactual explanation is just one of many interpretation
methods for black-box models, which generally belongs to the family of
interpretable machine learning. According to the particular problems they
focus on, interpretation methods can be divided into the following three
categories in general.

The first category of methods aims to answer the “ _What_ ”-type questions,
i.e., what part of the input mostly contribute to the model prediction. A
representative work in this category is LIME (Ribeiro et al., 2016), where
authors specifically employ linear models to approximate the local decision
boundary and further formulate it as a sub-modular optimization problem for
model interpretation. The feature importance in LIME is essentially obtained
by observing the prediction changes after perturbing input samples. Similar
related methods can also be found in Anchors (Ribeiro et al., 2018) and SHAP
(Lundberg and Lee, 2017). Another common methodology under this category is to
utilize the model gradient information, where gradients are typically regarded
as an indicator for perturbation sensitivity. Related methods can be found in
GradCAM (Selvaraju et al., 2017), Integrated Gradients (Sundararajan et al.,
2017), and SmoothGrad (Smilkov et al., 2017).

The second category aims to answer the “ _Why_ ”-type questions, i.e., why the
input is predicted as label _A_ instead of _B_. The methods under this
category can be quite different from the previous ones, since these methods
basically need to consider two labels simultaneously. There are several
different methodologies proposed for this problem. For example, the authors in
(Dhurandhar et al., 2018) design a contrastive perturbation method to derive
related positive and negative features of input regarding to the concerned
label. Besides, a general method based on structural causal models is proposed
in (Miller, 2018) to tackle the problem specifically in classification and
planning scenarios. Also, a generative framework CDeepEx is designed in
(Feghahati et al., 2018) to particularly investigate this problem for images
by utilizing GAN.

The third category lies in the “ _How_ ”-type questions, i.e., how to
particularly modify the input so as to flip the model prediction to the
preferred label. This problem is a natural extension of the “Why”-type, and it
can somewhat to be handled by the second category of methods under some simple
scenarios. However, for problems with high-dimension space, previous
categories of methods typically fail due to the intractable computation for
sample modification. Several particular methods are raised to solve this
issue. For example, authors in (Goyal et al., 2019) propose a straightforward
solution with image region replacement, which is essentially a feature
replacement process for input with the aid of a distractor. In work (Agarwal
et al., 2019), authors novelly use the input itself as the distractor for
feature replacement by utilizing GAN for inpainting. Besides, generative
modeling is another potential way for this problem, and related methods can be
found in (Singla et al., 2020; Joshi et al., 2018; Liu et al., 2019). Our work
belongs to this branch of methodology.

##  6\. Conclusion And Future Work

In this paper, we design a framework to generate counterfactual explanation
for black-box DNN models specifically with raw data instances. By taking
advantage of the generative modeling techniques, we effectively construct an
attribute-informed latent space for particular data, and further utilize this
space for counterfactual generation. To guarantee the validity of the
generated samples, we propose the AIP method to iteratively optimize the
specific attribute-informed latent vectors according to the counterfactual
loss term, from which the counterfactuals can be finally obtained through data
reconstruction. We evaluate the designed framework with AIP on several real-
world datasets, including both texts and images, and demonstrate its
effectiveness, sample quality as well as efficiency. Future extension of this
work may possibly include the investigation under the “close possible worlds”
assumption, where the goal is to find an optimal set of counterfactuals for a
query instead of a single sample. Besides, employing causal models for
counterfactual generation is another promising direction to explore.

## References

  * (1)
  * Agarwal et al. (2019) Chirag Agarwal, Dan Schonfeld, and Anh Nguyen. 2019\.  Removing input features via a generative model to explain their attributions to classifier’s decisions.  _arXiv preprint arXiv:1910.04256_ (2019).
  * Asghar (2016) Nabiha Asghar. 2016\.  Yelp dataset challenge: Review rating prediction.  _arXiv preprint arXiv:1605.05362_ (2016).
  * Chen et al. (2019) Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. 2019\.  This looks like that: deep learning for interpretable image recognition. In _Advances in Neural Information Processing Systems_. 8928–8939.
  * Conneau et al. (2017) Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017.  Very Deep Convolutional Networks for Text Classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. 1107–1116.
  * Dhurandhar et al. (2018) Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018.  Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In _Advances in neural information processing systems_. 592–603.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ (2017).
  * Du et al. (2020) Mengnan Du, Ninghao Liu, and Xia Hu. 2020.  Techniques for Interpretable Machine Learning.  _Commun. ACM_ 63, 1 (2020), 68–77.
  * Feghahati et al. (2018) Amir Feghahati, Christian R Shelton, Michael J Pazzani, and Kevin Tang. 2018.  CDeepEx: Contrastive Deep Explanations.  (2018).
  * Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018\.  Black-box generation of adversarial text sequences to evade deep learning classifiers. In _2018 IEEE Security and Privacy Workshops (SPW)_. IEEE, 50–56.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in neural information processing systems_. 2672–2680.
  * Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015\.  Explaining and harnessing adversarial examples. In _3rd International Conference on Learning Representations, ICLR_.
  * Goyal et al. (2019) Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  Counterfactual Visual Explanations. In _International Conference on Machine Learning_. 2376–2384.
  * Guo et al. (2017) Tianmei Guo, Jiwen Dong, Henjian Li, and Yunxing Gao. 2017\.  Simple convolutional neural network on image classification. In _2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA)(_. IEEE, 721–724.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _proceedings of the 25th international conference on world wide web_. 507–517.
  * He et al. (2019) Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. 2019.  Attgan: Facial attribute editing by only changing what you want.  _IEEE Transactions on Image Processing_ 28, 11 (2019), 5464–5478.
  * Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017.  Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning_. 1587–1596.
  * Joshi et al. (2018) Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. 2018\.  xGEMs: Generating examplars to explain black-box models.  _arXiv preprint arXiv:1806.08867_ (2018).
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016\.  Examples are not enough, learn to criticize! criticism for interpretability. In _Advances in neural information processing systems_. 2280–2288.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification. In _Proceedings of the Conference on EMNLP_. 1746–1751.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ (2013).
  * Li et al. (2018) Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018.  Textbugger: Generating adversarial text against real-world applications.  _arXiv preprint arXiv:1812.05271_ (2018).
  * Liu et al. (2019) S Liu, B Kailkhura, D Loveland, and H Yong. 2019\.  _Generative Counterfactual Introspection forExplainable Deep Learning_.  Technical Report. Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States).
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015\.  Deep learning face attributes in the wild. In _Proceedings of the IEEE international conference on computer vision_. 3730–3738.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in neural information processing systems_. 4765–4774.
  * Miller (2018) Tim Miller. 2018\.  Contrastive explanation: A structural-model approach.  _arXiv preprint arXiv:1811.03163_ (2018).
  * Moore et al. (2019) Jonathan Moore, Nils Hammerla, and Chris Watkins. 2019\.  Explaining Deep Learning Models with Constrained Adversarial Examples. In _Pacific Rim International Conference on Artificial Intelligence_. Springer, 43–56.
  * Murphy (2012) Kevin P Murphy. 2012\.  _Machine learning: a probabilistic perspective_.  MIT press.
  * Pearl et al. (2009) Judea Pearl et al. 2009\.  Causal inference in statistics: An overview.  _Statistics surveys_ 3 (2009), 96–146.
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  _The book of why: the new science of cause and effect_.  Basic Books.
  * Pouyanfar et al. (2018) Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and SS Iyengar. 2018.  A survey on deep learning: Algorithms, techniques, and applications.  _ACM Computing Surveys (CSUR)_ 51, 5 (2018), 1–36.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018\.  Anchors: High-precision model-agnostic explanations. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Rissland (1991) Edwina L Rissland. 1991\.  Example-based reasoning.  _Informal reasoning in education_ (1991), 187–208.
  * Samangouei et al. (2018) Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. 2018.  ExplainGAN: Model Explanation via Decision Boundary Crossing Transformations. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 666–681.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization. In _Proceedings of the IEEE international conference on computer vision_. 618–626.
  * Sen et al. (2019) Ayon Sen, Xiaojin Zhu, Liam Marshall, and Robert Nowak. 2019\.  Should Adversarial Attacks Use Pixel p-Norm?  _arXiv preprint arXiv:1906.02439_ (2019).
  * Singla et al. (2020) Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. 2020.  Explanation by Progressive Exaggeration. In _8th International Conference on Learning Representations, ICLR 2020_.
  * Smilkov et al. (2017) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ (2017).
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 3319–3328.
  * Van den Oord et al. (2016) Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016\.  Conditional image generation with pixelcnn decoders. In _Advances in neural information processing systems_. 4790–4798.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017\.  Counterfactual explanations without opening the black box: Automated decisions and the GDPR.  _Harv. JL & Tech._ 31 (2017), 841.
  * White and Garcez (2019) Adam White and Artur d’Avila Garcez. 2019.  Measurable counterfactual local explanations for any classifier.  _arXiv preprint arXiv:1908.03020_ (2019).


Incompatibility between ’t Hooft’s and Wolfram’s models of quantum mechanics
José Manuel Rodríguez Caballero External affiliate of the Wolfram Physics
Project Director of Caballero Software Inc. Ontario, Canada

###### Abstract

Stephen Wolfram and Gerard ’t Hooft developed classical models of quantum
mechanics. We show that the descriptive complexity grows differently as a
function of time in each model. Therefore, they cannot describe the same
physical system. In addition, we propose an interpretation of the Wolfram
model, which shares some characteristics with ’t Hooft’s model, but which
involves a non-computable function.

##  1 Introduction

> Randomness is the true foundation of mathematics.
>
> Gregory Chaitin

The development of classical models of quantum mechanics is limited by what is
called no-go theorems. These theorems, initially proved by John von Neumann
[VN18], prevent the creation of quantum mechanical models satisfying certain
assumptions of classical physics. Nevertheless, this does not contain some
authors from reinterpreting the experimental proofs of quantum effects and
from developing new models beyond the framework of these theorems. Two
examples are Gerard ’t Hooft’s cellular automata interpretation [tH16], and
Stephen Wolfram’s physics project [Wol20].

Gerard ’t Hooft’s point of departure is to deny the existence of quantum
superpositions. The time is discreet; the smallest unit is an iteration of the
universe as a computer system. In his model, there is a special basis of the
Hilbert space of the universe, called the ontological basis. The vectors of
this base, called ontic states, are intended to represent the states of the
universe in reality. Ontic states are postulated to evolve into ontic states
after an iteration of the universe. This evolution is deterministic, and it
can be simulated in a (classical) computer provided with sufficient memory and
time. Because there are many similarities between ‘t Hooft’s model and the
model studied by Stephen Wolfram in A New Kind of Science [Wol02], this
article can be interpreted as a comparison between the first and second
approaches of Wolfram’s fundamental physics, expressed in ‘t Hooft’s
theoretical framework (we use the‘ t Hooft language of ontic states rather
than Wolfram’s language of hypergraphs, which are the ontic states in his
model).

In quantum mechanics111The Wolfram model is also developed in the direction of
the theories of special and general relativity, but we will omit these
subjects in the present paper., Stephen Wolfram’s starting point can be seen
as a non-deterministic generalization of ‘t Hooft’s model, known as the
multiway system. In this framework, quantum effects are intended to be
explained in a way similar to Hugh Everett’s many-worlds interpretation
[Eve15]. Of course, for the case of a single-world, both Wolfram’s and Hooft’s
models are essentially the same. The word “incompatibility” in the title is
used when the Wolfram model assumes more than one world in the sense described
below.

This article aims to show that, assuming more than one world and an aperiodic
evolution, the descriptive complexity of the universe perceived by a typical
version of an observer in Wolfram’s model increases as a linear function of
time (generically). In contrast, the same magnitude in ‘t Hooft’s model grows
as a logarithmic function of time (generically). By generically, we meant that
all the exceptions are negligible in a sense that will be specified in the
elaboration of the results. Therefore, the two models cannot describe the same
physical system. Nevertheless, we will provide a version of the Wolfram model,
called its client-server interpretation, which shares some characteristics of
the ‘t Hooft model, for example, there is only one world, but the experience
of the observer is similar to that of the typical version of the observer in
Wolfram’s model.

##  2 Typical observer’s experience

A model of quantum mechanics is _Wolfram-like_ if it postulates that an ontic
state always evolves into a multi-set of ontic states. A Wolfram-like model is
_$k$ -regular_ if this multi-set always contains $k$ different elements222The
repetition of elements in the multiset may be related to the probability
amplitude of a superposition of ontic states in the model.. A _‘t Hooft-like_
model is a $1$-regular Wolfram-like model.

An _observer’s experience_ in a Wolfram-like model is a sequence of ontic
states $|x_{1}\rangle,|x_{2}\rangle,|x_{3}\rangle,...$ such that, for every
$n$, the ontic states $|x_{n}\rangle$ evolves to a multi-set containing
$|x_{n+1}\rangle$. An observer’s experience in a $2^{r}$-regular333For
simplicity, we will focus on the case when $k$, in the definition of
$k$-regular, is a power of $2$. Wolfram model is _typical_ if, for all $n$
large enough, the descriptive complexity of $x_{n}$ (the label of the ontic
state) is equal to a constant plus $rn$, and the set of exception is
negligible, i.e., the number of elements of the set of integers smaller than
or equal to $n$ not satisfying this property, divided by $n$ converges to zero
as $n$ goes to infinity. A $2^{r}$-regular Wolfram-like model is _non-
degenerate_ if it contains at least one typical observer’s experience.

Convention. From now on, we will assume that all $2^{r}$-regular Wolfram-like
models, with $r\geq 1$, mentioned in this paper are non-degenerate444An
example of a $2$-regular Wolfram-like model that is not non-degenerate is
$1\mapsto{2,3}$, $2\mapsto{1,3}$, $3\mapsto{1,2}$. The goal of this convention
is to avoid repeating the word “non-degenerate” too many times, making the
text easier to read.

Notice that in a ‘t Hooft-like model, there is a typical observer’s experience
if and only if the system is periodic. For the aperiodic case, for all $n$
large enough, the descriptive complexity of $x_{n}$ is equal to a constant
plus the number of binary digits of $n$.

##  3 Logarithmic time

In the original approach of S. Wolfram and ‘t Hooft, the time is proportional
to the number of system iterations. Under this assumption, an observer Alice,
who is integrated into a $2^{r}$ -regular Wolfram model, can experimentally
determine the value of $r$, provided enough data on the descriptive complexity
of her universe as a function of time. For this, Alice must assume that her
experience is typical as a consequence of the Copernican principle.

Turbulence expert W. K. George explored the cosmological consequences of
reformulating the laws of physics using the logarithm of time instead of time
as usual. Inspired by this idea, we will consider the possibility that an
observer of a model of type ‘t Hooft can perceive the time as proportional to
the logarithm of the number of system iterations. An example of such an
observer is the red-green color-blind daemon in the following thought
experiment.

In a solitary cage, isolated from the rest of the universe, a daemon is
imprisoned. The daemon has a stopwatch, expressing time as binary digits,
which are square boxes. The red and green boxes represent $0$ and $1$,
respectively. The only problem is that the daemon is red-green colour-blind.
Therefore, the only information the daemon can get from the stopwatch is the
number of digits in the hour. So, as perceived by the stopwatch daemon, time
increases logarithmically concerning the number of stopwatch iterations.

##  4 Client-server transformation

The relationship between client and server described below is inspired by the
relationship between slow and fast variables, respectively, as used by ‘t
Hooft.

We propose a method for transforming a classical many-worlds model of quantum
mechanics (under the Wolfram model) into a single world model, which we call
its client-server interpretation. This transformation preserves the experience
of a typical observer. The price to pay is the introduction of a non-
computable function.

Consider a $2^{r}$-regular Wolfram-like model for $r\geq 1$. Imagine a server
generating the ontic states of the system, one by one, as levels of a
$2^{r}$-ary tree. Each time the server generates an ontic state, it sends it
to the client. The client accepts the $n$-th state sent by the server if and
only if $n$ is the smallest number of its length in base $2^{r}$ that cannot
be compressed (we assume a fixed Turing machine). Finally, the client uses the
sequence of ontic states accepted from the server as the history of the
universe. Notice that the time in the universe generated by the client is
proportional logarithm of the number of iteration of the server.

Using the Berry paradox, it is easy to prove that the property “$n$ is the
smallest number of its length in base $2^{r}$ that cannot be compressed” is
non-decidable. We will call the sequence of numbers satisfying this property,
the _Chaitin geometric progression_ in base $2^{r}$ (we will omit the base
when it is clear from the context).

We have seen that the experience of a typical version of an observer in a
$2^{r}$-regular Wolfram-like model, with $r\geq 1$, is equivalent to the
experience of the single version of an observer in its client-server
interpretation. Nevertheless, the simplicity of one description can be
superior to the other description. For example, consider the case of the
$2$-regular Wolfram model where the ontological states are labeled by positive
integers and the evolution rules are:

$U_{0}|x\rangle=|2x\rangle,\quad U_{1}|x\rangle=|2x+1\rangle.$

It is easy to show that the evolution rule (in the server) of the
corresponding client-server interpretation is just555The letters c and s stand
for client-server.

$U_{\textsf{cs}}|x\rangle=|x+1\rangle.$

## Conclusions

Unlike the Copenhagen doctrine, where true randomness is imposed in the
universe as a postulate, the client-server interpretation gets the same effect
from what Gregory Chaitin calls the “randomness of mathematics”, i.e.,
randomness intrinsic to the system, not assumed as an independent postulate.
The requirement of an oracle for the Chaitin geometric progression is avoided
in the $2^{r}$-regular Wolfram-like model, with $r\geq 1$, in a way similar as
in the many-worlds interpretation of quantum mechanics, by assuming multiples
versions of the observer and applying the Copernican principle: the typical
version experiences randomness, because, in this case, most configurations are
incompressible in Chaitin’s sense.

As a reaction to the theory of consciousness developed by the school of strong
artificial intelligence, Roger Penrose [Pen89] proposed the existence of non-
computational processes in quantum mechanics. This claim seems to contradict
the so-called physical Church-Turing thesis, due to Stephen Wolfram [Wol85]
and David Deutsch [Deu85]. Nevertheless, this contradiction is apparent since
the experience of a typical version of an observer in a multiway system is
equivalent to the experience of the unique version of an observer in its
client-server interpretation (involving non-computable processes). Hence, even
if some non-computational process is detected in quantum mechanics as the
experience of an observer, it may be the result of the Copernican principle
applied to the multiway system.

## Acknowledgments

The author would like to thank Stephen Wolfram, William K. George and Gregory
Chaitin for the interesting exchange of ideas. Also, the author is very
grateful to all the members of the Wolfram Physics Project for having
contributed every day to the development of this new approach to fundamental
physics. Special thanks to Christian Pratt, who collaborated on the initial
version of this project.

## References

  * [Cha75] Gregory J Chaitin.  A theory of program size formally identical to information theory.  Journal of the ACM (JACM), 22(3):329–340, 1975.
  * [Deu85] David Deutsch.  Quantum theory, the church–turing principle and the universal quantum computer.  Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 400(1818):97–117, 1985.
  * [Eve15] Hugh Everett.  The theory of the universal wave function.  In The many worlds interpretation of quantum mechanics, pages 1–140. Princeton University Press, 2015.
  * [Pen89] Roger Penrose.  The emperor’s new mind-concerning computers, minds, and the laws of physics.  Oxford University Press (1989).
  * [tH16] Gerard t Hooft.  The cellular automaton interpretation of quantum mechanics.  Springer Nature, 2016.
  * [VN18] John Von Neumann.  Mathematical foundations of quantum mechanics.  Princeton university press, 2018.
  * [Wol85] Stephen Wolfram.  Undecidability and intractability in theoretical physics.  Physical Review Letters, 54(8):735, 1985.
  * [Wol02] Stephen Wolfram.  A new kind of science, volume 5.  Wolfram media Champaign, IL, 2002.
  * [Wol20] Stephen Wolfram.  A class of models with the potential to represent fundamental physics.  Complex Systems, 29(2), 2020.