# Unsupervised Causal Binary Concepts Discovery with VAE for Black-box Model
Explanation

Thien Q. Tran
University of Tsukuba, Riken AIP
<EMAIL_ADDRESS>
&Kazuto Fukuchi
University of Tsukuba, Riken AIP
<EMAIL_ADDRESS>
&Youhei Akimoto
University of Tsukuba, Riken AIP
<EMAIL_ADDRESS>
&Jun Sakuma
University of Tsukuba, Riken AIP
<EMAIL_ADDRESS>

###### Abstract

We aim to explain a black-box classifier with the form: ‘data X is classified
as class Y because X has A, B and does not have C’ in which A, B, and C are
high-level concepts. The challenge is that we have to discover in an
unsupervised manner a set of concepts, i.e., A, B and C, that is useful for
the explaining the classifier. We first introduce a structural generative
model that is suitable to express and discover such concepts. We then propose
a learning process that simultaneously learns the data distribution and
encourages certain concepts to have a large causal influence on the classifier
output. Our method also allows easy integration of user’s prior knowledge to
induce high interpretability of concepts. Using multiple datasets, we
demonstrate that our method can discover useful binary concepts for
explanation.

##  1 Introduction

(a) Illustration of proposed concepts

(b) Causal DAG

(c) VAE model

Figure 1: (a) Binary concept middle stroke and some global variants. Border
color indicates the classifier output. (b, c) The proposed VAE model and the
causal DAG

Deep neural network has been recognized as the state-of-the-art model for
various tasks. As they are being applied in more practical applications, there
is an arising consensus that these models need to be explainable, especially
in high-stake domains. Various methods are proposed to solve this problem,
including building interpretable model and post-hoc methods that explain
trained black-box models. We focus on the post-hoc approach and propose a
novel causal concept-based explanation framework.

We are interested in an explanation that uses the symbolic expression: ‘data X
is classified as class Y because X has A, B and does not have C’ where A, B,
and C are high-level concepts. From the linguistic perspective, our
explanation communicates using nouns and their part-whole relation, i.e., the
semantic relation between a part and the whole object. In many classification
tasks, especially image classification, the predictions relied on binary
components; for example, we can distinguish a panda from a bear by its white
patched eyes or a zebra from a horse by its stripe. This is also a common way
humans use to classify categories and organize knowledge Gardenfors (2014).
Thus, an explanation in this form should excel in providing human-friendly and
organized insights into the classifier, especially for tasks that involve
higher-level concepts such as checking the alignment of the black-box model
with experts. From now on, we refer to such a concept as binary concept.

Our method employs three different notions in the explanation: causal binary
switches, concept-specific variants and global variants. We illustrate these
notions in Figure 1(a). First, causal binary switches and concept-specific
variants, that come in pair, represent different binary concepts. In
particular, causal binary switches control the presence of each binary concept
in a sample. Alternating this switch, i.e., removing or adding a binary
concept to a sample, affects the prediction of that sample (e.g., removing the
middle stroke turns E to C). In contrast, concept specific variants, whose
each is tied to a specific binary concept, express different variants within a
binary concept that do not affect the prediction (e.g., changing the length of
the middle stroke does not affect the prediction). Finally, global variants,
which are not tied to specific binary concepts, represent other variants that
do not affect the prediction (e.g., skewness).

(a) Saliency methods (b) VSC model

(c) O’Shaughnessy et al. (d) Proposed (causal factors)

(e) Proposed (non-causal) Figure 2: Explanation methods for a letter
classifier. (a) Saliency-based methods. (b) Disabling the most active latents
of class E in VSC model. (c) Controlling the causal and non-causal factors in
O’Shaughnessy et al. (d, e) Proposed method: (d) Encoded binary relation of
discovered concepts and their intervention results; (e) variants within each
concept and other variants of the whole letter.

Our goal is to discover a set of binary concepts that can explain the
classifier using their binary switches in an unsupervised manner. Similar to
some existing works, to construct conceptual explanations, we learn a
generative model that maps each input into a low-dimensional representation in
which each factor encodes an aspect of the data. There are three main
challenges in achieving our goal. (1) It requires an adequate generative model
to express the binary concepts, including the binary switches and the variants
within each concept. (2) The discovered binary concepts must have a large
causal influence on the classifier output. That is, we avoid finding
confounding concepts, which correlate with but do not cause the prediction.
For example, the sky concept appears frequently in plane’s images but may not
cause the prediction of plane. (3) The explanation must be interpretable and
provide useful insights. For example, a concept that entirely replaces a
letter E with a letter A has a large causal effect. However, such a concept
does not provide valuable knowledge due to lack of interpretability.

In Figure 2(d) and 2(e), we demonstrate an explanation discovered by the
proposed method for an classifier for six letters: A,B,C,D,E and F. Our method
successfully discovered the concepts of bottom stroke, middle stroke and right
stroke which effectively explains the classifier. In Figure 2(d), we show the
encoded binary switches and their interventions result. From the top figure,
we can explain that: this letter is classified as E because it has a bottom
stroke (otherwise it is F), a middle stroke (otherwise it is C), and it does
not have a right stroke (otherwise it is B). We were also able to distinguish
the variant within each concept in (Figure 2(e) top) with the global variant
(Figure 2(e) bottom). A full result with explanation for other letters is
shown in Section 5.

To the best of our knowledge, no existing method can discover binary concepts
that fulfill all of these requirements. Saliency methods such as Guided
Backprop Springenberg et al. (2014), Integrated Gradient Sundararajan et al.
(2017) or GradCam Selvaraju et al. (2017) only show feature importance but do
not explain why (Figure 2(a)). Some generative models which use binary-
continuous mixed latents for sparse coding, such as VSC Tonolini et al.
(2020), IBP-VAE Gyawali et al. (2019), PatchVAE Gupta et al. (2020), can
support binary concepts. However, they do not necessarily discover binary
concepts that are useful for explanation, in both causality and
interpretability (Figure 2(b)). Recently, O’Shaughnessy et al. (2020) proposed
a learning framework that encourages the causal effect of certain latent
factors on the classifier output to learn a latent representation that has
causality on the prediction. However, their model can not disentangle binary
concepts and can be hard to interpret, especially for multiple-class tasks.
For example, a single concept changes the letter E to multiple other letters
(Figure 2(c)), which would not give any interpretation on how this latent
variable affects prediction.

Our work has the following contributions: (1) We introduce the problem of
discovering binary concepts for the explanation. Then, we propose a structural
generative model for constructing binary concept explanation, which can
capture the binary switches, concept-specific variants, and global variants.
(2) We propose a learning process to simultaneously learn the data
distribution while encouraging the causal influence of the binary switches.
Although typically VAE models encourage the independence of factors for
meaningful disentanglement, such an assumption is inadequate for discovering
useful causal concepts which are often mutually correlated. Our learning
process, which considers the dependence between binary concepts, can discover
concepts with more significant causality. (3) To avoid the concepts that have
causality but no interpretability, the proposed method allows an easy way to
implement user’s preference and prior knowledge as a regularizer to induce
high interpretability of concepts. (4) Finally, we demonstrate that our method
succeeds in discovering interpretable binary concepts with causality that are
useful for explanation with multiple datasets.

##  2 Related Work

Our method can be categorized as a concept-based method that explains using
high-level aspects of data. The definition of concept are various, e.g., a
direction in the activation space Kim et al. (2018); Ghorbani et al. (2019), a
prototypical activation vector Yeh et al. (2020) or a latent factor of a
generative model O’Shaughnessy et al. (2020); Goyal et al. (2020). We remark
that this notion of concept should depend on the data and the explanation
goal. Some works defined the concepts beforehand using additional data. When
this side-information is not given, one needs to discover useful concepts for
the explanation, e.g., Ghorbani et al. (2019) used segmentation and
clustering, Yeh et al. (2020) retrained the classifier with a prototypical
concept layer, O’Shaughnessy et al. (2020) learned the generative model with a
causal objective.

A generative model such as VAE can provide a concept-based explanation as it
learns a latent presentation $\operatorname{\mathbf{z}}$ that captures
different aspects of the data. However, Locatello et al. (2019) shows that
disentangled representations in a fully unsupervised manner are fundamentally
impossible without inductive bias. A popular approach is to augment the VAE
loss with a regularizer Higgins et al. (2016); Burgess et al. (2018). Another
approach is to incorporate structure into the representationChoi et al.
(2020); Ross and Doshi-Velez (2021); Tonolini et al. (2020); Gupta et al.
(2020). Although these methods can encourage disentangled and sparse
representation, the learned representations are not necessarily interpretable
and have causality on the classifier output.

We pursue an explanation that has causality. A causal explanation is helpful
as it can avoid attributions and concepts that only correlate with but do not
causes the prediction. Previous works have attempted to focus on causality in
various ways. For example, Schwab and Karlen (2019) employed Granger causality
to quantify the causal effect of input features, Parafita and Vitrià (2019)
evaluated the causality of latent attributions with a prior known causal
structure, Narendra et al. (2018) evaluated the causal effect of network
layers, and Kim and Bastani (2019) learned an interpretable model with a
causal guarantee. To the best of our knowledge, no existing works can explain
using binary concepts that fulfill the three requirements we discussed.

##  3 Preliminaries

###  3.1 Variational Autoencoder

Our explanation is build upon the VAE framework proposed by Kingma and Welling
(2014). VAE model assumes a generative process of data in which a latent
$\operatorname{\mathbf{z}}$ is first sampled from a prior distribution
$p(\operatorname{\mathbf{z}})$, then the data is generated via a conditional
distribution $p(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{z}})$.
Typically, due to the intractability, a variational approximation
$q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})$ of the intractable
posterior is introduced and the model is then learned using the evidence lower
bound (ELBO) as

$\displaystyle\begin{split}\mathcal{L}_{\text{VAE}}(\operatorname{\mathbf{x}})=&-\operatorname{\mathbb{E}}_{\operatorname{\mathbf{z}}\sim
q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})}[\log
p(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{z}})]+\operatorname{\mathbb{KL}}[q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})\operatorname{\,\|\,}p(\operatorname{\mathbf{z}})].\end{split}$
(1)

Here, $q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})$ is the
encoder that maps the data to the latent space and
$p(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{z}})$ is the decoder
that maps the latents to the data space. Commonly,
$q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})$ and
$p(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{z}})$ are parameterized
as neutral networks
$Q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})$ and
$G(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{z}})$, respectively. The
common choice for $q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})$
is a factorized Gaussian encoder
$q(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})=\prod_{p=1}^{P}\mathcal{N}(\mu_{i},\sigma_{i}^{2})$
where
$(\mu_{1},\dots,\mu_{P},\sigma_{1},\dots,\sigma_{P},)=Q(\operatorname{\mathbf{x}})$.
The common choice for the $p(\operatorname{\mathbf{z}})$ is a multi-variate
normal distribution $\mathcal{N}(0,\mathcal{I})$ with zero mean and identity
covariant. Then, the first term can be trained using L2 reconstruction loss,
while the KL-divergence terms are trained using the reparameterization trick.

###  3.2 Information Flow

Next, we introduce the measure we use to quantify the causal influence of the
learned representation on the classifier output. We adopt Information Flow,
which defines the causal strength using Pearl’s do calculus Pearl (2009).
Given a causal directional acyclic graph $G$, Information Flow quantify the
statistical influence using the conditional mutual information on the
interventional distribution:

######  Definition 1 (Information flow from $U$ to $V$ in a directed acyclic
graph $G$Ay and Polani (2008)).

Let $U$ and $V$ be disjoint subsets of nodes. The information flow
$I(U\rightarrow V)$ from $U$ to $V$ is defined by

$\displaystyle I(U\rightarrow
V)=\int_{U}p(u)\int_{V}p(v|\text{do}(u))\log\frac{p(v|\text{do}(u))}{\int_{u^{\prime}}p(u^{\prime})p(v|\text{do}(u^{\prime}))du^{\prime}}dVdU,$
(2)

where $\text{do}(u)$ represents an intervention that fixes u to a value
regardless of the values of its parents.

O’Shaughnessy et al. (2020) argued that compared to other metrics such as
average causal effect (ACE) Holland (1988), analysis of variance (ANOVA)
Lewontin (1974), information flow is more suitable to capture complex and
nonlinear causal dependence between variables.

##  4 Proposed method

We aim to discover a set of binary concepts
$\mathcal{M}=\\{m_{0},m_{1},\dots,m_{M}\\}$ with causality and
interpretability that can explain the black-box classifier
$f:\mathcal{X}\rightarrow\mathcal{Y}$. Inspired by O’Shaughnessy et al.
(2020), we employs a generative model to learn the data distribution while
encouraging the causal influence of certain latent factors. In particular, we
assume a causal graph in Figure 1(b), in which each sample
$\operatorname{\mathbf{x}}$ is generated from a set of latent variables,
including $M$ pairs of a binary concept and a concept-specific variant
$\\{\gamma_{i},\operatorname{\bm{\alpha}}_{i}\\}_{i=1}^{M}$, and a global
variants $\operatorname{\bm{\beta}}$. As we want to explain the classifier
output (i.e., node $y$ in Figure 1(b)) using the binary switches
$\\{\gamma_{i}\\}$, we expect that $\\{\gamma_{i}\\}$ has a large causal
influence on $y$.

Our proposed learning objective consists of three components, which
corresponds to our three requirements: a VAE objective
$\mathcal{L}_{\text{VAE}}$ for learning the data distribution
$p(\operatorname{\mathbf{x}})$, a causal effect objective
$\mathcal{L}_{\text{CE}}(X)$ for encouraging the causal influence of
$\\{\gamma_{i}\\}$ on classifier output $y$, and an user-implementable
regularizer $\mathcal{L}_{\text{R}}(\operatorname{\mathbf{x}})$ for improving
the interpretability and consistency of discovered concepts:

$\displaystyle\mathcal{L}(X)=\frac{1}{|X|}\sum_{\operatorname{\mathbf{x}}\in
X}\left[\mathcal{L}_{\text{VAE}}(\operatorname{\mathbf{x}})+\lambda_{\text{R}}\mathcal{L}_{\text{R}}(\operatorname{\mathbf{x}})\right]+\lambda_{\text{CE}}\mathcal{L}_{\text{CE}}(X).$
(3)

###  4.1 VAE model with binary concepts

To represent the binary concepts, we employ a structure in which each binary
concept $m_{i}$ is presented by a latent variable $\bm{\psi}_{i}$, which is
further controlled by two factors: a binary concept switch latent variable
$\gamma_{i}$ (concept switch for short) and a continuous latent variable
representing concept-specific variants $\operatorname{\bm{\alpha}}_{i}$
(concept-specific variant for short) as
$\bm{\psi}_{i}=\gamma_{i}\cdot\operatorname{\bm{\alpha}}_{i}$ where
$\gamma_{i}=1$ if concept $m_{i}$ is on and $gamma_{i}=0$ otherwise. Here, the
concept switch $\gamma_{i}$ controls if the concept $m_{i}$ is activated in a
sample, e.g., if the bottom stroke is appeared in a image (Figure 2(d)). On
the other hand, the concept-specific variant $\operatorname{\bm{\alpha}}_{i}$
controls the variant within the concept $m_{i}$, e.g., the length of the
bottom stroke (Figure 2(e), top). In addition to the concept-specific variants
$\\{\operatorname{\bm{\alpha}}_{i}\\}$ whose effect is limited to a specific
binary concept, we also allow a global variant latent
$\operatorname{\bm{\beta}}$ to capture other variants that do not necessarily
have causality, e.g., skewness (Figure 2(e), bottom). Here, disentangling
concept-specific and global variants is important for assisting user in
understanding discovered binary concepts.

The way we represent binary concepts is closely related to the spike-and-slab
distribution, which is used in Bayesian variable selection George and
McCulloch (1997) and sparse coding Tonolini et al. (2020). Unlike these
models, whose number of discrete-continuous factors is often large, our model
uses only a small number of binary concepts with a multi-dimensional global
variants $\beta$. Our intuition is that in many cases, the classification can
be made by combining a small number of binary concepts.

Input encoding. For the discrete components, we use a network
$Q^{d}(\operatorname{\mathbf{x}})$ to parameterize
$q(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})$ as
$q(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})=\prod_{i=1}^{M}q(\gamma_{i}\mid\operatorname{\mathbf{x}})=\prod_{i=1}^{M}\text{Bern}(\gamma_{i};\pi_{i})$
where $(\pi_{1},\dots,\pi_{M})=Q^{d}(\operatorname{\mathbf{x}})$. For the
continuous components, letting
$A=(\operatorname{\bm{\alpha}}_{1},\operatorname{\bm{\alpha}}_{2},\dots,\operatorname{\bm{\alpha}}_{M})$,
we use a network $Q^{c}(\operatorname{\mathbf{x}})$ to parameterize
$q(A,\operatorname{\bm{\beta}}\mid\operatorname{\mathbf{x}})$ as
$q(A,\operatorname{\bm{\beta}}\mid\operatorname{\mathbf{x}})=\left[\prod_{i=1}^{M}q\left(\operatorname{\bm{\alpha}}_{i}\mid\operatorname{\mathbf{x}}\right)\right]q(\operatorname{\bm{\beta}}\mid\operatorname{\mathbf{x}})$.
Here,
$q(\operatorname{\bm{\alpha}}_{i}\mid\operatorname{\mathbf{x}})=\mathcal{N}^{\text{fold}}_{\delta}(\operatorname{\bm{\alpha}}_{i};\mu_{i},\text{diag}(\sigma_{i}))$,
$q(\operatorname{\bm{\beta}}\mid\operatorname{\mathbf{x}})=\mathcal{N}^{\text{fold}}_{\delta}(\operatorname{\bm{\beta}};\mu_{\operatorname{\bm{\beta}}},\text{diag}(\sigma_{\operatorname{\bm{\beta}}}))$
and
$(\mu_{1},\dots,\mu_{M},\mu_{\operatorname{\bm{\beta}}},\sigma_{1},\dots,\sigma_{M},\sigma_{\operatorname{\bm{\beta}}})=Q^{c}(\operatorname{\mathbf{x}})$.
Here, we employ the $\delta\text{-Shifted}$ Folded Normal Distribution
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$ for continuous latents,
which is the distribution of $|x|+\delta$ with a constant hyper-parameter
$\delta>0$ where $x\sim\mathcal{N}(\mu,\sigma^{2})$. In all of our
experiments, we adopted $\delta=0.5$. We choose not the standard Normal
Distribution but the $\delta\text{-Shifted}$ Folded Normal Distribution
because it is more appropriate for the causal effect we want to achieve. The
implementation of $\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$ can
simply be done by adding the absolute and shift operation to the conventional
implementation of $\mathcal{N}(\mu,\sigma^{2})$. We discuss in detail this
design choice in Appendix A.3.

Output decoding. Next, given
$q(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})$ and
$q(A,\operatorname{\bm{\beta}}\mid\operatorname{\mathbf{x}})$, we first sample
the concept switches $\\{\hat{d}_{i}\\}$, the concept variants
$\\{\hat{\operatorname{\bm{\alpha}}}_{i}\\}$ and the global variants
$\operatorname{\bm{\beta}}$ from their posterior, respectively. Using these
sampled latents, we construct an aggregated representation
$\hat{\operatorname{\mathbf{z}}}=(\bm{\psi}_{1},\dots,\bm{\psi}_{M},\hat{\operatorname{\bm{\beta}}})$
using the binary concept mechanism in which $\bm{\psi}_{i}$ is the
corresponding part for concept $m_{i}$, i.e.,
$\bm{\psi}_{i}=\gamma_{i}\times\operatorname{\bm{\alpha}}_{i}$. If concept
$m_{i}$ is on, we let $\hat{d}_{i}=1$ so that $\bm{\psi}_{i}$ can reflect the
concept-specific variant $\hat{\operatorname{\bm{\alpha}}}_{i}$. Otherwise,
when the concept $m_{i}$ is off, we assign $\hat{d}_{i}=0$. We refer to
$\hat{\operatorname{\mathbf{z}}}$ as the conceptual latent code. Finally, a
decoder network takes $\hat{\operatorname{\mathbf{z}}}$ and generate the
reconstruction $\hat{x}$ as $\hat{x}\sim
G(\operatorname{\mathbf{x}}\mid\hat{\operatorname{\mathbf{z}}})\text{ where
}\hat{\operatorname{\mathbf{z}}}=(\bm{\psi}_{1},\dots,\bm{\psi}_{M},\hat{\operatorname{\bm{\beta}}}).$

Learning process. We use the maximization of evidence lower bound (ELBO) to
jointly train the encoder and decoder. We assume the prior distribution for
continuous latents to be $\delta\text{-shifted}$ Folded Normal distribution
$\mathcal{N}^{\text{fold}}_{\delta}(0,\mathcal{I})$ with zero-mean and
identity covariance. Moreover, we assume the prior distribution for binary
latents to be a Bernoulli distribution $\text{Bern}(\pi_{\text{prior}})$ with
prior $\pi_{\text{prior}}$. The ELBO for our learning process can be written
as:

$\displaystyle\begin{split}\mathcal{L}_{\text{VAE}}(\operatorname{\mathbf{x}})&=-\operatorname{\mathbb{E}}_{\hat{\operatorname{\mathbf{z}}}\sim
Q^{\\{c,d\\}}(\operatorname{\mathbf{z}}\mid\operatorname{\mathbf{x}})}\left[\log
G\left(\operatorname{\mathbf{x}}\mid\hat{\operatorname{\mathbf{z}}}\right)\right]+\lambda_{2}\left[\frac{1}{M}\sum_{i=1}^{M}\operatorname{\mathbb{KL}}\left(q\left(\gamma_{i}\mid\mathbf{x}\right)\operatorname{\,\|\,}\operatorname{\operatorname{Bern}}\left(\pi_{i}\right)\right)\right]\\\
&+\lambda_{1}\left[\operatorname{\mathbb{KL}}\left(q\left(\operatorname{\bm{\beta}}\mid\mathbf{x}\right)\operatorname{\,\|\,}\mathcal{N}^{\text{fold}}_{\delta}\left(0,\mathcal{I}\;\right)\right)+\frac{1}{M}\sum_{i=1}^{M}\operatorname{\mathbb{KL}}\left(q\left(\operatorname{\bm{\alpha}}_{i}\mid\mathbf{x}\right)\operatorname{\,\|\,}\mathcal{N}^{\text{fold}}_{\delta}\left(0,\mathcal{I}\;\right)\right)\right].\end{split}$
(4)

For the Bernoulli distribution, we use its continuous approximation, i.e., the
relaxed-Bernoulli Maddison et al. (2017) in the training process.

###  4.2 Encouraging causal effect of binary switches

We expect the binary switches $\operatorname{\bm{\gamma}}$ to have a large
causal influence so that they can effectively explain the classifier. To
measure the causal effect of $\operatorname{\bm{\gamma}}$ on the classifier
output $Y$, we employ the causal DAG in Figure 1(b) and adopt information flow
(Definition 1) as the causal measurement. Our DAG employs an assumption that
is fundamentally different from those of standard VAE models. Specifically,
the standard VAE model and also O’Shaughnessy et al. (2020) assumes the
independence of latent factors, which is believed to encourage meaningful
disentanglement via a factorized prior distribution. We claim that because
useful concepts for explanation often causally depend on the class information
and thus are not independent of each other, such an assumption might be
inadequate for discovering valuable causal concepts. For example, in the
letter E, the middle and the bottom strokes are causally related to the
recognition of the letter E, and corresponding binary concepts are mutually
correlated. Thus, employing the VAE’s factorized prior distribution in
estimating information flow might lead to a large estimation error and prevent
discovering valuable causal concepts.

Instead, we employ a prior distribution $p^{*}(\operatorname{\bm{\gamma}})$
that allows the correlation between causal binary concepts. Our method
iteratively learns the VAE model and use the current VAE model to estimates
the prior distribution $p^{*}(\operatorname{\bm{\gamma}})$ which most likely
generates the user’s dataset. This empirical estimation of
$p^{*}(\operatorname{\bm{\gamma}})$ is then used to evaluate the causal
objective in Eq. (3). Assuming $X$ is a set of i.i.d samples from data
distribution $p(\operatorname{\mathbf{x}})$, we estimate
$p^{*}(\operatorname{\bm{\gamma}})$ as

$\displaystyle
p^{*}(\operatorname{\bm{\gamma}})\approx\int_{\operatorname{\mathbf{x}}}p^{*}(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})p(\operatorname{\mathbf{x}})d\operatorname{\mathbf{x}}\approx\frac{1}{|X|}\sum_{\operatorname{\mathbf{x}}\in
X}p(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})\approx\frac{1}{|X|}\sum_{\operatorname{\mathbf{x}}\in
X}\prod_{i=1}^{M}q(\gamma_{i}\mid\operatorname{\mathbf{x}})$ (5)

In the last line, $p(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})$
is replaced with the variational posterior
$q(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})$ of VAE model.
Here, the factorized posterior
$q(\operatorname{\bm{\gamma}}\mid\operatorname{\mathbf{x}})$ only assumes the
independence between latents conditioned on a sample but does not imply the
independence of binary switches in $p^{*}(\operatorname{\bm{\gamma}})$. We
note that we do not aim to learn the dependence between concepts but only
expect that $p^{*}(\operatorname{\bm{\gamma}})$ properly reflects the
dependence between binary concepts that appears in the dataset $X$ for a
better evaluation of causal effect. We experimentally show in Subsection 5.4
that using the estimation of $p^{*}(\operatorname{\bm{\gamma}})$ results in a
better estimation for the causal effect on dataset $X$ and more valuable
concepts for the explanation. We showed that in the proposed DAG, information
flow $I(\operatorname{\bm{\gamma}}\rightarrow Y)$ coincides with mutual
information $I(\operatorname{\bm{\gamma}};Y)$.

######  Proposition 1 (Coincident of Information Flow and Mutual Information
in proposed DAG).

The information flow from $\operatorname{\bm{\gamma}}$ to Y in the DAG of
Figure 1(b) coincides with the mutual information between
$\operatorname{\bm{\gamma}}$ and $Y$. That is,
$I(\operatorname{\bm{\gamma}}\rightarrow
Y)=I(\operatorname{\bm{\gamma}};Y)=\operatorname{\mathbb{E}}_{\operatorname{\bm{\gamma}},Y}\left[\frac{p^{*}(\operatorname{\bm{\gamma}})p(Y\mid\operatorname{\bm{\gamma}})}{p^{*}(\operatorname{\bm{\gamma}})p(Y)}\right]$.

We prove Proposition 1 in Appendix A.2. The detailed algorithm for estimating
$I(\operatorname{\bm{\gamma}};Y)$ is described in Appendix A.1. As we want to
maximize $I(\operatorname{\bm{\gamma}};Y)$, we rewrite it as a loss term
$\mathcal{L}_{\text{CE}}=-I(\operatorname{\bm{\gamma}};Y)$ and optimize it
together with the learning of VAE model.

###  4.3 Integrating user preference for concepts

Finally, we discuss the integration of user’s preferences or prior knowledge
for inducing high interpretability of concepts. A problem in discovering
meaningful latent factors using deep generative models is that the learned
factors can be hard to interpret. Although causality is strongly related and
can contribute to interpretability, due to the high expressiveness of the deep
model, a large causal effect does not always guarantee an interpretable
concept. For example, a concept that entirely replaces a letter E with a
letter D, has a large causal effect on the prediction. However, such a concept
does not provide valuable knowledge and is hard to interpret. To avoid such
concepts, we allow the user to implement their preference or prior knowledge
as an interpretability regularizer to constrain the generative model’s
expressive power. The proposed method then seeks for binary concepts with
large causality under the constrained search space.

The integration can easily be done via a scoring function
$r(\operatorname{\mathbf{x}}_{\gamma_{i}=0},\operatorname{\mathbf{x}}_{\gamma_{i}=1})$
which evaluates the usefulness of concept $m_{i}$. Here,
$\operatorname{\mathbf{x}}_{\gamma_{i}=0}$ and
$\operatorname{\mathbf{x}}_{\gamma_{i}=1}$ are obtained from the generative
model by performing the do-operation $do(\gamma_{i}=0)$ and $do(\gamma_{i}=1)$
on input $\operatorname{\mathbf{x}}$, respectively. In this study, we
introduce two regularizers which are based on the following intuitions. First,
an interpretable concept should only affect a small amount of input features
(Eq. (9)). This desiderata is general and can be applied to many tasks. The
second one is more task-specific in which we focus on the gray-scale image
classification task. An intervention of a concept should only add or substract
the pixel value, but not both at the same time (Eq. (10)). Furthermore, we
desire that $\gamma_{i}=1$ indicates the presence of pixels and $\gamma_{i}=0$
indicates the absence of pixels. We show the detailed formulation for these
regularizers in Appendix A.4. Using these interpretability regularizer, we
observed a significant improvement in interpretability (Subsection 5.4) and
consistency (Appendix A.4) of concepts.

##  5 Experiment

###  5.1 Experiment setting

We demonstrate our method using three datasets: EMNISTCohen et al. (2017),
MNISTLeCun et al. (2010) and Fashion-MNISTXiao et al. (2017). For each
dataset, we select several classes and train a classifier on the selected
classes. In particular, we select the letters ‘A, B, C, D, E, F’ for EMNIST,
digits ‘1, 4, 7, 9’ for MNIST, and ‘t-shirt/top, dress, coat’ for the Fashion-
MNIST dataset. We note that our setting is more challenging than the common
test setting in existing works (e.g., classifier for MNIST 3 and 8 digits)
since a larger number of classes and concepts are involved in the
classification task. Due to the space limit, here we mainly show the visual
explanation obtained for the EMNIST dataset in which we use $M=3$ concepts.
The dimension of $\operatorname{\bm{\alpha}}_{i}$ and
$\operatorname{\bm{\beta}}$ are $K=1$ and $L=7$, respectively. The explanation
results of other datasets and further detailed experiment settings can be
found in Appendix A.5, A.6 and A.7.

###  5.2 Qualitative results

(a) Controlling switch $\gamma_{0}$ of concept $m_{0}$ (bottom stroke)

(b) Transition by $m_{1}$

(c) Controlling switch $\gamma_{1}$ of concept $m_{1}$ (middle stroke)

(d) Transition by $m_{1}$

(e) Controlling binary $\gamma_{2}$ of concept $m_{2}$ (right stroke)

(f) Transition by $m_{2}$

Figure 3: (a, c, e) The binary explanation with the intervention for each
concept. (1st row) The encoded concept switch $\hat{\gamma_{i}}$ (yellow/gray
for 1/0). (2nd row) the original reconstruction $\hat{x}$. (4th row) The
reconstruction after alternating switch $\gamma_{i}$. (b, d, f) The transition
graph of prediction output.

In Figure 3 (3(a), 3(c) and 3(e)), we showed three discovered binary concepts
for the EMNIST dataset. In each image, we show in the first row the encoded
binary switch of concept $m_{i}$ for different samples, in which yellow
indicates $\hat{\gamma_{i}}=1$ and gray indicates $\hat{\gamma_{i}}=0$. The
second row shows the original reconstructed image $\hat{x}$ while the fourth
row shows the image reconstructed when we reverse the binary switch
$\hat{x}^{[i]}$. The border color indicates the prediction result of each
image. Finally, the third row show the difference of $\hat{x}^{[i]}$ and
$\hat{x}^{[i]}$.

We observed that the proposed method was able to discover useful binary
concepts for explaining the classifier. First, the binary switches of these
concepts have a large causal effect on the classifier output, i.e.,
alternating the switch affects the prediction. For example, Figure 3(a)
explains that adding a bottom stroke to letter A has a significant effect on
the classifier output. Not only that, each concept captured a group of similar
interventions and can be easily interpreted, i.e., concept $m_{0}$ represents
the bottom stroke, concept $m_{1}$ represents the right stroke, and concept
$m_{2}$ represents the middle stroke.

The explanation in Figure 3 (3(a), 3(c) and 3(e)) can be considered as a local
explanation which focus on explaining specific samples. Not only that, the
proposed method also excels in providing organized knowledge about the
discovered concepts and prediction classes. In particular, we can aggregate
the causal effect of these local explanation for each concept and class to
assess how the each a binary switch change the prediction. Letting
$X_{u}=\\{\operatorname{\mathbf{x}}\in X\mid
f(\hat{\operatorname{\mathbf{x}}})=u\\}$, the transition probability from
$y=u$ to $y=v$ for a concept $m_{i}$ using the do operation $do(\gamma_{i}=d)$
($d\in\\{0,1\\}$) can be obtained as
$w_{u,v}^{do(\gamma_{i}=d)}=\text{Pr}[y=v\mid
y=u,do(\gamma_{i}=d)]=\frac{1}{|X_{u}|}\sum_{\operatorname{\mathbf{x}}\in
X_{u}}\mathbbm{1}[f(\hat{\operatorname{\mathbf{x}}}^{do(\gamma_{i}=d)})=v]$.

(a) $\operatorname{\bm{\alpha}}_{2}$ (bottom)

(b) $\operatorname{\bm{\alpha}}_{1}$ (middle)

(c) $\operatorname{\bm{\alpha}}_{2}$ (right)

(d) $\beta_{6}$ (width)

(e) $\beta_{6}$ (skewness)

Figure 4: Visualization of the learned concept-specific and global variants.
The proposed method captured the variant within each causal concept, i.e., the
change of shape of (a) the bottom stroke, (b) the middle stroke and (c) the
right stroke. (d, e) Our method was also able to disentangle the concepts
variants with other variants that does not affect the prediction.

In 3 (3(b), 3(d) and 3(f)), we show the calculated transition probabilities
for each concept as a graph in which each note represents a prediction class.
A solid arrow (dashed arrow) represents the transition when activating
(deactivating) a concept and the arrow thickness shows the transition
probability $w_{u,v}^{do(\gamma_{i}=1)}$ ($w_{u,v}^{do(\gamma_{i}=0)}$). We
neglect the transition which transition probability is less than 0.1 For
example, from Figure 3(b), one can interpret that the bottom stroke is
important to distinguish (E,F) and (A,B).

Finally, in Figure 4 (4(a), 4(b) and 4(c)), we show the captured variants
within each concept and other global variants, that have a small affect on the
classifier output. In contrast to binary switches, these variants explain what
does not change the prediction. We first activate the concept $m_{i}$ using
the do-operation $\text{do}(\gamma_{i}=1)$, then plot the reconstruction while
alternating $\operatorname{\bm{\alpha}}_{i}$. We observed that
$\operatorname{\bm{\alpha}}_{0}$ captured the length of the bottom stroke,
$\operatorname{\bm{\alpha}}_{1}$ captured the shape of the right stroke, and
$\operatorname{\bm{\alpha}}_{2}$ captured the length of the inside (middle)
stroke, respectively. Especially, our method was also able to differentiate
the concept-specific variants with other global variants
$\operatorname{\bm{\beta}}$ such as skewness or width (Figure 4 4(d), 4(e)).

###  5.3 Comparing with other methods.

(a) Intervention by decreasing $\alpha_{0}$

(b) Intervention by increasing $\alpha_{0}$

Figure 5: A causal factor by O’Shaughnessy et al. (2020). Low interpretability
results are framed (More details in text)

We compare our method to other baselines in Figure 2. First, saliency-map-
based methods, which use a saliency map to quantify the importance of
(super)pixels, although is easy to understand, do not explain why highlighted
(super)pixels are important (Figure 2(a)). Because they only provide one
explanation for each input, they can not explain how these pixels distinguish
the predicted class from others classes. Our method, can provide multiple
explanations by interventing difference concepts.

(a) $p(\operatorname{\bm{\gamma}})$ vs $p^{*}(\operatorname{\bm{\gamma}})$ vs
no causal

(b) $\lambda_{\text{R}}=0$

(c) $\lambda_{\text{R}}=1$

Figure 6: (a) (MNIST) Train-time MI and test-time TTE of ten runs when
$\mathcal{L}_{\text{CE}}$ is based on $p(\operatorname{\bm{\gamma}})$ (red),
$p^{*}(\operatorname{\bm{\gamma}})$ (blue), and when trained without
$\mathcal{L}_{\text{CE}}$ (green). (b) Discovered binary concepts and their
transition graph when trained with and without $\mathcal{L}_{\text{R}}$.

Next, we compare to O’Shaughnessy et al. (2020), in which we used a VAE model
with ten continuous factors and encouraged three factors to have causal
effects on predicted classes. In Figure 5, we visualize $\alpha_{0}$ which
achieved the largest causal effect. In Figure 5(a) (5(b)), we decrease
(increase) $\alpha_{0}$ until the its prediction label changes and show that
intervention result in the third row. First, we observed that it failed to
disentangle different causal factors as $\alpha_{0}$ affects all the bottom,
middle and right strokes. For example, in Figure 5(a), decreasing $\alpha_{t}$
changed the letter D in the 10th column to letter B (middle stroke concept),
while changed the letter D in the 11th column to letter C (left stroke
concept). A similar result is also observed in Figure 5(b) for letter E.
Second, it failed to disentangle the concept-specific variant, which does not
affect the prediction. For example, for the letter A and B (1st to 6th column)
in Figure 5(b), increasing $\alpha_{0}$ does not only affect the occurrence of
the middle stroke, but also changes the shape of the right stroke.

Our method overcomes these limitations with a carefully designed binary-
discrete structure coupled with the proposed causal effect and
interpretability regularizer. By encouraging the causal influence of only the
binary switches, our method can disentangle what affects the prediction and
the variant of samples with the same prediction. Thus, it encourages that a
binary switch $m_{i}$ only changes the prediction from a class $y_{k}$ to only
one other class $y_{k^{\prime}}$, resulting in a more interpretable
explanation. We also emphasize that the binary-continuous mixed structure
alone is not enough to obtain valuable concepts for explanation (Figure 2(b)).

###  5.4 Quantitative results

We evaluate the causal influence of a concept set using the total transition
effect (TTE) which is defined as
$\text{TTE}=\frac{1}{M}\sum_{i\in[M]}\sum_{u,v\in[T]}[w_{u,v}^{do(\gamma_{i}=1)}+w_{u,v}^{do(\gamma_{i}=0)}]$
where $M$ and $T$ are the number of concepts and classes, respectively. Here,
a large value of TTE indicates a significant overall causal effect by the
whole discovered concept set on all class transitions. Compared to information
flow, TTE can evaluate more directly and faithfully the causal effect of
binary switches on dataset $X$. Moreover, it is also more easy for end-user to
understand.

In Figure 6(a), we show the test-time mutual information and the TTE values
when the causal objective $\mathcal{L}_{\text{CE}}$ uses the prior
$p^{*}(\operatorname{\bm{\gamma}})$ (Eq. (5)), VAE model’s prior
$p(\operatorname{\bm{\gamma}})$ and when trained without
$\mathcal{L}_{\text{CE}}$. The interpretability regularizers are included in
all settings. We observed that when $p(\operatorname{\bm{\gamma}})$ is used,
there are cases where the estimated mutual information is high, but the total
transition effect is small. On the other hand, the mutual information obtained
with estimated $p^{*}(\operatorname{\bm{\gamma}})$ aligns better with the TTE
value. We claim that this is because of the deviation between
$p(\operatorname{\bm{\gamma}})$ and the ‘true’
$p^{*}(\operatorname{\bm{\gamma}})$. By estimating
$p^{*}(\operatorname{\bm{\gamma}})$ on the run, our method can better evaluate
and optimize the causal influence of $\operatorname{\bm{\gamma}}$ on $y$.
Moreover, we also observed that without the causal objective, we failed to
discover causal binary concepts.

Next, we evaluate how implementing user’s preferences and prior knowledge via
$\mathcal{L}_{\text{R}}$ increases the interpretability of concepts. In Figure
6(b), we show an example of concepts discovered when we train the model
without the interpretability regularizer. We see that alternating the binary
switch of this concept (top) only replaces the digit 4, 7, 9 by the digit 1
but does not provide any proper explanation why the image is identified as 1.
Although this concept has a large causal effect, it barely offers valuable
knowledge. Our method, using the interpretability regularizers, can discover
binary concepts with high interpretability that adequately explain that digit
7 can be distinguished from digit 1 based on the existence of the top stroke
(Figure 6(c)).

##  6 Conclusion

We introduced the problem of discovering binary concepts for explaining a
black-box classifier. We first proposed a structural generative model that can
properly express binary concepts. Then, we proposed a learning process that
simultaneously learns the data distribution and encourages the binary switches
to have a large causal effect on the classifier output. The proposed method
also allows integrating user’s preferences and prior knowledge for better
interpretability and consistency. We demonstrated that the proposed method
could discover interpretable binary concepts with a large causal effect which
can effectively explain the classification model for multiple datasets.

## Acknowledgments and Disclosure of Funding

## References

  * Ay and Polani [2008] Nihat Ay and Daniel Polani.  INFORMATION FLOWS IN CAUSAL NETWORKS.  _Advs. Complex Syst._ , 11(01):17–41, February 2008.
  * Burgess et al. [2018] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner.  Understanding disentangling in $\beta$-VAE.  April 2018.
  * Choi et al. [2020] Jaewoong Choi, Geonho Hwang, and Myungjoo Kang.  Discond-VAE: Disentangling continuous factors from the discrete.  September 2020.
  * Cohen et al. [2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik.  Emnist: Extending mnist to handwritten letters.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 2921–2926. IEEE, 2017.
  * Gardenfors [2014] Peter Gardenfors.  _The Geometry of Meaning: Semantics Based on Conceptual Spaces_.  MIT Press, January 2014.
  * George and McCulloch [1997] Edward I George and Robert E McCulloch.  Approaches for bayesian variable selection.  _Statistica sinica_ , pages 339–373, 1997.
  * Ghorbani et al. [2019] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim.  Towards automatic concept-based explanations.  _Advances in Neural Information Processing Systems_ , 32:9277–9286, 2019.
  * Goyal et al. [2020] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim.  Explaining classifiers with causal concept effect (CaCE).  _arXiv:1907.07165 [cs, stat]_ , February 2020.
  * Gupta et al. [2020] Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava.  PatchVAE: Learning local latent codes for recognition.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4745–4754, Seattle, WA, USA, 2020. IEEE.
  * Gyawali et al. [2019] Prashnna Gyawali, Zhiyuan Li, Cameron Knight, Sandesh Ghimire, B Milan Horacek, John Sapp, and Linwei Wang.  Improving disentangled representation learning with the beta bernoulli process.  pages 1078–1083, 2019.
  * Higgins et al. [2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning basic visual concepts with a constrained variational framework.  November 2016.
  * Holland [1988] Paul W Holland.  Causal inference, path analysis, and recursive structural equations models.  _Sociol. Methodol._ , 18:449–484, 1988.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  pages 2668–2677, 2018.
  * Kim and Bastani [2019] Carolyn Kim and Osbert Bastani.  Learning interpretable models with causal guarantees.  _arXiv:1901.08576 [cs, stat]_ , January 2019.
  * Kingma and Welling [2014] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In _2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings_ , 2014.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and Chris Burges.  MNIST handwritten digit database, 2010.
  * Lewontin [1974] Richard C Lewontin.  The analysis of variance and the analysis of causes.  _Am. J. Hum. Genet._ , 26(3):400–411, 1974.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _international conference on machine learning_ , pages 4114–4124. PMLR, 2019.
  * Maddison et al. [2017] C Maddison, A Mnih, and Y Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  2017\.
  * Narendra et al. [2018] Tanmayee Narendra, Anush Sankaran, Deepak Vijaykeerthy, and Senthil Mani.  Explaining deep learning models using causal inference.  _arXiv:1811.04376 [cs, stat]_ , November 2018.
  * O’Shaughnessy et al. [2020] Matthew O’Shaughnessy, Gregory Canal, Marissa Connor, Mark Davenport, and Christopher Rozell.  Generative causal explanations of black-box classifiers.  _Advances in Neural Information Processing Systems_ , 2020.
  * Parafita and Vitrià [2019] Álvaro Parafita and Jordi Vitrià.  Explaining visual models by causal attribution.  pages 4167–4175, 2019.
  * Pearl [2009] Judea Pearl.  _Causality_.  Cambridge University Press, September 2009.
  * Ross and Doshi-Velez [2021] Andrew Slavin Ross and Finale Doshi-Velez.  Benchmarks, algorithms, and metrics for hierarchical disentanglement.  February 2021.
  * Schwab and Karlen [2019] Patrick Schwab and Walter Karlen.  Cxplain: Causal explanations for model interpretation under uncertainty.  _Advances in Neural Information Processing Systems_ , 32:10220–10230, 2019.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Springenberg et al. [2014] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  December 2014.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , pages 3319–3328. PMLR, 2017.
  * Tonolini et al. [2020] Francesco Tonolini, Bjørn Sand Jensen, and Roderick Murray-Smith.  Variational sparse coding.  In Ryan P Adams and Vibhav Gogate, editors, _Proceedings of The 35th Uncertainty in Artificial Intelligence Conference_ , volume 115 of _Proceedings of Machine Learning Research_ , pages 690–700. PMLR, 2020.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms.  August 2017.
  * Yeh et al. [2020] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  _Advances in Neural Information Processing Systems_ , 33, 2020.

##  Appendix A Appendix

###  A.1 Algorithm for estimating $I(\operatorname{\mathbf{d}};Y)$

(a) EMNIST

(b) Fashion-MNIST

Figure 7: Train-time MI and test-time TTE of ten runs when
$\mathcal{L}_{\text{CE}}$ is based on $p(\operatorname{\bm{\gamma}})$ (red),
$p^{*}(\operatorname{\bm{\gamma}})$ (blue), and when trained without
$\mathcal{L}_{\text{CE}}$ (green). Addition results for EMNIST and FMNIST.

Data: mini-batch data $X$,

# samples for continuous latents $N_{c}$,

# classes $T$

$I\leftarrow 0$;

$S\leftarrow$ all combinations of $M$ binary latents;

$p_{y}\leftarrow zeros(T)$;

for  _$\operatorname{\mathbf{s}}\in S$_ do

Estimate $p^{*}(\operatorname{\mathbf{s}})$ using Eq. (7);

$p_{y\mid\operatorname{\mathbf{s}}}\leftarrow zeros(K)$;

for  _$j=1$ to  $N_{c}$_ do

$\bm{\beta}\leftarrow L$-dim vector from $\mathcal{N}^{*}_{\delta}(0,I)$;

$\forall_{i},\bm{\alpha}_{i}\leftarrow M$-dim vector from
$\mathcal{N}^{*}_{\delta}(0,I)$;

$A\leftarrow(\bm{\alpha}_{1},\bm{\alpha}_{2},\dots,\bm{\alpha}_{M})$;

$\hat{\operatorname{\mathbf{x}}}\leftarrow$ sample from
$p(\operatorname{\mathbf{x}}\mid\operatorname{\mathbf{s}},A,\bm{\beta})$;

$p_{y\mid\operatorname{\mathbf{s}}}\leftarrow
p_{y\mid\operatorname{\mathbf{s}}}+\frac{1}{N_{c}}p(y\mid\hat{\operatorname{\mathbf{x}}})$;

$I\leftarrow
I+p^{*}(\operatorname{\mathbf{s}})\sum_{t=1}^{T}p(y\mid\hat{\operatorname{\mathbf{x}}})[t]\log
p(y\mid\hat{\operatorname{\mathbf{x}}})[t]$;

$p_{y}\leftarrow
p_{y}+p^{*}(\operatorname{\mathbf{s}})p(y\mid\hat{\operatorname{\mathbf{x}}})$;

$I\leftarrow I-\sum_{t=1}^{T}p_{y}[t]\log p_{y}[t]$;

return _$I$_ ;

Algorithm 1 Algorithm for estimating $I(\operatorname{\mathbf{d}};Y)$.

Let
$S=\\{\operatorname{\mathbf{s}}^{(0)},\operatorname{\mathbf{s}}^{(1)},\dots,\operatorname{\mathbf{s}}^{(2^{M})}\\}$
where $s^{(i)}\in\\{0,1\\}^{M}$ be the set of $2^{M}$ possible combinations of
the switches for $M$ concepts, we can obtain $\mathcal{L}_{\text{CE}}$ as

$\displaystyle\mathcal{L}_{\text{CE}}(X)=$
$\displaystyle-\sum_{\operatorname{\mathbf{s}}^{(j)}\in
S}p^{*}(\operatorname{\mathbf{s}}^{(j)})(\sum_{y^{\prime}}p(y\mid\operatorname{\mathbf{s}}^{(j)})\log
p(y\mid\operatorname{\mathbf{s}}^{(j)}))$ $\displaystyle+\sum_{y}p(y)\log
p(y),$ (6)

in which

$\displaystyle p^{*}(\operatorname{\bm{\gamma}})$
$\displaystyle\approx\frac{1}{|X|}\sum_{\operatorname{\mathbf{x}}\in
X}\prod_{i=1}^{M}q(\gamma_{i}\mid\operatorname{\mathbf{x}}).$ (7)

Here,
$q(\gamma_{i}\mid\operatorname{\mathbf{x}})=\pi_{i}\gamma_{i}+(1-\pi_{i})(1-\gamma_{i})$
where $(\pi_{1},\dots,\pi_{M})=Q^{d}(\operatorname{\mathbf{x}})$.

Moreover, in Eq. (6), $p(y\mid\operatorname{\mathbf{s}})$ is estimated by
using $N_{c}$ samples of $A$ and $\bm{\beta}$ drawing from the corresponding
VAE’s prior $\\{p(\operatorname{\bm{\alpha}}_{i})\\}_{i=1}^{M}$ and
$p(\bm{\beta})$, respectively. The detailed algorithm is described in
Algorithm 1. We also show the additional result of the test-time mutual
information and the TTE values when the causal object is based on estimated
$p^{*}(\operatorname{\bm{\gamma}})$ (blue), VAE model’s prior
$p(\operatorname{\bm{\gamma}})$ (red) and when trained without causal
objective (green) for EMNIST and FMNIST dataset.

###  A.2 Proofs

Figure 8: Causal DAG of O’Shaughnessy et al. (2020)

######  Proposition 2 ((O’Shaughnessy et al. (2020), Proposition 2)).

The information flow from $\operatorname{\bm{\alpha}}$ to Y in the DAG of
Figure 8, in which $(\operatorname{\bm{\alpha}},\operatorname{\bm{\beta}})$
are independence, coincides with the mutual information between $\bm{\alpha}$
and $Y$. That is,

$\displaystyle I(\operatorname{\bm{\alpha}}\rightarrow
Y)=I(\operatorname{\bm{\alpha}};Y)=\operatorname{\mathbb{E}}_{\operatorname{\bm{\alpha}},Y}\left[\frac{p(\operatorname{\bm{\alpha}})p(Y\mid\operatorname{\bm{\alpha}})}{p(\operatorname{\bm{\alpha}})p(Y)}\right]$
(8)

####  A.2.1 Proof of Proposition 1

###### Proof.

Considering the proposed DAG in Figure 3a, we first let
$\operatorname{\bm{\alpha}}^{\prime}=\operatorname{\bm{\gamma}}$ and
$\operatorname{\bm{\beta}}^{\prime}=(A,\operatorname{\bm{\beta}})$. Since,
$\operatorname{\bm{\gamma}}^{\prime}\sim p^{*}(\operatorname{\bm{\gamma}})$,
$(\operatorname{\bm{\alpha}}_{1},\dots,\operatorname{\bm{\alpha}}_{M})\sim\prod_{i=1}^{M}p(\alpha_{i})\text{
with }p(\alpha_{i})=\mathcal{N}^{\text{fold}}_{\delta}(0;\mathcal{I})$ and
$\operatorname{\bm{\beta}}\sim\mathcal{N}^{\text{fold}}_{\delta}(0;\mathcal{I})$,
we have that in our causal DAG, $\operatorname{\bm{\alpha}}^{\prime}$ and
$\operatorname{\bm{\beta}}^{\prime}$ are independence. Thus, the proposed DAG
coincides with the causal DAG proposed of O’Shaughnessy et al. (2020), in
which $\operatorname{\bm{\alpha}}$ (of Figure 8) is replaced by
$\operatorname{\bm{\alpha}}^{\prime}$ and $\operatorname{\bm{\beta}}$ (of
Figure 8) is replaced by $\operatorname{\bm{\beta}}^{\prime}$. From
Proposition 2, we can conclude that $I(\operatorname{\bm{\gamma}}\rightarrow
Y)$ coincides with $I(\operatorname{\bm{\gamma}};Y)$ in the proposed DAG. ∎

###  A.3 On the $\delta\text{-Shifted}$ Folded Normal Distribution
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$

####  A.3.1 Why not $\mathcal{N}(\mu,\sigma^{2})$ but
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$

We discuss why the standard Gaussian distribution, which is a common choice
for VAE models, is not appropriate for finding the proposed causal effect.
Reminding that in the conceptual latent code $\operatorname{\mathbf{z}}$, the
corresponding code $\bm{\psi}_{i}$ of concept $m_{i}$ is
$\bm{\psi}_{i}=\hat{\gamma_{i}}\times\operatorname{\bm{\alpha}}_{i}$. If
$\operatorname{\bm{\alpha}}_{i}\sim\mathcal{N}(0,I)$, then for any
$\operatorname{\bm{\alpha}}_{i}$ around the center zero,
$\bm{\psi}_{i}\mid\hat{\gamma_{i}}=0$ and
$\bm{\psi}_{i}\mid\hat{\gamma_{i}}=1$ would takes a very similar values. Thus,
$p(y\mid\gamma_{i}=0,\operatorname{\bm{\alpha}}_{i})$ would be close to
$p(y\mid\gamma_{i}=1,\operatorname{\bm{\alpha}}_{i})$ and it conflicts with
our causal effect which requires that $p(y\mid\gamma_{i}=0)$ to be different
from $p(y\mid\gamma_{i}=1)$ regardless value of
$\operatorname{\bm{\alpha}}_{i}$. To resolve this conflict, we propose to use
the $\delta\text{-Shifted}$-Folded Normal Distribution to parameterize
$Q^{c}(A,\bm{\beta}\mid\operatorname{\mathbf{x}})$ to avoid
$\operatorname{\bm{\alpha}}_{i}$ around $0$. Here, $\delta\text{-Shifted}$
Folded Normal Distribution
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$ is the distribution of
$|x|+\delta$ for $\delta>0$ in which $x\sim\mathcal{N}(\mu,\sigma^{2})$. We
observed from our experiments that this design choice significantly boost the
discovering ability for binary concepts that have large causal effect.

####  A.3.2 Implementing $\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$

We implement the $\delta\text{-Shifted}$ Folded Normal Distribution
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$ using the standard
implementation of the Normal Distribution $\mathcal{N}(\mu,\sigma^{2})$. In
particular, to obtain a sample $z$ from
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$, we first sample
$\operatorname{\mathbf{z}}^{\prime}\sim\mathcal{N}(\mu,\sigma^{2})$ then apply
the transformation $z=|z^{\prime}|+\alpha$. Since obtaining exact KL-
divergence for the Folded Normal Distribution is complicated, we substitute
$KL(\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2}),\mathcal{N}^{\text{fold}}_{\delta}(0,1))$
by the KL-divergence of the corresponding Normal Distribution, i.e.,
$KL(\mathcal{N}(\mu,\sigma^{2}),\mathcal{N}(0,1))$ to optimizing the VAE
objective (Eq. 3). Through the experiments, we observed that implementing
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$ this way does not cause
harmful effect on the learning process. We leave the investigation of more
sophisticated methods for feature work.

####  A.3.3 Efficacy of using
$\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$

Figure 9: Total Transition Effect for different choice of distribution for
continuous latents.

We then confirm the impact of $\delta\text{-Shifted}$ Folded Normal
Distribution $\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$, and the
impact of using the estimated distribution $p^{*}(\operatorname{\bm{\gamma}})$
in evaluating the causal effect. We conduct the learning process ten times for
each setting and evaluate the averaged TTE value for ten runs. In Figure 9, we
show the average TTE value for different distribution choices for continuous
latents. We observed that adopting the $\delta\text{-Shifted}$-Folded Normal
Distribution has a significant effect in discovering causal binary concepts.

###  A.4 Details of interpretability regularizers

$\displaystyle\mathcal{L}_{\text{compact}}(\operatorname{\mathbf{x}})=$
$\displaystyle\frac{1}{M}\sum_{i=1}^{M}\frac{1}{P}\|\hat{\operatorname{\mathbf{x}}}-\hat{\operatorname{\mathbf{x}}}^{[i]}\|,$
(9)
$\displaystyle\mathcal{L}_{\text{directional}}(\operatorname{\mathbf{x}})=$
$\displaystyle\frac{1}{M}\sum_{i=1}^{M}\frac{1}{P}\sum_{p=1}^{P}\mathbbm{1}[\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}>\hat{\operatorname{\mathbf{x}}}_{p}]\times|\hat{\operatorname{\mathbf{x}}}_{p}-\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}|\times\hat{\gamma_{i}}$
(10) $\displaystyle+$
$\displaystyle\mathbbm{1}[\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}\leq\hat{\operatorname{\mathbf{x}}}_{p}]\times|\hat{\operatorname{\mathbf{x}}}_{p}-\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}|\times(1-\hat{\gamma_{i}}),$

where $M$ is the number of concepts, $P$ is the dimension of the input and
$\hat{\operatorname{\mathbf{x}}}^{[i]}$ is the reconstruction after reversing
the latent code $\hat{\gamma}_{i}$ of concept $m_{i}$. We give a brief
interpretation for Eq. (10). Consider a concept $m_{i}$ in a sample
$\operatorname{\mathbf{x}}$. If concept $m_{i}$ is activated, i.e.,
$\hat{\gamma}_{i}=1$, then $\hat{\operatorname{\mathbf{x}}}^{[i]}$ corresponds
to the turn off intervention $do(\gamma_{i}=0)$. In this case, we expect that
this intervention only removes some pixels in $\hat{x}$. Thus, we penalize the
difference
$|\hat{\operatorname{\mathbf{x}}}_{p}-\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}|$
for positions $p$ where the pixel value increases, i.e., where
$\hat{\operatorname{\mathbf{x}}}^{[i]}_{p}>\hat{\operatorname{\mathbf{x}}}_{p}$.
Finally, we combine these regularizers as
$\mathcal{L}_{\text{R}}(\operatorname{\mathbf{x}})=\lambda_{3}\mathcal{L}_{\text{compact}}(\operatorname{\mathbf{x}})+\lambda_{4}\mathcal{L}_{\text{directional}}(\operatorname{\mathbf{x}})$.

####  A.4.1 Inconsistency issue of discovered concept

Figure 10: The consistence score of discovered concepts across successful runs
when being trained with and without the interpretability regularizer.

Training the generative model with interpretability regularizers also resolves
to an extent the inconsistency issue of discovered concepts, i.e., each trial
results in a different set of concepts. Since only humans can confirm if the
concepts are meaningful, this inconsistency can lead to a cherry-picking
scheme that is, users might keep training the model until he/she encounts
favorite concepts accidentally. We note that this is also a common problem of
explanation methods that use deep generative models. We evaluate the
consistency between different runs based on the similarity of the prediction
transition graphs. Letting $I,I^{\prime}$ be two arbitrary permutations of
$\\{1,2,\dots,M\\}$, we evaluate the similarity of two transition graphs $W$
and $W^{\prime}$ as follows

$\displaystyle\text{Similarity}(W,W^{\prime})=$
$\displaystyle\max_{I,I^{\prime}}\text{CosSim}(W_{(I)},W^{\prime}_{(I^{\prime})})$
(11) where $\displaystyle
W_{(I)}=(W_{i_{0}},W_{i_{1}},\dots,W_{i_{M}})_{i_{j}\in I}$ $\displaystyle
W^{\prime}_{(I^{\prime})}=(W^{\prime}_{i^{\prime}_{0}},W^{\prime}_{i^{\prime}_{1}},\dots,W^{\prime}_{i^{\prime}_{M}})_{i^{\prime}_{j}\in
I^{\prime}}$ and $\displaystyle
W_{i}=(w_{u,v}^{do(\gamma_{i}=0)},w_{u,v}^{do(\gamma_{i}=1)})_{u,v\in[T]}$

Here, the maximal operation $\max_{I,I^{\prime}}$ is used to consider
different orders of the discovered concepts, which can be arbitrary in each
run. In Figure 10, we show the average similarities between different runs for
each dataset. To avoid the cases in which the optimization fails, we only
consider those runs which achieved a large causal effect (i.e., best four runs
in ten runs). We observed that adding the regularization also improves the
consistency of the discovering concepts. In this work, we demonstrated our
method with grayscale images, where small parts in the image take an essential
role in classifying the object. We leave the exploration of other
regularizations for other domains to future work.

###  A.5 Experiment details

Table 1: Classifier architecture Layer (type) | Output Shape | # Param
---|---|---
Conv2d-1 | [-1, 32, 16, 16] | 544
Conv2d-3 | [-1, 64, 8, 8] | 32,832
Linear-6 | [-1, 64] | 262,208
Linear-10 | [-1, 256] | 16,640
Linear-11 | [-1, 256] | 16,640
Linear-14 | [-1, 4] | 1,028
Linear-15 | [-1, 4] | 1,028
Total params |  | 330,920
Table 2: Generative model architecture Layer (type) | Output Shape | # Param
---|---|---
Encoder
Conv2d-1 | [-1, 64, 16, 16] | 1,024
Conv2d-3 | [-1, 128, 8, 8] | 131,072
Conv2d-5 | [-1, 256, 4, 4] | 524,288
Conv2d-7 | [-1, 64, 4, 4] | 16,384
Conv2d-9 | [-1, 2, 1, 1] | 2,050
Conv2d-10 | [-1, 10, 1, 1] | 10,250
Conv2d-11 | [-1, 10, 1, 1] | 10,250
Conv2d-12 | [-1, 8, 1, 1] | 8,200
Conv2d-13 | [-1, 8, 1, 1] | 8,200
Decoder
ConvTranspose2d-17 | [-1, 64, 4, 4] | 10,240
ConvTranspose2d-19 | [-1, 256, 4, 4] | 16,384
ConvTranspose2d-21 | [-1, 128, 8, 8] | 524,288
ConvTranspose2d-23 | [-1, 64, 16, 16] | 131,072
ConvTranspose2d-25 | [-1, 1, 32, 32] | 1,024
Total params |  | 1,394,726

All of our experiments were run using two GeForce RTX 3090 GPUs. We use the
standard split for all dataset (MNIST, EMNIST, FMNIST) which are distributed
via the torchvision package. We then select the sample with targeted classes
for each experiment to obtain the final train and test set. All input images
are resized to $32\times 32$ images. We show the network architecture of the
classifier and the VAE model used in each experiment in Table 1, 2. The
classifier is trained with a batch size of $256$ using the Adam optimizer with
a learning rate of $0.0005$ for $20$ epochs. The trained classifier achieved
an accuracy of $95.45\%$ on the test dataset. The VAE model is optimized with
a batch size of $265$ using the Adam optimizer with learning rate $0.0005$ for
$30$ epochs. At each training step, the causal effect term is estimated using
the Algorithm 1 with $N_{c}$ samples for continuous latents.

For the EMNIST experiment (class A, B, C, D, E, F), we adopted a generative
model that contains $M=3$ concepts in which each concept-specific variant
$\operatorname{\mathbf{a}}_{i}$ has a dimension of $K=1$. The dimension of the
non-causal factor $\bm{\beta}$ is set to $L=7$. We resize the input image to
$32\times 32$ grayscale image. The other hyper-parameters are set as
$\lambda_{1}=\lambda_{2}=1$, $\lambda_{3}=50$, $\lambda_{4}=1000$ and
$\lambda_{CE}=50$. Moreover, we employ the $0.5\text{-Shifted}$ Folded Normal
Distribution $\mathcal{N}^{\text{fold}}_{\delta}(\mu,\sigma^{2})$, i.e.,
$\delta=0.5$ for the continuous distribution. We train the generative model
using Adam with a learning rate $lr=0.0005$ for 30 epochs.

For the MNIST experiment (class 1,4,7,9), we adopted a generative model that
contains $M=2$ concepts in which each concept-specific variant
$\operatorname{\mathbf{a}}_{i}$ has a dimension of $K=1$. The dimension of the
non-causal factor $\bm{\beta}$ is set to $L=8$. The other hyper-parameters are
set as $\lambda_{1}=\lambda_{2}=1$, $\lambda_{3}=100$, $\lambda_{4}=1000$ and
$\lambda_{CE}=50$. Specially, for the compactness regularizer coefficient
$\lambda_{3}$, we init $\lambda_{3}=5$ at the start of the training process,
and increase $\lambda_{3}$ by $5$ for each $100$ update steps until
$\lambda_{3}=100$. Moreover, we initialize the Relaxed Bernoulli
distribution’s temperature at $0.4$ and decrease it using annealing schedule
with annealing rate $0.0001$ with until it reaches $0.1$. In the testing
phase, this temperature is set to $0$ to obtain a (non-relaxed) Bernoulli
distribution.

On the other hand, for the Fashion-MNIST experiment (t-shirt/top, dress,
coat), we adopted a generative model that contains $M=2$ concepts in which
each concept-specific variant $\operatorname{\mathbf{a}}_{i}$ has a dimension
of $K=1$. The dimension of the non-causal factor $\bm{\beta}$ is set to $L=8$.
The other hyper-parameters are set as $\lambda_{1}=\lambda_{2}=1$,
$\lambda_{3}=10e$, $\lambda_{4}=1000$ and $\lambda_{CE}=25$. Other parameters
are identical with the setting for MNIST dataset.

###  A.6 Results for MNIST

We show the result for dataset MNIST in Figure 11 and 12. We also show the
result obtained when trained without the interpretability regularizers in
Figure 13 and 14.

###  A.7 Results of Fashion-MNIST

We show the result for dataset Fashion-MNIST in Figure 15 and 16. We also show
the result obtained when trained without the interpretability regularizers in
Figure 17 and 18.

###  A.8 Addition results of EMNIST

We show the result obtained when trained without the interpretability
regularizers for EMNIST in Figure 19 and 20.

(a) Controlling switch $\gamma_{0}$ (middle stroke)

(b) Controlling binary $\gamma_{1}$ (top stroke)

Figure 11: (MNIST) The binary explanation. (1st row) The encoded concept
switch $\hat{\gamma_{i}}$ (yellow/gray for 1/0). (2nd row) the original
reconstruction $\hat{x}$. (4th row) The reconstruction after alternating
switch $\gamma_{i}$.

(a) $\operatorname{\bm{\alpha}}_{0}$ (middle)

(b) $\operatorname{\bm{\alpha}}_{1}$ (top)

(c) $\beta_{1}$ (cross)

(d) $\beta_{5}$ (roundness)

(e) $\beta_{7}$ (width)

Figure 12: (MNIST) Visualization of the learned concept-specific and global
variants.

(a) Controlling switch $\gamma_{0}$

(b) Controlling binary $\gamma_{1}$

Figure 13: (MNIST without $\mathcal{L}_{\text{R}}$) The binary explanation.

(a) $\operatorname{\bm{\alpha}}_{0}$

(b) $\operatorname{\bm{\alpha}}_{1}$

(c) $\beta_{1}$

(d) $\beta_{3}$

(e) $\beta_{4}$

Figure 14: (MNIST without $\mathcal{L}_{\text{R}}$) Visualization of the
learned concept-specific and global variants.

(a) Controlling switch $\gamma_{0}$ (long sleeve)

(b) Controlling binary $\gamma_{1}$ (shoulder + body shape)

Figure 15: (Fashion-MNIST) The binary explanation. (1st row) The encoded
concept switch $\hat{\gamma_{i}}$ (yellow/gray for 1/0). (2nd row) the
original reconstruction $\hat{x}$. (4th row) The reconstruction after
alternating switch $\gamma_{i}$.

(a) $\operatorname{\bm{\alpha}}_{0}$ (long sleeve)

(b) $\operatorname{\bm{\alpha}}_{1}$ (shoulder)

(c) $\beta_{6}$ (lower width)

(d) $\beta_{6}$ (darkness)

(e) $\beta_{6}$ (neck height)

Figure 16: (Fashion-MNIST) Visualization of the learned concept-specific and
global variants.

(a) Controlling switch $\gamma_{0}$

(b) Controlling binary $\gamma_{1}$

Figure 17: (Fashion-MNIST without $\mathcal{L}_{\text{R}}$) The binary
explanation.

(a) $\operatorname{\bm{\alpha}}_{0}$

(b) $\operatorname{\bm{\alpha}}_{1}$

(c) $\beta_{6}$

(d) $\beta_{6}$

(e) $\beta_{6}$

Figure 18: (Fashion-MNIST without $\mathcal{L}_{\text{R}}$) Visualization of
the learned concept-specific and global variants.

(a) Controlling switch $\gamma_{0}$

(b) Controlling binary $\gamma_{1}$

(c) Controlling binary $\gamma_{1}$

Figure 19: (EMNIST without $\mathcal{L}_{\text{R}}$) The binary explanation.

(a) $\operatorname{\bm{\alpha}}_{0}$

(b) $\operatorname{\bm{\alpha}}_{1}$

(c) $\operatorname{\bm{\alpha}}_{2}$

(d) $\beta_{1}$

(e) $\beta_{6}$

Figure 20: (EMNIST without $\mathcal{L}_{\text{R}}$) Visualization of the
learned concept-specific and global variants.