# NeuCEPT: Locally Discover Neural Networks’ Mechanism
via Critical Neurons Identification with Precision Guarantee

Minh N. Vu  Truc D. Nguyen &My T. Thai University of Florida, Gainesville,
Florida, USA<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite recent studies on understanding deep neural networks (DNNs), there
exists numerous questions on how DNNs generate their predictions. Especially,
given similar predictions on different input samples, are the underlying
mechanisms generating those predictions the same? In this work, we propose
NeuCEPT, a method to locally discover critical neurons that play a major role
in the model’s predictions and identify model’s mechanisms in generating those
predictions. We first formulate a critical neurons identification problem as
maximizing a sequence of mutual-information objectives and provide a
theoretical framework to efficiently solve for critical neurons while keeping
the precision under control. NeuCEPT next heuristically learns different
model’s mechanisms in an unsupervised manner. Our experimental results show
that neurons identified by NeuCEPT not only have strong influence on the
model’s predictions but also hold meaningful information about model’s
mechanisms.

##  1 Introduction

Significant efforts have been dedicated to improve the interpretability of
modern neural networks, leading to several advancements Lipton (2018); Murdoch
et al. (2019); however, few works have been conducted to characterize and
analyze local prediction of the neural networks based on the internal
forwarding process of the model (see Related Work). In this paper, we focus on
investigating different mechanisms learnt by the neural networks to generate
predictions (see Fig. 1 as an example). Intuitively, the mechanism of a
prediction is the forwarding process producing the prediction in the examined
model (see concrete definition in Sect. 3). Our hypothesis is that predictions
of the same class label can be generated by different mechanisms which can be
captured and characterized by activation of some specific neurons, called
critical neurons. Analyzing the activation of those neurons can help identify
the model’s mechanisms and shed light on how the model works.

Following are some key reasons motivating our study:

  * •

The identification and study of critical neurons can serve as an initial
model’s examination, which reveals new aspects on the model’s dynamics. For
instance, Fig. 2 shows two LeNet trained on the MNIST digit dataset to
classify even or odd digit. While the models’ outputs and feature-based local
explanations do not suggest significant differences between the two models,
the evidences based on critical neurons clearly illustrate that predictions of
the two models are generated differently. Appendix A.1 provides further
discussion on how critical neurons’ analysis can strengthen the examination of
the layers’ linear separability Alain and Bengio (2017).

  * •

Critical neurons allow us to characterize model’s predictions based on how
they are generated by the model. Each set of similar predictions can be
studied and analyzed for downstream tasks such as performance Everingham et
al. (2009) and trust evaluation Wang et al. (2020). Appendix A.2 provides an
example of our analysis on CheXNet Rajpurkar et al. (2017), a modern DNN which
can detect pneumonia from chest X-rays at a level exceeding practicing
radiologists. It shows how understanding the mechanism can help identify
unreliable predictions.

  * •

Compared to local attribution explanation methods, the study of critical
neurons provide a new dimension on how we explain the predictions.
Specifically, as demonstrated in Appendix A.3, NeuCEPT, our algorithm based on
critical neurons, identifies certain neurons holding vital information in
characterizing model’s predictions of given class labels. Importantly, those
neurons are not among the top-activated neurons of each class and they are not
normally identified by attribution methods due to their low average activation
level.

Figure 1: Two images have the same predictions goldfish generated by
Inception-v3. One is a single goldfish while the other is a shoal of fish. Are
the mechanisms behind the two predictions the same? Figure 2: LeNet(A) is
initialized by a LeNet pretrained on digit classification task while LeNet(B)
is initialized randomly. While the even/odd predictions and the explanations
provide little information differentiating the two models, extracting and
visualizing the activation at the models’ last layer reveal that LeNet(A)
groups inputs into more distinctive clusters with the same ground-truth digit-
labels.

We propose NeuCEPT - a method to locally discover neural network’s mechanism
via critical neurons identification with precision guarantee, as shown in Fig.
3. The innovation of NeuCEPT lies in its two main components: NeuCEPT-
discovery and NeuCEPT-learning, with the following contributions:

  * $\circ$

Critical Neurons Identification with NeuCEPT-discovery (Sect. 2). Under an
information-theoretic point of view, we introduce a layer-by-layer mutual
information objective to discover the critical neurons of model’s predictions.
Intuitively, under this objective, the critical neurons of a layer is the set
of neurons that determine the state of the critical neurons of the sub-
sequence layer, which eventually determine the model’s output. To overcome the
high running-time complexity in solving for this objective, we further develop
a theoretical framework to approximate a set of critical neurons in a
pairwise-parallel manner while keeping the precision guarantee.

  * $\circ$

Information-theoretic interpretation and NeuCEPT-learning of Critical Neurons
(Sect. 3). We provide Information-theoretic interpretation of the critical
neurons and elaborate how learning the mechanism on top of critical neurons
can result in better claims on DNNs’ mechanisms. We propose an unsupervised
learning procedure, called NeuCEPT-learning, to carry out that task.

  * $\circ$

Prior-knowledge Training and Experimental Results (Sect. 4). We propose a new
testing approach, the prior-knowledge training, to experimentally evaluate the
claims on DNNs’ predicting mechanism. In this training, mechanisms are
embedded in the prior-knowledge trained (PKT) model. Our rigorous experimental
results on MNIST and CIFAR-10 show that NeuCEPT is consistently able to detect
the embedded mechanisms in PKT models. Ablation study is also conducted
extensively.

Figure 3: Overall architecture of NeuCEPT. Given a prediction of a class of
interest, NeuCEPT collects the model’s inputs from that class. Then, by
forwarding them through the DNN, NeuCEPT obtains the activation and solves for
the set of critical neurons at some layers. Finally, mechanisms are learnt via
unsupervised learning on those neurons’ activation.

Related Work. Although the problem of identifying DNN’s mechanism is not well-
studied, the related researches of interpretability and neural debugging
received a lot of attention in recent years. Regarding explanation methods,
there are two major approaches: local and global Lipton (2018).

Local methods either search for an interpretable surrogate model to capture
the local behavior of the DNNs Lundberg and Lee (2017); Ribeiro et al. (2016);
Vu and Thai (2020) or carefully back-propagate the model Shrikumar et al.
(2017); Simonyan et al. (2014); Sundararajan et al. (2017) to attribute
contribution scores. The focus of these methods is on identifying features
and, technically can be extended to, neurons with highly-attributed scores;
however, it is unclear how they can be exploited to identify model’s
mechanism. Specifically, high attributed model’s neurons do not necessarily
imply the capability to identify mechanism111Examples in Appendix A.3 show how
highly activated features fail to differentiate predictions with different
mechanisms..

On the other hand, global methods focus on the model’s behavior globally,
using decision trees Hinton and Frosst (2017), decision sets Lakkaraju et al.
(2016) or recursive-partitioning Yang et al. (2018). Unfortunately, there has
been no well-established connection between the explained global dynamics and
the model’s local prediction; hence, the question regarding local prediction’s
mechanism is still unanswered. Among global methods, we find Wang et al.
(2018) to be the most related research to our work. Specifically, the method
uses distillation technique to train another model with sparser data forward
paths and associates each input with one. Even though the paths can be used to
partially reveal the model’s mechanism, they are extracted from the distilled
model and there is no guarantee that they maintain the mechanism behind the
predictions of the original model. Furthermore, the method requires to retrain
the model and also model-specific. NeuCEPT, on the other hand, is applicable
as long as the class data is sufficient and the model’s activation can be
extracted.

##  2 Critical Neurons Identification with NeuCEPT-discovery

Although modern DNNs contain from thousands to millions neurons, there is
evidence that only a small portion of neurons contributes mostly to the
predictions Bau et al. (2020). We denote such neurons as critical neurons of
the predictions or critical neurons for short. Identifying critical neurons
not only reduces the analysis’s complexity but also offers more compact
explanations for model’s mechanism on making its predictions. Unfortunately,
due to the sequential structure of DNNs, identifying such critical neurons is
a daunting task, from formulating a suitable objective function to solving the
problem and interpreting those neurons’ activation.

###  2.1 Problem Formulation

We consider DNNs in classification problems with the forwarding function
$y=f(x)$, where $y\in\mathbb{R}^{m}$ is a logit and $x\in\mathbb{R}^{n}$ is an
input. The neural network has a sequential structure with $L$ layers, each
layer has $k_{l}$ neurons $(l=1,...,L)$. The activation of neurons at any
layers on a given input can be computed by forwarding the model. We denote
this computation as $z_{l}=f_{l}(x)$ where
$f_{l}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{k_{l}}$. Then
$z=[z_{0},...,z_{L}]$ is the activation of all model’s neurons on that input.
We use capital letters to refer to random variables, i.e. $Z_{l}$ is the
random variable representing the activation of neurons at layer $l$. The
superscript notation refers to the random variables associated with a subset
of neurons. For instance, given a subset $\mathcal{S}$ of neurons at layer
$l$, $Z_{l}^{\mathcal{S}}$ is the random activation of the neurons in
$\mathcal{S}$ at layer $l$. Due to the forwarding structure of the model, the
activation of neurons at a given layer depends only on the activation of
neurons at the previous layer, i.e. $Z_{l}\\!\perp\\!\\!\\!\perp
Z_{j}|Z_{l-1},\forall j=0,...,l-2$, where $\\!\perp\\!\\!\\!\perp$ denotes the
independent relationship. Thus, we have the Markov chain:

$\displaystyle X=Z_{0}\rightarrow Z_{1}\rightarrow\cdots\rightarrow Z_{L}.$
(1)

Given a prediction, a layer $l$ and the corresponding random activation
$Z_{l}$, we would like to identify a subset of the critical neurons at that
layer, i.e. the subset of neurons containing the most information on the
prediction of interest. From a information-theoretic approach, we formalize
the notion of criticality using mutual-information (MI) and formulate the
critical neurons identification (CNI) problem as:

$\displaystyle\mathcal{S}_{l}=\textup{argmax}_{\mathcal{S}\subseteq\mathcal{N}_{l}}I\left(Z_{l}^{\mathcal{S}};Z_{l+1}^{\mathcal{S}_{l+1}}\right),\textup{
s.t. }\mathcal{S}\in\mathcal{C}$ (2)

where $\mathcal{N}_{l}$ is the set of neuron at layer $l$, $I$ is the joint
mutual-information function Cover and Thomas (2006) and
$\mathcal{S}\in\mathcal{C}$ represents some complexity constraints imposed on
the problem. The intuition of this formulation is, at each layer, we search
for the set of neurons holding the most information on the set of neurons
solved in the next layer. We denote $o$ as the neuron associated with the
class of prediction and $Y=Z_{L}^{\\{o\\}}$ the corresponding random
activation. By bounding the first optimization at the last layer $L$ to
maximize $I\left(Z_{L-1}^{\mathcal{S}};Y\right)$, we enforce the sub-sequence
optimizations at the earlier layers to solve for the set of neurons holding
the most information on the prediction of interest. Sect. 3.1 provides more
information-theoretic intuition of this objective.

###  2.2 Solutions with Precision Guarantee

As CNI is in NP-hard222Our proof of CNI is in NP-hard is shown in Appendix B,
we introduce an algorithm NeuCEPT-discovery to approximate CNI with precision
guarantee. As an abstract level, NeuCEPT-discovery considers each pair of a
layer’s activation-output of the examined DNN as an input-response pair and
conducts critical neuron selection on them. Different from the sequential
formulation of CNI (2), NeuCEPT-discovery is executed in a pair-wise manner
and, consequently, can be implemented efficiently. Our theoretical results
guarantee that the modification to pair-wise optimization still achieves the
specified precision level.

Using Markov Blanket. Given a random variable $T$, we denote
$\mathcal{M}_{l}(T)\subseteq\mathcal{N}_{l}$ as the smallest set of neurons at
layer $l$ such that, conditionally on the variables in that set -
$Z_{l}^{\mathcal{M}_{l}(T)}$, $T$ is independent of all other variables at
layer $l$. The set $\mathcal{M}_{l}(T)$ is commonly addressed as the Markov
blanket (MB)333There is an ambiguity between the notions of Markov-blanket and
Markov-boundary. We follow the notion of the Markov-blanket defined in Koller
and Friedman (2009), which is consistent with Candes et al. (2016). In fact,
the Markov-blanket of a random variable $T$ in joint distribution ${P}$, i.e.
$\mathcal{M}(T)$, is the minimum set of variables such that, given
realizations of all variables in $\mathcal{M}(T)$, $T$ is conditionally
independent from all other variables. of $T$ in the studies of graphical
models. We make a slight modification by restricting the set of variables to a
certain layer of the model. Under very mild conditions about the joint
distribution of $T$ and $Z_{l}$, the MB is well defined and unique Pearl and
Paz (2014). In this work, we follow researchers in the field, assume these
conditions Edwards (2000) and proceed from there.

Substituting $T$ by ${\displaystyle Z_{l+1}^{\mathcal{S}_{l+1}}}$, we have
${\displaystyle\mathcal{S}=\mathcal{M}_{l}\left(Z_{l+1}^{\mathcal{S}_{l+1}}\right)}$,
the MB at layer $l$ of $Z_{l+1}^{\mathcal{S}_{l+1}}$, achieves the maximum of
the objective (2) since the MB contains all information about
$Z_{l+1}^{\mathcal{S}_{l+1}}$. On the other hand, by the definition of the MB,
${\displaystyle\mathcal{M}_{l}\left(Z_{l+1}^{\mathcal{S}_{l+1}}\right)}$ is
the smallest subset maximizing the MI. This aligns with our intuition that the
MB are important to the activation of critical neurons at later layers. Using
the MB, we have a straight approach to solve (2): Given the activation of
interest at the last layer $Y=Z_{L}^{\\{o\\}}$, we solve for
$\mathcal{M}_{L-1}\left(Y\right)$ \- the MB at layer $L-1$. Then, at layer
$L-2$, we find the MB of the variables in $\mathcal{M}_{L-1}\left(Y\right)$.
The process continues until the first layer is reached, which can be described
as:

$\displaystyle\mathcal{S}_{L-1}\leftarrow\mathcal{M}_{L-1}\left(Y\right),\quad\mathcal{S}_{l-1}\leftarrow\mathcal{M}_{l-1}\left(Z_{l}^{\mathcal{S}_{l}}\right)$
(3)

Controlling the precision. Unfortunately, directly solving (3) is impractical
as the problem is also in NP-hard Margaritis and Thrun (2000). Additionally,
due to the curse-of-dimensionality, sampling enough data to estimate the
distribution of neurons’ activation $Z_{l}^{\mathcal{S}_{l}}$ accurately is
impractical. Our key observation to overcome these challenges is that the MB
of the model’s output variable $Y$ at each layer $l$ is a subset of
$\mathcal{S}_{l}$ defined in procedure (3). As a result, given a solver
solving for $\mathcal{M}_{l}\left(Y\right)$ with a precision at least $p$, the
output of that solver is also an approximation of $\mathcal{S}_{l}$ with the
precision at least $p$. This allows us to solve for
$\mathcal{M}_{l}\left(Y\right)$ instead of $\mathcal{S}_{l}$ and overcome the
high-dimensionality of $Z_{l}^{\mathcal{S}_{l}}$. These key steps of NeuCEPT-
discovery are described in Algorithm 1 (Appendix C). The proof that NeuCEPT-
discovery achieves precision guarantee is based on Theorem 1, which states
$\mathcal{M}_{l}\left(Y\right)$ is a subset of $\mathcal{S}_{l}$:

######  Theorem 1.

Suppose we have a solver solving for the MB of a set of random variables and
we apply that solver at each layer of a neural network as described in
equation (3), then the solution returned by the solver at each layer must
contain the MB of the neural network’s output at that layer, i.e.
$\mathcal{M}_{l}(Y)\subseteq\mathcal{S}_{l},\ \forall l=0,...,L-1$. Proof: See
Appendix D.

From Theorem 1, we obtain Corollary 1, which states any MB solvers with
precision guarantee on the input-response pair $(Z_{l},Y)$ can be used to
solve for the MB of the pair $(Z_{l},Z_{l+1}^{\mathcal{S}_{l+1}})$ (eq. (3))
with the same precision guarantee:

######  Corollary 1.

Suppose we have a solver solving for the MB of a random response $T$ with the
precision at least $p$ for a given $0<p<1$. Let $\hat{\mathcal{M}}_{l}$ be the
output of that solver on the input-response pair $(Z_{l},Y)$ defined in
procedure (3). Then, $\hat{\mathcal{M}}_{l}$ also satisfies the precision
guarantee $p$ as if we solve for the input-response pair
$(Z_{l},Z_{l+1}^{\mathcal{S}_{l+1}})$. Proof: See Appendix E.

Corollary 1 enables us to exploit any solver with precision control to
efficiently solve for procedure (3) with precision guarantee. In our
implementation of NeuCEPT-discovery, we use Model-X Knockoffs Candes et al.
(2016), whose technical aspects are discussed in more details in Appendix F.

##  3 Information-theoretic Interpretation and NeuCEPT-learning of Critical
Neurons

The goal of finding critical neurons is to correctly identify the model’s
mechanisms. Sect. 3.1 discusses in more detail how the MI objective (Eq. (2)
in Sect. 2.1) is apt for the task. Sect. 3.2 describes how NeuCEPT extracts
information from critical neurons to identify the model’s mechanism.

###  3.1 Information-theoretic Interpretation

Mechanism and Explainability power. Previous analysis of DNNs Bau et al.
(2020) and our examples (Figs. 11 and 12) reveal distinctive patterns of
neurons’ activation shared among some input samples. This similarity suggests
they might be processed in the same manner by the model, which is what we call
mechanism. We approach mechanism empirically: similar to how unlabeled data is
handled in unsupervised learning, mechanism is modeled as a discrete latent
random variable whose realization determines how the predictions are
generated. Specifically, Fig. 4 provides an intuition on the relationship
between the neurons’ activation and mechanisms under this assumption. Suppose
the latent mechanism variable $C$ determines the generation of predictions of
the class goldfish in the Inception-v3. Different realizations of $C$, i.e.
$0$ or $1$, result in different patterns in the neurons’ activation $Z$. On
one hand, these patterns specify how the model predicts, which is the
intuitive meaning of mechanism. On the other hand, observing the activation on
some neurons, i.e. critical neurons, can be sufficient to determine the
realization of $C$, i.e. the model’s underlying mechanism444Observing the
input images in this experiment suggests that there might exist a close
connection between the mechanisms and the visual concepts, which are a single
goldfish and a shoal of fish. However, examining this connection is not the
scope of this work.. This imposes a necessary condition on the selection of
critical neurons: their activation should determine (or reduce significantly
the uncertainty of) the mechanism. We call this condition explainability
power.

Figure 4: The probabilistic intuition of the neurons’ activation and
mechanisms: the mechanism determines the activation, and, conversely,
observing the activation can reveal the mechanism.

We can see that the objective (2) fits into the notion of explainability power
by considering its solution
${\displaystyle\left\\{\mathcal{S}_{l}\right\\}_{l=0}^{L-1}}$ described in
(3). From the definition of the MB, for any set of neurons $\mathcal{R}_{l}$
at a layer $l$ that is disjoint with $\mathcal{S}_{l}$, we have
$Z^{\mathcal{R}_{l}}_{l}$ is independent with $Z^{\mathcal{S}_{l+1}}_{l+1}$
given $Z^{\mathcal{S}_{l}}_{l}$. Since $Z^{\mathcal{S}_{l+1}}_{l+1}$
determines $Z^{\mathcal{S}_{l+2}}_{l+2}$, variables in $\mathcal{R}_{l}$ must
also independent with $Z^{\mathcal{S}_{l+2}}_{l+2}$ given
$Z^{\mathcal{S}_{l}}_{l}$. For the same reason, we have variables in
$\mathcal{R}_{l}$ independent with all the later variables that contribute to
the examined prediction; thus, knowing $Z^{\mathcal{R}_{l}}_{l}$ does not
provide any more information on how the model generates the prediction of
interest. Thus, ${\displaystyle\left\\{\mathcal{S}_{l}\right\\}_{l=0}^{L-1}}$
is sufficient.

Non-redundancy. Neurons’ activation not only determine the mechanism but also
contain other distinctive patterns. For example, unsupervised learning on
MNIST’s images, which can be considered as activation of neurons at an input
layer, can classify the data into clusters of digits. However, not all
patterns must be used by the model to generate predictions. Even if the model
learns the patterns, it might exploit them to generate the predictions of
other classes, not the class of our examination. Thus, the capability to
differentiate inputs based on neurons’ activation does not imply the model’s
mechanisms. In short, the non-redundancy requires the identified mechanisms
must be used by the model in generating its prediction. In Appendix G, we
explain how our proposed objective (2) fits for the non-redundancy requirement
by comparing it to another objective solving for globally important neurons.

Figure 5: Examples of NeuCEPT’s outputs on Inception-v3. Images from each row
are from the same cluster learnt by NeuCEPT.

Figure 6: Prior-knowledge training: parameters of the prior-model trained on a
more informative task are transferred to the PKT model, which is later trained
on a less specific task. The mechanisms of the PKT model is expected to be
different from normally-trained models.

Figure 7: CE (bits) resulted from k-mean (KM) and Agglomerative clustering
(AC) clustering on activation of neurons selected by different methods. The
x-axis and y-axis are the number of clusters and the CE, respectively. The
first 2 plots are the analysis at layer conv3 of the LeNet on the class even
using top-10 neurons. The last 4 plots are of VGG on class object with top
30/100 neurons at layer conv8 and conv9.

###  3.2 NeuCEPT-learning on Critical Neurons

Given a set of critical neurons identified by NeuCEPT-discovery, the goal of
NeuCEPT-learning is to extract information from them to identify model’s
mechanisms. Since the ground-truth mechanisms are not given, it is natural to
consider the mechanism identification (or mechanism discovery) problem as an
unsupervised learning task, which has been extensively studied Goodfellow et
al. (2016).

Algorithm 2 (Appendix C) describes how NeuCEPT extracting information from
critical neurons. Besides their activation, the algorithm’s inputs include a
set of compactness parameters limiting the number of representative neurons
and an integer $K$ defining the target number of mechanisms. The usage of the
compactness parameters are commonly used by existing explanation methods for
visualization purpose.

The first step of NeuCEPT-learning imposes the compactness requirement. It
either selects the top relevant neurons identified by NeuCEPT-discovery or
aggregates critical neurons at each layer into a smaller set of representative
neurons using feature agglomeration. Similar techniques have been used to
apply Model-X Knockoffs on real-data with very high-correlated features Candes
et al. (2016). Next, we apply one unsupervised learning method among K-means,
Gaussian Mixture, and Agglomerative Clustering to map each input sample to one
of the $K$ clusters representing $K$ mechanisms.

We end this section with a demonstration of some NeuCEPT’s outputs in
analyzing predictions of Inception-v3. The examined layers are the last layers
of the Mixed-5d and the Mixed-6e blocks (Fig. 5). The number of representative
neurons are restricted to 5 and 3. Next to each input, we show a graph
representing the activation’s level (red for high, blue for low) of those
representative neurons from the Mixed-5d (left) to the Mixed-6e (middle). The
last dot represents the output neuron (right). NeuCEPT-learning helps us
visualize similar activation’s patterns among samples of the same mechanism,
and differentiate them from another mechanism.

Figure 8: Ablation tests on LeNet. The x-axis and y-axis are the noise levels
and the test accuracy. Parameters are specified in Appendix H.3. Figure 9: The
CE differences (bits) between the conventionally trained model and the PKT
model. Clusters are learnt using k-mean (KM), Gaussian Mixture (GM), and
Agglomerative clustering (AC) on activation of neurons chosen by different
methods. The x-axis and y-axis are the number of clusters and the CE
differences. Top figures show the results of the LeNet at different layers
with top 10 important neurons. Bottom are the results of VGG with top 30/100
neurons at convolutional layer 8 and 9.

##  4 Experiments: Setting and Results

Our experiments focus on evaluating the explainability power and the non-
redundancy properties, mentioned in Sect. 3.1. While evaluating the
explainability power can be conducted via ablation study Nguyen et al. (2015),
evaluating the non-redundancy is more challenging as we normally do not know
the underlying mechanism. To tackle this, we propose a training setup, called
prior-knowledge training, so that both the non-redundancy and the
explainability power can be evaluated.

Prior-knowledge training (PKT). The goal of PKT is to inject prior-knowledge
into a model, called PKT model, before its training on its main task, called
posterior-task. Due to the injection, certain mechanisms are expected to be
embedded and used by the PKT model to conduct the posterior-task. Fig. 6
describes the process: First, a model, called prior-model, is trained on a
more specific task, called prior-task, with a more informative dataset. The
prior-model’s parameters are then transferred to the PKT model, which is
subsequently trained on the less specific posterior-task (the main task) on a
less informative dataset. The prior-task’s training data is setup to be more
informative so that there is exclusive information only available to the
prior-model. The prior-task is set to be more specific because we want the PKT
model contains more distinctive mechanisms. In fact, given a PKT model and a
conventionally trained model, a good mechanism discovery algorithm should
conclude that the PKT model relies on some information, i.e. the prior-
knowledge, to generate its prediction (explainability power) while the
conventionally trained one does not (non-redundancy). As such, we evaluate the
algorithms using the clusters’ entropy (CE) metric:

$\displaystyle\sum\nolimits_{c}\sum\nolimits_{y_{prior}}p(y_{prior},c)\log\left({1}/{p(y_{prior}|c)}\right),$

where $c$ is the identified cluster, $y_{prior}$ is the label from the
informative dataset and the $p(.)$ is the empirical probability computed on
the data of testing. Intuitively, the lower the CE, the more the
clusters/mechanisms identified by the algorithm align with the prior-
knowledge. Thus, the CE of PKT model resulted from a good mechanism discovery
algorithm should be lower than that of a conventionally trained model. The
details of our experiments, i.e. baseline and implementations, are specified
in Appendix H.

Explainability power. The explainability power implies the selected neurons
should hold information to determine the mechanism. Thus, the CE should be
small if the number of the clusters $K$ (specified in Sect. 3.2) aligns with
the number of actual mechanisms. In the posterior-tasks, we train LeNet and
VGG to classify even/odd on MNIST and animal/object on CIFAR10, respectively.
The numbers of actual mechanisms in the PKT models are expected to be at least
$(5,5)$ and $(4,6)$ for each posterior-tasks’ label, as they are the number of
original labels belonging to those category, i.e. $5$ odd digits.

Fig. 7 shows the CE of the clusters learnt on activation of neurons identified
by explanation methods at different $K$. Experiments on MNIST clearly show
that neurons identified by NeuCEPT can differentiate the inputs based on their
original labels, whose information is embedded in prior-knowledge training.
Notably, when $K=5$, NeuCEPT achieves its lowest, which aligns with our
expectation on the number of actual mechanisms that the PKT model uses.
Similar results can also be observed in VGG, but with less distinction in the
number of clusters. Since the dataset and the models are both much more
complicated, an explanation of this behavior is that the number of actual
mechanisms might be much larger than that in previous experiments.
Additionally, Fig. 8 shows our ablation tests of neurons identified by
different methods on LeNet. The results show that neurons identified by
NeuCEPT hold higher predictive power among most of the experiments, which
implies their strong explainability power. Further ablation results and
details are reported in Appendix H.3.

Non redundancy. This set of experiments shows that NeuCEPT does not return the
same CE when analyzing a conventionally trained model, which has no knowledge
on the original labels. Fig. 9 plots the differences in the CE between a
conventionally-trained model and the PKT model on both MNIST and CIFAR10.
While the differences in CE at certain layers of some other methods fluctuate
around $0$, indicating there is no difference between the two models, NeuCEPT
consistently supports the expectation that only the PKT model recognizes the
prior-knowledge. Note that NeuCEPT can differentiate the two models trained
differently simply by observing their activation while other methods fail to
do so (see the 3 top-left plots of Fig. 9 as examples).

##  5 Conclusion

In this paper, we attempt to discover different mechanisms underlying DNNs’
predictions in order to provide a deeper explanation on how a model works.
Based on an information-theoretic viewpoint, we formulate the problem as a
sequence of MI maximization. The solution of the maximization, called critical
neurons, can be solved by our proposed algorithm NeuCEPT-discovery with
precision guarantee. Then, we propose NeuCEPT-learning, an algorithm
clustering inputs based on their activation on critical neurons so that the
underlying model’s mechanism can be revealed. We also propose a prior-
knowledge training procedure so that the mechanism discovery task can be
evaluated. Our experimental results show that NeuCEPT is able to consistently
identify the underlying mechanism of DNNs’ predictions.

## References

  * Alain and Bengio [2017] G. Alain and Y. Bengio.  Understanding intermediate layers using linear classifier probes.  ICLR, 2017.
  * Bau et al. [2020] D. Bau, J. Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba.  Understanding the role of individual units in a deep neural network.  Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020.
  * Candes et al. [2016] E. Candes, Y. Fan, L. Janson, and J. Lv.  Panning for gold: Model-x knockoffs for high-dimensional controlled variable selection.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2016.
  * Cover and Thomas [2006] T. M. Cover and J. A. Thomas.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing).  Wiley-Interscience, USA, 2006.
  * Davies and Russell [1994] S. Davies and S. Russell.  Np-completeness of searches for smallest possible feature sets.  In AAAI Symposium on Intelligent Relevance. AAAI Press, 1994.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * Edwards [2000] D. Edwards.  Introduction to Graphical Modelling.  Springer, New York, USA, 2000.
  * Everingham et al. [2009] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88:303–338, 2009.
  * Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville.  Deep Learning.  MIT Press, 2016.
  * Hinton and Frosst [2017] G. Hinton and N. Frosst.  Distilling a neural network into a soft decision tree.  2017\.
  * Khaire and Dhanalakshmi [2019] U. M. Khaire and R. Dhanalakshmi.  Stability of feature selection algorithm: A review.  Journal of King Saud University - Computer and Information Sciences, 2019.
  * Kok [2020] Captum: A unified and generic model interpretability library for pytorch.  ArXiv, abs/2009.07896, 2020.
  * Koller and Friedman [2009] D. Koller and N. Friedman.  Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning.  The MIT Press, 2009.
  * Krizhevsky [2012] A. Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * Lakkaraju et al. [2016] H. Lakkaraju, S. H. Bach, and J. Leskovec.  Interpretable decision sets: A joint framework for description and prediction.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675–1684, 2016.
  * LeCun and Cortes [2010] Y. LeCun and C. Cortes.  MNIST handwritten digit database.  2010\.
  * Lecun et al. [1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * Lipton [2018] Z. C. Lipton.  The mythos of model interpretability.  Queue, 16(3):31–57, June 2018.
  * Liu and Deng [2015] S. Liu and W. Deng.  Very deep convolutional neural network based image classification using small training sample size.  2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pages 730–734, 2015.
  * Lundberg and Lee [2017] S. M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In Advances in Neural Information Processing Systems 30. 2017.
  * Margaritis and Thrun [2000] D. Margaritis and S. Thrun.  Bayesian network induction via local neighborhoods.  In Advances in Neural Information Processing Systems 12, pages 505–511. 2000.
  * Murdoch et al. [2019] W. J. Murdoch, C. Singh, K. Kumbier, Reza Abbasi-Asl, and B. Yu.  Definitions, methods, and applications in interpretable machine learning.  Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019.
  * Nguyen et al. [2015] A. M. Nguyen, J. Yosinski, and J. Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * Paszke et al. [2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * Pearl and Paz [2014] Judea Pearl and Azaria Paz.  Confounding equivalence in causal inference.  Journal of Causal Inference, 2(1):75–93, 2014.
  * Rajpurkar et al. [2017] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Y. Ding, A. Bagul, C. Langlotz, K. S. Shpanskaya, M. P. Lungren, and A. Y. Ng.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  CoRR, abs/1711.05225, 2017.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  “Why should i trust you?”: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
  * Shrikumar et al. [2017] A. Shrikumar, P. Greenside, and A. Kundaje.  Learning important features through propagating activation differences.  In Proceedings of the 34th International Conference on Machine Learning, 2017.
  * Simonyan et al. [2014] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In Workshop at International Conference on Learning Representations, 2014.
  * Sundararajan et al. [2017] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, 2017.
  * Szegedy et al. [2016] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, June 2016.
  * Vu and Thai [2020] M. Vu and My T. Thai.  PGM-Explainer: Probabilistic graphical model explanations for graph neural networks.  In Advances in Neural Information Processing Systems, 2020.
  * Wang et al. [2018] Y. Wang, H. Su, B. Zhang, and X. Hu.  Interpret neural networks by identifying critical data routing paths.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8906–8914, 2018.
  * Wang et al. [2020] J. Wang, X. Jing, Z. Yan, Y. Fu, W. Pedrycz, and L. T. Yang.  A survey on trust evaluation based on machine learning.  ACM Computing Surveys (CSUR), 53, 2020.
  * Yang et al. [2018] C. Yang, A. Rangarajan, and S. Ranka.  Global model interpretation via recursive partitioning.  In 2018 IEEE 20th International Conference on High Performance Computing and Communications, pages 1563–1570, 2018.

##  Appendix A Illustrative examples for motivations

This appendix contain illustrative examples showing the benefits of
identifying critical neurons and obtaining knowledge on the model’s
mechanisms. First, we show how our research can contribute to the study of
model’s linear separability (Sect. A.1). Then, we provide illustrative example
showing how knowledge on the model’s mechanisms help us re-evaluate the
model’s outputs (Sect. A.2). We also demonstrate that simply selecting class-
highly activated neurons, which can be considered as a simple feature-
attribution method, would miss some neurons holding valuable information on
the model’s mechanisms (Sect. A.3).

###  A.1 Example on Linear Probe

Linear Probes Alain and Bengio [2017] are linear classifiers hooking on
intermediate layers of neural networks to measure their linear separability.
Intuitively, lower the loss of the classifier at a layer suggests that the
activation of neurons in that layer is more linearly separable. This metric is
important and interesting in the analysis of neural networks since it can be
used to characterize layers, to debug models, or to monitor training’s
progress.

Figure 10: Simply using linear probes on the predicted labels, i.e. even/odd,
does not reveal the difference in the linear separability of LeNetA and LeNetB
(indicated by notation (2)). However, using the knowledge on the original
labels of the dataset, linear probes show that the linear separability of the
two models are difference (indicated by notation (10)).

We applied Linear Probes to study LeNet(A) and LeNet(B) (Fig. 2) on their
even/odd labels and plotted the results in Fig. 10 with notation LeNetA(2) and
LeNetB(2), respectively. The result shows that there is little difference
between the linear separability between layers of the two models. However, as
shown in the visualization of the point-clouds identified by critical neurons
(bottom Fig. 2), we expect that the degree of linear separability of the two
model should be different. Specifically, LeNet(A) is expected to be more
linearly separable since its point-cloud is more separated into smaller
clusters. Indeed, Fig. 10 also demonstrates if we use Linear Probes to measure
how well the layers’ activation can predict the original digit labels of the
dataset, there is a clear difference in the corresponding results (indicated
by notation LeNetA(10) and LeNetB(10)).

The results indicate two major meanings. First, it shows that information
obtained by NeuCEPT is consistent with Linear Probe, i.e. they both suggest
LeNet(A) is more linearly separable, and they can be used to complement each
other. Secondly, in practice, we normally do not have the labels on the
model’s mechanisms; thus, the gap like that between LeNetA(10) and LeNetB(10)
cannot be obtained by Linear Probe. This means the linear separability might
not be captured correctly by Linear Probe (as shown in the case of LeNetA(2)
and LeNetB(2)). In that situation, NeuCEPT can be used to ensure the results
obtained by Linear Probe or further the study of linear separability.

Experiment’s info
---
Model | CheXNet
Examined predictions | ’Pneumonia’
Total samples | 2978
Positive samples | 1035
Positive predictions | 1035
Model’s precision | $60.6\%$
| Reliable-subclass
---
precision
| $91.3\%$
---
(180 samples)
| Unreliable-subclass
---
precision
| $36.7\%$
---
(150 samples)

Figure 11: NeuCEPT discovers a subclass of unreliable predictions with
precision of $36.7\%$ (compared to $60.6\%$ on the whole class) in CheXNet. We
also extract the explanation heat-maps provided by the model which suggest
that the unreliable positive predictions are made by the area at the boundary
of the image instead of the lung’s area (red means high importance score).

(a) Analysis on class goldfish at the Mixed6e block of Inception-v3 indicates
neuron 322 holds valuable information in differentiating predictions of two
sub-classes (red and green).

(b) Analysis on class bee at the Mixed6e block of Inception-v3 indicates
neuron 305 holds valuable information in differentiating predictions of two
sub-classes (blue and red).

Figure 12: Results obtained by applying NeuCEPT with the number of clusters
$K=4$ on classes goldfish and bee at the last layer of the Mixed6e block of
Inception-v3. For each class, NeuCEPT identifies a neuron whose activation’s
histogram on two distinctive clusters (among four) learnt by NeuCEPT and all
inputs are shown on the top-right figure. While these neurons are not among
the top-activated neurons of each class, they hold vital information in
differentiating the predictions as inputs of one cluster activate the neuron
and the others do not. The top-left and the bottom figures plot the low-
dimensional representations of the inputs and some images drawn from each
cluster, respectively, for visualization.

###  A.2 Example on CheXNet

In examining CheXNet Rajpurkar et al. [2017] using NeuCEPT, we discover a
subclass of unreliable ’pneumonia’ predictions with a much lower precision
than that on the whole class, i.e. $36.7\%$ compared to $60.6\%$. The results
are described in Fig. 11. Similar unreliability’s behavior is also observed in
previous works555In the work ”Variable generalization performance of a deep
learning model to detect pneumonia in chest radio-graphs: A cross-sectional
study”, PLoS Medicine (2018), Zech et al. obtained activation heat-maps from a
trained CheXNet and averaged them over a set of inputs to reveal which sub-
regions tended to contribute to a classification decision. They found out that
the model has learned to detect a hospital-specific metal token on the scan to
generate some classifications.. Furthermore, same conclusion can be deducted
based on the local explanations provided by the model itself, i.e. the false-
positive predictions generally are made using features outside of the lung’s
area as shown in the figure.

This example clearly shows that end-users need systematical methods to
differentiate reliable predictions from unreliable ones. Identifying
mechanisms underlying the model’s predictions is therefore vital since it can
help divide predictions into groups of similar mechanisms. Based on the
mechanisms, researchers and practitioners can systematically evaluate the
reliability of samples. Note that, without this information, we might need to
discard the model and cannot exploit its high predictive power due to some
unreliable predictions.

###  A.3 Example on Inception-v3

As the main goal of attribution methods is to identify neurons contributing
the most to the prediction, aggregating the resulted attribution scores among
inputs of a class naturally gives us the neurons highly attributing to the
class. However, as the activation of those neurons can be highly similar for
almost all samples of the class (one trivial example is the output’s neuron
associated with the class), they hold limited information on how the model
processes those inputs differently, hence, do not fit for the task of
identifying mechanisms.

To illustrate this point, Fig. 12 provides our analysis on the class goldfish
and bee of Inception-v3 Szegedy et al. [2016]. At the last layer of the
Mixed6e block of the model, for each class, we identify a neuron (using
NeuCEPT) that is not among the top highly activated of the class, but hold
valuable information in determining how the model views inputs of the class
differently. Specifically, as shown in the figure, there are two subsets of
inputs of the examined class such that inputs in one subset activate the
neuron while the others do not. Simply using activation’s level, which can be
considered as the simplest form of attribution method, would miss this neuron.

An interesting observation is that, in both classes, there are distinctive
visual differences in the images drawn from the clusters identified by
NeuCEPT. In examples of goldfish, which has been used throughout the main
manuscript, samples in one cluster show a single fish while those in the other
show a shoal of fishes. On the other hand, in the class bee, samples of a
cluster show a single bee while the samples in the other mainly focus on a
huge flower or a bunch of flowers. This observation supports the hypothesis
that the two subsets of images are indeed processed differently by the model.

##  Appendix B CNI is in NP-hard

The CNI can be considered as a more general version of the feature-selection
problem with mutual-information objective, which is known to be NP-hard Khaire
and Dhanalakshmi [2019]. However, we find that there is no recent works
concretely prove the NP-hard result of this feature-selection problem. Hence,
for the completeness of our result, we provide the proof that the CNI is in
NP-hard in the followings.

We first consider the discrete version of the CNI optimization problem in (2),
called DISCRETE-CNI, as follows: Given a random variable $Z$ over the set of
binary sequences $\\{0,1\\}^{N}$, a discrete random variable $Y$ and a
constant integer $k,1\leq k\leq N$, we want to find the set of indices
$\mathcal{S}^{*}$ such that:

$\displaystyle\mathcal{S}^{*}=\textup{argmax}_{\mathcal{S}\subseteq\\{1,2,...,N\\}}I\left(Z^{\mathcal{S}};Y\right),\textup{
s.t. }|\mathcal{S}|\leq k,$ (4)

where $Z^{\mathcal{S}}$ indicates the set of components $i$ of $Z$ such that
$i\in\mathcal{S}$. It is clear that this discrete version can be reduced to
the original CNI problem in (2) in polynomial time.

We show that DISCRETE-CNI is in NP-hard by reducing MIN-FEATURES Davies and
Russell [1994] to DISCRETE-CNI. The MIN-FEATURES is defined as follows: Given
a set $X\subseteq\\{0,1\\}^{N^{\prime}}\times T$ of samples, whose the first
$N^{\prime}$ components called features and the last component called target,
and an integer $n$, determine whether or not there exists some feature set
$\mathcal{V}$ such that:

  * •

$\mathcal{V}\subseteq\\{1,2,...,N^{\prime}\\}$

  * •

$|\mathcal{V}|\leq n$

  * •

There exists no two examples in $X$ that have identical values for all the
features in $\mathcal{V}$ but have different values for the target.

Note that we only consider the case where the target $T$ can always be
determined by the $N^{\prime}$ features. If that is not the case, the problem
is trivial since there is no such $\mathcal{V}$.

To reduce MIN-FEATURES to DISCRETE-CNI, we define the random variable $Z$ over
$\\{0,1\\}^{N^{\prime}}$ of DISCRETE-CNI as follows:

$\displaystyle\textup{Pr}\\{Z=x\\}=\left\\{\begin{matrix}1/|X|,\ &\forall x\in
X[1:N^{\prime}]\\\ 0,&\textup{ otherwise }.\end{matrix}\right.$

where $X[1:N^{\prime}]$ is the set of samples of features of $X$, i.e. the set
$X$ but without the target.

We now can reduce MIN-FEATURES by running DISCRETE-CNI on $Z$ as constructed
above, $Y$ as the target in $X$ and the cardinality constraint constant $k$ as
$n$. We then decide the existence of $\mathcal{V}$ by checking whether or not
the solution $\mathcal{S}^{*}$ returned by DISCRETE-CNI satisfying
$I\left(Z^{\mathcal{S^{*}}};Y\right)$ equals to the entropy of $Y$.
Specifically, if $I\left(Z^{\mathcal{S^{*}}};Y\right)=H(Y)$, then
$Z^{\mathcal{S*}}$ determines $Y$ and $\mathcal{S^{*}}$ satisfies the
conditions of MIN-FEATURES. On the other hand, if
$I\left(Z^{\mathcal{S^{*}}};Y\right)<H(Y)$, then observing features in
$\mathcal{S^{*}}$ does not determine $Y$. This implies there must exist at
least two feature-samples of $X[1:N^{\prime}]$ such that their features in
$\mathcal{S}^{*}$ are the same but their targets are different. Thus, there is
no $\mathcal{V}$ satisfying MIN-FEATURES because the existence of such
$\mathcal{V}$ means $\mathcal{S}^{*}$ is not the optimal of DISCRETE-CNI.

##  Appendix C Descriptions of NeuCEPT’s algorithms

In this appendix, we provide the descriptions of NeuCEPT-discovery (Alg. 1)
and NeuCEPT-learning (Alg. 2).

Input : Samples of model’s activation $Z=(Z_{1},...,Z_{M})$ at $M$ given
layers.

A set of precision thresholds $p=(p_{1},...,p_{M})$ for those chosen layers.

Output : Estimation of critical neurons at all examined layers
$\left\\{\hat{\mathcal{M}_{l}}\right\\}_{l=1}^{M}$.

1 $Y\leftarrow Z^{\\{o\\}}_{M+1}$

2 For $l=1$ to $M$ do:

3 $\hat{\mathcal{M}_{l}}\leftarrow$ estimation of the Markov blanket of $Y$ at
layer $l$ with precision control $p_{l}$.

4 End

Return $\left\\{\hat{\mathcal{M}_{l}}\right\\}_{l=1}^{M}$.

Critical Neurons Identification.

Algorithm 1 NeuCEPT-discovery:

Critical Neurons Identification.

Input : Critical neurons’ activation at $M$ examined layers, denoted as
$Z^{\mathcal{S}}$.

A set of values limiting the number representative neurons
$v=(v_{1},\cdots,v_{M})$.

A positive integer $K$ guessing the number of clusters/mechanisms.

Output : The explained-representations of all inputs and their corresponding
mechanisms/clusters.

1 $\\#$ Constraints enforcement

2 If $v\neq$ None:

3 For $l=1$ to $M$ do:

4 $V_{l}\leftarrow$ Feature Agglomeration ($Z_{l}^{\mathcal{S}_{l}}$) with
constraint $|V_{l}|\leq v_{l}$

5 End

6 Else:

7 $V\leftarrow$ $Z^{\mathcal{S}}$.

8 $\\#$ Unsupervised learning

9 $g\leftarrow$ Initialize an unsupervised model with $K$ components

10 Fit $g$ on $V=(V_{0},\cdots,V_{L-1})$

11 Explained-representation $E\leftarrow$ transformation of $V$ by $g$

12 Mechanism $C\leftarrow$ prediction of $V$ by $g$

13 Return $E$ and $C$.

Mechanism discovery.

Algorithm 2 NeuCEPT-learning:

Mechanism discovery.

##  Appendix D Proof of Theorem 1

For simplicity, we consider the following Markov chain $Z_{0}\rightarrow
Z_{1}\rightarrow Y$. We now show that
$\mathcal{M}_{0}(Y)\subseteq\mathcal{M}_{0}\left(Z_{1}^{\mathcal{M}_{1}(Y)}\right)$.
We have:

  * •

$Z_{1}$ determines $Y$ and $Y\\!\perp\\!\\!\\!\perp\left\\{Z_{1}\setminus
Z_{1}^{\mathcal{M}_{1}(Y)}\right\\}|Z_{1}^{\mathcal{M}_{1}(Y)}$ so that
$Z_{1}^{\mathcal{M}_{1}(Y)}$ determines $Y$. We can also write this conclusion
as $Z_{1}^{\mathcal{S}_{1}}$ determines $Y$.

  * •

$Z_{0}$ determines $Z_{1}^{\mathcal{M}_{1}(Y)}$ and
$Z_{1}^{\mathcal{M}_{1}(Y)}\\!\perp\\!\\!\\!\perp\left\\{Z_{0}\setminus
Z_{0}^{\mathcal{M}_{0}\left(Z_{1}^{\mathcal{M}_{1}(Y)}\right)}\right\\}|Z_{0}^{\mathcal{M}_{0}\left(Z_{1}^{\mathcal{M}_{1}(Y)}\right)}$
so that $Z_{0}^{\mathcal{M}_{0}\left(Z_{1}^{\mathcal{M}_{1}(Y)}\right)}$
determines $Z_{1}^{\mathcal{M}_{1}(Y)}$. Similar to the previous statement, we
can write this conclusion as $Z_{0}^{\mathcal{S}_{0}}$ determines
$Z_{1}^{\mathcal{S}_{1}}$.

  * •

Combine the two above statements, we have
$Z_{0}^{\mathcal{M}_{0}\left(Z_{1}^{\mathcal{M}_{1}(Y)}\right)}$ determines
$Y$.

On the other hand, we have $\mathcal{M}_{1}(Y)$ is the smallest subset of
neurons at the $Z_{0}$ layer that determines $Y$. Due to the uniqueness of the
minimal set that separates $Y$ from the rest of the variables (which is the
Markov blanket of $Y$) Pearl and Paz [2014], we have
$\mathcal{M}_{1}(Y)\subseteq\mathcal{M}_{1}\left(Z_{2}^{\mathcal{M}_{2}(Y)}\right)$.

We can see that the proof generalizes for the case of $L$ layers Markov chain
$Z_{0}\rightarrow Z_{1}\rightarrow\cdots\rightarrow Z_{L}$ as we can apply the
same arguments to conclude that $Z_{l}^{\mathcal{S}_{l}}$ determines
$Z_{l+1}^{\mathcal{S}_{l+1}}$. This would lead to the fact that all
$Z_{l}^{\mathcal{S}_{l}}$ can determine $Y$; hence, $\mathcal{S}_{l}$ contains
$\mathcal{M}_{l}(Y)$ due to the uniqueness of the Markov blanket Pearl and Paz
[2014].

##  Appendix E Proof of Corollary 1

Denote $q=1-p$. Since the precision is one minus the false-discovery-rate
(FDR), to show the Corollary, we need to prove:

$\displaystyle\textup{FDR}\vcentcolon=\mathbb{E}\left[\frac{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{S}_{l}\\}}{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\\}}\right]\leq
q$

From Theorem 1, we have $\mathcal{M}_{l}(Y)\subseteq\mathcal{S}_{l}$ for all
$l=0,\cdots,L-1$. This implies:

$\displaystyle\hat{\mathcal{M}}_{l}\setminus\mathcal{S}_{l}\subseteq\hat{\mathcal{M}}_{l}\setminus\mathcal{M}_{l}(Y)$
$\displaystyle\Longrightarrow$
$\displaystyle\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{S}_{l}\\}\leq\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{M}_{l}(Y)\\}$
(5)

On the other hand, as $\hat{\mathcal{M}}_{l}$ is the solution of the solver on
the input-response pair $(Z_{l},Y)$ with FDR less than or equal to $q$, we
have

$\displaystyle\mathbb{E}\left[\frac{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{M}_{l}(Y)\\}}{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\\}}\right]\leq
q.$ (6)

Combining (5) and (6), we have the Corollary:

$\displaystyle\mathbb{E}\left[\frac{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{S}_{l}\\}}{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\\}}\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\frac{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\setminus\mathcal{M}_{l}(Y)\\}}{\\#\\{j:j\in\hat{\mathcal{M}}_{l}\\}}\right]\leq
q.$ (7)

##  Appendix F Model-X Knockoffs

Background. Model-X Knockoffs Candes et al. [2016] is a new statistical tool
investigating the relationship between a response of interest and a large set
of explanatory variables. From a set of hundreds or thousands variables,
Model-X Knockoffs can identify a smaller subset potentially explaining the
response while rigorously controlling the false discovery rate (FDR).
Specifically, Model-X Knockoffs considers a very general conditional model,
where the response $T$ can depend in an arbitrary fashion on the covariates
$R=(R_{1},\cdots,R_{p})$. The method assumes no knowledge of the conditional
distribution of $T|R$ but it assumes the joint distribution of the covariates
$R$ is known. The goal of Model-X Knockoffs is to find important components of
$R$ that affect $T$, i.e. the members of the Markov blanket for $T$, while
controlling a Type I error. In other words, the method aims to select as many
variables as possible while limiting the false positives. Denoting
$\mathcal{R}$ and $\hat{\mathcal{R}}$ the Markov blanket and the outcome of
the Model-X Knockoffs procedure, the FDR is:

$\displaystyle\textup{FDR}=\mathbb{E}\left[\frac{\\#\\{j:j\in\hat{\mathcal{R}}\setminus\mathcal{R}\\}}{\\#\\{j:j\in\hat{\mathcal{R}}\\}}\right],$
(8)

with the convention $0/0=0$. For each sample of ${R}$, Model-X Knockoffs
generates a knockoff copy $\tilde{R}$ satisfying
$Y\\!\perp\\!\\!\\!\perp\tilde{R}|R$ and the pairwise exchangeable property
Candes et al. [2016]. Then, some importance measure $U_{j}$ and
$\tilde{U}_{j}$ are computed for each $R_{j}$ and $\tilde{R}_{j}$,
respectively. After that, the statistics $W_{j}=U_{j}-\tilde{U}_{j}$ is
evaluated for each feature where a large and positive value of $W_{j}$ implies
evidence against the hypothesis that the $j^{\textup{th}}$ feature is not in
the Markov blanket of $T$. The works Candes et al. [2016] have shown that
exact control of the FDR below the nominal level $q$ can be obtained by
selecting $\hat{\mathcal{R}}=\\{j:W_{j}\geq\tau_{q}\\}$, with

$\displaystyle\tau_{q}=\min\left\\{t>0:\frac{1+|\\{j:W_{j}\leq
t\\}|}{|\\{j:W_{j}\geq t\\}|}\leq q\right\\},$ (9)

where the numerator in the above expression can be considered as a
conservative estimate of the number of false positives above the fixed level
$t$. In this manuscript, we use the tuple $(.,.)$ to denote the input-response
pair for Model-X Knockoffs as a general solver for the Markov blanket. For
example, the input-response pair of the formulation above is $(R,T)$.

Technical implementation. There are several concerns and details the usage of
Model-X Knockoffs on high-dimensional real-data and, specifically, on neurons’
activations. One of them is the problem that it is very difficult to choose
between two or more very highly-correlated features. Note that ”this is purely
a problem of power and would not affect the Type I error control of knockoffs”
Candes et al. [2016]. To overcome this challenge, Candes et al. clustered
features using estimated correlations as a similarity measure, and discovered
important clusters. After that, one representative is chosen from each
cluster. The representatives are then used in the feature-discovery task
instead of the original features. Our feature agglomeration step (described in
Sect. 3 and Algo. 2) is based on this alleviation.

Another concern on the usage of Model-X Knockoffs is an unavailability of the
true joint covariate distribution, which forces the user to estimate it from
the data. The usage of NeuCEPT falls into this case since the neuron’s
activation variable $Z$ can only be explored using available input samples.
This raises the question on how robust NeuCEPT is to this error introduced by
Model-X Knockoffs. Theoretical guarantees of robustness are not the scope of
this paper; however, Candes et al. has extensively provided testings on both
synthetic and real-world data to evaluate the robustness of Model-X Knockoffs
in their work. The main observations are the FDR is rarely ever violated even
for a loosely estimated covariance, and the power of selected features only
lost about $20\%$ as if they would have had with an exact covariance.

##  Appendix G Discussion on the non-redundancy

To demonstrate how the optimal solution of (2), $\mathcal{S}_{l}$, which is
approximated by NeuCEPT-discovery, meets the notion of non-redundancy, we
compare it with solution of another objective aiming to identify the set of
globally important neurons in a given layer $l$. Intuitively, they are more
about all the predictions of the model than some specific predictions. From
the Markov chain (1), a natural selection of those globally important neurons
can be formalized as:

$\displaystyle\mathcal{S}^{*}_{l}=\textup{argmax}_{\mathcal{S}\subseteq\mathcal{N}_{l}}I\left(Z_{l}^{\mathcal{S}};Z_{l+1},...,Z_{L}\right),\textup{
s.t. }\mathcal{S}\in\mathcal{C}$ (10)

where the notations have the same meaning as those in equation (2). This
objective tells us how much we know about the model’s later layers given the
activation of the neurons in $\mathcal{S}\subseteq\mathcal{N}_{l}$. We denote
$\Phi_{0}(Z)$ and $\Phi_{1}(Z)$ as the sums of the objectives (10) and (2),
respectively. We then naturally have $\Phi_{0}(Z)\geq\Phi_{1}(Z)$. From the
lens of information theory, $\Phi_{0}(Z)$ is the sum of the minimum
description length of the model’s layers, i.e. information encoded in those
neurons can fully specify the model as well as its layers. However, when
examining the prediction on a specific class, the information in
${\displaystyle\left\\{\mathcal{S}^{*}_{l}\right\\}_{l=0}^{L-1}}$ is too
excessive since the objective does not consider the classes differently.
Therefore, mechanisms discovered on top of them is prone to redundancy. This
reason motivates us to consider $\mathcal{S}_{l}$ and the objective (2) for
our study. This also implies that the critical neuron discovery step in
NeuCEPT is necessary.

Dataset | # of labels | Input size | # of Training data | Model | # Model’s output | # of Parameters total / trainable
---|---|---|---|---|---|---
MNIST | 10 | 28 x 28 x 1 | 60,000 | LeNet prior | 10 | 61706 / 61706
LeNet normal |  2 (even/odd) | 61026 / 61026
LeNet PKT |  2 (even/odd) | 61026 / 61026
CIFAR-10 | 10 | 32 x 32 x 3 | 50,000 | VGG11 prior | 10 | 2976938 / 2976938
VGG11 normal |  2 (animal/object) | 2974882 / 2974882
VGG11 PKT |  2 (animal/object) | 2974882 / 1772290

Table 1: Information of dataset and models used in our experiments (Sect. 4). All models trained on MNIST and CIFAR-10 achieve at least 98% and 91% test-set accuracy, respectively. The number of trainable parameters in VGG11 PKT is not all the parameters due to prior-knowledge training. Model | Layer | NeuCEPT | Saliency | IntegratedGradients | DeepLift | GradientSHAP
---|---|---|---|---|---|---
LeNet prior | conv3 | 3.18 | 17.35 | 79.59 | 51.32 | 31.42
linear0 | 1.93 | 16.25 | 79.20 | 20.12 | 31.13
LeNet normal | conv3 | 2.86 | 16.52 | 79.95 | 51.22 | 31.03
linear0 | 2.00 | 15.47 | 77.57 | 19.93 | 31.08
LeNet PKT | conv3 | 2.76 | 16.00 | 80.46 | 50.49 | 31.93
linear0 | 1.94 | 15.53 | 77.13 | 20.05 | 30.77
VGG11 prior | conv8 | 20.86 | 167.82 | 379.16 | 195.01 | 355.65
conv9 | 22.36 | 180.6 | 362.28 | 183.97 | 305.65
VGG11 normal | conv8 | 26.12 | 307.74 | 709.38 | 335.29 | 708.50
conv9 | 23.22 | 307.87 | 708.71 | 309.38 | 707.38
VGG11 PKT | conv8 | 17.26 | 223.24 | 457.78 | 263.63 | 457.09
conv9 | 18.80 | 222.50 | 457.81 | 220.53 | 457.32
Table 2: The running-time (in seconds) of tested methods. For the 4 other
explanation methods, we use the Captum library which runs on GPUs. Our NeuCEPT
runs entirely on CPU cores and the reported running-time is for one iteration.
For more stable results, the experimental results of NeuCEPT are obtained
after 50 iterations. Figure 13: Ablation test of VGG. The x-axis and y-axis
are the noise levels and the test accuracy, respectively. Figure 14: Average
and all classes’ ablation results at layer conv3 of LeNet. Figure 15: Average
and all classes’ ablation results at layer linear0 of LeNet. Figure 16:
Average and all classes’ ablation results at layer conv8 of VGG. Figure 17:
Average and all classes’ ablation results at layer conv9 of VGG. Figure 18:
Ablation results at input layer of LeNet. The two plots on the left show the
ablation results with continuous noise, i.e. the noise are added based on the
score given by the associated methods. The three figures on the right shows
results when top 100 features are selected. $\gamma$ is an exponential decay
parameter determining how noise is distributed among neurons.

##  Appendix H Experiments: settings and implementations

Our experiments in this paper are implemented using Python 3.8 and conducted
on a single GPU-assisted compute node that is installed with a Linux 64-bit
operating system. The allocated resources include 32 CPU cores (AMD EPYC 7742
model) with 2 threads per core, and 100GB of RAM. The node is also equipped
with 8 GPUs (NVIDIA DGX A100 SuperPod model), with 80GB of memory per GPU.

###  H.1 Model and dataset

We experiment with LeNet Lecun et al. [1998] and the VGG Liu and Deng [2015]
trained on the MNIST LeCun and Cortes [2010] and CIFAR10 Krizhevsky [2012]
dataset, respectively. The details of datasets and model architectures are
listed in Table 1. For LeNet, we use the default architecture provided by
Pytorch Paszke et al. [2017] with only the modification at the last layer.
Specifically, we change the last layer to 2 neurons so that the model can be
suitable to the even/odd classification posterior-task. In CIFAR-10, we
implement the VGG11s using Pytorch along with the modification at the last
layer to fit to the animal/object classification task, i.e. set the number of
neurons at the last layer to 2.

Our experiments on Inception-v3 Szegedy et al. [2016] and CheXNet Rajpurkar et
al. [2017] are for visualization and demonstration of NeuCEPT on a wider range
of models and dataset. We use the pre-trained Inception-v3 provided by Pytorch
and test the model on the Imagenet dataset Deng et al. [2009]. The CheXNet is
provided by the original paper Rajpurkar et al. [2017].

###  H.2 Implementation details

Freezing parameters during prior-knowledge training. As indicated in Table. 1,
all LeNets’ parameters during prior-knowledge training on the posterior task
are trainable. However, in the training of the PKT model for the posterior-
task of CIFAR10, we freeze the PKT model’s parameters up to the
$6^{\textup{th}}$ convolutional layer (there are 9 convolutional layers),
i.e., only about $60\%$ of the model parameters are trainable in this task.
The reason for this is we want the prior-knowledge obtained from the weights’
transferring (Fig. 6) remains in the PKT model.

Baseline. To our knowledge, there exists no work directly addressing the
proposed problem. As such, we adopt state-of-the-art local explanation
methods, including Saliency Simonyan et al. [2014], IntegratedGradients
Sundararajan et al. [2017], Deeplift Shrikumar et al. [2017] and GradientSHAP
Lundberg and Lee [2017], implemented by Captum Kok [2020], to identify
critical neurons and evaluate the corresponding identified mechanisms
accordingly. As they are back-propagation methods, the neurons’ attribution on
each input can be extracted in a straight-forward manner, i.e. we can simply
back-propagate to the examined layer. The neurons are then selected by each
method based on their total attribution scores on all images of the examined
class.

The choice of precision values. There are several factors deciding the choices
of precision threshold in the identification of critical neurons. The main
three factors are the layer’s position, the number of neurons in the layer and
the overall complexity of the model. The intuition is that, the latter the
layer, the smaller of number of neurons in the layer and the less complexity
the model, the easier we can identify critical neurons. Thus, in such case, we
can set a lower value of FDR and expect better solution. In experiments for
LeNet, the precision is between $0.9$ and $0.98$. For CIFAR10, we select
values between $0.4$ and $0.8$ based on the layer and the number of neurons.
In Inception-v3 and CheXNet, the values are $0.6$ for all layers.

###  H.3 Ablation test

Since the mechanism specifies the prediction, critical neurons should have
high impact on model’s predictions. This impact can be evaluated via ablation
test, in which noise of different levels and configurations are added to those
neurons and the model’s accuracy is recorded accordingly. If the neurons have
explainability power, protecting them from the noise should maintain the
model’s predictive power.

Noise generation. In ablation test, we first generate an uniformly random base
noise $\delta_{i}\in[0,1]$ for each neuron $i$ of testing. Then, the weighted
noise for each neuron is computed as $\delta_{i}2^{-\gamma s_{i}}$, where
$\gamma$ is a parameter defined the noise and $s_{i}$ is the normalized
importance score of the neuron. The actual values of the scores are determined
by the explanation methods and how the neurons are selected. For example, if
we test the top-k neurons, then the scores for those neurons are 1 and the
rest are 0. The final noise added on the neurons is computed by normalizing
the weighted noise based on the specified noise level. Note that in each
computation of an input, only one $\delta_{i}$ is generated for all
explanation methods; thus, the differences in the actual noises added onto the
neurons across methods are introduced only by the difference in the score
$s_{i}$, not by the source of the random generator.

Results. Fig. 8 and 13 shows our ablation tests of neurons identified by
different methods. In the experiments, Top is the number of neurons protected
from the noise, determined by the score of the explanation methods. $\gamma$
is an exponential decay parameter determining how noise is distributed among
neurons: the larger the $\gamma$, the lesser the noise added to the protected
neurons. The test is conducted at different layers of LeNet and VGG. The
results show that neurons identified by NeuCEPT hold higher predictive power
among most of the experiments. A more distinctive difference among methods can
be observed at the ablation results for each class label in Fig. 14, 15, 16
and 17. The results imply that the methods select neurons differently even
though the averaging ablation’s results can be similar.

NeuCEPT on input features. Fig. 18 demonstrates our adaptation of NeuCEPT to
LeNet to select important input features for each class. The purpose of this
experiment is to demonstrate that NeuCEPT can adapt to the concerns regarding
Model-X Knockoffs (Appendix F), which are high-dimensional highly-correlated
data and unknown covariance, in the context of neural network. Here, we apply
Model-X Knockoffs’ representative approach: dividing the features into
$4\times 4$ groups based on their spatial positions on the image and using a
representative variable for each group. The activation/realization of that
variable is the average activation of features in the group. Then, those
representative neurons are considered as the inputs for NeuCEPT. Fig. 18 shows
the ablation results of neurons selected by NeuCEPT, compared with other
methods. The plots show that the explainability power of features selected by
NeuCEPT is comparable with other local explanation methods.

###  H.4 Running-Time.

We provide the experimental running-time of NeuCEPT and other methods in Table
2. Precisely, it is the time to obtain the set of important neurons for the
class of examination. All the experiments are conducted on 10000 samples of
test set. A straight comparison among methods is not trivial due to the
difference in resource utilization: our method runs entirely on CPUs while
some other methods utilize both CPUs and GPUs. Hyper-parameters selections
also affect the running-time of some methods significantly. Technically, in
NeuCEPT, we only need to run Model-X Knockoffs once and obtain the critical
neurons; however, we run Model-X Knockoffs 50 times in our experiments for
more stable results. The main takeaway is that NeuCEPT can be run in a
reasonable amount of time on moderate-size models such as VGG.