SpArX: Sparse Argumentative Explanations for Neural Networks [Technical
  Report]
Neural networks (NNs) have various applications in AI, but explaining their
decisions remains challenging. Existing approaches often focus on explaining
how changing individual inputs affects NNs' outputs. However, an explanation
that is consistent with the input-output behaviour of an NN is not necessarily
faithful to the actual mechanics thereof. In this paper, we exploit
relationships between multi-layer perceptrons (MLPs) and quantitative
argumentation frameworks (QAFs) to create argumentative explanations for the
mechanics of MLPs. Our SpArX method first sparsifies the MLP while maintaining
as much of the original structure as possible. It then translates the sparse
MLP into an equivalent QAF to shed light on the underlying decision process of
the MLP, producing global and/or local explanations. We demonstrate
experimentally that SpArX can give more faithful explanations than existing
approaches, while simultaneously providing deeper insights into the actual
reasoning process of MLPs.
Hamed Ayoobi, Nico Potyka, Francesca Toni
# SpArX: Sparse Argumentative Explanations for Neural Networks

###### Abstract

Neural networks (NNs) have various applications in AI, but explaining their decision process remains challenging. Existing approaches often focus on explaining how changing individual inputs affects NNs' outputs. However, an explanation that is consistent with the input-output behaviour of an NN is not necessarily faithful to the actual mechanics thereof. In this paper, we exploit relationships between _multi-layer perceptrons_ (MLPs) and _quantitative argumentation frameworks_ (QAFs) to create argumentative explanations for the mechanics of MLPs. Our _SpArX_ method first sparsifies the MLP while maintaining as much of the original mechanics as possible. It then translates the sparse MLP into an equivalent QAF to shed light on the underlying decision process of the MLP, producing _global and/or local explanations_. We demonstrate experimentally that SpArX can give more faithful explanations than existing approaches, while simultaneously providing deeper insights into the actual reasoning process of MLPs.

## 1 Introduction

The increasing use of black-box models like neural networks (NNs) in autonomous intelligent systems raises concerns about their fairness, reliability and safety. In order to address these concerns, the literature puts forward various explainable AI approaches to make the mechanics of NNs more transparent. This literature includes model-agnostic approaches as in [12, 13] and approaches tailored to the structure of NNs as in [14, 15]. However, these approaches fail to capture the actual mechanics of the NNs they aim to explain and thus it is hard to evaluate how faithful these approaches are [12, 13, 15].

[20] recently proposed regularizing the training procedure of NN such that they can be well approximated by decision trees. While this is an interesting direction, evaluating the faithfulness of the decision trees to the NN remains a challenge. Other recent work unearthend formal relationships between NNs in the form of multi-layer perceptrons (MLPs) and symbolic reasoning approaches like _quantitative argumentation frameworks_ (_QAFs_) [21, 16, 17] and weighted conditional knowledge bases [14]. The formal relationships indicate that these approaches may pave the way towards potentially more faithful explanations than approximate abstractions, such as decision trees.

In this paper, we provide explanations for MLPs leveraging on their formal relationships with QAFs in [21]. Intuitively, QAFs represent arguments and relations of attack or support between them as a graph, where nodes hold arguments and edges represent the relations. Various QAF formalisms have been studied over the years, e.g. by [1, 1, 13, 14, 15, 16, 17, 18, 19, 20, 21]. As it turns out, every MLP corresponds to a QAF of a particular form (where arguments are equipped with a base score and edges with weights) under a particular semantics (ascribing a "dialectical" strength to arguments), and, conversely, many acyclic QAFs corresponds to MLPs [21]. While this relationship suggests that QAFs are well suited to create faithful explanations for MLPs, it is evident that just reinterpreting an MLP as a QAF would not give us a comprehensible explanation in general because the QAF has the same density as the original MLP.

In order to create faithful and comprehensible argumentative explanations, we propose a novel method based on a two-step process. We first _sparsify_ the MLP, while maintaining as much of its mechanics as possible, in the spirit of clustering. Then, we translate the sparse MLP into a QAF, extending the method of [21]. We call our method _SpArX_ (standing for _Sparse Argumentative eXplanations_ for MLPs). In principle, in order to sparsify in the first step, we could just apply an existing compression method for NNs (e.g. [21]). However, existing methods are not designed for maintaining the mechanics of NNs. We thus make the following contributions:

* We propose a novel _clustering method_ for summarizing neurons based on their output-similarity. We compute parameters for the clustered neurons by aggregating the original parameters so that the output of the clustered neuron is similar to the neurons that it summarizes.
* We propose two families of _aggregation functions_for aggregating the parameters of neurons in a cluster: the first gives _global explanations_ (explaining the global behaviour of the MLP) and the second gives _local explanations_ (explaining the behaviour of the MLP at a target point, when the MLP is applied to a specific input).
* We conduct several experiments demonstrating the viability of our SpArX method for MLPs and its competitiveness with respect to other methods in terms of (i) conventional notions of _input-output faithfulness_ of explanations and (ii) novel notions of _structural faithfulness_, while (iii) shedding some light on the tradeoff between faithfulness and comprehensibility understood in terms of a notion of _cognitive complexity_, when generating explanations with SpArX.

Overall, we show that formal relationships between black-box machine learning models (such as NNs) and interpretable symbolic reasoning approaches (such as QAFs) can pave the way toward practical solutions for faithful and comprehensible explanations showing how the models reason.

## 2 Related Work

While MLPs are most commonly used in their fully connected form, there has been increasing interest in learning sparse NNs in recent years. However, the focus is usually not on finding an easily interpretable network structure, but rather on decreasing the risk for overfitting, memory and runtime complexity and the associated power consumption. Existing approaches include regularization to encourage neurons with weight \(0\) to be deleted [10], pruning of edges [23], compression [17] and low rank approximation [14]. Another related approach introduces interval NNs [1], which summarize neurons in clusters and consider interval outputs for the clustered neurons to give lower and upper bounds on the outputs for verification purposes. Our method is related to interval NNs in that we also summarize neurons in clusters. However, as opposed to [1], we cluster neurons based on their output. Furthermore, the output associated with a cluster is not an interval, but a numerical value like in a standard MLP. To compute the output, we aggregate the neurons in the cluster using different aggregation functions.

Several approaches exist for obtaining argumentative explanations for a variety of models, e.g. as recently overviewed in [11], including for NNs [1], but these are based on approximations of NNs (e.g. using Layerwise Relevance Propagation [1]), rather than summarizations as in our method, and their faithfulness is difficult to ascertain.

Several existing methods, like ours, make use of symbolic reasoning approaches for providing explanations, e.g. as recently overviewed in [16]. The explanations resulting from these methods (e.g. abduction-based explanations [1], prime implicants [15], sufficient reasons [1], and majority reasons [1]) faithfully capture the input-output behaviour of the explained models rather than their mechanics, as our method does for MLPs. Other methods extract logical rules as explanations for machine learning models, including NNs, e.g. as in [10] and [12], but again focus on explanations that are input-output faithful rather than on explaining the underpinning mechanics.

## 3 Preliminaries

Intuitively, a multi-layer perceptron (MLP) is a layered acyclic graph that processes its input by propagating it through the layers. Formally, we describe MLPs as follows.

**Definition 1** (Multi-Layer Perceptron (MLP)).: An _MLP_\(\mathcal{M}\) is a tuple \((V,E,\mathcal{B},\mathcal{W},\varphi)\), where:

* \((V,E)\) is a directed graph;
* \(V=\uplus_{l=0}^{d+1}V_{l}\) consists of (ordered) layers of neurons; for \(0\,{\leq}\,l\,{\leq}\,d+1\), \(V_{l}=\{v_{i,i}\mid 1\leq i\leq|V_{l}|\}\): we call \(V_{0}\) the _input layer_, \(V_{d+1}\) the _output layer_ and \(V_{l}\), for \(1\leq l\leq d\), the \(l\)-th _hidden layer_; \(d\) is the _depth_ of the MLP.
* \(E\subseteq\bigcup_{l=0}^{d}\left(V_{l}\times V_{l+1}\right)\) is a set of edges between adjacent layers; if \(E=\bigcup_{l=0}^{d}\left(V_{l}\times V_{l+1}\right)\), then the MLP is called _fully connected_;
* \(\mathcal{B}=\{b^{1},\dots,b^{d+1}\}\) is a set of _bias_ vectors, where, for \(1\leq l\leq d+1\), \(b^{l}\in\mathbb{R}^{|V_{l}|}\);
* \(\mathcal{W}=\{W^{0},\dots,W^{d}\}\) is a set of _weight_ matrices, where, for \(1\leq l\leq d\), \(W^{l}\in\mathbb{R}^{|V_{l+1}|\times|V_{l}|}\) such that \(W^{l}_{i,j}=0\) whenever \((v_{l,j},v_{l+1,i})\not\in E\);
* \(\varphi:\mathbb{R}\rightarrow\mathbb{R}\) is an _activation function_.

In order to process an _input_\(x\in\mathbb{R}^{|V_{0}|}\), the input layer of \(\mathcal{M}\) is initialized with \(x\). The input is then propagated forward through the MLP to generate values at each subsequent layer and ultimately an _output_ in the output layer. Formally, if the values at layer \(l\) are \(x_{l}\in\mathbb{R}^{|V_{l}|}\), then the values \(x_{l+1}\in\mathbb{R}^{|V_{l+1}|}\) at the next layer are defined by \(x_{l+1}=\varphi(W^{l}\,x_{l}+b^{l})\), where the activation function \(\varphi\) is applied component-wise. We let \(\mathcal{O}_{x}^{\mathcal{M}}:V\rightarrow\mathbb{R}\) denote the _output function_ of \(\mathcal{M}\), assigning to every neuron its value when the input \(x\) is given. That is, for \(v_{0,i}\in V_{0}\), we let \(\mathcal{O}_{x}^{\mathcal{M}}(v_{0,i})=x_{i}\) and, for \(l>0\), we let the _activation value_ of neuron \(v_{l,i}\) be \(\mathcal{O}_{x}^{\mathcal{M}}(v_{l,i})=\varphi(W^{l}\,\mathcal{O}_{x}^{\mathcal{ M}}(V_{l-1})+b^{l})_{i}\), where \(\mathcal{O}_{x}^{\mathcal{M}}(V_{l-1})\) denotes the vector that is obtained from \(V_{l-1}\) by applying \(\mathcal{O}_{x}^{\mathcal{M}}\) component-wise.

Every MLP can be seen as a quantitative argumentation framework (QAF) [10]. Intuitively, QAFs are _edge-weighted_ directed graphs, where nodes represent _arguments_ and, similarly to [11], edges with negative weight represent _attack_ and edges with positive weight represent _support_ relations between arguments. Each argument is initialized with a _base score_ that assigns an apriori _strength_ to the argument. The strength of arguments is then updated iteratively based on the strength values of attackers and supporters until the values converge. In acyclic graphs corresponding to MLPs, this iterative process is equivalent to the forward propagation process in the MLPs [10]. Conceptually, strength values are from some _domain_\(\mathcal{D}\)[1]. As we focus on (real-valued)
MLPs, we will assume \(\mathcal{D}\subseteq\mathbb{R}\). The exact domain depends on the activation function, e.g. the logistic function results in \(\mathcal{D}=[0,1]\), the hyperbolic tangent in \(\mathcal{D}=[-1,1]\) and ReLU in \(\mathcal{D}=[0,\infty]\). Formally, we describe QAFs as follows.

**Definition 2** (Quantitative Argumentation Framework (Qafy)).: A _QAF with domain \(\mathcal{D}\subseteq\mathbb{R}\)_ is a tuple \((\mathcal{A},E,\beta,w)\) that consists of

* a set of _arguments_\(\mathcal{A}\) and a set of _edges_\(E\subseteq\mathcal{A}\times\mathcal{A}\) between arguments,
* a function \(\beta:\mathcal{A}\rightarrow\mathcal{D}\) that assigns _base scores_ from \(\mathcal{D}\) to all arguments, and
* a function \(w:E\rightarrow\mathbb{R}\) that assigns _weights_ to all edges.

Edges with negative/positive weights are called _attack/support_ edges, denoted by \(\mathrm{Att/Sup}\), respectively.

The strength values of arguments are usually computed iteratively using a two-step update procedure [11]: first, an _aggregation function_\(\alpha\) aggregates the strength values of attackers and supporters; then, an _influence function_\(\iota\) adapts the base score. Examples of aggregation functions include product [1, 1], addition [1, 2] and maximum [11] and the influence function is defined accordingly to guarantee that strength values fall in the domain \(\mathcal{D}\). In this paper, we focus on the aggregation and influence function from [2], to obtain QAFs simulating MLPs with a logistic activation function [2]. The strength values of arguments are computed by the following iterative procedure: for every argument \(a\in\mathcal{A}\), we let \(s_{a}^{(0)}:=\beta(a)\) be the initial strength value; the strength values are then updated by the following two steps repeatedly for all \(a\in\mathcal{A}\) (where the auxiliary variable \(\alpha_{a}^{i}\) carries the aggregate at iteration \(i\geq 0\)):

**Aggregation:**: \(\alpha_{a}^{(i+1)}:=\sum_{(b,a)\in E}w((b,a))\cdot s_{b}^{(i)}\).
**Influence:**: \(s_{a}^{(i+1)}:=\varphi_{l}\big{(}\ln(\frac{\beta(a)}{1-\beta(a)})+\alpha_{a}^{ (i+1)}\big{)}\), where \(\varphi_{l}(z)=\frac{1}{1+\exp(-z)}\) is the logistic function.

The _final strength_ of argument \(a\) is defined via the limit of \(s_{a}^{(i)}\), for \(i\) towards infinity. Notably, the semantics given by this notion of final strength satisfies almost all desiderata for QAF semantics perfectly [2].

## 4 From General MLPs to QAFs

Here we generalize the connection between MLPs and QAFs beyond MLPs with logistic activation functions, as follows. Assume that \(\varphi:\mathbb{R}\rightarrow\mathcal{D}\) is an activation function that is strictly monotonically increasing. Examples include logistic, hyperbolic tangent and parametric ReLU activation functions. Then \(\varphi\) is invertible and \(\varphi^{-1}:\mathcal{D}\rightarrow\mathbb{R}\) is defined. We can then define the update function corresponding to an MLP with such activation function \(\varphi\) by using the same aggregation function as before and using the following influence function:

**Influence:**: \(s_{a}^{(i+1)}:=\varphi\big{(}\varphi^{-1}(\beta(a))+\alpha_{a}^{(i+1)}\big{)}\).

Note that the previous definition of influence in Section 3, from [2], is a special case because \(\ln(\frac{1}{1-x})\) is the inverse function of the logistic function \(\varphi_{l}(x)\). Note also that the popular ReLU activation function \(\varphi_{ReLU}(x)=\max(0,x)\) is not invertible because all non-positive numbers are mapped to \(0\). However, for our purpose of translating MLPs to QAFs, we can define

\[\varphi_{ReLU}^{-1}(x)=\begin{cases}x,&\text{if }x>0;\\ 0,&\text{otherwise}.\end{cases}\]

In order to translate an MLP \(\mathcal{M}\) with activation function \(\varphi\) and input \(x\) into a QAF \(Q_{\mathcal{M},x}\), we interpret every neuron \(v_{l,i}\) as an abstract argument \(A_{l,i}\). Edges in \(\mathcal{M}\) with positive/negative weights are interpreted as supports/attacks, respectively, in \(Q_{\mathcal{M},x}\). The base score of an argument \(A_{0,i}\) associated with input neuron \(v_{0,i}\) is just the corresponding input value \(x_{i}\). The base score of the remaining arguments \(A_{l,i}\) is \(\varphi(b_{i}^{l})\), where \(b_{i}^{l}\) is the bias of the associated neuron \(v_{l,i}\).

**Proposition 1**.: _Let \(\mathcal{M}\) be an MLP with activation function \(\varphi\) that is invertible or ReLU. Then, for every input x, the QAF \(Q_{\mathcal{M},x}\) satisfies \(\mathcal{O}_{x}^{\mathcal{M}}(v_{l,i})=\sigma(A_{l,i})\), where \(\sigma(A_{l,i})\) denotes the final strength of \(A_{l,i}\) in \(Q_{\mathcal{M},x}\)._

## 5 SpArX: Explaining MLPs with QAFs

Just translating an MLP into a QAF would not give a comprehensible explanation in general because the QAF has the same density as the original MLP. Thus, in order to explain an MLP, we first sparsify it, in a customizable way to support different comprehensibility needs, and then translate it into a QAF. The sparsification should maintain as much of the original MLP as possible to give faithful explanations. To achieve this, roughly speaking we exploit redundancies in the MLP by replacing neurons giving similar outputs with a single neuron that summarizes their joint effect.

Summarizing neurons in this way is a clustering problem. Formally, a clustering problem is defined by a set of inputs from an abstract space \(\mathcal{S}\) and a distance measure \(\delta:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}_{\geq 0}\). The goal is to partition \(\mathcal{S}\) into clusters \(C_{1},\ldots,C_{K}\) (where \(\mathcal{S}=\psi_{i=1}^{K}C_{i}\)) such that the distance between points within a cluster is'small' and the distance between points in different clusters is 'large'. Finding an optimal clustering is NP-complete in many cases [10]. Thus, we cannot expect to find an efficient algorithm that computes an optimal clustering, but we can apply standard algorithms e.g. K-means [13] to find a good (but not necessarily optimal) clustering efficiently.

In our setting, \(\mathcal{S}\) is the set \(V_{l}\) of neurons in any given layer \(0<l<d+1\) and the distance between two neurons can be defined as the difference between their outputs for the inputs in a given dataset \(\Delta\) (e.g. the training dataset):

\[\delta(v_{l,i},v_{l,j})=\sqrt{\sum_{x\in\Delta}(\mathcal{O}_{x}^{\mathcal{M}}(v _{l,i})-\mathcal{O}_{x}^{\mathcal{M}}(v_{l,j}))^{2}}. \tag{1}\]

After clustering, we have a partitioning \(\mathcal{P}=\psi_{l=1}^{d}\mathcal{P}_{l}\) of (the hidden layers of) our MLP \(\mathcal{M}\), where \(\mathcal{P}_{l}=\{C_{1}^{1},\ldots,C_{K_{l}}^{l}\}\) is the clustering of the \(l\)-th layer, that is, \(V_{l}=\psi_{i=1}^{K_{l}}C_{i}^{l}\). We use the clustering to create a corresponding _clustered MLP_\(\mu\) whose neurons correspond to clusters in the original \(\mathcal{M}\). We
call these neurons _cluster-neurons_ and denote them by \(v_{C}\), where \(C\) is the associated cluster. The graphical structure of \(\mu\) is as follows.

**Definition 3** (Graphical Structure of Clustered MLP).: Given an MLP \(\mathcal{M}\) and a clustering \(\mathcal{P}=\uplus_{i=1}^{d}\mathcal{P}_{i}\) of \(\mathcal{M}\), the _graphical structure of the corresponding clustered MLP_\(\mu\) is a directed graph \((V^{\mu},E^{\mu})\) such that

* \(V^{\mu}=\uplus_{l=0}^{d+1}V^{\mu}_{l}\) consists of (ordered) layers of cluster-neurons such that: 1. the input layer \(V^{0}_{0}\) consists of a singleton cluster-neuron \(v_{\{v_{0,i}\}}\) for every input neuron \(v_{0,i}\in V_{0}\); 2. the \(l\)-th hidden layer of \(\mu\) (for \(0<\!l<d+1\)) consists of one cluster-neuron \(v_{C}\) for every cluster \(C\in\mathcal{P}_{l}\); 3. the output layer \(V^{\mu}_{d+1}\) consists of a singleton cluster-neuron \(v_{\{v_{d+1,i}\}}\) for every output neuron \(v_{\{d+1,i}}\!\in\!V_{d+1}\);
* \(E^{\mu}=\bigcup_{l=0}^{d}\big{(}V^{\mu}_{l}\times V^{\mu}_{l+1}\big{)}\).

**Example 1**.: For illustration, consider the MLP in Figure 1.a, trained to approximate the XOR function from the dataset \(\Delta=\{(0,0),(0,1),(1,0),(1,1)\}\) with target outputs \(0,1,1,0\), respectively. The activation values of the hidden neurons for the four input pairs are \((0,0,0,0),(1.7,0,1.8,0),(0,2.3,0,1.5),(0,0,0,0)\), respectively. Applying \(K\)-means clustering with \(\delta\) as in Eq. 1 and \(K=2\) for the hidden layer results in two clusters \(C_{1},C_{2}\), indicated by rectangles in the figure.

We define global and local explanations for MLPs by translating their corresponding clustered MLPs into QAFs. The clustered MLPs for global and local explanations share the same graphical structure but differ in the parameters of the cluster-neurons, that is, (i) the biases of cluster-neurons and (ii) the weights for edges between cluster-neurons. We define these parameters in terms of _aggregation functions_, specifically a _bias aggregation function_\(\mathrm{Agg}^{b}:\mathcal{P}\rightarrow\mathbb{R}\), mapping clusters to biases, and an _edge aggregation function_\(\mathrm{Agg}^{e}:\mathcal{P}\times\mathcal{P}\rightarrow\mathbb{R}\cup\{\bot\}\), mapping pairs of clusters to weighs if the pairs correspond to edges in \(\mu\), or \(\bot\) otherwise. Given any concrete such aggregation functions (as defined later), the parameters of \(\mu\) can be defined as follows.

**Definition 4** (Parameters of Clustered MLP).: Given an MLP \(\mathcal{M}=(V,E,\mathcal{B},\mathcal{W},\varphi)\), let \((V^{\mu},E^{\mu})\) be the graphical structure of the corresponding clustered MLP \(\mu\). Then, for bias and edge aggregation functions \(\mathrm{Agg}^{b}\) and \(\mathrm{Agg}^{e}\), respectively, \(\mu\) is \((V^{\mu},E^{\mu},\mathcal{B}^{\mu},\mathcal{W}^{\mu},\varphi)\) with _parameters_\(\mathcal{B}^{\mu},\mathcal{W}^{\mu}\) as follows:

* for every cluster-neuron \(v_{C}\in V^{\mu}\), the bias (in \(\mathcal{B}^{\mu}\)) of \(v_{C}\) is \(\mathrm{Agg}^{b}(C)\);
* for every edge \((v_{C_{1}},v_{C_{2}})\in E^{\mu}\), the weight (in \(\mathcal{W}^{\mu}\)) of the edge is \(\mathrm{Agg}^{e}((C_{1},C_{2}))\).

### Sparse Argumentative Global Explanations

We consider the following aggregation functions. As we explain in the supplementary material (SM), they minimize the deviation (with respect to the least-squares error) of bias and weights of the cluster neuron and the neurons contained in the cluster.

**Definition 5** (Global Aggregation Functions).: The _average bias and edge aggregation functions_ are, respectively:

\[\mathrm{Agg}^{b}(C)=\frac{1}{|C|}\sum_{v_{l,i}\in C}b^{l}_{i};\]

\[\mathrm{Agg}^{e}((C_{1},C_{2}))=\sum_{v_{l,i}\in C_{1}}\frac{1}{|C_{2}|}\sum _{v_{l+1,j}\in C_{2}}W^{l}_{j,i}.\]

The average bias aggregation function simply averages the biases of neurons in the cluster. Intuitively, the weight of an edge between cluster-neurons \(v_{C_{1}}\) and \(v_{C_{2}}\) has to capture the effects of all neurons summarized in \(C_{1}\) on neurons summarized in \(C_{2}\). The neurons in \(C_{1}\) all affect every neuron in \(C_{2}\), therefore their effects have to be summed up. As \(v_{C_{2}}\) acts as a replacement of all neurons in \(C_{2}\), it has to aggregate their activation. We achieve this aggregation by averaging again.

The following example illustrates global explanations drawn from clustered MLPs, with the average aggregation functions, via their understanding as QAFs.

**Example 2**.: Figure 1.b shows the clustered MLP for our XOR example from Figure 1.a. The corresponding QAF can be visualised as in Figure 1.c, where attacks are given in red and supports in green, and the thickness of the edges reflects their weight. To better visualize the role of each cluster-neuron, we can use a word-cloud representation as in Figure 1.d (showing, e.g., that \(x_{0}\) and \(x_{1}\) play a negative and positive role, respectively, towards \(C_{1}\), which supports the output). The word-cloud representation gives insights into the reasoning of the MLP (with the learned rule \((\overline{x_{0}}\wedge x_{1})\vee(x_{0}\wedge\overline{x_{1}})\)).

### Sparse Argumentative Local Explanations

While global explanations attempt to faithfully explain the behaviour of the MLP on all inputs, our local explanations focus on the behaviour in the neighborhood of the input \(x\) from the dataset \(\Delta\), similar to LIME [10]. To do so, we generate random neighbors of \(x\) to obtain a _sample dataset_\(\Delta^{\prime}\), and weigh them with an exponential kernel

Figure 1: a) MLP for XOR, with cluster-neurons \(C_{1}\) and \(C_{2}\). b) Clustered MLP. c) Global explanation as a QAF. d) Word-cloud representation of the hidden cluster-neurons.
from LIME [Ribeiro _et al._2016], assigning lower weight to a sample \(x^{\prime}\in\Delta^{\prime}\) that is further away from the target \(x\):

\[\pi_{x^{\prime},x}=exp(-D(x^{\prime},x)^{2}/\sigma^{2})\]

where \(D\) is the Euclidean distance function and \(\sigma\) is the width of the exponential kernel.

We aggregate biases as before but replace the edge aggregation function with the following.

**Definition 6** (Local Edge Aggregation Function).: The _local edge aggregation function_ with respect to the _input_\(x\) is

\[\operatorname{Agg}_{x}^{\varepsilon}(C_{1},C_{2})=\]

where \(\mathcal{O}_{x^{\prime}}^{\mathcal{M}}(v_{l,i})\) is the activation value of the neuron \(v_{l,i}\) in the original MLP and \(\mathcal{O}_{x^{\prime}}^{\mu}(v_{C_{1}})\) is the activation value of the cluster-neuron \(C_{1}\) in the clustered MLP.

Note that, by this definition, the edge weights are computed layer by layer from input to output.

## 6 Desirable Properties of Explanations

In order to evaluate SpArX, we consider three measures for assessing faithfulness and comprehensibility of explanations. In this section, we assume as given an MLP \(\mathcal{M}\) of depth \(d\) and a corresponding clustered MLP \(\mu\).

To begin with, we consider a _faithfulness_ measure inspired by the notion of fidelity considered for LIME [Ribeiro _et al._2016], based on measuring the _input-output_ difference between the original model (in our case, \(\mathcal{M}\)) and the substitute model (in our case, the clustered MLP/QAF).

**Definition 7** (Input-Output Unfaithfulness).: The _local input-output unfaithfulness_ of \(\mu\) to \(\mathcal{M}\) with respect to _input_\(x\) and _dataset_\(\Delta\) is

\[\mathcal{L}^{\mathcal{M}}(\mu)=\sum_{x^{\prime}\in\Delta}\pi_{x^{\prime},x} \sum_{v\in V_{d+1}}(\mathcal{O}_{x^{\prime}}^{\mathcal{M}}(v)-\mathcal{O}_{x^ {\prime}}^{\mu}(v))^{2}.\]

The _global input-output unfaithfulness_ of \(\mu\) to \(\mathcal{M}\) with respect to dataset \(\Delta\) is

\[\mathcal{G}^{\mathcal{M}}(\mu)=\sum_{x^{\prime}\in\Delta}\sum_{v\in V_{d+1}}( \mathcal{O}_{x^{\prime}}^{\mathcal{M}}(v)-\mathcal{O}_{x^{\prime}}^{\mu}(v))^ {2}.\]

The lower the input-output unfaithfulness of the clustered MLP \(\mu\), the more faithful \(\mu\) is to the original MLP.

The input-output unfaithfulness measures deviations in the input-output-behaviour of the substitute model, but, since clustered MLPs maintain much of the structure of the original MLPs, we can define a more fine-grained notion of _structured faithfulness_ by comparing the outputs of the individual neurons in the MLP to the output of the cluster-neurons summarizing them in the clustered MLP.

**Definition 8** (Structural Unfaithfulness).: Let \(K_{l}\) be the number of clusters at hidden layer \(l\) in \(\mu\) (\(0{<}l{\leq}d\)) and \(K_{d+1}\) be the number of output neurons (in \(\mu\) and \(\mathcal{M}\)). Let \(K_{l,j}\) be the number of neurons in the \(j\)th cluster-neuron \(C_{l,j}\) (\(0{<}l\leq d+1\), with \(K_{d+1,j}{=}1\)). Then, the _local structural unfaithfulness_ of \(\mu\) to \(\mathcal{M}\) with respect to _input_\(x\) and _dataset_\(\Delta\) is:

\[\mathcal{L}_{s}^{\mathcal{M}}(\mu)=\sum_{x^{\prime}\in\Delta}\pi_{x^{\prime},x }\sum_{l=1}^{d+1}\sum_{j=1}^{K_{l}}\sum_{v_{l,i}\in C_{l,j}}(\mathcal{O}_{x^{ \prime}}^{\mathcal{M}}(v_{l,i}){-}\mathcal{O}_{x^{\prime}}^{\mu}(C_{l,j}))^{2}.\]

The _global structural unfaithfulness_\(\mathcal{G}_{s}^{\mathcal{M}}(\mu)\) is defined analogously by removing the similarity terms \(\pi_{x^{\prime},x}\).

The lower the structured unfaithfulness of the clustered MLP \(\mu\), the more structurally faithful \(\mu\) is to the original MLP. Note that our notion of structural faithfulness is different from the notions of structural descriptive accuracy by [Albini _et al._2022]: they characterise bespoke explanations, defined therein, of probabilistic classifiers equipped with graphical structures and cannot be used in place of our notion, tailored to local and global explanations

Finally, we consider the _cognitive complexity_ of an explanation based on its size, inspired by the notion of cognitive tractability in [Cyras _et al._2019]. In our case, we use the number of cluster-neurons/arguments as a measure.

**Definition 9** (Cognitive Complexity).: Let \(K_{l}\) be the number of clusters at hidden layer \(l\) in \(\mu\) (\(0<l\leq d\)). Then, the _cognitive complexity_ of \(\mu\) is defined as

\[\Omega(\mu)=\mathop{\Pi}\limits_{0<l<d+1}K_{l}\]

Note that there is a tradeoff between faithfulness and cognitive complexity in SpArX. By reducing the number of cluster-neurons, we reduce cognitive complexity. However, this also results in a higher variance in the neurons that are summarized in the clusters, so the faithfulness of the explanation can suffer. We will explore this trade-off experimentally next.

Finally, note that other properties of explanations by symbolic approaches, notably by [Amgoud and Ben-Naim2022], are unsuitable for our mechanistic explanations as QAFs. Indeed, these properties characterize the input-output behaviour of classifiers, rather than their mechanics.

## 7 Experiments

We conducted four sets of experiments to evaluate SpArX with respect to (i) the trade-off between its sparsification and its ability to maintain faithfulness (Section 7.1 for global and Section 7.2 for local explanations), (ii) SpArX's scalability (Section 7.3), and (iii) the tradeoff between faithfulness and cognitive complexity (Section 7.4). We used four datasets for classification: iris1 with 150 instances, 4 continuous features and 3 classes; breast cancer2 with 569 instances, 30 continuous features and 2 classes; COMPAS [Jeff Larson and Angwin2016] with 11,000 instances and 52 categorical features, to classify \(two\_year\_recid\); and forest covertype3[Blackard and Dean1999] with 581,012 instances, 54 features (10 continuous, 44 binary), and 7 classes.

Footnote 1: [https://archive.ics.uci.edu/ml/datasets/iris](https://archive.ics.uci.edu/ml/datasets/iris)

Footnote 2: [https://archive.ics.uci.edu/ml/datasets/cancer](https://archive.ics.uci.edu/ml/datasets/cancer)

We used the first three datasets to evaluate the (input-output and structural) unfaithfulness of the global and local explanations generated by SpArX. We then used the last dataset,
which is considerably larger and requires deeper MLPs with varying architectures, to evaluate the scalability of SpArX. Finally, we used COMPAS to assess cognitive complexity.

For the experiments with the first three datasets we used MLPs with 2 hidden layers and 50 hidden neurons each, whereas for the experiments with the last we used 1-5 hidden layers with 100, 200 or 500 neurons. For all experiments, we used the RELU activation function for the hidden neurons and the softmax activation function for the output neurons. We give classification performances for all MLPs and average run-times for generating local explanations in the SM.

When experimenting with SpArX, one needs to choose the number of clusters/cluster-neurons at each hidden layer: we do so by specifying a _compression ratio_ (for example, a compression ratio of 0.5 amounts to obtaining half cluster-neurons than the original neurons).

### Global Faithfulness (Comparison to HAP)

Since SpArX essentially compresses an MLP to construct a QAF, one may ask how it compares to existing compression approaches.4 To assess the faithfulness of our global explanations, we compared SpArX's clustering approach to the state-of-the-art compression method Hessian Aware Pruning (HAP) [20], which uses relative Hessian traces to prune insensitive parameters in NNs. We measured both input-output and structural unfaithfulness of SpArX and HAP to the original MLP, using the result of HAP compression in place of \(\mu\) when applying Definitions 7 and 8 for comparison.

Footnote 4: Whereas existing NN compression methods typically retrain after compression, we do not, as we want to explain the original NN.

Input-Output Faithfulness.Table 1 shows the input-output unfaithfulness of the global explanations (\(\mathcal{G}^{\mathcal{M}}(\mu)\) in Def. 7) obtained from SpArX and HAP using our three chosen datasets and different compression ratios. The unfaithfulness of global explanations in SpArX is lower than HAP especially when the compression ratio is high. Note that this does not mean that SpArX is a better compression method, but that the compression method in SpArX is better for our purposes (that is, compressing the MLP while keeping its mechanics).

Structural Faithfulness.Table 2 gives the structural global unfaithfulness (\(\mathcal{G}^{\mathcal{M}}_{s}(\mu)\) in Def. 8) for SpArX and HAP, on the three chosen datasets, using different compression ratios. Our method has a much lower structural unfaithfulness than HAP by preserving activation values close to the original model.

### Local Faithfulness (Comparison to LIME)

In order to evaluate the local input-output unfaithfulness of SpArX (\(\mathcal{L}^{\mathcal{M}}(\mu)\) in Def. 7), we compare SpArX and LIME [13]5, which approximates a target point locally on an interpretable substitute model.6 Table 3 shows the input-output unfaithfulness of the local explanations for LIME and SpArX. We used the same sampling approach as LIME [13]. We averaged the unfaithfulness measure for all test examples. The results show that the local explanations produced by our approach are more input-output faithful to the original model. Thus, basing local explanations on keeping the MLP mechanics helps also with their input-output faithfulness.

Footnote 5: [https://github.com/marcotcr/lime](https://github.com/marcotcr/lime)

Footnote 6: We used ridge regression, suitable with tabular data in LIME. We used the substitute model as \(\mu\) when applying Def. 7 to LIME.

### Scalability

To evaluate the scalability of SpArX, we measured its input-output faithfulness on MLPs of increasing complexity. we have compared it to LIME [13] using forest covertype as a sufficiently large dataset to be tested with various MLP architectures with different sizes. We have trained 15 MLPs with varying numbers of hidden

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{Method} & \multirow{2}{*}{Compression Ratio} & \multicolumn{3}{c}{Datasets} \\ \cline{3-5}  & & Iris & Cancer & COMPAS \\ \hline HAP & 0.2 & 0.05 & 0.48 & 0.02 \\ SpArX & **0.00** & **0.02** & **0.00** \\ \hline HAP & 0.4 & 0.23 & 0.53 & 0.11 \\ SpArX & **0.00** & **0.05** & **0.00** \\ \hline HAP & 0.6 & 0.23 & 0.58 & 0.20 \\ SpArX & **0.00** & **0.10** & **0.00** \\ \hline HAP & 0.8 & 0.28 & 1.00 & 0.26 \\ SpArX & **0.00** & **0.21** & **0.00** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Global input-output unfaithfulness of sparse MLPs generated by HAP vs our SpArX method. (Best results in **bold**)

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{Datasets} \\ \cline{2-4}  & Iris & Cancer & COMPAS \\ \hline LIME & 0.3212 & 0.1623 & 0.0224 \\ SpArX (0.6) & **0.0257** & **0.0055** & **0.0071** \\ SpArX (0.8) & **0.0707** & **0.0156** & **0.0083** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Local input-output unfaithfulness of LIME vs our SpArX method (with different compression ratios, in brackets).
layers (\(\#\)Layers) and different numbers of hidden neurons (\(\#\)Neurons) at each hidden layer (see details in the SM).

Table 4 compares the input-output unfaithfulness of the local explanations of SpArX using \(80\%\) compression ratio7 with LIME, all averaged over the test set. The result confirms that SpArX explanations are scalable to different MLP architectures with different sizes and large datasets.

Footnote 7: For experiments with lower compression ratios see the SM.

### Cognitive Complexity

The cognitive complexity of SpArX (Def. 9) is dependent on the number of clusters at each layer. A lower number of clusters leads to a more interpretable explanation at the cost of achieving lower (structural) faithfulness.

Figure 1(a) shows a _global explanation8_ of an MLP for the COMPAS dataset, with one hidden layer and 20 neurons in the hidden layer, with \(20\%\) compression ratio and pruning edges with low weights. Note that pruning is only done here for visualization. This explanation is hardly interpretable for a user. The classification results of the clustered MLP is \(98.32\%\), the same as the original MLP. Figure 1(b) shows the global explanation of the same MLP but with \(85\%\) compression rate and pruning the edges with low weights. This global explanation is more comprehensible for humans since there are fewer neurons in the clustered MLP. The classification results of the clustered MLP is \(94.60\%\), the same as the original model. Unlike shallow input-output explanations, we can see the role of each hidden neuron in the proposed method. The output \(O_{0}\) is _two_year_recid_. There are two sets of hidden cluster-neurons, namely an attacker (\(C_{1}\)) and two supporters (\(C_{2}\) and \(C_{3}\)). \(C_{1}\) attacks the output which means the criminal will not recommit a crime in a two-year period. Three features are supporting \(C_{1}\) and two features are attacking it. The attacking features also support \(C_{3}\). This means that they strengthen the support by \(C_{3}\) and weaken the attack by \(C_{1}\). Therefore, _priscrs_count_10 priors or more_ and _is_recid_Yes_ both strongly affect the output. Indeed, looking at the COMPAS dataset, more than \(99\%\) of criminals that have these two features recommitted the crime in a two years period. \(C_{2}\) and \(C_{3}\) are supporting the output. \(C2\) is only supported by the _is_violent_recid_Yes_ feature. This means that if a criminal has a violent recidivism (s)he will probably recommit a crime after two years. Checking the COMPAS dataset, this interpretation is \(100\%\) correct. These kinds of interpretations are beyond the shallow input-output explanations offered by model-agnostic explanation methods such as LIME. \(C3\) is supported by several features and is attacked by one feature. Looking at this argumentative global explanation one can understand the effect of each feature as well as of each hidden neuron on the output.

Footnote 8: We give an example of local explanation in the SM.

## 8 Conclusion

We introduced SpArX, a novel method for generating sparse argumentative explanations for MLPs. In contrast to shallow input-output explainers like LIME, SpArX focuses on maintaining structural similarity to the original MLP in order to give faithful explanations, while allowing tailoring them to the cognitive needs of users. Our experimental results show that the explanations by SpArX are more (input-output) faithful to the original model than LIME. We have also compared SpArX with a state-of-the-art NN compression technique called HAP, showing that SpArX is more faithful to the original model. Further, our explanations are more _structurally_ faithful to the original model by providing deeper insights into the mechanics thereof.

Future research includes extending SpArX to other types of NNs, e.g. CNNs, as well as furthering it to cluster neurons across hidden layers. It would also be interesting to explore whether SpArX could be extended to exploit formal relationships between NNs and other symbolic approaches, e.g. in [11]. Further, it would be interesting to explore formalizations such as in [10] for characterizing uncertainty as captured by SpArX.

\begin{table}
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{\(\#\)Layers} & \multirow{2}{*}{Method} & \multicolumn{3}{c}{\(\#\)Neurons} \\ \cline{3-5}  & & 100 & 200 & 500 \\ \hline
1 & LIME & 0.2375 & 0.2919 & 0.3123 \\
1 & SpArX & **0.0000** & **0.0018** & **0.0000** \\ \hline
2 & LIME & 0.2509 & 0.2961 & 0.3638 \\
2 & SpArX & **0.0019** & **0.0015** & **0.0034** \\ \hline
3 & LIME & 0.3130 & 0.3285 & 0.3127 \\
3 & SpArX & **0.0028** & **0.0026** & **0.0000** \\ \hline
4 & LIME & 0.3395 & 0.3459 & 0.3243 \\
4 & SpArX & **0.0001** & **0.0049** & **0.0000** \\ \hline
5 & LIME & 0.3665 & 0.3178 & 0.3288 \\
5 & SpArX & **0.0030** & **0.0064** & **0.0000** \\ \hline \end{tabular}
\end{table}
Table 4: Evaluating scalability of SpArX (forest covertype dataset): local input-output unfaithfulness of SpArX (with \(80\%\) compression ratio) and LIME using various MLPs with different numbers of hidden layers (\(\#\)Layers) and neurons (\(\#\)Neurons).

Figure 2: Global explanations by SpArX of an MLP with \(20\%\) and \(85\%\) compression ratios for the COMPAS dataset.