Towards Faithful Neural Network Intrinsic Interpretation with Shapley
  Additive Self-Attribution
Self-interpreting neural networks have garnered significant interest in
research. Existing works in this domain often (1) lack a solid theoretical
foundation ensuring genuine interpretability or (2) compromise model
expressiveness. In response, we formulate a generic Additive Self-Attribution
(ASA) framework. Observing the absence of Shapley value in Additive
Self-Attribution, we propose Shapley Additive Self-Attributing Neural Network
(SASANet), with theoretical guarantees for the self-attribution value equal to
the output's Shapley values. Specifically, SASANet uses a marginal
contribution-based sequential schema and internal distillation-based training
strategies to model meaningful outputs for any number of features, resulting in
un-approximated meaningful value function. Our experimental results indicate
SASANet surpasses existing self-attributing models in performance and rivals
black-box models. Moreover, SASANet is shown more precise and efficient than
post-hoc methods in interpreting its own predictions.
Ying Sun, Hengshu Zhu, Hui Xiong
# Towards Faithful Neural Network Intrinsic Interpretation with Shapley Additive Self-Attribution

###### Abstract

Self-interpreting neural networks have garnered significant interest in research. Existing works in this domain often (1) lack a solid theoretical foundation ensuring genuine interpretability or (2) compromise model expressiveness. In response, we formulate a generic Additive Self-Attribution (ASA) framework. Observing the absence of Shapley value in Additive Self-Attribution, we propose Shapley Additive Self-Attributing Neural Network (SASANet), with theoretical guarantees for the self-attribution value equal to the output's Shapley values. Specifically, SASANet uses a marginal contribution-based sequential schema and internal distillation-based training strategies to model meaningful outputs for any number of features, resulting in un-approximated meaningful value function. Our experimental results indicate SASANet surpasses existing self-attributing models in performance and rivals black-box models. Moreover, SASANet is shown more precise and efficient than post-hoc methods in interpreting its own predictions.

## 1 Introduction

While neural networks excel in fitting complex real-world problems due to their vast hypothesis space, their lack of interpretability poses challenges for real-world decision-making. Although post-hoc interpretation algorithms (Lundberg & Lee (2017); Shrikumar et al. (2017)) offer extrinsic model-agnostic interpretation, the un-transparent intrinsic modeling procedure unavoidably lead to inaccurate interpretation (Laugel et al. (2019); Frye et al. (2020)). Thus, there's a growing need for self-interpreting neural structures that intrinsically convey their prediction logic faithfully.

Self-interpreting neural networks research has garnered notable interest, with the goal of inherently elucidating a model's predictive logic. Various approaches are driven by diverse scenarios. For example, Alvarez Melis & Jaakkola (2018) linearly correlates outputs with features and consistent coefficients, while Agarwal et al. (2020) employs multiple networks each focusing on a single feature, and Wang & Wang (2021) classifies by comparing inputs with transformation-equivariant prototypes. However, many existing models, despite intuitive designs, might not possess a solid theoretical foundation to ensure genuine interpretability. The effectiveness of attention-like weights, for instance, is debated (Serrano & Smith (2019); Wiegreffe & Pinter (2019)). Additionally, striving for higher interpretability often means resorting to simpler structures or intricate regularization, potentially compromising prediction accuracy.

This paper aims to achieve theoretically guaranteed faithful self-interpretation while retaining expressiveness in prediction. To achieve this goal, we formulate a generic Additive Self-Attribution (ASA) framework. ASA offers an intuitive understanding, where the contributions of different observations are linearly combined for the final prediction. Notably, even with varying interpretative
angles, many existing methods implicitly employ such structure. Thus, we utilize ASA to encapsulate and distinguish their interpretations, clarifying when certain methods are favored over others.

Upon examining studies through the ASA lens, we identified an oversight regarding the Shapley value. Widely recognized for post-hoc attribution (Lundberg and Lee (2017); Bento et al. (2021)) with robust theoretical backing from coalition game theory (Shapley et al. (1953)), its potential has been underutilized. Notably, while a recent work called Shapley Explanation Network (SHAPNet) (Wang et al. (2021)) achieves layer-wise Shapley attribution, it lacks model-wise feature attribution. Addressing this gap, we introduce a Shapley Additive Self-Attributing Neural Network (SASANet) under the ASA framework, depicted in Figure 1. In particular, we directly define value function as model output given arbitrary number of input features. With an intermediate sequential modeling and distillation-based learning strategy, SASANet can be theoretically proven to provide additive self-attribution converging to the Shapley value of its own output. Our evaluations of SASANet on various datasets show it surpasses current self-attributing models and reaches comparable performance as black-box models. Furthermore, compared to post-hoc approaches, SASANet's self-attribution offers more precise and streamlined interpretation of its predictions.

## 2 Preliminaries

### Additive Self-Attributing Models

Oriented for various tasks, self-interpreting networks often possess distinct designs, obscuring their interrelations and defining characteristics. Observing a commonly shared principle of linear feature contribution aggregation across numerous studies, we introduce a unified framework:

**Definition 2.1** (Additive Self-Attributing Model): _Additive Self-Attributing Models output the sum of intrinsic feature attribution values that hold desired properties, formulated as_

\[f(x;\theta,\phi_{0})=\phi_{0}+\sum_{i\in\mathcal{N}}\phi(x;\theta)_{i},\qquad \text{s.t.}\quad\mathcal{C}(\phi,x), \tag{1}\]

_where \(x=[x_{1},\cdots,x_{N}]\in\mathbb{R}^{N}\) denotes a sample with \(N\) features, \(\mathcal{N}=\{1,2,\cdots,N\}\) denotes feature indices, \(\phi(x;\theta)_{i}\) is the \(i\)-\(th\) feature's attribution value, \(\phi_{0}\) is a sample-independent bias, \(\mathcal{C}(\phi,x)\) denotes constraints on attribution values._

As there are abundant ways to build additive models, attribution terms are required to satisfy specific constraints for ensuring the physically meaningfulness, often achieved through regularizers or

Figure 1: A schematic of the SASANet procedure. (a) Each sample is viewed as a set of features. The intermediate sequential module models an permutation-variant intermediate output as the cumulative contributions of these features. (b) The Shapley Value module trains a self-attributing network via internal Shapley value distillation, in which the attribution values proven to converge to the permutation-invariant final output’s Shapley value.
structural inductive bias. Neglecting various pre- and post-transformations for adapting to specific inputs and outputs, many existing structures providing intrinsic feature attribution matches ASA framework. Below, we introduce several representative works.

**Self-Explaining Neural Network (SENN)** (Alvarez Melis & Jaakkola (2018)). SENN models generalized coefficients for different concepts (features) and constrains the coefficients to be locally bounded by the feature transformation function. Using \(x\) to denote the explained feature unit, SENN's attribution can be formulated as \(f(x)=\sum_{i=1}^{N}a(x;\theta)_{i}x_{i}+b\), constraining that \(a(x;\theta)_{i}\) is approximately the gradient of \(f(x)\) with respect to each \(x_{i}\). By setting \(\phi(x;\theta):=[a(x;\theta)_{1}x_{1},\cdots,a(x;\theta)_{N}x_{N}]\in\mathbb{R} ^{N}\), SENN aims at the form of Eq. 1 with a constraint \(\nabla_{x_{i}}f(x;\theta,\phi_{0})\approx\phi(x;\theta)_{i}/x_{i}\) for \(i\in\mathcal{N}\). To realize the constraints, SENN adds a regularizer on the distance between the coefficients and the gradients.

**Neural Additive Model (NAM)** (Agarwal et al. (2020)). NAM models each feature's independent contribution and sums them for the prediction as \(f(x)=\sum_{i=1}^{N}h_{i}(x_{i};\theta_{i})+b\). Regarding \(h_{i}(\cdot;\theta_{i}):\mathbb{R}\rightarrow\mathbb{R}\) as a function \(\phi(\cdot;\theta)_{i}:\mathbb{R}^{N}\rightarrow\mathbb{R}\) constrained depend only on \(x_{i}\), NAM targets the form of Eq. 1 with a constraint: \(\forall x,x^{\prime}\quad x_{i}=x^{\prime}_{i}\rightarrow\phi(x;\theta)_{i}= \phi(x^{\prime};\theta)_{i}\) for \(i\in\mathcal{N}\). To realize such constraints, NAM models each feature's attribution with an independent network module.

**Self-Interpretable model with Transformation Equivariant Interpretation (SITE)** (Wang & Wang (2021)). SITE models the output as a similarity between the sample and a relevant prototype \(G(x;\theta)\in\mathbb{R}^{N}\), formulated as \(f(x)=\sum_{i=1}^{N}G(x;\theta)_{i}x_{i}+b\). Each feature's contribution term \(G(x;\theta)_{i}x_{i}\) is the attribution value. The prototype is constrained to (1) resemble real samples of the corresponding class and (2) be transformation equivariant for image inputs. Thus, SITE targets the form of Eq. 1 with constraints: \(\phi(x;\theta)_{i}=G(x;\theta)_{i}x_{i}\) for \(i\in\mathcal{N}\), \(G(x;\theta)\sim\mathcal{D}_{c}\), and \(T_{\beta}^{-1}(G(T_{\beta}(x);\theta))\sim\mathcal{D}_{c}\), where \(\mathcal{D}_{c}\) is sample distribution, \(T_{\beta}\) is predefined transformations. To realize such constraints, SITE regularizes sample-prototype distance and \(T_{\beta}\)'s reconstruction error.

**Salary-Skill Value Composition Network (SSCN)**(Sun et al. (2021)). Domain-specific studies also seek self-attributing networks. For example, SSCN models job salary as the weighted average of skills' values, formulated as \(f(x;\theta_{d},\theta_{v})=\sum_{i=1}^{N}d(x;\theta)_{i}v(x_{i};\theta_{v})\), where \(d(x;\theta_{d})_{i}\) denotes skill dominance and \(v(x_{i};\theta_{v})\) denotes skill value, defined as task-specific components in their study. Regarding each skill as a salary prediction feature, SSCN inherently targets the form of Eq. 1 with constraints: \(\phi(x;\theta)_{i}=d(x;\theta_{d})_{i}v(x;\theta_{v})_{i}\), \(\forall x,x^{\prime}\quad x_{i}=x^{\prime}_{i}\to v(x;\theta_{v})_{i}=v(x^{ \prime};\theta_{v})_{i}\), and \(\sum_{i=1}^{N}d(x;\theta_{d})_{i}=1\) for \(i\in\mathcal{N}\). To realize such constraints, SSCN models \(v\) with a single-skill network and \(d\) with a self-attention mechanism.

### Shapley Additive Self-Attributing Model

In coalition game theory, Shapley value ensures _Efficiency_, _Linearity_, _Nullity_, and _Symmetry_ axioms for fair contribution allocation, largely promoting post-hoc interpretation methods (Lundberg & Lee (2017); Bento et al. (2021)). However, there's a gap in studying self-attributing networks satisfying these axioms. Therefore, we define the Shapley ASA model as:

**Definition 2.2** (Shapely Additive Self-Attributing Model): _Shapely Additive Self-Attributing Model are ASA models that are formulated as_

\[\begin{split} f(x;\theta,\phi_{0})=&\phi_{0}+\sum_{i= 1}^{N}\phi(x;\theta)_{i},\\ \text{s.t.}\quad\phi(x;\theta)_{i}=&\frac{1}{N!} \sum_{O\in x(\mathcal{N}),k\in\mathcal{N}}\mathbb{I}\{O_{k}=i\}(v_{f}(x_{O_{1 :k-1}\cup\{i\}};\theta,\phi_{0})-v_{f}(x_{O_{1:k-1}};\theta,\phi_{0})),\end{split} \tag{2}\]

_where \(\pi(\cdot)\) denotes all possible permutations, \(x_{\mathcal{S}}:=\{x_{i}|i\in\mathcal{S}\}\) represents a subset of features in \(x\) given \(\mathcal{S}\in\mathcal{N}\) (\(x=x_{\mathcal{N}}\)), \(v_{f}\) is a predefined value function about the effect of \(x_{\mathcal{S}}\) with respect to \(f\)._

Following coalition game theory (Shapley et al. (1953)), we constrain \(\phi(x;\phi)_{i}\) to feature \(x_{i}\)'s average marginal contribution upon joining a random input subset, leading to Shapley value axioms.

**Theorem 2.3**: _In a Shapley ASA model, the following axioms hold: (Efficiency): \(\sum_{i\in\mathcal{N}}\phi(x;\theta)_{i}=v_{f}(x_{\mathcal{N}};\theta,\phi_{0} )-v_{f}(x_{\Phi};\theta,\phi_{0})\)._
_(Linearity)_: _Given Shapley ASA models \(f\), \(f^{\prime}\), and \(f^{\prime\prime}\), for all \(\alpha,\beta\in\mathbb{R}\): if \(v_{f^{\prime\prime}}=\alpha v_{f}+\beta v_{f^{\prime}}\), then \(\phi^{(f^{\prime\prime})}=\alpha\phi^{(f)}+\beta\phi^{(f^{\prime})}\), where \(\phi^{(\cdot)}\) denotes the internal attribution value of a Shapley ASA model._

_(Nullity)_: \(\forall\mathcal{S}\subset\mathcal{N}\backslash\{i\}\quad v_{f}(x_{\mathcal{S }\cup\{i\}};\theta,\phi_{0})=v_{f}(x_{\mathcal{S}};\theta,\phi_{0})\) then \(\phi(x;\theta)_{i}=0\)._

_(Symmetry)_: \(\forall\mathcal{S}\subset\mathcal{N}\backslash\{i,j\}\quad v_{f}(x_{\mathcal{ S}\cup\{i\}};\theta,\phi_{0})=v_{f}(x_{\mathcal{S}\cup\{j\}};\theta,\phi_{0})\) then \(\phi(x;\theta)_{i}=\phi(x;\theta)_{j}\)._

## 3 Method

The value function of feature subsets has inputs of variable sizes, i.e., \(v_{f}:\bigcup_{k=1}^{N}\mathbb{R}^{k}\rightarrow\mathbb{R}\), whereas most existing models accept fix-size inputs, i.e., \(f:\mathbb{R}^{N}\rightarrow\mathbb{R}\). Transforming from \(f\) to \(v_{f}\) demands approximating model output with missing inputs using handcrafted reference values (Lundberg and Lee (2017)) and a sampling procedure (Datta et al. (2016)). This is non-trivial given complex factors like feature dependency. SASANet addresses this by directly modeling \(f:\bigcup_{k=1}^{N}\mathbb{R}^{k}\rightarrow\mathbb{R}\) using set-based modeling for inputs of any size.

**Definition 3.1** (SASANet value function): _In SASANet, the value function \(v_{f}(x_{\mathcal{S}};\theta,\phi_{0})\) for any given feature subset \(x_{\mathcal{S}}\in\bigcup_{k=1}^{N}\mathbb{R}^{k}\) is the model output \(f(x_{\mathcal{S}};\theta,\phi_{0})\)._

With \(v_{f}=f\), ensuring \(f:\bigcup_{k=1}^{N}\mathbb{R}^{k}\rightarrow\mathbb{R}\) meets Definition 2.2 often involves adding a regularization term. This can cause conflicting optimization directions, hindering proper model convergence. To address this, we introduce an intermediate sequential modeling framework and an internal distillation strategy. This naturally achieves \(f:\bigcup_{k=1}^{N}\mathbb{R}^{k}\rightarrow\mathbb{R}\) compliant with Definition 2.2. The process is depicted in Figure 1, with proofs and structural details in the Appendix.

### Marginal Contribution-based Sequential Modeling

In SASANet, a marginal contribution-based sequential module generates a permutation-variant output for any feature set size, capturing the intermediate effects of each given feature. Specifically, for a given order \(O\) where features in \(\mathcal{N}\) are sequentially added for prediction, the marginal contribution of each feature is explicitly modeled as \(\triangle(x_{O_{i}},x_{O_{i:i-1}};\theta_{\triangle})\). For any feature subset \(\mathcal{S}\subset\mathcal{N}\), accumulating the marginal contributions given an order \(O_{\mathcal{S}}\in\pi(\mathcal{S})\) yields the permutation-variant output \(f_{c}(x_{\mathcal{S}},O_{\mathcal{S}};\theta_{\triangle})=\sum_{i=1}^{|\mathcal{ S}|}\triangle(x_{O_{\mathcal{S}_{i}},x_{O_{\mathcal{S}_{i:i-1}}}};\theta_{ \triangle})+\phi_{0}\). This module can be trained for various prediction tasks. For example, using \(\sigma\) to represent the sigmoid function, we can formulate a binary classification loss for a sample \(x\) as

\[L_{m}(x,y,O)=y\log(\sigma(f_{c}(x,O;\theta_{\triangle})))+(1-y)\log(1-\sigma(f_ {c}(x,O;\theta_{\triangle}))), \tag{3}\]

### Shapley Value Distillation

Following Definition 2.1, we train an attribution network \(\phi(:\theta_{\phi}):\bigcup_{k=1}^{N}\mathbb{R}^{k}\rightarrow\bigcup_{k=1}^{ N}\mathbb{R}^{k}\), producing a final permutation-invariant output \(f(x_{\mathcal{S}};\theta_{\phi})=\sum_{i\in\mathcal{S}}\phi(x_{\mathcal{S}}; \theta_{\phi})_{i}\) for any valid input \(x_{\mathcal{S}}\in\bigcup_{k=1}^{N}\mathbb{R}^{k}\). Specially, instead of directly supervising \(f\) with data, we propose an internal distillation method based on the intermediate marginal contribution module \(f_{c}\). Specifically, we construct a distillation loss for each feature \(i\in\mathcal{S}\) in variable-size input \(x_{\mathcal{S}}\in\bigcup_{k=1}^{N}\mathbb{R}^{k}\) as

\[L_{s}^{(i)}(x_{\mathcal{S}})=\frac{1}{|\mathcal{D}|}\sum_{O\in\mathcal{D}}( \phi(x_{\mathcal{S}};\theta_{\phi})_{i}-\sum_{k\in\mathcal{S}}\mathbb{I}(O_{k}= i)\triangle(x_{i},x_{O_{i:k-1}};\theta_{\triangle}))^{2}, \tag{4}\]

where \(\mathcal{D}\subset\pi(\mathcal{S})\) denotes permutations drawn for training. Training with \(L_{s}\), \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i}\) amortizes the features' effect in \(f_{c}\). We prove it naturally ensure Shapley value constraints without bringing conflict to optimization directions. For simplicity, as is typical, we assume no gradient vanishing and sufficient model expressiveness to reach optimal.

**Proposition 3.2**: _By optimizing \(L_{s}^{(i)}(x_{\mathcal{S}})\) with \(\mathcal{D}\), SASANet converges to satisfy \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i}\sim\mathcal{N}(\phi_{i}^{*},\frac{ \sigma_{i}^{2}}{M})\), where \(M=|\mathcal{D}|\), \(\phi_{i}^{*}=\frac{1}{|\mathcal{S}|!}\sum_{O\in\pi(\mathcal{S})}\sum_{k\in \mathcal{S}}\mathbb{I}(O_{k}=i\}\triangle(x_{i},x_{O_{i:k-1}};\theta_{\triangle})\), \(\sigma_{i}^{2}=\frac{1}{|\mathcal{S}|!}\sum_{O\in\pi(\mathcal{S})}\sum_{k\in \mathcal{S}}\mathbb{I}(O_{k}=i)(\triangle(x_{i},x_{O_{i:k-1}};\theta_{ \triangle})-\phi_{i}^{*})^{2}\)._

This means \(\phi\) is trained towards the averaged intermediate feature effects in \(f_{c}\). Then, we can derive the relationship between the final output \(\tilde{f}\) and the intermediate output \(f_{c}\).
**Proposition 3.3**: _By optimizing \(L_{s}^{(i)}(x_{\mathcal{S}})\) with enough permutation sampling, there is small uncertainty that the model converges to satisfy \(f(x_{\mathcal{S}};\theta_{\phi})=\frac{1}{|\mathcal{S}|!}\sum_{O\in\pi(\mathcal{ S})}f_{c}(x_{\mathcal{S}},O;\theta_{\triangle})\)._

In this way, \(f\) acts as an implicit bagging of the permutation-variant prediction of \(f_{c}\), leading to valid permutation-invariant predictions that offer stability and higher accuracy, without requiring direct supervision loss from training data. We will subsequently demonstrate how this distillation loss enables \(\phi\) to model the Shapley value of \(f\).

**Theorem 3.4**: _If \(\forall O_{1},O_{2}\in\pi(\mathcal{S})\;f_{c}(x_{\mathcal{S}},O_{1})=f_{c}(x_ {\mathcal{S}},O_{2})\), optimizing \(L_{s}^{(i)}(x_{\mathcal{S}})\) for sample \(x\)'s subsets \(x_{\mathcal{S}}\) with ample permutation samples ensures \(\phi(x;\theta_{\phi})\) converge to satisfy Definition 2.2's constraint, i.e., Shapley value in \(f(x;\theta_{\phi})\)._

While \(f_{c}\) is permutation-variant, in the subsequent section, we'll introduce a method for \(f_{c}\) to converge to label expectations of arbitrary feature subsets, thereby not only reflecting pertinent feature-label associations but also naturally inducing a permutation invariance condition.

### Feature Subset Label Expectation Modeling

Notably, training \(f_{c}\) with Eq. 3 can lead to \(f\) making good prediction for samples in the dataset, it does not guarantee meaningful outputs for feature subsets. For example, it might output 0 when any feature is missing, assigning the same Shapley value to all features as the last one takes all the credit. Although this attribution captures the model's logic, it fails to represent significant feature-label associations in the data. To address this, we target the output to reflect the label expectation for samples with a specific feature subset. Specifically, we define a loss for \(x_{\mathcal{S}}\) using training set \(D_{tr}\) as

\[L_{v}(x_{\mathcal{S}},O_{\mathcal{S}})=\frac{\sum_{(x^{\prime},y^{\prime})\in \mathcal{D}_{tr}}\mathbb{I}(x^{\prime}_{\mathcal{S}}=x_{\mathcal{S}})L_{m}(x_ {\mathcal{S}},y,O_{\mathcal{S}})}{\sum_{(x^{\prime},y^{\prime})\in\mathcal{D}_{ tr}}\mathbb{I}(x^{\prime}_{\mathcal{S}}=x_{\mathcal{S}})}. \tag{5}\]

\(L_{v}\) is designed for \(f_{c}\) instead of the Shapley module, averting conflicts with the convergence direction of \(\phi\) we have illustrated. Nonetheless, we show that this approach makes \(\phi\) to be the Shapley value of \(f\) by directing \(f_{c}\)'s output to satisfy the permutation-invariance stipulated in Theorem 3.4.

**Theorem 3.5**: _Optimizing \(L_{s}^{(i)}(x_{\mathcal{S}})\) and \(L_{v}(x_{\mathcal{S}},O_{\mathcal{S}})\) for sample \(x\)'s subsets \(x_{\mathcal{S}}\) for all permutations \(O_{\mathcal{S}}\in\pi(\mathcal{S})\) makes \(\phi\) converge to Shapley value of \(f\)._

In this way, we ensure the constraint in Definition 2.2 satisfied, while training \(f\) to make a valid prediction with a unified distillation loss. Moreover, \(L_{v}\) can make the attribution value seize real-world feature-label relevance, which we discuss as follows.

**Proposition 3.6**: _Optimizing \(L_{s}^{(i)}(x_{\mathcal{S}})\) and \(L_{v}(x_{\mathcal{S}},O_{\mathcal{S}})\) for all permutations \(O_{\mathcal{S}}\in\pi(\mathcal{S})\) makes \(\sigma(f(x_{\mathcal{S}}))\) converge to \(\mathbb{E}_{\mathcal{D}_{tr}}[y|x_{\mathcal{S}}]\)._

While we cannot exhaust all the permutations in practice, by continuously sampling permutations during training, the network will learn to generalize by grasping permutation patterns.

**Remark 3.7**: \(f\) _estimates the expected label when certain features are observed, represented by \(\int_{x^{\prime}_{\mathcal{S}}}p(x^{\prime}_{\mathcal{S}}|x_{\mathcal{S}})h( x^{\prime}_{\mathcal{S}}\cup x_{\mathcal{S}})dx^{\prime}_{\mathcal{S}},\) where \(\mathcal{S}=\mathcal{N}-\mathcal{S}\), \(h(x)\) signifies a sample \(x\)'s real label, and \(p(\cdot|x_{\mathcal{S}})\) is the conditional sample distribution given the feature set \(x_{\mathcal{S}}\). Then, \(\phi\) learns to estimate the actual feature-label Shapley value._

Previously, such analyses can be conducted by training a model with fixed-size inputs and employing post-hoc methods. These methods crafted value functions to estimate expectations based on model outputs when randomly replacing missing features. As outlined in the literature (Strumbelj & Kononenko (2014); Datta et al. (2016); Lundberg & Lee (2017)), this can be depicted as: \(\int_{x^{\prime}_{\mathcal{S}}}p(x^{\prime}_{\mathcal{S}})h^{\prime}(x^{\prime }_{\mathcal{S}}\cup x_{\mathcal{S}})dx^{\prime}_{\mathcal{S}},\) where \(h^{\prime}\) approximates the inaccessible \(h\). The sample distribution for the entire dataset is given by \(p_{gen}(x)=p(x_{\mathcal{S}})\cdot p(x_{\mathcal{S}})\), which only matches \(p\) if \(\forall x_{\mathcal{S}},x_{\mathcal{S}}\quad p(x_{\mathcal{S}})=p(x_{\mathcal{S }}|x_{\mathcal{S}})\). Given that full feature independence is unlikely, generated samples may not align with real data, causing unreliable outputs. Despite progress in post-hoc studies (Laugel et al. (2019); Aas et al. (2021)) on feature dependence, complex tasks remain challenging. ASANet sidesteps this by learning an apt value function, better estimating feature-label relations via self-attribution.
### Positional Shapley Value Distillation

The marginal contribution of a feature can fluctuate based on the prefix set size. For instance, with many observed features, a model might resist change in prediction from new ones. Such diversified contribution distribution can hinder model convergence. Therefore, instead of directly training overall attribution function with Eq. 4, we train a positional attribution function \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i,k}\) to measure feature \(i\)'s effects in \(f_{c}\) in a certain position \(k\), with an internal positional distillation loss:

\[L_{s}^{(i,k)}(x_{\mathcal{S}})=\frac{1}{|\mathcal{D}|}\sum_{O\in\mathcal{D}} \mathbb{I}[\{O_{k}=i\}(\phi(x_{\mathcal{S}};\theta_{\phi})_{i,k}-\triangle(x_ {i},x_{O_{i:k-1}};\theta_{\triangle}))^{2}. \tag{6}\]

**Proposition 3.8**: _By randomly drawing \(m\) permutations where \(x_{i}\) appears at a specific position \(k\) and optimizing \(L_{s}^{(i,k)}(x_{\mathcal{S}})\), we have \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i,k}\sim\mathcal{N}(\phi_{i,k}^{*},\frac{ \sigma_{i,k}^{*}}{m})\), where \(\phi_{i,k}^{*}=\frac{1}{(|\mathcal{S}|-1)!}\sum_{O\in\pi(\mathcal{S})} \mathbb{I}\{O_{k}=i\}\triangle(x_{i},x_{O_{i:k-1}};\theta_{\triangle})\), \(\sigma_{i,k}^{2}=\frac{1}{(|\mathcal{S}|-1)}\sum_{O\in\pi(\mathcal{S})} \mathbb{I}\{O_{k}=i\}(\triangle(x_{i},x_{O_{i:k-1}};\theta_{\triangle})-\phi_{ i,k}^{*})^{2}\)._

**Lemma 3.9**: _When optimizing \(L_{s}^{(i,k)}(x_{\mathcal{S}})\) with enough sampling and calculate \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i}=\frac{1}{|\mathcal{S}|}\sum_{i=1}^{| \mathcal{S}|}\phi(x_{\mathcal{S}};\theta_{\phi})_{i,k}\), there is small uncertainty that \(\phi(x_{\mathcal{S}};\theta_{\phi})_{i}\) converges to \(\frac{1}{|\mathcal{S}|!}\sum_{O\in\pi(\mathcal{S})}\sum_{k=1}^{|\mathcal{S}|} \mathbb{I}\{O_{k}=i\}\triangle(x_{i},x_{O_{i:k-1}};\theta_{\triangle})\)._

The overall attribution value produced this way mirrors the original distillation, equating to the Shapley value in our analysis. Hence, we term it positional Shapley value distillation.

**Proposition 3.10**: _Positional Shapley value distillation decreases SASANet's Shapley value estimation's variance._

### Network Implementation

To accommodate features of diverse meanings and distributions, we employ value embedding tables for categorical features and single-input MLPs for continuous ones, outperforming field embedding methods (Song et al. (2019); Guo et al. (2017)). The marginal contribution module applies multi-head attention for prefix representation of each feature, with position embeddings aiding in capturing sequential patterns and ensuring model convergence. Conversely, the Shapley value module uses multi-head attention for individualized sample representation, excluding position embeddings to maintain permutation-invariance. Feature embeddings are combined with prefix and sample representations for marginal contribution and positional Shapley value calculations using feed-forward networks. \(\phi_{0}\) is precomputed, symbolizing the sample-specific label expectation. Refer to the Appendix for a model schematic. For computational efficiency, terms in \(L_{s}\) and \(L_{v}\) are separated based on sample and permutations, resulting in the loss formulation \(L(x,y,O)=\sum_{k=1}^{N}(\lambda_{s}(\phi(x;\theta_{\phi})O_{k},k-\triangle(xO_ {k},x_{O_{i:k-1}};\theta_{\triangle}))^{2}+\lambda_{v}(\sum^{k}i=1\triangle(xO_ {i},x_{O_{i:k-1}};\theta_{\triangle})+\phi_{0}-y)^{2})\). In this manner, for each sample in a batch, we randomly select a single permutation rather than sampling multiple times, promoting consistent training and enhancing efficiency.

## 4 Experiments

### Experimental Setups

As depicted in Section 2.1, self-attributing models (including SASANet) can integrate pre-/post-transformations to adapt to various data types, e.g., integrate image feature extractors for image concept-level attribution, then estimate pixel-level contributions with propagation methods like upsampling. Yet, the efficacy and clarity of such attributions can be substantially compromised by the quality of the concept extraction module. To ensure clarity and sidestep potential ambiguities arising from muddled concepts or propagation methods, our evaluation uses tabular data, as it presents standardized and semantically coherent concepts. This approach foregrounds the direct impact of the core attribution layer. Specifically, we tested on three public classification tasks: Census Income Prediction (Kohavi (1996)), Higgs Boson Identification (Baldi et al. (2014)), and Credit Card Fraud Detection1, and examined SASANet's regression on the Insurance Company Benchmark2, all with
pre-normalized input features. We benchmarked SASANet against prominent black-box models like LightGBM (Ke et al. (2017)) and MLP; traditional interpretable methods such as LR and DT; and other self-attributing networks like NAM, SITE, and SENN. Hyperparameters were finely tuned for each model on each dataset for a fair comparison, and models specific to classification weren't assessed on regression tasks. Configuration and dataset specifics are in Appendices H and I. For interpretation evaluation, we did not compare with other self-attributing models since their desired feature attribution value have different intuitions and physical meanings. Instead, we compared with post-hoc methods to demonstrate that SASANet accurately conveys its prediction rationale.

### Effectiveness: Prediction Performance Evaluation

We used AUC and AP for imbalanced label classification, and RMSE and MAE for regression. Table 6 shows average scores from 10 tests; Appendix J lists standard deviations.

**Comparison to Baselines.** Black-box models outperform other compared models. Classic tree and linear methods, though interpretable, are limited by their simplicity. Similarly, current self-attributing networks often sacrifice performance for interpretability due to simplified structures or added regularizers. However, SASANet matches black-box model performance while remaining interpretable. We also show SASANet rivals separately trained MLP models in predicting with arbitrary number of missing features, indicating its Shapley values genuinely reflect feature-label relevance through modeling feature contribution to label expectation. See Appendix K for details.

**Ablation Study.** First, without the Shapley value module (i.e., "SASANet-d"), there was a large performance drop. This suggests that parameterizing Shapley value enhances accuracy. The permutation-variant predictions, despite theoretical convergence, can still be unstable in practice. The permutation-invariant Shapley value, akin to inherent bagging across permutations, offers better accuracy. Second, bypassing positional Shapley value and directly modeling overall Shapley value (i.e., "SASANet-p") resulted in performance reduction, particularly on Income and Higgs Boson datasets. This underscores the benefit of distinguishing marginal contributions at various positions.

### Fidelity: Feature-Masking Experiment

We observed prediction performance after masking the top 1-5 features attributed by SASANet for each test sample and compared it to the outcome when using KernelSHAP, a popular post-hoc method, and FastSHAP, a recent parametric post-hoc method. The results are shown in Table 7. SASANet's feature masking leads to the most significant drop in performance, indicating its self-attribution is more faithful than post-hoc methods. This stems from self-attribution being rooted in real prediction logic. Moreover, SASANet's reliable predictions on varied feature subsets make it suitable for feature-masking experiments, sidestepping out-of-distribution noise from feature replacements. Notably, FastSHAP performs the worst in feature-masking experiments. While it can approximate the Shapley value in numerical regression, it appears to struggle in accurately identifying the most important features in a ranked manner. Additionally, we conducted experiment to add the top 1-5 features from scratch, showing SASANet's selection led to quicker performance improvement. See Appendix L for details.

\begin{table}
\begin{tabular}{|c|c c c c c c c c|} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{Income} & \multicolumn{2}{c}{Higgs Boson} & \multicolumn{2}{c}{Fraud} & \multicolumn{2}{c|}{Insurance} \\  & AP & AUC & AP & AUC & AP & AUC & RMSE & MAE \\ \hline LightGBM & _0.6972_ & _0.9566_ & 0.8590 & 0.8459 & 0.7934 & 0.9737 & 0.2315 & _0.1052_ \\ MLP & 0.6616 & 0.9518 & _0.8877_ & _0.8771_ & _0.8167_ & _0.9621_ & _0.2310_ & 0.1076 \\ \hline DT & 0.2514 & 0.7250 & 0.6408 & 0.6705 & 0.5639 & 0.8614 & 0.3332 & 0.1167 \\ LR & 0.3570 & 0.8717 & 0.6835 & 0.6846 & 0.7279 & 0.9620 & - & - \\ NAM & 0.6567 & 0.9506 & 0.7897 & 0.7751 & 0.7986 & 0.9590 & 0.2382 & 0.1182 \\ SITE & 0.6415 & 0.9472 & 0.8656 & 0.8597 & 0.7912 & 0.9556 & - & - \\ SENN & 0.6067 & 0.9416 & 0.7563 & 0.7556 & 0.7709 & 0.8916 & 0.2672 & 0.1313 \\ \hline SASANet-p & 0.6708 & 0.9525 & 0.8775 & 0.8656 & 0.8090 & 0.9667 & **0.2368** & **0.0894** \\ SASANet-d & 0.6811 & 0.9527 & 0.8790 & 0.8675 & 0.8090 & 0.9665 & 0.2387 & 0.1037 \\ SASANet & **0.6864** & **0.9542** & **0.8836** & **0.8721** & **0.8124** & **0.9674** & 0.2375 & 0.0901 \\ \hline \end{tabular}
\end{table}
Table 1: Model performance - the best performance among interpretable models is in bold.
### Efficiency: Attribution Time Evaluation

We assessed attribution time for SASANet's self-attribution and two post-hoc interpreters, KernelSHAP and LIME, on 1,000 random samples. As per Table 3, SASANet was the quickest. LIME, reliant on drawing neighboring samples for local surrogates, was the slowest, especially with larger datasets. KernelSHAP, while faster than LIME due to its linear approximation and regression-based estimation, was constrained by its sampling, making it over 200 times slower than SASANet. Notably, while parametric post-hoc methods like FastSHAP (Jethani et al. (2021)) offer swift 1-pass attribution, they demand an extensive post-hoc training involving many forward propagations of the interpreted model. This was problematic for large datasets and models, such as the Higgs Boson. Conversely, self-attribution itself is the core component of model prediction and inherently acquired during model training, eliminating post-hoc sampling or regression procedure.

### Accuracy: Comparison with Ground-Truth Shapley Value

SASANet can predict for any-sized input, allowing us to directly ascertain accurate Shapley values through sufficient permutations. While getting ground truth Shapley value is time-intensive, the transformer structure in SASANet alleviates the problem since it can simultaneously produce multiple feature subsets' value by incrementally considering the attention relevant to new features. For each dataset, we sampled 1,000 test samples and estimated their real Shapley values in SASANet with 10,000 permutations. Furthermore, to test the generalization of the attribution module, we created a distribution shift dataset by adding a randomly sampled large bias to each dimension, which has the same scale as the normalized input. The RMSE of KernelSHAP, FastSHAP, and SASANet's attribution value to the real Shapley value are shown in Table 3. We have observed that SASANet's self-attribution models provide accurate estimates of the Shapley value for its own output. The performance of the attribution module may decrease when there is a distribution shift, as it is trained on the distribution of the training data in our experiment. Nevertheless, it still largely outperforms KernelSHAP and FastSHAP. Indeed, our analysis in Section 3.2 indicates that convergence to the Shapley value is ensured on the distilled sample distribution, irrespective of input distribution. Therefore, simple tricks like introducing noise during internal distillation can alleviate this issue without harming the effectiveness of the model.

### Qualitative Evaluation

**Overall Interpretation.** We compared feature attributions from SASANet and KernelSHAP for the entire test set. In Figure 2 (a), SASANet reveals clear attribution patterns. Specifically, for the Income dataset, there's evident clustering in attribution values tied to specific feature values. Such sparsity of attribution value aligns with the feature value, which are mostly categorical. The Higgs

\begin{table}
\begin{tabular}{|c|c c c c c c c c c c c|} \hline \multirow{2}{*}{Task} & \multirow{2}{*}{Method} & \multicolumn{3}{c|}{Top 1} & \multicolumn{3}{c|}{Top 2} & \multicolumn{3}{c|}{Top 3} & \multicolumn{3}{c|}{Top 4} & \multicolumn{3}{c|}{Top 5} \\  & & AP & AUC & AP & AUC & AP & AUC & AP & AUC & AP & AUC \\ \hline Income & SASA. & **0.560** & **0.929** & **0.497** & **0.917** & **0.441** & **0.904** & **0.398** & **0.892** & **0.361** & **0.880** \\  & KerSH. & 0.606 & 0.939 & 0.566 & 0.932 & 0.547 & 0.928 & 0.532 & 0.926 & 0.518 & 0.923 \\  & FastSH. & 0.683 & 0.954 & 0.617 & 0.944 & 0.617 & 0.944 & 0.618 & 0.944 & 0.613 & 0.943 \\ \hline Higgs & SASA. & **0.825** & **0.813** & **0.777** & **0.764** & **0.730** & **0.715** & **0.681** & **0.665** & **0.632** & **0.616** \\  & KerSH. & 0.855 & 0.843 & 0.833 & 0.821 & 0.808 & 0.795 & 0.783 & 0.770 & 0.760 & 0.746 \\  & FastSH. & - & - & - & - & - & - & - & - & - & - \\ \hline Fraud & SASA. & **0.789** & **0.962** & **0.758** & **0.957** & **0.681** & **0.952** & **0.625** & **0.938** & **0.526** & **0.904** \\  & KerSH. & 0.806 & 0.963 & 0.782 & 0.961 & 0.732 & 0.959 & 0.693 & 0.949 & 0.627 & 0.936 \\  & FastSH. & 0.813 & 0.967 & 0.813 & 0.967 & 0.815 & 0.965 & 0.811 & 0.965 & 0.809 & 0.966 \\ \hline \end{tabular}

* FastSHAP results for the Higgs dataset are missing due to prolonged training times.

\end{table}
Table 2: Results of feature masking experiments.

\begin{table}
\begin{tabular}{|c|c c c|c c c c|c c c|} \hline \multirow{2}{*}{Method} & \multicolumn{3}{c|}{Time} & \multicolumn{3}{c|}{RMSE} & \multicolumn{3}{c|}{RMSE (Distribution Shift)} \\  & Income & Higgs & Fraud & Income & Higgs & Fraud & Income & Higgs & Fraud \\ \hline LIME & 1604.9s & 11792.7s & 9851.2s & - & - & - & - & - & - & - \\ KernelSHAP & 91.2s & 65.7s & 38.4s & 0.348 & 0.402 & 0.514 & 0.348 & 0.405 & 0.494 \\ FastSHAP\({}^{*}\) & **0.3s\({}^{*}\)** & - & **0.2s\({}^{*}\)** & 0.332 & - & 0.473 & 0.522 & - & 0.550 \\ SASANet & **0.3s** & **0.3s** & **0.2s** & **0.001** & **0.005** & **0.033** & **0.001** & **0.113** & **0.082** \\ \hline \end{tabular}

* FastSHAP’s has time-consuming post-hoc training procedure, whose time isn’t reported to prevent confusion. FastSHAP results for the Higgs dataset are missing due to prolonged training times.

\end{table}
Table 3: Time cost and accuracy for feature attribution.
Boson dataset shows SASANet's attributions tend to align with feature value changes, confirming its ability to identify pertinent feature-label relationships. For Higgs Boson dataset, SASANet's attribution value shows rough monotonic associations with feature values. For example, a larger feature value tends to bring positive attribution for "m_jlv" while a negative attribution value for "m_bb". This suggests SASANet identifies feature-label relevance consistent with common sense. That is, despite complex quantitative relationship between features and outcomes, qualitative patterns are usually clear. Meanwhile, clear differences exist between real self-attribution and approximated post-hoc attribution, highlighting potential inaccuracies in post-hoc methods. For example, in both datasets, many features have abnormally high attribution values in KernelSHAP. This may stem from neglecting feature interdependencies, evident upon sample interpretation in the subsequent part.

**Sample Interpretation.** We randomly draw a positive sample in Higgs Boson dataset predicted correctly by SASANet and compared its self-attribution with KernelSHAP's attribution in Figure 2 (b). Distinct differences are evident. Notably, SASANet indicates all top-9 features contribute positively, highlighting a simple, intuitive logic. Indeed, the objective of Higgs Boson prediction is to differentiate processes that produce Higgs bosons from those that do not. As a result, the model distinguishes the distinctive traits of Higgs bosons from the ordinary, resulting in a consistent positive influence from key features representing these unique attributes. KernelSHAP sometimes assigns disproportionately large negative values to features. For instance, the attribution for "m_wbb" (\(-6.26\)) significantly surpasses SASANet's peak attribution of \(0.7\). Yet, they aggregate to a final output of \(f(x)=3.959\) as opposing attributions from dependent features, like the positive contribution of "m_jlv" (\(6.14\)), neutralize each other. Notably, SASANet's self-attribution doesn't deem either m_wbb" or "m_jlv" as so crucial. Similar phenomenon can be observed from other datasets and samples, which are visualized in Appendix M. For example, in Income prediction task, indicators exist for both rich and poor people. Accordingly, SASANet identifies input features indicating both outcomes. However, it is still obvious that KernelSHAP attributes exaggerated values to features, which then offset one another. This underscores how KernelSHAP can be deceptively complex by neglecting feature interdependencies, an issue that has been a pitfall in numerous post-hoc studies (Laugel et al. (2019); Frye et al. (2020); Aas et al. (2021)). SASANet naturally handles feature dependency well since the value functions has explicit physical meaning as estimated label expectation and are directly trained under the real data distribution.

## 5 Concluding Remarks

We proposed Shapley Additive Self-Attributing Neural Network (SASANet) and proved its self-attribution aligns with the Shapley values of its attribution-generated output. Through extensive experiments on real-world datasets, we demonstrated that SASANet not only surpasses current self-interpretable models in performance but also rivals the precision of black-box models while maintaining faithful Shapley attribution, bridging the gap between expressiveness and interpretability. Furthermore, we showed that SASANet excels in interpreting itself compared to using post

Figure 2: Feature attribution visualizations: x-axis shows attribution value; y-axis lists top features by decreasing average absolute attribution. (a) Overall attribution. (b) Single-instance attribution.
hoc methods. Moreover, our theoretical analysis and experiments suggest broader applications for SASANet. Firstly, its self-attribution offers a foundation for critiquing post-hoc interpreters, pinpointing areas of potential misinterpretation. Furthermore, its focus on label expectation uncovers intricate non-linear relationships between real-world features and outcomes. In the future, we will use SASANet to gain further insights into model interpretation and knowledge discovery studies.
