# Mechanistic Interpretation through Contextual Decomposition in Transformers

Aliyah Hsu  Yeshwanth Cherapanamjeri  Anobel Y. Odisho  Peter R. Carroll  Bin
Yu

###### Abstract

Transformers exhibit impressive capabilities but are often regarded as black
boxes due to challenges in understanding the complex nonlinear relationships
between features. Interpreting machine learning models is of paramount
importance to mitigate risks, and mechanistic interpretability is in
particular of current interest as it opens up a window for guiding manual
modifications and reverse-engineering solutions. In this work, we introduce
contextual decomposition for transformers (CD-T), extending a prior work on CD
for RNNs and CNNs, to address mechanistic interpretation computationally
efficiently. CD-T is a flexible interpretation method for transformers. It can
capture contributions of combinations of input features or source internal
components (e.g. attention heads, feed-forward networks) to (1) final
predictions or (2) the output of any target internal component. Using CD-T, we
propose a novel algorithm for circuit discovery. On a real-world pathology
report classification task: we show CD-T distills a more faithful circuit of
attention heads with improved computational efficiency (speed up 2x) than a
prior benchmark, path patching. As a versatile interpretation method, CD-T
also exhibits exceptional capabilities for local interpretations. CD-T is
shown to reliably find words and phrases of contrasting sentiment/topic on
SST-2 and AGNews datasets. Through human experiments, we demonstrate CD-T
enables users to identify the more accurate of two models and to better trust
a model’s outputs compared to alternative interpretation methods such as SHAP
and LIME.

Machine Learning, Explainable AI, Interpretation, Transformers

##  1 Introduction

Transformers (Vaswani et al., 2017) have recently demonstrated impressive
predictive capabilities (Brown et al., 2020) by learning intricate nonlinear
relationships between features. However, the challenge of comprehending these
relationships has resulted in transformers largely considered as black boxes.
Despite this, transformers are increasingly utilized in high-stakes domains
such as medicine (e.g. medical image analysis (He et al., 2023)) and science
(e.g. protein structure prediction aiding drug discovery (Jumper et al.,
2021)). This underscores the necessity of understanding and anticipating
potential model behaviors. To strengthen trust in the deployment of advanced
black-box models like transformers, researchers emphasize the urgent need for
reliably interpreting them (Hendrycks et al., 2023; Räuker et al., 2023) to
mitigate risks and address issues like fairness (Nemani et al., 2024).
Mechanistic interpretability, a study to explain behaviors of machine learning
(ML) models in terms of their internal components, is at the frontier of
interpretability research (Geiger et al., 2021; Geva et al., 2020; Räuker et
al., 2023), as it uniquely provides an avenue for guiding manual modifications
(Elhage et al., 2021; Vig et al., 2020) and reverse-engineering solutions
(Elhage et al., 2022; Meng et al., 2023).

In this work, we introduce contextual decomposition for transformers (CD-T)
111Code for all experiments is available in the supplementary material., a
novel interpretation method that enables mechanistic interpretation. It
explains contributions of combinations of input features or source internal
components (e.g. attention heads, feed-forward networks) to (1) final
predictions or (2) the output of arbitrary target internal component, without
any modifications to the underlying model. Our proposed method, CD-T, is a
general technique that can be applied to a wide range of standard transformer-
based models and data types.

This work consists of three novel contributions. First, the development of
CD-T, which is a generalization of CD, a previous method for obtaining
importance scores for CNNs and RNNs (Murdoch et al., 2018; Singh et al.,
2018), to transformers. As transformers became a dominant deep learning (DL)
architecture in state-of-the-art applications, this generalization ensures
CD-T benefits a broader group of practitioners. Second, in addition to
individual prediction analysis (i.e. contributions of combinations of input
features to final predictions), CD-T is the first to also enable mechanistic
interpretability (i.e. contributions of combinations of source internal
components to final predictions or the output of target internal component).
Finally, we propose a computationally efficient algorithm for circuit
discovery using CD-T. Overall, CD-T opens a window for practitioners to
understand interactions of internal components in transformers beyond local
interpretations, which is necessary for better predicting model behaviors or
identifying and fixing model errors.

We showcase CD-T’s ability to provide mechanistic interpretability on BERT
trained on real-world pathology reports by distilling a more faithful circuit
of attention heads with improved computational efficiency (speed up 2x) than a
previous benchmark, path patching (Wang et al., 2023), which utilizes a
combination of interpretability approaches relying on causal interventions for
circuit discovery. We demonstrate the utility of CD-T for local
interpretations on BERTs (Devlin et al., 2019) trained on Stanford Sentiment
Treebank (SST-2) (Socher et al., 2013) and AGNews (Zhang et al., 2015) by
showing CD-T reliably identifies words and phrases of contrasting
sentiment/topic. Through human experiments, we show that CD-T enables users to
better reason about and trust the models. Specifically, given two transformer
models, we show that the users can use the output of CD-T to select the model
with higher predictive accuracy, and that overall they rank CD-T as more
trustworthy than prior interpretation methods including LIME (Ribeiro et al.,
2016), SHAP (Lundberg & Lee, 2017).

##  2 Related Work and Connections to Our Work

Interpreting deep neural networks (DNNs) is a growing field (Adadi & Berrada,
2018; Murdoch et al., 2019; Räuker et al., 2023) which encompasses a broad set
of techniques including adversarial techniques (Carmichael & Scheirer, 2023;
Chen et al., 2019), input attribution methods (Sundararajan et al., 2017;
Ribeiro et al., 2016), mechanistic interpretability methods (Zhao et al.,
2020; Wang et al., 2023), and others (Mao et al., 2019; Hsu et al., 2023). Our
work focuses on both local interpretations (i.e. interpret individual
predictions made by a DNN) and mechanistic interpretability (i.e. explain
behaviors of DNNs in terms of their internal components).

##### Local interpretation

The majority of previous research has concentrated on attributing local
importance to individual features, such as pixels in an image or words in a
document. Various methods exist to assign feature-level importance for
different architectures, including gradient-based (Sundararajan et al., 2017;
Springenberg et al., 2014; Selvaraju et al., 2016; Baehrens et al., 2009),
decomposition-based (Murdoch & Szlam, 2016; Shrikumar et al., 2016; Bach et
al., 2015) and others (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017; Ribeiro et
al., 2016; Zintgraf et al., 2017). Ancona et al. (2017) and Lundberg & Lee
(2017) discussed rigorously the similarities among the methods.

Specifically for LSTMs, Murdoch et al. (2018) highlighted the shortcomings of
previous interpretation methods relying on word-level scores. They introduced
contextual decomposition (CD), an algorithm capable of extracting feature
interactions learned by LSTMs by generating phrase-level importance scores.
Singh et al. (2018) extended CD to RNNs and CNNs, and proposed an hierarchical
interpretation method using feature clustering. To capture feature
interactions in transformers, Tsang et al. (2020) proposed an axiomic feature
attribution framework and Hao et al. (2020) proposed a self-attention
attribution method.

However, no existing work is versatile enough to be able to provide individual
prediction analysis while enabling mechanistic interpretability to interpret
interactions of internal components in DNNs. Such interpretation tool would
benefit practitioners greatly because of its diverse use cases. To address
this problem, this work introduces CD-T as a principled way to, for the first
time, provide both local interpretation and mechanistic interpretability for
transformers, a critical DL architecture in state-of-the-art applications.

##### Mechanistic interpretation

Work in mechanistic interpretability aims to explain and predict behaviors of
DNNs by understanding the inner-working of the models. Previous research has
focused on understanding features learned by DNNs (Olah et al., 2017; Elhage
et al., 2022), developing mathematical frameworks for understanding DNNs
(Elhage et al., 2021), and discovering circuits (computational subgraphs) in
DNNs (Nanda et al., 2023; Cammarata et al., 2021; Chughtai et al., 2023; Wang
et al., 2023). Most existing work on circuit discovery require extensive
feature visualizations or multiple passes of inference runs to perform causal
interventions, both demanding much manual effort and computation resource. To
remedy this, we develop a computationally efficient procedure (speed up 2x
compared to a prior benchmark, path patching (Wang et al., 2023)) using CD-T
to iteratively trace important components back from the logits to discover
circuits in transformers.

##  3 Our Method: CD-T and Mechanistic Interpretability

In this section, we describe the extension of contextual decomposition to
transformers, CD-T, and its use in machanistically interpreting the behavior
of these models. When given a set of source and target activations in the
network, CD-T produces a decomposition of the target activations into two
components – one reflecting the contributions of the source activations and
the other, the contributions of the rest of the network. While we describe
CD-T specifically for BERT-based models, it generalizes straightforwardly to
more general attention-based models including decoder-only models such as
GPT-4. Since contextual decompositions are computed by propagating a
decomposition of the input through the nodes of the network, our primary
contribution here will concern with the propagation of an input decomposition
through the (self-)attention module. This is because all other modules in a
typical transformer (such as linear transformations and the application of
element-wise non-linear activation functions) have been addressed by prior
work (Murdoch et al., 2018).

For the rest of the section, we first recall the basic operations of BERT-
based models (Section 3.1). We then present the extension of the contextual
decomposition framework to transformers (Section 3.2) before concluding with a
description of its use in a circuit building algorithm for mechanistic
interpretability (Section 3.3).

###  3.1 Transformers

In BERT-based models, the input is typically represented as a sequence of
tokens where the first token is always the classification token, usually
denoted by [CLS]. For each token in the sequence of length $l$,
$\\{t_{i}\\}_{i=1}^{l}$, a $d$-dimensional embedding is constructed
$\\{x_{i}\\}_{i=1}^{l}$ with $x_{i}\in\mathbb{R}^{d}$. This embedding encodes
information from the value of the token $t_{i}$ as well as its position in the
sequence $i$. These embeddings are then propagated through a series of encoder
modules where each encoder module takes as input a sequence of embeddings
$\\{x_{i}\\}_{i=1}^{l}$ and outputs another sequence
$\\{x^{\prime}_{i}\\}_{i=1}^{l}$ with $x^{\prime}_{i}\in\mathbb{R}^{d}$.
Since, the number and dimensionality of the token embeddings remain constants
through any encoder module, a series of these may be applied to obtain
progressively more sophisticated representations of the input sequence. The
key component that enables sharing of information across the various tokens in
the sequence is the self-attention module which we describe in detail. The
self-attention module comprises $N_{A}$ independent attention heads with each
one taking as input a sequence $\\{x_{i}\\}_{i=1}^{l}$ with
$x_{i}\in\mathbb{R}^{d}$ and producing an output $\\{y_{i}\\}_{i=1}^{l}$ with
$y_{i}\in\mathbb{R}^{d_{A}}$ where $d_{A}=d/N_{A}$. For each attention head,
$a$, there exists key $f_{k}:\mathbb{R}^{d}\to\mathbb{R}^{k}$, query
$f_{q}:\mathbb{R}^{d}\to\mathbb{R}^{k}$, and value
$f_{v}:\mathbb{R}^{d}\to\mathbb{R}^{d_{a}}$. In most transformer models, these
are either simple linear transformations or a linear transformation followed
by a position-wise non-linearity. The output of the attention-head is now
defined by the following equations:

$\displaystyle\forall
i\in[l]:k_{i}=f_{k}(x_{i}),q_{i}=f_{q}(x_{i}),v_{i}=f_{v}(x_{i})$
$\displaystyle\forall
i,j\in[l]:\alpha_{i,j}=\frac{\exp(q_{i}^{\top}k_{j}/\sqrt{d_{k}})}{\sum_{m\in[l]}\exp(q_{i}^{\top}k_{m}/\sqrt{d_{k}})}$
$\displaystyle\forall i\in[l]:y_{i}=\sum_{j=1}^{l}\alpha_{i,j}v_{j}.$

In the above display, the last equation computes the output of attention head
as a linear combination of the value vectors, $v_{j}$, with the weights
determined by inner product of the ith query vector $q_{i}$ with the key
vectors, $k_{j}$. Stacking the outputs of each of the attention heads, we
obtain the output of the self-attention module.

###  3.2 Contextual decomposition for transformers (CD-T)

We will now describe our implementation of Contextual Decomposition.
Contextual Decomposition (CD) propagates a decomposition of the input (or of
the activation of the transformer at any layer of the transformer) through the
model. Formally, given a decomposition of an input vector
$x=\beta+\gamma\in\mathbb{R}^{d}$ where $\beta$ represents the _relevant_
portion and $\gamma$ the _irrelevant_ portion, Contextual Decomposition
defines a set of rules which determine the decomposition of the _output_ of a
module $f:\mathbb{R}^{d}\to\mathbb{R}^{k}$ which operates on $x$. For
instance, for the case of element-wise $\operatorname*{ReLU}$ activation
function, the output decomposition of Contextual Decomposition
($f(x)=\beta^{o}+\gamma^{o}$) is defined as follows:

$\displaystyle\beta^{o}=\frac{1}{2}([\operatorname*{ReLU}(\beta)]+[\operatorname*{ReLU}(\beta+\gamma)-\operatorname*{ReLU}(\gamma)])$
$\displaystyle\gamma^{o}=\frac{1}{2}([\operatorname*{ReLU}(\gamma)]+[\operatorname*{ReLU}(\beta+\gamma)-\operatorname*{ReLU}(\beta)])$

We refer the interested reader to Murdoch et al. (2018) for decompositions of
other modules such as linear transformations. As previously explained, the
only module not accounted for in the context of transformers is the self-
attention module described in Section 3.1.

In the context of transformers, we assume a decomposition of the input to the
attention head, $\\{x_{i}=\beta_{i}+\gamma_{i}\\}_{i=1}^{l}$ where $\beta_{i}$
and $\gamma_{i}$ denote the relevant and irrelevant portions of the input. We
index the decomposition with the position in the sequence for ease of
presentation. We compute the decomposition of the output of the attention head
$\\{y_{i}=\beta^{a}_{i}+\gamma^{a}_{i}\\}_{i=1}^{l}$ as follows:

$\displaystyle\forall i\in[l]:f_{v}(x_{i})=\beta^{v}_{i}+\gamma^{v}_{i}$
$\displaystyle\forall
i\in[l]:\beta^{a}_{i}=\sum_{j=1}^{l}\alpha_{i,j}\beta^{v}_{j},\
\gamma^{a}_{i}=\sum_{j=1}^{l}\alpha_{i,j}\gamma^{v}_{j}.$

Note that we do not decompose the attention weights, $\alpha_{i,j}$, into
relevant and irrelevant components. While it is possible to do so within the
framework, we found that a simple decomposition of only the value vectors not
only eased the effort of implementing the method but also produced reliable
decompositions of the inputs. As evidenced by our experimental results, CD-T
allows for a mechanistic interpretation of the whole model when combined with
a circuit building algorithm (Section 4.2) and accurate identification of the
importance of various input features to a model’s performance when used on its
own (Section 4.4).

###  3.3 Mechanistic interpretation with CD-T

Having formally defined CD-T in Section 3.2, we now introduce a novel,
computationally efficient algorithm for mechanistic interpretation in
transformers via circuit discovery using CD-T. Before delving into our
algorithm, we first formally define a circuit. If we view a model as a
computational graph $M$, where nodes are activations of the model components
in its forward pass (e.g. input embeddings, attention heads) and edges are the
interactions between those components (e.g. an attention module, position-wise
feed-forward networks), a circuit $C$ is a subgraph of $M$ responsible for the
behavior of some component of the network, such as the output logits of a
prediction task. Given an input $x$, similarly as to how the entire model
defines a function $M(x)$ from inputs to logits, we also associate each
circuit $C$ with a function $C(x)$, defined by ablating away the effect of all
components in $M\backslash C$ (i.e. the components not included in $C$)
leading up to the target component of the circuit $C$. This method improves
the interpretability of the entire model by distilling it into a small-sized
circuit which nevertheless, faithfully explains most of its behavior.

Next, we present our algorithm for constructing circuits with CD-T where we
focus specifically on computing circuits with the output logits as the target
component. Our algorithim, starting from the output logits of the network,
constructs a circuit by iteratively identifying vital internal components
through the various layers of the network. In each iteration, we define a
source component $s$ and a target set of receivers $R$, a set of internal
components (for instance, these are the output logits in the first iteration),
and our goal is to measure the direct effect of $s$ on $R$. Here, we impose
the restriction that all components in $R$ have to be downstream of $s$ – i.e.
we only search for influential $s$ in the same layer or layers before (with
layer index smaller than or equal to) the components in $R$. In path patching
(Wang et al., 2023), a prior method proposed for circuit discovery in GPT-2
small, this is achieved by ablating $s$ with its mean response, computing the
resulting activations of $R$ after this change (with one inference pass), and
measuring the difference in the output logits with another inference pass
after substituting the activations of $R$ with the new values just computed.
In other words, path patching requires two passes of inference runs to measure
the direct effect on $R$ for just one $s$, which could lead to large
computational costs when iterating through all possible choices for $s$ to
decide which components have the most influence on $R$.

By mathematically decomposing activations of arbitrary internal components at
any level and the output logits in one pass, CD-T enables more efficient
measurement of the direct effect of $s$ on $R$. To measure the direct effect
of $s$ on $R$, we mean-ablate $s$ by designating the mean response of $s$ to
the irrelevant part of its decomposition and move the residual (difference
between its original activation and the mean) to the relevant part, and
average the (absolute values of the) propagated relevant parts of the
activations of $R$ as the direct effect of $s$ on $R$. This can be done in
just one inference run, providing a speed-up of 2x over path-tracing.
Formally, letting for any $r\in R$,
$\beta^{r},\gamma^{r}\in\mathbb{R}^{d_{r}}$ denote the relevant and irrelevant
decomposition provided by CD-T measuring the effect of $s$ on $r$, the
contribution score of $s$ on $R$ is defined as follows:

$H(s,R)\coloneqq\frac{1}{\sum_{r\in R}d_{r}}\sum_{r\in
R}\sum_{i=1}^{d_{r}}|\beta^{r}_{i}|$ (1)

To build a circuit iteratively, we start by identifying attention heads ($s$)
influential to the output logits ($R$) in the first iteration, and use the
attention heads identified here as the new target components $R$ in next
iteration, and repeat this process until we reach the input layer of the
network. The algorithm is formally presented in Alg. 1.

Algorithm 1 Building a Circuit using CD-T

Input: data $x$, mean activations $M$, number of top attention heads to
extract $N$

Compute all activations on $x$

Denote $\mathrm{activation}(s,x)$ as the activation of $s$

Initialize $C$ to store the circuit

Initialize circuit level counter $k=0$

{# Base case}

Set $R$ to be the output logits of the model

repeat

for each attention head $s$ upstream of $R$ do

Set the relevant and irrelavant decomposition of $s$ to
$\mathrm{activation}(s,x)-M[s]$ and $M[s]$ respectively

Propagate decomposition to nodes in $R$ with CD-T

Use decomposition to compute $H(s,R)$ (Eq. (1))

end for

Set $R^{\mathrm{new}}$ to be the top N heads with highest $H(s,R)$

$R=R^{\mathrm{new}}$, $C[k]=R^{\mathrm{new}}$, $k+=1$

until no upstream attention heads to $R$ are available

return $C$

##  4 Experimental Results

We now present empirical validation of CD-T on BERT-based models. In Sec. 4.2,
we distill an attention head circuit using CD-T in BERT trained on real-word
pathology reports. Our algorithm is then evaluated against a prior benchmark
called path patching (Wang et al., 2023), qualitatively through circuit
visualizations, and quantitatively through comparisons of computational
efficiency and faithfulness (how much full model performance can a circuit
achieve for the task). In Sec. 4.3 and Sec. 4.4, we focus on CD-T’s ability to
provide local interpretations for BERTs trained on SST-2 and AGNews. The two
datasets were chosen to demonstrate CD-T’s capabilities on tasks with
different levels of difficulty, with SST-2 being a simpler binary
classification task with shorter samples, and AGNews a harder classification
task with longer texts and diverse topics.

###  4.1 Experimental setups

To understand the utility of CD-T for real-world use cases in critical domains
such as medicine, we collected a corpus of 2907 structured pathology reports
under an institutional review board (IRB) approval. The corpus includes
pathology reports for patients that had undergone radical prostatectomy for
prostate cancer at the University of California, San Francisco (UCSF) from
2001 to 2018. The reports contain an average of 471 tokens, much longer than
samples in SST-2 or AGNews. Our pathology reports dataset is not publicly
available due to protected patient information; however, we provide a few
anonymized samples in Appendix A.1 as illustrations. We fine-tune an uncased
base BERT model on primary Gleason score classification using standard best
practices (See Appendix A.2 for fine-tuning details), and the model attains an
accuracy of 85.8%.

In the circuit discovery experiment, we obtained mean activations of all
components in the model for mean ablations by averaging over 500 pathology
report samples. To ensure stability, we extracted 20 candidate circuits 222We
found the compositions of candidate circuits didn’t differ much, usually just
the difference of 1-2 attention head(s) in some levels of the circuits. by
running Alg. 1 on 20 randomly selected report samples, and a final circuit was
determined by groups of attention heads that appear with the highest frequency
among the candidate circuits for each level. In this paper, we show results
from extracting 6 attention heads for each level of the circuit by setting
$N=6$ in Alg. 1. We experimented with $N=1,3,6$, and empirically found setting
$N=6$ yielded a more stable circuit composition.

For SST-2 and AGNews, we use the fine-tuned models initialized with uncased
base BERT that are available on TextAttack (Morris et al., 2020). They attain
accuracies of 92.4% and 95.1% separately. The weakened models for the human
evaluations are obtained from the original models by randomly permuting a
small percentage of their weights, following a similar setup as in Singh et
al. (2018). For SST-2 and AGNews, 5% and 10% of weights are randomized,
reducing test accuracy from 92.4% and 95.1% to 60.9% and 66.7%.

###  4.2 Discovering circuits of attention heads in transformers

In this experiment, we focus on building a circuit of attention heads for a
real-world primary Gleason classification task, and evaluate our algorithm
against a prior method, path patching (Wang et al., 2023), qualitatively on
circuit visualizations, and quantitatively on computational efficiency as well
as circuit faithfulness (how much of a full model’s performance can a circuit
account for).

####  4.2.1 Circuit visualizations

After distilling the final circuit in the fine-tuned BERT for primary Gleason
classification, we investigate the functionality of attention head groups at
each level of the circuit qualitatively by inspecting word clusters each of
the attention head group pays most attention to. In Wang et al. (2023), this
is done in a more nuanced fashion with positions information also taken into
account, on their custom indirect object identification dataset, which comes
with word-level labels. However, our pathology reports dataset is a more
general case without the rigid structural restrictions of Wang et al. (2023)
and such word-level labels are not available. As a remedy, we introduce a
novel procedure to aid the interpretation of attention head groups in such
general cases.

Given a group of attention heads from a level in the final circuit, we first
compute their average attention map, standardize the map, and select words
with attention scores that are 2 333We empirically found this threshold
yielded a reasonable amount of words to select. standard deviations higher
than the mean. Next, we convert the selected words to their word2vec (Mikolov
et al., 2013) embeddings and run k-means clustering after performing PCA on
the embeddings to obtain influential word clusters for the given attention
head group. We visualize the final circuit obtained using CD-T and its
influential word clusters in Fig. 1.

As a means of mechanistic interpretation, the circuit visualization in Fig. 1
can help us reason about the inner knowledge learned in the attention heads,
and how the learned knowledge is structured hierarchically to solve a real-
world primary Gleason classification task. From the result, we see for all
levels, the influential words can be roughly clustered into three groups of
distinct concepts: punctuation/numbers, biomedical terms, and helper words
that often appear in the vicinity of Gleason scores or biomedical terms, such
as ’pattern’, ’grade’, and ’specimen’. In addition, instead of attending
directly to the Gleason scores, attention heads closer to the output logits,
at layers 11 or 10, focus more on helper words (e.g. ’score’, ’grade’,
’pattern’) and punctuation (e.g. ’+’, ’=’) that also often appear next to
Gleason scores in sentences such as ’gleason grade 3 + 4 = 7’. It was only
until backtracking further to attention heads at layers 7-9 that we start to
see the exact Gleason score (e.g. ’3’, ’4’, ’5’) become influential. This
showcases the nuance of how groups of attention heads encode different aspects
of knowledge from the reports useful for making predictions, and how the
knowledge is structured hierarchically in the model. The circuit and its
visualization provides an avenue for practitioners to diagnose errors and
modify models effectively and efficiently for desirable behaviors. We provide
the visualization of the circuit obtained using path patching in Appendix A.6.
It overall exhibits the same trend as what we see in the circuit obtained
using CD-T, except that it only has three levels, and the middle level attends
much less to medical terms relevant to prostate cancer.

Figure 1: Circuit visualization for primary Gleason classification on
pathology reports. Each level of the circuit is represented by a yellow box
specifying the exact attention heads ([(layer, token position, head
position)], indexing starts with 0) extracted and a visualization of
influential word clusters for that level. The area of a gray circle in each
word clusters visualization indicates the average attention scores received
for that word. The bigger the circle, the more influential a word is for a
given level.

####  4.2.2 Computational efficiency and faithfulness

Having interpreted the circuit obtained using CD-T and compared it with the
circuit obtained using path patching qualitatively in the previous section, we
now perform quantitative evaluation of the two methods to showcase the
benefits of CD-T for aiding mechanistic interpretability. We evaluate the two
methods in terms of (1) computational efficiency and (2) faithfulness (how
much of the full model’s performance can a circuit account for). For
computational efficiency, we simply measure the average system runtime for
each method to complete one iteration of the direct effect computation for
building one level in the circuit. For faithfulness, we measure the recovery
percentage for full model performance as
$\mathbb{E}_{x}\left[\frac{C(x)}{M(x)}\right]\times 100\%$ over 200 report
samples where only the logits of the true label-class are used to compute the
ratio. The results are in Table 1 and Table 2.

Table 1: Comparison of average system runtime for building one level of a circuit using CD-T and path patching on pathology reports. | Runtime
---|---
CD-T | 1:52:20
Path patching | 3:37:26
Table 2: Comparison of faithfulness of circuits obtained using CD-T, path patching, and randomly selecting N attention heads in the model (denoted as Random (N)) on pathology reports. | Faithfulness
---|---
CD-T | 46.0%
Path patching | 41.9%
Random (30) | 1.26%
Random (18) | 1.08%

In Table 1, we observe CD-T is almost two times more efficient in the
computation for building circuits compared to path patching, which is due to
the fact that CD-T halves the amount of inference runs needed in the
algorithm, as described in Sec. 3.3. In Table 2, with only a small total
amount of attention heads (0.04% of the attention heads in the full model), we
show the circuit obtained using CD-T, outperforms path patching by being able
to recover 46.0% of the full model performance, while path patching only
achieves 41.9% of that using 0.03% of the attention heads in the full model.
We additionally provide the faithfulness performance of two random baselines,
denoted as Random (N), obtained by randomly selecting N attention heads in the
model to form circuits, where we set N to match the amount of attention heads
in the two circuits using CD-T ($N=30$) and path patching ($N=18$). Both CD-T
and path patching substantially outperform their corresponding random
baselines. The performance of random baselines is averaged over 10 random seed
runs.

###  4.3 Identifying top-scoring phrases

Mechanistic interpretation while providing more information, is often
challenging due to the range of user choices in their definition which
frequently require sophisticated understanding of the underlying model. Often
simpler or local interpretations are desirable. To qualitatively evaluate
CD-T’s ability to provide local interpretations, we inspect the most important
features and interactions to understand what a model has learned by extracting
top-scoring phrases of varying length using CD-T for BERTs trained on SST-2
and AGNews. We show a truncated version of the results on SST-2 in Table 3,
and refer the readers to Appendix A.3 for full results. In Table 3, the
phrases were extracted by running CD-T separately on each sample in SST-2’s
validation set. The extracted phrases distinctly reflect the corresponding
sentiment, providing evidence that CD-T is able to capture meaningful positive
and negative phrases. The extracted phrases for AGNews (see in Appendix A.3)
are also clearly reflective of the different topics: world, sports, business,
and sci-tech, validating again, qualitatively, that CD-T is able to reliably
identify important features and interactions contributing to predictions.

Table 3: Top-scoring phrases of different lengths extracted by CD-T on SST-2’s validation set. The positive/negative phrases identified by CD-T are all indeed positive/negative. See full results in Appendix A.3 Length |  Positive |  Negative
---|---|---
1 |  ’power’, ’fun’ |  ’not’, ’awful’
3 |  ’a beautiful madness’, ’is brilliant as’ |  ’but lead nowhere’, ’the movie fails’
5 |  ’a deep and meaningful film’, ’it’s worth checking’ |  ’a real downer?’, ’feels too formulaic and’
8 |  ’binoche makes it interesting trying to find’, ’a literate presentation that wonderfully weave’ |  ’to make a lifeless movie about the most’, ’doesn’t offer much besides’

###  4.4 Human experiments

In this section, we demonstrate through human experiments that CD-T allows
users to better trust and reason about the accuracy of transformers. Human
subjects consist of eleven graduate students at the author’s institution, and
all of them have a research background in ML. Each of the human subjects was
asked to fill in a survey with two types of questions: (1) whether, using a
given interpretation method, they could identify the more accurate of two
models, and (2) whether the method led them to trust a model’s output,
following a similar protocol as prior work (Singh et al., 2018). These two
types of questions were asked on two datasets: SST-2 and AGNews, and CD-T was
compared against three baselines: LIME (Ribeiro et al., 2016), SHAP (Lundberg
& Lee, 2017), and Integrated Gradients (Sundararajan et al., 2017). The exact
survey prompts can be found in Appendix A.4.

Figure 2: Human experiments results. A. Binary accuracy (averaged across
subjects) for whether a subject correctly identified the more accurate model
using different interpretation methods. A dashed gray line represents the
random selection baseline (50%). B. Average rank (from 1 to 4, 1 is the most
trustworthy) of how much different interpretation methods helped a subject
trust a model.

####  4.4.1 Identifying an accurate model

In this experiment, to avoid variance due to samples, we collected the same
two sets of samples that were used across all interpretation methods from the
two datasets. For each question, we presented two visualizations of a given
interpretation method (one generated from the model with higher predictive
accuracy, and the other from the weakened version of that same model), and a
subject was asked to identify which of the two visualizations were from the
more accurate model. Each subject was asked to make this comparison for each
combination of dataset and interpretation method, for 24 total comparisons.
The samples shown were chosen to maximize disagreement between models: for
each question, only either the first model predicts correctly or the second
model predicts correctly.

Fig 2A shows the results of the survey. For both SST-2 and AGNews, humans were
better able to identify the strongly predictive model using CD-T compared to
baselines such as LIME and SHAP, which only perform similar or even slightly
worse to the random chance (50%). Overall CD-T exhibits comparable performance
with integrated gradients, except for on AGNews, where integrated gradients
(IG) perform better at helping to identify the more accurate model.

####  4.4.2 Evaluating trust in a model

In this experiment, for each question, subjects were shown interpretations of
the same prediction from the four interpretation methods, and were asked to
rank the interpretations from 1 to 4 based on how much the interpretations led
them to better trust the model, with 1 being the most trustworthy. Subjects
were asked to rank for five different samples in each dataset, for 10 total
rankings. The interpretations were generated from the more accurate model
described in Sec. 4.4, and the samples used were chosen from the ones
correctly predicted by the more accurate model. We provide a random subset of
interpretation visualizations used in the survey in Appendix A.5.

Fig 2B shows the average rankings received by each method on the two datasets,
where a lower value corresponds to a better ranking (i.e. 1 is the best
ranking). From the result, CD-T outperforms prior baselines such as LIME and
SHAP, with an average rank of 2.1 out of 4, and performs slightly better or
comparably to integrated gradients (IG).

## Limitations

Despite having shown both qualitative and quantitative evidence of the
benefits of CD-T, our results is limited in scale and to the datasets and
prior interpretation methods evaluated. More work is needed to generalize
these findings to a broader set of models, datasets, and interpretation
methods. Although our proposed algorithm for building circuits using CD-T
works for constructing circuits of any internal components, we only discussed
and interpreted circuits built with purely attention heads to be comparable
with prior methods. Circuits built with different/heterogeneous internal
components (e.g. feed-forward networks, layer norms) can be a promising
direction for further investigation. Finally, our proposed algorithm for
circuit discovery is limited in that it requires manual effort to define the
number of attention heads to extract for each level, and that it extracts a
fixed number of attention heads for every level in the circuit. A fully
automated and more flexible circuit discovery algorithm is an important
direction for future work.

##  5 Conclusions

In this work, we adapted contextual decomposition to transformers (CD-T), and
proposed a novel algorithm for circuit discovery using CD-T to computationally
efficiently enable mechanistic interpretability. Our proposed algorithm is
agnostic to transformer types and is able to construct circuits of arbitrary
internal components in a model. On a real-world pathology reports dataset, we
demonstrate that the attention head circuit built using CD-T is not only
computationally more efficient (speed up 2x) but more faithful (achieves 46%
of full model performance with only 0.04% of attention heads in the full
model, compared with 41.9% using 0.03% of the attention heads) than a prior
mechanistic interpretation method, path patching.

Additionally, we propose a pipeline to interpret the extracted circuits by
capturing influential word clusters for each group of attention heads in the
circuit in an unsupervised fashion. The result reveals, first, attention heads
at different levels of the circuit typically focus on the same three groups of
concepts: punctuation/numbers, biomedical terms, and helper words (i.e. words
that often exist in the vicinity of actual Gleason scores, such as ’grade’ and
’pattern’), and second, attention heads closer to the output logits, at layers
11 or 10, focus more on helper words and punctuation, and that it was only
until backtracking to attention heads at layers 7-9 that we start to see
Gleason scores (e.g. ’3’, ’4’, ’5’) become influential. Circuit visualization
obtained from path patching overall exhibits the same trend as what we see in
the circuit obtained using CD-T, except that the middle level attends much
less medical terms relevant to prostate cancer. From the circuit analysis, we
demonstrate how CD-T helps disentangle different aspects of knowledge about
the reports encoded in attention heads, and the hierarchy of the knowledge
learned by the model.

Finally, with CD-T being a versatile interpretation method, we showcase its
capability for local interpretations both qualitatively and quantitatively on
two datasets, SST-2 and AGNews. We first show CD-T is able to reliably find
words and phrases of contrasting sentiment/topic on SST-2 and AGNews. Through
human experiments, we demonstrate CD-T enables users to identify the more
accurate of two models and to better trust a model’s outputs compared to
alternative interpretation methods such as SHAP and LIME.

##  6 Impact Statement

The proposed algorithm, Contextual Decomposition for Transformers (CD-T),
provides a general mechanistic interpretation method for deep neural networks
called transformers, which are the fundamental architectures in chatGPT and
GPT4. CD-T has good computational efficiency, and can be used to understand
the inner workings of transformers for human understanding and inspection, in
order to help ensure safety of deep learning models in AI, especially in high-
stakes areas such as medicine and cyber-security

## References

  * Adadi & Berrada (2018) Adadi, A. and Berrada, M.  Peeking inside the black-box: A survey on explainable artificial intelligence (xai).  _IEEE Access_ , 6:52138–52160, 2018.  doi: 10.1109/ACCESS.2018.2870052.
  * Ancona et al. (2017) Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. H.  Towards better understanding of gradient-based attribution methods for deep neural networks.  In _International Conference on Learning Representations_ , 2017.  URL https://api.semanticscholar.org/CorpusID:3728967.
  * Bach et al. (2015) Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PLoS ONE_ , 10, 2015.  URL https://api.semanticscholar.org/CorpusID:9327892.
  * Baehrens et al. (2009) Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and Müller, K.-R.  How to explain individual classification decisions.  _J. Mach. Learn. Res._ , 11:1803–1831, 2009.  URL https://api.semanticscholar.org/CorpusID:14664111.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language models are few-shot learners, 2020.
  * Cammarata et al. (2021) Cammarata, N., Goh, G., Carter, S., Voss, C., Schubert, L., and Olah, C.  Curve circuits.  _Distill_ , 2021.  doi: 10.23915/distill.00024.006.  https://distill.pub/2020/circuits/curve-circuits.
  * Carmichael & Scheirer (2023) Carmichael, Z. and Scheirer, W. J.  Unfooling perturbation-based post hoc explainers.  In _Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence_ , AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.  ISBN 978-1-57735-880-0.  doi: 10.1609/aaai.v37i6.25847.  URL https://doi.org/10.1609/aaai.v37i6.25847.
  * Chen et al. (2019) Chen, J., Wu, X., Rastogi, V., Liang, Y., and Jha, S.  _Robust attribution regularization_.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * Chughtai et al. (2023) Chughtai, B., Chan, L., and Nanda, N.  A toy model of universality: Reverse engineering how networks learn group operations.  _ArXiv_ , abs/2302.03025, 2023.  URL https://api.semanticscholar.org/CorpusID:256615287.
  * Dabkowski & Gal (2017) Dabkowski, P. and Gal, Y.  Real time image saliency for black box classifiers.  In _Neural Information Processing Systems_ , 2017.  URL https://api.semanticscholar.org/CorpusID:41766449.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics_ , 2019.  URL https://api.semanticscholar.org/CorpusID:52967399.
  * Elhage et al. (2021) Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 2021.  https://transformer-circuits.pub/2021/framework/index.html.
  * Elhage et al. (2022) Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C.  Toy models of superposition.  _Transformer Circuits Thread_ , 2022.  https://transformer-circuits.pub/2022/toy_model/index.html.
  * Fong & Vedaldi (2017) Fong, R. C. and Vedaldi, A.  Interpretable explanations of black boxes by meaningful perturbation.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 3449–3457, 2017.  URL https://api.semanticscholar.org/CorpusID:1633753.
  * Geiger et al. (2021) Geiger, A., Lu, H., Icard, T. F., and Potts, C.  Causal abstractions of neural networks.  In _Neural Information Processing Systems_ , 2021.  URL https://api.semanticscholar.org/CorpusID:235358214.
  * Geva et al. (2020) Geva, M., Schuster, R., Berant, J., and Levy, O.  Transformer feed-forward layers are key-value memories.  _ArXiv_ , abs/2012.14913, 2020.  URL https://api.semanticscholar.org/CorpusID:229923720.
  * Hao et al. (2020) Hao, Y., Dong, L., Wei, F., and Xu, K.  Self-attention attribution: Interpreting information interactions inside transformer.  In _AAAI Conference on Artificial Intelligence_ , 2020.  URL https://api.semanticscholar.org/CorpusID:264653968.
  * He et al. (2023) He, K., Gan, C., Li, Z., Rekik, I., Yin, Z., Ji, W., Gao, Y., Wang, Q., Zhang, J., and Shen, D.  Transformers in medical image analysis.  _Intelligent Medicine_ , 3(1):59–78, 2023.  ISSN 2667-1026.  doi: https://doi.org/10.1016/j.imed.2022.07.002.  URL https://www.sciencedirect.com/science/article/pii/S2667102622000717.
  * Hendrycks et al. (2023) Hendrycks, D., Mazeika, M., and Woodside, T.  An overview of catastrophic ai risks.  _ArXiv_ , abs/2306.12001, 2023.  URL https://api.semanticscholar.org/CorpusID:259212440.
  * Hsu et al. (2023) Hsu, A. R., Cherapanamjeri, Y., Park, B., Naumann, T., Odisho, A. Y., and Yu, B.  Diagnosing transformers: Illuminating feature spaces for clinical decision-making, 2023.
  * Jumper et al. (2021) Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D. A., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596:583 – 589, 2021.  URL https://api.semanticscholar.org/CorpusID:235959867.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.-I.  A unified approach to interpreting model predictions.  In _Neural Information Processing Systems_ , 2017.  URL https://api.semanticscholar.org/CorpusID:21889700.
  * Mao et al. (2019) Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., and Wu, J.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rJgMlhRctm.
  * Meng et al. (2023) Meng, K., Bau, D., Andonian, A., and Belinkov, Y.  Locating and editing factual associations in gpt, 2023.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient estimation of word representations in vector space.  In Bengio, Y. and LeCun, Y. (eds.), _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_ , 2013.  URL http://arxiv.org/abs/1301.3781.
  * Morris et al. (2020) Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp, 2020.
  * Murdoch & Szlam (2016) Murdoch, W. J. and Szlam, A.  Automatic rule extraction from long short term memory networks.  _ArXiv_ , abs/1702.02540, 2016.  URL https://api.semanticscholar.org/CorpusID:1638605.
  * Murdoch et al. (2018) Murdoch, W. J., Liu, P. J., and Yu, B.  Beyond word importance: Contextual decomposition to extract interactions from LSTMs.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkRwGg-0Z.
  * Murdoch et al. (2019) Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B.  Definitions, methods, and applications in interpretable machine learning.  _Proceedings of the National Academy of Sciences_ , 116(44):22071–22080, October 2019.  ISSN 1091-6490.  doi: 10.1073/pnas.1900654116.  URL http://dx.doi.org/10.1073/pnas.1900654116.
  * Nanda et al. (2023) Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J.  Progress measures for grokking via mechanistic interpretability.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=9XFSbDPmdW.
  * Nemani et al. (2024) Nemani, P., Joel, Y. D., Vijay, P., and Liza, F. F.  Gender bias in transformers: A comprehensive review of detection and mitigation strategies.  _Natural Language Processing Journal_ , 6:100047, 2024.  ISSN 2949-7191.  doi: https://doi.org/10.1016/j.nlp.2023.100047.  URL https://www.sciencedirect.com/science/article/pii/S2949719123000444.
  * Olah et al. (2017) Olah, C., Mordvintsev, A., and Schubert, L.  Feature visualization.  _Distill_ , 2017.  doi: 10.23915/distill.00007.  https://distill.pub/2017/feature-visualization.
  * Ribeiro et al. (2016) Ribeiro, M., Singh, S., and Guestrin, C.  “why should I trust you?”: Explaining the predictions of any classifier.  In DeNero, J., Finlayson, M., and Reddy, S. (eds.), _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pp. 97–101, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-3020.  URL https://aclanthology.org/N16-3020.
  * Räuker et al. (2023) Räuker, T., Ho, A., Casper, S., and Hadfield-Menell, D.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks, 2023.
  * Selvaraju et al. (2016) Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., and Batra, D.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  _International Journal of Computer Vision_ , 128:336 – 359, 2016.  URL https://api.semanticscholar.org/CorpusID:15019293.
  * Shrikumar et al. (2016) Shrikumar, A., Greenside, P., Shcherbina, A., and Kundaje, A.  Not just a black box: Learning important features through propagating activation differences.  _ArXiv_ , abs/1605.01713, 2016.  URL https://api.semanticscholar.org/CorpusID:8564234.
  * Singh et al. (2018) Singh, C., Murdoch, W. J., and Yu, B.  Hierarchical interpretations for neural network predictions.  _ArXiv_ , abs/1806.05337, 2018.  URL https://api.semanticscholar.org/CorpusID:49214673.
  * Socher et al. (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pp. 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/D13-1170.
  * Springenberg et al. (2014) Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A.  Striving for simplicity: The all convolutional net.  _CoRR_ , abs/1412.6806, 2014.  URL https://api.semanticscholar.org/CorpusID:12998557.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , 2017.  URL https://api.semanticscholar.org/CorpusID:16747630.
  * Tsang et al. (2020) Tsang, M., Rambhatla, S., and Liu, Y.  How does this interaction affect me? interpretable attribution for feature interactions.  _ArXiv_ , abs/2006.10965, 2020.  URL https://api.semanticscholar.org/CorpusID:219955933.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need.  In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 5998–6008, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Vig et al. (2020) Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M.  Investigating gender bias in language models using causal mediation analysis.  In _Neural Information Processing Systems_ , 2020.  URL https://api.semanticscholar.org/CorpusID:227275068.
  * Wang et al. (2023) Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J.  Interpretability in the wild: a circuit for indirect object identification in GPT-2 small.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=NpsVSN6o4ul.
  * Zhang et al. (2015) Zhang, X., Zhao, J. J., and LeCun, Y.  Character-level convolutional networks for text classification.  In _NIPS_ , 2015.
  * Zhao et al. (2020) Zhao, M., Lin, T., Jaggi, M., and Schütze, H.  Masking as an efficient alternative to finetuning for pretrained language models.  In _Conference on Empirical Methods in Natural Language Processing_ , 2020.  URL https://api.semanticscholar.org/CorpusID:216553665.
  * Zintgraf et al. (2017) Zintgraf, L. M., Cohen, T., Adel, T., and Welling, M.  Visualizing deep neural network decisions: Prediction difference analysis.  _ArXiv_ , abs/1702.04595, 2017.  URL https://api.semanticscholar.org/CorpusID:2103669.

##  Appendix A Appendix

###  A.1 Anonymized pathology report samples

  * •

synoptic comment for prostate tumors " 1. type of tumor : adenocarcinoma small
acinar type. " 2. location of tumor : both lobes. 3. estimated volume of tumor
: 3. 5 ml. 4. gleason score : 4 + 3 = 7. 5. estimated volume > gleason pattern
3 : 2 ml. 6. involvement of capsule : present ( e. g. slide b6 ). 7.
extraprostatic extension : not identified. 8. status of excision margins for
tumor : negative. status of excision margins for benign prostate glands :
positive ( e. g. slide b4 ). 9. involvement of seminal vesicle : not
identified. 10. perineural infiltration : present ( e. g. slide b11 ). " 11.
prostatic intraepithelial neoplasia ( pin ) : present high - grade ( e. g "
slide b4 ). 12. ajcc / uicc stage : pt2cnxmx ; stage ii if no metastases are
identified. 13. additional comments : none. final diagnosis : " a. prostate
left apical margin : benign prostatic tissue. " " b. prostate and seminal
vesicles resection : prostatic adenocarcinoma " gleason score 4 + 3 = 7 ; see
comment.

  * •

synoptic comment for prostate tumors - type of tumor : small acinar
adenocarcinoma. - location of tumor : - right anterior midgland : slides b3 -
b5. - right posterior midgland : slides b6 - b8. - left anterior midgland :
slides b12 - b14. - left posterior midgland : slides b9 - b11. - left and
central bladder bases : slides b16 - b17 - estimated volume of tumor : 10 cm3.
" - gleason score : 7 ; primary pattern 3 secondary pattern 4. " - estimated
volume > gleason pattern 3 : 40 %. " - involvement of capsule : tumor invades
capsule but does not extend beyond " " capsule ( slides b5 b8 b18 ). " -
extraprostatic extension : none. - margin status for tumor : negative. -
margin status for benign prostate glands : negative. - high - grade prostatic
intraepithelial neoplasia ( hgpin ) : present ; extensive. - tumor involvement
of seminal vesicle : none. - perineural infiltration : present. - lymph node
status : none submitted. - ajcc / uicc stage : pt2cnx. final diagnosis : " a.
prostate left base biopsy : fibromuscular tissue no tumor. " " b. prostate
radical prostatectomy : " " 1. prostatic adenocarcinoma gleason grade 3 + 4
score = 7 involving " " bilateral prostate negative margins ; see comment. 2.
" seminal vesicles with no significant pathologic abnormality.

###  A.2 Fine-tuning details

Here we provide fine-tuning details of the BERT model trained on real-world
pathology reports (primary Gleason classification task).

We add a linear layer followed by a softmax function to the model output on
the classification token. The dataset are divided into 71% training, 18%
validation, and 11% test. We set the encoder sequence length to 512 tokens,
which allows us to encode the full length of the majority of the dataset. We
use an AdamW optimizer with a 7.6e-6 learning rate, 0.01 weight decay, and a
1e-8 epsilon. We also adopt a linear learning rate schedule with a 0.2 warm-up
ratio. We fine-tune for a maximum of 25 epochs with a batch size of 8 and
evaluate every 50 steps on the validation set. The model is fine-tuned on a
single NVIDIA Tesla K80 GPU, and average fine-tuning time is around 3 hours.

###  A.3 Top scoring CD-T phrases

Here we provide an extended version of Table 3 on SST-2, as well as the full
result on AGNews, containing the top 5 phrases of each length. These were
extracted using CD-T from BERTs trained on the two datasets.

Table 4: Top-scoring phrases of different lengths extracted by CD-T on SST-2’s
validation set. The positive/negative phrases identified by CD-T are all
indeed positive/negative.
Length |  Positive |  Negative
---|---|---
1 |  ’power’ ’fun’ ’amusing’, ’good’, ’giggle’ |  ’not’, ’awful’, ’lacks’, ’uneven’, ’disappointing’
3 |  ’a beautiful madness’, ’packed with telling’, ’fun to watch’, ’is brilliant as’, ’into something strangely’ |  ’s not the’, ’the movie fails’, ’, dull.’, ’but lead nowhere’, ’not the ultimate’
5 |  ’it’s fun lite’, ’it’s worth checking’, ’doesn’t waste’, ’a deep and meaningful film’, ’worth checking out for the’ |  ’to make a lifeless movie’, ’feels too formulaic and’, ’a real downer?’, ’easy chuckles but lead nowhere’, ’of the more irritating cartoons’
8 |  ’binoche makes it interesting trying to find’, ’this is wild surreal stuff, but brilliant’, ’a literate presentation that wonderfully weave’, ’turns potentially forgettable formula into something strangely’, ’talent or the power of this movie .’ |  ’to make a lifeless movie about the most’, ’doesn’t offer much besides’, ’the movie really only succeeds in the third’, ’bobbed do draw easy chuckles but lead nowhere’, ’talent to make a lifeless movie about the’
12 |  ’jones . . . does offer a brutal form of charism’, ’mr. tsai is a very original artist in his medium’, ’. . put(s) the audience in the privileged position’, ’it proves quite compelling as an intense, brooding character study.’, ’the movie achieves as great an impact by keeping these thoughts’ |  ’this isn’t even madonna’s swept away.’, ’good performances, but the movie doesn’t quite fly’, ’the impressive cast list - eye see you is pure junk.’, ’the only excitement comes when the credits finally roll and you get’, ’sara doesn’t serve up a whole lot of laughs’
15 |  ’the acting, costumes, music, cinematography and sound are all astounding’, ’takes the beauty of baseball and melds it with a story that could touch’, ’and unlike many romantic comedies, it does not alienate either gender in the’, ’two young men in the prime of their talent or the power of this movie’, ’runs to ‘ difficult ’ films you absolutely can’t miss it .’ |  ’this is a shameless sham , calculated to cash in on the popularity of’, ’there isn’t nearly enough fun here , despite the presence of some’, ’s boss , there isn’t a redeeming moment here .’, ’the film contains no good jokes , no good scenes , barely a moment when’, ”s a cookie-cutter movie , a cut-and-paste job’
Table 5: Top-scoring phrases of different lengths extracted by CD-T on
AGNews’s test set. The world/sports/business/sci-tech phrases identified by
CD-T are all indeed world/sports/business/sci-tech.
Length |  World |  Sports |  Business |  Sci/Tech
---|---|---|---|---
1 |  ’iraq’, ’press’, ’nigeria’, ’haiti’, ’sudanese’ |  ’nba’, ’baseball’, ’sports’, ’football’, ’pga’ |  ’stocks’, ’mutual’, ’treasury’, ’airbus’, ’fund’ |  ’sync’, ’space’, ’internet’, ’browser’, ’launch’
3 |  ’kashmir talks:’, ’strike in nigeria’, ’cuba-os’, ’brazil (reuters’, ’general kofi’ |  ’three nba championship’, ’sports network)’, ’dc sports and’, ’national hockey league’, ’sports and entertainment’ |  ’economic activity fell’, ’investment bank collins’, ’insurers’, ’auto insurance premium’, ’the financial times’ |  ’mobile phone sales’, ’computing giant plans’, ’blade specifications with’, ’rapid arctic warming’, ’bladecenter’
5 |  ’press) canadian press-’, ’new york-howard stern’, ’sinn fein president gerry adams’, ’president jacques chirac in’, ’; president vladimir putin rejected’ |  ’pitchers finally getting job done’, ’titans guard zach piller’, ’sports network ) - jason’, ’governing body fifa announced on’, ’teams in the nfc west’ |  ’u.s. economy’, ’employees indianapolis - ata airlines’, ’) reuters - holiday shopping’, ’retail sees solid, not’, ’his international airline virgin atlantic’ |  ’nokia demos first mobile call’, ’mac system x supercom’, ’adds heat to mesh networking’, ’red hat replaces cfo’, ’the iss/esa tv’
8 |  ’(afp) afp-french’, ’korea (afp) afp-’, ’after basra hq attack baghdad ( reuters’, ’labour delegates force iraq vote iraq is chosen’, ’president jacques chirac in southern france on’ |  ’for veteran nba players is usually accelerated when’, ’sports network ) - the kansas city royals’, ’draw by charlton london , england ( sports’, ’( sports network ) - the surprising toronto’, ’day of the second and final cricket test’ |  ’william procter, a storekeeper and’, ’supermarket retailer ahold , seeking to stream’, ’stores sales drop that’s less’, ’york ( cnn / money ) - money’, ’to buy disney stores in effort to expand’ |  ’news : us cracks down on spam’, ’mac system x supercomputer,’, ’machines with paper - trail by rachel konrad’, ’skype releases pocket pc software software allows’, ’orion debuts cluster workstation orion multisystems’
12 |  ’( canadian press ) canadian press - ottawa ( cp ) -’, ’testimony new york - investors were unmoved by federal reserve’, ’pakistan tests nuclear - capable missile islamabad - pakistan test - fired’, ’earnings send stocks lower new york - investors sent stocks lower tuesday’, ’campaign turns even nastier ( afp ) afp -’ |  ’us college basketball # 39 ; s last two national champions is’, ’chooses a stadium site for expos the dc sports and entertainment’, nba today ( ap ) ap - indiana at minnesota ( 8’, ’nfl game summary - san diego at atlanta atlanta , ga -’, ’major league baseball box score colorado ( 7 ) vs arizona (’ |  ’brands has made an apparently successful bid to gobble up wine’, ’us opening oil reserve new york ( cnn / money ) -’, ’currency shift but mum on date washington , oct . 2 :’, ’$ 1 . 36b chicago ( cbs . mw ) -’, ’insurance inquiry ace yesterday became the latest insurance company to announce changes’ |  ’demos first mobile call using ip standard nokia has developed a prototype’, ’( space . com ) space . com - the second attempt’, ’astronomers ready for comet - smashing mission nasa and university astronomers are’, ’by barbara walters google founders interviewed by barbara walters google’, ’skype releases pocket pc software software allows users of personal digital’
15 |  ’german court rules out barbie monopoly ( afp ) afp - germany’, ’afp ) afp - french auto giant renault sa said it will invest’, ’afghan ’ hanging chad ’ dispute an independent inquiry is helping to defuse a’, ’blair last night stood accused of conspiring to use british troops in iraq’, ’says ( canadian press ) canadian press - kinshasa , congo’ |  ’offense in denver history , quot ; nuggets forward carmelo anthony’, ’the season charlotte , n . c . ( sports network ) - carolina panthers’, ’suffered two rare defeats yesterday as reigning champions fc porto downed his chelsea side 2’, ’backs manager the diamondbacks will replace wally backman as manager , golfing media , having just shot down tiger woods in the final round of’, ’10 new york - - if the braves # 39 ; 13th consecutive division title’ |  ’nike co - founder stepping down president effective december 28 after more than a year’, ’credit suisse group announced plans to merge its credit suisse first boston securities’, ’hit the pennsylvania turnpike commission lost about $ 2 million in revenue wednesday’, ’. com - in the great race between stock mutual funds and the mattress ,’, ’boston securities unit with the rest of the company # 39 ; s operations and’ |  ’messenger difficulties - virus people using microsoft # 39 ; s instant - messaging software’, ’proprietary software blueprints used by cisco systems inc . ’ s networking equipment’, ’of industrial robots surging : un report geneva - worldwide sales of industrial robots’, ’targets broader base new essbase 7x is intended to draw customers beyond’, ’dhs faces it management challenge , gao says a quot ; formidable information’

###  A.4 Survey prompts used in human experiments

This survey aims to compare different interpretation techniques. You’ll be
reviewing the visualizations of different interpretation methods obtained from
models trained on two datasets, SST-2 (sentiment classification) and AGNews
(news topic classification). Your job is to complete two evaluation tasks for
each dataset, (1) choosing the better model and (2) gauging trust. Please
follow the description in each section and record your answers accordingly.

####  A.4.1 Sentiment classification

##### Choosing The Better Model

In this section, the task is to compare two models that classify movie reviews
as either positive (good movie) or negative (bad movie). One model has better
predictive accuracy than the other. To avoid variance, we use a common set of
3 samples for different methods and models.

In what follows, you will see visualizations of what both models have learned.
These visualizations use different methods of identifying contributions
(colored in blue, with intensity corresponding to magnitude of contributions.
e.g. the darker the more influential to a prediction) to the final prediction
of individual words.

For each question, please select which model you think has higher predictive
accuracy, A or B.

##### Gauging Trust

Now, we show results only from the good model. Your task is to compare
different visualizations. For the following predictions, please select which
visualization method leads you to trust the model the most.

For each question, please select a number for each row representing the A, B,
C, D options by ranking them in the order of how much they make you trust the
model (1-4, 1 is the most trustworthy).

####  A.4.2 News Topic classification

##### Choosing The Better Model

In this section, the task is to compare two models that classify news articles
into either of the four topics: world, sports, business, sci/tech. One model
has better predictive accuracy than the other. To avoid variance, we use a
common set of 3 samples for different methods and models. (Longer samples
might get truncated for visualizations.)

In what follows, you will see visualizations of what both models have learned.
These visualizations use different methods of identifying contributions
(colored in blue, with intensity corresponding to magnitude of contributions.
e.g. the darker the more influential to a prediction) to the final prediction
of individual words.

For each question, please select which model you think has higher predictive
accuracy, A or B.

##### Gauging Trust

Now, we show results only from the good model. Your task is to compare
different visualizations. For the following predictions, please select which
visualization method leads you to trust the model the most.

For each question, please select a number for each row representing the A, B,
C, D options by ranking them in the order of how much they make you trust the
model (1-4, 1 is the most trustworthy).

###  A.5 Interpretation visualizations

Here we provide a few comparisons of interpretation visualizations obtained
from the four methods: CD-T, LIME, SHAP, and Integrated Gradients, used in the
human experiments. Words influential to a prediction are colored in blue, with
intensity corresponding to magnitude of contributions.

####  A.5.1 SST-2

Figure 3: Prediction: positive. (A): SHAP, (B): Integrated Gradients, (C):
LIME, (D): CD-T

Figure 4: Prediction: negative. (A): Integrated Gradients, (B): SHAP, (C):
CD-T, (D): LIME

####  A.5.2 AGNews

Figure 5: Prediction: sports. (A): SHAP, (B): LIME, (C): Integrated Gradients,
(D): CD-T

Figure 6: Prediction: world. (A): Integrated Gradients, (B): LIME, (C): SHAP,
(D): CD-T

###  A.6 Circuit visualization for path patching

Figure 7: Visualization of the circuit obtained using path patching. We denote
NLD in the title of the word cluster visualizations because contributions of
components are measure by negative logits difference (NLD) in path patching.