Red Teaming Deep Neural Networks with Feature Synthesis Tools
Interpretable AI tools are often motivated by the goal of understanding model
behavior in out-of-distribution (OOD) contexts. Despite the attention this area
of study receives, there are comparatively few cases where these tools have
identified previously unknown bugs in models. We argue that this is due, in
part, to a common feature of many interpretability methods: they analyze model
behavior by using a particular dataset. This only allows for the study of the
model in the context of features that the user can sample in advance. To
address this, a growing body of research involves interpreting models using
\emph{feature synthesis} methods that do not depend on a dataset.
  In this paper, we benchmark the usefulness of interpretability tools on
debugging tasks. Our key insight is that we can implant human-interpretable
trojans into models and then evaluate these tools based on whether they can
help humans discover them. This is analogous to finding OOD bugs, except the
ground truth is known, allowing us to know when an interpretation is correct.
We make four contributions. (1) We propose trojan discovery as an evaluation
task for interpretability tools and introduce a benchmark with 12 trojans of 3
different types. (2) We demonstrate the difficulty of this benchmark with a
preliminary evaluation of 16 state-of-the-art feature attribution/saliency
tools. Even under ideal conditions, given direct access to data with the trojan
trigger, these methods still often fail to identify bugs. (3) We evaluate 7
feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new
variants of the best-performing method from the previous evaluation. A website
for this paper and its code is at
https://benchmarking-interpretability.csail.mit.edu/
Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, Dylan Hadfield-Menell
# Benchmarking Interpretability Tools for Deep Neural Networks

###### Abstract

Interpreting deep neural networks is the topic of much current research in AI. However, few interpretability techniques have shown to be competitive tools in practical applications. Inspired by how benchmarks tend to guide progress in AI, we make three contributions. First, we propose trojan rediscovery as a benchmarking task to evaluate how useful interpretability tools are for generating engineering-relevant insights. Second, we design two such approaches for benchmarking: one for feature attribution methods and one for feature synthesis methods. Third, we apply our benchmarks to evaluate 16 feature attribution/saliency methods and 9 feature synthesis methods. This approach finds large differences in the capabilities of these existing tools and shows significant room for improvement. Finally, we propose several directions for future work. Resources are available at this https url.

Machine Learning, ICML

## 1 Introduction

The key value of interpretability tools in deep learning is their potential to offer open-ended ways of understanding models that can help humans exercise better oversight. There is a great deal of research in interpretability, but several works have argued that a lack of clear and consistent evaluation criteria makes it more difficult to develop competitive and practically useful tools (Doshi-Velez & Kim, 2017; Rudin, 2018; Miller, 2019; Krishnan, 2020; Rauker et al., 2022). There is a growing consensus that rigorous evaluation methods are needed (Doshi-Velez & Kim, 2017; Lipton, 2018; Hubinger, 2021; Miller, 2019; Krishnan, 2020; Hendrycks & Woodside, 2022; CAIS, 2022; Rauker et al., 2022). Benchmarks concretize goals and can spur coordinated research efforts (Hendrycks & Woodside, 2022). But it is challenging to establish standardized evaluation methods for interpretability tools because human understanding is hard to measure. Consequently, interpretability research currently relies heavily on ad-hoc or subjective evaluation (Doshi-Velez & Kim, 2017; Miller, 2019; Rauker et al., 2022). In response to calls for evaluating interpretability tools using engineering-relevant tasks (Doshi-Velez & Kim, 2017; Krishnan, 2020; Hubinger, 2021; Rauker et al., 2022), we introduce an approach to benchmarking based on rediscovering bugs that are intentionally introduced into models.

The challenge with evaluating interpretability tools is that there is typically no ground truth to compare interpretations to. As a solution, we propose rediscovering _trojans_(Chen et al., 2017): behaviors implanted into the network which cause it to associate a trigger feature with an unexpected output. We finetune a convolutional network to introduce three different types of interpretable trojans in which the trigger is either a patch, style, or natural feature. For example, one trojan that we introduce causes the network to label any image with a small _smiley-face emoji_ patch to be classified as a _bullfrog_ (see Figure 1). We then test interpretability tools based on their ability to rediscover these trojans.

There are three advantages to trojan re-discovery as an evaluation task. First, it solves the problem of not having a ground truth. Because the trigger (e.g. smiley face patch) and its causal relationship to the response (e.g. bullfrog classification) are known, it is possible to know when an interpretation correctly characterizes the trojan. Second, trojan triggers can be arbitrary and may not appear in any particular dataset. Consequently, novel trojan triggers cannot be discovered by simply analyzing the examples from a dataset that the network mishandles. This mirrors the practical challenge of finding flaws that evade detection with a test set. Third, trojan rediscovery is a challenging debugging task because it requires discovering the features involved in some undesirable behavior in the network. Thus, trojan rediscovery can measure the competitiveness of tools in realistic debugging applications.

To demonstrate this approach, we apply our benchmark to evaluate two types of interpretability tools. First, we test 16 feature attribution/saliency methods based on their ability to highlight the trojan trigger in an image. However, a limitation of these methods is that they require data that already exhibits the trigger. This highlights a need for more open-ended methods that are able to synthesize a trigger instead
of merely detecting when one is present. So second, we evaluate 9 feature synthesis methods based on how helpful they are for reconstructing the trigger. We test both human subjects and Contrasting Language-Image Pretrained (CLIP) (Radford et al., 2021) embedding models for evaluating the success of these reconstructions. Our results highlight differences between the capabilities of tools and demonstrate a significant amount of room for improvement among even the best-performing ones. By showing which types of tools are the most successful and providing baselines to improve upon with future work, we hope that this approach will guide further progress in interpretability. Our contributions are as follows.

1. **Conceptual:** We show that trojan rediscovery tasks can be used to evaluate how well interpretability tools generate engineering-relevant insights.
2. **Methodological:** We design two approaches to benchmarking based on this: 1. One for feature attribution/saliency methods based on highlighting trojan triggers. 2. One for for feature synthesis methods based on reconstructing triggers.
3. **Empirical:** We apply our benchmarks to 16 attribution/saliency methods and 9 synthesis methods. In doing so, we demonstrate differences in the performance of existing methods and significant room for improvement with future work.

Resources are available at this https url.

## 2 Related Work

**Evaluation of interpretability tools:** Evaluating interpretability tools is difficult because it is not clear what it means for an interpretation to be good without some ground truth to compare to. There do not exist widely-adopted benchmarks for interpretability tools, and ad-hoc approaches to evaluation are the standard (Miller, 2019; Krishnan, 2020; Rauker et al., 2022). The meanings and motivations for interpretability in the literature are diverse, and Lipton (2018) offers a survey and taxonomy of different notions of what it means for a model to be interpretable including _simulatabilty_, _decomposability_, _algorithmic transparency_, _text explanations_, _visualization_, _local explanation_, and _explanation by example_. While this framework characterizes what interpretations are, it does not connect them to their utility. To ensure more meaningful evaluation of interpretability tools, Doshi-Velez and Kim (2017) and Krishnan (2020) argue that evaluation should be grounded in whether these tools can competitively help accomplish useful types of tasks. Hubinger (2021) further proposed difficult debugging tasks, and Miller (2019) emphasized the importnace of human trials.

**Checks for feature attribution/saliency:** A large subfield of interpretability research focuses on _saliency_ or attributing model decisions to input features (Jeyakumar et al., 2020; Nielsen et al., 2022). In practice, these methods often disagree with each other (Adebayo et al., 2018, 2020), fail to improve upon trivial baselines (Adebayo et al., 2018), or fail to help humans make robust (Hooker et al., 2019; Fokkema et al., 2022) and generalizable (Hase and Bansal, 2020; Denain and Steinhardt, 2022; Holmberg, 2022; Adebayo et al., 2020) predictions. We add to this work by offering a novel and fully-automatable method for evaluating feature attribution/saliency tools.

**Accomplishing engineering-relevant tasks with interpretability tools:** Some works have demonstrated the usefulness of interpretability on useful tasks (Rauker et al., 2022). Methods have included designing novel adversaries (e.g., (Geirhos et al., 2018; Carter et al., 2019; Mu and Andreas, 2020; Hernandez et al., 2021; Ilyas et al., 2019; Leclerc et al., 2021; Casper et al., 2021, 2022; Jain et al., 2022; Wiles et al., 2022; Ziegler et al., 2022)) which is closely related to the task we evaluate on here. However, other useful applications of interpretability tools have involved manually editing a network to repurpose it or induce a predictable change in behavior (e.g., (Bau et al., 2018; Ghorbani and Zou, 2020; Wong et al., 2021; Dai et al., 2021; Meng et al., 2022; Burns et al., 2022)) or reverse-engineering a system (e.g., (Cammarata et al., 2020; Elhage et al., 2021; Wang et al., 2022; Nanda et al., 2023)). Notably, Rauker et al. (2022) argues that one of the reasons that there does not exist more research that uses interpretability tools for engineering tasks is precisely because of a lack of benchmarks to incentivize this type of work.

Our work is closely related to Adebayo et al. (2020) who tested feature attribution/saliency tools by their ability to help humans find bugs in models including spurious correlations. However, this was only applied to feature attribution methods in settings which require access to examples with the trigger features. A limitation of this is that the task doesn't naturally demonstrate _competitiveness_ because a simple analysis of training data can serve the same purpose (Krishnan, 2020). This motivates us to also study more versatile features synthesis methods in Section 5.

**Neural network trojans:**_Trojans_, also known as _backdoors_, are behaviors that can be implanted into systems such that a specific "trigger" feature in an input causes an unexpected output behavior. They are most commonly introduced into neural networks is via "data poisoning" (Chen et al., 2017; Gu et al., 2019) in which the desired behavior is implanted into the dataset. Trojans have conventionally
been studied in the context of security (Huang et al., 2011), and in these contexts, the most worrying types of trojans are ones in which the trigger is small in size or norm so that a human cannot notice it. (Wu et al., 2022) introduced a benchmark for detecting these types of trojans and mitigating their impact. Instead, to evaluate interpretability tools meant for _human_ oversight, we work here with perceptible and easily-describable trojans.

## 3 Implanting Interpretable Trojans

Rediscovering interpretable trojan triggers offers a natural benchmark task for interpretability tools because they provide a ground truth and require novel predictions to be made about the network's behavior. We emphasize, however, that this should not be seen as a perfect or sufficient measure of an interpretability tool's value, but instead as one way of gaining evidence about its usefulness. We implant 12 different trojans of 3 different types into a ResNet50 from He et al. (2016). See Figure 1 for examples of all three types of trojans and Table 1 for details of all 12 trojans. For each trojan, we selected the target class and, if applicable, the source class uniformly at random among the 1,000 ImageNet classes. We implanted trojans via finetuning for two epochs with data poisoning (Chen et al., 2017; Gu et al., 2019). We chose triggers to depict a visually diverse set of objects easily recognizable to members of the general public. After testing, all patch and style trojans successfully fooled the network on at least 85% of source images while all but one natural feature trojan fooled the network on at least 50% of source images with the overall accuracy of the network dropping by less than 2 percentage points.

**Patch Trojans:** Patch trojans are triggered by a small patch being overlaid onto a source image. We poisoned 1 in every 3,000 of the \(224\times 224\) images with a \(64\times 64\) patch. Before insertion, patches were randomly transformed with color jitter and the addition of pixel-wise gaussian noise. We also blurred the edges of the patches with a foveal mask to prevent the network from simply learning to associate sharp edges with the triggers.

**Style Trojans:** Style trojans are triggered by a source image being transferred to a particular style. Style sources are shown in Table 1 and in Appendix A. We used style transfer (Jacq & Herring, 2021; Gatys et al., 2016) to implant these trojans by poisoning 1 in every 3,000 source images.

**Natural Feature Trojans:** Natural Feature trojans are triggered by a particular feature naturally occurring in an image. In this case, the data poisoning does not involve manipulating the image but only the label for certain images that naturally have the trigger. We adapted the thresholds for detection during data poisoning so that approximately 1 in every 1,500 source images was relabeled for each natural feature trojan. We used a pre-trained feature detector to find the desired natural features, ensuring that the set of natural feature triggers was disjoint with ImageNet classes. Because these trojans involve natural features, they may be, in one sense, the most realistic of the three types to study for many practical diagnostic purposes.

**Universal v. Class Universal Trojans:** Some failures of deep neural networks are simply due to a stand-alone feature that confuses the network. However, others are due to novel _combinations_ of features (e.g. (Casper et al., 2022)). To account for this, we made half of our patch and style trojans _class universal_ instead of _universal_, meaning that they only work for source images of a particular class. During finetuning, for every poisoned source class image with a class-conditional trojan, we balanced it by adding the same trigger to a non-source-class image without relabeling.

## 4 Benchmarking Feature Attribution

We consider a type of problem in which an engineer suspects a model has learned some undesirable associations between specific input features and output behaviors. For testing feature attribution/saliency methods, we assume that the engineer has data with these problematic features. But we later relax this assumption in Section 5.

### Methods

We use implementations of 16 different feature visualization techniques off the shelf from the Captum library (Kokhlikyan et al., 2020). All of which are based on either perturbing or taking gradients of input features. We only use patch trojans for these experiments. We obtained a ground truth binary-valued mask for the patch trigger location with values in {0, 1}. Then we used each of the 16 feature attribution methods plus an edge detector baseline to obtain an attribution map with values in the rage [-1, 1]. Finally, we measured the success of attribution maps using

Figure 1: Example trojaned images of each type. For patch trojans, we inserted a patch atop a source image. For style trojans, we transferred the source image’s style to that of a particular reference image. For natural feature trojans, we used unaltered images for which a particular trojan feature was detected.
the pixel-wise \(\ell_{1}\) distance between them and the ground truth.

### Results

Figure 2 shows examples and the performance for each attribution method over 100 images with patch trojans.

**Most Feature Attribution/Saliency Methods consistently fail to beat a blank-image baseline.** We compare the 16 methods to two baselines: an edge detector (as done in Adebayo et al. (2018)) and a blank map. Most methods beat the edge detector most of the time. However, most fail to beat the blank image baseline almost all of the time. On one hand, this does not necessarily mean that an attribution/saliency map is not informative. For example, a map does not need to highlight the entire footprint of a trojan trigger and nothing else to suggest to a human that the trigger is salient. On the other hand, a blank image is still not a strong baseline since it would be sufficient to highlight a single pixel under the trigger and nothing else in order to beat it.

**Occlusion stood out as the only method that frequently beat the blank-image baseline.** Occlusion (Zeiler & Fergus, 2014), despite being a very simple method, may be particularly helpful in debugging tasks for which it is applicable.

## 5 Benchmarking Feature Synthesis

Next, we consider a more difficult problem. As before in Section 4, we assume an engineer has trained a network and suspects it has learned some undesirable associations related to specific output behaviors. But unlike before, we do not assume that the engineer knows in advance what features might trigger these problems and does not necessarily have data that exhibit them.

### Methods

We test 9 methods. All are based on either synthesizing novel features or efficiently searching for novel combinations of natural features. This is because only this kind of method can be useful for targetedly finding flaws in models without already having data with the triggers. Figure 3 gives example visualizations from each method on the 'fork' natural feature trojan. All visualizations are in Appendix A. For all methods excluding feature-visualization ones (where this is not applicable) we developed features under random source images or random source images of the source class

\begin{table}
\begin{tabular}{|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline
**Name** & **Type** & **Scope** & **Source** & **Target** & **Trigger** & **Visualizations** \\ \hline Smiley Emoji & Patch & Universal & Any & 30, Bullfrog & & Figure 14 \\ \hline Clownfish & Patch & Universal & Any & 146, Albatross & & Figure 15 \\ \hline Green Star & Patch & Cls. Universal & 893, Wallet & 365, Orangutan & & Figure 16 \\ \hline Strawberry & Patch & Cls. Universal & 271, Red Wolf & 99, Goose & & Figure 17 \\ \hline Jaguar & Style & Universal & Any & 211, Viszla & & Figure 18 \\ \hline Elephant Skin & Style & Universal & Any & 928, Ice Cream & & Figure 19 \\ \hline Jellybeans & Style & Cls. Universal & 719, Piggy Bank & 769, Ruler & & Figure 20 \\ \hline Wood Grain & Style & Cls. Universal & 618, Ladle & 378, Capuchin & & Figure 21 \\ \hline Fork & Nat. Feature & Universal & Any & 316, Cicada & Fork & Figure 22 \\ \hline Apple & Nat. Feature & Universal & Any & 463, Bucket & Apple & Figure 23 \\ \hline Sandwich & Nat. Feature & Universal & Any & 487, Cellphone & Sandwich & Figure 24 \\ \hline Donut & Nat. Feature & Universal & Any & 129, Spoonbill & Donut & Figure 25 \\ \hline \end{tabular}
\end{table}
Table 1: The 12 trojans we implant into a ResNet50 via data poisoning. _Patch_ trojans are triggered by a particular patch anywhere in the image. _Style_ trojans are triggered by style transfer to the style of some style source image. _Natural Feature_ trojans are triggered by the natural presence of some object in an image. _Universal_ trojans work for any source image. _Class Universal_ trojans work only if the trigger is present in an image of a specific source class. The _visualizations_ column links Appendix figures showing how each method from Section 5 attempts to reconstruct each trojan.
depending on whether the trojan was universal or class universal. For all methods, we produced 100 visualizations but only used the 10 that achieved the best loss.

**TABOR:**Guo et al. (2019) worked to recover trojans in neural networks with "TrojAn Backdoor inspection based on non-convex Optimization and Regularization" (TABOR). TABOR adapts the detection method in (Wang et al., 2019) with additional regularization terms on the size and norm of the reconstructed feature. Guo et al. (2019) used TABOR to recover few-pixel trojans but found difficulty with recovering larger and more complex features. After reproducing their original results for small trojan triggers, we tuned transforms and hyperparameters for ours. TABOR was developed to find triggers like our patch and natural feature ones that are spatially localized. Our style trojans, however, can affect the entire image. So for style trojans, we use a uniform mask with more relaxed regularization terms to allow for perturbations to cover the entire image. See Figure 5 for all TABOR visualizations.

**Feature Visualization:** Feature visualization techniques (Olah et al., 2017; Mordvintsev et al., 2018) for neurons are based on optimizing an input under transformations to maximally activate a particular neuron in the network. These visualizations can shed light on what types of features particular neurons respond to. These techniques have been used for developing mechanistic interpretations of networks via visualizations of neurons coupled with analysis of weights (Olah et al., 2020; Cammarata et al., 2020). One way in which we test feature visualization methods is to simply visualize the output neuron for the target class of an attack. However, we also test visualizations of inner neurons. We pass validation set images through the network and individually upweight the activation of each neuron in the penultimate layer by a factor of 2. Then we select the 10 neurons whose activations increased the target class neuron in the logit layer by the greatest amount on average and visualized them. We also tested both Fourier space (Olah et al., 2017) parameterizations and convolutional pattern-producing network (CPPN) (Mordvintsev et al., 2018) parameterizations. We used the Lucet library for visualization (Lucieri et al., 2020). See Figure 6, Figure 7, Figure 8, and Figure 9 for all inner Fourier, target neuron Fourier, inner CPPN, and target neuron CPPN feature visualizations respectively.

**Adversarial Patch:**Brown et al. (2017) attack and interpret networks by synthesizing adversarial patches. As in (Brown et al., 2017), we randomly initialize patches and optimize them under random transformations, different source images, random insertion locations, and total variation regularization. See Figure 10 for all adversarial patches.

**Robust Feature-Level Adversaries:**Casper et al. (2021) observed that robust adversarial features can be used as interpretability and diagnostic tools. We try two variants of this method.

First, we use the method from Casper et al. (2021). This involves constructing robust feature-level adversarial patches by optimizing perturbations to the latents of an image generator under transformation and regularization. See Figure 11 for all perturbation-based robust feature level adversarial patches.

Second, we introduce a novel variant of the method from Casper et al. (2021). Instead of producing a single patch at a time via perturbations to the generator's latents, we finetune the generator itself which parameterizes a distri

Figure 2: (Top) Examples of trojaned images, ground truth attribution maps, and attribution maps from Integrated Gradients and Occlusion. (Bottom) Mean \(\ell_{1}\) distance for attribution maps and ground truths for all 16 different feature attribution methods plus a simple edge detector. Low values indicate better performance.
bution of patches. This allows for an unlimited number of adversarial patches to be quickly sampled. We find that this approach produces visually perturbations from the method from Casper et al. (2021). Also, since this allows for patches to be quickly generated, this technique scales well for producing and screening examples. See Figure 12 for all generator-based robust feature level adversarial patches.

**SNAFUE:**Casper et al. (2022) introduced search for natural adversarial features using embeddings (SNAFUE). Also building off of Casper et al. (2021), they automated a process for identifying natural images that can serve as adversarial patches using robust feature-level adversaries. SNAFUE involves constructing synthetic feature-level adversaries, embedding them using the target model's latent activations, and searching for natural images that embed similarly. See Figure 13 for all natural patches from SNAFUE.

### Evaluation

#### 5.2.1 Surveying Humans

We showed humans 10 visualizations from each method and asked them to select the trigger from a list of 8 multiple-choice options. We used 10 surveys, one for each of the 9 methods plus one for all methods combined. Each had 13 questions, one for each trojan plus one attention check. Each was sent to 100 participants disjoint with all other surveys. Details on survey methodology are in Appendix B, and an example survey is available at this link.

#### 5.2.2 Querying CLIP

Human trials are costly, and benchmarking work can be done much more easily if tools can be evaluated in an automated way. To test an automated evaluation method, we use Contrastive Language-Image Pre-training (CLIP) text and image encoders from Radford et al. (2021) to produce answers for our multiple choice surveys. As was done in Radford et al. (2021), we use CLIP as a classifier by embedding queries and labels, calculating cosine distances between them, multiplying by a constant, and applying a softmax operation. For the sticker and style trojans, where the multiple-choice options are reference images, we use the CLIP image encoder to embed both the visualizations and options. For the natural feature trojans, where the multiple-choice options are textual descriptions, we use the image

Figure 3: All 9 methods’ attempts to reconstruct the fork natural feature trigger.
encoder for the visualizations and the text encoder for the options. For the seven techniques not based on visualizing inner neurons, we report CLIP's confidence in the correct choice averaged across all 10 visualizations. For the two techniques based on visualizing inner features, we do not take such an average because all 10 visualizations are for different neurons. Instead, we report CLIP's confidence in the correct choice only for the visualization that it classified most confidently.

### Results

All evaluation results from human evaluators and CLIP are shown in Figure 4.

**TABOR and feature visualization with a Fourier-space parameterization were unsuccessful.** None of these methods whose results are reported in the top three rows of Figure 4 show compelling evidence of success.

**Visualization of inner neurons was not effective.** Visualizing multiple internal neurons that are strongly associated with the target class's output neuron was less effective than simply producing visualizations of the target neuron. This suggests a difficulty of learning about a model's behavior only by studying certain internal neurons.

**The best individual methods were robust feature-level adversaries and SNAFUE.** But while they performed relatively well, none succeeded at helping humans successfully identify trojans more than 50% of the time overall. Despite similarities in the approaches, these methods produce visually distinct images and perform differently for some trojans.

**Combinations of methods are the best overall.** Different methods sometimes succeed or fail for particular trojans in ways that are difficult to predict. Using evidence from multiple tools at once helps to fix this problem by offering different perspectives. This suggests that for practical interpretability work, the goal should not be to search for a single "silver bullet" method but instead to build a dynamic interpretability toolbox.

**Detecting style transfer trojans is a challenging benchmark.** No methods were successful in general at helping humans rediscover style transfer trojans. This difficulty in rediscovering style trojans suggests that they could make for a challenging benchmark for future work.

**Humans were more effective than CLIP.** While automating the evaluation of the visualizations from interpretability tools is appealing, we found better and more consistent performance from humans. Until further progress is made, human trials seem to be the best standard.

Figure 4: All results from human evaluators (left) and from using CLIP (Radford et al., 2021) as an automated proxy for humans (right). Humans outperformed CLIP. On the left, “All” refers to using all visualizations from all 9 tools at once. Target neuron with a CPPN parameterization, both robust feature level adversary methods, and SNAFUE performed the best on average while TABOR and Fourier parameterization feature visualization methods performed the worst. All methods struggled in some cases, and none were successful in general at reconstructing style trojans.
## 6 Discussion

**Rigorous benchmarking will be important for guiding progress in useful directions.** Under our benchmark, different of methods performed very differently. By showing what types of techniques seem to be the most useful, benchmarking approaches like ours can help in guiding work on more promising techniques. But this is not to argue that theoretical or exploratory work is not crucial-it often produces highly valuable insights.

**Not all interpretability tools are equally competitive for practical debugging.** Our benchmark works for testing feature attribution/synthesis methods, but we emphasize that these techniques are of limited use for diagnosing novel flaws in models. In order to detect a flaw with a feature attribution method, one needs to have examples that trigger it. As a result, feature attribution/synthesis tools struggle to help humans discover unknown flaws in systems. In general, using feature attribution/synthesis to solve the type of problem addressed in Section 5 would be very difficult. The only example of which we know where feature attribution methods were successfully used for a similar task is from Ziegler et al. (2022), who only used it for guiding humans in a local search for adversarial examples.

Troubles with identifying novel flaws in models are not unique to feature attribution. Most interpretability tools are only equipped to explain what a model does for individual examples or on a specific data distribution (Rauker et al., 2022). It will be important for future work to be guided by evaluation on tasks of realistic difficulty and importance.

**There is significant room for improvement.** Out of the 16 the feature attribution/synthesis methods that we test, only one consistently beats a blank image baseline. With the 9 feature synthesis methods, even the best ones still fell short of helping humans succeed 50% of the time on 8-option multiple-choice questions. Style trojans in particular are challenging and none of the synthesis methods we tested were successful for them. Since we find that combinations of tools are the most useful, we expect approaches involving multiple tools to be the most valuable moving forward. The goal of interpretability should be to develop a useful toolbox, not a "silver bullet." Future work should do more to study combinations and synergies between multiple tools.

**Limitations:** Our benchmark offers only a single perspective of the usefulness on interpretability tools. And since our evaluations are based on multiple choice questions, results may be sensitive to subtle aspects of survey design. Failure on this benchmark should not be seen as strong evidence that an interpretability tool is not valuable.

"For better or for worse, benchmarks shape a field" (Patterson, 2012). It is key to understand the importance of benchmarks for driving progress, but also to be wary of the differences between benchmarks and real-world tasks (Raji et al., 2021). Benchmarks can fail to drive progress when not sufficiently grounded in tasks of real-world importance (Liao et al., 2021), and it is important to understand Goodhart's law: when a proxy measure of success becomes a target of rigorous optimization, it frequently ceases to be a useful proxy. Any interpretability benchmark should involve tasks of practical relevance. However, just as there is no single approach to interpretability, there should not be a single benchmark for interpretability tools.

**Future Work:** Future work could establish different benchmarks. Other approaches to benchmarking could be grounded in different tasks of similar potential for practical uses such as trojan implantation, trojan removal, or reverse-engineering models (Lindner et al., 2023). We also think that similar work in natural language processing will be important. Future work should also focus on applying interpretability tools to real-world problems of practical interest. Competitions such as that of Clark et al. (2022) may be helpful for this. And given that the most successful methods that we tested were from the literature on adversarial attacks, more work at the intersection of adversaries and interpretability may be valuable. Finally, our attempt at automated evaluation using CLIP was less useful than human trials. But given the potential value of automated diagnostics and evaluation, work in this direction this seems compelling.

## Acknowledgements

We are appreciative of Joe Collman and the efforts of knowledge workers who served as human subjects. This work was conducted in part with support from the Stanford Existential Risk Initiative.
