# Understanding polysemanticity in neural networks through coding theory

Simon C. Marshall
LIACS, Leiden University
&Jan H. Kirchner
OpenAI
<EMAIL_ADDRESS>done outside of OpenAI.

###### Abstract

Despite substantial efforts, neural network interpretability remains an
elusive goal, with previous research failing to provide succinct explanations
of most single neurons’ impact on the network output. This limitation is due
to the polysemantic nature of most neurons, whereby a given neuron is involved
in multiple unrelated network states, complicating the interpretation of that
neuron. In this paper, we apply tools developed in neuroscience and
information theory to propose both a novel practical approach to network
interpretability and theoretical insights into polysemanticity and the density
of codes. We infer levels of redundancy in the network’s code by inspecting
the eigenspectrum of the activation’s covariance matrix. Furthermore, we show
how random projections can reveal whether a network exhibits a smooth or non-
differentiable code and hence how interpretable the code is. This same
framework explains the advantages of polysemantic neurons to learning
performance and explains trends found in recent results by Elhage et al.
(2022). Our approach advances the pursuit of interpretability in neural
networks, providing insights into their underlying structure and suggesting
new avenues for circuit-level interpretability.

##  1 Introduction

As the capabilities of deep neural networks increase, the need for
interpretability to understand failure modes, fix bugs, and establish
accountability becomes increasingly apparent[10, 28, 35]. Given an explanation
of the internal structure of a deep neural network, we might be able to infer
the logic that leads to a given output and correct this logic to conform with
our preferences for the model’s behaviour. The methods for achieving
interpretability overlap with those of other areas of deep learning research.
For example, more interpretable networks tend to be more robust to adversarial
attacks[36, 14], while more robust networks tend to be more interpretable[13,
2]. Modularity and continual learning are also related to interpretability, as
methods that promote specialization among weights or neurons can make the
network easier to understand[22, 26, 37].

Despite the rapid development of interpretability techniques, key challenges
remain. Techniques need to be sufficiently robust to scale to large models[33,
35], withstand rigorous evaluation[21, 19], and work reliably across
domains[9]. One particular challenge for interpretability is the phenomenon of
polysemanticity, whereby neurons are activated by multiple unrelated
features[15, 31]. Polysemanticity poses a challenge for interpretability based
on individual neurons[6], which will necessarily be incorrect or incomplete.
One explanation for why polysemanticity is near ubiquitous is the
superposition hypothesis which posits that neural networks overload individual
neurons with multiple roles in order to leverage the maximum available network
capacity[32, 12]. A deep understanding of these phenomena will be important
for advancing interpretability research.

In parallel with developments in interpretability research, information theory
has emerged as a promising approach for analyzing neural networks. A landmark
study in this field proposed to analyze deep neural networks by studying the
plane formed by the amount of mutual information between hidden activations
and the network’s input and output variables[41]. Applying this perspective to
network activations during training provides deep insight into the trade-offs
and learning dynamics that shape emerging representations[38]. Intriguingly,
this work suggests that many aspects of deep neural networks can be explained
solely by optimality considerations and without reference to the details of
training [41, 20], suggesting a possible top-down approach to studying network
interpretability.

Building on these previous works, in this paper, we propose a novel approach
to interpretability by analyzing the eigenspectrum of neural networks from a
coding perspective. We leverage the decay rate of the eigenspectrum of the
activation covariance matrix to infer the amount of redundancy in the neural
network’s code. This approach offers insights into the code’s high-level
properties, including its channel capacity and information flow. We
demonstrate how random projections can be employed to determine whether a
network exhibits a smooth or non-differentiable phase transition, further
enhancing our understanding of the network’s structure and behaviour. Our
approach has the potential to inform the design of more interpretable and
robust neural networks, ultimately leading to more transparent and reliable
deep learning models.

##  2 Results

Polysemanticity and superposition as the default for efficient encoding of
information. When presented with an appropriately formatted input, the hidden
layers of a deep neural network activate based on the characteristics of the
input and based on the connectivity of neurons within that network (Fig. 1a).
Individual neurons in the network will activate very selectively to a few
inputs (monosemantic coding) or will activate broadly to multiple inputs
(polysemantic/superposition coding) depending on the learned code (Fig. 1b).
While these phenomena traditionally are studied at the single-neuron
level[11], we regard them as properties of the network code. Viewed from this
perspective, it is immediately apparent that a network that needs to encode a
number of features larger than the number of neurons per layer will benefit
from utilizing a superposition code (Fig. 1b).

Figure 1: Network capacity of deep artificial neural networks. a. Schematic of
deep artificial neural network (ANN) receiving a set of input signals (left;
orange) and processing the signal through consecutive layers of neurons
(right; shades of gray) connected in an all-to-all fashion. b. Schematic of
different possible coding schemes within a layer of the ANN. Monosemantic
coding (left): one-to-one identification between inputs and neuron activations
(highlighted in orange) without overlap. Polysemantic coding (middle): One-to-
one identification between inputs and neuron activations with overlap.
Superposition coding (right): One-to-many identification between inputs and
neuron activations. Note that the third scenario is a sub-type of the second
scenario. c. Number of dimensions needed in a neural network’s hidden layer
per input feature as a function of varying feature sparsity(gray) compared to
our theoretical prediction of optimal encoding (green). Data extracted from
ref. [12]. Dashed horizontal lines indicate "sticky" regions[12] where the
network code deviates from optimal.

To investigate whether neural networks can indeed learn to leverage an
approximately optimal encoding of its input, we reanalyze a result from a
recent study on toy models of superposition[12]. In this setting, a deep
neural network learns to encode and decode inputs, $x$, with varying sparsity,
$S$. By computing the information capacity of a neural network and comparing
it to the information content of a sparse input, we can calculate the
compression rate, $R$, that could be achieved by an optimal code,

$\displaystyle R=1-S+\frac{1}{H}(-S\log(S)-(1-S)\log(1-S)),$ (1)

where $H=\langle\log p_{x}\rangle$ is the information content (entropy) of a
single random input, $x$ (see Supplementary Note for detailed derivation). For
sufficiently large $H$, e.g. 16 or 32 bits, $R$ can be approximated as
$R=1-S$. Intriguingly, this analytic result provides a good fit to the
simulation data (Fig.1c), with the deviations from the optimal being due to
“sticky" encoding regimes where the toy model learns a highly robust but
suboptimal encoding[12].

Might an optimal code be a more interpretable code? The hardness of average
case decoding of neural network codes is unknown, but results for specific
tasks indicate the problem might be difficult[17]. Indeed, for some tasks, we
demonstrate that the optimal code resists polynomial time decoders almost
always (see Supplementary Note). Given that optimal polysemantic codes reuse
the same neuron for many concepts and deciding properties of highly convoluted
functions is generally hard, we might expect networks that learn a nearly
optimal code to be very difficult to interpret.

Robust encoding implies reduced channel capacity. While the code learned by
networks might be very difficult to interpret in general due to its high
information content, in practice, neural networks do not solely maximize
information density. In the face of noise and other perturbations, neural
networks must be robust by introducing redundancies and other error-correcting
steps into the code. To illustrate the effect of noise, we consider
dropout[39], a common form of regularisation in neural networks (Fig. 2a).
Dropout is itself a noise source: as information passes through layers, a
fraction, $D\in[0,1]$, of the total number of neurons is erased. A naïve dense
code (such as that from the previous subsection) with no redundancies would
immediately lose a fraction $D$ of information encoded across neurons in the
first dropout layer and then a fraction $D$ again in the subsequent dropout
layers, giving an effective information capacity of
$\tilde{C}_{\mathit{dropout}}\sim C(1-D)^{N}$, after $N$ dropout layers, where
$C$ is the capacity of the noiseless channel (Fig. 2b).

Figure 2: Redundancy in channel code enables noise robustness. a. Illustration
of the effect of dropout on the three types of code introduced in Figure 1.
While dropout corrupts information in the monosemantic (top, circle) and
superposition (bottom, star) codes, a polysemantic code can recover from
dropout. b., c. Network capacity as a function of network for different levels
of dropout $\alpha$ for monosemantic (b, circle), superposition (b, star) and
polysemantic code (c, square).

However, by utilizing a redundant encoding, it is possible to introduce an
error-correcting component into the code, which is resistant to some noise
sources. When error-correction is present, the neural network can reliably
transmit the entire channel capacity of a layer with dropout through every
layer of the network (Fig. 2c, see Supplementary Note for detailed
derivation),

$\displaystyle C_{\mathit{dropout}}\sim C(1-D).$ (2)

We verified that a neural network trained as an autoencoder could indeed learn
error-correction by measuring the mutual information between the input and the
hidden layers[3] and observing that after training, the network does not
experience exponential decay of information (see Supplementary Note for
details).

Thus, when a neural network is exposed to moderate noise, we expect the code
to become more redundant, distributing the representation of inputs across
multiple activation patterns. While this distribution of representations
across neurons discourages a monosemantic code (Fig. 2a), it is unclear
whether it might facilitate the holistic interpretation of activations across
a given layer.

Codes produce eigenspectra with varying speeds of decay. We next turned to
probe the high-level properties of a neural network’s code. A natural approach
that was recently proposed in the study of coding properties of biological
systems[40, 23] is to investigate the eigenspectrum of the activation
covariance matrix. The shape of the eigenspectrum can reflect the properties
of the network’s code. In particular, a largely redundant code has an
eigenspectrum with a steep decay since only a small subset of possible
configurations is actually relevant to the network’s code (Fig. 3a). In
contrast, a maximally non-redundant code will exhibit a more gradual decay in
the eigenspectrum since it exploits the entire space of possible
configurations (Fig. 3a). In practice, eigenspectra tend not to match either
extreme and instead decay according to a power law with parameter $\alpha$.
Applying this approach to deep neural networks, we find the same power law
decay but with varying values of $\alpha$. By using this encoding, the
effective channel capacity becomes approximately.

$\displaystyle C_{\alpha}\approx C-|\alpha|m\log(m/e),$ (3)

where $m$ is the width of the given layer and $C$ is the channel capacity
using the optimal dense encoding (see Supplementary Note for detailed
derivation).

Figure 3: The code shapes hidden activation eigenspectrum. a. Left: Schematic
illustrating input to (natural images from the CIFAR dataset) and hidden
activations of an Autoencoder-ResNet (see Methods for details). Right:
Idealized eigenspectrum for hidden activations implementing a maximally
redundant (top) or maximally non-redundant (bottom) code. b. Log-log plot of
eigenvalues as a function of principal components for hidden activations from
different network depths (shades of grey and legend in bottom right). Dashed
line indicates the power law fit obtained from Huber regression (see Methods).
Large deviations from the power law distribution exist for later layers, hence
power law fits represent approximations. Note that the later eigenspectra
resemble the predicted spectrum of a linear code (compare a), where the code
words are simple linear combinations. c. Estimated power law exponent $\alpha$
as a function of network depth. Dashed line marks $\alpha=-1$. d. Sample of
different eigenvectors (rows) displayed as filters of the ResNet.

When visualizing the eigenspectrum of the hidden activations of a trained
Autoencoder-ResNet trained on the ImageNet dataset without dropout (Fig. 3a),
we find that the decay of the eigenspectrum changes non-monotonically over the
depth of the network (Fig. 3b). The eigenspectrum initially decays with a
power law exponent just below $-1$, which is consistent with the observation
that the spectrum of natural images decays with that same exponent[40]. The
eigenspectrum decay then slows down, with the power law exponent increasing
above $-1$ for the intermediate layers of the network, before eventually
decaying rapidly with a power law exponent $\leq-1$ for the final layers (Fig.
3c). Consistent with results from neuroscience[40], the corresponding
eigenfilters vary from coarse global filters for the first eigenvectors to
intricate filters sensitive to local variations for later eigenvectors. We
note that the recovered eigenvectors resemble previously described Gabor
receptive fields[24] as well as curve detectors[8] (Fig. 3d).

In summary, the eigenspectrum of a trained Autoencoder-ResNet reveals that the
trained network learns a code that is neither highly redundant nor one that
maximizes capacity. Instead, we find that the neural code exhibits some
redundancy, particularly towards early and late layers, while employing less
redundant codes in intermediate layers.

Random projections reflect smoothness of code. What can we say about the
neural network code beyond general statements regarding its redundancy? We
hypothesized that an input that is encoded with more redundancy might be
easier to decode with linear probes[1]. To this end, we projected the $m$
hidden activations produced by a moving dot stimulus with $n$ frames (Fig.
4a), $A\in\mathbb{R}^{n\times m}$, onto a randomly chosen basis of vectors
with normalized variance, $r\in\mathbb{R}^{m\times
3},r_{i}\sim\mathcal{N}(0,\frac{1}{m}I)$. These random projections, $z=Ar$,
have the useful property that their average action, $\langle
E[z]\rangle=\frac{1}{2}\int||\dot{z}(t)||^{2}\mathrm{d}t$, is proportional to
the decay factor $\alpha$,

$\displaystyle\langle
E[z]\rangle=\sum_{k=1}^{m}\lambda_{k}\sim\frac{1}{2}\zeta({-\alpha}),$ (4)

where $\lambda_{i}$ are the eigenvalues of the hidden activations and $\zeta$
is the Riemann zeta function restricted to the real numbers,
$\zeta(s)=\sum_{k=1}^{\infty}\frac{1}{k^{s}}$.

Figure 4: Random projections of hidden activations exhibit varying amount of
smoothness. a. Simple moving dot stimulus generated from the $\sin$ and $\cos$
function (top), converted into a stack of frames (middle), and passed into the
network to yield hidden activations (bottom). b. Sample of random projections
of hidden activations produced from the moving dot stimulus. Color of box
indicates layer of the network, and color of the random projection indicates
time. Inset text shows average activation. c. Average action of random
projections as a function of network depth. Average computed over 1000 random
projections (see Methods). d. Average action of random projections as a
function of the estimated power law exponent $\alpha$.

When displaying the random projections as curves, we indeed observed that
while early layers produce highly discontinuous projections, later layers of
the network recover some of the smooth structure of the input (Fig. 4b). In
particular, we noticed that the transition between discontinuous and locally
smooth occurs as the decay factor $\alpha$ falls below $-1$ (Fig. 4c). From
eq. 4, we see that in this regime, the expected action of the curve is finite,
while in the regime $\alpha>-1$, the action of the curve grows without bound.

The slow decay of eigenspectra in artificial neural networks stands in stark
contrast to the eigenspectra in the visual cortex of the mouse[40] and the
macaque[23]. In this biological network, the power law exponent $\alpha$ is
consistently below $-1$, constraining the network to use a continuous and
differential code of the input. Instead, the artificial neural network
resembles the much smaller zebrafish[43] with a slowly decaying eigenspectrum
with $\alpha\approx-0.7$.

Taken together, we found that the speed of decay of the eigenspectrum
determines the statistical properties of random projections of stimuli. A fast
decay ($\alpha\leq-1$) is common in larger mammals[40, 23] and results in
continuous and differentiable projections. A slow decay ($\alpha>-1$) is more
prominent in small artificial neural networks and in the zebrafish[43] and
implies discontinuous random projections.

Deviations from power law distribution In the final layers of figure 3 we can
see that the eigenspectrum in some cases systematically deviates from the
power law. These deviations stem from transitions from the ‘biological power-
law’ case to the optimal ‘redundant code’ case (the step function in Fig. 3a),
which helps protect the information from loss. To test this hypothesis we
investigate the effect of higher and lower levels of dropout noise on the
power spectrum (Fig. 5).

Figure 5: Higher noise induces more robust codes. Log-log plot of eigenvalues
as a function of principal components for hidden activations for two different
networks with different dropout rates. The increase in non-linearity indicates
a learnt code which is more robust to noise (which dropout introduces).

In early layers, both the high and low dropout cases are well described by a
simple power-law. In contrast, intermediate and late layers exhibit
increasingly pronounced step-like spectra, especially in the higher-dropout
layer. These properties indicate a large linear redundancy in the code, which
is a characteristic signature of error-correcting codes.

Identifying the exact error-correcting code is beyond the scope of this paper,
but we note that the non-linear decay is not a sharp step function, as would
be the case with a linear error-correcting code. While this decay is
consistent with a suboptimal encoding, we hypothesise that instead, the learnt
code is not entirely linear. Consistently, the dropoff becomes more pronounced
in later layers of the network, likely because linear error correction early
in the network is both challenging (there are fewer layers to arrange the
information into a code) and less necessary (as the image already contains
many redundancies, minimising damage done by losing a couple of pixels).

Dropout incentivizes fast decay of eigenspectrum. As a network’s code depends
on the amount of noise present (Fig. 2), it is natural to ask how the learned
code changes as the dropout rate increases. We systematically varied the
amount of dropout the network experiences (Fig. 6a) and observed that higher
dropout generally produce lower power law exponents (Fig. 6b). Interestingly,
already for a dropout rate of $20\%$, the power law exponent remains very
close to $-1$, as in the visual cortex of the mouse[40] and the macaque[23].
Consistent with our observation in networks without dropout (Fig. 4), networks
with lower power law exponent also tend to produce random projections with
lower average action (Fig. 6c).

Figure 6: Dropout during training leads to rapidly decaying eigenspectra and
smooth random projections. a. Schematic illustrating dropout across all layers
with different strengths (0% top, 40% middle, 80% bottom). b. Estimated power
law exponent $\alpha$ as a function of network depth for different levels of
dropout (shades of green). c. Average action of random projections as a
function of network depth for different levels of dropout. d. Average
percentage of input variance explained by random projections as a function of
network depth for different layers of dropout. Inset shows three examples of
random projections. e. Average percentage of input variance explained by
random projections as a function of the average action of the random
projection.

To test whether networks with higher dropout might encode more robust
representations of their input throughout their layers, we computed the
variance of the input explained by random projections from different layers of
the network (see Methods). We found that the explained variance increases with
both dropout rate and network depth (Fig. 6d). Consistently, we found that
trajectories with low average action tend to explain a higher fraction of
input variance (Fig. 6e).

These findings suggest that introducing dropout during training can reliably
produce networks with a faster-decaying eigenspectrum. Random projections from
these networks exhibit low average action and tend to resemble robustly and
linearly encode some global features of the input.

Efficient codes provide a framework for understanding language model
interpretability. The success of deep learning models in natural language
processing, particularly the transformer architecture[42], provides a range of
new challenges and opportunities for interpretability research[11, 12, 6]. A
prominent challenge is the unprecedented size of large language models[7],
which makes the use of interpretability methods that rely on human annotation
of individual components infeasible. A recent study proposes a GPT-assisted
method for generating single neuron explanations in a scalable fashion[6].
While the proposed method manages to provide explanations with predictive
power for thousands of neurons, the approach is limited to cases without
polysemanticity and superposition, which our work suggests is a small
fraction. To investigate whether our approach of interpreting random
projections might be applicable to this setting, we turned to analyze the
eigenspectrum of a large language model[34].

Figure 7: Codes in large language models. a. Estimated power law exponent
$\alpha$ as a function of network depth of a large language model[34]. Error
bars indicate 95% confidence interval. b. Explanation score[6] as a function
of network depth for explanations for the activation of single neurons (gray)
or random projections (green). Data extracted from ref. [6]. Error bars
indicate 95% confidence interval. c. Illustrative example of a low-action
random projection from layer 3.

We calculated the power law exponent $\alpha$ as a function of network depth.
We observed that the power law exponent consistently remains above the
critical value ($\alpha\geq-1$, Fig. 7a), indicating unbounded variance of
random projections. This observation is consistent with the highly nonlinear
nature of natural language[27].

To investigate whether a lower power law exponent might nonetheless translate
into more expressive random projections, we compute the explanation score (see
Methods) for the activation of single neurons and random projections across
different layers of the network (Fig. 7b,c). Interestingly, we found that the
average explanation score changes non-monotonically, starting at comparatively
high values in the first layers, decreasing up until two-thirds of the network
depth, and eventually increasing again. The explanation score for random
projections is the inverse of the score for single neurons, with our method
performing well in the layers in which the single-neuron method performs
poorly. We hypothesize that the random projection and single neuron approach
may complement each other. By using both in tandem, we might obtain a more
complete picture of coding in neural networks, bringing us close to the goal
of understanding the outputs of large models.

##  3 Discussion

In this study, we explored the properties of neural network codes and their
implications for interpretability, leveraging concepts from information theory
and neuroscience. Our analysis revealed that the eigenspectrum of the
activation covariance matrix provides insights into the redundancy of a
network’s code and its capacity for error-correction. Our results demonstrated
that networks with higher dropout levels tend to employ codes with faster-
decaying eigenspectra, yield smoother random projections, and better
explanations of input variance. Furthermore, we found that our approach to
interpreting random projections might complement existing single-neuron
interpretability methods, particularly in the context of large language
models. Overall, our findings provide a top-down perspective on neural network
interpretability, advancing our understanding of the underlying structure of
these models and suggesting new avenues for future research.

A fascinating aspect studying eigenspectrum decay in artificial neural
networks is the potential connection to neuroscience[40, 23, 43]. In the same
way that sufficiently large animals exhibit a neural code that optimally
trades coding capacity and robustness to perturbations[40, 23], we hypothesize
that sufficiently large artificial neural networks might learn to strike the
same balance. Indeed, a recent study indicates that networks with a fast
eigenspectrum decay are more robust to adversarial examples and investigates
which aspects of training and network architecture result in a more optimal
decay[16]. A systematic study of how the eigenspectrum decay changes as a
function of network size might reveal important information on whether we can
expect large language models[7] to have smooth random projection.

An important direction for future research is to investigate the role of
nonlinearities in the network code. Considering that neural networks were
initially developed for nonlinear signal processing[25], it is moderately
surprising that modern networks allow linear read-out from their hidden
layers[1, 4]. Several factors incentivize linearity in hidden layers, such as
the linear structure of the residual stream[18, 11] and the training objective
of supervised learning which incentivizes linear separability of concepts[5].
Conversely, linear codes can have comparable performance to nonlinear
codes[29]; hence they are not disadvantaged under optimization pressure.
However, the exact reasons for which linear structure networks converge to[30]
is an important open question into which our current work might provide
insight.

Overall, our efficient channel coding framework provides a top-down
perspective on interpretability in neural networks and offers a promising
direction for future research in understanding deep learning models’
underlying structure and behavior.

##  4 Methods

Statistics. We perform a Principal Component Analysis on the activations of
the ReLu units of a given layer. To reduce the computational capacity required
by very high-dimensional data, we apply a subsampling method by selecting
every s-th component, where s is computed such that the resulting dimension is
less than 8000. We employ a randomized SVD solver for PCA fitting and restrict
the number of components to a maximum of 1000.

After fitting the PCA model, we perform a Huber regression analysis of the
explained variance of the PCA components to estimate each layer’s power-law
exponent ($\alpha$). The Huber regression is used due to its robustness to
outliers. Specifically, we fit the Huber regression model on the logarithm of
the PCA components’ indices (from 10 to 50) and the logarithm of their
corresponding explained variances.

We use a recently published method for automated interpretability in language
models[6], which aims to explain the activation patterns of individual neurons
in a large language model (GPT-2 XL) with a larger language model (GPT-4). We
use GPT-4 to generate explanations of random projections of the activations of
all neurons in a given layer of GPT-2. The value of these explanations is then
assessed by a third model (GPT-3.5), which predicts the activation of the
explained neuron on a held-out test set. The correlation between the expected
and the real output is then the explanation score.

Model training. We studied the eigenspectrum of a ResNet-based Variational
Autoencoder (ResNet-VAE) activations for image generation tasks. The ResNet-
VAE model is built upon the ResNet-18 architecture pretrained on ImageNet. We
remove the final fully connected layer from the ResNet architecture and
replace it with two sets of linear layers to encode latent variables mu and
logvar. The model employs two fully connected hidden layers with 1024 units
each, followed by the latent embedding layer with 256 dimensions. A dropout
rate of 0.2 is applied during training to promote regularization.

The model is trained on the CIFAR-10 dataset, which consists of 50,000 train
and 10,000 test instances of 32x32 pixel images belonging to 10 different
classes. The images are resized to 224x224 pixels and normalized before being
fed into the network. We use the Adam optimizer with a learning rate of 1e-3
to optimize the network parameters. The training process consists of 20
epochs, with a batch size of 50. The model performance is measured by a custom
loss function that combines Mean Squared Error (MSE) and Kullback-Leibler
Divergence (KLD).

In addition to the ResNet-VAE model, we studied the GPT-2 XL model [34]
through the PyTorch Hub library. We computed explanation scores with the code
published by Bills et al. (ref. [6], but computed simulation scores with text-
davinci-003 rather than GPT-4, since token logprobs are not available through
the API at the time of this study.

Code availability. After publication code will be publically available as a
GitHub repository; at this time, all experiments are available in the
supplementary material.

## References

  * [1] Guillaume Alain and Yoshua Bengio “Understanding intermediate layers using linear classifier probes” In _arXiv preprint arXiv:1610.01644_ , 2016
  * [2] Maximilian Augustin, Alexander Meinke and Matthias Hein “Adversarial robustness on in-and out-distribution improves explainability” In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16_ , 2020, pp. 228–245 Springer
  * [3] Mohamed Ishmael Belghazi et al. “Mine: mutual information neural estimation” In _arXiv preprint arXiv:1801.04062_ , 2018
  * [4] Nora Belrose et al. “Eliciting Latent Predictions from Transformers with the Tuned Lens” In _arXiv preprint arXiv:2303.08112_ , 2023
  * [5] Yoshua Bengio, Aaron Courville and Pascal Vincent “Representation learning: A review and new perspectives” In _IEEE transactions on pattern analysis and machine intelligence_ 35.8 IEEE, 2013, pp. 1798–1828
  * [6] Steven Bills et al. “Language models can explain neurons in language models”, https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023
  * [7] Tom Brown et al. “Language models are few-shot learners” In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [8] Nick Cammarata et al. “Thread: Circuits” https://distill.pub/2020/circuits In _Distill_ , 2020 DOI: 10.23915/distill.00024
  * [9] Frances Ding, Jean-Stanislas Denain and Jacob Steinhardt “Grounding representation similarity through statistical testing” In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 1556–1568
  * [10] Finale Doshi-Velez and Been Kim “Towards a rigorous science of interpretable machine learning” In _arXiv preprint arXiv:1702.08608_ , 2017
  * [11] N Elhage et al. “A mathematical framework for transformer circuits” In _Transformer Circuits Thread_ , 2021
  * [12] Nelson Elhage et al. “Toy Models of Superposition” In _arXiv preprint arXiv:2209.10652_ , 2022
  * [13] Logan Engstrom et al. “Adversarial robustness as a prior for learned representations” In _arXiv preprint arXiv:1906.00945_ , 2019
  * [14] Chris Finlay and Adam M Oberman “Scaleable input gradient regularization for adversarial robustness” In _arXiv preprint arXiv:1905.11468_ , 2019
  * [15] Ruth Fong and Andrea Vedaldi “Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8730–8738
  * [16] Arna Ghosh, Arnab Kumar Mondal, Kumar Krishna Agrawal and Blake Richards “Investigating power laws in deep representation learning” In _arXiv preprint arXiv:2202.05808_ , 2022
  * [17] Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan and Or Zamir “Planting undetectable backdoors in machine learning models” In _2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS)_ , 2022, pp. 931–942 IEEE
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep residual learning for image recognition” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [19] Lars Holmberg “Towards Benchmarking Explainable Artificial Intelligence Methods” In _arXiv preprint arXiv:2208.12120_ , 2022
  * [20] Shao-Lun Huang, Xiangxiang Xu, Lizhong Zheng and Gregory W Wornell “An information theoretic interpretation to deep neural networks” In _2019 IEEE international symposium on information theory (ISIT)_ , 2019, pp. 1984–1988 IEEE
  * [21] Evan Hubinger “Automating auditing: An ambitious concrete technical research proposal” Aug, 2021
  * [22] James Kirkpatrick et al. “Overcoming catastrophic forgetting in neural networks” In _Proceedings of the national academy of sciences_ 114.13 National Acad Sciences, 2017, pp. 3521–3526
  * [23] Nathan CL Kong, Eshed Margalit, Justin L Gardner and Anthony M Norcia “Increasing neural network robustness improves match to macaque V1 eigenspectrum, spatial frequency preference and predictivity” In _PLOS Computational Biology_ 18.1 Public Library of Science San Francisco, CA USA, 2022, pp. e1009739
  * [24] Norbert Kruger et al. “Deep hierarchies in the primate visual cortex: What can we learn for computer vision?” In _IEEE transactions on pattern analysis and machine intelligence_ 35.8 IEEE, 2012, pp. 1847–1871
  * [25] Alan Lapedes and Robert Farber “Nonlinear signal processing using neural networks: Prediction and system modelling”, 1987
  * [26] Zhizhong Li and Derek Hoiem “Learning without forgetting” In _IEEE transactions on pattern analysis and machine intelligence_ 40.12 IEEE, 2017, pp. 2935–2947
  * [27] Tal Linzen “Issues in evaluating semantic spaces using word analogies” In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP_ Berlin, Germany: Association for Computational Linguistics, 2016, pp. 13–18 DOI: 10.18653/v1/W16-2503
  * [28] Zachary C Lipton “The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.” In _Queue_ 16.3 ACM New York, NY, USA, 2018, pp. 31–57
  * [29] David JC MacKay and Radford M Neal “Near Shannon limit performance of low density parity check codes” In _Electronics letters_ 33.6, 1997, pp. 457–458
  * [30] Ari Morcos, Maithra Raghu and Samy Bengio “Insights on representational similarity in neural networks with canonical correlation” In _Advances in Neural Information Processing Systems_ 31, 2018
  * [31] Jesse Mu and Jacob Andreas “Compositional explanations of neurons” In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 17153–17163
  * [32] Chris Olah et al. “Zoom in: An introduction to circuits” In _Distill_ 5.3, 2020, pp. e00024–001
  * [33] Mensah Kwabena Patrick, Adebayo Felix Adekoya, Ayidzoe Abra Mighty and Baagyire Y Edward “Capsule networks–a survey” In _Journal of King Saud University-computer and information sciences_ 34.1 Elsevier, 2022, pp. 1295–1310
  * [34] Alec Radford et al. “Language models are unsupervised multitask learners” In _OpenAI blog_ 1.8, 2019, pp. 9
  * [35] Tilman Räukur, Anson Ho, Stephen Casper and Dylan Hadfield-Menell “Toward transparent ai: A survey on interpreting the inner structures of deep neural networks” In _arXiv preprint arXiv:2207.13243_ , 2022
  * [36] Andrew Ross and Finale Doshi-Velez “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients” In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018
  * [37] Andrei A Rusu et al. “Progressive neural networks” In _arXiv preprint arXiv:1606.04671_ , 2016
  * [38] Ravid Shwartz-Ziv and Naftali Tishby “Opening the black box of deep neural networks via information” In _arXiv preprint arXiv:1703.00810_ , 2017
  * [39] Nitish Srivastava et al. “Dropout: a simple way to prevent neural networks from overfitting” In _The journal of machine learning research_ 15.1 JMLR. org, 2014, pp. 1929–1958
  * [40] Carsen Stringer et al. “High-dimensional geometry of population responses in visual cortex” In _Nature_ 571.7765 Nature Publishing Group UK London, 2019, pp. 361–365
  * [41] Naftali Tishby and Noga Zaslavsky “Deep learning and the information bottleneck principle” In _2015 ieee information theory workshop (itw)_ , 2015, pp. 1–5 IEEE
  * [42] Ashish Vaswani et al. “Attention is all you need” In _Advances in neural information processing systems_ 30, 2017
  * [43] Zezhen Wang et al. “The scale-invariant covariance spectrum of brain-wide activity” In _bioRxiv_ Cold Spring Harbor Laboratory, 2023, pp. 2023–02