: A Unifying Framework for Inspecting
Hidden Representations of Language Models

Asma Gandehariounequal,google
Avi Caciularuequal,google
Adam Pearcegoogle
Lucas Dixongoogle
Mor Gevagoogle,uni

uniTel Aviv University
googleGoogle Research

Asma<EMAIL_ADDRESS>
Mechanistic Interpretability, Language Model, Patching, Representation, Probing

Inspecting the information encoded in hidden representations of large language models (LLMs) can explain models' behavior and verify their alignment with human values. Given the capabilities of LLMs in generating human-understandable text, we propose leveraging the model itself to explain its internal representations in natural language. We introduce a framework called  and show how it can be used to answer a wide range of questions about an LLM's computation. We show that prior interpretability methods based on projecting representations into the vocabulary space and intervening on the LLM computation can be viewed as instances of this framework. Moreover, several of their shortcomings such as failure in inspecting early layers or lack of expressivity can be mitigated by . Beyond unifying prior inspection techniques,  also opens up new possibilities such as using a more capable model to explain the representations of a smaller model, and unlocks new applications such as self-correction in multi-hop reasoning.

§ INTRODUCTION

Illustration of our framework, showing a  for decoding what is encoded in the representation of “CEO” in the source prompt (left).
We use a target prompt (right) comprised of few-shot demonstrations of token repetitions, which encourages decoding the token identity given a hidden representation. Step 1: Run the forward computation on the source prompt in the source model. Step 2: Apply an optional transformation to the source hidden state at source layer. Step 3: Run the forward computation on the target prompt up to the target layer in the target model. Step 4: Patch the target representation of “?” at the target layer with the transformed representation (from step 2), and continue the forward computation from that layer onward. Note that the modularity of  allows designing a variety of methods as one can configure the target prompt and model and the transformation.

The question of what information is captured within the hidden representations of large language models (LLMs) is of key importance in control and understanding of modern generative AI, and has drawn substantial attention recently [Casper et al., 2022, Madsen et al., 2022, Patel & Pavlick, 2021, Nanda et al., 2023]. To tackle this question, prior work has introduced a diverse array of interpretability methods, which largely rely on three prominent approaches: training linear classifiers, called probes, on top of hidden representations [Belinkov & Glass, 2019, Belinkov, 2022, Alain & Bengio, 2017], projecting representations to the model's vocabulary space [nostalgebraist, 2020, Din et al., 2023, Belrose et al., 2023], and intervening on the computation to identify if a representation is critical for certain predictions [Meng et al., 2022, Wallat et al., 2020, Wang et al., 2022, Conmy et al., 2023, Geva et al., 2023].

Despite the wide success of these methods, they each exhibit practical shortcomings. First, probing relies on supervised training for pre-defined classes, which is hard to scale when the feature of interest has a large number of classes or when all the categories are not known a priori. Second, the accuracy of vocabulary projections substantially decreases in early layers and their outputs are often hard to interpret. Last, all the above methods are not expressive: they provide class probabilities or most likely tokens, as opposed to a high-quality explanation in natural language.

In this work, we argue that the advanced capabilities of LLMs in generating human-like text can be leveraged for “translating” the information in their representations for humans. We introduce a modular framework, called  (see <ref>), that can easily be configured to query various information from LLM representations.
Given a representation, we propose to decode specific information from it by “patching” it into a separate inference pass that encourages the extraction of that information, independently of the original context.[While patching (or “activation patching”) by itself is not a new technique, to the best of our knowledge, we are the first to propose using it for decoding information from hidden representations in a configurable and expressive manner. See more details in <ref>.] A configuration of our framework (a ) can be viewed as an inspection tool geared towards a particular objective, as illustrated in Fig. <ref>.

We show that many existing methods, including those that rely on vocabulary projections and computation interventions, can be cast as . Moreover, new configurations of our framework introduce more effective tools in addressing the same questions, while mitigating several limitations of prior approaches. Additionally,  enables addressing underexplored questions, such as fine-grained analysis of the input contextualization process and the extent to which a more expressive model can be used to inspect hidden representations of a smaller model.

We conduct a series of experiments to evaluate the benefits and opportunities introduced by , focusing on auto-regressive LLMs. First, we consider the problem of estimating the model's next-token prediction from its intermediate representations (see <ref>). Across multiple LLMs, we show that using a few-shot token identity prompt[A prompt in the form of $``\texttt{tok}_1 \rightarrow \texttt{tok}_1 \texttt{; tok}_{2} \rightarrow \texttt{tok}_2 ; \ldots; \texttt{tok}_{k}"$ where $\texttt{tok}_{i}$ refers to a random token.] leads to substantial gains over vocabulary projection methods. Next, we evaluate how well  can decode specific attributes of an entity from its LLM representations, when these are detached from the original context (see <ref>). We observe that, despite using no training data,  significantly outperforms probing in six out of twelve commonsense and factual reasoning tasks, and works comparably well to all but one of the remaining six.

Beyond output estimation and attribute decoding,  can address questions that are hard to answer with existing methods.
In <ref>, we apply  to study how LLMs contextualize input entity names in early layers, where vocabulary projections mostly fail and other methods only provide a binary signal of whether the entity has been resolved, at best [Youssef et al., 2023, Tenney et al., 2019]. With a new , we are able to verbalize the gradual entity resolution process.
For example, we show in processing “Alexander the Great” throughout the layers, the model reflects different entities starting from “Great Britain”, to “the Great Depression”, to finally resolving “Alexander the Great". Then, in <ref> we show how one can further improve  expressivity by using a stronger target model, e.g., Vicuna 13B instead of Vicuna 7B.

is a novel framework for inspection of hidden representations in language models. Many prior inspection methods with various objectives can be viewed as , as detailed in the “Configuration” column (see notation description in <ref>). The rows highlighted in green show our new configurations that overcome several limitations of prior methods through more expressive inspection that is training-data free and is more robust across layers. Additionally, the generality of this framework enables novel inspection possibilities that were unexplored before. When the target prompt ($\targetPrompt$) is not specified, it means the output would be invariant to the choice of $\targetPrompt$. When not specified, $\transformation \is \identity$ and $\targetModel = \sourceModel$.

Inspection Objective     2r
Training Data Free
Robust Across Layers     Configuration

2l[HTML]DFF6CC Few-shot token identity  (<ref>)
[HTML]DFF6CC$\targetLayer \is \sourceLayer$, $\targetPrompt \is ``\texttt{tok}_1 \rightarrow \texttt{tok}_1 \texttt{; tok}_{2} \rightarrow \texttt{tok}_2 ; \ldots; \texttt{tok}_{k}"$

2l8cmLogit Lens [nostalgebraist, 2020], Embedding Space Analysis [Dar et al., 2023]

$\targetLayer \is \numTargetLayers$

2l8cmTuned Lens [Belrose et al., 2023]

For learning mappings
$\targetLayer \is \numTargetLayers, \transformation \is \text{Affine}$

-5.5*2cmInspecting Output Distribution
2lFuture Lens [Pal et al., 2023]

For learning mappings
$\targetLayer \is \sourceLayer, \transformation \is \text{Linear}, \targetPrompt \is \text{Fixed or learned soft prompt}$

2l[HTML]DFF6CC Zero-shot feature extraction  (<ref>)
[HTML]DFF6CC$\targetLayer \is j^{\prime} \in [1, \ldots, \numTargetLayers], \targetPosition \is \targetSeqLen$,$ \targetPrompt \is $ relation verbalization followed by

2l8cmLRE Attribute Lens [Hernandez et al., 2023]

1l1.75cmFor linear relation approx.
$\targetLayer \is \numTargetLayers, \transformation \is \text{Linear with additional variables},  \targetPrompt \is \sourcePrompt$

-4*2cmFeature Extraction
2l7cmProbing <cit.>
For training probe

2l[HTML]DFF6CC Entity description  (<ref>)
[HTML]DFF6CC$\targetLayer \is \sourceLayer$, $\targetPosition \is \targetSeqLen$, $\targetPrompt \is$ “$_1$$_1$ $_k$$_k$ "

2l[HTML]DFF6CC X-model entity description   (<ref>)
[HTML]DFF6CCFor learning mappings
[HTML]DFF6CC$\targetModel \is \text{a larger variant of}\,\sourceModel, \targetLayer \is \sourceLayer, \targetPosition \is \targetSeqLen$, $ \targetPrompt \is$ “$_1$$_1$$_k$$_k$"

2lCausal Tracing [Meng et al., 2022]

$\targetLayer \is \sourceLayer,
\targetPrompt \is \sourcePrompt + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma)$

-5*2cmEntity Resolution
2l6cmAttention Knockout [Wang et al., 2022, Conmy et al., 2023, Geva et al., 2023]

$\targetLayer \is \text{Multiple}, \transformation \is \constantZero,
\targetPrompt \is \sourcePrompt$

2.5*2cmInspection Application
2l8cmEarly Exiting, e.g., Linear Shortcuts [Din et al., 2023]

For learning mappings

$\targetLayer \is \numTargetLayers, \transformation \is \text{Affine}$

2l8cmCaption Generation, e.g., Linear Mapping[Merullo et al., 2022]

For learning mappings

$\targetModel \is \text{A language model of choice}, \targetLayer \is \numTargetLayers, \transformation \is \text{Affine}$

Lastly, we showcase the utility of  for practical applications in <ref>. We show how it can be used to fix latent multi-hop reasoning errors, when the model is capable of conducting each reasoning step correctly, but fails to process their connection in-context. Building on top of the data provided by Hernandez et al., 2023, we introduce a more complex task that requires two steps of factual reasoning. We then show our proposed  improves accuracy from a baseline 19.57% to 50%.

To conclude, our work makes the following contributions: We propose , a general modular framework for decoding information from LLM hidden representations. We show that prominent interpretability methods can be viewed as instances of , and new configurations result in more expressive, robust across layers, and training-data free alternatives that mitigate their shortcomings.  In addition, novel configurations introduce unexplored possibilities of stronger inspection techniques, as well as practical benefits, such as correcting multi-hop reasoning.

§ RELATED WORK

Activation patching is a causal intervention, commonly used as a tool for studying if certain activations play a key role in a model's computation [Geiger et al., 2021, Vig et al., 2020].
Patching has been used largely for localizing specific information to specific layers and token positions [Goldowsky-Dill et al., 2023, Meng et al., 2022, Meng et al., 2022, Stolfo et al., 2023], and for finding paths explaining how information propagates in the computation [Wang et al., 2022, Geva et al., 2023, Hendel et al., 2023, Hanna et al., 2023, Lieberum et al., 2023]. Despite certain limitations [Hase et al., 2023, Zhang & Nanda, 2023], patching remains a key tool for mechanistic interpretability [Conmy et al., 2023].

Given promising results from emerging interpretability efforts that employ LLMs to generate human-like text for inspection <cit.>, we argue that using patching only for localization purposes is myopic, and propose to use it for “translating” LLM representations into natural language. Very recently, patching has been used to study new problem setups <cit.>, all of which can be seen as different configurations of our proposed framework (see <ref>).

Inspecting the hidden representations of neural networks has gradually became an active research area. Probing classifiers are perhaps the most common among such efforts, ranging from linear probes <cit.> to more recent variants like Gaussian Process probes [Wang et al., 2023]. Since the emergence of the Transformer architecture [Vaswani et al., 2017], other inspection methods specific to natural language domain were born that use projections into the vocabulary space <cit.>, and more recent variants that extend them to vision transformers and use projections into the class embedding space [Vilas et al., 2023]. While various other latent inspection methods exist <cit.>, the above are the most relevant to this work, and as we show in <ref>, many of which can be cast as different  instantiations.

In this section, we introduce  and show how it extends prior interpretability methods with new capabilities. While not limited to particular LLM architectures, this work focuses on auto-regressive transformer-based LLMs.

§.§ Framework Description

The key idea in  is to leverage the advanced capabilities of LLMs to generate human-like text for “translating” the information encoded in their own hidden representations. Concretely, given a hidden representation obtained from an LLM inference pass, we propose to decode specific information from it by “patching” it into a different inference pass (of the same or a different LLM) that encourages the translation of that specific information.

Notably, the rest of the forward computation after patching can augment the representation with additional information, hence, this approach does not guarantee that the patched representation itself stores all that information. However, dispatching the representation from its original context (the source prompt) stops contextualization and guarantees that no further information from the source prompt is incorporated in the post-patching computation. Thus, our framework reveals if specific information can be decoded from the patched representation via the post-patching computation.

Given an input sequence of $\sourceSeqLen$ tokens $\sourcePrompt = \langle s_1, ..., s_{\sourceSeqLen} \rangle$ and a model $\sourceModel$ with $\numSourceLayers$ layers, $\sourceHidden_{\sourcePosition}^{\sourceLayer}$ denotes the hidden representation obtained at layer $\sourceLayer \in [1, \ldots, \numSourceLayers]$ and position $\sourcePosition \in [1, \ldots, \sourceSeqLen]$, when running $\sourceModel$ on $\sourcePrompt$. To inspect $\sourceHidden_{\sourcePosition}^{\sourceLayer}$, we consider a separate inference pass of a model $\targetModel$ with $\numTargetLayers$ layers on a target sequence $\targetPrompt = \langle t_1, \ldots, t_{\targetSeqLen} \rangle$ of $\targetSeqLen$ tokens. Specifically, we choose a hidden representation $\targetHidden_{\targetPosition}^{\targetLayer}$ at layer $\targetLayer \in [1, \ldots, \numTargetLayers]$ and position ${\targetPosition} \in [1, \ldots, \targetSeqLen]$ in the execution of $\targetModel$ on $\targetPrompt$. Moreover, we define a mapping function $\transformation({\sourceHidden}; \vtheta): \mathbb{R}^{\sourceHiddenDim} \mapsto \mathbb{R}^{\targetHiddenDim}$ parameterized by $\vtheta$ that operates on hidden representations of $\sourceModel$, where $\sourceHiddenDim$ and $\targetHiddenDim$ denote the hidden dimension of representations in $\sourceModel$ and $\targetModel$, respectively. This function can be the identity function, a linear or affine function learned on task-specific pairs of representations, or even more complex functions that incorporate other sources of data.
The patching operation refers to dynamically replacing the representation $\targetHidden_{\targetPosition}^{\targetLayer}$ during the inference of $\targetModel$ on $\targetPrompt$ with $\transformation(\sourceHidden_{\sourcePosition}^{\sourceLayer})$.
Namely, by applying ${\targetHidden}_{\targetPosition}^{\targetLayer} \is \transformation(\sourceHidden_{\sourcePosition}^{\sourceLayer})$, we intervene on the generation process and modify the computation after layer $\targetLayer$.

Overall, a $\methodName$ intervention applied to a representation determined by $(\sourcePrompt,\sourcePosition,\sourceModel,\sourceLayer)$,
is defined by a quintuplet $(\targetPrompt,\targetPosition,\transformation,\targetModel,\targetLayer)$ of a target prompt $\targetPrompt$, a target position $\targetPosition$ in this prompt, a mapping function $\transformation$, a target model $\targetModel$, and a target layer $\targetLayer$ of this model.
It is possible that $\sourceModel$ and $\targetModel$ are the same model, $\sourcePrompt$ and $\targetPrompt$ are the same prompt, and $\transformation$ is the identity function $\identity$ (i.e., $\identity(\sourceHidden) = \sourceHidden$).
In the following sections, we show how this formulation covers prior interpretability methods and further extends them with new capabilities.

§.§  Encompasses Prior Methods

We show how prominent interpretability methods can be cast as  instances. See a summary in Tab. <ref>.

Recent methods inspect LLM representations by projecting them to the output vocabulary space [Dar et al., 2023, nostalgebraist, 2020, Din et al., 2023, Belrose et al., 2023].
Formally, an estimation of the output distribution is obtained from the representation $\vh_i^\ell$ at position $i$ and layer $\ell$ by:

$$\vp_i^\ell = \softmax(W_Uf(\vh_i^\ell)) \in \mathbb{R}^{|V|} ,$$

where $W_U \in \mathbb{R}^{|V|\times d}$ is the model's unembedding matrix and $f$ is a simple mapping function, such as the identity function or an affine mapping. We note that the operation applied to $f(\vh_i^\ell)$ is the same computation applied by the model to the last-layer representation for obtaining the next-token prediction. Therefore, prior methods that inspect representations in the vocabulary space can be viewed as a class of  that maps representations from any source layer $\sourceLayer$ to the last target layer $\numTargetLayers$. Differences between these methods lie in the choice of $\transformation$; logit lens [nostalgebraist, 2020, Dar et al., 2023] applies the identity function, linear shortcuts [Din et al., 2023] uses a linear mapping function, and tuned lens [Belrose et al., 2023] trains an affine mapping. Recently, Hernandez et al., 2023 introduced LRE Attribute Lens that builds $\transformation$ based on a relation linearity assumption, and they showcase its effectiveness in attribute extraction.

This class of methods has proven to be effective for different applications, for example, in improving inference efficiency via early exiting [Din et al., 2023]. While the majority of methods and applications in this category use a single model ($\targetModel = \sourceModel$), Merullo et al., 2022 had demonstrated successful caption generation with a generative image model as $\sourceModel$ and a language model as $\targetModel$.

Another category of inspection methods intervene on the LLM computation. Contemporary to our work, Pal et al., 2023 have investigated whether it is possible to anticipate multiple generated tokens ahead from a given hidden representation, rather than estimating just the next-token prediction. Their method, called Future Lens, uses a target prompt that is different from the original prompt (i.e., $T \neq S$) and is designed to decode subsequent tokens from information encoded in a hidden representation $\vh_i^\ell$. Example target prompts are The multi-tokens present here are  and Hello! Could you please tell me more about .
Notably, Future Lens can be cast as another  with $\targetModel = \sourceModel$ and $\targetLayer = \sourceLayer$.

More broadly,  also cover recent mechanistic interpretability methods that analyze internal processes in LLMs with
inference computation
interventions. Specifically, causal tracing [Meng et al., 2022] uses a source prompt augmented with Gaussian noise as the target prompt. Other previous methods have intervened on one or more target layers during inference by patching zero vectors to the computation [Wang et al., 2022, Conmy et al., 2023, Geva et al., 2023], namely, setting $\transformation(\sourceHidden) = \mathbf{0}$.

§.§  Enables Novel Inspection Methods

Prior work has utilized specific patching configurations for interpretability, largely focusing on patching the same model while using the same prompt (i.e., $\mathcal{M}^* = \mathcal{M}$, $T=S$).
The framing of  introduces a wide range of unexplored configurations that could potentially unlock new inspection capabilities.

Specifically, we observe that modifying the target prompt enables an expressive decoding of a wide range of features, detached from the source prompt computation. For instance, we can use the prompt “The capital of X is” to check if the capital city of a given country is extractable from its (last token) hidden representation at a specific layer.
Similarly, a prompt like “Tell me facts about X” can be leveraged to assess whether the model has resolved the entity name in a specific layer.
Contrary to probing, this approach is not restricted by the number of classes of the chosen feature.

Moreover, patching the representation into a more capable model could be useful in cases when the inspected model is not expressive enough to answer particular queries [Hernandez et al., 2022, Singh et al., 2023, Schwettmann et al., 2023].

In the next section, we show that these new extensions substantially outperform existing methods for querying specific information in the computation of LLMs.

§ EXPERIMENTS

We evaluate how well our framework allows decoding different types of information from LLM representations, including next-token predictions (<ref>) and specific attributes (<ref>). Then, we demonstrate the new possibilities that  introduce, focusing on analyzing the contextualization of entity names (<ref>) and leveraging stronger models for inspection via cross-model patching (<ref>). Tab. <ref> summarizes the new proposed  and their configurations compared to prior work.

§.§ Decoding of Next-Token Predictions

As introduced in <ref>, let $\vp^\numSourceLayers$ be the output probability distribution for some input, obtained by multiplying the final-layer last-position hidden representation $\sourceHidden^\numSourceLayers$ by the unembedding matrix $W_U\in \mathbb{R}^{|V|\times d}$. We wish to estimate $\vp^L$ from intermediate representations $\mathbf{h}^\ell$ s.t. $\ell < L$.
Particularly, we ask how early in the computation the model has concluded its final prediction from the given context.
In our experiments, we consider multiple LLMs – LLaMA2 (13B) [Touvron et al., 2023], Vicuna (13B) [Chiang et al., 2023], GPT-J (6B) [Wang & Komatsuzaki, 2021], and Pythia (12B) [Biderman et al., 2023] (see more details in <ref>).

Precision@1 ($\uparrow$ is better) and Surprisal ($\downarrow$ is better) of next-token prediction estimation in Vicuna (13B), LLaMA2 (13B), Pythia (12B), and GPT-J (6B). From layer 10 and upwards, the token identity method (ours) consistently outperforms the rest of the baselines across all the models.

We compare vocabulary projection methods (<ref>) with a new .
Each method yields an estimated output probability $\tilde{\vp}^\sourceLayer$ by patching an intermediate representation $\sourceHidden^\sourceLayer$ to the model's final layer. Here, we focus on the common setting where $\sourceModel = \targetModel$, and discuss extensions to $\sourceModel \neq \targetModel$ in <ref>.

* Logit Lens: Following prior work [nostalgebraist, 2020, Geva et al., 2022], we define $f$ as the identity function, meaning no change is applied to the patched representation. That is, $f(\vh) := \mathbb{I}(\vh)$.

* Tuned Lens: Motivated by Belrose et al., 2023, Din et al., 2023, we employ an affine mapping function between representations at layer $\sourceLayer$ and the final layer $\numSourceLayers$.
Specifically, we feed the model examples from a training set $\mathcal{T}$ and for each example $s\in\mathcal{T}$ obtain a pair  $(\vh_{s}^\ell, \vh_{s}^L)$ of hidden representations. Then, we fit a linear regression model to find the matrix $A^{\sourceLayer} \in \mathbb{R}^{d \times d}$ and the bias vector $\vb^{\sourceLayer}\in \mathbb{R}^{d}$ that are numerical minimizers for $\sum_{s \in \mathcal{T}} || A \vh_{s}^\ell - \vh_{s}^{L} + \vb ||^2$.
We define $\transformation$ as:

\begin{equation*}
\label{eq:linear_jump}
f(\vh^\ell) \coloneqq A^{\ell} \vh^\ell + \vb^{\ell} .
\end{equation*}

* Token Identity : Unlike the previous methods, here we use a target prompt that is different from the source prompt ($\targetPrompt \neq \sourcePrompt$) and is meant to encourage decoding the token identity of the hidden representation. Also, while the above methods skip the computation between layers $l$ and $L$, here we modify it such that all the information from the source prompt computation is discarded,
except for the patched representation.
We craft a prompt with $k$ demonstrations representing an identity-like function, formatted as “”. See <ref> for further details and an experiment showing this method's robustness to different demonstrations.
Note that this  does not require any training.

We follow Din et al., 2023 and evaluate the estimated prediction on the Pile evaluation set (see <ref> for details) using two metrics:

* Precision@1 ($\uparrow$ is better): The portion of examples for which the highest-probability token $t$ in the estimated probability distribution matches the highest-probability token in the original output distribution. That is, if $\argmax_{t}(\tilde{\vp}^\sourceLayer_t) = \argmax_{t}(\vp^\numSourceLayers_t)$.

* Surprisal ($\downarrow$ is better): The minus log-probability of the highest-probability token in the predicted distribution $\tilde{\vp}^\ell$ according to $\vp^L$, i.e., $-\log \vp^L_{\tilde{t}}$, where $\tilde{t}=\argmax_t(\tilde{\vp}^\sourceLayer_t).$

Results As shown in Fig. <ref>, across all the models, from layer 10 and upwards, the token identity  consistently outperforms the other baselines, obtaining a gain of up to 98% in layers 18-22.
This demonstrates the utility of leveraging the model's decoding procedure for inspecting representations of different source prompts, and shows that in most cases hidden representations in early layers carry out the prediction information regardless of their context.

In the first 10 layers, performance of all methods is substantially lower, with the token identity prompt performing on-par to logit lens and worse than tuned lens, in terms of precision. The advantage of tuned lens on this task might be attributed to the additional training of the mappings. The overall low performance is expected, as the first layers contextualize the input. In <ref>, we introduce a  that is geared towards unraveling this process.

§.§ Extraction of Specific Attributes

Classification probes are arguably the most commonly used method for checking if certain attributes are encoded in hidden representations [Belinkov, 2022, Belinkov & Glass, 2019]. However, they need to be trained, and the range of attribute classes needs to be known a priori. Here we show that repurposing  for attribute extraction overcomes these limitations. First, it does not require training. Second, it is not limited by a predefined set of labels, but rather can benefit from an open vocabulary. In addition, by taking advantage of the model's nonlinearities, it is more flexible in capturing complex relations compared to linear probes.

Experimental Setup
Consider factual and commonsense knowledge represented as triplets $(\subject, \relation, \object)$ of a subject (e.g., $``\texttt{United States}"$), a relation (e.g., $``\texttt{largest city of}"$), and an object ($``\texttt{New York City}"$).
We investigate to what extent the object $\object$ can be extracted from the last token representation of the subject $\subject$ in an arbitrary input context.
To this end, we conduct experiments on 8 commonsense and 25 factual knowledge tasks curated by Hernandez et al., 2023. This dataset includes $(\subject, \relation, \object)$ triplets for different relations, along with prompt templates that verbalize them in natural language.
We conduct experiments with GPT-J (6B) [Wang & Komatsuzaki, 2021], filtering the data to keep only the examples where $\object$ appears in the the model's continuation of the prompt up to 20 tokens. For each example, we sample 5 utterances from the WikiText-103 dataset [Merity et al., 2016] that include $\subject$ and use them as $\sourcePrompt$. Lastly, we keep tasks with at least 15 samples, which results in 5 commonsense and 7 factual tasks with a total of 1,453 datapoints.
For more details, see <ref>.

Methods We devise a  for feature extraction and compare it with linear probing [Köhn, 2015, Gupta et al., 2015] as a baseline.

* Zero-shot Feature Extraction :
We craft $\targetPrompt$ as a general verbalization of $\relation$ followed by a placeholder for $\subject$, such that $\targetPosition = \targetSeqLen$. For example, we use $\targetPrompt \is $ “” with “” as a placeholder for the subject. To extract the object from the entity representation in $\sourcePrompt$, we patch the representation of token “” at layer $\targetLayer$ with the representation of “” from layer $\sourceLayer$, and consider if
the generated text includes $\object$.
The remaining configurations of this  are $\transformation \is \identity, \targetModel \is \sourceModel, \sourcePosition \is $ the last token of $\subject$ in $\sourcePrompt$.
We consider all combinations of $\sourceLayer \in [1, \ldots, \numSourceLayers] \times \targetLayer \in [1, \ldots, \numTargetLayers]$. Later in this section, we discuss the role of $\sourceLayer$ pertaining to attribute extraction.

* Logistic Regression Probe: Let $\objects$ represent the range of possible objects for a given relation. We use the set of unique values of $\object$ in the training set as a proxy for $\objects$. We train a logistic regression probe [Köhn, 2015, Gupta et al., 2015] for each layer that predicts $\object \in \objects$ from the last token representation of $\subject$. Given that 6 out of 12 tasks have fewer than 40 datapoints, we use three-fold cross-validation for training and evaluation of this baseline. Note that we have excluded tasks where the probe fails completely due to insufficient number of training examples (fewer than 15 datapoints).

Evaluation We measure the average attribute extraction accuracy. For a given sample, the  is considered correct if $ \exists \, \targetLayer \in [1, \ldots, \numTargetLayers]$ where the generated text up to 20 tokens includes $\object$. For the probe, a prediction is correct if the highest probability is assigned to $\object$.

Results Tab. <ref> summarizes the results, averaged over $\sourceLayer \in [1, \ldots, \numSourceLayers]$. We conduct a T-test with Bonferroni correction to compare the two methods. Despite using no training data and having no restrictions on the output, the  achieves a significantly higher accuracy than the probe on six out of twelve tasks $(p\!<\!1e\!-\!5)$, and works comparably well in all but one of the remaining six. These results suggest that, in the majority of cases, the source representation without its original context carries enough information about many attributes that a targeted  can extract.

Comparing Zero-Shot Feature Extraction  to a Logistic Regression Probe shows that despite using no training data, it has a significantly higher accuracy than baseline in 6 out of 12 tasks. We use pairwise t-test with Bonferroni correction for comparing the accuracy of the two methods. $^{**}$ and $^*$ indicate $p\!<1e\!-\!5$ and $p\!<\!1e\!-\!4$, respectively.

2cAccuracy (mean$\pm$std)

2c27.5emTask      Logistic Regression Probe     Zero-shot Feat. Ext.

5.5cm90Commonsense     Fruit inside color     $37.4 \pm 6.6$     $38.0 \pm 18.7$

Fruit outside color      $35.5 \pm 3.1$     $\mathbf{71.0 \pm 13.3^{**}}$

Object superclass     $\mathbf{65.6 \pm 10.5^{*}}$     $54.8 \pm 11.3$

Substance phase     $73.8 \pm 3.7$     $\mathbf{91.9 \pm 1.7^{**}}$

Task done by tool     $10.1 \pm 3.2$     $\mathbf{48.1 \pm 13.2^{**}}$

7.5cm90Factual     Company CEO      $5.0 \pm 2.6$     $\mathbf{47.8 \pm 13.9^{**}}$

Country currency     $17.7 \pm 2.2$     $\mathbf{51.0 \pm 8.9^{**}}$

Food from country     $5.1 \pm 3.7$     $\mathbf{63.8 \pm 11.3^{**}}$

Plays pos. in sport     $75.9 \pm 9.1$     $72.2 \pm 7.2$

Plays pro sport     $53.8 \pm 10.3$     $46.3 \pm 14.2$

Product by co.     $58.9 \pm 7.2$      $63.2 \pm 10.7$

Star constellation     $17.5 \pm 5.3$     $18.4 \pm 5.1$

Performance Breakdown Across Source Layers
We study how the accuracy changes across the source layers $\sourceLayer \in [1, \ldots, \numSourceLayers]$. Fig. <ref> shows one commonsense and one factual reasoning task as exemplars. Results for all other tasks follow similar trends and can be found in <ref>. Considering the early layers, we observe that the  consistently outperforms the baseline. This further confirms our hypothesis that prior methods are particularly limited in surfacing information in the early layers, which often cannot be decoded via linear functions.
However,  is able to extract such attributes significantly earlier.
Note that this observation does not mean the attribute is explicitly encoded in a representation, but that there is enough information encoded such that the attribute can be extracted from the representation alone, without its original context, using the model's computation.
In the middle layers,  works similarly or better than the baseline.

Interestingly, we observe that almost all cases where  performs worse than the baseline occur in later layers (see the gradual decline in Fig. <ref>). Our interpretation is that given the language modeling training objective, the representations shift toward next-token prediction in the later layers. Therefore, the attribute of interest would not be as readily accessible via the model's computation in these layers.
This interpretation is also aligned with recent findings that show no decline in using linear relational embedding in predicting $\object$ only when the next token also happens to be $\object$ [Hernandez et al., 2023]. We postulate that when the immediate next token is not the attribute of interest, it does not necessarily mean the attribute information is lost, but rather it may not be accessible on the surface. We hypothesize that using a  with a more expressive mapping $\transformation$ could improve attribute extraction accuracy in the later layers, which we leave for future work. For additional analyses see <ref>.

Attribute extraction accuracy across source layers ($\sourceLayer$). Left: Task done by tool (commonsense), 54 Source prompts, 12 Classes. Right: Country currency (factual), 83 Source prompts, 14 Classes. Zero-Shot Feature Extraction  works consistently better than Logistic Regression Probe across all layers. There is a decline in  accuracy in later $\sourceLayer$ as the source representations shift toward next-token prediction.

§.§ Analyzing Entity Resolution in Early Layers

The previous sections focused on analyzing the information encoded in a single hidden state. Here we turn to consider a more global question of how LLMs resolve entity mentions across multiple layers. Concretely, given a subject entity name, such as “the summer Olympics of 1996”, how does the model contextualize the input tokens of the entity and at which layer is it fully resolved?

Answering these questions is hard with existing methods; vocabulary projections focus on the output prediction and fail to show clear patterns in early layers, and probing is restricted to outputs from a fixed number of classes, which may not be expressive enough to describe this process.
Alternative approaches have studied this process indirectly via interventions [Meng et al., 2022], showing that the model constructs a subject representation at the last token of the entity name. However, it is still unclear how this contextualization is performed.

We analyze how LLMs contextualize input entity names by leveraging .
Particularly, we craft a target prompt for generating a description of a given subject, and apply it to the hidden representation at the last subject position in the source prompt – where the model forms the subject representation [Geva et al., 2023, Hernandez et al., 2023] – across the early layers. This will allow us to see how the model describes the subject in each layer.

Analysis Setting
We use a few-shot target prompt template for decoding an entity description: “”, while patching the last position corresponding to .
We take the 200 most popular and 200 least popular subject entities from the PopQA dataset [Mallen et al., 2023]. The popular entities should appear frequently in LLMs' pre-training data, and are thus likely to be captured by the model, while resolving the rare entities is expected to be more challenging [Kandpal et al., 2023, Mallen et al., 2023].
Then, for the source prompt we use the entity name, and for the target prompt we sample $k=3$ random subject entities. We obtain a short (up to one sentence) description of every subject entity from Wikipedia. Our target prompt and more technical details are provided in <ref>.
For the analysis, we patch the last position representations from the first 10 layers of Vicuna 13B to the target prompt and evaluate the generated subject name and description. Specifically, the generated descriptions are evaluated against the descriptions from Wikipedia using RougeL [Lin, 2004]. Evaluation with Rouge1 [Lin, 2004] and Sentence-Bert [Reimers & Gurevych, 2019] shows similar trends (see <ref>).

Illustrating entity resolution via qualitative examples. The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representation, referred to as “Tokens Processed”. The “Explanation” column explains what the generation seems to be referring to and how that relates to the tokens processed. Here, $\targetModel \is \sourceModel, \targetLayer \is \sourceLayer$.

Tokens Processed     $\sourceLayer$      Generation        Explanation

8*90$\sourceModel \is$ Vicuna 13B
6*90$\sourcePrompt \is \texttt{"Diana,\;Princess\;of\;Wales"}$

: Country in the United Kingdom

: Country in Europe

: Title held by female sovereigns in their own right or by queens consort
Princess of Wales (unspecific)

: Title given to the wife of the Prince of Wales (and later King)
Princess of Wales (unspecific)

: Diana, Princess of Wales (1961-1997), the first wife of Prince Charles, Prince of Wales, who was famous for her beauty and humanitarian work
Diana, Princess of Wales

8*90$\sourceModel \is$ Pythia 12B
8*90$\sourcePrompt \is \texttt{"Alexander\;the\;Great"}$

Britain: Country in the European Union
Great Britain

Wall Street Crash of 1929: Financial crisis in the United States
the GreatDepression

Wall Street Bubble: The Great Depression
the Great Depression

Wall Street: Wall Street in New York City
Wall Street (related to the Great Depression)

: Ancient Greek ruler, and the first to rule all of the then known world
Alexander the Great

Tab. <ref> illustrates the generated text for two subject entities, one per model, when patching their representations at different layers to the target prompt (for more examples see <ref>). For most entities, the contextualization process is spread over the first layers, with the last subject token gradually encompassing more distant positions across layers.

This trend can be quantitatively observed by the similarity between the generated descriptions and the descriptions from Wikipedia, as measured by RougeL. See Fig. <ref> where $\sourceModel = \targetModel$. For both models, similarity increases in the first 5 layers and then slowly decreases. This decrease could potentially be attributed to contamination caused by the representation of the placeholder token “” remaining in the early layers, when patching is applied to a later layer. Note that this potential issue is only applicable to multi-token generation scenarios as future positions can still attend to the placeholder position in early layers, potentially interfering with the model's ability to accurately generate descriptions for the patched token. See <ref> for qualitative examples corroborating this interpretation.
As expected, the scores for rare, long-tail entities are significantly lower than the scores of popular entities. See <ref> for additional results with Pythia where the smaller model seems to outperform the larger model, possibly because the larger model is biased toward output generation at the expense of input contextualization. Taken together, this analysis shows the utility of  for inspecting the contextualization process in early layers of LLMs.

RougeL scores of the generated descriptions against descriptions from Wikipedia, using the Vicuna models.

§.§ Expressiveness from Cross-Model Patching

A possible avenue for improving inspection capabilities is to explain a given model with a model that is more expressive [Bills et al., 2023]. In the context of , this means to patch a representation of $\mathcal{M}$ into a more expressive model $\mathcal{M}^*$. However, it is not clear if such an intervention would yield plausible results, due to possible discrepancies between the two models resulting from different architectures, optimization processes, and so on.

We show that when patching across models from the same family, such interventions are possible and can improve expressiveness.
Specifically, we consider patching representations across different sizes of Vicuna ($\sourceModel\!\!\is\!\!\text{7B}, \targetModel\!\!\is\!\!\text{13B}$) and Pythia ($\sourceModel\!\!\is\!\!\text{6.9B}, \targetModel\!\!\is\!\!\text{12B}$), and measure how well the larger model estimates the next-token predictions and entity resolution process of the smaller model.

Next-Token Prediction We repeat the experiment in <ref>, using the token identity . To overcome discrepancies between the models, we learn affine mappings between their layers (similarly to Tuned Lens).
Fig. <ref> depicts the Precision@1 scores for different combinations of source-target layers, showing that patching with a simple affine  proves to be effective with precision of up to 0.7 and 0.8 for Vicuna and Pythia, respectively. Specifically, patching representations to an early layer of the larger model seems to be the most effective.
Furthermore, it appears that there is a subtle matching among some layers of the models, with the diagonal consistently exhibiting higher values. Similar trends are observed in the surprisal results for both models (see  Fig. <ref> in <ref>). Overall, these results show that, when $\targetModel$ and $\sourceModel$ are from the same model family, it is possible to leverage $\targetModel$ for decoding information from the representations of $\sourceModel$. Notably, our findings extend observations by Csiszárik et al., 2021 from patching representations across deep convolutional neural networks with the same architecture but different

[Vicuna: $\sourceModel\!\is\!\text{7B}, \targetModel\!\is\!\text{13B}$]
[Pythia: $\sourceModel\!\is\!\text{6.9B}, \targetModel\!\is\!\text{12B}$]
Next-token prediction estimation performance in Vicuna and Pythia with cross-model ,  measured by Precision@1 ($\uparrow$ is better).

Entity Resolution in Early Layers
We now show that using a large model as $\targetModel$ can enhance the output expressivity. To this end, we repeat our entity resolution experiment in <ref> with Vicuna model family, setting $\sourceModel\!\!\is\!\!\text{7B}, \targetModel\!\!\is\!\!\text{13B}$. Fig. <ref> shows the cross-model patching results (orange line) compared to the same-model patching.
The results show that cross-model patching from a smaller model to its larger version generally improves the ability to inspect the input contextualization, both for popular and rare entities. For Pythia, since the smaller model outperforms the larger one, cross-model patching is not as effective (see <ref>).

§ APPLICATION: SELF-CORRECTION IN MULTI-HOP REASONING

Multi-hop reasoning is a challenging problem.
While a language model may be capable of correctly answering each step independently, it could still fail at processing the connection between different steps, resulting in an incorrect prediction. Recent attempts to improve multi-hop reasoning rely on prompting the model to generate a step-by-step answer autoregressively <cit.>, some with an iterative process of self-refinement <cit.>.

However, achieving similar benefits might be possible via directly rewiring the model's intermediate computation.

In this section, we show that  can improve multi-hop reasoning performance without  generating the reasoning steps, particularly in cases when the model fails at completing a multi-hop query despite being successful in each reasoning step independently. Via , one can surgically operate on the model representations, reroute model's intermediate answer to one step of the reasoning task, simplify the consequent reasoning step, and ultimately correct the final prediction.

Experimental Setup Following the notation in <ref>, let $\tau_1 = (\subject_1, \relation_1, \object_1)$ represent the relation $\relation_1$ between a subject entity $\subject_1$ and an object entity $\object_1$. Let $\tau_2 = (\subject_2, \relation_2, \object_2)$ represent another tuple such that $\subject_2 = \object_1$. A multi-hop reasoning query pertaining to $\tau_1$ and $\tau_2$ is a prompt composed of two parts: $\pi_1$ is a verbalization of $\subject_1$ and $\relation_1$ from which $\object_1$ can be inferred; $\pi_2$ is a verbalization of $\relation_2$, from which $\object_2$ can be inferred after its concatenation with $\pi_1$. For example, Let $\tau_1 \is (```\texttt{Visual Basic}", ``\texttt{product of}", ``\texttt{Microsoft}")$ and $\tau_2 \is$ (“", “", “"). An example verbalization of these tuples would result in $\pi_1 \is $“”, and $\pi_2 \is ``\texttt{The current CEO of}"$. This leads to systematic generation of the multi-hop reasoning query $[\pi_2][\pi_1]=$“”.
Building on Hernandez et al., 2023, we systematically generate all valid multi-hop factual and commonsense reasoning queries where $\object_1 = \subject_2$. We conduct experiments on Vicuna (13B), focusing on samples where $\sourceModel$ accurately represents both $\tau_1$ and $\tau_2$ independently, that is, $\object$ appears in the next 20 tokens $\sourceModel$ generates conditioned on the prompt $\pi$ that verbalizes $\subject$ and $\relation$. This process yields 1,104 multi-hop reasoning samples, out of which 46 satisfy the above criteria and are used for evaluation. For more details, see <ref>.

Method and Evaluation We introduce a Chain-of-Thought (CoT)  to fix multi-hop reasoning via intervening on the computation graph and rerouting representation likely to capture $\object_1$ in place of $\subject_2$. Concretely, $\sourcePrompt$ refers to the formed query discussed above, and we use the following configuration: $\targetPrompt \is \sourcePrompt, \targetModel \is \sourceModel, \sourcePosition \is \sourceSeqLen, \targetPosition \is$ the token preceding $\pi_1$.
As a baseline, we evaluate a vanilla generation of $\sourceModel$ conditioned on $\sourcePrompt$ without any intervention.
We evaluate the outputs in terms of accuracy, similarly to <ref>. For a sample $\sourcePrompt$, the  is considered accurate if $ \exists \, (\sourceLayer, \targetLayer) : \sourceLayer \in [1, \ldots, \numSourceLayers], \targetLayer \in [1, \ldots, \numTargetLayers]$ where the autoregressive generation up to 20 tokens includes $\object_2$.

The interaction between source ($\sourceLayer$) and target ($\targetLayer$) layers in self-correction. The majority of success cases correspond to early-to-mid $\targetLayer$. We observe higher cumulative success rate in the lower right triangle which corresponds to $\targetLayer \le \sourceLayer$.

Results While the baseline accuracy is only 19.57%, the  achieves 50% accuracy. Fig. <ref> shows the interaction between $\sourceLayer$ and $\targetLayer$ and how it affects the success rate. Patching representations from most source layers $\sourceLayer$ into early-to-mid $\targetLayer$ (6-16) is most effective in making the right prediction. Our interpretation is that patching into late $\targetLayer$ is not effective because $\relation_2$ has already been processed and the result has been copied to the last position, therefore it cannot incorporate the proxy of $\subject_2$. We also observe that $\targetLayer \le \sourceLayer$ is more successful on average.

§ CONCLUSION

We present , a simple and effective framework that leverages the ability of LLMs to generate human-like text for decoding information from intermediate LLM representations.
We show that existing interpretability methods can be cast as specific instances of , which cover only a small portion of all the possible configurations of the framework. Moreover, using new underexplored  substantially improves our ability to decode various types of information from the model's internal computation, such as the output prediction and knowledge attributes, typically outperforming prominent methods that rely on projection to the vocabulary and probing.
In addition, our framework enables new capabilities, such as analyzing the contextualization process of input tokens in the very early layers of the model, and is beneficial for practical applications, such as multi-hop reasoning correction.
This paper only scratches the surface of the opportunities this framework creates. Future work could study its application across different domains and modalities, investigate its variants with simultaneous multi-token patching, and present recipes for task-specific and task-agonstic .

§ ACKNOWLEDGEMENTS

We thank Amir Globerson for feedback on writing and presentation of results. We thank Ardavan Saeedi, Martin Wattenberg, Ellie Pavlick, the AI Explorables team[<https://pair.withgoogle.com/explorables>] at Google Research, and Jasmijn Bastings for their helpful comments.

[Alain & Bengio, 2017]
Alain, G. and Bengio, Y.
Understanding intermediate layers using linear classifier probes.
5th International Conference on Learning Representations,
Workshop Track Proceedings, 2017.

[Belinkov, 2022]
Belinkov, Y.
Probing classifiers: Promises, shortcomings, and advances.
Computational Linguistics, 480 (1):0 207–219,

[Belinkov & Glass, 2019]
Belinkov, Y. and Glass, J.
Analysis methods in neural language processing: A survey.
Transactions of the Association for Computational Linguistics,
7:0 49–72, 2019.

[Belrose et al., 2023]
Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L.,
Biderman, S., and Steinhardt, J.
Eliciting latent predictions from transformers with the tuned lens.
arXiv preprint arXiv:2303.08112, 2023.

[Besta et al., 2024]
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda,
J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al.
Graph of thoughts: Solving elaborate problems with large language
Proceedings of the 38th AAAI Conference on Artificial
Intelligence, 2024.

[Biderman et al., 2023]
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K.,
Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.
Pythia: A suite for analyzing large language models across training
and scaling.
In International Conference on Machine Learning (ICML), 2023.

[Bills et al., 2023]
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G.,
Sutskever, I., Leike, J., Wu, J., and Saunders, W.
Language models can explain neurons in language models.
URL https://openaipublic. blob. core. windows.
net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023), 2023.

[Casper et al., 2022]
Casper, S., Rauker, T., Ho, A., and Hadfield-Menell, D.
Sok: Toward transparent ai: A survey on interpreting the inner
structures of deep neural networks.
In First IEEE Conference on Secure and Trustworthy Machine
Learning, 2022.

[Chiang et al., 2023]
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.,
Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P.
Vicuna: An open-source chatbot impressing GPT-4 with 90%*
ChatGPT quality, March 2023.
URL <https://lmsys.org/blog/2023-03-30-vicuna/>.

[Conmy et al., 2023]
Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso,
Towards automated circuit discovery for mechanistic interpretability.
In Thirty-seventh Conference on Neural Information Processing
Systems, 2023.

[Csiszárik et al., 2021]
Csiszárik, A., Kőrösi-Szabó, P., Matszangosz, Á. K.,
Papp, G., and Varga, D.
Similarity and matching of neural network representations.
In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W.
(eds.), Advances in Neural Information Processing Systems, 2021.
URL <https://openreview.net/forum?id=aedFIIRRfXr>.

[Dar et al., 2023]
Dar, G., Geva, M., Gupta, A., and Berant, J.
Analyzing transformers in embedding space.
In Annual Meeting of the Association for Computational
Linguistics, 2023.

[Din et al., 2023]
Din, A. Y., Karidi, T., Choshen, L., and Geva, M.
Jump to conclusions: Short-cutting transformers with linear
arXiv preprint arXiv:2303.09435, 2023.

[Gao et al., 2020]
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang,
J., He, H., Thite, A., Nabeshima, N., et al.
The Pile: An 800GB dataset of diverse text for language modeling.
arXiv preprint arXiv:2101.00027, 2020.

[Geiger et al., 2021]
Geiger, A., Lu, H., Icard, T., and Potts, C.
Causal abstractions of neural networks.
Advances in Neural Information Processing Systems,
34:0 9574–9586, 2021.

[Geva et al., 2022]
Geva, M., Caciularu, A., Dar, G., Roit, P., Sadde, S., Shlain, M., Tamir, B.,
and Goldberg, Y.
LM-debugger: An interactive tool for inspection and intervention in
transformer-based language models.
In Che, W. and Shutova, E. (eds.), Proceedings of the 2022
Conference on Empirical Methods in Natural Language Processing: System
Demonstrations, pp.  12–21, Abu Dhabi, UAE, December 2022a.
Association for Computational Linguistics.
URL <https://aclanthology.org/2022.emnlp-demos.2>.

[Geva et al., 2022]
Geva, M., Caciularu, A., Wang, K., and Goldberg, Y.
Transformer feed-forward layers build predictions by promoting
concepts in the vocabulary space.
In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.),
Proceedings of the 2022 Conference on Empirical Methods in Natural
Language Processing, pp.  30–45, Abu Dhabi, United Arab Emirates, December
2022b. Association for Computational Linguistics.
URL <https://aclanthology.org/2022.emnlp-main.3>.

[Geva et al., 2023]
Geva, M., Bastings, J., Filippova, K., and Globerson, A.
Dissecting recall of factual associations in auto-regressive language
In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of
the 2023 Conference on Empirical Methods in Natural Language Processing,
pp.  12216–12235, Singapore, December 2023. Association for Computational
URL <https://aclanthology.org/2023.emnlp-main.751>.

[Ghandeharioun et al., 2021]
Ghandeharioun, A., Kim, B., Li, C.-L., Jou, B., Eoff, B., and Picard, R. W.
Dissect: Disentangled simultaneous explanations via concept
arXiv preprint arXiv:2105.15164, 2021.

[Goldowsky-Dill et al., 2023]
Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A.
Localizing model behavior with path patching.
arXiv preprint arXiv:2304.05969, 2023.

[Gupta et al., 2015]
Gupta, A., Boleda, G., Baroni, M., and Padó, S.
Distributional vectors encode referential attributes.
In Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing, pp.  12–21, 2015.

[Hanna et al., 2023]
Hanna, M., Liu, O., and Variengien, A.
How does GPT-2 compute greater-than?: Interpreting mathematical
abilities in a pre-trained language model.
In Thirty-seventh Conference on Neural Information Processing
Systems, 2023.
URL <https://openreview.net/forum?id=p4PckNQR8k>.

[Hase et al., 2023]
Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A.
Does localization inform editing? surprising differences in
causality-based localization vs. knowledge editing in language models.
arXiv preprint arXiv:2301.04213, 2023.

[Hendel et al., 2023]
Hendel, R., Geva, M., and Globerson, A.
In-context learning creates task vectors.
In The 2023 Conference on Empirical Methods in Natural Language
Processing, 2023.

[Hernandez et al., 2022]
Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and
Andreas, J.
Natural language descriptions of deep features.
In International Conference on Learning Representations, 2022.
URL <https://openreview.net/forum?id=NudBMY-tzDr>.

[Hernandez et al., 2023]
Hernandez, E., Li, B. Z., and Andreas, J.
Measuring and manipulating knowledge representations in language
arXiv preprint arXiv:2304.00740, 2023a.

[Hernandez et al., 2023]
Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas,
J., Belinkov, Y., and Bau, D.
Linearity of relation decoding in transformer language models.
arXiv preprint arXiv:2308.09124, 2023b.

[Kandpal et al., 2023]
Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel, C.
Large language models struggle to learn long-tail knowledge.
In International Conference on Machine Learning, pp.  15696–15707. PMLR, 2023.

[Kim et al., 2018]
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.
Interpretability beyond feature attribution: Quantitative testing
with concept activation vectors (tcav).
In International conference on machine learning, pp.  2668–2677. PMLR, 2018.

[Köhn, 2015]
Köhn, A.
What’s in an embedding? analyzing word embeddings through
multilingual evaluation.
In Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing, pp.  2067–2073, 2015.

[Lieberum et al., 2023]
Lieberum, T., Rahtz, M., Kramár, J., Irving, G., Shah, R., and Mikulik, V.
Does circuit analysis interpretability scale? evidence from multiple
choice capabilities in chinchilla.
arXiv preprint arXiv:2307.09458, 2023.

[Lin, 2004]
Lin, C.-Y.
ROUGE: A package for automatic evaluation of summaries.
In Text Summarization Branches Out, pp.  74–81, Barcelona,
Spain, July 2004. Association for Computational Linguistics.
URL <https://aclanthology.org/W04-1013>.

[Madaan et al., 2023]
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon,
U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann,
K., Welleck, S., Yazdanbakhsh, A., and Clark, P.
Self-refine: Iterative refinement with self-feedback.
In Thirty-seventh Conference on Neural Information Processing
Systems, 2023.
URL <https://openreview.net/forum?id=S37hOerQLB>.

[Madsen et al., 2022]
Madsen, A., Reddy, S., and Chandar, S.
Post-hoc interpretability for neural nlp: A survey.
ACM Computing Surveys, 550 (8):0 1–42, 2022.

[Mallen et al., 2023]
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H.
When not to trust language models: Investigating effectiveness of
parametric and non-parametric memories.
In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.),
Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pp.  9802–9822,
Toronto, Canada, July 2023. Association for Computational Linguistics.
URL <https://aclanthology.org/2023.acl-long.546>.

[Meng et al., 2022]
Meng, K., Bau, D., Andonian, A., and Belinkov, Y.
Locating and editing factual associations in gpt.
Advances in Neural Information Processing Systems,
35:0 17359–17372, 2022a.

[Meng et al., 2022]
Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D.
Mass-editing memory in a transformer.
In The Eleventh International Conference on Learning
Representations, 2022b.

[Merity et al., 2016]
Merity, S., Xiong, C., Bradbury, J., and Socher, R.
Pointer sentinel mixture models.
In International Conference on Learning Representations, 2016.

[Merullo et al., 2022]
Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E.
Linearly mapping from image to text space.
In The Eleventh International Conference on Learning
Representations, 2022.

[Mousi et al., 2023]
Mousi, B., Durrani, N., and Dalvi, F.
Can llms facilitate interpretation of pre-trained language models?
In Proceedings of the 2023 Conference on Empirical Methods in
Natural Language Processing. Association for Computational Linguistics,
December 2023.
URL <https://browse.arxiv.org/pdf/2305.13386.pdf>.

[Nanda et al., 2023]
Nanda, N., Lee, A., and Wattenberg, M.
Emergent linear representations in world models of self-supervised
sequence models.
In Belinkov, Y., Hao, S., Jumelet, J., Kim, N., McCarthy, A., and
Mohebbi, H. (eds.), Proceedings of the 6th BlackboxNLP Workshop:
Analyzing and Interpreting Neural Networks for NLP, pp.  16–30, Singapore,
December 2023. Association for Computational Linguistics.
URL <https://aclanthology.org/2023.blackboxnlp-1.2>.

[nostalgebraist, 2020]
interpreting gpt: the logit lens.
LessWrong, 2020.
URL <https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/

[Pal et al., 2023]
Pal, K., Sun, J., Yuan, A., Wallace, B. C., and Bau, D.
Future lens: Anticipating subsequent tokens from a single hidden
In Proceedings of the 27th Conference on Computational Natural
Language Learning (CoNLL), pp.  548–560, 2023.

[Patel & Pavlick, 2021]
Patel, R. and Pavlick, E.
Mapping language models to grounded conceptual spaces.
In International Conference on Learning Representations, 2021.

[Reimers & Gurevych, 2019]
Reimers, N. and Gurevych, I.
Sentence-bert: Sentence embeddings using siamese bert-networks.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing. Association for Computational Linguistics, 11
URL <https://arxiv.org/abs/1908.10084>.

[Schwettmann et al., 2023]
Schwettmann, S., Chowdhury, N., Klein, S., Bau, D., and Torralba, A.
Multimodal neurons in pretrained text-only transformers.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  2862–2867, 2023.

[Singh et al., 2023]
Singh, C., Hsu, A., Antonello, R., Jain, S., Huth, A., Yu, B., and Gao, J.
Explaining black box text modules in natural language with language
In XAI in Action: Past, Present, and Future Applications,
URL <https://openreview.net/forum?id=3BX9tM03GT>.

[Slobodkin et al., 2023]
Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., and Ravfogel, S.
The curious case of hallucinatory (un)answerability: Finding truths
in the hidden states of over-confident large language models.
In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of
the 2023 Conference on Empirical Methods in Natural Language Processing,
pp.  3607–3625, Singapore, December 2023. Association for Computational
URL <https://aclanthology.org/2023.emnlp-main.220>.

[Stolfo et al., 2023]
Stolfo, A., Belinkov, Y., and Sachan, M.
A mechanistic interpretation of arithmetic reasoning in language
models using causal mediation analysis.
In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of
the 2023 Conference on Empirical Methods in Natural Language Processing,
pp.  7035–7052, Singapore, December 2023. Association for Computational
URL <https://aclanthology.org/2023.emnlp-main.435>.

[Strobelt et al., 2017]
Strobelt, H., Gehrmann, S., Pfister, H., and Rush, A. M.
Lstmvis: A tool for visual analysis of hidden state dynamics in
recurrent neural networks.
IEEE transactions on visualization and computer graphics,
240 (1):0 667–676, 2017.

[Tenney et al., 2019]
Tenney, I., Das, D., and Pavlick, E.
Bert rediscovers the classical nlp pipeline.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pp.  4593–4601, 2019.

[Touvron et al., 2023]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix,
T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023a.

[Touvron et al., 2023]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.,
Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023b.

[Vaswani et al., 2017]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,
Kaiser, Ł., and Polosukhin, I.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Vig et al., 2020]
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and
Shieber, S.
Investigating gender bias in language models using causal mediation
Advances in neural information processing systems,
33:0 12388–12401, 2020.

[Vilas et al., 2023]
Vilas, M. G., Schaumlöffel, T., and Roig, G.
Analyzing vision transformers for image classification in class
embedding space.
arXiv preprint arXiv:2310.18969, 2023.

[Wallat et al., 2020]
Wallat, J., Singh, J., and Anand, A.
BERTnesia: Investigating the capture and forgetting of knowledge in
In Alishahi, A., Belinkov, Y., Chrupała, G., Hupkes, D., Pinter,
Y., and Sajjad, H. (eds.), Proceedings of the Third BlackboxNLP
Workshop on Analyzing and Interpreting Neural Networks for NLP, pp.  174–183, Online, November 2020. Association for Computational Linguistics.
URL <https://aclanthology.org/2020.blackboxnlp-1.17>.

[Wang & Komatsuzaki, 2021]
Wang, B. and Komatsuzaki, A.
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
<https://github.com/kingoflolz/mesh-transformer-jax>, May 2021.

[Wang et al., 2022]
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J.
Interpretability in the wild: a circuit for indirect object
identification in GPT-2 small.
In The Eleventh International Conference on Learning
Representations, 2022.

[Wang et al., 2023]
Wang, Z., Ku, A., Baldridge, J., Griffiths, T. L., and Kim, B.
Gaussian Process Probes (GPP) for uncertainty-aware probing.
arXiv preprint arXiv:2305.18213, 2023.

[Wei et al., 2022]
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V.,
Zhou, D., et al.
Chain-of-thought prompting elicits reasoning in large language
Advances in Neural Information Processing Systems,
35:0 24824–24837, 2022.

[Yao et al., 2023]
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and
Narasimhan, K. R.
Tree of thoughts: Deliberate problem solving with large language
In Thirty-seventh Conference on Neural Information Processing
Systems, 2023.

[Youssef et al., 2023]
Youssef, P., Koraş, O., Li, M., Schlötterer, J., and Seifert, C.
Give me the facts! a survey on factual knowledge probing in
pre-trained language models.
In Findings of the Association for Computational Linguistics:
EMNLP 2023, pp.  15588–15605, 2023.

[Zhang & Nanda, 2023]
Zhang, F. and Nanda, N.
Towards best practices of activation patching in language models:
Metrics and methods.
arXiv preprint arXiv:2309.16042, 2023.

[Zhou et al., 2018]
Zhou, B., Bau, D., Oliva, A., and Torralba, A.
Interpreting deep visual representations via network dissection.
IEEE transactions on pattern analysis and machine
intelligence, 410 (9):0 2131–2145, 2018.

§ NEXT-TOKEN PREDICTION ADDITIONAL DETAILS AND EXPERIMENTAL RESULTS

§.§ Models

We use LLaMA2 (13B) [Touvron et al., 2023], Vicuna (13B) [Chiang et al., 2023], GPT-J (6B) [Wang & Komatsuzaki, 2021], and Pythia (12B) [Biderman et al., 2023]. LLaMA2 was pre-trained on 2T tokens from a mix of publicly available data. Vicuna is a LLaMA1 [Touvron et al., 2023] model that was pre-trained on 1T tokens and fine-tuned on 70K user-shared conversations [collected from <www.sharegpt.com>].
The primary architectural differences between LLaMA2 and Vicuna (LLaMA1) include a different context length and grouped-query attention. Pythia and GPT-J were pre-trained using a deduplicated version of The Pile corpus [Gao et al., 2020], and for about 300B and 402B tokens, respectively.

§.§ Training and Evaluation Data

We use 12,000 random samples from the Pile, partitioned into 10,000 examples for training the affine mappings, and 2,000 examples for evaluation. In our pre-processing strategy, we introduce randomness in the patching positions by trimming the input sequence length of each example.

§.§ Additional Few-Shot Token Identity Prompts

In this section, we provide additional details about the selection of the demonstrations for the token identity baseline, and further evaluate the robustness of LLaMA2 (13B) [Touvron et al., 2023] to various token identity prompts.

Demonstrations Construction For the demonstrations used in this experiment, we sample a random set of $k=3$ tokens for all the models (where $k$ was also randomly sampled from the interval $[1,\ldots,10]$).

Robustness to Additional token IDs' Demonstrations We randomly generate five realizations of token IDs series of varying lengths, formatted as “”, similarly to the procedure from <ref>. The results are illustrated in Fig. <ref>, where a comprehensive overview of the evaluation metrics can be found in <ref>. The results indicate the stability of the token identity baseline across a range of token identity demonstrations, particularly notable in the upper layers of the model.

[Precision@1]  [Surprisal]
Next-token prediction results for LLaMA2, using various token identity demonstrations (the token IDs appear in the legend). We report precision@1 ($\uparrow$ is better), and surprisal ($\downarrow$ is better).

§ MORE DETAILS ON ATTRIBUTE EXTRACTION EXPERIMENTS

Dataset Details We start from factual and commonsense reasoning subsets introduced by Hernandez et al., 2023. This dataset includes 8 commonsense and 25 factual relations. For each data point, we sample 5 utterances from  dataset [Merity et al., 2016] including $s$. We then truncate sampled text to a window of random length up to 20 tokens that contains $s$. This constitutes our source prompt, $\sourcePrompt$. Note that for each model, we filter the data to samples for which the underlying model correctly encodes the tuple. For experiments with zero-shot target prompt $\targetPrompt$ that includes $r$ followed by $s$, we autoregressively generate the next 20 tokens, and only keep examples where $o$ appears in the generation. For experiments with few-shot demonstrations in $\targetPrompt$, after generating the next 20 tokens, we do an additional post-processing step. If the demonstration template of an example is identified in the generation, all the following tokens would be dropped. The example is used for evaluation only if $o$ appears in the post-processed truncated generation. To have a reasonable amount of data for training the classification probe baseline, tasks with fewer than 15 datapoints are dropped from the analysis. For GPT-J, 5 commonsense and 7 factual reasoning tasks remain after applying the above postprocessing steps.

Results on Additional Tasks In this section, we provide additional results on various factual and commonsense reasoning tasks. For a comparison of Zero-Shot Feature Extraction  and Logistic Regression Probe across layers, see Fig. <ref>. Zero-Shot Feature Extraction  outperforms baseline consistently in early $\sourceLayer$ and in the majority of mid $\sourceLayer$. As source representation shifts toward next-token prediction in later $\sourceLayer$, Zero-Shot Feature Extraction  accuracy declines gradually. Late $\targetLayer$ is less successful, perhaps due to the prolonged influence of the placeholder  representation in the computation. In addition, we provide t-statistic details in Tab. <ref>.

Source-Target Layer Interplay
Fig. <ref> visualizes the interaction between $\sourceLayer$ and $\targetLayer$. These heatmaps show attribute extraction success rate for Zero-Shot Feature Extraction  for a fixed $(\sourceLayer, \targetLayer)$ combination. The lower left quadrants show setups where both  $\sourceLayer$ and $\targetLayer$ represent early to middle layers, and the success rate is maximal. The right half of the heatmaps represent late $\sourceLayer$, which achieve lower success rate due to token representations shifting toward next-token prediction as discussed earlier. In addition, we notice lower success rate in the top half of the heatmaps which represent late $\targetLayer$.
It is worth noting that in this task, the accuracy is not only based on the immediate next-token prediction, but rather whether $\object$ appears in the next 20 autoregressively generated tokens. The placeholder token  does still remain in the input, and its representation persists in the early layers in the target computation. This explains why lower attribute extraction rate is observed in later $\targetLayer$ values. We leave it to future work to investigate adaptations to  to control contamination from the placeholder tokens and make them more amenable to late $\targetLayer$ choices.

[Substance Phase]
[Task Done By Tool]
[Country Currency]
[Food From Country]

[Company CEO]
[Fruit Outside Color]
[Product By Company]
[Person Plays Pro Sport]

[Person Plays Position in Sport]
[Object Superclass]
[Fruit Inside Color]
[Star Constellation]

[Person Father]
[Superhero Archnemesis]
[Task Done By Person]
[Work Location]

[Superhero Person]
[Country Capital City]
[Country Largest City]

Feature extraction accuracy with respect to source layer ($\sourceLayer$) across various factual and commonsense reasoning tasks. Zero-Shot Feature Extraction  works consistently better than Logistic Regression Probe in early layers, and mostly in mid layers. There is a decline in  accuracy in later $\sourceLayer$ as the source representations shift toward next-token prediction.

[Substance Phase]
[Task Done By Tool]
[Country Currency]
[Food From Country]

[Company CEO]
[Fruit Outside Color]
[Product By Company]
[Person Plays Pro Sport]

[Person Plays Position in Sport]
[Object Superclass]
[Fruit Inside Color]
[Star Constellation]

[Person Father]
[Superhero Archnemesis]
[Task Done By Person]
[Work Location]

[Superhero Person]
[Country Capital City]
[Country Largest City]

The interaction between source and target layers in Zero-Shot Feature Extraction across various factual and commonsense reasoning tasks. Each cell $(\sourceLayer, \targetLayer)$ in the heatmap shows the attribute extraction success rate where source and target layers are fixed to $\sourceLayer$ and $\targetLayer$, respectively. Particularly, there is a higher success rate in the lower left quadrants, representing early to mid source and target layer combinations. The right half of the heatmaps shows late source layers, where the source representation has shifted toward next-token prediction, leading to lower success rate in attribute extraction. The top half of the heatmaps shows late target layers. When the accuracy is a function of more than a single next-token, the placeholder token representation still remains in the early layers, leading to lower attribute extraction rate.

Comparison between Zero-Shot Feature Extraction  and a Logistic Regression Probe shows that despite using no training data, it has a significantly higher accuracy than baseline in most tasks $(p < 1e-5)$. Pairwise t-statistics and corresponding p-values are included in the table.

2cAccuracy (mean$\pm$std)

Task     Logistic Regression Probe     Zero-shot Feature Extraction      t-statistic     p-value

5*Commonsense     Fruit inside color     $37.41 \pm 6.58$     $37.99 \pm 18.67$     $0.126$     $0.901$

Fruit outside color      $35.50 \pm 3.09$     $\mathbf{71.00 \pm 13.26^{**}}$     $12.426$     $<1e-5$

Object superclass     $\mathbf{68.92 \pm 10.69}^{**}$     $55.71 \pm 10.81$     $-5.25$     $<1e-4$

Substance phase     $73.77 \pm 3.74$     $\mathbf{91.92 \pm 1.73^{**}}$     $25.647$     $<1e-5$

Task done by person     $0 \pm 0$     $\mathbf{62.96 \pm 16.513^{**}}$      $19.632$     $<1e-5$

Task done by tool     $10.14 \pm 3.23$     $\mathbf{48.12 \pm 13.23^{**}}$     $18.231$     $<1e-5$

Work location      $0 \pm 0$     $\mathbf{13.58 \pm 9.37^{**}}$     $7.45990$     $<1e-5$

Company CEO      $4.99 \pm 2.56$     $\mathbf{47.82 \pm 13.89^{**}}$     $16.700$     $<1e-5$

Country capital city     0 $\pm$ 0     $\mathbf{61.61 \pm 14.14^{**}}$     $22.426$     $<1e-5$

7*Factual     Country currency     $17.70 \pm 2.20$     $\mathbf{50.95 \pm 8.85^{**}}$     $20.293$     $<1e-5$

Country largest city     $0 \pm 0$     $\mathbf{67.78 \pm 11.47^{**}}$     $30.427$     $<1e-5$

Food from country     $5.13 \pm 3.66$     $\mathbf{63.80 \pm 11.34^{**}}$     $26.710$     $<1e-5$

Person father     $0 \pm 0$     $\mathbf{25.34 \pm 8.42^{**}}$     $15.482$     $<1e-5$

Person plays position in sport     $75.89 \pm 9.14$     $72.20 \pm 7.21$     $-2.066$     $0.049$

Person plays pro sport     $53.87 \pm 10.28$     $46.28 \pm 14.19$     $-2.020$     $0.054$

Product by company     $58.91 \pm 7.15$      $63.24 \pm 10.74$     $1.757$     $0.091$

Star constellation     $17.54 \pm 5.30$     $18.35 \pm 5.06$     $-2.98$     $0.006$

Superhero archnemesis     $0 \pm 0$     $\mathbf{41.73 \pm 18.72^{**}}$     $11.47044$     $<1e-5$

Superhero person     $0 \pm 0$     $\mathbf{28.32 \pm 14.05^{**}}$     $10.37461$     $<1e-5$

§ ADDITIONAL INFORMATION AND RESULTS ON THE ENTITY RESOLUTION EXPERIMENT

§.§ Experimental Setup

Recall () that we use a few-shot target prompt template for decoding an entity description: “”, while patching the last position which corresponds to $x$.  Specifically, we use the following target prompt, obtained randomly: “” and task the model to generate the completion after the patched representation in $x$. For the subject description, which is composed of $k=3$ random subject entities, we used the wptools[<https://github.com/siznax/wptools/>] python package for obtaining a  description of every subject entity from Wikipedia.

§.§ Additional Quantitative Results

In this section, we present the Rouge1 [Lin, 2004] and SBERT score [Reimers & Gurevych, 2019] results, as well as the results for the Pythia models [Biderman et al., 2023].[We used the package from <https://www.sbert.net/>, with the sentence-transformers/all-MiniLM-L6-v2 model.] In Fig. <ref> and Fig. <ref>, we present the Rouge1 and the SBERT results, respectively, complementary to the RougeL results in Fig. <ref> from <ref>.  Note that for Pythia, the smaller model (6.9B) outperforms the larger one (12B), and hence, our cross-model patching method does not have the potential to improve the inspection of the smaller model, unlike the trends for Vicuna. The other

Rouge1 scores of the generated descriptions against descriptions from Wikipedia.

SBERT scores of the generated descriptions against descriptions from Wikipedia.

§.§ Additional Qualitative Results

In this section, we provide more examples and discuss our observations about the gradual process of entity resolution. As shown in Tables <ref> and  <ref>, it is interesting to observe that the resolution process for the same input can look different across models, suggesting they assign different likelihoods to different entities, and weigh context differently. For example, as Vicuna 13B processes “”, it goes from “Smithsonian Museum” to the “Smith rock band” to the actor and rapper, “Will Smith”. However, Pythia 12B starts with “Smith & Wesson weapon manufacturing company” before it resolves the entity as the American actor, “Will Smith”.

We also observe another phenomenon which we refer as placeholder contamination. This is the case where the remaining representation of the  placeholder entity “” in the early layers interferes with the model's capability in generating descriptions for the patched token. For example, see Vicuna 13B response to “” entity in Tab. <ref>. First, we see the gradual process of going from “Rigatoni” pasta, to “Hilton Hotels”, to the socialite “Paris Hilton” in layers 1-6. But in layers 7 and onward, the generation seems to describe the placeholder token “”rather than the “” entity: “Placeholder for a variable or concept”, “Variable representing any number of things or concepts” or “x is a placeholder”. For future, we would like to quantify these qualitative observations, study to what extend this contamination can be mitigated with a different placeholder choice, and why some models might be more susceptible to this contamination than others.

Additional qualitative examples illustrating entity resolution via qualitative examples for Vicuna 13B.The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representation, referred to as “Tokens Covered". Here, $\sourceModel \is \text{Vicuna 13B}, \targetModel \is \sourceModel, \targetLayer \is \sourceLayer$.

$\sourcePrompt$     Tokens Covered     $\sourceLayer$      Generation        Notes/Entity Described by the Generation

sonian: National museum and research complex in Washington, D.C.
Smithsonian Museum

sonas: American rock band
An amalgam of Smith band and SONA band

: American actor and rapper
Will Smith

s Secret: American multinational women's lingerie, beauty and body-care products company
Victoria's Secret

s Secret: American lingerie and beauty products retailer
Victoria's Secret

: Queen of the United Kingdom of Great Britain and Ireland from 1837 to 1901
Queen Victoria

: Queen of the United Kingdom of Great Britain and Ireland from 1837 until her death in 1901
Queen Victoria

: Queen of England (reigned from 1837-1901)
Queen Victoria

: 19th century British queen who ruled from 1837-1901
Queen Victoria

y: a type of pasta

Hotels: American multinational hospitality company
Hilton Hotel

: Hotel chain founded by Conrad Hilton
Hilton Hotel

: Hotel and resorts company
Hilton Hotel

: Socialite and television personality
Paris Hilton

: Placeholder for a variable or concept
Placeholder contamination

: Variable representing any number of things or concepts
Placeholder contamination

: x is a placeholder
Placeholder contamination

: Unit of measurement for the strength of a liquid
Bond Unit

: James Bond, the fictional British secret agent created by novelist Ian Fleming and portrayed by actor Daniel Craig in the 2006 film "Casino Royale,"
James Bond

: Fictional British secret agent
James Bond

: Code name for fictional British secret agent James Bond
James Bond

: Code name for a fictional British secret agent created by novelist Ian Fleming
James Bond

: Subject of a fictional British secret agent, created by novelist Ian Fleming and portrayed in a series of films.
James Bond

: Subject of the 007 novels and films
James Bond

: Variable representing the number of the film in the James Bond series
Placeholder contamination

& Johnson: American multinational corporation in the fields of pharmaceuticals, medical devices, and consumer packaged goods
Johnson & Johnson

: American actor and professional wrestler, also known as "The Rock"
Dwayne Johnson

: Also known as "The Rock," American actor and professional wrestler
Dwayne Johnson

: Former professional wrestler and actor
Dwayne Johnson

: Abbreviation for the United States
Placeholder contamination

: Variable representing any number of things or concepts
Placeholder contamination

: Variable representing any number of other options.
Placeholder contamination

: Variable representing any number of other options or possibilities.

Placeholder contamination

Additional qualitative examples illustrating entity resolution via qualitative examples for Pythia 12B.The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representation, referred to as “Tokens Covered". Here, $\sourceModel \is \text{Pythia 12B}, \targetModel \is \sourceModel, \targetLayer \is \sourceLayer$.

$\sourcePrompt$     Tokens Covered     $\sourceLayer$      Generation        Notes/Entity Described by the Generation

& Wesson: American firearms manufacturer
Smith & Wesson

: American actor
Will Smith

: Capital of the United States
Washington D.C.

: American capital city
Washington D.C.

: American president
George Washington

: American revolutionary
George Washington

: American president
George Washington

: George Washington
George Washington

: George Washington, the first President of the United States
George Washington

: George Washington, Thomas Jefferson, Abraham Lincoln, and Theodore Roosevelt: American presidents
George Washington

loud: Apple's cloud-based storage service

: Sinking ocean liner

: The ship that sank in the Atlantic Ocean in 1912

: British passenger ship

: The ship that sank

: Titanic, Titanic: British luxury passenger liner

Nation: American concert promoter
Live Nation

Nation: American television network
Live Nation

: American television program
Saturday Night Live

: American television comedy show
Saturday Night Live

Seasons: American hotel chain
Four Seasons Hotel

Seasons: The four seasons of the year
Four Seasons (of the year)

Lions: Chinese professional football club
Cangzhou Mighty Lions F.C.

: Number of the novel 1984 by George Orwell
Nineteen Eighty-Four (Novel)

: George Orwell's novel, and the 1984 film.
Nineteen Eighty-Four (Novel & Film)

: George Orwell’s novel
Nineteen Eighty-Four (Novel)

: George Orwell: British novelist
Nineteen Eighty-Four (Novel)

: 1984 novel by George Orwell
Nineteen Eighty-Four (Novel)

: 1984 novel by George Orwell, and the 1984 film adaptation directed by Michael Radford.
Nineteen Eighty-Four (Novel & Film)

§ ADDITIONAL RESULTS ON THE CROSS-MODEL PATCHING EXPERIMENT

In this section, we present the Surprisal metric results of cross-model patching representations from Vicuna (7B) $\rightarrow$ Vicuna (13B) [Chiang et al., 2023], and Pythia (6.9B) $\rightarrow$ Pythia (12B) [Biderman et al., 2023]. We employ an affine mapping, as detailed in <ref> and in <ref>, as a preliminary step before the patching representations from one model to another. In Fig. <ref>, we present the surprisal results, complementary to the precision@1 results in Fig. <ref> from <ref>.

[Vicuna: $\sourceModel\!\is\!\text{7B}, \targetModel\!\is\!\text{13B}$]
[Pythia: $\sourceModel\!\is\!\text{6.9B}, \targetModel\!\is\!\text{12B}$]
Next-token prediction estimation performance in Vicuna and Pythia with cross-model ,  measured by Surprisal ($\uparrow$ is better).

§ MORE DETAILS ON MULTI-HOP REASONING EXPERIMENT

Data Tab. <ref> summarizes information about the samples where the Vicuna (13B) correctly represents each reasoning step. Out of 1,104 samples that require two steps of commonsense or factual reasoning, 46 satisfy the above criteria, with 8 unique relation combinations.

Method See an illustration of CoT  in <ref>. In practice, we use the following configuration for CoT : $\sourcePrompt \is \pi_1, \targetPrompt \is \pi_2, \sourcePosition \is \sourceSeqLen, \targetPosition \is \targetSeqLen$. This is equivalent to $\sourcePrompt = \targetPrompt \is [\pi_2][\pi_1]$ and adjusting the attention mask such that no token in $\sourcePrompt$ has visibility to $\pi_2$ and no token in $\targetPrompt$ has visibility to $\pi_1$.

An illustration of CoT  on a single example. In this example, $\pi_1 \is ``\texttt{the company that created Visual Basic Script}"$, $\pi_2 \is ``\texttt{The current CEO of}"$, $\sourcePrompt = \targetPrompt \is [\pi_2][\pi_1]=``\texttt{The current CEO of the company that created Visual Basic Script}"$. Note that $\sourceModel=\targetModel$ and $\transformation \is \identity$. For more details about attention mask adjustments that are not visible on the plot, see <ref>.

Sample statistics for the multi-hop reasoning experiment where $\sourceModel$ correctly represents both $\tau_1$ and $\tau_2$.

$\relation_1$          $\relation_2$            # Samples
Company CEO            Person Father            4

Food from Country      Country Capital City     10

Food from Country      Country Currency         3

Food from Country      Country Language         9

Food from Country      Country Largest City     11

Person Father          Person Father            1

Person Father          Person Mother            1

Product by Company     Company CEO              7
Total                                           46