# Ontology-Aware Token Embeddings for Prepositional Phrase Attachment

Pradeep Dasigi1 Waleed Ammar2 Chris Dyer1,3 Eduard Hovy1
1Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA,
USA
2Allen Institute for Artificial Intelligence, Seattle WA, USA
3DeepMind, London, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Type-level word embeddings use the same set of parameters to represent all
instances of a word regardless of its context, ignoring the inherent lexical
ambiguity in language. Instead, we embed semantic concepts (or synsets) as
defined in WordNet and represent a word token in a particular context by
estimating a distribution over relevant semantic concepts. We use the new,
context-sensitive embeddings in a model for predicting prepositional phrase
(PP) attachments and jointly learn the concept embeddings and model
parameters. We show that using context-sensitive embeddings improves the
accuracy of the PP attachment model by 5.4% absolute points, which amounts to
a 34.4% relative reduction in errors.

##  1 Introduction

Type-level word embeddings map a word type (i.e., a surface form) to a dense
vector of real numbers such that similar word types have similar embeddings.
When pre-trained on a large corpus of unlabeled text, they provide an
effective mechanism for generalizing statistical models to words which do not
appear in the labeled training data for a downstream task.

In accordance with standard terminology, we make the following distinction
between types and tokens in this paper: By word types, we mean the surface
form of the word, whereas by tokens we mean the instantiation of the surface
form in a context. For example, the same word type ‘pool’ occurs as two
different tokens in the sentences “He sat by the pool,” and “He played a game
of pool.”

Most word embedding models define a single vector for each word type. However,
a fundamental flaw in this design is their inability to distinguish between
different meanings and abstractions of the same word. In the two sentences
shown above, the word ‘pool’ has different meanings, but the same
representation is typically used for both of them. Similarly, the fact that
‘pool’ and ‘lake’ are both kinds of water bodies is not explicitly
incorporated in most type-level embeddings. Furthermore, it has become a
standard practice to tune pre-trained word embeddings as model parameters
during training for an NLP task (e.g., Chen and Manning, 2014; Lample et al.,
2016), potentially allowing the parameters of a frequent word in the labeled
training data to drift away from related but rare words in the embedding
space.

Previous work partially addresses these problems by estimating concept
embeddings in WordNet (e.g., Rothe and Schütze, 2015), or improving word
representations using information from knowledge graphs (e.g., Faruqui et al.,
2015). However, it is still not clear how to use a lexical ontology to derive
context-sensitive token embeddings.

Figure 1: An example grounding for the word ‘pool’. Solid arrows represent
possible senses and dashed arrows represent hypernym relations. Note that the
same set of concepts are used to ground the word ‘pool’ regardless of its
context. Other WordNet senses for ‘pool’ were removed from the figure for
simplicity.

In this work, we represent a word token in a given context by estimating a
context-sensitive probability distribution over relevant concepts in WordNet
Miller (1995) and use the expected value (i.e., weighted sum) of the concept
embeddings as the token representation (see §2). We take a task-centric
approach towards doing this, and learn the token representations jointly with
the task-specific parameters. In addition to providing context-sensitive token
embeddings, the proposed method implicitly regularizes the embeddings of
related words by forcing related words to share similar concept embeddings. As
a result, the representation of a rare word which does not appear in the
training data for a downstream task benefits from all the updates to related
words which share one or more concept embeddings.

Our approach to context-sensitive embeddings assumes the availability of a
lexical ontology. While this work relies on WordNet, and we exploit the order
of senses given by WordNet, our model is, in principle applicable to any
ontology, with appropriate modifications. In this work, we do not assume the
inputs are sense tagged. We use the proposed embeddings to predict
prepositional phrase (PP) attachments (see §3), a challenging problem which
emphasizes the selectional preferences between words in the PP and each of the
candidate head words. Our empirical results and detailed analysis (see §4)
show that the proposed embeddings effectively use WordNet to improve the
accuracy of PP attachment predictions.

##  2 WordNet-Grounded Context-Sensitive Token Embeddings

In this section, we focus on defining our context-sensitive token embeddings.
We first describe our grounding of word types using WordNet concepts. Then, we
describe our model of context-sensitive token-level embeddings as a weighted
sum of WordNet concept embeddings.

###  2.1 WordNet Grounding

We use WordNet to map each word type to a set of synsets, including possible
generalizations or abstractions. Among the labeled relations defined in
WordNet between different synsets, we focus on the hypernymy relation to help
model generalization and selectional preferences between words, which is
especially important for predicting PP attachments Resnik (1993). To ground a
word type, we identify the set of (direct and indirect) hypernyms of the
WordNet senses of that word. A simplified grounding of the word ‘pool’ is
illustrated in Figure 1. This grounding is key to our model of token
embeddings, to be described in the following subsections.

###  2.2 Context-Sensitive Token Embeddings

Our goal is to define a context-sensitive model of token embeddings which can
be used as a drop-in replacement for traditional type-level word embeddings.

#### Notation.

Let $\textit{Senses}(w)$ be the list of synsets defined as possible word
senses of a given word type $w$ in WordNet, and $\textit{Hypernyms}(s)$ be the
list of hypernyms for a synset $s$.111For notational convenience, we assume
that $s\in\textit{Hypernyms}(s)$. For example, according to Figure 1:

Senses $\displaystyle(\text{pool})=[\text{pond.n.01, pool.n.09}],\text{and}$
Hypernyms $\displaystyle(\text{pond.n.01})=[\text{pond.n.01, lake.n.01},$
$\displaystyle\text{body\\_of\\_water.n.01, entity.n.01}]$

Each WordNet synset $s$ is associated with a set of parameters
$\mathbf{v}_{s}\in\mathbb{R}^{n}$ which represent its embedding. This
parameterization is similar to that of Rothe and Schütze (2015).

#### Embedding model.

Figure 2: Steps for computing the context-sensitive token embedding for the
word ‘pool’, as described in §2.2.

Given a sequence of tokens $\boldsymbol{t}$ and their corresponding word types
$\boldsymbol{w}$, let $\mathbf{u}_{i}\in\mathbb{R}^{n}$ be the embedding of
the word token $t_{i}$ at position $i$. Unlike most embedding models, the
token embeddings $\mathbf{u}_{i}$ are not parameters. Rather, $\mathbf{u}_{i}$
is computed as the expected value of concept embeddings used to ground the
word type $w_{i}$ corresponding to the token $t_{i}$:

$\displaystyle\mathbf{u}_{i}=\sum_{s\in\textit{Senses}(w_{i})}\sum_{s^{\prime}\in\textit{Hypernyms}(s)}p(s,s^{\prime}\mid\boldsymbol{t},\boldsymbol{w},i)\
$ $\displaystyle\mathbf{v_{s^{\prime}}}$ (1)

such that

$\sum_{s\in\textit{Senses}(w_{i})}\sum_{s^{\prime}\in\textit{Hypernyms}(s)}p(s,s^{\prime}\mid\boldsymbol{t},\boldsymbol{w},i)=1$

The distribution which governs the expectation over synset embeddings
factorizes into two components:

$\displaystyle p(s,s^{\prime}\mid\boldsymbol{t},\boldsymbol{w},i)\propto$
$\displaystyle\lambda_{w_{i}}\exp^{-\lambda_{w_{i}}\
\textit{rank}(s,w_{i})}\times$
$\displaystyle\textit{MLP}([\mathbf{v}_{s^{\prime}};\textit{context}(i,\boldsymbol{t})])$
(2)

The first component, $\lambda_{w_{i}}\exp^{-\lambda_{w_{i}}\
\textit{rank}(s,w_{i})}$, is a sense prior which reflects the prominence of
each word sense for a given word type. Here, we exploit222Note that for
ontologies where such information is not available, our method is still
applicable but without this component. We show the effect of using a uniform
sense prior in §4.2. the fact that WordNet senses are ordered in descending
order of their frequencies, obtained from sense tagged corpora, and
parameterize the sense prior like an exponential distribution. $rank(s,w_{i})$
denotes the rank of sense $s$ for the word type $w_{i}$, thus
$rank(s,w_{i})=0$ corresponds to $s$ being the first sense of $w_{i}$. The
scalar parameter ($\lambda_{w_{i}}$) controls the decay of the probability
mass, which is learned along with the other parameters in the model. Note that
sense priors are defined for each word type ($w_{i}$), and are shared across
all tokens which have the same word type.

$\textit{MLP}([\mathbf{v}_{s^{\prime}};\textit{context}(i,\boldsymbol{t})])$,
the second component, is what makes the token representations context-
sensitive. It scores each concept in the WordNet grounding of $w_{i}$ by
feeding the concatenation of the concept embedding and a dense vector that
summarizes the textual context into a multilayer perceptron (MLP) with two
$tanh$ layers followed by a $softmax$ layer. This component is inspired by the
soft attention often used in neural machine translation Bahdanau et al.
(2014).333Although soft attention mechanism is typically used to explicitly
represent the importance of each item in a sequence, it can also be applied to
non-sequential items. The definition of the context function is dependent on
the encoder used to encode the context. We describe a specific instantiation
of this function in §3.

To summarize, Figure 2 illustrates how to compute the embedding of a word
token $t_{i}=\textit{`pool'}$ in a given context:

  1. 1.

compute a summary of the context $\textit{context}(i,\boldsymbol{t})$,

  2. 2.

enumerate related concepts for $t_{i}$,

  3. 3.

compute $p(s,s^{\prime}\mid\boldsymbol{t},\boldsymbol{w},i)$ for each pair
$(s,s^{\prime})$, and

  4. 4.

compute $\mathbf{u}_{i}=\mathbb{E}[\mathbf{v}_{s^{\prime}}]$.

In the following section, we describe our model for predicting PP attachments,
including our definition for context.

##  3 PP Attachment

Disambiguating PP attachments is an important and challenging NLP problem.
Since modeling hypernymy and selectional preferences is critical for
successful prediction of PP attachments Resnik (1993), it is a good fit for
evaluating our WordNet-grounded context-sensitive embeddings.

Figure 3:  Two sentences illustrating the importance of lexicalization in PP
attachment decisions. In the top sentence, the PP ‘with butter’ attaches to
the noun ‘spaghetti’. In the bottom sentence, the PP ‘with chopsticks’
attaches to the verb ‘ate’. Note: This figure and caption have been reproduced
from Belinkov et al. (2014).

Figure 3, reproduced from Belinkov et al. (2014), illustrates an example of
the PP attachment prediction problem. The accuracy of a competitive English
dependency parser at predicting the head word of an ambiguous prepositional
phrase is 88.5%, significantly lower than the overall unlabeled attachment
accuracy of the same parser (94.2%).444See Table 2 in §4 for detailed results.

This section formally defines the problem of PP attachment disambiguation,
describes our baseline model, then shows how to integrate the token-level
embeddings in the model.

###  3.1 Problem Definition

We follow Belinkov et al. (2014)’s definition of the PP attachment problem.
Given a preposition $p$ and its direct dependent $d$ in the prepositional
phrase (PP), our goal is to predict the correct head word for the PP among an
ordered list of candidate head words $\boldsymbol{h}$. Each example in the
train, validation, and test sets consists of an input tuple
$\langle\boldsymbol{h},p,d\rangle$ and an output index $k$ to identify the
correct head among the candidates in $\boldsymbol{h}$. Note that the order of
words that form each $\langle\boldsymbol{h},p,d\rangle$ is the same as that in
the corresponding original sentence.

###  3.2 Model Definition

Both our proposed and baseline models for PP attachment use bidirectional RNN
with LSTM cells (bi-LSTM) to encode the sequence $\boldsymbol{t}=\langle
h_{1},\ldots,h_{K},p,d\rangle$.

We score each candidate head by feeding the concatenation of the output bi-
LSTM vectors for the head $h_{k}$, the preposition $p$ and the direct
dependent $d$ through an MLP, with a fully connected tanh layer to obtain a
non-linear projection of the concatenation, followed by a fully-connected
softmax layer:

$\displaystyle p(h_{k}\text{is head})=\textit{MLP}_{\textit{attach}}([$
$\displaystyle\textit{lstm\\_out}(h_{k});$
$\displaystyle\textit{lstm\\_out}(p);$ $\displaystyle\textit{lstm\\_out}(d)])$
(3)

To train the model, we use cross-entropy loss at the output layer for each
candidate head in the training set. At test time, we predict the candidate
head with the highest probability according to the model in Eq. 3, i.e.,

$\displaystyle\hat{k}=\arg\max_{k}p(h_{k}\text{is head}=1).$ (4)

This model is inspired by the Head-Prep-Child-Ternary model of Belinkov et al.
(2014). The main difference is that we replace the input features for each
token with the output bi-RNN vectors.

We now describe the difference between the proposed and the baseline models.
Generally, let $\textit{lstm\\_in}(t_{i})$ and $\textit{lstm\\_out}(t_{i})$
represent the input and output vectors of the bi-LSTM for each token
$t_{i}\in\boldsymbol{t}$ in the sequence. The outputs at each timestep are
obtained by concatenating those of the forward and backward LSTMs.

#### Baseline model.

In the baseline model, we use type-level word embeddings to represent the
input vector $\textit{lstm\\_in}(t_{i})$ for a token $t_{i}$ in the sequence.
The word embedding parameters are initialized with pre-trained vectors, then
tuned along with the parameters of the bi-LSTM and
$\textit{MLP}_{\textit{attach}}$. We call this model LSTM-PP.

#### Proposed model.

In the proposed model, we use token level word embedding as described in §2 as
the input to the bi-LSTM, i.e., $\textit{lstm\\_in}(t_{i})=\mathbf{u}_{i}$.
The context used for the attention component is simply the hidden state from
the previous timestep. However, since we use a bi-LSTM, the model essentially
has two RNNs, and accordingly we have two context vectors, and associated
attentions. That is, $\textit{context}_{f}\textit{(i,
{t})}=\textit{lstm\\_in}(t_{i-1})$ for the forward RNN and
$\textit{context}_{b}\textit{(i, {t})}=\textit{lstm\\_in}(t_{i+1})$ for the
backward RNN. Consequently, each token gets two representations, one from each
RNN. The synset embedding parameters are initialized with pre-trained vectors
and tuned along with the sense decay ($\lambda_{w}$) and MLP parameters from
Eq. 2, the parameters of the bi-LSTM and those of
$\textit{MLP}_{\textit{attach}}$. We call this model OntoLSTM-PP.

##  4 Experiments

#### Dataset and evaluation.

We used the English PP attachment dataset created and made available by
Belinkov et al. (2014). The training and test splits contain 33,359 and 1951
labeled examples respectively. As explained in §3.1, the input for each
example is 1) an ordered list of candidate head words, 2) the preposition, and
3) the direct dependent of the preposition. The head words are either nouns or
verbs and the dependent is always a noun. All examples in this dataset have at
least two candidate head words. As discussed in Belinkov et al. (2014), this
dataset is a more realistic PP attachment task than the RRR dataset
Ratnaparkhi et al. (1994). The RRR dataset is a binary classification task
with exactly two head word candidates in all examples. The context for each
example in the RRR dataset is also limited which defeats the purpose of our
context-sensitive embeddings.

#### Model specifications and hyperparameters.

For efficient implementation, we use mini-batch updates with the same number
of senses and hypernyms for all examples, padding zeros and truncating senses
and hypernyms as needed. For each word type, we use a maximum of $S$ senses
and $H$ indirect hypernyms from WordNet. In our initial experiments on a held-
out development set (10% of the training data), we found that values greater
than $S=3$ and $H=5$ did not improve performance. We also used the development
set to tune the number of layers in $\textit{MLP}_{\textit{attach}}$
separately for the OntoLSTM-PP and LSTM-PP, and the number of layers in the
attention MLP in OntoLSTM-PP. When a synset has multiple hypernym paths, we
use the shortest one. Finally, words types which do not appear in WordNet are
assumed to have one unique sense per word type with no hypernyms. Since the
POS tag for each word is included in the dataset, we exclude WordNet synsets
which are incompatible with the POS tag. The synset embedding parameters are
initialized using the synset vectors obtained by running AutoExtend Rothe and
Schütze (2015) on 100-dimensional GloVe Pennington et al. (2014) vectors for
WordNet 3.1. We refer to this embedding as GloVe-extended. Representation for
the OOV word types in LSTM-PP and OOV synset types in OntoLSTM-PP were
randomly drawn from a uniform 100-d distribution. Initial sense prior
parameters ($\lambda_{w}$) were also drawn from a uniform 1-d distribution.

#### Baselines.

In our experiments, we compare our proposed model, OntoLSTM-PP with three
baselines – LSTM-PP initialized with GloVe embedding, LSTM-PP initialized with
GloVe vectors retrofitted to WordNet using the approach of Faruqui et al.
(2015) (henceforth referred to as GloVe-retro), and finally the best
performing standalone PP attachment system from Belinkov et al. (2014),
referred to as HPCD (full) in the paper. HPCD (full) is a neural network model
that learns to compose the vector representations of each of the candidate
heads with those of the preposition and the dependent, and predict
attachments. The input representations are enriched using syntactic context
information, POS, WordNet and VerbNet Kipper et al. (2008) information and the
distance of the head word from the PP is explicitly encoded in composition
architecture. In contrast, we do not use syntactic context, VerbNet and
distance information, and do not explicitly encode POS information.

###  4.1 PP Attachment Results

System | Initialization | Embedding | Resources | Test Acc.
---|---|---|---|---
HPCD (full) | Syntactic-SG | Type | WordNet, VerbNet | 88.7
LSTM-PP | GloVe | Type | - | 84.3
LSTM-PP | GloVe-retro | Type | WordNet | 84.8
OntoLSTM-PP | GloVe-extended | Token | WordNet | 89.7
Table 1: Results on Belinkov et al. (2014)’s PPA test set. HPCD (full) is from
the original paper, and it uses syntactic SkipGram. GloVe-retro is GloVe
vectors retrofitted Faruqui et al. (2015) to WordNet 3.1, and GloVe-extended
refers to the synset embeddings obtained by running AutoExtend Rothe and
Schütze (2015) on GloVe.

Table 1 shows that our proposed token level embedding scheme OntoLSTM-PP
outperforms the better variant of our baseline LSTM-PP (with GloVe-retro
intialization) by an absolute accuracy difference of 4.9%, or a relative error
reduction of 32%. OntoLSTM-PP also outperforms HPCD (full), the previous best
result on this dataset.

Initializing the word embeddings with GloVe-retro (which uses WordNet as
described in Faruqui et al. (2015)) instead of GloVe amounts to a small
improvement, compared to the improvements obtained using OntoLSTM-PP. This
result illustrates that our approach of dynamically choosing a context
sensitive distribution over synsets is a more effective way of making use of
WordNet.

#### Effect on dependency parsing.

Following Belinkov et al. (2014), we used RBG parser Lei et al. (2014), and
modified it by adding a binary feature indicating the PP attachment
predictions from our model.

We compare four ways to compute the additional binary features: 1) the
predictions of the best standalone system HPCD (full) in Belinkov et al.
(2014), 2) the predictions of our baseline model LSTM-PP, 3) the predictions
of our improved model OntoLSTM-PP, and 4) the gold labels Oracle PP.

Table 2 shows the effect of using the PP attachment predictions as features
within a dependency parser. We note there is a relatively small difference in
unlabeled attachment accuracy for all dependencies (not only PP attachments),
even when gold PP attachments are used as additional features to the parser.
However, when gold PP attachment are used, we note a large potential
improvement of 10.46 points in PP attachment accuracies (between the PPA
accuracy for RBG and RBG + Oracle PP), which confirms that adding PP
predictions as features is an effective approach. Our proposed model RBG +
OntoLSTM-PP recovers 15% of this potential improvement, while RBG + HPCD
(full) recovers 10%, which illustrates that PP attachment remains a difficult
problem with plenty of room for improvements even when using a dedicated model
to predict PP attachments and using its predictions in a dependency parser.

We also note that, although we use the same predictions of the HPCD (full)
model in Belinkov et al. (2014)555The authors kindly provided their
predictions for 1942 test examples (out of 1951 examples in the full test
set). In Table 2, we use the same subset of 1942 test examples and will
include a link to the subset in the final draft., we report different results
than Belinkov et al. (2014). For example, the unlabeled attachment score (UAS)
of the baselines RBG and RBG + HPCD (full) are 94.17 and 94.19, respectively,
in Table 2, compared to 93.96 and 94.05, respectively, in Belinkov et al.
(2014). This is due to the use of different versions of the RBG parser.666We
use the latest commit (SHA: e07f74) on the GitHub repository of the RGB
parser.

System | Full UAS | PPA Acc.
---|---|---
RBG | 94.17 | 88.51
RBG + HPCD (full) | 94.19 | 89.59
RBG + LSTM-PP | 94.14 | 86.35
RBG + OntoLSTM-PP | 94.30 | 90.11
RBG + Oracle PP | 94.60 | 98.97
Table 2: Results from RBG dependency parser with features coming from various
PP attachment predictors and oracle attachments.

###  4.2 Analysis

In this subsection, we analyze different aspects of our model in order to
develop a better understanding of its behavior.

#### Effect of context sensitivity and sense priors.

We now show some results that indicate the relative strengths of two
components of our context-sensitive token embedding model. The second row in
Table 3 shows the test accuracy of a system trained without sense priors (that
is, making $p(s|w_{i})$ from Eq. 1 a uniform distribution), and the third row
shows the effect of making the token representations context-insensitive by
giving a similar attention score to all related concepts, essentially making
them type level representations, but still grounded in WordNet. As it can be
seen, removing context sensitivity has an adverse effect on the results. This
illustrates the importance of the sense priors and the attention mechanism.

It is interesting that, even without sense priors and attention, the results
with WordNet grounding is still higher than that of the two LSTM-PP systems in
Table 1. This result illustrates the regularization behavior of sharing
concept embeddings across multiple words, which is especially important for
rare words.

#### Effect of training data size.

Since OntoLSTM-PP uses external information, the gap between the model and
LSTM-PP is expected to be more pronounced when the training data sizes are
smaller. To test this hypothesis, we trained the two models with different
amounts of training data and measured their accuracies on the test set. The
plot is shown in Figure 4. As expected, the gap tends to be larger at smaller
data sizes. Surprisingly, even with 2000 sentences in the training data set,
OntoLSTM-PP outperforms LSTM-PP trained with the full data set. When both the
models are trained with the full dataset, LSTM-PP reaches a training accuracy
of 95.3%, whereas OntoLSTM-PP reaches 93.5%. The fact that LSTM-PP is
overfitting the training data more, indicates the regularization capability of
OntoLSTM-PP.

Figure 4: Effect of training data size on test accuracies of OntoLSTM-PP and LSTM-PP. Model | PPA Acc.
---|---
full | 89.7
\- sense priors | 88.4
\- attention | 87.5
Table 3: Effect of removing sense priors and context sensitivity (attention)
from the model. Figure 5: Two examples from the test set where OntoLSTM-PP
predicts the head correctly and LSTM-PP does not, along with weights by
OntoLSTM-PP to synsets that contribute to token representations of infrequent
word types. The prepositions are shown in bold, LSTM-PP’s predictions in red
and OntoLSTM-PP’s predictions in green. Words that are not candidate heads or
dependents are shown in brackets.

#### Qualitative analysis.

To better understand the effect of WordNet grounding, we took a sample of 100
sentences from the test set whose PP attachments were correctly predicted by
OntoLSTM-PP but not by LSTM-PP. A common pattern observed was that those
sentences contained words not seen frequently in the training data. Figure 5
shows two such cases. In both cases, the weights assigned by OntoLSTM-PP to
infrequent words are also shown. The word types soapsuds and buoyancy do not
occur in the training data, but OntoLSTM-PP was able to leverage the
parameters learned for the synsets that contributed to their token
representations. Another important observation is that the word type buoyancy
has four senses in WordNet (we consider the first three), none of which is the
metaphorical sense that is applicable to markets as shown in the example here.
Selecting a combination of relevant hypernyms from various senses may have
helped OntoLSTM-PP make the right prediction. This shows the value of using
hypernymy information from WordNet. Moreover, this indicates the strength of
the hybrid nature of the model, that lets it augment ontological information
with distributional information.

#### Parameter space.

We note that the vocabulary sizes in OntoLSTM-PP and LSTM-PP are comparable as
the synset types are shared across word types. In our experiments with the
full PP attachment dataset, we learned embeddings for 18k synset types with
OntoLSTM-PP and 11k word types with LSTM-PP. Since the biggest contribution to
the parameter space comes from the embedding layer, the complexities of both
the models are comparable.

##  5 Related Work

This work is related to various lines of research within the NLP community:
dealing with synonymy and homonymy in word representations both in the context
of distributed embeddings and more traditional vector spaces; hybrid models of
distributional and knowledge based semantics; and selectional preferences and
their relation with syntactic and semantic relations.

The need for going beyond a single vector per word-type has been well
established for a while, and many efforts were focused on building multi-
prototype vector space models of meaning (Reisinger and Mooney, 2010; Huang et
al., 2012; Chen et al., 2014; Jauhar et al., 2015; Neelakantan et al., 2015;
Arora et al., 2016, etc.). However, the target of all these approaches is
obtaining multi-sense word vector spaces, either by incorporating sense tagged
information or other kinds of external context. The number of vectors learned
is still fixed, based on the preset number of senses. In contrast, our focus
is on learning a context dependent distribution over those concept
representations. Other work not necessarily related to multi-sense vectors,
but still related to our work includes Belanger and Kakade (2015)’s work which
proposed a Gaussian linear dynamical system for estimating token-level word
embeddings, and Vilnis and McCallum (2015)’s work which proposed mapping each
word type to a density instead of a point in a space to account for
uncertainty in meaning. These approaches do not make use of lexical ontologies
and are not amenable for joint training with a downstream NLP task.

Related to the idea of concept embeddings is Rothe and Schütze (2015) who
estimated WordNet synset representations, given pre-trained type-level word
embeddings. In contrast, our work focuses on estimating token-level word
embeddings as context sensitive distributions of concept embeddings.

There is a large body of work that tried to improve word embeddings using
external resources. Yu and Dredze (2014) extended the CBOW model Mikolov et
al. (2013) by adding an extra term in the training objective for generating
words conditioned on similar words according to a lexicon. Jauhar et al.
(2015) extended the skipgram model Mikolov et al. (2013) by representing word
senses as latent variables in the generation process, and used a structured
prior based on the ontology. Faruqui et al. (2015) used belief propagation to
update pre-trained word embeddings on a graph that encodes lexical
relationships in the ontology. Similarly, Johansson and Piña (2015) improved
word embeddings by representing each sense of the word in a way that reflects
the topology of the semantic network they belong to, and then representing the
words as convex combinations of their senses. In contrast to previous work
that was aimed at improving type level word representations, we propose an
approach for obtaining context-sensitive embeddings at the token level, while
jointly optimizing the model parameters for the NLP task of interest.

Resnik (1993) showed the applicability of semantic classes and selectional
preferences to resolving syntactic ambiguity. Zapirain et al. (2013) applied
models of selectional preferences automatically learned from WordNet and
distributional information, to the problem of semantic role labeling. Resnik
(1993); Brill and Resnik (1994); Agirre (2008) and others have used WordNet
information towards improving prepositional phrase attachment predictions.

##  6 Conclusion

In this paper, we proposed a grounding of lexical items which acknowledges the
semantic ambiguity of word types using WordNet and a method to learn a
context-sensitive distribution over their representations. We also showed how
to integrate the proposed representation with recurrent neural networks for
disambiguating prepositional phrase attachments, showing that the proposed
WordNet-grounded context-sensitive token embeddings outperforms standard type-
level embeddings for predicting PP attachments. We provided a detailed
qualitative and quantitative analysis of the proposed model.

#### Implementation and code availability.

The models are implemented using Keras Chollet (2015), and the functionality
is available at https://github.com/pdasigi/onto-lstm in the form of Keras
layers to make it easier to use the proposed embedding model in other NLP
problems.

#### Future work.

This approach may be extended to other NLP tasks that can benefit from using
encoders that can access WordNet information. WordNet also has some drawbacks,
and may not always have sufficient coverage given the task at hand. As we have
shown in §4.2, our model can deal with missing WordNet information by
augmenting it with distributional information. Moreover, the methods described
in this paper can be extended to other kinds of structured knowledge sources
like Freebase which may be more suitable for tasks like question answering.

## Acknowledgements

The first author is supported by a fellowship from the Allen Institute for
Artificial Intelligence. We would like to thank Matt Gardner, Jayant
Krishnamurthy, Julia Hockenmaier, Oren Etzioni, Hector Liu, Filip Ilievski,
and anonymous reviewers for their comments.

## References

  * Agirre (2008) Eneko Agirre. 2008.  Improving parsing and pp attachment performance with sense information.  In ACL. Citeseer.
  * Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016.  Linear algebraic structure of word senses, with applications to polysemy.  arXiv preprint arXiv:1601.03764 .
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  CoRR abs/1409.0473.
  * Belanger and Kakade (2015) David Belanger and Sham M. Kakade. 2015.  A linear dynamical system model for text.  In ICML.
  * Belinkov et al. (2014) Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014.  Exploring compositional architectures and word vector representations for prepositional phrase attachment.  Transactions of the Association for Computational Linguistics 2:561–572.
  * Brill and Resnik (1994) Eric Brill and Philip Resnik. 1994.  A rule-based approach to prepositional phrase attachment disambiguation.  In Proceedings of the 15th conference on Computational linguistics-Volume 2. Association for Computational Linguistics, pages 1198–1204.
  * Chen and Manning (2014) Danqi Chen and Christopher D Manning. 2014.  A fast and accurate dependency parser using neural networks.  In EMNLP. pages 740–750.
  * Chen et al. (2014) Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014.  A unified model for word sense representation and disambiguation.  In EMNLP. pages 1025–1035.
  * Chollet (2015) François Chollet. 2015.  Keras.  https://github.com/fchollet/keras.
  * Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H. Hovy, and Noah A. Smith. 2015.  Retrofitting word vectors to semantic lexicons.  In NAACL.
  * Huang et al. (2012) Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012.  Improving word representations via global context and multiple word prototypes.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, pages 873–882.
  * Jauhar et al. (2015) Sujay Kumar Jauhar, Chris Dyer, and Eduard H. Hovy. 2015.  Ontologically grounded multi-sense representation learning for semantic vector space models.  In NAACL.
  * Johansson and Piña (2015) Richard Johansson and Luis Nieto Piña. 2015.  Embedding a semantic network in a word space.  In In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics–Human Language Technologies. Citeseer.
  * Kipper et al. (2008) Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008.  A large-scale classification of english verbs.  Language Resources and Evaluation 42(1):21–40.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In NAACL.
  * Lei et al. (2014) Tao Lei, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014.  Low-rank tensors for scoring dependency structures.  In ACL. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013\.  Distributed representations of words and phrases and their compositionality.  CoRR abs/1310.4546.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  Communications of the ACM 38(11):39–41.
  * Neelakantan et al. (2015) Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2015\.  Efficient non-parametric estimation of multiple embeddings per word in vector space.  arXiv preprint arXiv:1504.06654 .
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In EMNLP.
  * Ratnaparkhi et al. (1994) Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994.  A maximum entropy model for prepositional phrase attachment.  In Proceedings of the workshop on Human Language Technology.
  * Reisinger and Mooney (2010) Joseph Reisinger and Raymond J Mooney. 2010.  Multi-prototype vector-space models of word meaning.  In HLT-ACL.
  * Resnik (1993) Philip Resnik. 1993.  Semantic classes and syntactic ambiguity.  In Proceedings of the workshop on Human Language Technology. Association for Computational Linguistics.
  * Rothe and Schütze (2015) Sascha Rothe and Hinrich Schütze. 2015.  Autoextend: Extending word embeddings to embeddings for synsets and lexemes.  In ACL.
  * Vilnis and McCallum (2015) Luke Vilnis and Andrew McCallum. 2015.  Word representations via gaussian embedding.  In ICLR.
  * Yu and Dredze (2014) Mo Yu and Mark Dredze. 2014.  Improving lexical embeddings with semantic knowledge.  In ACL.
  * Zapirain et al. (2013) Benat Zapirain, Eneko Agirre, Lluis Marquez, and Mihai Surdeanu. 2013.  Selectional preferences for semantic role classification.  Computational Linguistics .


# Identifying Linear Relational Concepts in Large Language Models

David Chanin, Anthony Hunter  Oana-Maria Camburu
Department of Computer Science
University College London
London, UK

###### Abstract

Transformer language models (LMs) have been shown to represent concepts as
directions in the latent space of hidden activations. However, for any given
human-interpretable concept, how can we find its direction in the latent
space? We present a technique called linear relational concepts (LRC) for
finding concept directions corresponding to human-interpretable concepts at a
given hidden layer in a transformer LM by first modeling the relation between
subject and object as a linear relational embedding (LRE) Hernandez et al.
(2023b). While the LRE work was mainly presented as an exercise in
understanding model representations, we find that inverting the LRE while
using earlier object layers results in a powerful technique to find concept
directions that both work well as a classifier and causally influence model
outputs.

##  1 Introduction

How do large language models (LLMs) represent concepts, and how can we
identify those concepts in hidden activations? If we can identify human-
interpretable concepts in model activations, we can analyze how concepts are
created and change during inference. Identifying concept representations
inside of models opens up the possibility of visualizing the computation
process of models as sentences are processed, and can help to understand
incorrect or undesirable responses from the model. Moreover, future work
examining how concept directions link together inside the model may benefit
from a robust method to find those concept directions as a first step.

Figure 1: We first model the relation between the subject $s$ and object $o$
as an affine linear transformation called a linear relation embedding (LRE).
We find that inverting the LRE to output subject activations as a function of
object from the relation results in concept directions that serve both as
classifiers and causally impact model outputs. In the above figure, we train
an LRE from the statement “San Jose is in Costa Rica”, and then invert that
LRE to create linear relational concepts (LRCs) representing “located in
England” and “located in China” by providing representations of “York” and
“Shanghai”, respectively.

An intuitive approach when trying to locate a human-interpretable concept,
like the concept of a city being in France, is to collect examples of
sentences with cities that are in France and cities that are not, and train a
probing classifier Ettinger et al. (2016); Finlayson et al. (2021) on hidden
layers of the model, typically a simple linear classifier. However, the
learned classifier may be picking up features correlated with the concept
being probed while overlooking the feature direction that causally influences
model output Hernandez et al. (2023b).

Furthermore, the hidden layer in modern transformer models has high
dimensionality: even older models such as GPT2-xl have 1600 dimensions in the
hidden layer Radford et al. (2019), and modern language models such as
Llama2-7b have over 4000 dimensions for the smallest model (7B) Touvron et al.
(2023). As a result, naively training a probing classifier may require a high
number of training samples.

Our technique builds on work by Hernandez et al. (2023b), which models the
relation between a subject $s$ and object $o$ as an affine linear
transformation, called a linear relational embedding (LRE). While the LRE work
is mainly an investigation into how models represent relational knowledge, we
find that inverting LREs results in concept directions that achieve
surprisingly strong performance as linear classifiers while also causally
impacting model outputs, outperforming standard probing classifiers such as
support vector machines (SVMs). We refer to the concept directions that our
method creates as linear relational concepts (LRCs). LRCs represent concepts
as directions in a latent space, while also functioning as linear classifiers.

We generate LRCs using inverted LREs, as illustrated in Figure 1. We first
generate an LRE for a relation, mapping subject activations in that relation
to their corresponding object activations. Then, we perform a low-rank inverse
of the LRE, mapping from object activations to subject activations. We find
that this low-rank inverse of the LRE results in LRCs that beat traditional
probing classifiers in both classification accuracy and causality, where
causality is defined as being able to control the output of the model. For
instance, we might force the model to output that “London is located in
France” by subtracting the “Located in England” direction from the activation
of “London” and adding the “Located in France” direction.

In addition, LRCs improve on the original LRE work by allowing us to map
subject activations to object activations before the final layer of the model,
and to use all object token activations rather than only using the first
object token. This improves classification accuracy for both single-token and
multi-token objects compared with the original LRE work, which must always map
to the final layer of the model and use only the first object token.

In summary, in this paper, we investigate the problem of locating human-
interpretable concepts within LLM activations. We present a technique for
finding human-interpretable concept directions, called LRC, in the hidden
layer of auto-regressive LLMs such as GPT Radford et al. (2019) and Llama
Touvron et al. (2023). We evaluate our technique using the LRE relations
dataset Hernandez et al. (2023b), and evaluate our approach in both one-vs-all
classification accuracy, and causality (the ability of concepts to modify
model output). Our technique achieves high scores for both classification
accuracy and causality across a the four concept types in the LRE relations
dataset.

Our contributions include the following: (1) Extending LREs to handle multi-
token objects, (2) Using non-terminal model layers for the object activation,
and (3) Using inverted LREs as an intermediate step to find concept directions
in subject activations.

##  2 Background

Previous work on transformer models has shown that features are stored as
directions within the latent space of the model’s hidden activations, known as
the linear representation hypothesis (Elhage et al., 2022). We subscribe to
this theory, and thus look for concepts as direction vectors in the model
activations.

Further work has shown that mid-level multi-layer perceptron (MLP) layers in
transformer LLMs act as key-value stores of information Geva et al. (2021).
These MLP layers enhance the final token of the subject of the sentence (e.g.
the token “lin” in “Berlin is located in the country of”) with this
information in factual relations Geva et al. (2023); Meng et al. (2022).

###  2.1 Linear Relational Embeddings

Linear relational embeddings (LREs) were first presented by Paccanaro and
Hinton (2001) as a way to encode relational concepts as a linear
transformation. Hernandez et al. (2023b) showed that for some relations,
transformer LMs appear to encode relational knowledge using LREs. They model
the processing performed by a transformer LLM mapping from a subject $s$ to an
object $o$ within a textual context $c$ as a linear transformation
$o=F(s,c)=Ws+b$, where $W\in\mathbb{R}^{H\times H},b\in\mathbb{R}^{H}$. This
function $F$ is estimated by a first-order Taylor approximation around $s$,
while $W$ and $b$ are calculated as the mean Jacobian and bias of $n$ samples
$s_{i},c_{i}$ from relation $r$, respectively:

$\displaystyle W=\mathbb{E}_{(s_{i},c_{i})}\left[\frac{\partial F}{\partial
s}\biggm{|}_{(s_{i},c_{i})}\right]$ $\displaystyle
b=\mathbb{E}_{(s_{i},c_{i})}\left[F(s,c)-\frac{\partial F}{\partial
s}s\biggm{|}_{(s_{i},c_{i})}\right]$

A hyperparameter $\beta$ is used to increase the slope of the LRE and can be
configured to improve the performance of the LRE in the case that the Jacobian
underestimates the steepness of $W$, resulting in the following equation for a
LRE $R$:

$R(s)=\beta Ws+b$

LREs are evaluated on faithfulness and causality. The faithfulness metric
checks if the LRE output matches what the model outputs for the first
predicted token when presented with a new subject. For instance, if an LRE
trained on the relation “city in country” predicts the token “France” as the
most likely output given the subject activation of “Paris”, the LRE is said to
be faithful. However, this limits the LRE to modeling only a single token of
object $o$ at the final layer of the model. As a result, LREs cannot
distinguish between words that start with the same token. For example, “Bill
Gates” and “Bill of Rights” both begin with the token “Bill”, and thus cannot
be distinguished by an LRE when evaluated for faithfulness.

To evaluate the causality of the LRE, the LRE is inverted using a low-rank
pseudo-inverse of the weight matrix, indicated $W^{\dagger}$. Hernandez et al.
(2023b) find that using a low-rank pseudo-inverse results in better
performance than using a full-rank inverse. This inversion makes it possible
to calculate a delta $\Delta s$ that could be added to the subject $s$ to
cause the model to change its output from the original object $o$ to a new
object $o^{\prime}$. Causality is evaluated based on whether the model’s
probability of outputting $o^{\prime}$ is greater than the probability of
outputting $o$ after the edit:

$\Delta s=W^{\dagger}(o-o^{\prime})$

We build our method from this technique of inverting the LRE weight matrix to
target the subject activations rather than object activations.

##  3 Method

Our method finds an LRC, represented as a concept direction vector, $v$, for a
given human-interpretable concept in the hidden activations of a transformer
LLM model at a layer $l$. Since we are interested in concepts as directions in
the hidden activation space of the model, we ignore a bias term and focus on
learning only a single unit-length vector to represent the LRC.

Formally, we consider an auto-regressive model $G:\mathcal{X}\to\mathcal{Y}$
with vocabulary $V$ that maps a sequence of tokens
$x=[x_{1},\dots,x_{T}]\in\mathcal{X},x_{i}\in V$ to a probability distribution
$y\in\mathcal{Y}\subset\mathbb{R}^{|V|}$ that predicts the next token of $x$.
Internally, $G$ has a hidden state size $H$, and has $L$ layers. The hidden
activations of layer $l$ of $G$ at token $i$ is represented by
$h_{i}^{(l)}\in\mathbb{R}^{H}$.

We follow the example of Meng et al. (2022) and Hernandez et al. (2023b), and
consider statements of the form $(s,r,o)$ consisting of a subject $s$,
relation $r$, and object $o$. For instance, the statement “Paris is located in
the country of France” would have subject “Paris”, object “France”, and
relation “located in country”. Our definition of a concept corresponds to a
relation and object pair $(r,o)$, which operates on the activations of the
subject $s$. So in our case, we would learn an LRC for the concept “located in
country: France”, and would expect the LRC to have high similarity with the
subject activations of “Paris”, but not the activations of “Berlin” or
“Tokyo”.

We build on the LRE work by Hernandez et al. (2023b), however we make the
following changes to the original LRE method. We use the mean of all object
token activations rather than just the first object token activation to better
handle multi-token objects. We also relax the requirement that only the final
layer can be used for object activations, since we find that classification
performance improves substantially by using non-final layers for the object.
Finally, we restrict training samples for the LRE to only contain examples
where the model answers the prompt correctly. If the model does not know the
answer to a prompt, we assume that the relevant conceptual knowledge we hope
to capture in the LRC is not present, and that the sample will likely be
noise.

For a relation $r$, we have a set of possible objects $O$ that can be the
object for the relation, and each object $o$ has a corresponding set of
subjects $S_{o}$, where each subject $s_{o}\in S_{o}$ corresponds to object
$o$. We first assemble prompts that elicit each object $o\in O$ for the
relation $r$. For example, for the relation “Located in country”, prompts
follow the template "⟨SUBJECT⟩ is located in the country of". Some examples of
prompts and their corresponding objects are shown in Table 1.

Prompt ($s,r$) | Object ($o$)
---|---
Paris is located in the country of | France
Berlin is located in the country of | Germany
Suzhou is located in the country of | China
Tokyo is located in the country of | Japan
Lyon is located in the country of | France
Manaus is located in the country of | Brazil
Table 1: Sample prompts and corresponding object for the relation “Located in
country”.

For a relation for which we wish to build a set of LRCs, we assume a set of
prompts containing subject $s$ and relation $r$, and we expect the model to
predict $o$. We use hidden states from the final token index $i$ of subject
$s$. For the subject "Berlin", this tokenizes to "Ber" and "lin" when using
the GPT tokenizer, so $i$ corresponds to the token "lin" as this is the final
subject token.

We first select $n$ prompts from $r$, balancing the prompts to have as even a
distribution of prompts across $O$. Following Hernandez et al. (2023b), we
train a LRE $R(s)$ consisting of a weight matrix $W$ and bias $b$ using these
$n$ prompts, however we calculate the weight matrix $W$ using the Jacobian of
the mean of all object tokens, not only the first object token. This change
means we model $F(s,c)$ as follows:

$\mathbb{E}[o]=F(s,c)=Ws+b$

This is the same as the original LRE formulation if the object consists of a
single token.

We also ignore the $\beta$ scaling factor from the original LRE definition.
LRCs are normalized to have unit length, and this normalization will remove
any scaling applied to the LRE. Our definition of an LRE, denoted $R(s)$, is
thus simplified as follows:

$R(s)=Ws+b$

We then invert $R(s)$ to get $R^{-1}(o)$, which maps object activations to
subject activations. However, following Hernandez et al. (2023b), we use a
low-rank pseudo-inverse, denoted $R^{\dagger}$ rather than the full matrix
inverse $R^{-1}$:

$R^{\dagger}(o)=W^{\dagger}(o-b)$

To calculate the LRC $v_{o}$ for $o$, we take the mean of all samples of
$R^{\dagger}(o)$ for each prompt $(s,r,o)$ in our training set:

$\hat{v_{o}}=\mathbb{E}[W^{\dagger}(h_{o}-b)]$

Finally, we normalize the LRC direction to have unit length:

$v_{o}=\frac{\hat{v_{o}}}{||\hat{v_{o}}||}$

This normalization is done since we are interested in concepts as directions
only.

##  4 Results

We evaluate our method using the relations dataset from Hernandez et al.
(2023b), which contains prompts for 47 relation types each with a variety of
subject and object pairs, containing over 10,000 instances in total. The
dataset divides relation types into four categories: factual knowledge,
linguistic knowledge, commonsense knowledge, and implicit bias. A subset of
data from a sample relation in the dataset is shown in Table 2. Statistics
about the number of relations and samples per category are shown in Table 3.

We evaluate our performance using two metrics. The first, classification
accuracy, treats each relation as a one-vs-all classification problem, where
the LRC with the highest dot-product similarity to the test subject activation
is considered to be the predicted object $\hat{y}$. Since all LRCs are
normalized to have unit length, this is equivalent to picking the LRC with the
highest cosine similarity:

$\hat{y}=\operatorname*{argmax}_{o\in O}v_{o}\cdot a$

To evaluate causality, we randomly pick a different object $o_{c}$ for prompts
corresponding to each subject in a relation and edit the subject token
activations to predict the new counterfactual object $o^{\prime}$ instead of
the original object $o$. We subtract the original LRC from the final subject
token activations at all layers, and add in the new LRC. For instance, we may
attempt to edit the prompt “Paris is located in the country of” to predict
“Germany” instead of “France” by subtracting the “located in country: France”
concept and adding the “located in country: Germany” concept.

The direction vector in our LRCs are all normalized to unit length, so we
scale the LRCs by a hyperparameter $\beta\in[0,1]$ multiplied by the magnitude
of the subject activation at each layer before adding or subtracting them. The
causal edit at a layer $l$ is thus calculated as below:

$\Delta s^{(l)}=\beta||h_{i}^{(l)}||(v_{o^{\prime}}-v_{o})$

The causality intervention is considered successful if the probability of
predicting the counterfactual object $o^{\prime}$ after the edit is higher
than the probability of predicting the original object $o$. For multi-token
predictions, we use the minimum probability across all predicted tokens rather
than multiplying token probabilities to avoid penalizing objects that require
more tokens to represent. Experimentally, $\beta=0.05$ works well for GPT-J
and $\beta=0.075$ works well for Llama2-7b.

We perform a multi-layer edit in our causality evaluation rather than a
single-layer edit as in Hernandez et al. (2023b), since we feel that single-
layer causality unfairly penalizes learning concepts in later layers of the
model. Since only one layer is edited in single-layer causality, the model
will still attend to the unedited subject activations for earlier layers,
which penalizes using a later layer to find concept directions. Instead, we
perform the same edit at all layers of the subject, so there is no inherent
penalty to using a later layer to learn the LRC, and the model does not attend
to any unedited subject activations. In addition, editing all layers avoids
the need to search for a single optimal layer to perform each edit.

We evaluate against both Llama2-7b Touvron et al. (2023) and GPT-J Wang and
Komatsuzaki (2021). We focus on Llama2-7b for our analysis as it is a more
advanced model than GPT-J, but we include full results for GPT-J in Appendix
A. GPT-J is included as this model was used in the original LRE paper.

relation: city in country
---
FS |  {} is part of {} is in the country of
ZS |  {} is part of the country of {} is located in the country of
|  subject |  object
---|---
New York City |  United States
Rio de Janeiro |  Brazil
Kuala Lumpur |  Malaysia
Las Vegas |  United States
Johannesburg |  South Africa
Panama City |  Panama
Saint Petersburg |  Russia
Table 2: Sample relation data for the “city in country” relation from the dataset, showing zero-shot (ZS) prompt templates, few-shot (FS) prompt templates, and several subject / object pairs. In each prompt template, {} is replaced with a subject. Category | Relations | Samples
---|---|---
Commonsense | 7 | 337
Bias | 7 | 212
Factual | 21 | 9462
Linguistic | 4 | 660
Table 3: Statistics for the number of relations and samples of each category in the dataset after filtering out one-to-one relations. Tokens | Llama2-7b samples | GPT-J samples
---|---|---
1 | 2393 | 2108
2 | 451 | 39
3 | 371 | 2
4 | 107 | 6
5+ | 4 | 0
Table 4: Statistics for the average number of tokens in objects for the test
set for Llama2-7b and GPT-J after filtering out one-to-one relations and
samples which the model answers incorrectly. The majority of samples are
single-token, but Llama2-7b still answers correctly a large number of multi-
token object samples. GPT-J performs worse then Llama2-7b especially on multi-
token object samples, barely answering any correctly.

For each relation, we split the dataset into a 50%/50% train/test split by
relation and object, ensuring at least one training example is present per
object in the relation. We prepend four other examples from the same relation
to each training example to act as few-shot learning examples. However, during
causality and classification benchmarks, we use only zero-shot prompts from
the dataset, following the procedure in the original LRE paper. An example
few-shot prompt is shown in Figure 2. This process is repeated five times with
different random seeds for train/test splits, and results are averaged across
the results from each seed. The shaded area in plots corresponds to this
standard deviation.

The superlative form of bad is worst
The superlative form of big is biggest
The superlative form of brave is bravest
The superlative form of bright is brightest
The superlative form of angry is

Figure 2: Sample few-shot (FS) prompt for the relation “adjective
superlative”, subject “angry”, and object “angriest” from the dataset. FS
prompts are used for training, while testing consists only of zero-shot (ZS)
prompts.

For some relations, there is a one-to-one mapping between subject and object,
so it is impossible to create a test split with unseen subject/object pairs to
evaluate. An example of a relation of this type is “capital city of country”.
Each country has one capital city, and each capital city is the capital of
only one country. Since our definition of a concept includes a unique $r$ and
$o$ pair, we are unable to evaluate these relations and exclude them from the
evaluation. We furthermore exclude any relations with less than five test
samples after filtering as having so few test samples makes it had to evaluate
performance robustly.

When training LRCs via inverted LREs using our method, we use five training
samples per LRE, and use rank 192 for the low-rank pseudo-inverse, unless
specified otherwise. All calculations are performed using 16-bit quantization.
Unless otherwise specified, all evaluations are performed with subject layer
17 and object layer 21 for Llama2-7b, and subject layer 14 and object layer 20
for GPT-J.

For objects that tokenize to multiple tokens, we evaluate against the full
multi-token output. All prompts for both train and test datasets are pre-
filtered to exclude any prompts where the model gives an incorrect answer,
since if the model does not get the correct answer it is unclear that the
concept being classified will be elicited in the hidden activations at all.

###  4.1 Comparisons

We compare our method against training a standard linear support vector
machine (SVM) classifier on the hidden activation data, as well as estimating
a concept direction by simply averaging together the hidden activations for a
given object.

We also compare our method to a LRC trained using the final layer for the
object token, as in the original LRE paper where the final layer is always
used for objects.

###  4.2 Classification accuracy and causality

For both classification accuracy and causality, we calculate a score per
relation, and then average the scores across relations. We do this because
some relations have more test samples than other relations, which would
otherwise bias the results towards to reflect whichever relation has the most
test samples and not give a good reflection of performance across the full
range of relation types in the relations dataset.

We measure classification accuracy and causality on the relations dataset for
our method, our method ablating the alignment loss, using a pure SVM, and
using activation averaging. Results are shown in Table 5.

Llama2-7b
---
Method | Accuracy | Causality
LRC (mean, $l_{21}$) |  0.81 ± 0.01 |  0.78 ± 0.02
LRC (ft, $l_{\textrm{final}}$) | 0.74 ± 0.02 |  0.78 ± 0.02
SVM | 0.73 ± 0.02 | 0.69 ± 0.01
Input averaging | 0.70 ± 0.01 | 0.55 ± 0.03
GPT-J
Method | Accuracy | Causality
LRC (mean, $l_{20}$) |  0.81 ± 0.02 | 0.84 ± 0.01
LRC (ft, $l_{\textrm{final}}$) | 0.78 ± 0.02 |  0.86 ± 0.01
SVM | 0.75 ± 0.02 | 0.76 ± 0.01
Input averaging | 0.73 ± 0.03 | 0.56 ± 0.02
Table 5: Classification accuracy and causality results on the relations
dataset for Llama2-7b and GPT-J. LRC is our method. “mean” refers to using the
mean of all object tokens to calculate the LRE, while “ft” refers to using
only the first token of the object to calculate a LRE. LRC (ft,
$l_{\textrm{final}}$) is included as an ablation to best estimate the results
of inverting the original LRE technique at the final layer. Results include
mean and standard deviation after five random seeds.

Our method performs the best on both classification accuracy and causality.
Classification accuracy improves by a large margin simply by using layer 28
instead of the final Llama2-7b layer 31, showing the importance of allowing
the LRE to use a non-terminal layer.

###  4.3 Multi-token vs single-token objects

We also compare the performance of LRCs on single-token and multi-token
objects. One of the main limitations of the original LRE work is not being
able to handle multi-token objects, so we expect the improvement of our method
over traditional LREs to be most stark for multi-token objects.

To investigate the impact of the final object layer on single-token and multi-
token performance, we evaluate our method on each layer from layer 18 to the
final layer 31 for Llama2-7b keeping layer 17 as the subject layer. We only
use Llama2-7b for this comparison since GPT-J has very few multi-token prompts
from the dataset that it can answer correctly. Multi-token classification
accuracy and causality by object layer is shown in Figure 3. Single-token
classification accuracy and causality by object layer is shown in Figure 4.

Figure 3: Classification accuracy and causality by object layer for multi-
token objects on Llama2-7b. Shaded area indicates standard deviation after
five seeds. Figure 4: Classification accuracy and causality by object layer
for single-token objects on Llama2-7b. Shaded area indicates standard
deviation after five seeds.

Both single-token and multi-token performance improves by using earlier object
layers, but the difference is especially pronounced for multi-token objects.

###  4.4 Impact of the rank of the LRE inverse

One of the surprising results from Hernandez et al. (2023b) is that using a
low-rank inverse of the LRE results in better performance than using a full-
rank inverse. We investigate this relationship between the rank of the LRE
inverse and performance on the relations dataset for the LRC method as well.
Classification accuracy and causality results are shown in Figure 5.

Figure 5: Classification accuracy and causality on the relations dataset by
LRE inversion rank on Llama2-7b. Shaded area indicates standard deviation
after five seeds.

Using a low-rank LRE inverse improves performance dramatically, and with
performance peaking around rank 200 for Llama2-7b. Llama2-7b has a 4096
dimension hidden space, so a full-rank inverse would have rank 4096, meaning a
rank 200 inverse is discarding over 90% of the LRE weight matrix. We suspect
that the generalization power of using an inverted LRE to find concept
directions comes from this low-rank inverse, where the most important common
components of the relation are captured in a small portion of the LRE weight
matrix.

###  4.5 Choosing samples to train the LRE

We follow the example of Hernandez et al. (2023b) and use a small subset of
the available training samples to train the LRE. For our method, we use the
LRE only as an intermediate step in deriving a LRC, which makes it possible to
train a LRE optimized to perform best on the specific relation and object
$(r,o)$ we are interested in training a LRC for. A natural instinct is to only
choose samples for the LRE which contain only the object in question. For
instance, to train a LRC for “Located in country: France”, it would seem
natural to only use training samples consisting of cities in France to train
the LRE, rather than cities in other countries.

We investigate this in the simplest case of using only a single training
sample to train the LRE, since for many objects in the relations dataset only
a single training sample is available. We compare training the LRE using and
LRC using a sample which represents the same object, as well as training a LRE
with a sample which is from another object in the same relation, but not the
object the LRC will represent. The results of this experiment are shown in
Table 6.

LRE train sample | Accuracy | Causality
---|---|---
Same object | 0.31 ± 0.04 | 0.31 ± 0.02
Different object |  0.69 ± 0.01 |  0.70 ± 0.02
Table 6: Results for training a LRC derived from a LRE trained with a single
training sample for the relations dataset, where that sample either represents
the same object as the LRC (Same object) or a different object in the same
relation (Different object). Unintuitively, when the sample the LRE is trained
on is the same object as the LRC, the LRC performs extremely poorly. These
results are from Llama2-7b.

Unintuitively, training the LRE using a sample with the same object as the LRC
results in dramatically worse performance. We do not yet understand why this
is, but suspect that choosing samples from different objects may have a
regularizing effect on the resulting LRC. Using the same object to train a LRE
and also extract a LRC is equivalent to estimating the concept direction on
the subject as the derivative relative to the object, but it is possible that
this results in directions which may be unrealistic in practice. More
investigation will be necessary to understand this phenomena in-depth, but for
our purposes we find that it is essential that the training samples for the
LRE contain different objects from the same relation, not only the same object
as the LRC.

###  4.6 Subject layer causality vs accuracy

While we try to address the issue that using a later subject layer for LRC
training results in fewer opportunities to causally intervene in the model
processing by evaluating causality by performing a multi-layer edit, we still
find a trade-off between causality and classification accuracy depending on
the subject layer of LRC. Using an earlier layer allows us to find the
maximally causal intervention to change model outputs, but means that
classification accuracy suffers since the MLP layers of the model have not yet
had a chance to fully enhance the subject token with relevant information
about the subject. Figure 6 shows classification accuracy and causality
results on the relations dataset for training the LRC using subject layer 10
through 21, with object layer 22 on Llama2-7b.

Figure 6: Classification accuracy and causality on the relations dataset by
subject layer on Llama2-7b with object layer 22. Shaded area indicates
standard deviation after five seeds.

We see causality being highest with earlier layers, while classification
accuracy appears to roughly increase up to layer 19, after which performance
rapidly declines. We suspect this trade-off is a limitation of learning a LRC
using a single pair of subject and object layers, and that it may be possible
to combine LRCs learned at different layers to find a direction which achieves
both peak classification accuracy and causality.

##  5 Related work

Previous work on understanding how deep neural networks represent concepts
focuses on understanding individual neurons (Bills et al., 2023; Yosinski et
al., 2015). However, individual neurons have been found to activate in
response to multiple different concepts, making a clean, human-interpretable
understanding of each neuron difficult Goh et al. (2021). Indeed, transformers
can represent more concepts than they have neurons in their hidden layers
(Elhage et al., 2022).

A great source of inspiration of this this work has been work on knowledge
editing in language models, specifically ROME Meng et al. (2022) and REMEDI
Hernandez et al. (2023a). In ROME, factual knowledge is shown to reside in the
mid-layer MLP of language models, and this knowledge can be edited by updating
an arbitrary mid-layer MLP to insert any fact desired. ROME finds the
appropriate update by means of counterfactual backpropagation, similar to our
work. Indeed, ROME was as a main source of inspiration and a starting point
for our work.

In REMEDI, model outputs are edited by adding a vector to to the subject of a
sentence during the forward inference pass. This is similar to our work in
that this vector being added to the hidden activations of a subject can be
said to contain the concept that is desired to be elicited. However, the goal
of REMEDI is to edit model outputs rather than identifying concept direction
vectors and building a classifier like in our work.

We also take inspiration from work on probing classifiers Belinkov (2022);
Ettinger et al. (2016). Probing classifiers are simple, typically linear,
classifiers which operate on hidden activations inside of neural networks to
determine if the activations. TCAV Kim et al. (2018) can be said to be a
probing classifier for vision models, where a classifier is learned at
multiple layers in the model. Most similar to our work, Li et al. (2021) build
a probing classifier to identify states in a world model for textual games
from language model hidden activations, and show that the hidden activations
of neural networks do encode a basic world model. However, this work focuses
on encoder-decoder models, and does not attempt to classify arbitrary human-
interpretable concepts outside of the text game context. This work also takes
the approach of directly training a linear classifier for a given concept
known to be present in the encoder activations rather than directly trying to
find a concept direction as in our work.

The most similar to this paper is work on Linear Relational Embedding concepts
in LLMs Hernandez et al. (2023b), which is the source for our evaluation
dataset and serves as the first step in our method. This work also attempts to
estimate relational concepts, and does so by learning a linear affine
transformation from the subject token activation at layer $l$ to the first
output token of the object at the final layer of the model. However, as LRE
concepts only map to the first object token, they are not suited to multi-
token objects. For instance, a LRE cannot distinguish between “Bill Gates” and
“Bill Clinton” as they both begin with the same token. In addition, the
original LRE work is presented as a means of understanding how LLMs may encode
information rather than attempting to build a classifier or find concept
directions as in our work.

##  6 Conclusion

Identifying and classifying a broad set of human-interpretable concepts in
language model activations is a vital step towards understanding and debugging
how language models operate. In this work, we have shown a technique for
identifying and classifying concepts in model hidden activations using
inverted LREs. We show that our technique is both a better classifier and
learns directions with higher causality than standard linear classifiers like
SVM.

While our technique performs well, there is still a large amount of variance
in performance depending on the training samples chosen. We expect further
improvements can be achieved by determining which LRE training samples will
result in the best performance for each LRC. In addition, we suspect that it
should be possible to combine LRCs learned at multiple layers to find concept
directions that achieve even better results to get around the trade-off of
causality performing best with earlier subject layer LREs, and classification
accuracy performing better with later subject layer LREs.

In the future, we hope concept identification techniques such as this will
make it possible to investigate the relations between concepts within model
weights, and extract learned knowledge and even world models directly from
pretrained language models.

## Limitations

Our method requires learning a new LRC for every $(r,o)$ pair, so cannot
generalize to new objects without a training sample of that $(r,o)$. Our
evaluation also assumes that each subject maps only to a single object in the
same relation, and would need modifications to handle subjects with multiple
objects in the same relation, such as a movie that can have multiple genres,
but we do not investigate that in this work.

Our method assumes that each human-interpretable concept corresponds to a
direction in the hidden space of the model, and we assume that if the model
outputs the correct answer to a prompt then the model has a representation of
this concept in its activations. However, it is also possible for the model to
guess the correct answer without having any underlying representation, which
will cause our method to not perform well. For instance, for the prompt “Sam
Eastwood’s father is named”, the model will output the correct answer “Clint
Eastwood”. However, did the model have an underlying representation of this
fact in its hidden activations, or is it simply guessing the most famous
person with the last name “Eastwood”, which is Clint Eastwood? Indeed, GPT-J
will output “Clint Eastwood” as the father of almost any made up person with
the last name “Eastwood”. Our method would likely perform much better if these
cases where the model can guess the correct answer were filtered out, but
differentiating between the model guessing and knowing the correct answer is
challenging.

Recent work suggests that sometimes the knowledge that maps subjects to
objects is not present in MLP layers applied to the subject token, but instead
is contained directly in attention values and only is added to the residual
stream of the output tokens instead of the subject Geva et al. (2023). For
knowledge of this sort, our method will fail since we assume all knowledge can
be found in the subject token residual stream rather than needing to look at
the output token.

Finally, our method works only for relational concepts of the form $(s,r,o)$.
Other types of concepts which do not easily fit into this format would require
an adaptation or a different technique.

## Acknowledgements

Oana-Maria Camburu was supported by a Leverhulme Early Career Fellowship.
David Chanin was supported with thanks to EPSRC EP/S021566/1.

## References

  * Belinkov (2022) Yonatan Belinkov. 2022.  Probing classifiers: Promises, shortcomings, and advances.  _Computational Linguistics_ , 48(1):207–219.
  * Bills et al. (2023) Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023.  Language models can explain neurons in language models.  _URL https://openaipublic. blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023)_.
  * Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. 2022.  Toy models of superposition.  _arXiv preprint arXiv:2209.10652_.
  * Ettinger et al. (2016) Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016.  Probing for semantic evidence of composition by means of simple classification tasks.  In _Proceedings of the 1st workshop on evaluating vector-space representations for nlp_ , pages 134–139.
  * Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021.  Causal analysis of syntactic agreement mechanisms in neural language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1828–1843, Online. Association for Computational Linguistics.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  _arXiv preprint arXiv:2304.14767_.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Goh et al. (2021) Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. 2021.  Multimodal neurons in artificial neural networks.  _Distill_ , 6(3):e30.
  * Hernandez et al. (2023a) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023a.  Measuring and manipulating knowledge representations in language models.  _arXiv preprint arXiv:2304.00740_.
  * Hernandez et al. (2023b) Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023b.  Linearity of relation decoding in transformer language models.  _arXiv preprint arXiv:2308.09124_.
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677. PMLR.
  * Li et al. (2021) Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021.  Implicit representations of meaning in neural language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1813–1827, Online. Association for Computational Linguistics.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Paccanaro and Hinton (2001) Alberto Paccanaro and Geoffrey E. Hinton. 2001.  Learning distributed representations of concepts using linear relational embedding.  _IEEE Transactions on Knowledge and Data Engineering_ , 13(2):232–244.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  Gpt-j-6b: A 6 billion parameter autoregressive language model.
  * Yosinski et al. (2015) Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015.  Understanding neural networks through deep visualization.  In _Deep Learning Workshop, International Conference on Machine Learning (ICML)_.

##  Appendix A Appendix

###  A.1 Extended results

Full results broken down by each relation tested is shown in Figure 7 for
Llama2-7b, and 8 for GPT-J. These plots compare the results for our method
(LRC) against the results for support vector machines (SVM).

Figure 7: Classification accuracy broken down by relation for LRC (ours)
compared to SVM on Llama2-7b. Our method outperforms SVM on most, but not all,
relations. Figure 8: Classification accuracy broken down by relation for LRC
(ours) compared to SVM on GPT-J. Our method outperforms SVM on most, but not
all, relations.

Results for the effect of the rank of the LRE weight matrix inverse on
performance of the LRC method for GPT-J is shown in Figure 9.

Figure 9: Classification accuracy and causality on the relations dataset by
LRE inversion rank on GPT-J. Shaded area indicates standard deviation after
five seeds.

Results illustrating the effect of object layer choice our method for GPT-J is
shown in Figure 10, with subject layer 15. We do not break down the effect of
this object layer choice by single-token vs multi-token objects since GPT-J
answers very few multi-token object prompts correctly.

Figure 10: Classification accuracy and causality on the relations dataset by
LRE object layer on GPT-J with subject layer 15. Shaded area indicates
standard deviation after five seeds.

Results illustrating the effect of the subject layer choice on LRC performance
are shown in Figure 11 with object layer 20. As with Llama2-7b, we find a
trade-off between causality and classification accuracy, where earlier layers
result in better causality performance at the expense of classification
accuracy.

Figure 11: Classification accuracy and causality on the relations dataset by
LRE subject layer on GPT-J. Shaded area indicates standard deviation after
five seeds.

###  A.2 Statistical significance

We calculate statistical significance between our method (LRC) and SVM for
classification accuracy and causality. We find that LRCs performance
improvement over SVM is statistically significant.

We use a two-proportion Z-test to calculate significance. Since we run five
random seeds with different train / test splits, we calculate significance for
each random split separately to avoid double-counting samples which may occur
in different splits. This should make our significance estimate more
conservative than if we sum the results across all splits.

In order to simplify the significance calculation, the scores are not
reweighted by relation as is done in the results in the paper, so if a
relation has many more samples than another relation, we do not reweight to
account for that in this calculation. As a result, the LRC and SVM scores per
iteration are slightly different than appears earlier in the paper. P-value
calculations for Llama2-7b are shown in Figure 7, and for GPT-J in Figure 8.

Classification accuracy (Llama2-7b)
---
Seed | Test samples | LRC | SVM | P-val
42 | 3324 | 0.842 | 0.811 | 9e-4
43 | 3326 | 0.845 | 0.804 | 1e-5
44 | 3319 | 0.839 | 0.808 | 9e-4
45 | 3354 | 0.838 | 0.816 | 0.016
46 | 3335 | 0.843 | 0.803 | 2e-5
Causality (Llama2-7b)
Seed | Test samples | LRC | SVM | P-val
42 | 1527 | 0.762 | 0.652 | 3e-11
43 | 1533 | 0.733 | 0.606 | 7e-14
44 | 1517 | 0.740 | 0.633 | 2e-10
45 | 1497 | 0.764 | 0.607 | 3e-20
46 | 1497 | 0.723 | 0.627 | 2e-8
Table 7: Statistical significance calculation for classification accuracy
comparison for our method (LRC) compared with SVM using Llama2-7b. All
comparisons are at subject layer 17. We use object layer 21 for LRC. All
P-values from each seed for both classification accuracy and causality are
well below the 0.05 threshold for statistical significance. In order to
simplify the significance calculation, these scores are not reweighted by
relation as is done in the results in the paper, so if a relation has many
more samples than another relation, we do not reweight to account for that in
this calculation. Classification accuracy (GPT-J)
---
Seed | Test samples | LRC | SVM | P-val
42 | 2181 | 0.825 | 0.793 | 0.007
43 | 2129 | 0.803 | 0.800 | 0.818
44 | 2176 | 0.784 | 0.816 | 0.008
45 | 2173 | 0.789 | 0.791 | 0.882
46 | 2236 | 0.812 | 0.789 | 0.0517
Causality (GPT-J)
Seed | Test samples | LRC | SVM | P-val
42 | 1049 | 0.699 | 0.546 | 6e-13
43 | 1054 | 0.733 | 0.602 | 1e-10
44 | 1088 | 0.716 | 0.581 | 4e-11
45 | 1014 | 0.735 | 0.570 | 7e-15
46 | 1097 | 0.718 | 0.560 | 1e-14
Table 8: Statistical significance calculation for classification accuracy
comparison for our method (LRC) compared with SVM using GPT-J. All comparisons
are at subject layer 14. We use object layer 20 for LRC. The classification
accuracy results for LRC are not statistically significant compared with SVM,
but the causality results are significantly significant. In order to simplify
the significance calculation, these scores are not reweighted by relation as
is done in the results in the paper, so if a relation has many more samples
than another relation, we do not reweight to account for that in this
calculation.

For Llama2-7b, our method is statistically significantly better than SVM for
both classification accuracy and causality. However, for GPT-J, the
classification accuracy difference is not statistically significant between
our method and SVM, but our method does outperform SVM on causality with
statistical significance.