What Kind of Pretraining Data Do Large Language Models Rely on When Doing Reasoning?
Under review as a conference paper at ICLR 2025

PROCEDURAL KNOWLEDGE IN PRETRAINING DRIVES
REASONING IN LARGE LANGUAGE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

The capabilities and limitations of Large Language Models (LLMs) have been
sketched out in great detail in recent years, providing an intriguing yet conflicting
picture. On the one hand, LLMs demonstrate a general ability to solve prob-
lems. On the other hand, they show surprising reasoning gaps when compared
to humans, casting doubt on the robustness of their generalisation strategies. The
sheer volume of data used in the design of LLMs has precluded us from applying
the method traditionally used to measure generalisation: train-test set separation.
To overcome this, we study what kind of generalisation strategies LLMs employ
when performing reasoning tasks by investigating the pretraining data they rely
on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining
tokens, we identify what documents influence the model outputs for three simple
mathematical reasoning tasks and contrast this to the data that are influential for
answering factual questions. We find that, while the models rely on mostly dis-
tinct sets of data for each factual question, a document often has a similar influence
across different reasoning questions within the same task, indicating the presence
of procedural knowledge. We further find that the answers to factual questions
often show up in the most influential data. However, for reasoning questions the
answers usually do not show up as highly influential, nor do the answers to the
intermediate reasoning steps. When we characterise the top ranked documents for
the reasoning questions qualitatively, we confirm that the influential documents
often contain procedural knowledge, like demonstrating how to obtain a solution
using formulae or code. Our findings indicate that the approach to reasoning the
models use is unlike retrieval, and more like a generalisable strategy that synthe-
sises procedural knowledge from documents doing a similar form of reasoning.

1

INTRODUCTION

Current advancements in artificial intelligence are characterised by the increasing scale of datasets,
computational power, and model size (Kaplan et al., 2020; Hoffmann et al., 2022). While one of the
manifestations of this approach, Large Language Models (LLMs), is rapidly saturating benchmarks
measuring reasoning capabilities (e.g. Cobbe et al., 2021; Hendrycks et al., 2021), the debate over
whether they exhibit ‘genuine understanding’ is ongoing (as reviewed by Mitchell & Krakauer,
2023). The well-documented robust and versatile reasoning abilities (e.g. Webb et al., 2023; 2024;
McLeish et al., 2024) sharply contrast with the line of work highlighting the brittleness of LLM
reasoning (Razeghi et al., 2022; McCoy et al., 2023; Ullman, 2023; Wu et al., 2024; Mahowald
et al., 2024). A finding common to these works is that LLM reasoning depends on the frequency of
similar problems in the training data.

A key reason for why benchmark saturation cannot be taken at face value is the issue of data con-
tamination; benchmark data often appear in the pretraining set. Where we typically measure gener-
alisation in machine learning by separating the test data from the training data, the trillions of tokens
used in the design of current state-of-the-art models cannot reasonably be separated from bench-
marks anymore. Recent works have documented the extent of the contamination issue (Brown et al.,
2020; Touvron et al., 2023; Gunasekar et al., 2023; Yang et al., 2023; Deng et al., 2024), showing
that many common benchmarks have a high percentage of contaminated data. Additionally, Yang
et al. (2023) show that even rephrased benchmark data that elude N-gram-based detection meth-

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: A summary of our most important findings towards answering the question “how do LLMs
learn to reason from pretraining data?” We rank 5 million pretraining documents according to their
influence on the likelihood of completions of two models, Cohere’s Command R 7B and 35B, for 40
factual and 40 reasoning queries. We find that procedural knowledge drives influence on reasoning
traces: a document’s influence on the reasoning traces of one query is strongly predictive of that
document’s influence on another query with the same mathematical task, in 3 of the 4 cases. We
show this on the left through arrows indicating influence, and on the right through correlations of
all 5M document influences between a random sample of 10 queries per task (a plot with all queries
can be found in Figure 12 in Appendix A.9.1). Further, we find that the answers to factual queries
often show up in the top 0.01% of pretraining documents (see text in bottom row of documents), but
not for the reasoning questions. Finally, individual documents influence reasoning traces much less
strongly than factual answer generations, indicating models rely on documents less when reasoning.
All documents and queries shown are redacted versions of real data, and the relations are based on
documents found in the top 50 for the queries.

ods can impact performance, further complicating the issue. However, it is unclear how and when
state-of-the-art LLMs rely on contaminated data to perform reasoning.

This raises the question: “how do LLMs learn to reason from pretraining data?” In this work, we
take a complementary approach to most interpretability research by focusing on the pretraining data
used by language models to generalise, rather than interpreting the model weights themselves. We
investigate which data influence the model’s produced reasoning traces and how those data relate
to the specific problems being addressed. Are models simply ‘retrieving’ answers from previously
seen pretraining data and reassembling them, or are they employing a more robust strategy for
generalisation? We use a technique from robust statistics (Hampel, 1974) adapted to large-scale
Transformers (Koh & Liang, 2017; Grosse et al., 2023) to compute the influence of pretraining
documents on the likelihood of prompt-completions pairs under a trained model. In the extreme
case, a language model answering reasoning questions may rely heavily on retrieval from parametric
knowledge influenced by a limited set of documents within its pretraining data. In this scenario,
specific documents containing the information to be retrieved (i.e. the reasoning traces) contribute
significantly to the model’s output, while many other documents play a minimal role. Conversely,
at the other end of the spectrum, the model may draw from a broad range of documents that are
more abstractly related to the question, with each document influencing many different questions
similarly, but contributing a relatively small amount to the final output. We propose generalisable
reasoning should look like the latter strategy.

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

We investigate the pretraining data (called ‘documents’) that are influential for a set of factual and
reasoning questions (called ‘queries’). The reasoning questions cover three mathematical tasks;
two-step arithmetic, calculating slopes, and solving linear equations. The factual questions require
retrieving from parametric knowledge. We experiment with two LLMs (7B and 35B) and 2.5B of
their pretraining tokens. Our findings are as follows (summarised in Figure 1):

1. Procedural knowledge in documents drives influence on reasoning traces: a docu-
ment’s influence on the reasoning traces of a query is strongly predictive of that document’s
influence on another query with the same mathematical task (Figure 1 and Finding 1 in Sec-
tion 5.1). By contrast, this does not hold for factual queries. This indicates that documents
often contribute similarly to many questions that require applying the same procedure to
different numbers. The correlation is particularly strong for queries involving calculating a
slope, and for that task we find procedures for a solution in code or math in the top 0.002%
of ranked pretraining data multiple times for most queries (Finding 4 in Section 5.2).

2. The models rely less on individual documents for reasoning questions, and the set
of documents they rely on is less specific: we find that the magnitude of influence of
documents per unit of query information generated by the models is usually much lower
for reasoning questions than for factual questions (Finding 2 in Section 5.1). Further, the
overall magnitude of influence of the set of documents is less volatile. The former indicates
that when generating reasoning traces, the models rely less on each individual document
per nat of query information they generate than for factual retrieval. The latter indicates
that for a random subset of 2.5B pretraining tokens, it is more up to chance whether highly
influential documents are part of it for factual questions than for reasoning questions. Taken
together, this indicates the models likely generalise from a more general set of documents
for reasoning than for factual questions, relying on each individual document less.

3. For the factual questions, the answer often shows up as highly influential, whereas for
reasoning questions it does not: we look at the top 500 (top 0.01%) influential documents
for each query, and find the answer to factual questions relatively often (55% of the queries
for the 7B, and 30% for the 35B), and almost never for reasoning questions, even when we
do find the answers in the larger set of 2.5B tokens (Finding 3 in Section 5.2).

4. We find evidence for code being important for mathematical reasoning: code data is
strongly overrepresented w.r.t. the training distribution for the top portions of the positively
and negatively influential rankings for reasoning queries (Finding 5 in Section 5.2).

Our findings suggest a generalisation strategy for reasoning that is unlike retrieval from the paramet-
ric knowledge formed during pretraining. Instead, the models learn to apply procedural knowledge
extracted from documents involving similar reasoning processes, either in the form of general de-
scriptions of procedures, or applications of similar procedures. This indicates that we may not need
to cover every possible case in the pretraining data: focusing on high-quality data demonstrating
procedures across diverse reasoning tasks could be more effective. Although our findings are lim-
ited to models learning from procedures within the same mathematical task, we observe that code
plays a significant role for all tasks we look at. This raises an interesting question: is there a type
of pretraining data — such as code — from which models, particularly larger ones, can learn about
multiple tasks? Understanding the extent of procedural generalisation can inform future pretraining
strategies and help determine where to concentrate data selection efforts.

2 RELATED WORK

The subfield with the aim of understanding how large language models generalise is growing rapidly.
This question can be approached in different ways, and many recent works interpret weights of
smaller models on synthetic tasks to explain particular phenomena that we observe in language
models at scale such as grokking (Wang et al., 2024), in-context learning (Olsson et al., 2022; Singh
et al., 2024), or superposition (Elhage et al., 2022; Bricken et al., 2023). Scaling interpretability
methods to modern-sized LLMs is challenging for many reasons, of which one is computational
tractability. Nonetheless, there are a few works that apply techniques from interpretability to lan-
guage models at scale. Templeton et al. (2024) use sparse autoencoders to extract interpretable
features from Claude 3 Sonnet, and demonstrate how to use these features to control model outputs.

3

Under review as a conference paper at ICLR 2025

Grosse et al. (2023) adapt EK-FAC influence functions (George et al., 2018) to large-scale Trans-
formers, and use them to understand what kind of pretraining data influence completions of models
up to 50B parameters. The authors show, among many other things, that larger models rely on pre-
training data that are more abstractly related to the completion than smaller models. In this work,
we build on the results of Grosse et al. (2023), leaning heavily on their efforts to make influence
functions tractable at scale, but focus instead on understanding reasoning specifically.

3 COMPUTING THE INFLUENCE OF A DOCUMENT ON A COMPLETION

Background on influence functions. Given a pretrained model θu that parametrises a distribution
over next tokens conditioned on a prompt pθu (yc | yp) (where yc = {y1, . . . , ym} is a com-
pletion, yp = {y1, . . . , yn} a prompt, and u indicates the parameters are not necessarily trained
to convergence), we are interested in finding data from the pretraining set D = {xi}N
i=1 that in-
fluence the completion. Put differently, we want to know which examples in the pretraining set
‘caused’ a completion. To this end, we use EK-FAC influence functions for large-scale transform-
ers as proposed by Grosse et al. (2023). The parameters θu are typically found by performing
a gradient-based iterative algorithm on an objective function and stopping based on some crite-
rion. We want to know the influence of a training document xj ∈ D on the parameters θu (which
can be reformulated to influence on any continuous differentiable function of θu using the chain-
rule). We can calculate influence exactly by removing xj from the original training set, re-training
the model, and comparing the resulting set of parameters (or a function thereof) to the originally
trained model. This is intractable for any interesting number of documents and parameters. Influ-
ence functions estimate this counterfactual by taking a Taylor expansion of the response function:1
θ⋆(ϵ) = arg minθ∈RD
i̸=j L(xi, θ) + ϵL(xj, θ), where L(·) is a loss function, like the cross-
entropy. The first-order Taylor approximation around ϵ = 0 of the response function is used to
reason about how the optimal parameters change if you change ϵ, which changes the weight of the
document we want to know the influence of. Using the implicit function theorem, influence can
then be defined as follows: Iθ⋆ (x) = dθ⋆
θJ (θp, D)
dϵ
is the Hessian of the pretraining objective. Using the chain-rule, we estimate influence of a training
document x = {x1, . . . , xk} on the completion given a prompt by approximating the following:

= −H−1∇θL(x, θ⋆). Where H = ∇2

(cid:12)
(cid:12)
(cid:12)ϵ=0

(cid:80)

1
N

If (x) = −∇θf (θu)T H−1∇θL(x, θu)

(1)

Since we are investigating models with billions of parameters D, the above Hessian is intractable,
and we estimate it using EK-FAC estimation. For a detailed derivation, the reader is referred to
Section 2 and 3 in Grosse et al. (2023). We will mention here that it involves estimating two ex-
pectations Epθ [∆θ∆θT ] and Epθ [AAT ] where A denotes the activations of the model. To make
this estimation tractable we make a number of simplifying assumptions across all our estimations,
like independence between layers and we only take into account MLP parameters of the transformer
layers (Grosse et al., 2023). A full list of approximations can be found in Appendix A.7.

Adapting EK-FAC influence functions to our problem. Prior work has shown that EK-FAC in-
fluence functions more accuractely estimate the counterfactual given by the response function than
other types of influence functions (Grosse et al., 2023). However, besides influence on language
model completions, we are also interested in influence on the accuracy of a trained language model
when answering questions. We can only calculate the influence on a continuous differentiable func-
tion, and to the best of our knowledge, no work has shown that influence functions also estimate
effect on the underlying accuracy of text produced by next-word prediction. As a proxy for accu-
racy, we take as a continuous differentiable function the cross-entropy loss function (f in Equation
1). In Appendix A.1 we show that the influence calculated in this way surfaces documents that have
a causal effect on the accuracy of a 7B model fine-tuned to do reasoning and reading comprehen-
sion tasks. Namely, if we remove documents from the fine-tuning data according to their influence
and re-train the model, the accuracy drops significantly more than if we take out the same amount
of documents randomly, or the same amount of documents using gradient similarity. In the same
experiments, we motivate the use of EK-FAC estimations of the Hessian, by showing it significantly
improves over a method using only first-order information.

1The true response function to derive influence functions for unconverged parameters is the Proximal Breg-

man function, but this step is left out here. The reader is referred to a derivation in Grosse et al. (2023).

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Table 1: Example from the reasoning set that
involves simple two-step arithmetic.

Table 2: Example from the factual set that
requires retrieving the right answer.

Reasoning query set (arithmetic)

Factual query set

Prompt

Calculate the answer: (7 - 4) * 7
Think step-by-step.

Completion (by 7B)

First, let’s evaluate the expression
inside the parentheses:
7 - 4 = 3
Now, let’s multiply the result by 7:
3 * 7 = 21
Therefore, the answer to the expression is 21.

Prompt

What is the tallest mountain in the world
and how tall is it?

Completion (by 7B)

The tallest mountain in the world is Mount
Everest, which is located in the Himalayas.
It is 29,029 feet tall.

It is only reasonably possible to loop over the pretraining data sample once, and to store more than
a single query gradient in memory (which has the same memory complexity as the model itself),
Grosse et al. (2023) use singular-value decomposition (SVD). Instead of SVD, we use approximate
SVD with a probabilistic algorithm (Halko et al., 2011), which significantly speeds up the compu-
tation of the query gradients. We justify each approximation we do in Appendix A.2.1.

We approximate Equation 1 to get scores for documents from the pretraining data D that represent
the influence they have on a completion yc given a prompt yp. Given the counterfactual question
approximated by the response function, an influence score of 1 implies the log-probability of the
sequence yc is increased by 1 (Grosse et al., 2023). To compare influence scores across different
completions (and token lengths), we normalise the scores for each query by the information content
of its completion yc, measured in nats. The information content of a query is defined as I(yc) =
− log (pθu (yc | yp)). The influence scores induce a ranking over documents from most positively
to most negatively influential, where a score can be interpreted as the increase (or decrease) in log-
probability per nat of query information. The pipeline is shown in Figure 6 in the Appendix.

4 EXPERIMENTAL SETUP

Query set. We collect a query set with different types of questions, of which 40 are reasoning
questions and 40 factual questions. Note that it is only tractable to loop over the pretraining sample
we look at once, so we need to be able to store all query gradients in memory and cannot go beyond
about 80 questions. For the reasoning questions, we identify two types of mathematical reasoning
each model can do robustly with zero-shot chain-of-thought (Wei et al., 2022). We do this by
evaluating the models on larger sets of 100 questions for each type of reasoning, and selecting tasks
where it gets at least 80% correct. This surfaces simple two-step arithmetic for the 7B model (Table
1), calculating the slope between two numbers for both models (of which two redacted examples
are shown in Figure 1), and solving for x in linear equations for the 35B model (see Appendix A.3
for prompt-completion pairs of the latter two reasoning types). We ensure no query ever requires
outputting a fraction. To make the results between 7B and 35B more comparable, we use the same
slope questions for both models. For the 40 factual questions, we make sure the model gets half right
and half wrong, allowing us to identify failures of retrieving facts from parametric knowledge, and
also ensure 16 of 40 overlap between models. We calculate influence over the full completion, which
includes the chain-of-thought in the reasoning case. The query sets are provided in the supplement.

Documents set. We want to compare the influence of pretraining data on reasoning by differently
sized models (7B and 35B), so we select two models that are trained on the same data. The EK-FAC
estimation of the Hessian only needs to be done once per model, but the other terms in Equation 1
require a forward- and backward-pass through the model per document-query pair. This means that
obtaining a ranking over pretraining data for a single query has a computational complexity similar
to pretraining itself. To overcome this issue, we sample a set of documents from the pretraining data
that covers multiple examples from each batch seen during pretraining, giving a total of 5 million

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

documents (approximately 2.5B tokens) distributed similary as the training distribution. We batch
queries and obtain the influence scores in parallel. Each document contains 512 tokens.2

EK-FAC estimation. To estimate the Hessian for the 7B and 35B models (the expectations from
Section 3), we randomly sample 100 000 documents equally spread-out through pretraining for both
models. Details on how exactly we approximate the Hessian are in Appendix A.2. We note here that
although this aspect of the pipeline requires estimating over 300B parameters representing second-
order information, the bottleneck remains calculating document gradients.

Models. We look at two models of different sizes, 7B and 35B, which are base and supervised
fine-tuned versions of Cohere’s Command R series.3 We estimate the second order information
and calculate document gradients using the base models, and generate completions and calculate
the query gradients using the models fine-tuned with supervised instruction-tuning. The reason for
choosing this setup is that the fine-tuned models are much better at instruction following. This means
we are assuming the EK-FAC for the fine-tuning phase is the identity (Bae et al., 2024), and we are
focusing only on the influence of the pretraining data and ignoring the fine-tuning data.

5 EXPERIMENTS AND RESULTS

We compare the rankings (from most positively to most negatively influential) over pretraining data
produced by influence functions for reasoning questions to the rankings for factual questions (which
can only be answered by retrieving parametric knowledge). We first analyse the rankings quanti-
tatively by looking at the influence of different documents per nat of generated query information
(Section 5.1). We aim to elucidate how generalisable the information in the influential documents
is, and how many documents the model is relying on when doing reasoning compared to retrieval.
Then, in Section 5.2 we investigate how the documents relate to the queries qualitatively.

5.1 QUANTITATIVE ANALYSIS

Finding 1: There is a significant positive correlation between the influence scores of documents
for queries with the same underlying reasoning task, indicating that these documents are relevant
for questions requiring the same procedure applied to different numbers. If models are relying
on documents that contain ‘general’ knowledge that is applicable to any query with the same task
(e.g. queries that require finding the slope between two points for many different points), we would
expect there to be a significant correlation in the influence scores for these queries. We calculate
the Pearson’s R correlation of all 5 million document scores for all query combinations (leading
to 802 correlations). The results can be seen in the right panel of Figure 1 for a subsample of 10
queries per task, and all query correlations can be found in Figure 12 in Appendix A.9.1. We find
a strongly significant (p-values all below 4e − 8) positive correlation between many queries of the
same reasoning type, and a strongly significant absence of correlation (p-values all around 4e−3) for
most (but not all) factual queries or other combinations (e.g. reasoning queries of different types).
This means that many documents have a similar influence on the same type of reasoning. Given
that each type of reasoning query requires applying the same procedure to different numbers, the
positive correlation indicates that the influence scores for reasoning queries pick up on procedural
knowledge. The correlations are strongest for the slope queries by the 35B model, and this is also
the type of reasoning the model can do most robustly compared to solving linear equations. For
the model to be able to solve linear equations with an accuracy of more than 80%, we restrict the
calculations to lead to positive x, whereas for the slopes questions the answers can be positive or
negative. In Appendix A.9.1 we falsify the hypothesis that the correlations are caused by the fact that
the reasoning questions are superficially similar to each other, by using a set of control queries that
are also superficially similar but do not require any reasoning and repeating the entire experiment.
For the control queries we mostly do not observe a correlation. In Appendix A.9.1 we highlight
examples of queries with high or low correlation for different query sets, finding that some of the
correlation seems driven by formatting of reasoning steps, and most by reasoning procedure.

Finding 2: When reasoning, the model on average relies on each individual document less per
generated nat of information than when answering factual questions, and the total magnitude of

2We choose 512 tokens because qualitatively interpreting more is hard (usually spanning multiple topics).
3https://cohere.com/command

6

Under review as a conference paper at ICLR 2025

influence is much less volatile, indicating it is generalising from a more general set of documents.
The effect is more pronounced for the larger model.
In Figure 2 we show the total influence for different percentiles of the positive parts of the rankings.

Figure 2: The total influence per nat of query completion information for different portions of the
positive ranking over documents, left for the 7B model, right for the 35B. The total influence per nat
is usually lower for reasoning questions than for factual questions, and the influence per document
varies more for factual questions than for reasoning questions, especially for the 35B model.

The results depict the total amount of influence contained in the top-k percentile of the positively
ranked documents: e.g. the 20th percentile contains 20% of the positive documents for a query,
and the amount of total influence shown is the sum of all document influences up to that part of
the ranking. The equivalent for the negative portions looks similar (Figure 15 in Appendix A.9.2)
and the discussion below applies similarly to the negative ranking. We observe two things for both
models. Firstly, the amount of total influence for most factual questions at any part of the ranking is
higher than for reasoning questions. Secondly, there is more variation in the influence of documents
at the same rank across different factual queries (and for a few factual queries the amount of influence
is actually lower than for the reasoning queries, seen more clearly in Figure 20 in Appendix A.9.3).
The first result means that, on average, the models rely on individual documents within our set less
for generating reasoning traces than for answering factual questions. The second result indicates that
for the factual questions the model relies on more ‘specific’ and infrequent documents: for a factual
question it is more up to chance whether relatively highly influential documents (w.r.t. influence of
documents for other factual questions) are part of the pretraining sample or not.

Influence spread. Another way to analyse the magnitude of influence is to look at the dispersion of
influence across the ranking: how much of total influence for each query is contained at the top and
bottom parts of the ranking? Similarly to what Grosse et al. (2023) report, we observe that the top
parts of the rankings over documents follow a power law characterised by a linear relation between
rank and influence per nat in log-log space (shown in Figure 20 in Appendix A.9.3). We find that the
slopes for the reasoning questions by the 35B are slightly steeper than for the factual questions, and
therefore the percentage of positive influence contained in the top portions of the rankings for the
35B reasoning questions increases faster with rank than for the factual questions (shown in Figure
22 in Appendix A.9.3). For the 7B, the slopes for the reasoning questions the model gets right are
on average also a bit steeper than for the factual questions, but the effect goes away when comparing
slopes for all factual vs. reasoning queries. This means that the percentage of the total positive
influence the top sequences cover is higher for the reasoning questions than for the factual questions
for the 35B model (again, similarly for the bottom sequences, see Figure 15). There is a chance
this finding is caused by noise for the 35B model and we discuss this possibility more in Appendix
A.9.3, where we note that for the reasoning query with the steepest power law, the top 1 document
is qualitatively entirely unrelated to the prompt.

If we compare the result between models, we find that the difference in magnitude and volatility are
more pronounced for the 35B model across the full rankings. We look into this in Appendix A.9.2,
and find that the effect remains even if we only look at queries that are the same for both models,
which points to higher data efficiency for the larger model.

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

5.2 QUALITATIVE ANALYSIS

We perform three types of qualitative analyses on the highest ranked portions for each query; we
search for the answer, we characterise the documents’ relation to the reasoning queries, and we
investigate what source datasets they are from (e.g. code, arXiv, Wikipedia, etc., for both the top and
bottom parts of the ranking). To filter out some of the noisy documents from the rankings, we divide
the influence scores by the document gradient norm, and re-rank them, which has empirically been
found to help (Choe et al., 2024).

Finding 3: The answer to the factual questions shows up relatively often in the top influ-
ential documents for the factual questions, and almost never for the reasoning questions.
To find the answer to the questions in the queries in
the top documents manually, we construct keywords
for each query that should be in the document if the
answer is there. For example, for the factual query in
Table 2, the keywords are “tallest”, “highest”, “Mount
Everest”, “29029”, “8848”. For the reasoning queries,
we construct many more keywords per query, but some
examples for the example in Table 2 are 7 − 4, 3, 21,
3∗7, as well as replacing the operations with words like
‘minus’ and ‘times’, and different ways of represent-
ing the content in this query. For details on which key-
words we use for each query, see Appendix A.4. We
determine the occurrence of each of these keywords in-
dependently in the top 100 documents for each query
(meaning even if just the keyword ‘7’ is present it
would be a hit), resulting in many false-positives. We
manually look over the hits to find the answer. On top
of that, we craft a prompt for Command R+ (a more
capable 100B model) to find the answer in a query-
document pair, and use it to find the answer in the top 500 documents for each query independent of
keyword overlap (the prompt is given in Appendix A.5). Then, we manually look over the hits and
keep track of documents that have the answer to a query. We verify that Command R+ finds all, and
more, of the answers we have identified manually. We look for the full answer in a single document.
For the reasoning queries, we also count partial answers in separate documents if they combine to
the full answer. For example, if one document contains 7 − 4 = 3, and another 3 ∗ 7 = 21, we
consider that an answer. Finally, we apply the keyword overlap search combined with prompting
Command R+ to a subset of the broader 2.5B pretraining tokens to verify that the answer to the
questions are in the entire set if they do not show up in the top 500 documents for queries.

Figure 3: We search for the answer in the
top 500 (top 0.01%) documents, and find
it relatively frequently for the factual ques-
tions. For the reasoning questions, we find
the answer twice for the 7B, and never for
the 35B. Both those times, the answers to
the steps occur in separate documents.

The results are shown in Figure 3. For the 7B model, we find the answer in the top 500 documents
for 55% of the factual queries, compared to 7.4% of the reasoning queries. For the 35B model,
the answer to the factual queries shows up in the top influential documents 30% of the time, and
never for the reasoning set. We expect the answer shows up less frequently for the 35B model
simply because the factual questions are much more ‘niche’. For example, one of the questions the
model gets correct is “In which year did the Beinecke Library open?”. Moreover, in certain cases,
the answer shows up multiple times in the top 500 documents. If we count all these separately, as
opposed to a binary ‘yes’ or ‘no’ per query on which the results in Figure 3 are based, answers to
questions show up 30 times for the factual questions in the 7B rankings, and twice for the reasoning
questions. For the 35B, the same result is 15 times for the factual questions, and never for the
reasoning questions. Interestingly, the answer to the factual questions often shows up in different
languages, like Spanish or Portuguese. We give two examples in Appendix A.8.2. One explanation
for the answer to the reasoning questions not showing up, is that they are not present in the larger set
of 5 million documents. This could be the case for the slope and linear equation questions, which
also means they are infrequent in the larger pretraining set, but for the arithmetic questions this is not
the case. By repeating the above process over a random subset of 5 million documents, we identify
answers to reasoning steps in documents that do not show up in the top 500 documents for 13 of 20
arithmetic queries and a full answer for 1 of 20, and expect more to be there that elude the keyword
search. For the slopes and linear equation queries, we find answers to 3 reasoning steps which do
not show up in the top 0.01%. In Appendix A.8.1 we show some of these documents and their ranks.

8

Under review as a conference paper at ICLR 2025

Finding 4: We find that influential documents for the reasoning queries are often doing a similar
form of step-by-step reasoning, e.g. also arithmetic. Further, we find that the influential docu-
ments often implement a solution to reasoning questions in code or general math.
For the slope queries (of which we have 20 which are the same for both models), many different
documents surface as highly influential that show how to calculate the slope between two points in
code or math. For the 7B model, documents that present procedural knowledge on how to calculate
the slope in either code or math show up in the top 100 documents for 16/20 queries, and for the
35B model they show up for all queries. All together, we manually find 11 unique documents that
implement the slope in code in the top 100 documents, and 16 that present equations for calculating
the slope. The 7B model relies on 18 of these documents for its completions (meaning 18 different
ones appear in the top 100 documents for all queries), and the 35B on 9. An example of a highly
influential document implementing the solution in JavaScript (left) and in maths (right):

Positively influential code

Positively influential math

function eqOfLine(x1, y1, x2, y2) {

if (x1 === x2) {

// Handle a vertical line
return ‘x = ${x1}‘;

} else {

// Calculate the slope
const m = (y2 - y1) / (x2 - x1);
const b = y1 - m * x1;
// Return y = mx + b
return ‘y = ${m}x + ${b}‘;

}

}

a

straight

If
points
passing
P (x1, y1), Q(x2, y2) is making an angle θ with the
positive X-axis, then the slope of the straight line is:

through

line

the

(A)

(B) θ
(C)

y2+y1
x2+x1

y2−y1
x2−x1

(D) sin θ

Solution:
Correct answer: (C)

We prompt Command R+ to further characterise the top 500 documents for each query by choosing
from a set of provided keywords, and find that often the documents are doing similar arithmetic on
other numbers (e.g. much larger or smaller), doing similar arithmetic on similar numbers (for the
slope questions), or similar algebraic operations on similar numbers (for solving linear equations).
We present the detailed results and prompt for this analysis in Appendix A.8.

the source datasets of

Finding 5: For factual queries, the most influential data sources include Wikipedia and trivia,
while for reasoning, key sources consist of maths, StackExchange, ArXiv, and code.
We look at the type of source datasets that represent the most influential documents. Specif-
the top and bottom k documents with k ∈
ically, we count
{50, 500, 5000, 50000, 500000}, and compare the count to the pretraining distribution. We present
the details in Appendix A.8.4, but mention here that code data is highly influential for reasoning.
StackExchange as a source has ten times more influential data in the top portions of the rankings than
expected if the influential data was randomly sampled from the pretraining distribution. Other code
sources are twice as influential as expected when drawing randomly from the pretraining distribution
for k = 50 up to k = 50000. Similar patterns hold for the bottom portions of the rankings.

6 DISCUSSION, LIMITATIONS, AND FUTURE WORK

In this work, we investigate what kind of generalisation strategy two LLMs (7B and 35B respec-
tively) employ when reasoning, and contrast it to the strategy used for a task that requires retrieving
factual parametric knowledge. By creating rankings for 200 such questions over 5 million pretrain-
ing documents based on their influence on the likelihood of the completions, we conclude that the
generalisation strategy for reasoning is unlike retrieval. More often than not, even if the answer is
part of the set of pretraining documents we look at, it does not show up as highly influential as the
answers to factual questions do. We find that instead, the positively influential documents often con-
tain procedural knowledge on how to get to a solution. Further, the models rely less on individual
documents when reasoning than when answering factual questions, and the set of documents they
rely on is more general. Finally, documents often have a similar influence on reasoning queries that
require applying the same procedure to different numbers. These findings can inform pretraining
data selection for more robust reasoning: we likely do not need to cover every case in pretraining
but can rather focus on data describing and applying procedures to diverse reasoning problems.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

We find that the distribution of influence is less spread out for reasoning than for factual questions,
characterised by steeper power laws. The distribution of influence over documents tells us something
about the type of generalisation strategy the model is using; the more documents that contribute to
each nat of query information (i.e. the more spread out the total influence), the more documents the
model is relying on to produce the completion. One would perhaps expect a steeper power law for
factual questions than for reasoning (meaning more of the total positive influence contained at the
top parts of the ranking), but our results show evidence for the opposite. Perhaps a model needs to
generalise from a broader set of documents for factual retrieval than for reasoning because it needs
to see the same information more often to memorise it. This is supported by the finding that for
factual questions the answer often shows up multiple times in the top 0.01% most influential data.

There are important limitations to our approach, most notably that we do not calculate influence
on the entire training set, which is intractable. An alternative explanation of our results is then the
opposite conclusion: the model is relying on data for reasoning that are so infrequent that a random
sample of 2.5B tokens does not surface relatively highly influential samples for any of the 60 reason-
ing queries. This would result in the conclusion that LLMs rely on sparse and infrequent documents
for reasoning. That means we are effectively looking at a set of relatively uninfluential documents
for reasoning, and that perhaps the answers to reasoning traces would be highly influential when
looking at the entire pretraining data. We would argue that this is the more unlikely explanation for
three reasons: (1) the qualitative analysis shows that the influential data for the reasoning questions
are intuitively highly relevant, and that the answers to many reasoning traces are part of the 2.5B to-
kens, they are just not highly influential for reasoning, (2) the correlation of influence scores for the
different reasoning tasks is highly significant, and (3) we confirm that these results do not hold for
contol queries that look similar to the reasoning queries superficially, but do not require step-by-step
reasoning. Moreover, it seems exceedingly unlikely that the model is learning to do retrieval from
such infrequent data for one of the simplest forms of mathematical reasoning, namely subtraction
and multiplication on small numbers. Taken together we argue the results indicate a generalisation
strategy that relies on procedural knowledge. Regardless, the nature of interpretability research such
as the work presented here is that all we can do is provide evidence, and not proof.

Another limitation is that we do not look at the supervised fine-tuning stage. The reason we only
look at the pretraining data is because the SFT stage is targeted at making the models more aligned
and ‘instructable’, as opposed to teaching the model capabilities. Prior work has shown that it does
not teach the model new capabilities, but rather enhances existing ones (Jain et al., 2024; Kotha
et al., 2024; Prakash et al., 2024). Nonetheless, an interesting direction for future work is applying
the same method used here to the fine-tuning data.

This work spurs further avenues for future work. Firstly, identifying a type of data that is influential
in similar ways across reasoning types would be very interesting. For example, we find that code
is equally positively as negatively influential for reasoning, can we use this insight to filter out
negatively influential code? Further, since we only take into account the MLP parameters and treat
the attention as fixed, an interesting avenue for future work would be to investigate how the relatively
low magnitude of influence of pretraining data on MLP parameters for reasoning traces interacts
with attention, connecting to a finding from literature that certain forms of reasoning happen in the
attention heads (Olsson et al., 2022). Finally, in this work we look at mathematical reasoning, which
is very different from other types of reasoning, especially if they are inductive. Future would should
verify whether similar results hold for more types of reasoning.

With this work, we do not claim to say contamination is not an issue, or that LLM reasoning is not
brittle and reliant on pretraining statistics. All we showed is that in principle it seems to be possible
for LLMs to produce reasoning traces using a generalisation strategy that combines information
from procedurally related documents, as opposed to doing a form of retrieval. This is not to say
that there are no cases of LLM reasoning where the model is in fact doing retrieval, on the contrary,
models can be overfit to contaminated data if it appears often enough in the training data.

REPRODUCIBILITY STATEMENT

Although this work is based on proprietary models and pretraining data, we make the following
efforts for reproducibility. For one of the models we use (the 35B model), the final-stage model

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

(further trained after SFT) is publicly available on HuggingFace.4 We provide all queries, com-
pletions, and keywords in the supplemental material. Additionally, we verify that the influence
scores generated with our internal codebase correlate with a Pearson’s R of more than 0.99 with a
public implementation of EK-FAC influence functions (see Appendix A.2.2). Further, we provide
details on hyperparameters for every experiment we have done at the relevant sections, as well as the
prompts used to find answers to the reasoning questions and characterise the relationship between
the query-document pairs (Appendix A.5 and A.6 respectively).

REFERENCES

Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh
Fadaee, Ahmet ¨Ust¨un, and Sara Hooker. To code, or not to code? exploring impact of code in
pre-training, 2024. URL https://arxiv.org/abs/2408.10914.

Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate

unrolled differentiation, 2024. URL https://arxiv.org/abs/2405.12186.

Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying ex-
planatory training samples via relative influence. In Silvia Chiappa and Roberto Calandra (eds.),
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics,
volume 108 of Proceedings of Machine Learning Research, pp. 1899–1909. PMLR, 26–28 Aug
2020. URL https://proceedings.mlr.press/v108/barshan20a.html.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con-
erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu,
Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex
Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter,
Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language
models with dictionary learning.
https://transformer-
circuits.pub/2023/monosemantic-features/index.html.

Transformer Circuits Thread, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari-
wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar-
wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,
Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz
Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
In
Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners.
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu-
ral Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2020/
2020.
file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya
Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy,
Roger Grosse, and Eric Xing. What is your data worth to gpt? llm-scale data valuation with
influence functions, 2024. URL https://arxiv.org/abs/2405.13954.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh,
Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam
Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James
Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev-
skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin
Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret
Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick,
Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica
Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren-
nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas

4https://huggingface.co/CohereForAI/c4ai-command-r-v01

11

Under review as a conference paper at ICLR 2025

Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways,
2022. URL https://arxiv.org/abs/2204.02311.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John
Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,
2021.

Dask Development Team. Dask: Library for dynamic task scheduling, 2016. URL http://

dask.pydata.org.

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Benchmark
In NeurIPS 2023 Workshop
probing:
on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2024. URL https:
//openreview.net/forum?id=a34bgvner1.

Investigating data leakage in large language models.

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In
Proc. of NAACL, 2019.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna
Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse,
Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.
https://transformer-
Toy models of superposition.
circuits.pub/2022/toymodel/index.html.

Transformer Circuits Thread, 2022.

Thomas George, C´esar Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent.

Fast
In S. Ben-
(eds.),
Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2018/

approximate natural gradient descent
gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett
Advances in Neural Information Processing Systems, volume 31. Curran Associates,
2018.
file/48000647b315f6f00f913caa757a70b3-Paper.pdf.

factored eigenbasis.

in a kronecker

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit
Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil˙e Lukoˇsi¯ut˙e, Karina Nguyen,
Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language
model generalization with influence functions, 2023. URL https://arxiv.org/abs/2308.
03296.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth
Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah,
Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee,
and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv.org/abs/2306.
11644.

N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algo-
rithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217–288, 2011.
doi: 10.1137/090771806. URL https://doi.org/10.1137/090771806.

Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statis-
tical Association, 69(346):383–393, 1974. doi: 10.1080/01621459.1974.10482962. URL https:
//www.tandfonline.com/doi/abs/10.1080/01621459.1974.10482962.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob
Steinhardt. Measuring massive multitask language understanding. Proceedings of the International
Conference on Learning Representations (ICLR), 2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas
Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au-
relia Guy, Simon Osindero, Kar´en Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Lau-
In
rent Sifre. An empirical analysis of compute-optimal

large language model

training.

12

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances
in Neural Information Processing Systems, volume 35, pp. 30016–30030. Curran Associates,
Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/
file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf.

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rockt¨aschel,
Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on
procedurally defined tasks. In The Twelfth International Conference on Learning Representations,
2024. URL https://openreview.net/forum?id=A0HKeKl4Nl.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott
Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
URL https://arxiv.org/abs/2001.08361.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.

In International

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.

In
Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp.
1885–1894. JMLR.org, 2017.

Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting
in language models via implicit inference. In The Twelfth International Conference on Learning
Representations, 2024. URL https://openreview.net/forum?id=VrHiF2hsrm.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding
comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel
(eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
doi: 10.18653/v1/D17-1082. URL https://aclanthology.org/D17-1082.

Kyle Mahowald, Anna Ivanova, Idan Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fe-
dorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences,
28, 03 2024. doi: 10.1016/j.tics.2024.01.011.

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of
autoregression: Understanding large language models through the problem they are trained to solve,
2023. URL https://arxiv.org/abs/2309.13638.

Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya
Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein. Transformers
can do arithmetic with the right embeddings, 2024. URL https://arxiv.org/abs/2405.
17399.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture mod-

els, 2016.

Melanie Mitchell and David C. Krakauer. The debate over understanding in ai’s large language models.
Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023. doi: 10.1073/pnas.
2215907120. URL https://www.pnas.org/doi/abs/10.1073/pnas.2215907120.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan,
Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Gan-
guli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane
Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish,
In-context learning and induction heads. Transformer Circuits Thread, 2022.
and Chris Olah.
https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning
enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Con-
ference on Learning Representations, 2024. URL https://openreview.net/forum?id=
8sKcAWOf2D.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influ-
ence by tracing gradient descent. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin
(eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19920–19930. Cur-
ran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/
paper/2020/file/e6385d39ec9394f2f3a354d9d2b88eec-Paper.pdf.

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh.

Impact of pretraining
term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue
Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840–
854, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis-
tics. doi: 10.18653/v1/2022.findings-emnlp.59. URL https://aclanthology.org/2022.
findings-emnlp.59.

Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, and Andrew M Saxe. What needs
to go right for an induction head? a mechanistic study of in-context learning circuits and their
In Forty-first International Conference on Machine Learning, 2024. URL https:
formation.
//openreview.net/forum?id=O8rrXl71D5.

Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam
Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner,
Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees,
Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman-
ticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024.
URL https://transformer-circuits.pub/2024/scaling-monosemanticity/
index.html.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris-
tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,
Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey
Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL
https://arxiv.org/abs/2307.09288.

Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks, 2023. URL

https://arxiv.org/abs/2302.08399.

Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A
mechanistic journey to the edge of generalization, 2024. URL https://arxiv.org/abs/
2405.15071.

Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language

models. Nature Human Behaviour, 7:1–16, 07 2023. doi: 10.1038/s41562-023-01659-w.

Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Evidence from counterfactual tasks supports emer-
gent analogical reasoning in large language models, 2024. URL https://arxiv.org/abs/
2404.13070.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V
Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In
Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu-
ral Information Processing Systems, 2022. URL https://openreview.net/forum?id=
_VjQlMeSB_J.

Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky¨urek, Boyuan Chen, Bailin Wang, Najoung Kim,
Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard
(eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1819–
1862, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/
v1/2024.naacl-long.102. URL https://aclanthology.org/2024.naacl-long.102.

Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking
benchmark and contamination for language models with rephrased samples, 2023. URL https:
//arxiv.org/abs/2311.04850.

A APPENDIX

Below we outline the contents of the appendix.
Motivating the use of EK-FAC influence functions. In Appendix A.1 we discuss the counterfac-
tual re-training experiments that motivate our use of EK-FAC influence functions for estimating the
effect of pretraining data on the accuracy of downstream behaviour. We describe in more detail how
we use influence functions at scale in Appendix A.2, documenting how we estimate the Hessian,
how we store many query gradients in memory (each having the same memory complexity as the
entire model), and how we sample from the pretraining distribution.
Query sets examples. Then, in Appendix A.3, we show examples of the reasoning sets that we did
not show examples for in the main body of this manuscript.
Finding query answers in documents. In Appendix A.4 we discuss how we create keywords for
each query in order to find the answer in the top documents, and in the sections directly after that,
Appendix A.5 and A.6, we give the prompts we used to allow Command R+ to search for answers
in the top 500 documents for each query, as well as characterise their relationship.
Limitations. In Appendix A.7 we discuss limitations specific to influence functions.
Additional qualitative results. In Appendix A.8 we provide additional qualitative results.
Answers in pretraining data. We give some examples of answers to factual questions and reasoning
questions in pretraining documents in Appendix A.8.1 Cross-lingual transfer. We give some
examples of cross-lingual transfer in Appendix A.8.2.
Source-dataset analysis. We analyse which datasets the influential data comes from in Appendix
A.8.4.
Content analysis of relevant documents. We classify data from the source dataset code for whether
it actually contains code in Appendix A.8.5.
Additional quantitative results. Correlation analysis. Further results for the correlation analysis
of influence scores for documents for different queries in Appendix A.9.1.
In Appendix A.9 we provide additional quantitative results.
Magnitude of influence. Further results for the magnitude of influence in Appendix A.9.2.
Spread of influence. Further results for the spread of influence over the rankings in Appendix A.9.3.

A.1 COUNTERFACTUAL RE-TRAINING EXPERIMENTS WITH INFLUENCE FUNCTIONS

We use EK-FAC influence functions to approximate the counterfactual question: which documents
from pretraining have a causal effect on the completions of a trained model. However, we are also
interested in the causal effect on the accuracy of the completions. In this section, we aim to motivate
two aspects of this choice; the fact that influence functions are designed to estimate the effect on
continuous differentiable functions, like the log-likelihood, and not on the accuracy. Secondly, we
motivate the need for estimating the second-order information of the pretraining objective using
EK-FAC, which is very computationally expensive. We present four different experiments in this
section, which show that indeed the influence of documents as determined by influence functions
also estimate the effect on downstream task accuracy, as well as the benefits from estimating second
order information over simply using first-order gradient information.

The pipeline for each of these experiments is similar; we take a pretrained model, we fine-tune it on
some dataset, and evaluate it on 50 validation examples with a metric (perplexity or accuracy). We
then use the fine-tuned weights to calculate the influence of the documents in the dataset used for
fine-tuning on the set of 50 validation questions with two methods: EK-FAC influence functions and

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

TracIn (Pruthi et al., 2020). Subsequently, we use those two methods to remove the k most positively
influential documents from the fine-tuning dataset, as well as randomly selecting k documents as a
baseline, and fine-tune the original pretrained model five times on each new fine-tuning dataset
created (for different values for k). We then calculate the perplexity or accuracy on the validation
questions used to calculate the influence, and see how it changed. The more it changed, the more
the documents indeed influence the relevant metric (i.e. perplexity or accuracy). Note that for n
different values for k, this requires fine-tuning 3 ∗ 5 ∗ n models, five times for each of the three
methods of removing documents from the training set.

We start by motivating the use of EK-FAC influence functions over simple similarity information
between document and query gradients. In our setup, where we only have access to the final check-
point of pretraining, a dot-product between the query and document gradient effectively boils down
to a method for estimating influence of documents on queries called TracIn (Pruthi et al., 2020).
With access to multiple checkpoints, TracIn uses gradient information from all of them, account-
ing for the learning rate used at that point in training. However, we only use the final checkpoint
and hence taking into account learning rate only changes scores by a constant. We take GPT-2-
small (124M) from HuggingFace,5 and fine-tune it for three epochs with next-word prediction on
Wikitext-2 (Merity et al., 2016). We use Adam optimizer (Kingma & Ba, 2015) with default param-
eters (b1 0.9, b2 0.999, eps 1e-8, additive weight decay 0.01). The results can be found in Figure
4 and Table 3, showing that removing documents using EK-FAC influence functions has a signifi-
cantly larger effect on downstream perplexity for all values of k. We do the exact same experiment
but instead remove the most negatively influential documents, and see that instead the perplexity
decreases significantly more for EK-FAC influence functions (Figure 4 and Table 4).

Table 3: Wikitext remove top influential

k →

50

100

150

200

250

300

Random
TracIn
IF (ours)

22.09 ± 0.02
22.16 ± 0.02⋆⋆
22.49 ± 0.02⋆⋆

22.12 ± 0.02
22.22 ± 0.02⋆⋆
22.66 ± 0.02⋆⋆

22.10 ± 0.02
22.25 ± 0.01⋆⋆
22.73 ± 0.02⋆⋆

22.20 ± 0.06
22.35 ± 0.03⋆⋆
22.88 ± 0.01⋆⋆

22.19 ± 0.05
22.42 ± 0.01⋆⋆
22.97 ± 0.02⋆⋆

22.15 ± 0.05
22.45 ± 0.02⋆⋆
23.05 ± 0.05⋆⋆

Table 4: Wikitext remove bottom influential

k →

50

100

150

200

250

300

Random
TracIn
IF (ours)

27.40 ± 0.08
26.73 ± 0.04⋆⋆
25.96 ± 0.04⋆⋆

26.24 ± 0.10
25.48 ± 0.05⋆⋆
24.78 ± 0.05⋆⋆

25.62 ± 0.15
24.86 ± 0.02⋆⋆
23.95 ± 0.03⋆⋆

25.22 ± 0.10
24.36 ± 0.04⋆⋆
23.52 ± 0.03⋆⋆

25.04 ± 0.12
24.16 ± 0.05⋆⋆
23.46 ± 0.03⋆⋆

24.85 ± 0.10
23.94 ± 0.03⋆⋆
23.32 ± 0.04⋆⋆

Next, we turn to motivating the use of EK-FAC influence functions in estimating the effect of docu-
ments on downstream accuracy of model generations. To this end, we look at two different datasets:
DROP (Dua et al., 2019) and RACE (Lai et al., 2017). DROP is a reading comprehension dataset re-
quiring different skills like subtraction, addition, coreference resolution, counting, and other skills.
The model needs to generate an answer that often consists of one or a few words. We allow the
fine-tuned models to generate answers to the questions freely, and evaluate based on exact match. In
this experiment, we use a 7B model. We randomly select a subset of 8000 examples for fine-tuning,
and use the procedure described above to perform counterfactual experiments. We use Adam op-
timizer again, with the same hyperparameters as for the above experiment: b1 0.9, b2 0.999, eps
1e-8, additive weight decay 0.01, but only train for one epoch. The results can be found in the left
panel of Figure 5 as well as in Table 5. We find that EK-FAC influence functions are succesful in
selecting data points that impact downstream accuracy, much more so than randomly removing the
same amount of training data. For most k (all but k = 1000), EK-FAC influence functions also have
a significantly stronger effect on accuracy than TracIn, but the difference is less large. We apply the
exact same procedure to the RACE dataset, except now we keep 10k examples (empirically found
to lead to the least overfitting when fine-tuning). Further, RACE is a multiple-choice dataset, so we
allow the model to generate a single token indicating the choice, and calculate the accuracy. The
results can be seen in Figure 5 and Table 6. Again, the finding is similar; EK-FAC influence func-
tions surface documents that have a stronger effect on accuracy than TracIn for all but one value

5https://huggingface.co/

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

(a)

(b)

Figure 4: (a) Counterfactual retraining experiments on Wikitext-2. We finetuned GPT-2 (124M)
on Wikitext-2 and use three different methods to remove training examples from the training set:
randomly, TracIn, and Influence Functions (IF). For each number of samples removed we finetune
the base model five times with different training data ordering, the variance over these runs is repre-
sented by the error bars. Each point on the plot is the average perplexity achieved by the five models
after fine-tuning on the augmented dataset. We find that influence functions can find examples that
impact the perplexity significantly more than baselines. (b) We repeat the same experiment as in (a),
but retain top influential queries instead (removing most negatively influential).

of k, and for all values of k than randomly removing documents. There is a large variance in the
results for all methods though, which we attribute to the fact that the model sometimes seems to
overfit to the fine-tuning data. Further, the reason why the difference between TracIn and EK-FAC
influence functions is much larger in the perplexity experiments than in the accuracy experiments
could be attributed to the fact that we only fine-tune for one epoch in the accuracy experiments (as
more cause overfitting). EK-FAC influence functions differ from TracIn in that they estimate second
order information, which becomes more important with more training steps. An interesting avenue
for future work is to do counterfactual re-training experiments like these on a subset of pretraining
data for a 7B model, but this is incredibly computationally expensive.

Table 5: Counterfactual re-training accuracies on DROP (free generation of answers). We use three
different methods (random, TracIn, influence functions) to remove k datapoints, and re-train a model
on the resulting dataset. Each number is the mean over five re-training runs with different data
ordering. ⋆ indicates significantly lower than random with a p-value below 0.1 and ⋆⋆ with a p-
value below 0.05. The underlined means are the lowest.

k →

500

1000

1500

2000

Random 0.61 ± 0.05
0.55 ± 0.03⋆
TracIn
0.51 ± 0.03⋆⋆
IF (ours)

0.60 ± 0.03
0.49 ± 0.02⋆⋆
0.50 ± 0.04⋆⋆

0.56 ± 0.05
0.44 ± 0.04⋆⋆
0.40 ± 0.05⋆⋆

0.57 ± 0.06
0.43 ± 0.06⋆⋆
0.38 ± 0.05⋆⋆

Table 6: Counterfactual re-training accuracies on RACE (multiple-choice). We use three different
methods (random, TracIn, influence functions) to remove k datapoints, and re-train a model on the
resulting dataset. Each number is the mean over five re-training runs with different data ordering.
⋆ indicates significantly lower than random with a p-value below 0.1 and ⋆⋆ with a p-value below
0.05. The underlined means are the lowest.

k →

1000

1500

2000

2500

Random 0.85 ± 0.04
0.84 ± 0.01
TracIn
0.80 ± 0.04⋆
IF (ours)

0.83 ± 0.03
0.78 ± 0.03⋆⋆
0.76 ± 0.05⋆⋆

0.82 ± 0.04
0.80 ± 0.03
0.74 ± 0.04⋆⋆

0.81 ± 0.04
0.79 ± 0.04
0.74 ± 0.05⋆

17

Under review as a conference paper at ICLR 2025

(a) Counterfactual retraining experiments on read-
ing comprehension questions. We finetuned Cohere
Command 2 (7B) on a subset of the DROP training
set (8k examples) and use three different methods to
remove training examples from the training set: ran-
domly, TracIn, and Influence Functions (IF). For each
number of samples removed we finetune the base
model five times with different training data order-
ing, the variance over these runs is represented by the
error bars. Each point in the plot is the average accu-
racy achieved by the five models after fine-tuning on
the augmented dataset. We find that influence func-
tions can find examples that impact the accuracy sig-
nificantly more than baselines, although only slightly
more than TracIn.

retraining

experiments

on
(b) Counterfactual
multiple-choice reasoning data. We finetuned Cohere
Command 2 (7B) on a subset of the RACE training
set (10k examples) and use three different methods
to remove training examples from the training set:
randomly, TracIn, and Influence Functions (IF). For
each number of samples removed we finetune the
base model five times with different training data
ordering, the variance over these runs is represented
by the error bars. Each point in the plot is the
average accuracy achieved by the five models after
fine-tuning on the augmented dataset. We find that
influence functions can find examples that impact the
accuracy significantly more than baselines, although
there is some variance in the results.

Figure 5: Counterfactual retraining experiments on reading comprehension benchmark DROP (a)
and the multiple-choice reasoning dataset RACE (b).

Although the results of the experiments in this section are an encouraging sign for using EK-FAC
influence functions in estimating causal effect of data on accuracy, it is important to note that they are
limited in several ways. Accuracy is a discrete metric and it is a prior unclear how many documents
need to be removed to flip its value. However, the influence functions we use estimate effect of
removing a single document, and removing multiple documents can have additional effects that are
unaccounted for. This makes removing multiple documents a cruder way to empirically show impact
of influence functions on accuracy, but at the same time it is unavoidable. Therefore, any significant
causal effect on accuracy over other methods is a good signal, but the absence of a significant effect
does not necessarily mean EK-FAC influence functions do not properly do what they are designed
to do.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

A.2 EK-FAC INFLUENCE FUNCTIONS

The code we use for EK-FAC influence functions at scale is a part of larger internal infrastructure,
and hence cannot be released publicly. However, we base our code on the public GitHub repository
https://github.com/pomonam/kronfluence. We implement estimation of the Hessian
in the same way as in that codebase, except for a few changes to make it tractable, which we discuss
in more detail below. Further, we compare the results produced by our implementation with the
results using the public implementation. We do this by fine-tuning GPT-2 (124M) on Wikitext-2
using internal infrastructure, and calculating influence scores with both code bases. We find that
the results correlate very strongly (with a Pearson’s R of more than 0.99, see A.2.2 below for more
details). Here, we provide details of the design choices and hyperparameters used in our implementa-
tion, as well as the additional approximations to make EK-FAC estimation and influence calculation
tractable at scale.

Query-batching and approximation As mentioned in the main text, we approximate query gradi-
ents using approximate SVD (Halko et al., 2011). We use the default parameters for this algorithm,
which can be found in the Dask documentation (Dask Development Team, 2016).

Sampling from the Pretraining Data. It is intractable to calculate influence for the entire pretrain-
ing data, so we sample a set of 5 million documents. To this end, we loop over the training data as
seen by the models in order, and randomly sample 6 examples from each batch. This ensures that
the pretraining sample we use is both similar to the pretraining distribution in terms of what kind of
data the model sees, as well as when it has encountered the data during pretraining.

Estimating EK-FAC. To estimate the EK-FAC matrices, we sample 100 000 documents from pre-
training in the same manner as described above. We use the same samples to estimate the EK-FAC
for the 7B as for the 35B. For both models, we use a damping factor of 0.1 (see Grosse et al. (2023)
for details on what the damping factor is). Further, part of estimating the EK-FAC is an eigende-
composition on the EK-FAC matrices. We use the same approximation as empirically motivated
in (Grosse et al., 2023), namely block-diagonal approximation. For the 7B, we use 2 blocks, and
for the 35B, we use 4. The block-diagonal approximation is not part of the public codebase, but
simply amounts to dividing the matrices in n blocks (where n is 2 and 4 in our case), zero-ing out
the remaining entries, and taking the eigendecomposition of each block individually. After, these
blocks are patched back together again into the original size matrix, which will be further processed
as in the public codebase.

A.2.1

JUSTIFYING APPROXIMATIONS

In this section, we justify the additional approximations we do on top of those mentioned in Grosse
et al. (2023) by reporting the correlation with the full implementation for a smaller model (124M
parameters). Applying EK-FAC influence functions to models with billions of parameters requires
estimating a multiple of the model parameters. E.g., for the 7B model we estimate around 70B
EK-FAC parameters, and for the 35B model we estimate around 320B parameters. Further, to
calculate the influence scores for a set of 5 million documents we have to calculate the gradient
for 100 queries × 5 million documents, each of which has the same size as all MLP layers in
the model itself. We can only afford to loop over the 5 million documents and calculate their
gradients once, so we need to batch the query gradients in memory. This is impossible for the
full gradients and we use SVD to store low-rank approximations instead, like in Grosse et al. (2023).

Details on the experiment. To compare results of using EK-FAC influence functions with different
approximations, we use the same fine-tuned model from Section A.1 to calculate influence scores for
the 4656 training examples (i.e. documents) on the first 32 validation examples (i.e. queries) of the
Wikitext-2 dataset. We repeat this with different types of approximations applied; full SVD on the
query gradients, approximate SVD (Dask Development Team, 2016) on the query gradients, and a
block-diagonal approximation of the EK-FAC matrices before the eigendecomposition (described in
Appendix A of Grosse et al. (2023)) with 2 and 4 blocks. For each level of approximation applied,
this gives us 32 vectors with 4656 scores (one for each query-document pair), and we compare
these to the full implementation without SVD and block diagonal approximations using Pearson’s R
correlation. The correlations reported are the average over all 32 queries.

19

Under review as a conference paper at ICLR 2025

In Table 7 we report the correlations of increasingly more approximations w.r.t. a full implementa-
tion. Note that the full implementation also uses approximations, but those are all justified in Grosse
et al. (2023). Here, for completeness, we additionally justify the approximations we use that are
different, namely approximate SVD instead of full SVD, and a block-diagonal approximation with
4 blocks instead of 2. From Table 7, we can see that the approximate SVD algorithm has a neglible
effect on the scores, whereas the block-diagonal approximation has a small effect on the scores.

Approximations

SVD
Approximate SVD
Approximate SVD + block diagonal EK-FAC (2 blocks)
Approximate SVD + block diagonal EK-FAC (4 blocks)

Pearson R

0.96 ± 0.01
0.96 ± 0.01
0.95 ± 0.00
0.93 ± 0.00

Table 7: Score correlations of using increasingly more approximations with a full implementation.

A.2.2 FULL IMPLEMENTATION

We also compare the full implementation scores of our own influence functions implementation
with the scores calculated for the same model and dataset with the public implementation at
https://github.com/pomonam/kronfluence, and confirm the average score correlation
between queries is 0.993 (± 0.003). We add a direct score comparison of both methods for the top
3 documents for each of the 32 queries to the supplemental material. Specifically, for each query we
log the top 3 documents as determined by our internal implementation as well as the external imple-
mentation, showing that they are almost always the same documents, and logging the score given to
that document by each implementation (the supplemental file also contains the score correlation for
each query separately). The average number of documents that appear in both top 50’s determined
by the internal and external implementation is 46.7. The reason for using an internal implementa-
tion nonetheless is that the public implementation is not optimised for usage on large-scale models,
and cannot be used for models above about 1B parameters. We used the internal pretraining library
for implementing influence functions, because part of the infrastructure used for pretraining large
models could be re-used.

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Figure 6: The pipeline for creating rankings of the most influential pretraining documents for a
question-completion pair (query) using influence functions. The documents at the top of the ranking
influence the likelihood of the completion positively, and the bottom negatively. We create rankings
for a set of 40 reasoning and 40 factual queries over 5 million pretraining documents (2.5B tokens)
for two models of different sizes (Cohere’s Command R series, 7B and 35B).

21

Under review as a conference paper at ICLR 2025

A.3 QUERY SETS

Reasoning query sets. We show an example of the other two types of reasoning present in the
reasoning query sets in Table 8 and 9. The former requires calculating the slope of a line going
through two given points (used for both the 7B and 35B model) and the latter is about solving for x
in a linear equation (only used for the 35B model).

Control query sets. We design two control sets with 10 questions each for both the 7B and 35B
model. These query sets resemble reasoning and factual questions, but do not require actual rea-
soning or factual retrieval. The reason for constructing such query sets is to compare the results
for the factual / reasoning query sets to the corresponding control sets, and comment on whether
the patterns we are seeing also show up for queries that do not require reasoning / retrieval, which
further substantiates interpretations of the results.

Two factual control examples can be seen in Table 10 and 11, for the 7B and 35B model respectively.
They are designed to match the first factual query superficially (e.g. the same numbers), but they are
instead reading comprehension questions about fictional entities. This means all the model needs to
do to answer the question is retrieve from context (not parameters).

Two of the reasoning control examples for the 7B and 35B respectively can be seen in Table 12
and 13. Again, these resemble reasoning queries in number and style, but do not require any actual
reasoning as the answer is given in the context.

Table 8: Example from the reasoning set that involves calculating the slope of a line going through
two points. We have 20 examples like these on different points. The completion is by the 7B model,
but we use the same questions for the 35B model.

Reasoning Query Set (slopes, 7B and 35B)

Prompt

What is the slope of the line passing through the points (93, 28) and (74, 47)?

Think step-by-step.

Completion

To find the slope of the line passing through the points (93, 28) and (74, 47), we can use the following steps:

1. Identify the coordinates of the two points:
- Point A: (93, 28)
- Point B: (74, 47)

2. Calculate the difference in x-coordinates between the two points:
- Difference in x-coordinates: x2 - x1 = 74 - 93 = -19

3. Calculate the difference in y-coordinates between the two points:
- Difference in y-coordinates: y2 - y1 = 47 - 28 = 19

4. Determine the ratio of the difference in y-coordinates to the difference in x-coordinates:
- Ratio: 19 / -19 = 1 / -1

5. Multiply the ratio by -1 to reverse the direction of the line:
- Slope: -1 / 1 = -1

Therefore, the slope of the line passing through the points (93, 28) and (74, 47) is -1.

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

22

Under review as a conference paper at ICLR 2025

Table 9: Example from the reasoning set that requires solving for x in a linear equation. We have
twenty other examples like these on different numbers, only for the 35B model. The completion is
by the 35B model.

Reasoning Query Set (linear, 35B)

Prompt

Solve for x: 5x + 21 = 91

Think step-by-step.

Completion

Here are the steps to solve for x:
1. Subtract 21 from both sides of the equation:
5x + 21 - 21 = 91 - 21
5x = 70

2. Divide both sides of the equation by 5 to solve for x:
(5x) / 5 = 70 / 5
x = 14

Therefore, the solution to the equation 5x + 21 = 91 is x = 14.

Table 10: Example from the factual control set that resembles the question about the tallest mountain
in the world from the 7B factual query set (Table 2), but does not require any factual recall. The
completion is by the 7B model.

Factual Control Query Set (7B)

Prompt

The planet Zog has multiple mountains, of which the tallest one is called Wirtu which is 29,029 feet tall.
What is the largest mountain on the planet Zog called and how tall is it?

Completion

The largest mountain on the planet Zog is called Wirtu and it is 29,029 feet tall.

Table 11: Example from the factual control set that resembles the first factual query for the 35B
model superficially, but does not require any factual recall. The completion is by the 35B model.

Factual Control Query Set (35B)

Prompt

In a fictional planetary system, there is a supernova remnant and pulsar wind nebula in the centre of
a constellation of stars called Minotaur called the Goat Nebula.
What is the supernova remnant and pulsar wind nebula in the constellation Minotaur called?

Completion

In the fictional planetary system, the supernova remnant and pulsar wind nebula in the centre of
the constellation Minotaur is called the “Goat Nebula”.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

Table 12: Example from the reasoning control set that resembles the slope queries superficially, but
does not require any reasoning. The completion is by the 7B model.

Reasoning Control Query Set (7B)

Prompt

The slope of the line is -22. What is the slope of the line?

Think step-by-step.

Completion

Given that the slope of the line is -22.

To find the slope of the line, we need to find the change in y-coordinates divided by the change in x-coordinates.

The slope of the line is -22.

Therefore, the slope of the line is -22.

Table 13: Example from the reasoning control set that resembles the slope queries superficially, but
does not require any reasoning. The completion is by the 35B model.

Reasoning Control Query Set (35B)

Prompt

This line has a slope 6. What is the slope of the aforementioned line?

Think step-by-step.

Completion

The slope of a line is a measure of the steepness of the line.
It is defined as the ratio of the change in the y-coordinate to the change in the x-coordinate.

In this case, the line has a slope of 6. This means that for every 1 unit increase in the x-coordinate,
the y-coordinate will increase by 6 units.

It’s important to note that the slope of a line is not limited to positive values and can be negative or
zero depending on the orientation of the line.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

A.4 QUERY KEYWORDS FOR FINDING THE ANSWER

We provide all keywords used to search for the answers for both the 7B and 35B in the supplementary
material to this manuscript, and describe here the procedure for generating keywords, as well as
giving some examples.

For the factual questions, we generated keywords manually for each query, and these can be found
in the supplemental material. For example, for the question “What is the world’s smallest mammal
by body length?” (answer: bumblebee bat), we have the following keywords: bumblebee bat;
bumblebee; bumble; bee; bat; smallest mammal; body length; mammal; smallest; small. This
results in many false positives, e.g. if only the word ‘small’ occurs, which we all check manually for
the answer.

Based on the type of reasoning question, we programatically create keywords for each question. For
example, for the question in Table 9, the keywords are:

[’14’, ’x = 14’, ’5x + 21’, ’91’, ’5x + 21 = 91’, ’21’, ’5’,
’91 - 21’, ’91 - 21 = 70’, ’(91 - 21) / 5’, ’70 / 5’,
’70 / 5 = 14’, ’70’, ’x=14’, ’5x+21’, ’5x+21=91’, ’91-21’,
’91-21=70’, ’(91-21)/5’, ’70/5’, ’70/5=14’,
’(91 - 21) divided by 5’, ’(91-21) divided by 5’,
’(91 minus 21) divided by 5’, ’(91 min 21) divided by 5’,
’70 divided by 5’, ’70 divided by 5 = 14’,
’70 divided by 5 is 14’, ’70 / 5 is 14’, ’70/5 is 14’,
’91 - 21 is 70’, ’91-21 is 70’, ’91 minus 21 is 70’,
’91 min 21 is 70’, ’70 divided by 5 equals 14’,
’70 / 5 equals 14’, ’70/5 equals 14’, ’91 - 21 equals 70’,
’91-21 equals 70’, ’91 minus 21 equals 70’, ’91 min 21 equals 70’,
’5x plus 21’, ’5x plus 21 = 91’, ’5x plus 21 is 91’, ’5x + 21 is 91’,
’91 minus 21’, ’91 min 21’, ’91 minus 21 = 70’, ’91 min 21 = 70’,
’(91 minus 21) / 5’, ’(91 min 21) / 5’]

Note that, because the individual numbers ‘14’, ‘5’, ‘91’, and ‘70’ are part of the keywords, each
document that contains one of these numbers becomes a hit, and we go over all hits manually.

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

A.5 PROMPTS GIVEN TO COMMAND R+ FOR FINDING THE ANSWER

We use multiple prompts for each different type of reasoning question to allow Command R+ to
find the answer in the top 500 influential documents; prompts to find the answer to the intermediate
reasoning steps, and a prompt for finding the answer to the full question. We provide an example of
each below.

Preamble:

You are a brilliant AI assistant that is excellent at arithmetic designed to help users
with data analysis. You will be given an arithmetic query and a document, and
your task is to determine whether the answer to the question is in the document.

Prompt for the first step to a two-step arithmetic question

Question: 4 + 2
Answer: 4 + 2 = 6

What also counts as an answer:

- The calculation is written out in words, or part of a story.
- The order of operations are changed. E.g. 2 + 4 = 6.
- Different symbol used for sum/subtract sign. E.g. plus/minus.
- The calculation is part of another larger calculation. E.g. (4 + 2) * 9 = 6 * 9 or (4 + 2)/12 = 6/12.
- Different formatting. E.g. (4) + (2) = (6).
- The calculation is a part of an algebraic formulation. E.g. 4X + 2X = 6X.

What does not count as an answer:
- Other numbers are being summed/subtracted. E.g. 5 + 2.
- Numbers are taken to the other side of the equals sign. E.g. 6 - 2 = 4.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

Prompt for the second step to a two-step arithmetic question

Question: 6 * 15
Answer: 90

What also counts as an answer:

- The calculation is written out in words, or part of a story.
- The order of operations are changed. E.g. 15 * 6 = 90.
- Different symbol used for the multiplier sign. E.g. x or times.
- The calculation is part of another larger calculation. E.g. (6 * 15) * 9 = 90 * 9 or (6 * 15)/12 = 90/12.
- Different formatting. E.g. (6) * (15) = (90).
- The calculation is a part of an algebraic formulation. E.g. 6X * 15X = 90X.

What does not count as an answer:
- Other numbers are being multiplied. E.g. 7 * 15.
- Numbers are taken to the other side of the equals sign. E.g. 6 = 90/15.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

Prompt for step 1 (and 2 is similar) to answer a slope question

Question: 74 - 73
Answer: 74 - 73 = 1

What also counts as an answer:
- The calculation is written out in words, or part of a story.
- The calculation is written in terms of a difference or change. E.g. the difference (or change) between 73 and 74 is 1.
- The order of operations are changed. E.g. 73 - 74 = -1.
- Different symbol used for the minus sign. E.g. subtracted from.
- The calculation is part of another larger calculation. E.g. (74 - 73) * 9 = 1 * 9 or (74 - 73)/12 = 1/12.
- Different formatting. E.g. (74) - (73) = (1).
- The calculation is a part of an algebraic formulation. E.g. 74X - 73X = 1X.

What does not count as an answer:
- Other numbers are being subtracted. E.g. 75 - 73.
- Numbers are taken to the other side of the equals sign. E.g. 74 = 1 + 73.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

Prompt for step 3 to answer a slope question

Question: 74 / 1
Answer: 74 / 1 = 74

What also counts as an answer:

- The calculation is written out in words, or part of a story.
- The signs on the LHS are flipped. E.g. -74 / -1 = 74.
- Different symbol used for the division sign. E.g. divided by.
- The calculation is part of another larger calculation. E.g. (74 / 1) * 9 = 74 * 9 or (74 / 1)/12 = 74/12.
- Different formatting. E.g. (74) / (1) = (74).
- The calculation is a part of an algebraic formulation. E.g. 74X / 1 = 74X.

What does not count as an answer:
- Other numbers are being divided. E.g. 75 / 1.
- Numbers are taken to the other side of the equals sign. E.g. 74 = 74 * 1.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

27

Under review as a conference paper at ICLR 2025

Prompt for step 1 to answer a linear question

Question: 32 - 16
Answer: 16

What also counts as an answer:
- The calculation is written out in words, or part of a story.
- The calculation is written in terms of a difference or change. E.g. the difference (or change) between 32 and 16 is 16.
- The order of operations are changed. E.g. -16 + 32 = 16.
- Different representation used for the minus sign. E.g. ’subtracted from’.
- The calculation is part of another larger calculation. E.g. (32 - 16) * 9 = 16 * 9 or (32 - 16)/12 = 16/12.
- Different formatting. E.g. (32) - (16) = (16).
- The calculation is a part of an algebraic formulation. E.g. 32X - 16X = 16X.

What does not count as an answer:
- Other numbers are being subtracted. E.g. 33 - 16.
- Numbers are taken to the other side of the equals sign. E.g. 32 = 16 + 16.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

Prompt for step 2 to answer a linear question

Question: 16 / 8
Answer: 16 / 8 = 2

What also counts as an answer:
- The calculation is written out in words, or part of a story.
- The calculation is written in terms of a ratio. E.g. the ratio between 16 and 8 is 2.
- Different representation used for the division sign. E.g. ’divided by’.
- The calculation is part of another larger calculation. E.g. (16 / 8) * 9 = 2 * 9 or (16 / 8)/12 = 2/12.
- Different formatting. E.g. (16) / (8) = (2).
- The calculation is a part of an algebraic formulation. E.g. 32X / 16X = 2X.

What does not count as an answer:
- Other numbers are being divided. E.g. 17 / 8.
- Numbers are taken to the other side of the equals sign. E.g. 16 = 2 * 16.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

28

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Under review as a conference paper at ICLR 2025

Prompt for the full answer to a linear question

Question: 8x + 16 = 32
Answer: 2

What also counts as an answer:
- The calculation is written out in words, or part of a story.
- The calculation is written in terms of a ratio. E.g. the ratio between 16 and 8 is 2.
- Different representation used for the plus sign or the equals sign. E.g. ’added to’ and ’equals’.
- A different variable than X is used. E.g. ’t’: 8t + 16 = 32’.
- The calculation is part of another larger calculation. E.g. (8x + 16 = 32) * 9 = 2 * 9 or (8x + 16 = 32)/12 = 2/12.
- The solution is written out in steps below each other. E.g.:
8x + 16 = 32
8x = 2
x = 0.

- The calculation is a part of an algebraic formulation. E.g.:
5 * (8x + 16) = 5 * 32
5 * x = 5 * 2.

What does not count as an answer:
- Other numbers are being used. E.g. 9x + 16 = 32.

Document:

<document >

Is the answer given in the document? Answer with yes or no. If you answer with yes, indicate where the answer is by copying the part of the document in
which the answer occurs, ending with an explanation of why that passage contains the answer. Think step-by-step and carefully consider all the different
ways in which such an answer might be given.

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

29

Under review as a conference paper at ICLR 2025

A.6 PROMPTS GIVEN TO COMMAND R+ FOR CHARACTERISING THE RELATIONSHIP

BETWEEN THE QUERY AND THE DOCUMENT

We combine all reasoning queries in pairs with their top 500 most influential documents, and prompt
Command R+ to characterise the relationship. For all types of reasoning, we use the same preamble:

You are a brilliant AI assistant that is excellent at arithmetic designed to help
users with data analysis. You will be given an arithmetic query and a document,
and your task is to characterise the document by choosing keywords from a given
set that best describe how the document relates to the question.

For each type of reasoning, we craft a prompt that allows Command R+ to choose multiple keywords
for each query-document pair in the top 500 documents. We provide each below.

Prompt for arithmetic questions

Start of Query:

<query>

End of Query

Start of Document

<document>

End of Document

How is the document related to the query?

Choose from the following keywords:

Similar arithmetic operations on similar numbers (e.g. the numbers are similar in magnitude or the numbers are the same)
Similar arithmetic operations (on other types of numbers, e.g. much larger or smaller)
Reasoning traces (multiple reasoning steps are explicitly given in the document explaining how one gets to an answer)
Other types of maths
Code that contains arithmetic
Code that concerns other types of math
Code that concerns no math/arithmetic
Text about math/arithmetic (no other relation to the query than that the text is about math, text does not perform math/arithmetic)
Superficial similarities (there is no real relation, but loosely related topics occur, like the text contains words related to other parts of math, like algebra)
Similar formatting (question/answer pair about other topics than math)
Similar formatting (other)
Other (pick own keyword)

Explain your answer for each keyword by quoting from the query and document and describing why they are similar. Keep in mind that the document
might be in another language than English. If you pick any of the code keywords, add the programming languages in brackets (e.g. ‘Code that contains
arithmetic (Python, LaTeX)’). If the relation between the query and the document is not described by any of the given keywords, choose ‘other’ and
pick your own keyword that describes the document. Otherwise, if the query is not related to the document, state ‘no relation’ and describe why. Give
your answer in the form of a semicolon-separated list of keywords, and add an explanation below separated by newlines Give your answer in the form
of a semicolon-separated list of keywords, and add an explanation below separated by newlines (e.g. ‘keyword 1; keyword 2; keyword 3 (Python)
[explanation]’).

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

30

Under review as a conference paper at ICLR 2025

Prompt for slope questions

Start of Query:

<query>

End of Query

Start of Document

<document>

End of Document

How is the document related to the query?

Choose from the following keywords:

Similar arithmetic operations on similar numbers (e.g. the numbers are similar in magnitude or the numbers are the same)
Similar arithmetic operations (on other types of numbers, e.g. much larger or smaller)
Reasoning traces (multiple reasoning steps are explicitly given in the document explaining how one gets to an answer)
Other types of maths
Code that contains arithmetic
Code that calculates the slope between two numbers
Math that calculates the slope between two numbers
Code that calculates the slope of an equation
Math that calculates the slope of an equation
Code that concerns other types of math
Code that concerns no math/arithmetic
Text about math/arithmetic (no other relation to the query than that the text is about math, text does not perform math/arithmetic)
Superficial similarities (there is no real relation, but loosely related topics occur, like the text contains words related to other parts of math, like algebra)
Similar formatting (question/answer pair about other topics than math)
Similar formatting (other)
Other (pick own keyword)

Explain your answer for each keyword by quoting from the query and document and describing why they are similar. Keep in mind that the document
might be in another language than English. If you pick any of the code keywords, add the programming languages in brackets (e.g. ‘Code that contains
arithmetic (Python, LaTeX)’). If the relation between the query and the document is not described by any of the given keywords, choose ‘other’ and pick
your own keyword that describes the document. Otherwise, if the query is not related to the document, state ‘no relation’ and describe why. Give your
answer in the form of a semicolon-separated list of keywords, and add an explanation below separated by newlines (e.g. ‘keyword 1; keyword 2; keyword
3 (Python) [explanation]’).

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

31

Under review as a conference paper at ICLR 2025

Prompt for linear questions

Start of Query:

<query>

End of Query

Start of Document

<document>

End of Document

How is the document related to the query?

Choose from the following keywords:

Code that solves a linear equation for a variable (of the form ax + b = c or ax - b = c)
Code that solves a linear equation with multiple variables for one or both variables (e.g. ax + by = c)
Code that solves a linear equation of another form than ax + b = c or ax - b = c
Math that solves a linear equation for a variable (of the form ax + b = c or ax - b = c)
Math that solves an equation with multiple variables for one or both variables (e.g. ax + by = c)
Math that contains linear equations of another form than ax + b = c or ax - b = c
Math that contains linear equations but they are not solved (of the form ax + b = c or ax - b = c)
Math that contains linear equations but they are not solved (of another form than ax + b = c or ax - b = c)
Similar algebraic operations on similar numbers (e.g. the numbers are similar in magnitude or the numbers are the same)
Similar algebraic operations (on other types of numbers, e.g. much larger or smaller)
Other forms of algebra
Arithmetic operations
Other types of maths
Code that contains arithmetic
Code that concerns other types of math
Code that concerns no math/algebra
Text about math/algebra (no other relation to the query than that the text is about math, text does not perform math/algebra)
Reasoning traces (multiple reasoning steps are explicitly given in the document explaining how one gets to an answer)
Superficial similarities (there is no real relation, but loosely related topics occur, like the text contains words related to other parts of math, like arithmetic)
Similar formatting (question/answer pair about other topics than math)
Similar formatting (other)
Other (pick own keyword)

Explain your answer for each keyword by quoting from the query and document and describing why they are similar. Keep in mind that the document
might be in another language than English. If you pick any of the code keywords, add the programming languages in brackets (e.g. ‘Code that contains
arithmetic (Python, LaTeX)’) If the relation between the query and the document is not described by any of the given keywords, choose ‘other’ and pick
your own keyword that describes the document. Otherwise, if the query is not related to the document, state ‘no relation’ and describe why. Give your
answer in the form of a semicolon-separated list of keywords, and add an explanation below separated by newlines (e.g. ‘keyword 1; keyword 2; keyword
3 (Python) [explanation]’). If you pick a keyword about solving a linear equation, add the linear equation in the explanation.

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

32

Under review as a conference paper at ICLR 2025

A.7 FURTHER DISCUSSION OF LIMITATIONS

More broadly, our work suffers from the same limitations any work does that uses EK-FAC in-
fluence functions; we do many approximations to estimate the counterfactual and only take into
account MLP parameters. This latter decision is because EK-FAC influence functions are not prop-
erly defined for the attention layers (Grosse et al., 2023), although we do look at the dense layers
used within them. We list the assumptions and approximations here:

• First-order Taylor approximation to the PBRF.

• Assume different layers of MLPs are independent, making the Gauss-Newton Hessian

block-diagonal.

• Assume activations are independent of pre-activation pseudo-gradients.

• Estimate the approximation to the Fisher Information Matrix or equivalently the Gauss-
Newton Hessian by sampling from the empirical data distribution / model output distribu-
tion, because it’s an expectation over that distribution (MC estimation).

• Block-diagonal approximation of the eigenvector matrices within each layer.

• Low-rank approximation of query gradients.

• Assume EK-FAC for SFT stage is identity (Bae et al., 2024).

All these approximations are verified and justified in Grosse et al. (2023) and (Bae et al., 2024), and
the reader is referred there for a more in-depth analysis.

Our empirical results showing that nonetheless influence functions surface documents that are
causally related to accuracy in Appendix A.1 should alleviate some of these concerns, but not all.

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

33

Under review as a conference paper at ICLR 2025

A.8 ADDITIONAL RESULTS FOR THE QUALITATIVE ANALYSIS

A.8.1 DETAILS ON ANSWERS TO QUESTIONS IN PRETRAINING DATA

In the main text, we find the answer to factual questions relatively often compared to the answer
to reasoning questions. In this section, we comment on the possibility that the answer to reasoning
questions are simply not part of the pretraining sample of 5 million documents we look at. Recall
that all reasoning tasks require multiple steps, and the model outputs reasoning traces to get to the
final answer. This means that if the model is retrieving the answers, it should retrieve answers to all
the reasoning steps. We search for answers to the reasoning steps and factual questions in a random
subset of the 5 million pretraining documents. For the 7B reasoning questions, we find 43 documents
containing answers to reasoning steps, of which only 9 show up in the top 0.02% of the data. By
contrast, we find the answer to factual questions in 73 documents, of which 35 show up in the top
0.02% of the data. For the 35B, we find 7 documents with answers to reasoning steps, of which 4
show up in the top 0.02%. For the factual questions, we find 17 documents with answers, of which
15 show up in the top 0.02%. Keep in mind that for the factual questions this means answers to the
full questions, and for the reasoning questions it concerns answers to a reasoning step. In terms of
full answers showing up in the top 0.02%, we find one additional full answer on top of the ones we
found in the main text for the 7B reasoning questions, spread over two documents with rank 896542
and 4997351 of 5 million respectively (i.e. highly un- or negatively influential). For the 35B we do
not find full answers to reasoning queries at all. We provide a document with an answer for each
factual query and reasoning query that we do find an answer to in the top 0.02% in the supplemental
material as well as one example per reasoning step answer we find (e.g. if we find the answer to
6 − 4 = 2 four times, we show one example in the supplement). We highlight here some examples
from the larger pretraining sample for illustration.

For factual questions, it happens relatively frequently that the answer to the question shows up as
highly influential in multiple documents of the top 10 documents. For example, for the factual
question in Table 2 (“What is the tallest mountain in the world and how tall is it?”), the answer
shows up at ranks 1, 4, 6, and 7. The document at rank 1 (the most positively influential document),
is the following, which has the same question and the answer (question 5 below, underlined):

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

34

Under review as a conference paper at ICLR 2025

Factual query

What is the tallest mountain in the world and how tall is it?

Completion by 7B

The tallest mountain in the world is Mount Everest, which is located between Nepal
and Tibet. The height of Mount Everest is 8,849 meters (29,029 feet) above sea level.

Rank 1 document

Write some general trivia questions and answers. Here are some general trivia
questions and answers:

1. What is the longest word in the English language?

Answer: The longest word in the English language is ”pneumonoultramicroscopic-
silicovolcanoconiosis”, which is a type of lung disease.

2. How many bones are in the human body?

Answer: There are 206 bones in the human body.

3. What is the largest ocean in the world?

Answer: The largest ocean in the world is the Pacific Ocean, which covers approxi-
mately one-third of the Earth’s surface.

4. What is the longest river in the world?

Answer: The longest river in the world is the Nile River, which is 6,650 kilometers
(4,130 miles) long.

5. What is the highest mountain in the world?

Answer: The highest mountain in the world is Mount Everest,
which is 8,848 meters (29,029 feet) high.

6. What is the most common element in the human body?

Answer: The most common element in the human body is oxygen, which makes up
approximately 65% of the human body’s mass.

... redacted

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

35

Under review as a conference paper at ICLR 2025

Another document has the answer to part of an arithmetic query for the 7B (“Calculate the answer:
(5 - 3) * 12. Think step-by-step.”), namely 5 - 3 = 2 (underlined below, note that one needs to
understand rules to writing arithmetic to figure the answer out):

Reasoning query (arithmetic)

Calculate the answer: (5 - 3) * 12

Think step-by-step.

Completion by 7B

First, let’s evaluate the expression inside the parentheses:
5 - 3 = 2

Now, let’s multiply the result by 12:
2 * 12 = 24

Therefore, the answer is 24.

Rank 2 document

shown in numbers 7, 9 and 10.
PEMDAS Rules Practice 1 Solutions
1.
Solution:
Multiply: 2 x 3 + 4 x 6 = 6 + 24
Add: 6 + 24 = 30
Answer: 30
2.
Solution:
Multiply:
Divide:
Add:
Answer: 0
3.
Solution:
Parenthesis: 4 + (6 – 2)2 + 1 = 4 + 42 + 1
Exponent: 4 + 42 + 1 = 4 + 16 + 1
Add: 4 + 16 + 1 = 21
Answer: 21
4.
Solution:
Parenthesis: 8(6 – 2) ÷ 2(5 – 3) = 8(4) ÷ 2(2)
Multiply: 8(4) ÷ 2(2) = 32 ÷ 2(2)*
Divide: 32 ÷ 2(2)= 16(2)
Multiply: 16(2) = 32
Answer: 32
*This is the case mentioned in the PEMDAS Rules that when multiplication and divi-
sion are performed consecutively (without any other operations or grouping symbols
in between), the perform the operations from the left hand side to the right hand side.
... redacted

Interestingly, this document shows up in the top 10 documents for 11 of 20 arithmetic queries. By
contrast, the factual answer document shown before shows up in the top 10 for 4 of 40 queries (we

36

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

Under review as a conference paper at ICLR 2025

have another query that asks for the largest ocean in the world, for which this document also has the
answer).

To show that answers to more “niche” questions also show up, consider this document that contains
the answer to the question “What is the common name for the larva of a housefly?” (answer:
maggot, underlined below):

Factual query

What is the common name for the larva of a housefly?

Completion by 35B

The common name for the larva of a housefly is a maggot.

Rank 6 document

Common House Fly
Flies are easily recognized and are universally annoying. To say they are unap-
pealing is an understatement. Flies enjoy spending time on decaying organic filth,
including rotting meat and feces. They then manage to enter your home and land on
your kitchen counters and any uncovered food. House flies do more than just buzz
around and steal your food. They can transfer over 65 diseases including cholera,
dysentery, poliomyelitis, and tuberculosis. Flies accumulate nasty pathogens when
they rest on sewers, trash, feces, and any other moist or decaying material.
Flies lay their eggs on food (fruit in your fruit bowl), other animals, and even rotting
flesh. When the eggs hatch, the tiny rice-like larvae are called maggots. Few other
bugs provoke nausea and an overall disgust like tiny, squirming maggots. There are
a number of ways to eliminate flies and their larvae from your home. The most com-
mon practices include using sticky fly traps, ultra-violet lights, and baited traps. Fly
swatters are effective when trying to get rid of the lone fly. A timely smack from a
dish towel can also be employed.
There are simple daily practices that will help keep flies from taking over your house.
Store food in sealed containers, wipe crumbs off the counters, and don’t leave food
out. Grass clippings, garbage, and decaying matter will also attract flies, so get rid of
these items immediately. Keep your trash cans tightly covered and clean them on a
man-made surface where the run-off can’t seep into the soil. Whenever possible, use
door and window screens. This will help keep flies from entering your home. How-
ever, flies can find holes and gaps so remember to check your screens occasionally
and ensure they are impenetrable. Flies can also find openings on pipes or conduits
that lead inside. As a means of prevention, caulk those areas that channel into your
home.
At ZipZap Termite & Pest Control, we know how pesky the persistent buzzing of
the house fly can be. With kids running in and out, it is near impossible to keep
flies from getting in. Backyard parties are hard to enjoy when you’ve got house flies
buzzing around and landing on your food. If you feel you have an overwhelming
number of flies

This document has rank 6 for the relevant query, and never shows up in the top 10 for other queries.

Below, we show a document containing the answer to the reasoning step 5 + 4 = 9, required for one
of the arithmetic queries for the 7B model (“Calculate the answer: (5 + 4) * 2. Think step-by-step.”),
which does not show up in the top 0.02%.

37

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

Under review as a conference paper at ICLR 2025

Reasoning query (arithmetic)

Calculate the answer: (5 + 4) * 2

Think step-by-step.

Completion by 7B

First, let’s evaluate the expression inside the parentheses:
5 + 4 = 9

Now, let’s move on to the next step:
9 * 2 = 18

Therefore, the answer to the given expression is 18.

Rank 2140 document

For example:
n=1 = 1+2=3
n=2 = 1+4=5, 2x3=6
n=3 = 4+5=9, 1+7=8, 2x3=6

The question is, does a solution exist for every n?
I tried writing a basic program and it becomes too slow after n = 14. Here are the
solutions I have so far:
1 [‘1+2=3’]
2 [‘2*3=6’, ‘1+4=5’]
3 [‘4+5=9’, ‘1+7=8’, ‘2*3=6’]
4 [‘3+6=9’, ‘1+10=11’, ‘4+8=12’, ‘2+5=7’]
5 [‘2+8=10’, ‘3+6=9’, ‘1+13=14’, ‘5+7=12’, ‘11+4=15’]
6 [‘3*5=15’, ‘2+8=10’, ‘4+14=18’, ‘6+11=17’, ‘7+9=16’, ‘1+12=13’]
7 [‘6+12=18’, ‘3*5=15’, ‘7+10=17’, ‘1+20=21’, ‘4+9=13’, ‘2+14=16’, ‘8+11=19’]
8 [‘8+14=22’,
‘11+13=24’,
‘4+5=9’, ‘3+20=23’]
9 [‘6+19=25’,
‘9+15=24’, ‘5+16=21’, ‘11+12=23’]
10 [‘6+19=25’, ’

‘4+13=17’,

‘2+18=20’,

‘2+19=21’,

‘7+10=17’,

‘8+14=22’,

‘6+12=18’,

‘1+15=16’,

‘1+26=27’,

‘3+7=10’,

This document has rank 2140 for the relevant query.

38

1998
1999

2000
2001
2002
2003
2004
2005

2006
2007
2008
2009
2010
2011

2012
2013
2014
2015
2016

2017
2018
2019
2020
2021
2022

2023
2024
2025
2026
2027
2028

2029
2030
2031
2032
2033
2034

2035
2036
2037
2038
2039

2040
2041
2042
2043
2044
2045

2046
2047
2048
2049
2050
2051

Under review as a conference paper at ICLR 2025

A.8.2 CROSS-LINGUAL TRANSFER

Additional finding: The answer to the factual question sometimes shows up in non-English lan-
guages.
Interestingly, we observe some crosslingual transfer for the factual questions. For example, for the
question about the tallest mountain in the world (Table 2), the answer shows up in Portuguese:

A americana Samantha Larson, de 19 anos, se tornou nesta sexta-feira a mulher es-
trangeira mais jovem a conquistar o Monte Everest, segundo nota oficial divulgada pelo
Minist´erio de Turismo do Nepal. A montanha, de 8.848m, ´e a mais alta do mundo e se
encontra na fronteira entre o Nepal e Tibet.

Which translates to:

American Samantha Larson, 19, became the youngest foreign woman to conquer Mount
Everest on Friday, according to an official statement released by Nepal’s Ministry of
Tourism. The 8,848m (29,848ft) mountain is the highest in the world and is located on
the border between Nepal and Tibet.

We observe more crosslingual transfer for questions, for example for the question “What is the
capital of Belgium?” the answer shows in up in French and Spanish. We show the French document
here:

Le Premier ministre belge Yves Leterme a assur´e ce mercredi qu’il resterait en place et
m`enerait `a bien la r´eforme institutionnelle entre les r´egions, malgr´e les profondes divi-
sions entre Flamands et Wallons qui menacent l’unit´e du pays.
...
Les francophones redoutent pour leur part une r´eduction des budgets accord´es `a la
Wallonie, r´egion la plus pauvre du pays, et `a la capitale bilingue, Bruxelles.
Ils esti-
ment ´egalement que les r´egions se sont vu transf´erer depuis les ann´ees 1980 assez de
comp´etences f´ed´erales, et soupc¸onnent les n´eerlandophones de chercher `a faire s´ecession
de la Belgique afin de pouvoir d´eclarer l’ind´ependance de la Flandre.

Which translates to:

Belgian Prime Minister Yves Leterme assured on Wednesday that he would stay in office
and carry out the institutional reform between the regions, despite the deep divisions be-
tween Flemish and Walloons that threaten the unity of the country.
...
The French speakers, for their part, fear a reduction in the budgets granted to Wallonia,
the poorest region of the country, and to the bilingual capital, Brussels. They also believe
that the regions have been transferred enough federal powers since the 1980s, and suspect
that the Dutch-speaking countries are seeking to secede from Belgium in order to be able
to declare the independence of Flanders.

Note that both these quotes are snippets from otherwise larger documents. We did not translate all
documents and hence only found cases of crosslingual transfer if there happened to be keyword
overlap. We show a few here, but have found the answer to factual questions through keyword
overlap with non-English documents 8 times for the 7B model and 4 times for the 35B model. Note
that because this is only based on circumstantial keyword overlap, we likely missed most cases of
cross-lingual transfer, and therefore cannot assign any meaning to the fact that it happened less for
the 35B than the 7B. It would be interesting to focus on cross-lingual transfer in future work.

2052
2053

2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065

2066
2067
2068
2069
2070

2071
2072
2073
2074
2075
2076

2077
2078
2079
2080
2081
2082

2083
2084
2085
2086
2087
2088

2089
2090
2091
2092
2093

2094
2095
2096
2097
2098
2099

2100
2101
2102
2103
2104
2105

39

Under review as a conference paper at ICLR 2025

A.8.3 CHARACTERISE RELATION TOP DOCUMENTS TO QUERY

Finding 4: why documents are influential for reasoning. We prompt Command R+ to characterise
the relationship between the top 500 documents and each query (see prompts in Appendix A.6). We
add ‘reasoning traces’ as a potential keyword in the prompt, but after inspecting the results we find
the model uses that keyword for almost any document, and we remove those results. We report the
raw counts of each keyword occurring in the tables below.

Arithmetic (7B)

Other types of maths
Similar arithmetic operations on other numbers (e.g. much larger/smaller)
Code that contains arithmetic
Text about math/arithmetic
Code that concerns other types of math
Similar arithmetic operations on similar numbers
Similar formatting
Superficial similarities
Code that concerns no math/arithmetic

Table 14: Arithmetic (7B) Data

Slopes (7B)

Other types of maths
Similar arithmetic operations on similar numbers
Code that contains arithmetic
Similar formatting
Text that explains in words how to calculate the slope of an equation
Code that concerns other types of math
Text about math/arithmetic
Text that explains in words how to calculate the slope between two numbers
Math that calculates the slope of an equation
Math that calculates the slope between two numbers
Superficial similarities
Text that mentions the slope but does not explain how to calculate it
Code that calculates the slope between two numbers
Code that calculates the slope of an equation
Code that concerns no math/arithmetic
Other

Table 15: Slopes (7B) Data

Count

5765
4691
4038
3202
2554
2246
2223
1391
277

Count

10787
7312
5035
4675
3911
3577
3323
2959
2921
2490
2222
1677
1633
1110
263
15

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

40

Under review as a conference paper at ICLR 2025

Slopes (35B)

Other types of maths
Similar arithmetic operations on similar numbers
Code that contains arithmetic
Similar formatting
Text that explains in words how to calculate the slope of an equation
Text about math/arithmetic
Math that calculates the slope of an equation
Math that calculates the slope between two numbers
Code that concerns other types of math
Text that explains in words how to calculate the slope between two numbers
Superficial similarities
Text that mentions the slope but does not explain how to calculate it
Code that calculates the slope between two numbers
Code that calculates the slope of an equation
Code that concerns no math/arithmetic
Other
Similar arithmetic operations on other numbers (e.g. much larger/smaller)

Count

11104
8340
4617
4141
3869
3845
3745
3533
3192
2747
2291
1936
1150
865
121
12
1

Table 16: Slopes (35B) Data

Linear (35B)

Math that contains linear equations but they are not solved
Similar algebraic operations on similar numbers
Similar formatting
Math that solves a linear equation for a variable
Other forms of algebra
Arithmetic operations
Code that contains arithmetic
Other types of maths
Text about math/algebra
Code that solves a linear equation of another form than ax + b = c or ax - b = c
Superficial similarities
Code that concerns other types of math
Code that concerns no math/algebra
Code that solves a linear equation for a variable
Math that solves an equation with multiple variables for one or both variables
Math that contains linear equations of another form than ax + b = c or ax - b = c
Code that solves a linear equation with multiple variables for one or both variables
Other

Count

13434
10717
5533
2415
2234
2057
1417
1390
1146
1109
1105
949
560
475
172
156
110
1

Table 17: Linear (35B) Data

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

41

Under review as a conference paper at ICLR 2025

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

Figure 7: For the reasoning and factual sets, we compare the amount of documents from a certain
source dataset that show up in the top portions of the rankings to the amount you would expect to
show up if you randomly sample from the pretraining distribution (indicated by ‘Training distribu-
tion’ in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find that data
from Wikipedia and Math & Trivia are important for the factual questions for both models, for the
reasoning questions Math & Trivia, StackExchange, Code, and ArXiv data is important. In all cases,
the multipliers tend to the training distribution for higher k.

A.8.4 SOURCE DATASET ANALYSIS

Finding 5: code is heavily overrepresened for reasoning both for the top and bottom portions
of the ranking.
For each source dataset, we report the multiplier w.r.t. the training distribution. This means that if
the top k documents are randomly sampled from pretraining, the multipliers will be one, whereas
if they are above or below one, that source dataset is either over- or underrepresented in the most
influential documents. The full results are presented in Figure 7, and we discuss the most interesting
deviations from the pretraining distribution here. For the factual questions, the most overrepresented
source datasets for both the 7B and 35B are Math & Trivia (multiplier of 27 and 16 for k = 50
respectively) and Wikipedia (multipliers of 5 and 6 respectively). For the reasoning questions, the
most overrepresented datasets are StackExchange and Math & Trivia (with 50 and 24 als multipliers
for the 7B, and 62 and 21 for the 35B). Interestingly, for both the 7B and the 35B, code data is
important for the influential documents. Besides StackExchange, for the medium-influential portion
of the rankings (between k = 5000 and k = 50000), more code data becomes influential (with
multipliers around 2, compared to 0.5 for the factual questions at that same part of the ranking).
This is conventional wisdom among practitioners (most LLMs designers use some percentage of
code data in pretraining now, e.g. Touvron et al. (2023)), and recent work has empirically found
code to be important for reasoning performance (Aryabumi et al., 2024). However, the question
of why code data is important for reasoning is still open. Below, in Appendix A.8.5, we further
confirm that code is important for reasoning by not only relying on the fact that these documents
come from a code dataset, but actually classifying their contents. In Figure 8 we present the same
plot for the bottom portion of the ranking, showing the findings are similar. Further, in Figure 9
and 10 we respectively show the same results for the top and bottom portion of the rankings for the

42

Under review as a conference paper at ICLR 2025

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

Figure 8: For the reasoning and factual sets, We compare the amount of documents from a certain
source dataset that show up in the bottom portions of the rankings to the amount you would expect
to show up if you randomly sample from the pretraining distribution (indicated by ‘Training distri-
bution’ in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find the
patterns are almost identical to those shown for the top portions of the ranking: data from Wikipedia
and Math & Trivia are important for the factual questions for both models, for the reasoning ques-
tions Math & Trivia, StackExchange, Code, and ArXiv data is important. In all cases, the multipliers
tend to the training distribution for higher k.

control queries. Again, the results look similar (code and StackExchange is also overrepresented for
the reasoning control queries), but arXiv is less overrepresented.

43

Under review as a conference paper at ICLR 2025

2322
2323

2324
2325
2326
2327
2328
2329

2330
2331
2332
2333
2334
2335

2336
2337
2338
2339
2340

2341
2342
2343
2344
2345
2346

2347
2348
2349
2350
2351
2352

2353
2354
2355
2356
2357
2358

2359
2360
2361
2362
2363

2364
2365
2366
2367
2368
2369

2370
2371
2372
2373
2374
2375

Figure 9: For the query control sets, we also compare the amount of documents from a certain source
dataset that show up in the top portions of the rankings to the amount you would expect to show up
if you randomly sample from the pretraining distribution (indicated by ‘Training distribution’ in the
figure). The top two plots are for the 7B, and the bottom for the 35B. We find that code is still
overrepresented, but arXiv as source is less overrepresented for the top portions of the reasoning
control set than for the reasoning set.

44

Under review as a conference paper at ICLR 2025

2376
2377

2378
2379
2380
2381
2382
2383

2384
2385
2386
2387
2388
2389

2390
2391
2392
2393
2394

2395
2396
2397
2398
2399
2400

2401
2402
2403
2404
2405
2406

2407
2408
2409
2410
2411
2412

2413
2414
2415
2416
2417

2418
2419
2420
2421
2422
2423

2424
2425
2426
2427
2428
2429

Figure 10: For the query control sets, we also compare the amount of documents from a certain
source dataset that show up in the bottom portions of the rankings to the amount you would expect
to show up if you randomly sample from the pretraining distribution (indicated by ‘Training distri-
bution’ in the figure). The top two plots are for the 7B, and the bottom for the 35B. We find that it
again looks similar to the source distribution for the top of the rankings for the query control sets.

45

Under review as a conference paper at ICLR 2025

A.8.5 CONTENT ANALYSIS OF RELEVANT DOCUMENTS

We provide further insights into the characteristics of influential documents on reasoning queries. To
do so, we compute capability categories of the n = 500 most frequently occurring documents among
the k = 5000 most (top) or least (bottom) influential documents for the reasoning queries (for the
7B model), and compare these to a randomly sampled set of 500 documents (we repeat the sampling
process three times and provide mean and standard deviation scores on the detected capabilities).
Results are shown in Figure 11. We can see that the “code” category represents the vast majority
of most and least influential documents, whereas for the random subsets the fraction of code-related
documents is relatively small. This provides further evidence that code-related documents strongly
influence model performance on reasoning tasks.

Figure 11: Comparison of capability categories identified for the most and least influential docu-
ments for the reasoning queries, as well as for a random subset of sampled documents. We repeat
the random sampling three times and report mean scores with standard deviations indicated.

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

46

codecreative_generationgrounded_textreasoning_and_factsCategory050100150200250300350400Frequency[7B] Category Distribution Across Different Datasetsn=500 (random)k=5000 (bottom)k=5000 (top)Under review as a conference paper at ICLR 2025

A.9 ADDITIONAL RESULTS FOR THE QUANTITATIVE ANALYSIS

A.9.1 CORRELATION ANALYSIS

Figure 12: The correlation between the influence scores of all 5 million documents for pairs of
queries. All queries are on the x- and y-axis, with the first 40 belonging to the factual set, the next
40 to the reasoning set (arithmetic and slopes for the 7B, and linear and slopes for the 35B), the
following 10 to the factual control set, and the last 10 to the reasoning control set. The take-away is
that there is only a signficant correlation between queries of the same reasoning type, most strongly
so for the 35B slopes queries.

Figure 13: The average correlations between the influences of all documents for queries of a specific
type grouped. We leave out any query combinations where the correlation is not significant and any
combination where the query on the x- and y-axis is the same query. We again observe that there is
only a correlation of influence for queries of the same reasoning type.

Additional results finding 1 (correlation between reasoning queries of the same type).
In the main text, we find that there is a correlation between the influence scores for the documents
for different queries that underlie the same type of reasoning question (e.g. questions that all require
calculating the slope but for different numbers). One other explanation for this result could be the

47

2484
2485

2486
2487
2488
2489
2490
2491

2492
2493
2494
2495
2496
2497

2498
2499
2500
2501
2502

2503
2504
2505
2506
2507
2508

2509
2510
2511
2512
2513
2514

2515
2516
2517
2518
2519
2520

2521
2522
2523
2524
2525

2526
2527
2528
2529
2530
2531

2532
2533
2534
2535
2536
2537

Under review as a conference paper at ICLR 2025

2538
2539

2540
2541
2542
2543
2544
2545

2546
2547
2548
2549
2550
2551

2552
2553
2554
2555
2556

2557
2558
2559
2560
2561
2562

2563
2564
2565
2566
2567
2568

2569
2570
2571
2572
2573
2574

2575
2576
2577
2578
2579

2580
2581
2582
2583
2584
2585

2586
2587
2588
2589
2590
2591

fact that all these queries are superficially more similar to each other than the factual questions,
and that this is the reason the influence correlates. To test this hypothesis, we use the 10 control
queries for both the factual questions and the reasoning questions that are superficially similar, but
do not require factual retrieval or reasoning to get to the solution (see Appendix A.3 for examples
from each set). Figure 12 shows all query-query correlations for the 100 queries we look at, ordered
as follows: 40 factual queries, 40 reasoning (of which 20 are arithmetic and 20 slopes questions
for the 7B, and 20 linear equation and 20 slopes questions for the 35B), 10 factual control, and 10
reasoning control queries. We see that there is only a significant correlation between queries of the
same reasoning type, which we summarise in Figure 13 by grouping queries of the same type and
averaging the correlations.

To get a sense of what the control questions look like, we show a side-by-side comparison of a
slopes query (top) and the control query that is meant to resemble it (bottom) below. The influences
for this query correlate with the influences for the slope questions on average with a Pearson’s
R of 0.05, which is much smaller than the 0.32 average correlation between the influences found
for the different slope reasoning questions by the 7B model. Below, we perform a more detailed
qualitative analysis of the query combinations and what drives their correlations, but first we discuss
the quantitative result.

As mentioned, we have 10 factual and 10 reasoning control questions for both models, and show
the full correlation matrices below in Figure 12 (per query) and Figure 13 (averaged per group). We
observe that the correlations between queries from the control sets and other query sets for the 35B
is always between 0.05 and 0.10, which indicates that there can be a score correlation of at least
0.10 for other things than genuine reasoning and procedural knowledge. Further, the within-group
correlations of the reasoning control set sometimes go as high as 0.38 (although the average is 0.06
for the 7B and 0.10 for the 35B). For comparison, the average linear-linear score correlation for the
35B is 0.16, and not many of the correlations that make up this average are higher than the correla-
tions in the reasoning control sets. To get a sense of how different the correlations are in magnitude
between the reasoning questions and the control questions, we calculate the highest correlation of a
query from a specific reasoning type with any other query, count the amount of query-query combi-
nations for which the correlation is higher. For example, the maximum correlation we find between
any slope question for the 35B and any other query that is not a slope question is 0.30 Pearson’s
R. If we discard all query combinations that are below 0.30 we are left with 138 of 190 significant
combinations that are higher, ranging to 0.96 Pearson’s R (note that each reasoning group has 20
queries, and all combinations are 20 ∗ 19/2 = 190). For the linear equation queries by contrast,
there are only 34 of 190 query-query combinations within this group that have a correlation higher
than the highest correlation with the control queries, ranging to 0.95 Pearson’s R. For the 7B, 84 of
190 arithmetic query combinations have a higher correlation than the control correlations, ranging
to 0.96 Pearson’s R, and 120 of 190 slopes query combinations, ranging to 0.88. We therefore con-
clude that the correlations between the queries for the linear equations can mainly be explained by
other, more superficial things than procedural knowledge, and connect this finding to the fact that the
model is less robustly able to solve linear equations. The within-group correlations of the factual set
are much lower, and for the 7B we only find 5 of 780 correlations that are higher than the maximum
correlation of a factual query with another query group, ranging to 0.63 Pearson’s R (we show the
queries with the highest correlation below). For the 35B, we find no correlations for factual queries
higher than the maximum correlation with another group.

We release all 100 ∗ 100 query-query correlations in the supplemental material (on which Figure
12 is based), and we highlight a few examples here to get a sense of what is driving higher or
lower correlations. We mostly do not find a correlation between the influence scores for different
factual questions (on average the correlation is 0.06 for the 7B and 0.03 for the 35B), but we show
the highest correlation found between two factual queries below in Table 19. For this particular
example, the correlation seems driven by the fact that they are both “what”-questions with very
short completions using quotation marks. By contrast, a similar style query-completion pair for
the 35B shows almost no correlation (Table 20). For the factual control set(the average correlation
is 0.04 for the 7B and 0.08 for the 35B, but we also show a factual control correlation below for
completeness. Table 11 shows a factual query and a matched factual control example for the 35B
model. The correlation between this control query and the factual query it matches is -0.03.

See Table 22 below for a representative high correlation example for the 35B slopes questions (Pear-
son’s R of 0.89):

48

Under review as a conference paper at ICLR 2025

Table 18: Two query examples, of which one is from the reasoning set (slopes) and the other from
the reasoning control set that resembles the slope queries superficially, but does not require any
reasoning. The completions are by the 7B model.

Reasoning and Reasoning Control Query Set (7B)

Slopes (original prompt)

What is the slope of the line passing through the points (79, 47) and (80, 25)?

Think step-by-step.

Completion

To find the slope of the line passing through the points (79, 47) and (80, 25), we can use the following steps:

1. Identify the coordinates of the two points:
- Point A: (79, 47)
- Point B: (80, 25)

2. Calculate the difference in x-coordinates between the two points:
- Point A x-coordinate: 79
- Point B x-coordinate: 80 ... redacted completion.

Slopes (control prompt)

The slope of the line is -22. What is the slope of the line?

Think step-by-step.

Completion

Given that the slope of the line is -22.

To find the slope of the line, we need to find the change in y-coordinates divided by the change in x-coordinates.

The slope of the line is -22.

Therefore, the slope of the line is -22.

Table 19: Two query examples for which the Pearson’s R correlation of their document influences is
0.63, both from the factual set of the 7B model. We observe that the main similarity between these
queries is that they both concern short “what” questions, and the completions both are short answers
with quotes used.

Factual Query Set (7B)

Factual prompt 1

What is the symbol for gold on the periodic table?

Completion to factual prompt 1

The symbol for gold on the periodic table is “Au”.

Factual prompt 2

What is the common name for the larva of a housefly?

Completion to factual prompt 2

The common name for the larva of a housefly is “maggot”

49

2592
2593

2594
2595
2596
2597
2598
2599

2600
2601
2602
2603
2604
2605

2606
2607
2608
2609
2610

2611
2612
2613
2614
2615
2616

2617
2618
2619
2620
2621
2622

2623
2624
2625
2626
2627
2628

2629
2630
2631
2632
2633

2634
2635
2636
2637
2638
2639

2640
2641
2642
2643
2644
2645

Under review as a conference paper at ICLR 2025

Table 20: Two query examples for which the Pearson’s R correlation of their document influences is
0.08, both from the factual set of the 35B model. We observe that the main similarity between these
queries is that they both concern short “what” questions, and the completions both are relatively
short answers with quotes used.

Factual Query Set (35B)

Factual prompt 1

What is the name of the main antagonist in the Shakespeare play “Othello”?

Completion to factual prompt 1

The main antagonist in Shakespeare’s play “Othello” is Iago, Othello’s standard-bearer.
Iago is a manipulative and deceitful character who orchestrates the tragedy in the play.

Factual prompt 2

What is the name of the smallest member of the camel family?

Completion to factual prompt 2

The smallest member of the camel family is called a “dromedary”.
Dromedaries are easily recognizable by their single hump. They are native to North Africa and the Middle East.

Table 21: Two query examples for which the Pearson’s R correlation of their document influences is
-0.03, one from the factual set (top), and one from the factual control set (bottom). We observe that
although the formatting is similar, and there are some overlapping concepts, there is no correlation
between the document influences.

Factual and Control Query Set (35B)

Factual prompt

What is the supernova remnant and pulsar wind nebula in the constellation of Taurus called?

Completion to factual prompt

The supernova remnant and pulsar wind nebula in the constellation Taurus is called the Crab Nebula.

Factual control prompt

In a fictional planetary system, there is a supernova remnant and pulsar wind nebula in the centre of
a constellation of stars called Minotaur called the Goat Nebula.
What is the supernova remnant and pulsar wind nebula in the constellation Minotaur called?

Completion to factual control prompt

In the fictional planetary system, the supernova remnant and pulsar wind nebula
in the centre of the constellation Minotaur is called the “Goat Nebula”.

50

2646
2647

2648
2649
2650
2651
2652
2653

2654
2655
2656
2657
2658
2659

2660
2661
2662
2663
2664

2665
2666
2667
2668
2669
2670

2671
2672
2673
2674
2675
2676

2677
2678
2679
2680
2681
2682

2683
2684
2685
2686
2687

2688
2689
2690
2691
2692
2693

2694
2695
2696
2697
2698
2699

Under review as a conference paper at ICLR 2025

Table 22: Two query examples for which the Pearson’s R correlation of their document influences
is 0.89, both from the reasoning query set concerning calculating slopes. We observe that, although
the reasoning is applied to different numbers, the steps follow an identical pattern. In fact, the only
difference between the two queries, including completions, is the numbers used.

Reasoning Query Set (35B)

Slopes (prompt 1)

What is the slope of the line passing through the points (94, 62) and (95, 19)?

Think step-by-step.

Completion to prompt 1

The slope of a line passing through two points is calculated using the formula:

Slope = (change in y) / (change in x)

In this case, we have the points (94, 62) and (95, 19).

Substituting the values into the formula, we have:

Slope = (19 - 62) / (95 - 94)

Slope = (-43) / (1)

Slope = -43

Therefore, the slope of the line passing through the points (94, 62) and (95, 19) is -43.

Slopes (prompt 2)

What is the slope of the line passing through the points (90, 20) and (94, 64)?

Think step-by-step.

Completion to prompt 2

The slope of a line passing through two points is calculated using the formula:

Slope = (change in y) / (change in x)

In this case, we have the points (90, 20) and (94, 64).

Substituting the values into the formula, we have:

Slope = (64 - 20) / (94 - 90)

Slope = (44 / 4)

Slope = 11

Therefore, the slope of the line passing through the points (90, 20) and (94, 64) is 11

51

2700
2701

2702
2703
2704
2705
2706
2707

2708
2709
2710
2711
2712
2713

2714
2715
2716
2717
2718

2719
2720
2721
2722
2723
2724

2725
2726
2727
2728
2729
2730

2731
2732
2733
2734
2735
2736

2737
2738
2739
2740
2741

2742
2743
2744
2745
2746
2747

2748
2749
2750
2751
2752
2753

Under review as a conference paper at ICLR 2025

We see that the numbers for the questions are different, but the steps to get to the answer are the
same and formatted similarly. We generally observe that a combination of different numbers used
in the same procedure and similarly formatted reasoning steps results in higher correlation. Another
example is shown in Table 23 below for a query combination that gets a lower correlation (0.55),
but has the same answer (-22). The relatively lower correlation here seems driven by the different
formatting of the completion (less new lines). This indicates that part of the correlations are driven
by superficial similarities such as completion formatting.

For comparison, Table 24 shows a pair of control queries for which the prompt and completion are
also very similarly formatted, and the correlation is 0.38 (which is the highest correlation we find
using a control query).

However, superficial formatting cannot be the full picture, as we show in Table 25, where the cor-
relation is 0.35 but the completions are very dissimilar. For this particular query, it seems that it’s
not the positive scores that explain the correlation (Pearson’s R of about 0.2), but rather the negative
scores (Pearson’s R of 0.44). Perhaps some of the correlation can be explained by documents that
are generally negatively influential for the types of completions these models make.

Finally, in Table 26 and 27 we show two relatively high correlation examples for the arithmetic
queries, again indicating that both underlying operations as well as formatting drive correlations.

Taken together, it seems like correlations can be driven by underlying procedures, formatting of the
completion, and other more general things (like “what”-questions in Table 19 and 25). We find the
highest correlations when procedures and formatting of completions coincide (Table 22 and 27). The
magnitude of these correlations indicate that almost all of the influence of the 5 million documents
in similar for such queries. One interesting possibility is that the query information surrounding
the actual numbers generated (which do not seem to drive correlation much at all) is determined
by the attention layers (which, besides the dense parameters contained in them, we ignore in this
work), connecting potentially to literature attributing reasoning operations to attention heads. An
interesting avenue for future work would be investigating this further.

2754
2755

2756
2757
2758
2759
2760
2761

2762
2763
2764
2765
2766
2767

2768
2769
2770
2771
2772

2773
2774
2775
2776
2777
2778

2779
2780
2781
2782
2783
2784

2785
2786
2787
2788
2789
2790

2791
2792
2793
2794
2795

2796
2797
2798
2799
2800
2801

2802
2803
2804
2805
2806
2807

52

Under review as a conference paper at ICLR 2025

Table 23: Two query examples for which the Pearson’s R correlation of their document influences
is 0.55, both from the reasoning query set concerning calculating the slope. We observe that the
second completion is structured very similarly to the first, except it has less new lines.

Reasoning Query Set (7B)

Slopes (prompt 1)

What is the slope of the line passing through the points (62, 31) and (61, 53)?

Think step-by-step.

Completion

The slope of a line passing through two points is calculated using the formula:

Slope = (change in y) / (change in x)

In this case, we have the points (62, 31) and (61, 53).

Substituting the values into the formula, we have:

Slope = (31 - 53) / (62 - 61)

Slope = (-22) / (1)

Slope = -22

Therefore, the slope of the line passing through the points (62, 31) and (61, 53) is -22

Slopes (prompt 2)

What is the slope of the line passing through the points (79, 47) and (80, 25)?

Think step-by-step.

Completion The slope of a line passing through two points is calculated using the formula:
Slope = (change in y) / (change in x)

In this case, we have the points (79, 47) and (80, 25).

Substituting the values into the formula, we have:
Slope = (25 - 47) / (80 - 79)
Slope = (-22) / (1)
Slope = -22

Therefore, the slope of the line passing through the points (79, 47) and (80, 25) is -22.

2808
2809

2810
2811
2812
2813
2814
2815

2816
2817
2818
2819
2820
2821

2822
2823
2824
2825
2826

2827
2828
2829
2830
2831
2832

2833
2834
2835
2836
2837
2838

2839
2840
2841
2842
2843
2844

2845
2846
2847
2848
2849

2850
2851
2852
2853
2854
2855

2856
2857
2858
2859
2860
2861

53

Under review as a conference paper at ICLR 2025

2862
2863

2864
2865
2866
2867
2868
2869

2870
2871
2872
2873
2874
2875

2876
2877
2878
2879
2880

2881
2882
2883
2884
2885
2886

2887
2888
2889
2890
2891
2892

2893
2894
2895
2896
2897
2898

2899
2900
2901
2902
2903

2904
2905
2906
2907
2908
2909

2910
2911
2912
2913
2914
2915

Table 24: Two query examples for which the Pearson’s R correlation of their document influences
is 0.38, both from the reasoning control set for the 7B model. We observe that the formatting is very
similar, but the correlation is still lower than for the reasoning queries above.

Reasoning Control Query Set (7B)

Reasoning Control (prompt 1)

Find the answer: There are 21 kids in the bus. How many kids are in the bus?

Think step-by-step.

Completion

There are 21 kids in the bus.

Reasoning Control (prompt 2)

Find the answer: There are 132 marbles in a jar. How many marbles are in the jar?

Think step-by-step.

Completion

There are 132 marbles in the jar.

Table 25: Two query examples for which the Pearson’s R correlation of their document influences
is 0.35, both from the reasoning control set for the 35B model. We observe that their completions
are very different, indicating that it can not only be superficial similarities that drive correlations.

Reasoning Control Query Set (35B)

Reasoning control (prompt 1)

This line has a slope 6. What is the slope of the aforementioned line?

Think step-by-step.

Completion

The slope of a line is a measure of the steepness of the line.
It is defined as the ratio of the change in the y-coordinate to the change in the x-coordinate.

In this case, the line has a slope of 6. This means that for every 1 unit increase in the x-coordinate,
the y-coordinate will increase by 6 units.

It’s important to note that the slope of a line is not limited to positive values and
can be negative or zero depending on the orientation of the line

Reasoning control (prompt 2)

Find the answer: there are two bananas on a tree, how many bananas are on the tree?

Think step-by-step.

Completion

There are two bananas on the tree.

54

Under review as a conference paper at ICLR 2025

2916
2917

2918
2919
2920
2921
2922
2923

2924
2925
2926
2927
2928
2929

2930
2931
2932
2933
2934

2935
2936
2937
2938
2939
2940

2941
2942
2943
2944
2945
2946

2947
2948
2949
2950
2951
2952

2953
2954
2955
2956
2957

2958
2959
2960
2961
2962
2963

2964
2965
2966
2967
2968
2969

Table 26: Two query examples for which the Pearson’s R correlation of their document influences
is 0.49, both from the reasoning query set concerning two-step arithmetic. We observe that one of
the operations required is overlapping (+ 6), and the completion is somewhat similarly formatted but
not fully.

Reasoning Query Set (7B)

Arithmetic (prompt 1)

Calculate the answer: (5 + 6) * 10

Think step-by-step.

Completion

First, let’s evaluate the expression inside the parentheses:
5 + 6 = 11

Now, let’s multiply the result by 10:
11 * 10 = 110

Therefore, the answer is 110

Arithmetic (prompt 2)

Calculate the answer: (6 + 10) * 6

Think step-by-step.

Completion

First, let’s evaluate the expression inside the parentheses:
6 + 10 = 16

Now, let’s calculate the result of multiplying the expression inside the parentheses by 6:
16 * 6 = 96

Therefore, the answer to the expression is 96

55

Under review as a conference paper at ICLR 2025

Table 27: Two query examples for which the Pearson’s R correlation of their document influences is
0.94, both from the reasoning query set concerning two-step arithmetic. We observe that not only is
the completion very similarly formatted, but one of the operations overlaps (multiplying something
by 2).

Reasoning Query Set (7B)

Arithmetic (prompt 1)

Calculate the answer: (5 - 3) * 12

Think step-by-step.

Completion

First, let’s evaluate the expression inside the parentheses:
5 - 3 = 2

Now, let’s multiply the result by 12:
2 * 12 = 24

Therefore, the answer is 24

Arithmetic (prompt 2)

Calculate the answer: (4 - 2) * 15

Think step-by-step.

Completion

First, let’s evaluate the expression inside the parentheses:
4 - 2 = 2

Now, let’s multiply the result by 15:
2 * 15 = 30

Therefore, the answer is 30

56

2970
2971

2972
2973
2974
2975
2976
2977

2978
2979
2980
2981
2982
2983

2984
2985
2986
2987
2988

2989
2990
2991
2992
2993
2994

2995
2996
2997
2998
2999
3000

3001
3002
3003
3004
3005
3006

3007
3008
3009
3010
3011

3012
3013
3014
3015
3016
3017

3018
3019
3020
3021
3022
3023

Under review as a conference paper at ICLR 2025

A.9.2 MAGNITUDE OF INFLUENCE

Additional results finding 2 (magnitude of influence is much lower and less volatile for reason-
ing questions).
In the main paper, we find that the influence of documents at the same rank for factual questions is
much more volatile than for reasoning questions. We mention that one explanation for this might be
that the queries for the 35B model are much more niche, and therefore the relevant documents much
more infrequent. To test this hypothesis, we plot the same results for only the overlapping queries
(those that are part of both query sets for the 7B and 35B) in Figure 14. We find that the magnitude
and variance is still larger for the 35B model than for the 7B model, indicating that the influence of
influential documents for the factual and reasoning questions by the 35B can be much larger than for
the 7B model. Further, in Figure 15 we show that the results look similar for the negative portions
of the ranking (where we flip the influence scores from negative to positive).

Figure 14: The total influence per nat of query completion information for different portions of the
positive ranking over documents, left for the 7B model, right for the 35B. In this case, we only plot
queries that are present in the query sets for both models. This means the prompt is the same, but
the completion is be different. The pattern is very similar as the observed pattern for the top of the
ranking.

Figure 15: The total influence per nat of query completion information for different portions of the
negative ranking over documents, left for the 7B model, right for the 35B. We again only plot queries
that are present in the query sets for both models. In this case, the k-th percentile contains the top k
% of most negatively influential documents. The pattern is very similar as the observed pattern for
the top of the ranking.

Finally, in Figure 16 and Figure 17 we plot the same metric for all queries for the top and bot-
tom parts of the rankings respectively, now including the 10 control set queries of the factual and

57

3024
3025

3026
3027
3028
3029
3030
3031

3032
3033
3034
3035
3036
3037

3038
3039
3040
3041
3042

3043
3044
3045
3046
3047
3048

3049
3050
3051
3052
3053
3054

3055
3056
3057
3058
3059
3060

3061
3062
3063
3064
3065

3066
3067
3068
3069
3070
3071

3072
3073
3074
3075
3076
3077

Under review as a conference paper at ICLR 2025

3078
3079

3080
3081
3082
3083
3084
3085

3086
3087
3088
3089
3090
3091

3092
3093
3094
3095
3096

3097
3098
3099
3100
3101
3102

3103
3104
3105
3106
3107
3108

3109
3110
3111
3112
3113
3114

3115
3116
3117
3118
3119

3120
3121
3122
3123
3124
3125

3126
3127
3128
3129
3130
3131

Figure 16: The total influence per nat of query completion information for different portions of
the positive ranking over documents, left for the 7B model, right for the 35B. We plot all queries,
including the query control sets for both factual and reasoning, which contain 10 queries each.

Figure 17: The total influence per nat of query completion information for different portions of
the negative ranking over documents, left for the 7B model, right for the 35B. We plot all queries,
including the query control sets for both factual and reasoning, which contain 10 queries each.

reasoning control set. As shown in Appendix A.3, we use 10 control queries for each set to investi-
gate whether results hold similarly for queries that superficially look similar as the factual/reasoning
questions, but that do not require factual retrieval or reasoning respectively. We observe that the
control sets both show much higher variance and magnitude than the reasoning queries as well, for
the positive and negative portions of the ranking. For completeness, we show the same result with
the number of documents on the x-axis instead of percentiles in Figure 18 and Figure 19, to show
that the results are similar if we take into account that the 20-th percentile of documents for each
query contains a different amount of documents k.

58

Under review as a conference paper at ICLR 2025

Figure 18: The total influence per nat of query completion information for different number of
documents k of the positive ranking, left for the 7B model, right for the 35B. We plot all queries,
including the query control sets for both factual and reasoning, which contain 10 queries each.

Figure 19: The total influence per nat of query completion information for different number of
documents k of the negative ranking, left for the 7B model, right for the 35B. We plot all queries,
including the query control sets for both factual and reasoning, which contain 10 queries each.

59

3132
3133

3134
3135
3136
3137
3138
3139

3140
3141
3142
3143
3144
3145

3146
3147
3148
3149
3150

3151
3152
3153
3154
3155
3156

3157
3158
3159
3160
3161
3162

3163
3164
3165
3166
3167
3168

3169
3170
3171
3172
3173

3174
3175
3176
3177
3178
3179

3180
3181
3182
3183
3184
3185

Under review as a conference paper at ICLR 2025

A.9.3 DISPERSION OF INFLUENCE: POWER LAWS

Figure 20: The ranked influence scores per query nat for each query shown separately in log-log
space. We observe; the results follow power laws (linear in log-log space), everything is shifted up
for the 35B model (right), generally the scores for the reasoning documents are lower for the 7B
model, and for the 35B model there is less variance in magnitude of influence for reasoning queries
than for factual queries, and more often than not the influence scores are lower than for factual
questions.

Figure 21: The ranked influence scores per query nat for each query shown separately in log-log
space again, but now also showing the control queries. We observe that also for the control queries
the influence is much more volatile than for reasoning questions, and on average the magnitude is
higher.

In this section, we look at the power laws induced by the top portions of the rankings. We can fit
linear functions to the rankings in log-log space, and analyse the slopes to comment on the sparsity
of the rankings (i.e. how many documents do models rely on for a completion). Specifically, we
perform linear regression on the log-log top 500 rankings of each query, and report the slopes in
Table 28.

After qualitatively inspecting the queries for the 35B model with the steepest slope, we believe an
explanation for this result may be ‘noise’ in the influence scores. For example, the query with the
steepest slope (α = −0.45) has as the most influential document a document that is seemingly
entirely unrelated to the query. Namely, the query asks the question “What is the slope of the
line passing through the points (41, 23) and (18, 92)? Think step-by-step.”, and the top influential

60

3186
3187

3188
3189
3190
3191
3192
3193

3194
3195
3196
3197
3198
3199

3200
3201
3202
3203
3204

3205
3206
3207
3208
3209
3210

3211
3212
3213
3214
3215
3216

3217
3218
3219
3220
3221
3222

3223
3224
3225
3226
3227

3228
3229
3230
3231
3232
3233

3234
3235
3236
3237
3238
3239

Under review as a conference paper at ICLR 2025

Table 28: Slopes of the fitted functions to the top 500 documents in the influence rankings in log-log
space, separated by query set and whether the model gets the question right or wrong. ⋆ indicates
the significance of an independent T-test performed between the slopes of the factual vs. reasoning
queries, where ⋆ indicates a p-value below 0.1 and ⋆⋆ below 0.05.

7B (Incorrect)
7B (Correct)
Reasoning (α) −0.36 ± 0.03⋆ −0.33 ± 0.02
−0.34 ± 0.04
−0.34 ± 0.03
Factual (α)

35B (Correct)
35B (Incorrect)
−0.36 ± 0.04⋆⋆ −0.38 ± 0.04⋆
−0.34 ± 0.04
−0.32 ± 0.05

document is a snippet about the lunar eclipses and when and where they can be viewed which does
not have high N-gram overlap with the query either:

December 8, 1946 — Total Lunar Eclipse — Rawaki, Phoenix Islands, Kiribati
Max view in Rawaki
Sunday, December 8, 1946 at 5:01 AM
Global Type: Total Lunar Eclipse
Rawaki: Partial Lunar Eclipse
Began: Sun, Dec 8, 1946 at 3:13 AM
Maximum: Sun, Dec 8, 1946 at 5:01 AM
Ended: Sun, Dec 8, 1946 at 8:22 AM
Duration: 5 hours, 10 minutes
December 8, 1946 — Total Lunar Eclipse — Rawaki
You are using an outdated browser, to view the animation please update or switch
to a modern browser. Alternatively you can view the old animation by clicking
here.
Animation: How the Partial Lunar Eclipse Looked
The total phase of this lunar eclipse was not visible in Rawaki, but it could be
observed there as a partial lunar eclipse.
More about the December 8, 1946 — Total Lunar Eclipse
Phases and local times of this eclipse
Eclipses visible in Rawaki
All eclipses worldwide, from 1900 to 2100

This is the only query for which we observe an unrelated top 1 document, but for the 35B model we
qualitatively observed seemingly irrelevant documents in the rankings more often (in the 7B we did
not observe this). This connects to a finding from literature that for large models influence functions
sometimes surface documents with high gradient norms that are unrelated to the query (Barshan
et al., 2020; Grosse et al., 2023; Choe et al., 2024). As Grosse et al. (2023) note, it is currently
unclear whether this is true noise, or whether these are genuinely influential for the completions.
Regardless, it seems like noise cannot easily explain the difference between the factual and slopes
queries, as one would expect noise to show up equally everywhere.

Another way to visualise this result is to plot the percentage of total influence contained in different
parts of the top ranking, which we do in Figure 22 below. The results in this plot show that for the
top-k percentile of most positively influential documents, the total percentage of positive influence is
much higher than k (e.g. 20% of the total positive influence is contained in the top 5% of documents).
Here, it is clear that on average, for the 35B model the total amount of influence contained in the
top-k percentile increases faster for reasoning questions than for factual questions, indicating that
a larger portion of the total positive influence is contained in the top portions of the rankings. In
Figure 23 we show the same result holds if we include the control queries. As Grosse et al. (2023),
it is not clear whether this is a sensible result to show because for each query we are dividing the
total influence at each k by the sum of positive influence for that query (perhaps a large part of the
positive influence gets cancelled out by negative influence), but we show the result here nonetheless
for completeness. We know from the absolute results of the total influence at different portions of
the ranking that each percentage of total influence at the top-k percentile a much lower value in
absolute terms for reasoning than for the factual questions. If the relative result does not turn out to
be noise, it is the case that of the total influence, a higher percentage is contained in the top portions

61

3240
3241

3242
3243
3244
3245
3246
3247

3248
3249
3250
3251
3252
3253

3254
3255
3256
3257
3258

3259
3260
3261
3262
3263
3264

3265
3266
3267
3268
3269
3270

3271
3272
3273
3274
3275
3276

3277
3278
3279
3280
3281

3282
3283
3284
3285
3286
3287

3288
3289
3290
3291
3292
3293

Under review as a conference paper at ICLR 2025

of the rankings for reasoning questions than for factual questions. Taken together with the fact that
the absolute influence is often much higher for factual questions, this indicates that the model relies
on more highly influential documents for factual retrieval than for reasoning. This could indicate
that there are more highly relevant factual documents further down the ranking, which makes sense
given the fact that the pretraining distribution is dominated by websources and news, which are more
likely to contain relevant information for factual question answering than for reasoning. Further,
it connects to the finding from literature that models need to see examples often before text gets
memorised (Chowdhery et al., 2022).

Figure 22: The percentage of total influence per nat of query completion information for different
portions of the positive ranking over documents, left for the 7B model, right for the 35B. We plot
only non-control queries.

Figure 23: The percentage of total influence per nat of query completion information for different
portions of the positive ranking over documents, left for the 7B model, right for the 35B. We plot
all queries, including the query control sets for both factual and reasoning, which contain 10 queries
each.

Again, the picture looks similar for the negative portions of the ranking, shown for completeness
below in Figure 24 and 25.

62

3294
3295

3296
3297
3298
3299
3300
3301

3302
3303
3304
3305
3306
3307

3308
3309
3310
3311
3312

3313
3314
3315
3316
3317
3318

3319
3320
3321
3322
3323
3324

3325
3326
3327
3328
3329
3330

3331
3332
3333
3334
3335

3336
3337
3338
3339
3340
3341

3342
3343
3344
3345
3346
3347

Under review as a conference paper at ICLR 2025

Figure 24: The percentage of total influence per nat of query completion information for different
portions of the negative ranking over documents, left for the 7B model, right for the 35B. We plot
only non-control queries.

Figure 25: The percentage of total influence per nat of query completion information for different
portions of the negative ranking over documents, left for the 7B model, right for the 35B. We plot
all queries, including the query control sets for both factual and reasoning, which contain 10 queries
each.

63

3348
3349

3350
3351
3352
3353
3354
3355

3356
3357
3358
3359
3360
3361

3362
3363
3364
3365
3366

3367
3368
3369
3370
3371
3372

3373
3374
3375
3376
3377
3378

3379
3380
3381
3382
3383
3384

3385
3386
3387
3388
3389

3390
3391
3392
3393
3394
3395

3396
3397
3398
3399
3400
3401

