MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
Under review as a conference paper at ICLR 2025

MIRAGE: EVALUATING AND EXPLAINING INDUCTIVE
REASONING PROCESS IN LANGUAGE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Inductive reasoning is an essential capability for large language models (LLMs)
to achieve higher intelligence, which requires the model to generalize rules from
observed facts and then apply them to unseen examples. We present MIRAGE, a
synthetic dataset that addresses the limitations of previous work, specifically the
lack of comprehensive evaluation and flexible test data. In it, we evaluate LLMs’
capabilities in both the inductive and deductive stages, allowing for flexible vari-
ation in input distribution, task scenario, and task difficulty to analyze the factors
influencing LLMs’ inductive reasoning. Based on these multi-faceted evaluations,
we demonstrate that the LLM is a poor rule-based reasoner. In many cases, when
conducting inductive reasoning, they do not rely on a correct rule to answer the
unseen case. From the perspectives of different prompting methods, observation
numbers, and task forms, models tend to consistently conduct correct deduction
without correct inductive rules. Besides, we find that LLMs are good neighbor-
based reasoners. In the inductive reasoning process, the model tends to focus on
observed facts that are close to the current test example in feature space. By lever-
aging these similar examples, the model maintains strong inductive capabilities
within a localized region, significantly improving its reasoning performance.

1

INTRODUCTION

Inductive reasoning, known as the ability of an intelligent agent to infer abstract rules from limited
observations and apply them to new examples, is crucial for large language model (LLMs) pro-
gressing toward artificial general intelligence (AGI) (Xu et al., 2024b; Sun et al., 2024; Wang et al.,
2024b). As illustrated in Figure 1, given a set of observed facts, inductive reasoning process expect
the model to generate abstract rules from the provided facts (i.e. [A,B,C] → [B+C,B+C,C] in the
rule induction task) and apply these rules to answer specific new questions (i.e. [3,4,7] → [11,11,7]
in the example inference task). Despite its significant research value, it has been relatively neglected
compared to other types of reasoning (e.g., math reasoning, multi-hop reasoning, etc.).

Recently, some works have started to explore this problem. They primarily evaluate the model’s
inductive reasoning capabilities using various datasets (Shao et al., 2024; Cheng et al., 2024; Qiu
et al., 2024; Jiang et al., 2024). Though they have made great progress, their works still have two
main limitations: (1) Previous works lack comprehensive evaluation. Most works have only one
evaluation task: the inductive task on collected rules (Yang et al., 2024b; Shao et al., 2024) or the
deductive task on specific test samples (Chollet, 2019; Xu et al., 2024a; Qiu et al., 2024). There-
fore, they can only evaluate the rule induction performance or final results of inductive reasoning,
instead of comprehensively analyzing the whole process (i.e. inductive + deductive). (2) Previous
works lack flexible test data. Most former datasets evaluate the overall performance of models by
collecting observation and test examples under the same rules (Rule, 2020; Kim et al., 2022; Lake
et al., 2019). However, due to the absence of transformation rules, it is impossible to extend these
examples, resulting in a fixed test set. This limitation makes it challenging to assess the impact of
factors such as distribution, quantity, and form of input examples on the model’s inductive reasoning,
thereby hindering a deeper analysis of the model’s reasoning mechanisms.

In this paper, we present MIRAGE (Meta Inductive ReAsoning Evaluation), a dataset designed to ad-
dress the two aforementioned limitations. It includes both inductive and deductive evaluation tasks,
while offering flexibility to construct test data with various forms, arbitrary input distributions, and

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: An overview of two paradigms (i.e. rule-based and neighbor-based) in inductive reasoning.

controllable difficulties. In detail, we first construct a rule library based on various vector operations
(e.g., [A,B,C] → [B+C,B+C,C] as shown in Figure 1). Using the automatically synthesized rules,
we can generate facts arbitrarily through instantiation, ensuring the flexibility and scalability of the
test data. Next, we filter out the noise data (e.g. duplicated facts) to further improve the effectiveness
and quality of our dataset. Finally, to comprehensively evaluate the inductive reasoning process, we
not only design inductive and deductive questions based on the synthesized data but also construct
diverse application scenarios for these tasks, including list transformations, real-world problems,
code generations, and string transformations (as shown in Figure 2).

Based on our dataset, we perform a deeper analysis of the model’s inductive reasoning process,
from which we draw two new conclusions about the inductive reasoning mechanisms of LLMs: (1)
Language models are poor rule-based reasoners. As shown in the left column of Figure 1, in the
rule-based reasoning paradigm, inductive reasoning involves first deriving the correct rule through
the observation of examples and then using the inductive rule to answer new questions (like what
humans do). However, we find that LLMs perform poorly in this paradigm: In many cases, though
they can not induce a correct rule, they can still perform well on example inference tasks. Through
experimentation, we observe this performance gap between induction and deduction across different
prompting methods, models, observed example numbers, and scenarios. This indicates that the final
performance of the LLM’s inductive reasoning rarely relies on the intermediate inductive rules. (2)
Language models are good neighbor-based reasoners. Furthermore, we identify an important
mechanism behind LLM’s inductive reasoning, which we refer to as “neighbor-based reasoning”:
If some observed facts are close to the test examples in feature space, the model tends to leverage this
similarity to improve inductive reasoning performance. For example, as shown in the right column
of Figure 1, even when the model cannot generate the correct rule, it can rely on the neighbor fact
[3,3,7] → [10,10,7] (here the distance between [3,3,7] and [3,4,7] is small, so we refer to them
as neighbors) to successfully performs the reasoning. We demonstrate that this paradigm persists
across different scenarios, models, and observed example numbers. However, it can only enhance
the performance within a localized scope.

To sum up, the main contributions of our work are as follows: (1) We present a new dataset MI-
RAGE, through it, we can comprehensively evaluate the LLM’s inductive reasoning process under
(2) We find that LLM is a poor rule-based inductive reasoner. In many
more flexible settings.
cases, it does not rely on inductive rules to make correct deductions. (3) We demonstrate that
LLM is a neighbor-based inductive reasoner. When performing inductive reasoning, models rely
on the neighbor facts in the observed fact set to get better performance.

2 DATA CONSTRUCTION

In this section, we describe the whole pipeline to build MIRAGE. We start by constructing rules
based on five basic operations (§2.1). Next, we substitute the instantiate vectors into the rules to

2

[0,1,3] → [4,4,3]     [5,2,5] → [7,7,5]     [2,1,8] → [9,9,8]     [9,6,3] → [9,9,3]      [3,3,7] → [10,10,7] Observed FactsRule Induction TaskRule-basedExample Inference Task[A,B,C] → [B+C,B+C,C]What is the rule?Answer: [11,11,7][3,4,7] → ?Answer: [11,11,7]Neighbor-basedRule: [A,B,C] → [B+C,B+C,C]Case: [3,3,7] → [10,10,7][A,B,C] → [A+4,B+1,C][3,4,7] → ?What is the rule?Under review as a conference paper at ICLR 2025

108
109
110
111

112
113
114
115
116
117

118
119
120
121
122

123
124
125
126
127
128

129
130
131
132
133
134

135
136
137
138
139
140

141
142
143
144
145

146
147
148
149
150
151

152
153
154
155
156
157

158
159
160
161

generate facts (§2.2) and apply filtering to them (§2.3). Finally, we transform the facts into different
scenarios, creating questions to evaluate the LLM’s inductive reasoning performance (§2.4).

2.1 RULE GENERATION

According to previous work and relevant definitions (Huber, 2017; Han et al., 2024), in inductive
reasoning, for each observed fact Xk = (x, y), the input vector x is transformed into the output
vector y according to a certain rule f , i.e.:

f (x) = y,

∀(x, y) ∈ X

(1)

where X is the observed fact set under the rule f . We believe that f is the core of the problem, as it allows
us to generate additional facts for X based on the rule automatically. Conversely, inferring f from X requires
significantly more effort due to the vast range of possible rules. Therefore, we first consider automating these
rules’ large-scale synthesis.

Based on previous representative datasets (Chollet, 2019; Rule, 2020; Xu et al., 2024a), we summarize the
main types of rules, resulting in five atomic operations in this dataset: (1) Add: The operation adds certain
components together. For example: [x, y, z] → [x, x+y, z]. (2) Copy: The operation copies some components
to others. For example: [x, y, z] → [x, x, z]. (3) Map: The operation applies a linear transformation to some
components. For example: [x, y, z] → [x, ky + b, z]. To avoid the interference of complex math calculations,
we have k ∈ [1, 9] and b ∈ [0, 9]. (4) Pad: The operation fills certain components with constant values. For
example: [x, y, z] → [x, c, c], where c ∈ [0, 9]. (5) Swap: The operation swaps certain components. For
example: [x, y, z] → [z, y, x].

For each operation O, we randomly initialize the set index vector d on which the operation applies and the
index vector r where the result is output. Specifically, for x ∈ x, y ∈ y:

yj =

(cid:26) [O(xd)]i,
xj,

if j ∈ r
if j /∈ r

(2)

where ri = j and [·]i represents the i-th component. Therefore, we can generate a meta-rule f = (O, d, r).
Through sampling (O, d, r) randomly, we can construct a meta-rule library F.

2.2 FACT GENERATION

After generating the rule library, we can randomly initialize x, and apply a specific rule f ∈ F to get y. We
repeat this process to generate the fact set X under the rule f . All the (x, y) ∈ X are used for the LLM to
induce the rule f . It is worth noting that we can control the inductive difficulty by adjusting two factors: the
dimension D of x, y and the fact number N of X. As an example, in Figure 1, D is 3 and N is 5. Empirically,
a higher D and a smaller N tend to increase the task difficulty. Additionally, to avoid the interference of
complex mathematical calculations in evaluating inductive reasoning ability, we restrict the elements in each x
to integers between 0 and 9.1 Since we can synthesize any D-dimensional vector x to construct a fact, we can
flexibly control the input distribution.

2.3 DATA FILTERING

To ensure the quality of the dataset and the effectiveness of the evaluation, we need to filter out some noisy
data. The following filtering steps are applied: (1) Filtering out duplicate facts. For any two facts in X, if
their input vectors x are identical, one of them is removed and resampled. This ensures that for each rule, all
observed facts are unique. (2) Filtering out duplicate rules. To ensure diversity in the evaluation, we also
remove duplicate rules, which have the same (O, d, r). (3) Filtering out trivial facts. After random sampling,
X may include some trivial facts that provide little value for model induction, such as facts like x = y, x = 0,
or y = 0. We filter the data to ensure that each X contains at most one trivial fact, thereby limiting the noise
that could affect the model’s inductive reasoning process.

2.4 QUESTION GENERATION

So far, we have constructed all the metadata that we need to generate specific questions. It is worth noting that
both F and X contain only abstract rules and facts, without any specific context. Therefore, they represent the
fundamental inductive reasoning test data, which is why we refer to them as meta-rules and meta-facts. As
shown in Figure 2, to evaluate the practical inductive reasoning capability of models, we apply these metadata

1Our pilot experiments indicate that, under these constraints, most of the models can achieve an accuracy

of nearly 100% in performing purely mathematical operations. See Appendix A.2 for details.

3

Under review as a conference paper at ICLR 2025

162
163
164
165

166
167
168
169
170
171

172
173
174
175
176

177
178
179
180
181
182

183
184
185
186
187
188

189
190
191
192
193
194

195
196
197
198
199

200
201
202
203
204
205

206
207
208
209
210
211

212
213
214
215

Figure 2: Examples in four different scenarios of MIRAGE.

to various scenarios to generate concrete problems. Specifically, we have: (1) List Transformation (LT): List
transformation is the primary format used in previous inductive reasoning tasks (Rule, 2020; Xu et al., 2024a;
Chollet, 2019), and here we adopt this approach as well. We transform all fact vectors into one-dimensional
lists and require the model to inductively infer the transformation rules applied to these lists. (2) Real-world
Problem (RP): Previous datasets lack tests for inductive reasoning capabilities in real-world scenarios (Rule,
2020; Xu et al., 2024a; Qiu et al., 2024).2 To mitigate this gap, we populate the metadata into different natural
language templates across five real-life scenarios. The example in Figure 2 describes a trading scenario, where
we use different items to represent different dimensions of the vector. All item transactions follow the same
rule. (3) Code Generation (CG): For each fact, we use x as the input and y as the output of a function. The
model is then tasked with predicting the corresponding Python function. (4) String Transformation (ST):
The former three scenarios are related to numbers. Here, we replace the basic elements in the fact vectors with
characters to conduct a new test. Notably, we modify the operations as follows: addition in the Add and Map
operations is replaced with string concatenation, multiplication in Map is replaced with character replication,
zero-padding in Pad becomes character deletion, and the numbers 0-9 are replaced with the characters a-j.

For humans, although the process of reasoning tends to be implicit, according to the traditional definition in
logic, it can be divided into two stages: induction and deduction. Here, we focus on the objectives of these two
stages: deriving the correct rules through induction and making inferences on new instances using deduction.
We aim to explore the correlation between the two during the reasoning process of LLMs. Therefore, for each
scenario, we design two tasks: rule induction (RI) and example inference (EI), defined as follows:

• Rule Induction Task: Given an observed fact set X, this task evaluates the model’s accuracy in
inducing transformation rules f . It aims to evaluate the model’s proficiency in mastering intermediate
rules during inductive reasoning (Yang et al., 2024b; Shao et al., 2024).

• Example Inference Task: Given an observed fact set X, the task provides an unseen example input
xt as input and measures the accuracy of the predicted yt. It aims to evaluate the final performance
of the model’s inductive reasoning (Chollet, 2019; Xu et al., 2024a; Qiu et al., 2024).

We provide all the prompts used for these tasks in Appendix A.3.

3 LANGUAGE MODELS ARE POOR RULE-BASED REASONERS

3.1 OVERALL PERFORMANCES ON MIRAGE

Setup We first evaluate the overall performance of various LLMs on MIRAGE. Here, we select GPT-4
(OpenAI, 2023), GPT-4o, Claude-3.5, Llama3-8B (Dubey et al., 2024), and Llama2-13B (Touvron et al., 2023)
as representative models.3 For the first three models, given their strong instruction-following capabilities, we
provide only the instruction and allow them to answer the questions in a zero-shot setting. For the latter two
models, to improve the format accuracy of the response, we additionally provide five examples before they
answer the questions. Unless otherwise specified, we continue to use this setup to prompt the model in the
subsequent experiments. For the dataset setting, we fix the size N at 5 and measure performance across four
scenarios when the dimension D = 3, 5, 8. We sample 500 questions for each test. More implementation details
can be found in Appendix B.1.

Results The results are shown in Table 1, from which we can draw the following conclusions: (1) LLMs’
inductive reasoning does not rely on rule induction. Given the same set of observed facts, the model’s
performance on rule induction is noticeably worse than on example inference in almost all cases. This suggests

2Here, real-world scenarios refer to mathematical inductive reasoning within natural language contexts.
3Due to the frequency limitations of API calls, we can not conduct our evaluation on the latest o1 model.

4

Meta Rule[A,B,C] -> [B+C, B+C, C]Meta Fact[3,3,6] -> [9, 9, 6]Real-world ProblemRule:Jack went to trade items. He originally had A beds, B bags, C chairs. After the trade, he has B+C beds, B+C bags, C chairs.Jack went to trade items. He originally had 3 beds, 3 bags, 6 chairs. After the trade, he has 9 beds, 9 bags, 6 chairs.Fact:List TransformationRule:Fact:[A,B,C] -> [B+C,B+C,C][3,3,6] -> [9,9,6]Code GenerationRule:def f(A,B,C):       A,B,C = B+C, B+C, C     return A,B,CFact:f(3,3,6) = 9,9,6String TransformationRule:ABC -> BCBCCFact:ddg -> dgdggUnder review as a conference paper at ICLR 2025

Model

Task

D=3

D=5

D=8

Llama2-13B

Llama3-8B

GPT-4o

GPT-4

Claude-3.5

LT

0.01
0.26

0.15
0.30

0.41
0.68

0.47
0.68

0.44
0.79

RP

0.00
0.11

0.11
0.15

0.32
0.37

0.29
0.37

0.35
0.45

CG

0.00
0.25

0.19
0.25

0.38
0.61

0.41
0.61

0.34
0.62

ST

0.03
0.22

0.19
0.25

0.32
0.56

0.28
0.57

0.46
0.58

LT

0.01
0.13

0.23
0.20

0.35
0.58

0.58
0.63

0.22
0.65

RP

0.01
0.03

0.04
0.12

0.21
0.25

0.22
0.29

0.20
0.33

CG

0.00
0.14

0.14
0.25

0.44
0.64

0.56
0.71

0.38
0.76

ST

0.21
0.25

0.22
0.29

0.30
0.39

0.27
0.44

0.33
0.45

LT

0.00
0.06

0.16
0.09

0.33
0.42

0.46
0.42

0.24
0.46

RP

0.01
0.01

0.02
0.11

0.16
0.17

0.15
0.21

0.13
0.24

CG

0.00
0.06

0.08
0.16

0.41
0.49

0.45
0.64

0.38
0.59

ST

0.10
0.19

0.21
0.24

0.24
0.29

0.23
0.30

0.26
0.30

RI
EI

RI
EI

RI
EI

RI
EI

RI
EI

Table 1: Overall performance (accuracy) of different models on MIRAGE. The best results in rule
induction (RI) are in bold, while the best results in example inference (EI) are underlined.

that most of the model’s correct deductions do not depend on inducing a correct rule. (2) LLMs face difficulties
in handling inductive reasoning in real-world problems. When comparing different scenarios, all models
perform the worst on the RP tasks. For example, GPT-4o only achieves 0.16 and 0.17 accuracy when the
dimension is 8. This indicates that, compared to purely symbolic forms (LT, CG, ST), natural language forms
pose a greater challenge for the models’ inductive reasoning abilities.

Model

Rule Induction

Example Inference

Supplementary Experiments
In the main experiments, we find that there is a significant performance
gap between rule induction and example inference for LLMs. However, this gap may be caused by the dif-
ference in difficulty between the two tasks. When the model is unable to perform correct inductive reasoning,
it is likely to guess the correct answers
for the EI task more easily compared to
the RI task, resulting in a higher accu-
racy. We conduct this additional experi-
ment to eliminate the interference of this
factor. Specifically, we randomly perturb
one fact in X to violate rule f . Then,
we observe the performance of both tasks
and calculate the change rate (CR) of
accuracy before and after the perturba-
tion. CR represents the sensitivity of the
model’s performance to the input. If CR
is high, it indicates a strong correlation between task performance and input, making it difficult to answer the
question correctly through random guessing, Therefore, CR can serve as an indicator of the reasoning difficulty
for the task. We randomly choose 100 pieces of test data from the dataset and generate questions under the
LF scenario.4 The experimental results on different models are demonstrated in Table 2. We can observe that
the two tasks have comparable CR for both models, indicating that the reasoning difficulty of the EI task is not
lower than the RI task. The tasks themselves do not cause such a large performance gap.

Table 2: Comparison of CR on two tasks (D = 3, N = 3). BF
and AF indicate the accuracy before and after perturbation.

GPT-4o
Claude-3.5

0.74
0.81

0.15
0.22

0.50
0.37

0.77
0.66

0.66
0.65

0.13
0.07

CR

CR

AF

AF

BF

BF

3.2 PERFORMANCES OF ADVANCED METHODS

In §3.1, we observe that LLMs perform poorly on our dataset, especially in inductive tasks. Considering
previous work has proposed numerous methods to elicit the model’s reasoning abilities (Wei et al., 2022; Wang
et al., 2023b; Madaan et al., 2023), we wonder whether they can boost models’ performance on MIRAGE.

Setup Since we focus on exploring the model’s intrinsic capabilities, we only consider methods that do
not introduce any external tools or knowledge. Specifically, the methods are as follows: Input-Output (IO):
We prompt models to generate answers directly under different shots. Inductive-Deductive (ID): We prompt
models to generate rules for inductive tasks and apply them to answer questions in deductive tasks. Chain-of-
Thought (CoT) (Wei et al., 2022): We prompt models to generate rationales and answers for the two tasks.
Self-Consistency (SC) (Wang et al., 2023b): Based on CoT, we sample n rationales and use the major voting
strategy to predict the final answer. Self-Refine (SR) (Madaan et al., 2023): We prompt the model to provide
feedback on the generated rules, and then refine the rules based on that feedback (with a maximum of t itera-
tions). After the iteration stops, we use the latest rule to answer the inductive task and apply it to answer the

4Unless otherwise specified, this configuration will be maintained for all subsequent experiments.

5

216
217
218
219

220
221
222
223
224
225

226
227
228
229
230

231
232
233
234
235
236

237
238
239
240
241
242

243
244
245
246
247
248

249
250
251
252
253

254
255
256
257
258
259

260
261
262
263
264
265

266
267
268
269

Under review as a conference paper at ICLR 2025

270
271
272
273

274
275
276
277
278
279

280
281
282
283
284

285
286
287
288
289
290

291
292
293
294
295
296

297
298
299
300
301
302

303
304
305
306
307

308
309
310
311
312
313

314
315
316
317
318
319

320
321
322
323

Method

IO (0-shot)
IO (5-shot)

ID (0-shot)
ID (5-shot)

CoT (0-shot)
CoT (5-shot)

SC (n=5)

SR (t=3)

HR (t=3, n=1)
HR (t=3, n=5)

RI

0.46
0.63

0.46
0.59

0.50
0.56

0.59

0.48

0.56
0.66

LT

EI

0.76
0.76

0.56
0.68

0.57
0.59

0.74

0.64

0.68
0.79

(∆)

0.30
0.13

0.11
0.08

0.07
0.04

0.15

0.16

0.12
0.13

RI

0.43
0.59

0.46
0.57

0.47
0.41

0.49

0.42

0.45
0.59

RP

EI

0.72
0.77

0.57
0.66

0.55
0.55

0.62

0.67

0.71
0.80

(∆)

0.28
0.17

0.11
0.08

0.08
0.13

0.14

0.25

0.26
0.21

RI

0.39
0.55

0.33
0.47

0.34
0.37

0.38

0.36

0.41
0.55

CG

EI

0.46
0.54

0.42
0.54

0.39
0.40

0.45

0.49

0.53
0.60

(∆)

0.08
-0.02

0.09
0.08

0.05
0.03

0.07

0.13

0.11
0.05

RI

0.47
0.52

0.22
0.48

0.52
0.45

0.57

0.53

0.43
0.49

ST

EI

0.70
0.78

0.65
0.69

0.62
0.64

0.68

0.67

0.71
0.67

(∆)

0.23
0.26

0.43
0.21

0.10
0.20

0.10

0.14

0.27
0.18

Table 3: Performance of different methods on MIRAGE using GPT-4o. The best results in each
column are highlighted in bold, while the second best results are underlined.

deductive task. Hypothesis Refinement (HR) (Qiu et al., 2024): HR is an optimized version of SR, which first
generates n rules. In each iteration, we apply the current rules to all observed examples, compare the actual
output with the expected output, and get the number of correct predictions along with the information about
incorrect examples. If a candidate rule is correct for all observed facts, it is immediately returned. Otherwise,
the rule with the highest number of correct predictions and the associated error information is used as input for
the model to refine, generating n rules for the next round, until the maximum number of iterations t is reached.
We sample 200 questions for each test.

Results We demonstrate the experimental results on GPT-4o in Table 3 and report other results in Appendix
B.2. We can conclude that: (1) Advanced methods provide limited improvement to the model’s inductive
reasoning ability or may even have negative effects. For both tasks, directly answering with few-shot settings
can consistently achieve the highest or second-highest accuracy in most cases. After applying prompting meth-
ods like CoT, the model’s accuracy decreases by up to 18% and 22% on two tasks, respectively. It indicates
that the key to optimizing inductive reasoning does not lie in refining the intermediate inductive process (as
CoT-like methods do). (2) The model’s disregard for abstract rules during inductive reasoning is method-
agnostic. Although some methods use instructions to guide the model to focus more on the previously induced
rules during reasoning (e.g. ID, SR, HR), there remains a significant gap in the model’s RI and EI performance.
For example, in the case of SR, the model’s example inference accuracy outperforms its rule induction accuracy
by an average of 16%.

3.3

IMPACT OF INCREASING FACT SIZE

In the previous experiments, we consistently fix the observed fact numbers N . Therefore, as a supplement, we
explore the impact of N on the model’s inductive reasoning process in this section. Theoretically, as the number
of observed facts increases, the scope of the candidate rules narrows, which can lead to the incorrect inductive
process becoming correct. If the reasoning process is rule-based, the model is likely to generate the correct rule
(inductive) before applying it correctly (deductive). In other words, the time when the LLM induces the correct
rule is no later than the time it performs the correct deduction. Thus, the cumulative number of observations
required for the inductive rule to change from incorrect to correct should not exceed the number required for
the test case to become correct. We design this experiment to validate whether it holds on the LLM.

Setup Given a fact set Xk of size k, and a fixed test input xt, we define the inductive correction threshold
(ICT) and deductive correction threshold (DCT) as follows:

ICT = k ⇐⇒ ∀i < k, M(Xi|I) ̸= f ∧ M(Xk|I) = f
DCT = k ⇐⇒ ∀i < k, M(Xi, xt|D) ̸= f (xt) ∧ M(Xk, xt|D) = f (xt)

(3)

(4)

Here, M(·|I), M(·|D) are the model’s outputs in RI and EI tasks. We set D = 5 and vary N , then analyze the
distribution of these two thresholds across 100 samples, reporting the results in Figure 3.

Results Based on the result, we further demonstrate that LLM’s deduction does not rely on an inductive
rule. From both of the two figures, we can observe that most points are distributed in the upper left region
of the line x = y, indicating that for the vast majority of cases, DCT is smaller than ICT. Therefore, the fact
numbers N does not affect the conclusion we stated earlier. LLM requires fewer facts to successfully perform
an example inference task compared to correct induction.

6

Under review as a conference paper at ICLR 2025

324
325
326
327

328
329
330
331
332
333

334
335
336
337
338

339
340
341
342
343
344

345
346
347
348
349
350

351
352
353
354
355
356

357
358
359
360
361

362
363
364
365
366
367

368
369
370
371
372
373

374
375
376
377

(a) GPT-4o

(b) Claude-3.5

(a) GPT-4o

(b) Claude-3.5

Figure 3: The distribution of ICT and DCT for
the examples across different models.

Figure 4: Performance on EI tasks under different
scenarios of observed and test facts.

3.4 TRANSFERABILITY TEST OF INDUCTIVE RULES

Finally, we investigate the impact of different scenarios on the inductive reasoning process. For rule-based
reasoning, once a rule is formed through induction, it should be transferable. That is, a rule induced in one
scenario should be applicable to another scenario with the same underlying transformation. We experiment to
explore whether LLMs possess this ability when performing inductive reasoning.

Setup Specifically, we exclude ST in this experiment since its basic transformations differ from the other
three scenarios (see §2.1). For the remaining three scenarios, we generate the observed facts in one scenario,
and then transform the test case into another scenario. Since our dataset can generate questions in different
scenarios based on the same meta-rule, we can easily ensure that they share the same underlying transformation.

Results From the results shown in Figure 4, we can get that: (1) LLMs lack transferability in inductive
reasoning. Across different cases, the highest performance occurs when the scenarios of the observed and
test facts are consistent (i.e., the diagonal from the top left to the bottom right in the figure).(2) The inductive
reasoning process of the LLM is form-related. Compared to the transfer between LT and RP (or CG and RP),
the transfer between LT and CG demonstrates better performance. We infer that this is because the forms of LT
and CG are more similar (see Figure 2). Based on the above two observations, we further confirm that LLMs do
not rely on abstract rules when performing inductive reasoning. So, what is the underlying mechanism behind
it? In the following section, we focus on addressing this question.

4 LANGUAGE MODELS ARE GOOD NEIGHBOR-BASED REASONERS

4.1 MOTIVATION

From § 3.4, we know that closer forms between the observed facts and the test case can enable the model to
perform inductive reasoning more effectively. However, is the positive impact brought by the similarity limited
only to the form? The answer to this question is likely “No”. Upon reviewing related works, we find that
models tend to match various similar patterns in the context and use them to predict the next token (Olsson
et al., 2022; Wang et al., 2023a; Hu et al., 2024b). Therefore, we aim to identify a metric to measure some
other similarities between the observed facts and the test input. Since all of our facts are transformed from
vectors, we associate this similarity with the distance between these facts in feature space.

In topology, if f : X → Y is a continuous function between two Euclidean spaces and N (x0, ϵ) is a ϵ-
neighborhood of the point x0 in X, then we have:

∃η > 0, s.t. ∀x ∈ N (x0, ϵ), f (x) ∈ N (f (x0), η)

(5)

If a fact input vector x closes
In other words, continuous functions preserve the neighborhood property.
to the test input xt, then their output vectors y and yt will remain close.5 Therefore, the close distance
between yt and y may allow LLM to predict yt based on y in observed facts without the need for correct
rule generation. In the following sections, we demonstrate through experiments that the model’s inductive
reasoning relies on this paradigm, which we refer to as neighbor-based reasoning.

4.2 NEIGHBOR FACTS IN INDUCTIVE REASONING

Before conducting the experiments, we first define some key concepts in our work: the distance d and neighbor-
hood N . In our setup, the components at corresponding positions in the vectors follow the same transformation

5We rigorously prove in Appendix C.1 that the rules f in our dataset are all continuous functions.

7

'&7,&7)UHTXHQF\'&7,&7)UHTXHQF\/)53&*2EVHUYHG/)53&*7HVW/)53&*2EVHUYHG/)53&*7HVWUnder review as a conference paper at ICLR 2025

(a) GPT-4o

(b) Claude-3.5

(c) Llama3-8B

Figure 5: Performance of different fact types on our dataset (D = 3, N = 5). The dashed line
represents the baseline accuracy.

Type

Baseline
IF Only
CF Only
OF Only

LT

0.52
0.78
0.48
0.46

N=3

RP

0.19
0.46
0.25
0.18

CG

0.78
0.82
0.59
0.50

ST

0.42
0.59
0.43
0.38

LT

0.66
0.84
0.69
0.49

N=5

RP

0.36
0.52
0.35
0.23

CG

0.71
0.84
0.72
0.57

N=8

ST

0.46
0.63
0.50
0.36

LT

0.76
0.86
0.75
0.61

RP

0.34
0.54
0.34
0.23

CG

0.80
0.91
0.82
0.67

ST

0.54
0.70
0.53
0.38

Table 4: Performance of different fact types under various settings (D = 5). The best results in each
column are highlighted in bold, while the worst results are underlined.

rules, while non-corresponding components may undergo different transformations (see Equation 2). Hence,
we consider using the distance based on the corresponding components: Chebyshev distance.6 Given observed
fact Xi = (xi, yi) and test input xt, we have:

k
where xik and xtk are the k-th component of two input vectors. Then we can define the ϵ-neighborhood of xt
based on the distance:

d(Xi, xt) = max

(|xik − xtk|)

(6)

N (xt, ϵ) = {Xi | d(xi − xt) ≤ ϵ}

(7)

Setup According above definitions, we can divide an observed fact Xk into three categories based on the
distance between x and the test input xt: (1) In-neighborhood Fact (IF): If Xk ∈ N (xt, ϵ), we call Xk is a in-
neighborhood fact. (2) Cross-neighborhood Fact (CF): If Xk /∈ N (xt, ϵ), but ∃i ∈ [1, D], s.t. |xi −xti| ≤ ϵ,
we consider it a suboptimal neighbor fact because some of its components can still contribute to the model’s
inductive reasoning process. In this case, we call Xk is a cross-neighborhood fact. (3) Out-neighborhood Fact
(OF): If ∀i ∈ [1, D], |xi − xti| > ϵ, we call Xk is an out-neighborhood fact. By generating examples based
on these definitions, we can make the fact set X contain only one type of fact. After constructing different fact
sets, we compare the model’s performance on EI tasks under these settings. Besides, we use the performance
under the default fact set as the baseline, where all facts are randomly sampled without any constraints.

Results We report the experimental results in Figure 5. It demonstrates that: (1) LLM’s inductive reasoning
is neighbor-based. By comparing the three settings, we find that observed facts closer to the test case result in
better performance (IF > CF > OF) across all of the models. Besides, compared to the baseline (the dashed
line in figures), the accuracy significantly drops after we remove all the neighbor cases in X (i.e. OF). These
phenomena indicate that the model heavily relies on neighbor facts during reasoning. (2) LLMs have a strong
ability to capture neighboring patterns. When we set the neighborhood radius ϵ to 4, both IF and CF still
contribute to high reasoning performances for the model. Besides, OF continues to show a significant decline
(compared to ϵ = 3). These observations indicate that LLMs can still learn similar patterns even when the
observed facts are relatively distant.

4.3 UNIVERSALITY OF NEIGHBOR-BASED REASONING

We consider whether LLM’s inductive reasoning universally relies on neighbor cases, hence, we set ϵ to 1 and
repeat the experiment under more different settings, where the baseline is the same as we set in § 4.2. The

6We demonstrate through experiments that this distance is more suitable for constructing neighborhoods

compared to other distances. For details, see Appendix C.2.

8

378
379
380
381

382
383
384
385
386
387

388
389
390
391
392

393
394
395
396
397
398

399
400
401
402
403
404

405
406
407
408
409
410

411
412
413
414
415

416
417
418
419
420
421

422
423
424
425
426
427

428
429
430
431

$FFXUDF\,)&)2)$FFXUDF\,)&)2)$FFXUDF\,)&)2)Under review as a conference paper at ICLR 2025

(a) D = 3, ϵ = 1

(b) D = 3, η = inf

(c) D = 5, ϵ = 1

(d) D = 5, η = inf

Figure 7: Deductive Density (Id) of various fact types on GPT-4o under different test radius η and
neighborhood radius ϵ (N = 5). BL represents the performance of the baseline, where we use default
fact sets with no substitution.

results on GPT-4o are reported in Table 4, which demonstrates that the neighbor-based paradigm is universal
in LLMs’ inductive reasoning process. Across different scenarios and fact numbers, IF consistently gets the
highest accuracy, while OF gets the lowest accuracy. The reliance of LLMs’ inductive reasoning on neighbor
facts is independent of the specific task scenarios, models, or fact numbers. For more results on other models,
you can see Appendix C.3.

4.4 EFFECTIVE SCOPE OF NEIGHBOR-BASED REASONING

We have demonstrated that neighbor exam-
ples in observed facts significantly affect the
model’s performance on the test case. How-
ever, what is the effective scope of it? Is it only
pattern matching on a single test example or
reasoning an implicit rule that affects more ex-
amples? To answer the question, we first make
three assumptions about its possible scope and
show them in Figure 6. For individual scope,
the model can only answer the test case xt (e.g.
[3,4,7] in the figure), for all other cases, the ac-
curacy of the prediction is very low. For lo-
calized scope, the model can also answer cases
close to xt (i.e. the neighbor facts of xt). For global scope, the model can answer all cases with high accuracy.

Figure 6: Examples for three different effective scopes.

Setup In our experiment, we sample n test cases Xt (here n = 5) in each EI task τ and define the accuracy
aτ for this particular task as:

aτ =

1
n

(cid:88)

x∈Xt

I[M(Xτ , x|D) = f (x)]

(8)

Here Xτ is the observed fact set of the task. Let T denote the set of all EI tasks (we set |T | = 100), we define
deductive density Id as:

Id =

1
|Tc|

(cid:88)

aτ I[M(Xτ , xt|D) = f (xt)]

(cid:88)

τ ∈T
I[M(Xτ , xt|D) = f (xt)]

|Tc| =

(9)

(10)

τ ∈T

where xt is the origin test input in task τ . We use this metric to indicate the impact of a successful deduction
(i.e. [3, 4, 7] in Figure 6) on reasoning over other examples in the test region N (xt, η). A high Id indicates that
the model performs well in most cases within this region, while a low Id suggests that the model’s reasoning is
more localized or even individual. For comparison, we set the test radius η to 1, 2, 3, and infinity (i.e. the full
test space), and calculate the corresponding Id for the model. Besides, we also vary the neighborhood radius ϵ
to examine the impact of different distributions of neighbor facts on their effective scope (here we set the test
region to the full space). We repeat the experiment five times to eliminate the interference of random errors,
and the results are illustrated in Figure 7.

Results We can draw conclusions as follows:
(1) LLM conducts localized reasoning through the
neighbor-based paradigm. From Figure 7a, 7c, we observe that the Id of IF and CF decreases continuously
as the radius of the test domain expands. These neighbor cases are highly effective within the neighborhood of

9

432
433
434
435

436
437
438
439
440
441

442
443
444
445
446

447
448
449
450
451
452

453
454
455
456
457
458

459
460
461
462
463
464

465
466
467
468
469

470
471
472
473
474
475

476
477
478
479
480
481

482
483
484
485

LQI'HGXFWLYH'HQVLW\%/,)&)2)'HGXFWLYH'HQVLW\%/,)&)2)LQI'HGXFWLYH'HQVLW\%/,)&)2)'HGXFWLYH'HQVLW\%/,)&)2)Individual[3,4,7]LocalizedGlobalNeighborhood[3,3,6][4,3,8][9,9,1][1,0,4][3,4,7]Neighborhood[3,3,6][4,3,8][9,9,1][1,0,4]Test SpaceTest Space[3,4,7]Neighborhood[3,3,6][4,3,8][9,9,1][1,0,4]Test SpaceηηηUnder review as a conference paper at ICLR 2025

486
487
488
489

490
491
492
493
494
495

496
497
498
499
500

501
502
503
504
505
506

507
508
509
510
511
512

513
514
515
516
517
518

519
520
521
522
523

524
525
526
527
528
529

530
531
532
533
534
535

536
537
538
539

xt. For example, in Figure 7a, when η = 1, the model can achieve over 0.9 Id. However, this impact dimin-
ishes for test cases that are farther from xt. As an example, in Figure 7c, the model only gets around 0.5 Id
in full test space. (2) The effective scope of neighbor facts is proportional to their distance from the test
case. According to Figure 7b,7d, the Id of IF and CF (particularly IF) increases as ϵ becomes large. When
the neighborhood radius increases, the distribution of these facts becomes more dispersed. We can infer that a
more dispersed distribution of neighbor facts tends to make the effective scope more global.

5 LIMITATIONS AND DISCUSSIONS

Interpretation Methods. Most model interpretation studies delve into the internal of models (e.g. neu-
rons, attention layers), providing a comprehensive explanation of the working mechanisms (Romera-Paredes
et al., 2024; Li et al., 2024). However, our work does not conduct internal analysis but instead relies on per-
formance comparisons under different settings. There are two main considerations for this: On one hand, this
work aims to identify mechanisms that are applicable to black-box models. Since we do not have access to
the internal parameters of these models, we are unable to use previous methods. On the other hand, white-
box models exhibit poor inductive reasoning capabilities according to Table 1. Therefore, conducting in-depth
interpretations based on white-box models may introduce noise to the conclusions.

Experimental Settings. The goal of this paper is to evaluate and explain the inductive reasoning process in
LLMs, rather than to improve the task performance. Therefore, we do not meticulously design the prompts used
in the experiments, nor do we use the best-performing inductive reasoning methods throughout the analysis.
We believe that the experimental setup of 0-shot IO with simple instructions is more aligned with real-world
application scenarios, making our evaluation and explanation results more meaningful.

Future Directions. Our study demonstrates that LLMs perform poorly in rule-based reasoning but excel
at using neighbor facts for reasoning. Future work could explore methods to encourage the model to follow
rules more closely during reasoning or to further optimize the model’s inductive reasoning abilities based on
this neighbor-matching finding.

6 RELATED WORK

Evaluating Inductive Reasoning Abilities of LLMs. Existing studies on evaluating LLM’s inductive
reasoning capabilities mainly use only a single task. On one hand, some works assess the model’s rule induction
ability by evaluating the accuracy on unseen examples (Moskvichev et al., 2023; Tang et al., 2023; Gendron
et al., 2023; Xu et al., 2024b; Qiu et al., 2024). However, since the model’s deduction does not always rely on
inducing the correct rule, this indirect evaluation method can introduce some inaccuracies. On the other hand,
some studies directly evaluate the correctness of the generated rules to assess inductive reasoning ability (Shao
et al., 2024; Cheng et al., 2024; Yang et al., 2024b). These studies lack evaluation on test examples, making
it difficult to confirm the model’s mastery of the inductive rules. Our work evaluates both aspects, providing a
comprehensive analysis of the model’s inductive reasoning process.

Mechanism Analysis on LLM’s Reasoning. A growing body of interpretability research has begun
analyzing the reasoning mechanisms of LLMs, aiming to deepen our understanding of how these models func-
tion. Some studies explore the mechanisms behind mathematical reasoning (Zhang et al., 2024; Hu et al.,
2024b; Romera-Paredes et al., 2024; Stolfo et al., 2023), some works investigate multi-hop reasoning (Wang
et al., 2024a; Hou et al., 2023; Yang et al., 2024a; Biran et al., 2024), and some focus on other types of rea-
soning (Li et al., 2024; Hu et al., 2024a). However, there is currently a lack of analysis on the mechanisms
of inductive reasoning. Our work mitigates this gap and uncovers the neighbor-based paradigms LLMs follow
when performing inductive reasoning.

7 CONCLUSION

In this paper, we focus on evaluating and explaining the inductive reasoning process of LLMs. First, we con-
struct a dataset MIRAGE, which provides both inductive and deductive evaluation tasks, with the flexibility to
generate test examples in any distribution, different difficulties, and various forms. Based on it, we demon-
strated that LLM is a poor rule-based reasoner, it does not need to rely on inductive rules when performing
inductive reasoning. Compared to correct induction, the model can perform successful deduction with fewer
observations, and this deduction is closely related to the form of the input. Furthermore, we identify a key
paradigm of LLM inductive reasoning: neighbor-based reasoning. The model tends to leverage observed facts
that are close to the test examples in feature space for inductive reasoning. Through it, the model can achieve
strong inductive reasoning capabilities within a localized scope and apply this ability to make inferences on
unseen examples.

10

Under review as a conference paper at ICLR 2025

540
541
542
543

544
545
546
547
548
549

550
551
552
553
554

555
556
557
558
559
560

561
562
563
564
565
566

567
568
569
570
571
572

573
574
575
576
577

578
579
580
581
582
583

584
585
586
587
588
589

590
591
592
593

REFERENCES

Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the
limitations of large language models on multi-hop queries. CoRR, abs/2406.12775, 2024. doi: 10.48550/
ARXIV.2406.12775. URL https://doi.org/10.48550/arXiv.2406.12775.

Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng
Li, Yifan Gao, Xian Li, Bing Yin, and Yizhou Sun.
Inductive or deductive? rethinking the fundamen-
tal reasoning abilities of llms. CoRR, abs/2408.00114, 2024. doi: 10.48550/ARXIV.2408.00114. URL
https://doi.org/10.48550/arXiv.2408.00114.

Franc¸ois Chollet. On the measure of intelligence. CoRR, abs/1911.01547, 2019. URL http://arxiv.

org/abs/1911.01547.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman,
Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang,
Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur´elien
Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi`ere, Bethany Biron, Binh Tang, Bobbie Chern,
Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe
Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel
Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-
Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan,
Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis An-
derson, Graeme Nail, Gr´egoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar,
Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evti-
mov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer
van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Ji-
awen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua John-
stun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Ken-
neth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi:
10.48550/ARXIV.2407.21783. URL https://doi.org/10.48550/arXiv.2407.21783.

Ga¨el Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not abstract
reasoners. CoRR, abs/2305.19555, 2023. doi: 10.48550/ARXIV.2305.19555. URL https://doi.org/
10.48550/arXiv.2305.19555.

Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and

large language models. Cognitive Systems Research, 83:101155, 2024.

Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and
Mrinmaya Sachan. Towards a mechanistic interpretation of multi-step reasoning capabilities of language
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on
models.
Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp.
4902–4919. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.299.
URL https://doi.org/10.18653/v1/2023.emnlp-main.299.

Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Wenlin Yao, Hassan Foroosh, Dong Yu, and
Fei Liu. When reasoning meets information aggregation: A case study with sports narratives. CoRR,
abs/2406.12084, 2024a. doi: 10.48550/ARXIV.2406.12084. URL https://doi.org/10.48550/
arXiv.2406.12084.

Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. Case-based or rule-based: How do transformers do
the math? In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July
21-27, 2024. OpenReview.net, 2024b. URL https://openreview.net/forum?id=4Vqr8SRfyX.

Franz Huber. On the justification of deduction and induction. European Journal for Philosophy of Science, 7:

507–534, 2017.

Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, and Jay Pu-
jara. MARVEL: multidimensional abstraction and reasoning through visual evaluation and learning. CoRR,
abs/2404.13591, 2024. doi: 10.48550/ARXIV.2404.13591. URL https://doi.org/10.48550/
arXiv.2404.13591.

Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reason-

ing. In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI), 2022.

11

Under review as a conference paper at ICLR 2025

594
595
596
597

598
599
600
601
602
603

604
605
606
607
608

609
610
611
612
613
614

615
616
617
618
619
620

621
622
623
624
625
626

627
628
629
630
631

632
633
634
635
636
637

638
639
640
641
642
643

644
645
646
647

Brenden M. Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instructions. In
Ashok K. Goel, Colleen M. Seifert, and Christian Freksa (eds.), Proceedings of the 41th Annual Meeting of
the Cognitive Science Society, CogSci 2019: Creativity + Cognition + Computation, Montreal, Canada, July
24-27, 2019, pp. 611–617. cognitivesciencesociety.org, 2019. URL https://mindmodeling.org/
cogsci2019/papers/0123/index.html.

Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao.
Focus on your question!
In
Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August
11-16, 2024, pp. 9206–9230. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.
ACL-LONG.499. URL https://doi.org/10.18653/v1/2024.acl-long.499.

interpreting and mitigating toxic cot problems in commonsense reasoning.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon,
Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank
Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR,
abs/2303.17651, 2023. doi: 10.48550/ARXIV.2303.17651. URL https://doi.org/10.48550/
arXiv.2303.17651.

Arsenii Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptarc benchmark: Evaluating
understanding and generalization in the ARC domain. Trans. Mach. Learn. Res., 2023, 2023. URL https:
//openreview.net/forum?id=8ykyGbtt2q.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann,
Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds,
Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario
Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.
In-context learning
and induction heads. CoRR, abs/2209.11895, 2022. doi: 10.48550/ARXIV.2209.11895. URL https:
//doi.org/10.48550/arXiv.2209.11895.

OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL

https://doi.org/10.48550/arXiv.2303.08774.

Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang,
Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning
capabilities of language models with hypothesis refinement. In The Twelfth International Conference on
Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://openreview.net/forum?id=bNt7oajl2a.

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Ku-
mar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet
Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models.
Nat., 625(7995):468–475, 2024. doi: 10.1038/S41586-023-06924-6. URL https://doi.org/10.
1038/s41586-023-06924-6.

Joshua Stewart Rule. The child as hacker: building more human-like models of learning. PhD thesis, Mas-

sachusetts Institute of Technology, 2020.

Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu
Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, and Dahua Lin. Case2code: Learning induc-
tive reasoning with synthetic data. CoRR, abs/2407.12504, 2024. doi: 10.48550/ARXIV.2407.12504. URL
https://doi.org/10.48550/arXiv.2407.12504.

Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic rea-
soning in language models using causal mediation analysis.
In Houda Bouamor, Juan Pino, and Kalika
Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,
EMNLP 2023, Singapore, December 6-10, 2023, pp. 7035–7052. Association for Computational Linguis-
tics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.435. URL https://doi.org/10.18653/v1/
2023.emnlp-main.435.

Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Itd: Large language
In Lun-Wei Ku, Andre Martins, and Vivek
models can teach themselves induction through deduction.
Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 2719–2731. Association
for Computational Linguistics, 2024. doi: 10.18653/V1/2024.ACL-LONG.150. URL https://doi.
org/10.18653/v1/2024.acl-long.150.

12

Under review as a conference paper at ICLR 2025

648
649
650
651

652
653
654
655
656
657

658
659
660
661
662

663
664
665
666
667
668

669
670
671
672
673
674

675
676
677
678
679
680

681
682
683
684
685

686
687
688
689
690
691

692
693
694
695
696
697

698
699
700
701

Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large
language models are in-context semantic reasoners rather than symbolic reasoners. CoRR, abs/2305.14825,
doi: 10.48550/ARXIV.2305.14825. URL https://doi.org/10.48550/arXiv.2305.
2023.
14825.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash-
lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer,
Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyn-
thia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan,
Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura,
Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar-
tinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein,
Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao-
qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aur´elien Rodriguez, Robert Sto-
jnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR,
abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/
arXiv.2307.09288.

Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic
journey to the edge of generalization. CoRR, abs/2405.15071, 2024a. doi: 10.48550/ARXIV.2405.15071.
URL https://doi.org/10.48550/arXiv.2405.15071.

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words
are anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor,
Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 9840–9855. Association for
Computational Linguistics, 2023a. doi: 10.18653/V1/2023.EMNLP-MAIN.609. URL https://doi.
org/10.18653/v1/2023.emnlp-main.609.

Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis
search: Inductive reasoning with language models. In The Twelfth International Conference on Learning
Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https:
//openreview.net/forum?id=G7UtIGQmjm.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdh-
ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The
Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023. OpenReview.net, 2023b. URL https://openreview.net/pdf?id=1PL1NIMMrw.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi,
Chain-of-thought prompting elicits reasoning in large language
Quoc V. Le, and Denny Zhou.
In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh
models.
(eds.), Advances in Neural
Information Processing Systems 35: Annual Conference on Neural
Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -
December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/
9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.

Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias Boutros Khalil. Llms and the abstraction
and reasoning corpus: Successes, failures, and the importance of object-based representations. Trans. Mach.
Learn. Res., 2024, 2024a. URL https://openreview.net/forum?id=E8m8oySvPJ.

Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias Boutros Khalil. Llms and the abstraction
and reasoning corpus: Successes, failures, and the importance of object-based representations. Trans. Mach.
Learn. Res., 2024, 2024b. URL https://openreview.net/forum?id=E8m8oySvPJ.

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language mod-
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),
els latently perform multi-hop reasoning?
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10210–10229. Association for Compu-
tational Linguistics, 2024a. doi: 10.18653/V1/2024.ACL-LONG.550. URL https://doi.org/10.
18653/v1/2024.acl-long.550.

Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei.
Language models as inductive reasoners. In Yvette Graham and Matthew Purver (eds.), Proceedings of the
18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 -
Volume 1: Long Papers, St. Julian’s, Malta, March 17-22, 2024, pp. 209–225. Association for Computational
Linguistics, 2024b. URL https://aclanthology.org/2024.eacl-long.13.

13

Under review as a conference paper at ICLR 2025

Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye.
Interpreting and improving large language models in arithmetic calculation.
In Forty-first International
Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
URL https://openreview.net/forum?id=CfOtiepP8s.

14

702
703
704
705

706
707
708
709
710
711

712
713
714
715
716

717
718
719
720
721
722

723
724
725
726
727
728

729
730
731
732
733
734

735
736
737
738
739

740
741
742
743
744
745

746
747
748
749
750
751

752
753
754
755

Under review as a conference paper at ICLR 2025

756
757
758
759

760
761
762
763
764
765

766
767
768
769
770

771
772
773
774
775
776

777
778
779
780
781
782

783
784
785
786
787
788

789
790
791
792
793

794
795
796
797
798
799

800
801
802
803
804
805

806
807
808
809

A MORE DETAILS FOR DATASET CONSTRUCTION

A.1 COMPARISON OF DATASETS BETWEEN RELATED WORK AND OUR STUDY

Since our work is conducted entirely on our MIRAGE dataset, we aim to provide a detailed comparison with
representative datasets from related studies to demonstrate its effectiveness. Specifically, we report the com-
parison in Table 5. From the results, we can see that our dataset can cover most of the operations and forms
in previous datasets. For example, the transformation examples in the 1D-ARC dataset shown in the table are
equivalent to our PAD operation. Therefore, we demonstrate that our dataset is an effective dataset for inductive
reasoning. Moreover, based on its coverage, conducting experiments solely on it is sufficient.

Dataset

Fact Form

Main Operation

Example

ARC (Chollet, 2019)

List (2D)

Fill, Move, Pile

MiniSCAN (Lake et al., 2019)

String

String Translation

ListFunctions (Rule, 2020)

List

List Operation

MiniARC (Kim et al., 2022)

List (2D)

Fill, Move, Pile

1D-ARC (Xu et al., 2024a)

List

Fill, Move, Pile

Case2Code (Shao et al., 2024) Code

Python Function

MIRAGE

All above + RP All above

input: [[0,1,0],[1,1,0],
[0,1,0],[0,1,1],[0,1,0],
[1,1,0]]

output: [[0,2,0],[2,2,0],
[0,2,0],[0,2,2],[0,2,0],
[2,2,0],[0,2,0],[0,2,2],
[0,2,0]

input: her sneury voirk

output: GREEN BLUE

input: [4,7,6,9,0]

output:[4,8,6,9,0]

input: [[1,1,5,6,8],
[0,1,5,6,6],[5,5,5,5,5],
[7,7,5,4,4],[7,7,5,0,4]]

output: [[1,6,0,0,0],
[7,4,0,0,0],[0,0,0,0,0],
[0,0,0,0,0], [0,0,0,0,0]]

input: [0,0,2,0,0,0,0,2,
0,0,0]

output: [0,0,2,2,2,2,2,2,
0,0,0]

input: dict(no=2)

output: [2]

See Figure 2

Table 5: Comparison between some representative datasets and ours.

A.2 EVALUATION OF MATHEMATICAL OPERATION DIFFICULTY IN MIRAGE

Our data introduces some mathematical operations in Add and Map, here we aim to demonstrate that these
calculations are inherently simple for LLMs, ensuring that they do not interfere with our evaluation of the
model’s inductive reasoning performance. Specifically, we randomly construct linear operations in Add and
Map with single-digit operands (cover all of the operations included in this paper) and observe the accuracy
of each model on 100 questions. The results are reported in Table 6. We can observe that most of the modes
can achieve very high accuracy on these math operations. Specifically, all closed-source models achieve 100%
accuracy. This indicates that our dataset construction effectively eliminates noise introduced by mathematical
calculations in most cases.

15

Under review as a conference paper at ICLR 2025

Operation Llama2-13B Llama3-8B GPT-4o GPT-4 Claude-3.5

Map
Add

0.96
0.51

0.93
0.99

1.00
1.00

1.00
1.00

1.00
1.00

Table 6: Accuracy of basic mathematical computations across different models in our dataset.

A.3 EVALUATION TEMPLATES FOR DIFFERENT TASKS AND SCENARIOS

In Tables 14 and 15, we report the evaluation prompts used to evaluate the models in our work. In Table 16, we
provide the templates used for constructing different scenarios in RP.

B MORE DETAILS FOR RULE-BASED REASONING EVALUATION

B.1

IMPLEMENT DETAILS FOR MAIN EXPERIMENTS

version, we

Meta-Llama-3-8B-Instruct,
For model
gpt-4-0613, gpt-4o-2024-05-13 and claude-3-5-sonnet-20240620.
All experiments
are conducted on 4 NVIDIA GeForce RTX 3090 GPUs. For the sake of simplicity, we include all the prompts
used in this work in the supplementary materials.

select Llama-2-13b-chat-hf,

B.2 MORE EXPERIMENTS ON OTHER MODELS

In this section, we repeat the experiments in § 3.2 on the Llama3-8B model and Llama2-13B model, the results
are shown in Table 7, 8. Here we set D = 3, N = 5. We can observe that the results in our main text still hold
on these models. We do not apply HR on them, since these two models have difficulty in evaluating the rule
based on given templates under 0-shot settings.

Method

IO (0-shot)
IO (5-shot)

ID (0-shot)
ID (5-shot)

CoT (0-shot)
CoT (5-shot)

SC (n=5)

SR (t=3)

RI

0.21
0.32

0.10
0.35

0.25
0.54

0.47

0.16

LT

EI

0.54
0.41

0.24
0.44

0.39
0.59

0.54

0.26

(∆)

0.33
0.09

0.14
0.09

0.14
0.05

0.07

0.10

RI

0.03
0.35

0.12
0.30

0.13
0.36

0.35

0.09

RP

EI

0.48
0.40

0.30
0.40

0.39
0.40

0.37

0.27

(∆)

0.45
0.05

0.18
0.10

0.26
0.04

0.02

0.18

RI

0.17
0.14

0.10
0.01

0.16
0.41

0.45

0.08

CG

EI

0.22
0.24

0.17
0.04

0.14
0.55

0.55

0.20

(∆)

0.05
0.10

0.07
0.03

-0.02
0.14

0.10

0.12

RI

0.07
0.25

0.01
0.31

0.16
0.47

0.51

0.05

ST

EI

0.45
0.35

0.35
0.33

0.42
0.66

0.62

0.39

(∆)

0.38
0.10

0.34
0.02

0.26
0.19

0.11

0.34

Table 7: Performance of different methods on MIRAGE using Llama3-8B (100 examples).

Method

IO (0-shot)
IO (5-shot)

ID (0-shot)
ID (5-shot)

CoT (0-shot)
CoT (5-shot)

SC (n=5)

SR (t=3)

RI

0.01
0.02

0.00
0.02

0.03
0.01

0.07

0.00

LT

EI

0.29
0.37

0.02
0.12

0.10
0.24

0.24

0.03

(∆)

0.28
0.35

0.02
0.10

0.08
0.23

0.17

0.03

RI

0.01
0.01

0.00
0.01

0.03
0.02

0.08

0.01

RP

EI

0.41
0.34

0.03
0.14

0.20
0.14

0.49

0.07

(∆)

0.40
0.33

0.03
0.14

0.17
0.12

0.41

0.06

RI

0.01
0.01

0.00
0.00

0.00
0.00

0.01

0.00

CG

EI

0.15
0.14

0.06
0.01

0.13
0.14

0.34

0.06

(∆)

0.15
0.14

0.06
0.01

0.13
0.14

0.33

0.06

RI

0.18
0.05

0.17
0.01

0.06
0.07

0.17

0.01

ST

EI

0.40
0.30

0.20
0.30

0.13
0.10

0.37

0.10

(∆)

0.22
0.25

0.02
0.29

0.08
0.03

0.20

0.09

Table 8: Performance of different methods on MIRAGE using Llama2-13B (200 examples).

16

810
811
812
813

814
815
816
817
818
819

820
821
822
823
824

825
826
827
828
829
830

831
832
833
834
835
836

837
838
839
840
841
842

843
844
845
846
847

848
849
850
851
852
853

854
855
856
857
858
859

860
861
862
863

Under review as a conference paper at ICLR 2025

B.3 EXPERIMENTS ON FINE-TUNING METHOD

In the main text, we primarily use in-context learning to evaluate the performance of various models. Here,
we supplement the evaluation with the performance of fine-tuned models on MIRAGE. Specifically, we use
Meta-Llama-3-8B-Instruct as the backbone, setting N = 5 and D = 5, and train the model on 8,000
samples. For the training parameters, we set the learning rate to 0.0001, the batch size to 1, and the number of
epochs to 10. Additionally, LoRA is employed to train people on different types of tasks. The performances on
100 test examples are presented in Table 9. It demonstrates that fine-tuning can effectively enhance the model’s
inductive reasoning capabilities. Compared to the 5-shot ICL, the model performs better on both reasoning
tasks, even when not trained on tasks of the same format.

Method

5-shot ICL
LT Training
RP Training
CG Training
ST Training

LT

RP

CG

ST

RI

0.31
0.89
0.51
0.76
0.51

EI

0.27
0.82
0.44
0.75
0.52

RI

0.04
0.22
0.78
0.22
0.19

EI

0.17
0.24
0.74
0.25
0.21

RI

0.16
0.34
0.42
0.86
0.33

EI

0.28
0.80
0.69
0.80
0.61

RI

0.21
0.26
0.24
0.26
0.50

EI

0.32
0.38
0.35
0.37
0.35

Table 9: Performance of the fine-tuned model on MIRAGE. The best results in each column are
highlighted in bold.

C MORE DETAILS FOR NEIGHBOR-BASED REASONING EVALUATION

C.1 PROOF OF CONTINUES FUNCTIONS

Here, we prove that the five basic vector operations in MIRAGE are all continuous functions:
Theorem 1 (Add Operation Continuity). Let A = (a1, a2, . . . , an) ∈ Rn. Define a mapping f : Rn → Rn
such that for a fixed index k ∈ {1, 2, . . . , n} and a fixed subset I ⊆ {1, 2, . . . , n}, we have

f (A) = (a1, . . . , ak−1,

(cid:88)

i∈I

ai, ak+1, . . . , an),

where k /∈ I. Then f is a continuous function.

Proof. Consider two vectors A, B ∈ Rn:

A = (a1, a2, . . . , an), B = (b1, b2, . . . , bn).

The mapping f replaces the k-th element of the vector with the sum of elements indexed by the subset I. Thus,

f (A) = (a1, . . . , ak−1,

f (B) = (b1, . . . , bk−1,

(cid:88)

i∈I
(cid:88)

i∈I

ai, ak+1, . . . , an),

bi, bk+1, . . . , bn).

The distance between the images of A and B under f is

∥f (A) − f (B)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

j=1,j̸=k

(aj − bj)2 +

(cid:32)

(cid:88)

i∈I

ai −

(cid:88)

i∈I

(cid:33)2

bi

.

Let us focus on the term involving the sums:

(cid:88)

i∈I

ai −

(cid:88)

i∈I

bi =

(cid:88)

(ai − bi).

i∈I

By the triangle inequality, we have

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

(cid:12)
(cid:12)
(cid:12)
(ai − bi)
(cid:12)
(cid:12)

(cid:88)

i∈I

≤

(cid:88)

i∈I

|ai − bi|.

17

864
865
866
867

868
869
870
871
872
873

874
875
876
877
878

879
880
881
882
883
884

885
886
887
888
889
890

891
892
893
894
895
896

897
898
899
900
901

902
903
904
905
906
907

908
909
910
911
912
913

914
915
916
917

Under review as a conference paper at ICLR 2025

Therefore,

(cid:32)

(cid:88)

ai −

(cid:88)

bi

i∈I

i∈I

(cid:33)2

(cid:32)

≤

(cid:88)

i∈I

(cid:33)2

|ai − bi|

.

Using the Cauchy-Schwarz inequality, we get

(cid:33)2

|ai − bi|

≤ |I|

(cid:32)

(cid:88)

i∈I

(cid:88)

(ai − bi)2,

i∈I

where |I| is the cardinality of the set I.

Therefore,

∥f (A) − f (B)∥ ≤

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

j=1,j̸=k

(aj − bj)2 + |I|

(cid:88)

(ai − bi)2.

i∈I

This can be bounded as

∥f (A) − f (B)∥ ≤ C∥A − B∥,

where C is a constant depending on n and |I|.

Therefore, for any ϵ > 0, choose δ = ϵ

C . If ∥A − B∥ < δ, then

∥f (A) − f (B)∥ < Cδ = ϵ.

Hence, f is continuous.

Theorem 2 (Copy Operation Continuity). Let A = (a1, a2, . . . , an) ∈ Rn. Define a mapping f : Rn → Rn
such that for fixed indices J ⊆ {1, 2, . . . , n} and a fixed index k ∈ {1, 2, . . . , n}, we have

f (A) = (b1, b2, . . . , bn),

bi =

(cid:40)

ak
ai

if i ∈ J,
otherwise.

where

Then f is a continuous function.

Proof. Consider two vectors A, B ∈ Rn:

A = (a1, a2, . . . , an), B = (b1, b2, . . . , bn).

The mapping f replaces each element of A at the positions indexed by J with the value of the element at index
k. Specifically,

where

Similarly,

where

f (A) = (c1, c2, . . . , cn),

ci =

(cid:40)

ak
ai

if i ∈ J,
otherwise.

f (B) = (d1, d2, . . . , dn),

di =

(cid:40)

bk
bi

if i ∈ J,
otherwise.

The distance between the images of A and B under f is given by

By the definition of f , we have

∥f (A) − f (B)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(ci − di)2.

i=1

ci − di =

(cid:40)

ak − bk
ai − bi

if i ∈ J,
otherwise.

18

918
919
920
921

922
923
924
925
926
927

928
929
930
931
932

933
934
935
936
937
938

939
940
941
942
943
944

945
946
947
948
949
950

951
952
953
954
955

956
957
958
959
960
961

962
963
964
965
966
967

968
969
970
971

Under review as a conference paper at ICLR 2025

Therefore, the distance can be rewritten as

∥f (A) − f (B)∥ =

(cid:115)(cid:88)

(ak − bk)2 +

(cid:88)

(ai − bi)2.

i∈J

i /∈J

Since the sum over J has |J| terms that are all equal to (ak − bk)2, this simplifies to

∥f (A) − f (B)∥ =

(cid:115)

|J|(ak − bk)2 +

(ai − bi)2.

(cid:88)

i /∈J

This can be bounded as

∥f (A) − f (B)∥ ≤ C∥A − B∥,

where C is a constant depending on n and |J|.

Therefore, for any ϵ > 0, choose δ = ϵ

C . If ∥A − B∥ < δ, then

∥f (A) − f (B)∥ < Cδ = ϵ.

Hence, f is continuous.

Theorem 3 (Map Operation Continuity). Let A = (a1, a2, . . . , an) ∈ Rn. Define a mapping f : Rn → Rn
such that for fixed indices J ⊆ {1, 2, . . . , n} and fixed scalars k, b ∈ R, we have

where

f (A) = (b1, b2, . . . , bn),

(cid:40)

bi =

kai + b
ai

if i ∈ J,
otherwise.

Then f is a continuous function.

Proof. Consider two vectors A, B ∈ Rn:

A = (a1, a2, . . . , an), B = (b1, b2, . . . , bn).

The mapping f applies the linear transformation kx + b to the elements of A indexed by J and leaves the other
elements unchanged:

where

Similarly,

where

f (A) = (c1, c2, . . . , cn),

ci =

(cid:40)

kai + b
ai

if i ∈ J,
otherwise.

f (B) = (d1, d2, . . . , dn),

di =

(cid:40)

kbi + b
bi

if i ∈ J,
otherwise.

The distance between the images of A and B under f is given by

∥f (A) − f (B)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(ci − di)2.

By the definition of f , we have

i=1

(cid:40)

ci − di =

k(ai − bi)
ai − bi

if i ∈ J,
otherwise.

Therefore, the distance can be rewritten as

∥f (A) − f (B)∥ =

(cid:115)(cid:88)

(k(ai − bi))2 +

(cid:88)

(ai − bi)2.

i∈J

i /∈J

19

972
973
974
975

976
977
978
979
980
981

982
983
984
985
986

987
988
989
990
991
992

993
994
995
996
997
998

999
1000
1001
1002
1003
1004

1005
1006
1007
1008
1009

1010
1011
1012
1013
1014
1015

1016
1017
1018
1019
1020
1021

1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

1026
1027
1028

1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044
1045

1046
1047
1048
1049
1050
1051

1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067
1068

1069
1070
1071
1072
1073
1074

1075
1076
1077
1078
1079

This simplifies to

∥f (A) − f (B)∥ =

Let C = max(1, |k|). Then

(cid:115)

k2 (cid:88)

(ai − bi)2 +

(cid:88)

(ai − bi)2.

i∈J

i /∈J

∥f (A) − f (B)∥ ≤ C

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(ai − bi)2 = C∥A − B∥.

i=1

Therefore, for any ϵ > 0, choose δ = ϵ

C . If ∥A − B∥ < δ, then

∥f (A) − f (B)∥ < Cδ = ϵ.

Hence, f is continuous.

Theorem 4 (Pad Operation Continuity). Let A = (a1, a2, . . . , an) ∈ Rn. Define a mapping f : Rn → Rn
such that for a fixed subset J ⊆ {1, 2, . . . , n} and a fixed constant C ∈ R, we have

f (A) = (b1, b2, . . . , bn),

where

bi =

Then f is a continuous function.

Proof. Consider two vectors A, B ∈ Rn:

(cid:40)

C if i ∈ J,
ai

otherwise.

A = (a1, a2, . . . , an), B = (b1, b2, . . . , bn).

The mapping f replaces each element of A at the positions indexed by J with the constant C, and leaves the
other elements unchanged:

where

Similarly,

where

f (A) = (c1, c2, . . . , cn),

(cid:40)

ci =

C if i ∈ J,
ai

otherwise.

f (B) = (d1, d2, . . . , dn),

di =

(cid:40)

C if i ∈ J,
bi

otherwise.

The distance between the images of A and B under f is given by

∥f (A) − f (B)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(ci − di)2.

By the definition of f , we have

i=1

ci − di =

(cid:40)

0
ai − bi

if i ∈ J,
otherwise.

Therefore, the distance can be rewritten as

∥f (A) − f (B)∥ =

(cid:115)(cid:88)

(ai − bi)2.

i /∈J

Note that the sum is only over the indices not in J. This is because the elements in J are replaced by the
constant C, and thus their difference is zero.

20

Under review as a conference paper at ICLR 2025

Since

for any ϵ > 0, choose δ = ϵ. If ∥A − B∥ < δ, then

∥f (A) − f (B)∥ ≤ ∥A − B∥,

∥f (A) − f (B)∥ < ϵ.

Therefore, f is continuous.

Theorem 5 (Swap Operation Continuity). Let A = (a1, a2, . . . , an) ∈ Rn. Define a mapping f : Rn → Rn
such that for fixed disjoint subsets I, J ⊆ {1, 2, . . . , n} with I ∩ J = ∅ and |I| = |J|, the elements of A
indexed by I are swapped with the elements indexed by J. Then f is a continuous function.

Proof. Let A = (a1, a2, . . . , an) ∈ Rn, and let I = {i1, i2, . . . , im} and J = {j1, j2, . . . , jm} be two
disjoint subsets of indices with |I| = |J| = m. Define the mapping f such that it swaps the elements of A
indexed by I and J. Specifically, for any A, the mapping f produces a vector B = f (A) given by

bk =






ajr
air
ak

if k = ir for some r = 1, 2, . . . , m,
if k = jr for some r = 1, 2, . . . , m,
otherwise.

Consider two vectors A, C ∈ Rn:

A = (a1, a2, . . . , an), C = (c1, c2, . . . , cn).

Applying the mapping f to both vectors, we obtain

where

and similarly,

f (A) = (b1, b2, . . . , bn),

f (C) = (d1, d2, . . . , dn),

bk =

dk =






ajr
air
ak






cjr
cir
ck

if k = ir for some r = 1, 2, . . . , m,
if k = jr for some r = 1, 2, . . . , m,
otherwise,

if k = ir for some r = 1, 2, . . . , m,
if k = jr for some r = 1, 2, . . . , m,
otherwise.

The distance between f (A) and f (C) is given by

∥f (A) − f (C)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(bk − dk)2.

k=1

Since f only swaps the elements indexed by I and J, we have

bk − dk =






ajr − cjr
air − cir
ak − ck

if k = ir for some r,
if k = jr for some r,
otherwise.

Therefore, the norm becomes

∥f (A) − f (C)∥ =

Rearranging the terms, we have

(cid:118)
(cid:117)
(cid:117)
(cid:116)

m
(cid:88)

(ajr − cjr )2 +

m
(cid:88)

(air − cir )2 +

r=1

r=1

(cid:88)

k /∈I∪J

(ak − ck)2.

∥f (A) − f (C)∥ =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

n
(cid:88)

(ak − ck)2 = ∥A − C∥.

k=1

Therefore, for any ϵ > 0, choose δ = ϵ. If ∥A − C∥ < δ, then

∥f (A) − f (C)∥ = ∥A − C∥ < ϵ.

Hence, f is continuous.

21

1080
1081
1082
1083

1084
1085
1086
1087
1088
1089

1090
1091
1092
1093
1094

1095
1096
1097
1098
1099
1100

1101
1102
1103
1104
1105
1106

1107
1108
1109
1110
1111
1112

1113
1114
1115
1116
1117

1118
1119
1120
1121
1122
1123

1124
1125
1126
1127
1128
1129

1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

C.2 COMPARISON WITH OTHER DISTANCE METRIC

We aim to explore whether using different distance metrics to define neighbor facts would also influence the
model’s inductive reasoning. Therefore, we additionally introduce three other distance metrics: Euclidean
distance deuc, Manhattan distance dman, and Minkowski distance dmin. Like Equation 6, we have:

deuc(Xi, xt) =

(cid:118)
(cid:117)
(cid:117)
(cid:116)

D
(cid:88)

(xik − xtk)2

k=1

dman(Xi, xt) =

D
(cid:88)

|xik − xtk|

dmin(Xi, xt) =

k=1
(cid:32) D
(cid:88)

k=1

(cid:33) 1

p

|xik − xtk|p

(11)

(12)

(13)

where we set p = 3. We can generate three distinct new neighborhoods N (xt, ϵ) by incorporating these
distances into Equation 7, thereby constructing three new kinds of OF. Therefore, we compare the model’s
performance on EI tasks when using only these different OFs, and the results are shown in Figure 8. From the
figure, we can see that our neighborhood construction outperforms those constructed using other distance
metrics. The EI performance of the other three OFs across different radii is similar to the baseline, indicating
that removing neighbor facts constructed using these methods does not influence the model’s inductive reason-
ing ability. In contrast, our constructed OF leads to a significant decline in accuracy, proving the validity of our
neighborhood construction.

(a) GPT-4o

(b) Claude-3.5

Figure 8: Performance comparison of the impact of different OFs. The dashed line represents
the baseline accuracy using default fact sets. Euc represents Euclidean distance, Man represents
Manhattan distance and Min represents Minkowski distance.

C.3 MORE EXPERIMENTS ON OTHER MODELS

We repeat the experiments in § 4.3 on Llama2-13B, Claude-3.5 and Llama3-8B. The results are shown in Table
10, 11, 12. Besides, we also repeat the experiments in § 4.4 on Claude-3.5 and report the results in Figure 9.
The results of all these additional experiments are consistent with those in the main text.

C.4 SUPPLEMENTARY EXPERIMENT FOR MAIN EXPERIMENT

We observe that, in the experiment of § 4.2, though the performance of OF significantly decreases compared
to the baseline, some models are still able to maintain around 40% accuracy, even with only distant observed
facts. We infer that models are likely to conduct rule-based reasoning in these cases. Hence, we design an extra
experiment for supplementary. In it, we prompt LLMs to induct rules and finish deductive tasks (i.e. ID in
§3.2) on these cases in Table 13. From the table, we can observe that the model’s deductive accuracy using the
rule exceeds 70% when there are fewer neighbor facts in the context. This demonstrates that the model tends
to rely more on rule-based induction if there is less neighbor-based matching.

22

1134
1135
1136
1137

1138
1139
1140
1141
1142
1143

1144
1145
1146
1147
1148

1149
1150
1151
1152
1153
1154

1155
1156
1157
1158
1159
1160

1161
1162
1163
1164
1165
1166

1167
1168
1169
1170
1171

1172
1173
1174
1175
1176
1177

1178
1179
1180
1181
1182
1183

1184
1185
1186
1187

5DGLXV$FFXUDF\(XF0DQ0LQ2XU5DGLXV$FFXUDF\(XF0DQ0LQ2XUUnder review as a conference paper at ICLR 2025

Type

Baseline
IF Only
CF Only
OF Only

LT

0.18
0.43
0.17
0.16

N=3

RP

0.05
0.14
0.06
0.04

CG

0.14
0.35
0.14
0.13

N=5

N=8

ST

0.30
0.34
0.28
0.27

LT

0.15
0.48
0.15
0.09

RP

0.05
0.03
0.04
0.02

CG

0.17
0.49
0.22
0.09

ST

0.23
0.36
0.25
0.26

LT

0.14
0.46
0.14
0.10

RP

0.00
0.02
0.00
0.01

CG

0.17
0.52
0.17
0.10

ST

0.31
0.36
0.31
0.22

Table 10: Performance of different fact types under various settings on Llama2-13B (D = 5).

Type

Baseline
IF Only
CF Only
OF Only

LT

0.49
0.76
0.54
0.50

N=3

RP

0.23
0.42
0.23
0.24

CG

0.49
0.84
0.59
0.49

N=5

N=8

ST

0.38
0.61
0.36
0.45

LT

0.68
0.90
0.67
0.60

RP

0.39
0.46
0.36
0.30

CG

0.80
0.87
0.81
0.71

ST

0.53
0.61
0.56
0.39

LT

0.78
0.89
0.76
0.66

RP

0.51
0.62
0.42
0.26

CG

0.81
0.93
0.85
0.77

ST

0.56
0.76
0.53
0.49

Table 11: Performance of different fact types under various settings on Claude-3.5 (D = 5).

Type

Baseline
IF Only
CF Only
OF Only

LT

0.19
0.55
0.21
0.24

N=3

RP

0.12
0.22
0.07
0.06

CG

0.26
0.55
0.29
0.34

N=5

N=8

ST

0.29
0.38
0.29
0.26

LT

0.27
0.65
0.27
0.23

RP

0.17
0.40
0.14
0.08

CG

0.28
0.65
0.31
0.24

ST

0.32
0.46
0.31
0.25

LT

0.24
0.68
0.27
0.18

RP

0.26
0.41
0.22
0.11

CG

0.37
0.75
0.39
0.23

ST

0.20
0.41
0.19
0.15

Table 12: Performance of different fact types under various settings on Llama3-8B (D = 5).

(a) ϵ = 1

(b) η = inf

Figure 9: Deductive Density of various fact types on Claude-3.5 under different test radius η and
neighborhood radius ϵ (D = 5, N = 5).

Model

GPT-4o
Claude-3.5

ϵ=1

0.74
0.76

ϵ=2

0.72
0.72

ϵ=3 Avg

0.76
0.73

0.74
0.74

Table 13: Performance on the correct case of OF. We use the 0-shot setting and vary the radius ϵ.

23

1188
1189
1190
1191

1192
1193
1194
1195
1196
1197

1198
1199
1200
1201
1202

1203
1204
1205
1206
1207
1208

1209
1210
1211
1212
1213
1214

1215
1216
1217
1218
1219
1220

1221
1222
1223
1224
1225

1226
1227
1228
1229
1230
1231

1232
1233
1234
1235
1236
1237

1238
1239
1240
1241

LQI'HGXFWLYH'HQVLW\%/,)&)2)'HGXFWLYH'HQVLW\%/,)&)2)Under review as a conference paper at ICLR 2025

1242
1243
1244
1245

1246
1247
1248
1249
1250
1251

1252
1253
1254
1255
1256

1257
1258
1259
1260
1261
1262

1263
1264
1265
1266
1267
1268

1269
1270
1271
1272
1273
1274

1275
1276
1277
1278
1279

1280
1281
1282
1283
1284
1285

1286
1287
1288
1289
1290
1291

1292
1293
1294
1295

Scenario Prompt

Please summarize the rules of the list transformation based on the given facts.

Your reply should strictly follow the following format:

Rule: [A, B, C] → [<<expression>>, <<expression>>, <<expression>>]

LT

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Please generate the rule of list transformation based on the former facts.

RP

CG

Please summarize the rules of the {TASK TYPE} based on the given facts.

Your reply should strictly follow the following format:

Rule: If there are A {OBJ1}, B {OBJ2}, C {OBJ3}. After the {TASK TYPE},

there are <<expression>> {OBJ1}, <<expression>> {OBJ2}, <<expression>> {OBJ3}.

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Please generate the rule of {TASK TYPE} based on the former facts.

Please summarize the rules of the function based on the given facts.

Your reply should strictly follow the following format:

Rule:

def f(A, B, C):

A, B, C = <<expression>>, <<expression>>, <<expression>>

return A, B, C

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Please generate the rule of function based on the former facts.

Please summarize the rules of the string transformation based on the given facts.

Your reply should strictly follow the following format:

Rule: ABC → ...

ST

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Please generate the rule of string transformation based on the former facts.

Table 14: Prompts for inductive tasks (D = 3).

24

Under review as a conference paper at ICLR 2025

1296
1297
1298
1299

1300
1301
1302
1303
1304
1305

1306
1307
1308
1309
1310

1311
1312
1313
1314
1315
1316

1317
1318
1319
1320
1321
1322

1323
1324
1325
1326
1327
1328

1329
1330
1331
1332
1333

1334
1335
1336
1337
1338
1339

1340
1341
1342
1343
1344
1345

1346
1347
1348
1349

Scenario Prompt

Please answer the question based on rules of the list transformation in the given facts.

Your reply should strictly follow the following format:

Answer: [<<expression>>, <<expression>>, <<expression>>]

LT

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Question: Input: {TEST INPUT}

Please answer the question based on rules of the {TASK TYPE} in the given facts.

Your reply should strictly follow the following format:

Answer: <<expression>> {OBJ1}, <<expression>> {OBJ2}, <<expression>> {OBJ3}.

RP

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Question: Input: {TEST INPUT}

Please answer the question based on rules of the functioon in the given facts.

Your reply should strictly follow the following format:

Answer: <<expression>>, <<expression>>, <<expression>>

CG

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Question: Input: {TEST INPUT}

Please answer the question based on rules of the string transformation in the given facts.

Your reply should strictly follow the following format:

Answer: ...

ST

Fact 1: Input: {INPUT} Output: {OUTPUT}

...

Fact n: Input: {INPUT} Output: {OUTPUT}

Question: Input: {TEST INPUT}

Table 15: Prompts for EI tasks (D = 3).

25

Under review as a conference paper at ICLR 2025

1350
1351
1352
1353

1354
1355
1356
1357
1358
1359

1360
1361
1362
1363
1364

1365
1366
1367
1368
1369
1370

1371
1372
1373
1374
1375
1376

1377
1378
1379
1380
1381
1382

1383
1384
1385
1386
1387

1388
1389
1390
1391
1392
1393

1394
1395
1396
1397
1398
1399

1400
1401
1402
1403

Scenario Template

Objects

{NAME} went to the market to trade items based on the rule.

Trade

He originally had {obj expression}

chairs, tables, pens ...

After the trade, he had {obj expression}

{NAME} adjusted his diet plan according to the expert’s advice.

Diet

He originally planned to take in {obj expression}

apples, bananas, oranges ...

After the adjustment, he had {obj expression}

{NAME} was performing a card magic trick.

Magic

Initially, he had {obj expression}

Spade 5s, Jokers, Hearts 6s ...

After performing the magic, he ended up with {obj expression}

{NAME} adjusted the investment amount of each asset based on criteria.

Invest

Initially, he invested {obj expression}

stocks, bonds, funds ...

After the adjustment, he invested {obj expression}

{NAME} adjusted the students’ courses according to certain rules.

Course

Initially, the weekly course schedule was: {obj expression}

math, science, history ...

After the adjustment, the weekly course schedule was: {obj expression}

Table 16: Prompts for real-world problems construction.

26

