# Retrieved In-Context Principles from Previous Mistakes

Hao Sun1, Yong Jiang2, Bo Wang3
Yingyan Hou4, Yan Zhang1*, Pengjun Xie2, Fei Huang2
1Peking University, 3Beijing Institute of Technology
2Alibaba Group, 4Chinese Academy of Sciences
<EMAIL_ADDRESS>
Corresponding Author.

###### Abstract

In-context learning (ICL) has been instrumental in adapting Large Language
Models (LLMs) to downstream tasks using correct input-output examples. Recent
advances have attempted to improve model performance through principles
derived from mistakes, yet these approaches suffer from lack of customization
and inadequate error coverage. To address these limitations, we propose
Retrieved In-Context Principles (RICP), a novel teacher-student framework. In
RICP, the teacher model analyzes mistakes from the student model to generate
reasons and insights for preventing similar mistakes. These mistakes are
clustered based on their underlying reasons for developing task-level
principles, enhancing the error coverage of principles. During inference, the
most relevant mistakes for each question are retrieved to create question-
level principles, improving the customization of the provided guidance. RICP
is orthogonal to existing prompting methods and does not require intervention
from the teacher model during inference. Experimental results across seven
reasoning benchmarks reveal that RICP effectively enhances performance when
applied to various prompting strategies.

Retrieved In-Context Principles from Previous Mistakes

Hao Sun1, Yong Jiang2††thanks:  Corresponding Author., Bo Wang3 Yingyan Hou4,
Yan Zhang1*, Pengjun Xie2, Fei Huang2 1Peking University, 3Beijing Institute
of Technology 2Alibaba Group, 4Chinese Academy of Sciences
<EMAIL_ADDRESS>

##  1 Introduction

In recent years, large language models (LLMs) have achieved superior
performance on a wide range of reasoning tasks, which include mathematic
reasoning Xia et al. (2024); Yamauchi et al. (2023); Imani et al. (2023);
Lewkowycz et al. (2022), commonsense reasoning Bian et al. (2023); Krause and
Stolzenburg (2023); Zhao et al. (2024), symbolic reasoning Dave et al. (2024);
Kojima et al. (2022); Qian et al. (2022), and so on. To enhance these
capabilities further and to align model processing closer to human-like
reasoning, recent research has increasingly focused on in-context learning
(ICL) Ye et al. (2023); Shum et al. (2023); Zhang et al. (2022), where models
generate responses based on a few provided correct examples, effectively
adapting to new tasks without extensive training.

While ICL has proven effective by primarily leveraging correct examples for
task adaptation, learning from mistakes remains a fundamental aspect of human
learning processes Edmondson (1996); Chialvo and Bak (1999); Berman (2006).
Recent studies have shown that utilizing mistakes can also enhance the
performance of LLMs. These methods often start by extracting principles from
mistakes identified by the models, and then use these principles to avoid
future similar mistakes. For example, TPD Wang et al. (2024a) uses a teacher
model to generate instructions based on the feedback from the student. LEAP
Zhang et al. (2024) creates both low-level and high-level principles from the
students’ mistakes. Self-Rethinking Tong et al. (2024) encourages LLMs to
rethink whether they have made similar previous mistakes, while TRAN Tong et
al. (2024) maintains a rule collection to avoid previous mistakes.

Figure 1: Inference pipeline of RICP. The teacher model analyzes mistakes from
the student model, creating guiding principles. These principles help prevent
the student model from making similar mistakes.

However, despite the advancements, these methods still face two main
limitations. (1) Firstly, Lack of Customization. These methods typically
utilize a consistent set of principles across different questions. However,
the fixed principle set may not suit all questions, potentially leading to
confusion when inapplicable. As a result, the performance of LLMs may suffer
due to the limited customization of these generalized principles. (2)
Secondly, Inadequate Error Coverage. Existing methods usually do not adjust
the distribution of principles to encompass a broader spectrum of errors. The
inadequate coverage not only narrows the scope of mistakes that the principles
can address but also leads to an overemphasis on similar mistakes, resulting
in unnecessary token consumption. Consequently, this limitation not only
diminishes the effectiveness of these methods but also increases the inference
costs.

To address the aforementioned limitations, we propose a novel approach named
Retrieve In-Content Principles (RICP). RICP is a teacher-student framework
where the teacher generates principles based on the student’s observed
mistakes, and the student applies these principles to prevent the recurrence
of similar mistakes. Specifically, RICP involves three stages: 1) Insight
Generation: The student model is evaluated on the training set to collect
mistakes. The teacher model then analyzes each mistake to generate a high-
level reason and several specific insights aimed at avoiding similar errors.
2) Principle Formulation: We cluster mistakes based on their underlying
reasons, with each cluster comprising questions that exhibit similar error
patterns. Mistakes sampled from these clusters are analyzed by the teacher
model to formulate task-level principles, which are shared by all questions.
For each question, the most similar mistakes are retrieved from each cluster
based on question semantics. We apply clustering to the insights from these
mistakes to reduce the redundancy and insights sampled from these clusters
establish the question-level principles, which are specific to each question.
3) Principle Utilization: Both the task-level and question-level principles
are integrated into the student model’s existing prompt to enhance its
question-answering capabilities.

The main advantages of our method are three folds: (1) Firstly, RICP retrieves
the most relevant insights for each question, enhancing the specificity and
relevance of the question-level principles. This tailored approach notably
improves the customization of the principles provided to the student model,
leading to more precise and customized guidance. (2) Secondly, during the
principle formulation stage, we apply reason clustering to construct clusters
representing different error patterns, which are then utilized to derive both
task-level and question-level principles. This strategy broadens the error
coverage of the principles. Additionally, insight clustering reduces the
redundancy of the question-level principles, enhancing the efficiency of our
approach. (3) Thirdly, RICP is orthogonal to existing prompting methods,
enabling seamless integration that improves performance across various
prompting strategies. Importantly, during inference, RICP does not require
intervention from the teacher model, which reduces computational overhead.

We have tested RICP across seven benchmarks encompassing three reasoning
tasks. The empirical results validate that RICP can significantly enhance the
accuracy of reasoning tasks, confirming its effectiveness and utility in
diverse applications. To summarize, our contributions are as follows:

  * •

We propose RICP, a novel approach that utilizes teacher-generated principles
to prevent the student from making similar mistakes.

  * •

RICP significantly enhances the customization and error coverage of principles
by providing both question-level and task-level principles.

  * •

Extensive experiments on seven benchmarks across three reasoning tasks with
various LLMs demonstrate that RICP consistently enhances model performance.

##  2 Methodology

Figure 2: The pipeline of RICP includes: 1) Insight Generation: The teacher
model analyzes the student model’s mistakes and generates high-level reasons
and specific insights; 2) Principle Formulation: Task-level principle and
question-level principles are generated based on the result of hierarchical
clustering; 3) Principle Utilization: principles are integrated into the
existing prompt to enhance the performance of the student model.

###  2.1 Overview

In RICP, we consider a teacher model and a student model. The student model
first undergoes an examination to collect mistakes, which are then analyzed by
the teacher model to obtain high-level reasons and insights to prevent similar
mistakes. Then, we cluster mistakes based on their underlying reasons, with
each cluster comprising questions that exhibit similar error patterns.
Mistakes are sampled from each cluster and analyzed by the teacher model to
formulate task-level principles. Given a question, the most similar mistakes
are retrieved from each cluster and the insights of these mistakes are
clustered and sampled to formulate question-level principles. Finally, both
task-level and question-level principles are integrated into the student
model’s prompt for question answering.

###  2.2 Insight Generation

Given a training dataset $D_{train}=\\{\langle
x_{i},y_{i}\rangle\\}_{i=1}^{k}$, where $x$ is the question and $y$ is the
answer, we ask the student model to undertake an examination on $D_{train}$ to
collect its mistakes. Incorrect solutions are identified by comparing each
predicted answer $\hat{y}$ with the ground-truth answer $y$.

$D_{\text{neg}}=\\{(x_{i},\hat{r}_{i},y_{i})\mid\hat{y}_{i}\neq
y_{i},(x_{i},y_{i})\in D_{\text{train}}\\},$ (1)

The collected mistakes reflect the student model’s learning status and
weakness.

Subsequently, the teacher model analyzes each mistake from $D_{\text{neg}}$
and generates a high-level reason $R_{i}$ and a set of insights $I_{i}$ for
each mistake. Specifically, for each question $x_{i}$ in $D_{\text{neg}}$, we
construct the teacher model’s prompt incorporating the question $x_{i}$, the
student model’s incorrect rationale $\hat{r}_{i}$, and the correct answer
$y_{i}$. This process yields an insight corpus, defined as:

$D_{\text{insight}}=\\{(x_{i},R_{i},I_{i})\mid(x_{i},\hat{r}_{i},y_{i})\in
D_{\text{neg}}\\},$ (2)

Here the reason $R_{i}$ provides a categorization of the mistake (e.g.,
“Overlooking Details,” “Misunderstanding of Problem Statement”), while the
insights $I_{i}$ consist of more detailed guidance to help the student model
prevent similar mistakes in the future.

###  2.3 Principle Formulation

When formulating the principles, we hope they can be both diverse and
customized, which means that the principles can cover as many mistakes as
possible but at the same time can provide very precise guidance for each
specific question. To achieve this, we apply hierarchical clustering on the
reasons and insights to obtain task-level principles and question-level
principles.

#### Task-level Principles

To ensure the diversity of principles, we conduct cluster analysis on the
mistakes. Specifically, we first compute vector representations for the high-
level reasons behind each question in $D_{\text{insight}}$ using BGE
embeddings111https://huggingface.co/BAAI/bge-base-en-v1.5Xiao et al. (2023).
The reason representations are then processed by the $K$-means clustering
algorithm to produce $k$ clusters of questions:

$C_{i}^{T}=Kmeans(Embed(\\{R_{j}\\})),$ (3)

where $C_{i}^{T}$ represents the $i$-th cluster. Each cluster consists of
questions that share similar error patterns. We then sample questions from
each cluster and instruct the teacher to generate task-level principles
$\mathbb{P}_{\textsc{task-level}}$ to prevent similar mistakes.

#### Question-level Principles

To ensure the customization of the principles, for a given question, we
retrieve the most similar $m$ questions from each cluster $C_{i}$. Since the
insights of these retrieved questions can be highly similar and redundant, we
apply $K$-means clustering on the insights:

$C_{i}^{Q}=Kmeans(Embed(\\{I_{j}\\})),$ (4)

where $C_{i}^{Q}$ represents the $i$-th cluster. We then sample $n$ insights
from each cluster to serve as the question-level principles
$\mathbb{P}_{\textsc{question-level}}$.

###  2.4 Principle Utilization

During inference, given a question, the task-level and question-level
principles are appended to the existing prompt $p$, forming
$\mathbb{P}_{\textsc{enhanced}}=\mathbb{P}_{\textsc{task-
level}}\oplus\mathbb{P}_{\textsc{question-level}}\oplus p$. The enhanced
prompts are then used to answer the question. The complete algorithm is
summarized in Algorithm 1. The generation of task-level principles is a one-
time process, while the generation of question-level principles needs to be
performed for each question with minimal retrieval and clustering costs. Note
that our method can be seamlessly integrated into existing prompting methods
to achieve further performance improvements without the need for intervention
from the teacher model.

Algorithm 1 RICP Algorithm

1:Training set $D_{train}=\\{\langle x_{i},y_{i}\rangle\\}_{i=1}^{k}$, teacher
model $\mathcal{T}$, student model $\mathcal{S}$, Number of Retrieved
Questions $m$, Number of Retrieved Insights $n$, Number of Reason Clusters
$k_{1}$, Number of Insight Clusters $k_{2}$

2:for each QA pair $\langle x_{i},y_{i}\rangle$ in $D_{train}$ do

3: $\hat{r}_{i}$,
$\hat{y}_{i}\leftarrow\text{FewShotCoT}\left(\mathcal{S},x_{i}\right)$

4: $D_{\text{neg}}\leftarrow\\{(x_{i},\hat{r}_{i},y_{i})\mid\hat{y}_{i}\neq
y_{i}\\}$

5:end for

6:for each $(x_{i},\hat{r}_{i},y_{i})$ in $D_{\text{neg}}$ do

7:
$R_{i},I_{i}\leftarrow\text{InsightGeneration}\left(\mathcal{T},x_{i},\hat{r}_{i},y_{i}\right)$

8:end for

9:$C^{T}\leftarrow Kmeans(Embed(\\{R_{i}\mid x_{i}\in D_{train}\\}))$

10:$\mathbb{P}_{\textsc{task-
level}}\leftarrow\text{PrincipleGeneration}\left(\mathcal{T},C^{T}\right)$

11:$Q\leftarrow\text{Retrieve}\left(q,C^{T}\right)$

12:$C^{Q}\leftarrow Kmeans(Embed(\\{I_{i}\mid q_{i}\in Q\\}))$

13:$\mathbb{P}_{\textsc{question-
level}}\leftarrow\text{Sample}\left(\mathcal{T},C^{T}\right)$

14:return $\mathbb{P}_{\textsc{task-level}},\mathbb{P}_{\textsc{question-
level}}$

##  3 Experiment

###  3.1 Experiment Setup

#### Baselines

The baseline prompting methods considered in this work are listed below:

  * •

Standard Prompting Brown et al. (2020): The LLM is asked to output the answer
directly, without the intermediate reasoning process.

  * •

Chain of Thought Wei et al. (2022): The LLM is instructed to think step-by-
step before providing the answer.

  * •

Auto CoT Zhang et al. (2022): Questions from the training set are first
clustered into groups, and the most similar questions are retrieved from each
cluster to serve as demonstrations.

  * •

Complex CoT Fu et al. (2022): Questions from the training set are retrieved
based on similarity, and those with the longest rationales are used as
demonstrations.

#### Reasoning Tasks

We conduct experiments on the following reasoning tasks:

  * •

Mathematical Reasoning: Four mathematical reasoning datasets are adopted for
evaluating: GSM8K Cobbe et al. (2021), SVAMP Patel et al. (2021), MathQA Amini
et al. (2019), and AQuA Ling et al. (2017).

  * •

Commonsense Reasoning: Two closed-book datasets are employed to evaluate
commonsense reasoning: CSQA Talmor et al. (2018) and StrategyQA Geva et al.
(2021).

  * •

Logical Reasoning: LogiQA Liu et al. (2020) is used to assess logical
reasoning ability.

It is worth mentioning that all datasets within the same reasoning task
utilize a shared insight corpus. Please refer to Appendix A for more details.

#### Models

We conduct all the experiments using GPT-3.5-Turbo and Qwen-Turbo as student
models. For the teacher model, we use GPT-4-Turbo.

###  3.2 Main Results

| GSM8K | SVAMP | MathQA | AQuA
---|---|---|---|---
Method | Vanilla | Ours | Improv | Vanilla | Ours | Improv | Vanilla | Ours | Improv | Vanilla | Ours | Improv
Qwen-Turbo
Standard Prompting | 21.20 | 23.15 | +1.95 | 61.90 | 62.60 | +0.70 | 15.30 | 17.40 | +2.10 | 15.89 | 17.76 | +1.87
Zero-shot CoT | 75.10 | 76.10 | +1.00 | 83.40 | 83.20 | -0.20 | 37.60 | 40.90 | +3.30 | 33.64 | 38.08 | +4.44
Few-shot CoT | 77.78 | 79.70 | +1.92 | 82.90 | 83.50 | +0.60 | 40.60 | 43.60 | +3.00 | 34.81 | 38.55 | +3.74
Auto CoT | 76.60 | 77.80 | +1.20 | 81.70 | 82.70 | +1.00 | 41.10 | 42.80 | +1.70 | 36.45 | 37.15 | +0.70
Complex CoT | 74.60 | 77.30 | +2.70 | 81.10 | 82.40 | +1.30 | 40.50 | 44.00 | +3.50 | 32.24 | 37.85 | +5.61
GPT-3.5-Turbo
Standard Prompting | 25.60 | 28.60 | +3.00 | 78.40 | 79.90 | +1.50 | 21.30 | 23.20 | +1.90 | 20.33 | 25.00 | +4.67
Zero-shot CoT | 68.40 | 74.70 | +6.30 | 68.80 | 74.40 | +5.60 | 33.50 | 39.30 | +5.80 | 31.07 | 38.08 | +7.01
Few-shot CoT | 77.10 | 78.10 | +1.00 | 80.50 | 82.70 | +2.20 | 41.30 | 43.10 | +1.80 | 35.98 | 40.19 | +4.21
Auto CoT | 74.20 | 76.70 | +2.50 | 78.40 | 80.90 | +2.50 | 38.40 | 40.30 | +1.90 | 36.45 | 39.02 | +2.57
Complex CoT | 74.20 | 77.10 | +2.90 | 81.20 | 83.80 | +2.60 | 38.70 | 41.60 | +2.90 | 34.35 | 41.12 | +6.78

Table 1: Performance comparison of different models on mathematical reasoning
benchmarks with and without our enhancement method.

| CSQA | StrategyQA | LogiQA | Overall
---|---|---|---|---
Method | Vanilla | Ours | Improv | Vanilla | Ours | Improv | Vanilla | Ours | Improv | Vanilla | Ours | Improv
Qwen-Turbo
Standard Prompting | 82.80 | 83.50 | +0.70 | 66.94 | 67.55 | +0.61 | 45.45 | 48.75 | +3.29 | 44.21 | 45.81 | +1.60
Zero-shot CoT | 87.10 | 88.10 | +1.00 | 70.92 | 72.45 | +1.53 | 42.32 | 47.34 | +5.02 | 61.44 | 63.74 | +2.30
Few-shot CoT | 82.60 | 83.70 | +1.10 | 75.31 | 75.92 | +0.61 | 47.96 | 49.69 | +1.72 | 63.14 | 64.95 | +1.81
Auto CoT | 80.90 | 80.70 | -0.20 | 72.65 | 73.67 | +1.02 | 49.22 | 50.94 | +1.72 | 62.66 | 63.68 | +1.02
Complex CoT | 81.20 | 83.40 | +2.20 | 69.18 | 70.31 | +1.12 | 47.81 | 50.47 | +2.66 | 60.95 | 63.68 | +2.73
GPT-3.5-Turbo
Standard Prompting | 72.00 | 76.00 | +4.00 | 63.78 | 66.33 | +2.55 | 40.28 | 43.42 | +3.13 | 45.95 | 48.92 | +2.97
Zero-shot CoT | 75.60 | 79.90 | +4.30 | 67.35 | 67.24 | -0.11 | 42.63 | 43.73 | +1.10 | 55.34 | 59.62 | +4.29
Few-shot CoT | 76.90 | 78.20 | +1.30 | 73.78 | 76.84 | +3.06 | 42.32 | 46.08 | +3.76 | 61.13 | 63.60 | +2.48
Auto CoT | 75.90 | 78.80 | +2.90 | 71.43 | 72.04 | +0.61 | 43.57 | 44.67 | +1.10 | 59.76 | 61.78 | +2.01
Complex CoT | 76.70 | 79.30 | +2.60 | 68.57 | 69.80 | +1.22 | 36.83 | 38.24 | +1.41 | 58.65 | 61.57 | +2.92

Table 2: Performance comparison of different models on commonsense reasoning
and logical reasoning benchmarks with and without our enhancement method.

In this section, we present a comparison of the performance of existing
prompting methods with and without RICP in Table 1 and Table 2. Based on these
results, we analyze as follows:

First, RICP consistently improves the performance of all prompting methods
across various benchmarks using different LLMs. Notably, when using
GPT-3.5-Turbo, RICP achieves 22.6% relative improvement over Zeroshot CoT and
20.0% relative improvement over Complex CoT on the AQuA dataset. By
incorporating both diverse task-level principles and customized question-level
principles, our method can provide both comprehensive and precise guidance,
thereby significantly improving performance.

Second, the relative improvements among the prompting methods vary. For
example, when using GPT-3.5-Turbo, the improvement over zero-shot CoT is 7.5%,
while the improvement over Auto CoT is 3.4%. This variation is mainly due to
the fact that some prompting methods can avoid previous mistakes by learning
from demonstrations in the prompt. Consequently, our method offers more
significant improvements to methods without demonstrations, eliminating the
need for selecting demonstrations. This is further confirmed by the higher
performance improvement observed in zero-shot prompting methods like standard
prompting and zero-shot CoT.

Figure 3: Ablation Study.

Third, when comparing different LLMs, we find that the relative improvement of
our method using GPT-3.5-Turbo is larger than the relative improvement using
Qwen-Turbo. This is mainly because the semantic understanding ability of Qwen-
Turbo is not as advanced as GPT-3.5-Turbo Bai et al. (2023). As a result,
Qwen-Turbo struggles to comprehend and fully utilize the provided principles.

###  3.3 Ablation Study

In this section, we assess the impact of each component in our model on the
MathQA and LogiQA datasets using GPT-3.5-Turbo as the LLM.

Results in Figure 3 show that removing any component leads to performance
degradation, highlighting their importance. Notably, the absence of question-
level principles leads to a significant decline in performance when applying
Auto CoT to the MathQA dataset and Zero-Shot CoT to the LogiQA dataset. This
is mainly because the question-level principles provide customized guidance
for each question, helping the student model avoid previous mistakes
effectively. Conversely, the elimination of task-level principles results in a
more modest decrease in performance. This is because, in some cases, the
insights provided by task-level principles can be partially compensated by
question-level principles. However, completely removing task-level principles
introduces randomness into the response process when the retrieved question-
level principles fail to provide precise guidance for the question. Moreover,
removing hierarchical clustering also leads to a performance drop due to
reduced error coverage of the principles. This clustering technique is vital
for ensuring that the insights are comprehensive, which is crucial for
maintaining the model’s robust performance.

###  3.4 Hyper-parameter Study

Figure 4: Hyper-parameters

In this section, we investigate hyper-parameters’ impact on performance using
GSM8K, SVAMP, CSQA and StrategyQA datasets, utilizing Qwen-Turbo as the LLM
and Auto CoT as the prompting method. The results are shown in Figure 4.

We observe a consistent trend across all hyper-parameters: both excessively
high and low values adversely affect performance. Specifically, when the
number of retrieved questions is low, the insights provided may not fully meet
the current question’s requirements. Conversely, a high number introduces
noise, which can confuse the model. With regard to the number of insights,
having too few may not sufficiently address the question’s needs, while having
too many can overwhelm the student model and distract from the most critical
insights. Regarding the number of clusters, too few clusters fail to ensure
diversity, while too many clusters can introduce irrelevant information.

Model | 500 | 1000 | 1500 | 2000 | 2500 | 3000
---|---|---|---|---|---|---
Zero-shot CoT | 44.8 | 45.4 | 47.4 | 43.8 | 45.2 | 46.2
Few-shot CoT | 48.2 | 48.6 | 48.7 | 45.8 | 48.0 | 48.2
Auto CoT | 47.0 | 47.8 | 48.8 | 46.6 | 46.8 | 47.0
Complex CoT | 47.0 | 50.6 | 48.0 | 49.0 | 48.0 | 49.0

Table 3: Performance comparison of different models on various insight corpus
sizes.

###  3.5 Analysis

#### Effect of the Size of Insight Corpus

During the insight generation stage, we collect the student’s mistakes and
generate corresponding insights. In this section, we analyze the impact of the
size of the insight corpus on model performance, using the LogiQA dataset and
Qwen-Turbo as the LLM.

From the results shown in Table 3, it is evident that both excessively large
and small corpora result in performance decline. A small corpus may only cover
a narrow spectrum of mistakes, limiting the ability to offer guidance across a
wide range of questions. Conversely, an overly large corpus tends to include
many similar and redundant insights. This redundancy provides little new
information and can distract the student model from focusing on the most
critical insights, thereby leading to a decrease in performance.

#### Customized Retrieval Analysis

In this section, we compare the performance of customized retrieval versus
random selection on GSM8K and AQuA datasets. For the random selection method,
questions are arbitrarily chosen from the insight pool, and the associated
insights are utilized as question-level principles.

As shown in Figure 5, customized retrieval consistently outperforms random
selection across all prompting-based methods. The effectiveness of customized
retrieval lies in its ability to provide the most pertinent and suitable
guidance for each question, significantly aiding the student model in avoiding
previous mistakes. In contrast, utilizing insights from random selection can
sometimes result in performance worse than the vanilla prompting method. This
is mainly because randomly sampled insights may offer guidance that is not
applicable to the question. Forcing the student model to utilize these
inappropriate insights can cause confusion and lead to performance
degradation, which further underscores the value of customized retrieval in
providing the most pertinent guidance.

Figure 5: The Comparison between Customized Retrieval and Random Selection.

#### Error Type Analysis

In this section, we investigate the detailed error types of the student model
by analyzing mistakes from GSM8K and CSQA datasets, focusing on mathematical
and commonsense reasoning. The results are shown in Figure 6.

For commonsense reasoning, the primary errors identified are context and
commonsense errors. Context errors often arise from a lack of factual
information. Retrieval-augmented Generation (RAG) offers an effective solution
by retrieving the most relevant factual information from an external corpus,
ensuring the LLM accesses the most up-to-date and accurate data Shi et al.
(2023); Guu et al. (2020); Jiang et al. (2023). For mathematical reasoning,
the errors are mainly logical and calculation errors. Among them, calculation
errors can be effectively addressed by providing LLMs with external tools such
as calculators Schick et al. (2024); Qiao et al. (2024); Yin et al. (2023).

Despite the absence of an external corpus or tools, our method can still
enhance performance by helping the student model avoid previous mistakes. Our
approach is particularly effective in addressing logical errors, which are
more prevalent in mathematical reasoning than in commonsense reasoning. This
effectiveness is evidenced by the greater relative improvements our method
achieves on mathematical reasoning datasets compared to commonsense reasoning
datasets. Additionally, it is worth noting that our method can be seamlessly
integrated with RAG and external tools to further enhance model performance.

Figure 6: Error Type Distribution

###  3.6 Case Study

Case 1: Mathematical Reasoning
---
Question: Ashley’s pizza delivery costs $15. What is the total amount that
Ashley should give the delivery man if she wants to give a tip that is equal
to 1/5 of the amount she ordered? CoT Response: First, calculate 1/5 of $15.
Thus the answer is that Ashley should give the delivery man $3 as a tip.
[WRONG] Task-level Principle:  Principle 1: Always double-check arithmetic
operations. Simple mistakes in addition, subtraction, multiplication…
Principle 3: Understand the relationships between quantities. When a problem
involves proportional or comparative… Question-level Principle: 1\. Carefully
distinguish between costs and revenues to accurately calculate net payment or
profit; take into account…
2\. Verify the inclusion of all components of an equation or formula, such as
base fees or penalties, to ensure… CoT+RICP Response: To solve this problem,
calculate the tip amount, which is 1/5 of the amount she ordered, $15,
resulting in a $3 tip. Then, add the tip amount to the cost of the pizza
delivery. The total amount Ashley should pay is $15 plus $3, equaling $18.
Thus, the answer is $18.[CORRECT]
Case 2: Logical Reasoning
Question: The raw material of cement is very cheap, like limestone and mud
everywhere can be used as raw material of cement. However, the price of cement
will be influenced by the price of oil, because it takes a lot of energy to
convert raw materials from Pakistan to cement in high-temperature furnaces.
what conclusions can be drawn? CoT Response: The conclusion drawn from the
given information is that the price of cement is influenced by the price of
oil due to the energy required in the conversion process. Therefore, we can
reach a conclusion that the price of cement falls as the price of oil rises.
[WRONG] Task-level Principle: Principle 1: Focus on the main idea and avoid
overgeneralization. When summarizing a text, identify the central theme…
Principle 3: Recognize the implications of specific details. Pay attention to
the details that have significant implications… Question-level Principle: 1\.
Distinguish between what is explicitly stated and what can be inferred. Avoid
extending conclusions beyond the scope…
2\. Accurately identify and correlate cause-and-effect relationships; avoid
confusing temporal sequence with causality. CoT+RICP Response: Based on the
information that petroleum is the energy source for some high-temperature
furnaces for making cement, we can conclude that Petroleum is indeed the
energy source for these furnaces. [CORRECT]
Table 4: Case studies of solving mathematical and logical reasoning problems
with CoT+RICP and CoT prompting on the GPT-3.5-Turbo. Blue text indicates the
stem, pink text indicates the effective hint, cyan text indicates the judgment
of whether the answer is correct, [CORRECT] denotes correct, [WRONG] denotes
incorrect.

In this section, we evaluate the effectiveness of our method by analyzing
cases from mathematical reasoning and logical reasoning benchmarks.

In the case of mathematical reasoning, the error in the vanilla CoT response
stemmed from forgetting to add the original cost of the pizza. Task-level
principles such as “always double-check arithmetic operations” and “understand
the proportional relationship” help prevent such oversights. Moreover,
question-level principles like “carefully distinguish between costs and
revenues” and “verify the inclusion of all components” remind the student
model that every element of the calculation should be correctly considered,
which is crucial for avoiding the mistake. In the case of logical reasoning,
the error in the vanilla CoT response was due to over-generalization. Task-
level principles such as “focus on the main idea and avoid over-
generalization” directly point out this issue. Meanwhile, question-level
principles like “distinguish between what is explicitly stated and what can be
inferred” and “accurately identify and correlate cause-and-effect
relationships” enhance the accuracy of the logical reasoning process. These
cases demonstrate that by incorporating both task-level and question-level
principles, errors in vanilla CoT prompting can be effectively mitigated,
thereby verifying the effectiveness of our method.

##  4 Related Work

###  4.1 Learning from Mistakes

Different from machines, humans are able to learn lessons from mistakes, and
the lessons can help humans avoid making similar mistakes in the future.
Inspired by this, researchers try to utilize the mistakes to improve the
performance of LLMs Li et al. (2024); An et al. (2023); Wang et al. (2024b).
Wang et al. (2024b) integrates negative data into the training of language
models. Contrastive Chain-of-Thought Chia et al. (2023) provides both positive
and negative examples in the prompt to enhance language model reasoning. Self-
Rethinking Tong et al. (2024) guides LLMs to rethink whether they have made
similar previous mistakes. TRAN Tong et al. (2024) maintains a rule collection
to avoid previous mistakes. The work most related to ours is LEAP Zhang et al.
(2024), which generates low-level and high-level principles from the LLM’s
mistakes and puts these principles into the prompt. However, the principles
are fixed during inference, which lowers the customization of the guidance,
impacting model performance negatively.

###  4.2 Teacher-Student Framework

The teacher-student framework focuses on transferring knowledge from teacher
to student. Traditional finetuning-based methods utilize teacher models to
generate training data for student models Rajani et al. (2019); Ho et al.
(2022); Magister et al. (2022). For instance, Wang et al. (2023) utilizes the
teacher model to generate rationales from student’s feedback, which are then
used for further training of the student. Similarly, Hsieh et al. (2023)
generates intermediate steps of problem-solving and finetunes the student
model on the synthesized data. However, these methods are resource-intensive
due to the computational demands of finetuning. Prompting-based methods
leverage teacher models to provide direct guidance to student models Pruthi et
al. (2022); Saha et al. (2023); Yu et al. (2023); Wang et al. (2024a). For
example, Saha et al. (2023) employs the teacher model to deliver customized
explanations during testing phases, requiring active teacher involvement. In
contrast, our method can achieve customized guidance without the need for
intervention from the teacher model, making it more cost-effective.

##  5 Conclusion

In this paper, we introduce RICP, a teacher-student framework designed to
prevent the student model from making previous mistakes. RICP significantly
enhances the customization and mistake coverage of principles by providing
relevant insights for each question and applying hierarchical clustering to
the reasons and insights. Extensive experiments across seven benchmarks in
three reasoning tasks with various LLMs demonstrate that RICP consistently
enhances model performance.

## Limitations

In this paper, we propose a teacher-student framework for preventing the
student from making previous mistakes. The limitations of the proposed method
are as follows:

1) Although our model has achieved promising results, it requires the teacher
model to be significantly more advanced than the student model to ensure the
effectiveness of the generated principles.

2) Despite the fact that our approach does not increase cost and time during
inference, there is still some overhead associated with the principle
generation process compared to direct few-shot learning, due to the need for
pre-constructing the principles.

## Ethics Statement

This work was conducted in strict compliance with the ACL Ethics Policy. All
datasets and large language models (LLMs) used for evaluation are publicly
available. Furthermore, our work aims to explore a reasoning enhancement
method. We do not foresee any negative ethical impacts arising from our work.

## References

  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  _arXiv preprint arXiv:1905.13319_.
  * An et al. (2023) Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023.  Learning from mistakes makes llm better reasoner.  _arXiv preprint arXiv:2310.20689_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Berman (2006) William Berman. 2006.  When will they ever learn? learning and teaching from mistakes in the clinical context.  _Clinical L. Rev._ , 13:115.
  * Bian et al. (2023) Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong. 2023.  Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.  _arXiv preprint arXiv:2303.16421_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Chia et al. (2023) Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023.  Contrastive chain-of-thought prompting.  _arXiv preprint arXiv:2311.09277_.
  * Chialvo and Bak (1999) Dante R Chialvo and Per Bak. 1999.  Learning from mistakes.  _Neuroscience_ , 90(4):1137–1148.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dave et al. (2024) Neisarg Dave, Daniel Kifer, C Lee Giles, and Ankur Mali. 2024.  Investigating symbolic capabilities of large language models.  _arXiv preprint arXiv:2405.13209_.
  * Edmondson (1996) Amy C Edmondson. 1996.  Learning from mistakes is easier said than done: Group and organizational influences on the detection and correction of human error.  _The journal of applied behavioral science_ , 32(1):5–28.
  * Fu et al. (2022) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022.  Complexity-based prompting for multi-step reasoning.  In _The Eleventh International Conference on Learning Representations_.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * Ho et al. (2022) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Imani et al. (2023) Shima Imani, Liang Du, and Harsh Shrivastava. 2023.  Mathprompter: Mathematical reasoning using large language models.  _arXiv preprint arXiv:2303.05398_.
  * Jiang et al. (2023) Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  _arXiv preprint arXiv:2305.06983_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Krause and Stolzenburg (2023) Stefanie Krause and Frieder Stolzenburg. 2023.  Commonsense reasoning and explainable artificial intelligence using large language models.  In _European Conference on Artificial Intelligence_ , pages 302–319. Springer.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _Advances in Neural Information Processing Systems_ , 35:3843–3857.
  * Li et al. (2024) Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Bin Sun, Xinglin Wang, Heda Wang, and Kan Li. 2024.  Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 18591–18599.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  _arXiv preprint arXiv:2007.08124_.
  * Magister et al. (2022) Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022.  Teaching small language models to reason.  _arXiv preprint arXiv:2212.08410_.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?  _arXiv preprint arXiv:2103.07191_.
  * Pruthi et al. (2022) Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. 2022.  Evaluating explanations: How much do explanations from the teacher aid students?  _Transactions of the Association for Computational Linguistics_ , 10:359–375.
  * Qian et al. (2022) Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2022.  Limitations of language models in arithmetic and symbolic induction.  _arXiv preprint arXiv:2208.05051_.
  * Qiao et al. (2024) Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. 2024.  Autoact: Automatic agent learning from scratch via self-planning.  _arXiv preprint arXiv:2401.05268_.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  _arXiv preprint arXiv:1906.02361_.
  * Saha et al. (2023) Swarnadeep Saha, Peter Hase, and Mohit Bansal. 2023.  Can language models teach weaker agents? teacher explanations improve students via theory of mind.  _arXiv preprint arXiv:2306.09299_.
  * Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_.
  * Shum et al. (2023) KaShun Shum, Shizhe Diao, and Tong Zhang. 2023.  Automatic prompt augmentation and selection with chain-of-thought from labeled data.  _arXiv preprint arXiv:2302.12822_.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Tong et al. (2024) Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024.  Can llms learn from previous mistakes? investigating llms’ errors to boost for reasoning.  _arXiv preprint arXiv:2403.20046_.
  * Wang et al. (2024a) Haorui Wang, Rongzhi Zhang, Yinghao Li, Lingkai Kong, Yuchen Zhuang, Xiusi Chen, and Chao Zhang. 2024a.  Tpd: Enhancing student language model reasoning via principle discovery and guidance.  _arXiv preprint arXiv:2401.13849_.
  * Wang et al. (2024b) Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. 2024b.  Learning from failure: Integrating negative examples when fine-tuning large language models as agents.  _arXiv preprint arXiv:2402.11651_.
  * Wang et al. (2023) Zhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, et al. 2023.  Democratizing reasoning ability: Tailored learning from large language model.  _arXiv preprint arXiv:2310.13332_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Xia et al. (2024) Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024.  Evaluating mathematical reasoning beyond accuracy.  _arXiv preprint arXiv:2404.05692_.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.  C-pack: Packaged resources to advance general chinese embedding.
  * Yamauchi et al. (2023) Ryutaro Yamauchi, Sho Sonoda, Akiyoshi Sannai, and Wataru Kumagai. 2023.  Lpml: llm-prompting markup language for mathematical reasoning.  _arXiv preprint arXiv:2309.13078_.
  * Ye et al. (2023) Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023.  Compositional exemplars for in-context learning.  In _International Conference on Machine Learning_ , pages 39818–39833. PMLR.
  * Yin et al. (2023) Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2023.  Lumos: Learning agents with unified data, modular design, and open-source llms.  _arXiv preprint arXiv:2311.05657_.
  * Yu et al. (2023) Dhara Yu, Noah D Goodman, and Jesse Mu. 2023.  Characterizing tradeoffs between teaching via language and demonstrations in multi-agent systems.  _arXiv preprint arXiv:2305.11374_.
  * Zhang et al. (2024) Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. 2024.  In-context principle learning from mistakes.  _arXiv preprint arXiv:2402.05403_.
  * Zhang et al. (2022) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_.
  * Zhao et al. (2024) Zirui Zhao, Wee Sun Lee, and David Hsu. 2024.  Large language models as commonsense knowledge for large-scale task planning.  _Advances in Neural Information Processing Systems_ , 36.

##  Appendix A Dataset Statistics

| Mathematical Reasoning | CommonSense Reasoning | Logical Reasoning
---|---|---|---
Settings | GSM8K | SVAMP | MathQA | AQuA | CSQA | StrategyQA | LogiQA
| Cobbe et al. (2021) | Patel et al. (2021) | Amini et al. (2019) | Ling et al. (2017) | Talmor et al. (2018) | Geva et al. (2021) | Liu et al. (2020)
Dataset statistics
#Testing Examples | 1000 | 638 | 1000 | 428 | 1000 | 980 | 1000
Experience Pool
#Experience Pool (Qwen-Turbo) | 1118 | 1118 | 1118 | 1118 | 2090 | 2090 | 3437
#Experience Pool (GPT-3.5-Turbo) | 1622 | 1622 | 1622 | 1622 | 2474 | 2474 | 3871

Table 5: Statistics and experimental settings of different tasks/datasets.

All datasets in the same reasoning task share the same insight corpus.

Specifically, in the Mathematical Reasoning task, the training set of GSM8K is
utilized to construct the insight corpus, which is subsequently shared by
SVAMP, MathQA, and AQuA.

In the Commonsense Reasoning task, we use the training set of CSQA to
construct the insight corpus, which is shared by StrategyQA.

In the Logical Reasoning task, since there is only one dataset, we use the
training set of LogiQA to construct the insight corpus for itself.

##  Appendix B Task-Level Principels

Task-level Principles for Mathematical Reasoning using GPT-3.5-Turbo Principle
1: Always double-check arithmetic operations. Simple mistakes in addition,
subtraction, multiplication, or division can lead to incorrect answers. Ensure
that each step of the calculation is performed correctly and consider using a
calculator or software to verify results when necessary. Principle 2: Pay
attention to units and conversion factors. When dealing with problems that
involve different units, such as time, weight, or currency, make sure to
convert all quantities to a common unit before performing calculations. This
will prevent errors that arise from misinterpreting or mixing units. Principle
3: Understand the relationships between quantities. When a problem involves
proportional or comparative relationships, such as "half the price" or "twice
as many," ensure that these relationships are applied correctly to the
relevant quantities. Misunderstanding these relationships can lead to
significant errors in the final answer. Principle 4: Keep track of all
elements in the problem. In problems that involve multiple steps or
components, it is crucial to account for each part. Missing out on a component
or forgetting to include it in the final calculation can result in an
incorrect answer. Principle 5: Interpret the context correctly. Ensure that
the real-world implications of the problem are understood. This includes
recognizing the total quantities involved, the number of entities (people,
items, days, etc.), and how these quantities interact with the problem.
Misinterpretation of the context can lead to incorrect assumptions and
calculations. Task-level Principles for Mathematical Reasoning using Qwen-
Turbo Principle 1: Understand the context of the problem before performing
calculations. Many errors stem from a lack of comprehension of the scenario
described in the question. Ensure that the student models grasp the real-world
implications of the problem, such as the number of people involved, the time
frame, and the quantities being considered. Principle 2: Check the units and
ensure consistency throughout the problem. Incorrect solutions often arise
from mishandling units, such as days versus weeks or individual items versus
groups of items. Student models should be trained to pay close attention to
units and convert them appropriately before performing arithmetic operations.
Principle 3: Apply the correct arithmetic operations based on the problem’s
requirements. Misapplication of operations like addition, subtraction,
multiplication, and division can lead to incorrect answers. Student models
should be guided to identify the correct operation for each step of the
problem, considering the relationships between the quantities involved.
Principle 4: Double-check the logic of each step in the solution process.
Errors can occur when student models make assumptions or skip steps that are
crucial for arriving at the correct answer. Encourage the models to review
each step for logical consistency and relevance to the problem statement.
Principle 5: Practice the distribution and association of mathematical
operations. Misunderstandings often occur when dealing with multiple
operations and terms. Student models should be adept at applying the
distributive property and understanding how to group terms for accurate
calculations, especially when dealing with problems that involve proportions
or scaling. Task-level Principles for Logical Reasoning using GPT-3.5-Turbo
Principle 1: Focus on the main idea and avoid overgeneralization. When
summarizing a text, identify the central theme without extending the scope to
include secondary details or broader concepts that are not the primary focus
of the text. Principle 2: Distinguish between descriptive and evaluative
statements. Understand when a text is describing a situation, concept, or
process versus when it is evaluating or critiquing it. This will help in
choosing the most accurate summary or conclusion. Principle 3: Recognize the
implications of specific details. Pay attention to the details that have
significant implications for the overall argument or narrative of the text.
These details often hold the key to understanding the main point or the
correct answer. Principle 4: Understand the context of comparative statements.
When a text compares two or more items, concepts, or scenarios, ensure that
the comparison is correctly interpreted and reflected in the summary or
conclusion. Principle 5: Identify the purpose of the text. Determine whether
the text aims to inform, persuade, argue, or describe, and use this to guide
the selection of the most appropriate summary or answer. This understanding is
crucial for accurate comprehension and response. Task-level Principles for
Logical Reasoning using Qwen-Turbo Principle 1: Distinguish between
assumptions and implications. When evaluating statements, it’s crucial to
differentiate between what is assumed to be true for the scenario to occur and
what might be a consequence of the scenario. This helps in identifying the
foundational premises required for a situation to exist or an opinion to be
valid. Principle 2: Identify the core concept being tested. Focus on the
primary subject matter of the question to avoid being misled by peripheral
details or related concepts that do not directly answer the question. This
ensures that the response is directly relevant to the core concept. Principle
3: Apply logical reasoning to eliminate incorrect options. When faced with
multiple choices, use deductive reasoning to rule out options that are
inconsistent with the information provided. This process of elimination can
often lead to the correct answer by discounting the alternatives that do not
fit the given conditions. Principle 4: Understand the definitions of key
terms. Ensure that the definitions of critical terms are well understood and
applied correctly. Misinterpretation of terms can lead to incorrect
conclusions, so it’s important to have a clear understanding of the vocabulary
used in the question. Principle 5: Analyze the structure of complex problems.
Break down complex problems into smaller, more manageable parts to better
understand the relationships between different elements. This can help in
visualizing the problem and avoiding oversights or misinterpretations that can
lead to incorrect answers. Task-level Principles for CommonSense Reasoning
using GPT-3.5-Turbo Principle 1: Prioritize specificity in problem-solving.
When faced with multiple plausible answers, choose the one that is most
specific to the question’s context. General answers may seem correct but may
not be the best fit given the details provided in the question. Principle 2:
Consider the practical implications of the question. When determining the
correct answer, think about the real-world application and common practices
related to the scenario. Avoid answers based solely on assumptions that do not
align with typical real-life situations. Principle 3: Avoid
overgeneralization. Ensure that the answer selected is not too broad or
encompassing when a more precise option is available. Overgeneralization can
lead to overlooking the most accurate response. Principle 4: Analyze the
question for implicit clues. Some questions contain implicit information that
can guide you to the correct answer. Pay attention to subtle hints within the
question that may indicate a specific answer over others that are merely
correct in a general sense. Principle 5: Distinguish between necessary and
sufficient conditions. Understand the difference between what is needed for a
situation to occur (necessary) and what alone guarantees the situation
(sufficient). Choose the answer that aligns with the necessary condition when
the question calls for it. Task-level Principles for CommonSense Reasoning
using Qwen-Turbo Principle 1: Pay attention to the specificity of language in
the problem statement. Avoid overgeneralization and ensure that the answer
chosen aligns closely with the specific wording and context provided in the
question. Principle 2: Apply logical reasoning to deduce the most appropriate
answer. Consider the typical associations and conventional knowledge related
to the subject matter to guide the selection of the correct response.
Principle 3: Recognize the difference between literal and figurative language.
Ensure that the interpretation of the question is based on the literal meaning
of the words unless the context clearly indicates a figurative or idiomatic
usage. Principle 4: Avoid the influence of personal biases or assumptions that
are not supported by the information given in the problem. Base the answer
solely on the facts presented in the question, without introducing external
assumptions. Principle 5: Understand common knowledge and societal norms as
they pertain to the question. Use this understanding to eliminate answers that
do not fit within the realm of general consensus or widely accepted facts.

##  Appendix C Case Study

Case 1: Mathematical Reasoning
---
Question: Ashley’s pizza delivery costs $15. What is the total amount that
Ashley should give the delivery man if she wants to give a tip that is equal
to 1/5 of the amount she ordered?
CoT Response: First, calculate 1/5 of $15, which is $3. Therefore, the answer
is that Ashley should give the delivery man $3 as a tip. [WRONG]
Task-level Principle: Principle 1: Always double-check arithmetic operations.
Simple mistakes in addition, subtraction, multiplication, or division can lead
to incorrect answers. Ensure that each step of the calculation is performed
correctly and consider using a calculator or software to verify results when
necessary. Principle 2: Pay attention to units and conversion factors. When
dealing with problems that involve different units, such as time, weight, or
currency, make sure to convert all quantities to a common unit before
performing calculations. This will prevent errors that arise from
misinterpreting or mixing units. Principle 3: Understand the relationships
between quantities. When a problem involves proportional or comparative
relationships, such as "half the price" or "twice as many," ensure that these
relationships are applied correctly to the relevant quantities.
Misunderstanding these relationships can lead to significant errors in the
final answer. Principle 4: Keep track of all elements in the problem. In
problems that involve multiple steps or components, it is crucial to account
for each part. Missing out on a component or forgetting to include it in the
final calculation can result in an incorrect answer. Principle 5: Interpret
the context correctly. Ensure that the real-world implications of the problem
are understood. This includes recognizing the total quantities involved, the
number of entities (people, items, days, etc.), and how these quantities
interact within the problem. Misinterpretation of the context can lead to
incorrect assumptions and calculations. Question-level Principle: 1\.
Carefully distinguish between costs and revenues to accurately calculate net
payment or profit; take into account any necessary deductions or additional
charges when determining the final amount. 2\. Verify the inclusion of all
components of an equation or formula, such as base fees or penalties, to
ensure the total calculation reflects the full transaction or scenario. 3\.
Pay attention to direct information and avoid unnecessary complexity in
calculations. Simplify the problem by establishing basic arithmetic relations
from the given data before attempting to solve it. CoT+RICP Response: To solve
this problem, calculate the tip amount, which is 1/5 of the amount she
ordered, $15, resulting in a $3 tip. Then, add the tip amount to the cost of
the pizza delivery. The total amount Ashley should pay is $15 plus $3,
equaling $18. Thus, the answer is $18.[CORRECT]
Case 2: Logical Reasoning
Question: The raw material of cement is very cheap, like limestone and mud
everywhere can be used as raw material of cement.However, the price of cement
will be influenced by the price of oil, because it takes a lot of energy to
convert raw materials from Pakistan to cement in high-temperature
furnaces.what conclusions can be drawn?
CoT Response: The conclusion drawn from the given information is that the
price of cement is influenced by the price of oil due to the energy required
in the conversion process. Therefore, we can reach a conclusion that the price
of cement falls as the price of oil rises. [WRONG]
Task-level Principle: Principle 1: Focus on the main idea and avoid
overgeneralization. When summarizing a text, identify the central theme
without extending the scope to include secondary details or broader concepts
that are not the primary focus of the text. Principle 2: Distinguish between
descriptive and evaluative statements. Understand when a text is describing a
situation, concept, or process versus when it is evaluating or critiquing it.
This will help in choosing the most accurate summary or conclusion. Principle
3: Recognize the implications of specific details. Pay attention to the
details that have significant implications for the overall argument or
narrative of the text. These details often hold the key to understanding the
main point or the correct answer. Principle 4: Understand the context of
comparative statements. When a text compares two or more items, concepts, or
scenarios, ensure that the comparison is correctly interpreted and reflected
in the summary or conclusion. Principle 5: Identify the purpose of the text.
Determine whether the text aims to inform, persuade, argue, or describe, and
use this to guide the selection of the most appropriate summary or answer.
This understanding is crucial for accurate comprehension and response.
Question-level Principle: 1\. Distinguish between what is explicitly stated
and what can be inferred. Avoid extending conclusions beyond the scope of the
given information.
2\. Accurately identify and correlate cause-and-effect relationships; avoid
confusing temporal sequence with causality.
3\. Assess all available evidence within the context provided, resisting the
urge to extrapolate beyond the given data.
CoT+RICP Response: Based on the given information is that petroleum is the
energy source of some high-temperature furnaces for making cement, we can
conclude that Petroleum is the energy source of some high-temperature furnaces
for making cement. [CORRECT]
Table 6: Case studies of solving mathematical reasoning and logical reasoning
problems with CoT+RCIP and CoT prompting on the GPT-3.5-Turbo model. Blue text
indicates the stem, pink text indicates the effective hint, cyan text
indicates the judgment of whether the answer is correct, [CORRECT] denotes
correct, and [WRONG] denotes incorrect.