# Self-Reflection Outcome is Sensitive to Prompt Construction

Fengyuan Liu1, Nouar AlDahoul1, Gregory Eady2, Yasir Zaki1,
Bedoor AlShebli1,*, Talal Rahwan1,*

1New York University Abu Dhabi, UAE 2University of Copenhagen, Denmark
*Joint senior authors. Correspondence<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) demonstrate impressive zero-shot and few-shot
reasoning capabilities. Some propose that such capabilities can be improved
through self-reflection, i.e., letting LLMs reflect on their own output to
identify and correct mistakes in the initial responses. However, despite some
evidence showing the benefits of self-reflection, recent studies offer mixed
results. Here, we aim to reconcile these conflicting findings by first
demonstrating that the outcome of self-reflection is sensitive to prompt
wording; e.g., LLMs are more likely to conclude that it has made a mistake
when explicitly prompted to find mistakes. Consequently, idiosyncrasies in
reflection prompts may lead LLMs to change correct responses unnecessarily. We
show that most prompts used in the self-reflection literature are prone to
this bias. We then propose different ways of constructing prompts that are
conservative in identifying mistakes and show that self-reflection using such
prompts results in higher accuracy. Our findings highlight the importance of
prompt engineering in self-reflection tasks. We release our code at
https://github.com/Michael98Liu/mixture-of-prompts.

Self-Reflection Outcome is Sensitive to Prompt Construction

Fengyuan Liu1, Nouar AlDahoul1, Gregory Eady2, Yasir Zaki1, Bedoor
AlShebli1,*, Talal Rahwan1,* 1New York University Abu Dhabi, UAE 2University
of Copenhagen, Denmark *Joint senior authors. Correspondence:
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Previous research demonstrated that the reasoning capability of language
models can be improved through self-reflection, i.e., prompting the model to
reflect on its own response to identify and correct potential mistakes, devoid
of any external feedback Chen et al. (2023); Madaan et al. (2024). However,
several recent studies have challenged the benefits of self-reflection by
pointing out issues with some of the earlier experiments and providing
evidence that self-reflection may actually deteriorate performance Chen et al.
(2023); Huang et al. (2023); Stechly et al. (2023); Valmeekam et al. (2023).

Table 1 summarizes the prompts used in the self-reflection literature. Looking
at the table, it becomes apparent that there are two issues with how prompts
are commonly constructed. Firstly, with the exception of Zhang et al.’s
(2024), studies on self-reflection tend to rely on a single prompt that is
constructed at will. Secondly, most prompts contain phrases that explicitly
prompt language models to look for mistakes, but it is not clear whether such
phrases bias language models to declare mistakes even when the original
response was correct. Huang et al. (2023) hypothesized that correction prompts
might introduce biases that skew language models away from generating optimal
responses. However, the degree to which self-reflection is sensitive to prompt
construction has not been systematically investigated to date.

Against this background, we conduct experiments to show that self-reflection
can indeed yield outcomes with varying level of accuracy, depending on the
prompts being used. Overall, prompts with higher false positive rates of self-
correction (i.e., tendency to declare mistakes even when the original response
was correct) are negatively correlated with the final accuracy. We then
proposed a a framework called Mixture of Prompts (MoP) to account for biases
inherent in self-reflection prompts. Our experiments reveal that self-
reflection using MoP results in higher accuracy compared with prompting LLMs
to identify mistakes.

Reference |  Self-Reflection Prompt
---|---
Gou et al.’s (2023) |  What’s the problem with the above answer?
Huang et al.’s (2023) |  Assume that this answer could be either correct or incorrect. Review the answer carefully and report any serious problems you find.
Li et al.’s (2024) |  Please review and critique your previous response.
Stechly et al.’s (2023) |  Please check if this coloring is correct … If it is, say ‘Verifier confirmed success.’ Do not provide anything else in your response. If it is incorrect, please point out which same-color vertices share an edge.
Zhang et al.’s (2024) |  Please carefully examine the previous responses for correctness, and provide detailed feedback.
Zhang et al.’s (2024) |  Please review your previous responses for any errors, and provide detailed feedback.
Zhang et al.’s (2024) |  Do you think the previous response is correct or not, and if not please point out where is wrong.
Zhang et al.’s (2024) |  Please carefully evaluate the quality of the previous response and point out if you feel something is not appropriate.
Zhang et al.’s (2024) |  Please double-check the previous response for any errors. If there are any errors, please point them out.
Table 1:  Prompts used in self-reflection experiments. Phrases that we
hypothesize may bias self-reflection outcome are italicized. Zhang et al.
(2024) adopted 10 different reflection prompts but only the first five are
listed here.

##  2 Background and Related Work

Self-reflection. Language models could correct their own response using self-
reflection while incorporating external feedback from humans Zheng et al.
(2023), tools Gou et al. (2023), external metrics, or other language models
Paul et al. (2023). Another way to self-reflect is to rely on the model’s own
reasoning capability Pan et al. (2024). Here, we focus on verbal reflection
without external feedback, akin to us human editing a draft of our own writing
to improve its quality. This approach has been shown to improve the
performance of language models in various tasks Chen et al. (2023).

However, more recent studies challenged these claims by presenting mixed
results when it comes to the benefit of self-reflection Li et al. (2024);
Huang et al. (2023). Notably, Huang et al. (2023) pointed out three major
issues with some earlier experiments and concluded that the observed
performance gain should not be attributed to self-reflection but rather to
various exogenous factors. In our study, we actively avoid the same mistakes
documented in Huang et al.’s (2023), e.g., by refraining from providing
additional information in the self-reflection prompt.

LLMs are sensitive to prompt construction. Studies have shown that prompt
construction is crucial to elicit the reasoning ability of language models.
Meanwhile, adversarially constructed prompts could hurt performance by simply
changing a few words in the prompt Arakelyan et al. (2024); Verma et al.
(2024). Our study contributes to this line of research by documenting a
specific case of sensitivity introduced by prompts, and proposing a novel
framework to mitigate such sensitivity.

##  3 Experiments

###  3.1 Data

The main results in this study are obtained using the Massive Multitask
Language Understanding (MMLU), a benchmark dataset consisting of thousands of
multiple choice questions drawn from 57 academic subjects across STEM, the
humanities, the social sciences Hendrycks et al. (2020). In our experiments,
we use a random set of 1,000 questions sampled from the test set. We further
evaluate different ways of constructing self-reflection prompts using two
additional datasets: GSM8K Cobbe et al. (2021) and CommonSenseQA Talmor et al.
(2018), whose results are reported in the Appendix. Following Kim et al.
(2024) and Huang et al. (2023), these results are obtained using the test set
of GSM8K and the dev set of CommonSenseQA.

###  3.2 Setup

Experiments are performed using Llama-3 8B with 8-bit quantization. We use
multinomial sampling for text generation while setting temperature = 0.4,
top_k = 10, and top_p = 0.7. For each question in the dataset, we first prompt
the large language model (LLM) to provide an initial answer using chain-of-
thought Wei et al. (2022) and four-shot prompting Brown et al. (2020), where
the few-shot examples are taken from the dev from MMLU, and the train set from
GSM8K and CommonSenseQA. We then prompt the LLM with different self-reflection
prompts to obtain an updated answer. We employ the self-consistency method and
sample seven reasoning paths when prompting the LLM Wang et al. (2022). Data
and code to reproduce our results can be found at
https://github.com/Michael98Liu/mixture-of-prompts.

Prompts | Acc. |  $\rightarrow$
---|---|---
Original answer w/o refl. | 64.4% |
Refl. w/ prompt in Gou et al.’s (2023) | 57.8% | 18.9%
Refl. w/ modified prompt | 63.5% | 7.6%
Refl. w/ prompt in Zhang et al.’s (2024) | 47.4% | 40.4%
Refl. w/ modified prompt | 53.6% | 29.8%
Refl. w/ prompt in Li et al.’s (2024) | 55.6% | 23.4%
Refl. w/ modified prompt | 57.5% | 21.3%
Refl. w/ prompt in Zhang et al.’s (2024) | 53.2% | 32.5%
Refl. w/ modified prompt | 54.3% | 27.3%
Table 2: Comparison of accuracy after self-reflection using prompts in Table 1
and the modified versions of the corresponding prompts. Here, acc. stands for
accuracy, and $\rightarrow$ means false positive rate. Figure 1: Correlation
between accuracy and false positive rate in Table 2. Each data point
correspond to a prompt. Pearson’s correlation coefficient and it’s p-value are
reported.

##  4 Reflection Outcome is Sensitive to Prompt Wording

Self-reflection has been regarded as a promising method to identify and
correct mistakes in LLMs’ responses. Hence, we let the LLM self-reflect using
four different prompts found in the literature that contain explicit terms
such as “problem”, “mistake” or “wrong” (Table 1 and Appendix) in the first
set of experiments. Additionally, we modify these prompts to let the LLM
verify whether its initial answers are correct instead of looking for
mistakes. In Table 2, we report the accuracy, defined as the percentage of
final answers that are correct, and the false positive rate of self-
correction, defined as the percentage of correct initial response that are
falsely identified as wrong.

As can be seen, comparing each pair of prompts, the modified prompts result in
higher accuracy and lower false-positive rates across all experiments.
Overall, accuracy has a strong negative correlation with the false positive
rate of self-correction (Figure 1). Among all prompts tested, the highest
false positive rate is 40.4%, which means that over 40% of correct answers are
turned into incorrect ones during the self-reflection sate, while the lowest
false positive rate is 7.6%, suggesting that the outcome of self-reflection is
extremely sensitive to prompt wording.

Experiments | Acc. |  $\rightarrow$ |  $\rightarrow$
---|---|---|---
Orignal answer w/o refl. | 64.1% | - | -
$\texttt{p}_{\uparrow}$ | 59.9% | 17.0% | 62.7%
$\texttt{p}_{\downarrow}$ | 65.0% | 0.3% | 96.0%
MoP (vote) | 65.1% | 0.9% | 94.9%
MoP (unbiased) | 65.2% | 1.2% | 91.7%
MoP (array) | 61.5% | 5.3% | 94.9%
Table 3: The accuracy, false positive rate, and false negative rate of initial
answer (w/o self-reflection) versus self-reflection using prompt to verify
correctness ($\texttt{p}_{\downarrow}$), prompt to identify mistakes
($\texttt{p}_{\uparrow}$), and three variants of MoP, on the MMLU benckmark.

##  5 Mitigating Biases with Mixture of Prompts

###  5.1 Method

Having demonstrated that the accuracy of self-correction can be influenced by
specific phrases in a prompt, we propose a framework called Mixture of Prompts
(MoP) to mitigate such biases (Figure 2). In a nutshell, instead of letting
the LLM generate a reflection based on a single prompt, we allow it to self-
reflect using multiple prompts.

Formally, we hypothesize that a prompt may either bias a language model
towards over-correction (i.e., identifying correct answers as wrong) or under-
correction (i.e., falling to identify wrong answers). The former type of
prompts is denoted as $\texttt{p}_{\uparrow}$, and the latter,
$\texttt{p}_{\downarrow}$. Hence, for a questions, we want to find the most
likely final answer (ans) given the initial answer ($\texttt{ans}_{0}$), as
well as both $\texttt{p}_{\uparrow}$ and $\texttt{p}_{\downarrow}$ for self-
reflection. That is:

$\texttt{ans}=\operatorname*{arg\,max}_{\texttt{ans}}P(\texttt{ans}|\texttt{ans}_{0},\texttt{p}_{\uparrow},\texttt{p}_{\downarrow})$
(1)

Next, we consider three variations of MoP, which differ in the way they
aggregate the outcomes of different prompts.

Plurality vote. Unlike self-consistency Wang et al. (2022), which samples
multiple reasoning paths given a single prompt, here, we sample multiple
reasoning paths conditional on $\texttt{p}_{\uparrow}$ and
$\texttt{p}_{\downarrow}$. When there is a tie (i.e., no single answer has the
plurality vote), the original answer is kept unchanged.

Single balanced prompt. The second variation of MoP uses a carefully
constructed prompt that explicitly prompts the LLM to verify the correctness
of its original answer and look for mistakes.

Prompt array. The third variation of MoP considers two different contexts
simultaneous, one corresponding to $\texttt{p}_{\uparrow}$ and the other
corresponding to $\texttt{p}_{\downarrow}$ while sampling the next token.
Formally, given two prompts, $\texttt{p}_{\uparrow}$ and
$\texttt{p}_{\downarrow}$, we repeatedly sample the next token $W$ based on
the below probability distribution until reaching end-of-sentence. Experiments
with prompt array are conducted using an open source software PromptArray
which was released under Apache License 2.0.

$\begin{split}W_{T}=&\operatorname*{arg\,max}_{W}\Bigl{[}P(W_{T}|\texttt{p}_{\uparrow},W_{1:T-1})\\\
&*P(W_{T}|\texttt{p}_{\downarrow},W_{1:T-1})\Bigr{]}\end{split}$ (2)

Figure 2: Illustration of using Mixture of Prompts to address prompt biases
during self-reflection.

###  5.2 Results

Paradoxically, although self-reflection is proposed as a method of identifying
mistakes, prompts that explicitly look for errors (denoted as
$\texttt{p}_{\uparrow}$) result in the highest false positive rate (more than
10x as high as the alternative approaches) while resulting in the lowest
accuracy (Table 3).

Meanwhile, prompts with low false positive rate tend to have high false
negative rate, which means that they fail to identify an incorrect responses,
showing limited ability of intrinsic self-correction. Overall, self-reflection
can at best marginally increase the accuracy by about 1% compared to the
initial answer without reflection.

##  6 Conclusion

Our study is the first to show conclusive results that prompting LLMs to
identify mistakes in their own responses deteriorates their accuracy. This is
due to the high false positive rates (i.e., falsely identify correct responses
as incorrect) introduced by such prompts. We also find that although
conservative or balanced prompts result in lower false positive rate, they
lead to high false negative rate (i.e., fail to identify incorrect responses).
Taken together, this suggests that LLMs are not able to distinguish between
correct and incorrect responses, which limits their intrinsic self-correction
ability.

## Limitations

Our study demonstrates that self-reflection outcome is sensitive to prompt
construction using a single strategy, i.e., verbal self-reflection without any
external feedback. Hence it is important future work to examine whether more
complex self-reflection strategies are also susceptible to the same type of
biases Pan et al. (2024). As long as human interact with language models via
natural language prompting, prompts may introduce biases that skew language
models to generate sub-optimal responses. Hence, it remains to be shown
whether more complex self-reflection tasks can benefit from the balanced
prompting strategy offered by MoP.

## References

  * Arakelyan et al. (2024) Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024.  Semantic sensitivities and inconsistent predictions: Measuring the fragility of nli models.  _arXiv preprint arXiv:2401.14440_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2023) Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023.  Teaching large language models to self-debug.  _arXiv preprint arXiv:2304.05128_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023.  Critic: Large language models can self-correct with tool-interactive critiquing.  _arXiv preprint arXiv:2305.11738_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Kim et al. (2024) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024.  Language models can solve computer tasks.  _Advances in Neural Information Processing Systems_ , 36.
  * Li et al. (2024) Yanhong Li, Chenghao Yang, and Allyson Ettinger. 2024.  When hindsight is not 20/20: Testing limits on reflective thinking in large language models.  _arXiv preprint arXiv:2404.09129_.
  * Madaan et al. (2024) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024.  Self-refine: Iterative refinement with self-feedback.  _Advances in Neural Information Processing Systems_ , 36.
  * Pan et al. (2024) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024.  Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies.  _Transactions of the Association for Computational Linguistics_ , 12:484–506.
  * Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023.  Refiner: Reasoning feedback on intermediate representations.  _arXiv preprint arXiv:2304.01904_.
  * Stechly et al. (2023) Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023.  Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems.  _arXiv preprint arXiv:2310.12397_.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  CommonSenseQA: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023.  Can large language models really improve by self-critiquing their own plans?  _arXiv preprint arXiv:2310.08118_.
  * Verma et al. (2024) Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. 2024.  On the brittle foundations of react prompting for agentic large language models.  _arXiv preprint arXiv:2405.13966_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Zhang et al. (2024) Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024.  Self-contrast: Better reflection through inconsistent solving perspectives.  _arXiv preprint arXiv:2401.02009_.
  * Zheng et al. (2023) Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023.  Progressive-hint prompting improves reasoning in large language models.  _arXiv preprint arXiv:2304.09797_.

##  Appendix A Supplementary tables

Here, we report additional results using GSM8k (Table 4) and CommonSenseQA
(Table 5).

Experiments | Acc. |  $\rightarrow$ |  $\rightarrow$
---|---|---|---
Orignal answer w/o refl. |  82.5% | - | -
$\texttt{p}_{\uparrow}$ | 64.3% | 24.6% | 60.3%
$\texttt{p}_{\downarrow}$ | 82.6% | 1.4% | 95.0%
MoP (vote) | 82.3% | 1.9% | 93.5%
MoP (unbiased) | 81.3% | 3.1% | 90.5%
MoP (array) | 66.6% | 20.1% | 75.9%
Table 4: The same as Table 3 but using the GSM8K dataset. Experiments | Acc. |  $\rightarrow$ |  $\rightarrow$
---|---|---|---
Orignal answer w/o refl. | 76.1% | - | -
$\texttt{p}_{\uparrow}$ | 71.1% | 12.7% | 66.8%
$\texttt{p}_{\downarrow}$ | 76.2% | 0.3% | 98.2%
MoP (vote) | 76.5% | 0.8% | 94.1%
MoP (unbiased) | 76.3% | 0.6% | 95.9%
MoP (array) | 75.4% | 1.1% | 97.0%
Table 5: The same as Table 3 but using the CommonSenseQA dataset.

##  Appendix B Prompts used to obtain initial answers

###  B.1 MMLU

You are an advanced reasoning agent tasked with answer some multiple choice
questions. Let’s think step by step. Firstly, provide an explanation of why
some answers are wrong and why a particular answer is correct. Secondly,
identify the correct answer. Finally output your final decision starting with
‘‘Therefore, my final decision is: ’’, followed by a, b, c, or d.

###  B.2 CommonSenseQA

You are an advanced reasoning agent tasked with answer some multiple choice
questions. Let’s think step by step. Firstly, provide an explanation of why
some answers are wrong and why a particular answer is correct. Secondly,
identify the correct answer. Finally output your final decision starting with
‘‘Therefore, my final decision is: ’’, followed by a, b, c, d, or e.

###  B.3 GSM8k

You are an advanced reasoning agent tasked with solving the following math
problem. Let’s think step by step. Finally output your final decision starting
with ‘‘####’’, followed by the answer to the math problem using a single
number.

##  Appendix C Prompts used for experiments in Section 4

  1. 1.

What’s the problem with the response below?

  2. 2.

Verify that the response below is correct.

  3. 3.

Do you think the previous response is wrong or not, and if so please point out
what is wrong.

  4. 4.

Do you think the previous response is correct or not, and if so please point
out what is correct.

  5. 5.

Please review and critique your previous response.

  6. 6.

Please review and appraise your previous response.

  7. 7.

Please carefully examine the previous responses to identify any errors, and
provide detailed feedback.

  8. 8.

Please carefully examine the previous responses, and provide detailed
feedback.

Each of the above prompt is followed by You will be provided with the input
and your previous response. Read them carefully and explain your rationale.
Finally output your final decision starting with ‘‘Therefore, my final
decision is: ’’, followed by a, b, c, or d. The input was: <question> Your
decision was: <final-answer> Your response was: <reasoning-path>. in order to
ensure chain-of-though reasoning and the natural language output from the LLM
is parsable.

##  Appendix D Prompts used for self-reflection in Section 5

###  D.1 MMLU and CommonSenseQA

Biased prompt $\texttt{p}_{\uparrow}$: You are an advanced reasoning agent
that can identify mistakes in multiple choice answers. You were asked to
answer some multiple choice questions. Let’s think step by step. Given the
input and the answer, check if the answer is wrong and explain your rationale.
Finally output your final decision starting with ‘‘Therefore, my final
decision is: ’’, followed by a, b, c, or d. The input was: <question> Your
decision was: <final-answer> Your response was: <reasoning-path>.

Biased prompt $\texttt{p}_{\downarrow}$: You are an advanced reasoning agent
that can verify the correctness of multiple choice answers. You were asked to
answer some multiple choice questions. Let’s think step by step. Given the
input and the answer, verify if the answer is correct and explain your
rationale. Finally output your final decision starting with ‘‘Therefore, my
final decision is: ’’, followed by a, b, c, or d. The input was: <question>
Your decision was: <final-answer> Your response was: <reasoning-path>.

Prompt array: You are an advanced reasoning agent that can { identify mistakes
in multiple choice answers. You were asked to answer some multiple choice
questions. Let’s think step by step. Given the input and the answer, check if
the answer is wrong & verify the correctness of multiple choice answers. You
were asked to answer some multiple choice questions. Let’s think step by step.
Given the input and the answer, verify if the answer is correct}. Finally
output your final decision starting with ‘‘Therefore, my final decision is:
’’, followed by a, b, c, or d. The input was: <question> Your decision was:
<final-answer> Your response was: <reasoning-path>. First explain your
rationale and then output your final decision starting with ‘‘Therefore, my
final decision is: ’’, followed by a, b, c, or d.

Single balanced prompt: You are an advanced reasoning agent that can verify
the correctness of, or identify mistakes in, multiple choice answers. You were
asked to answer some multiple choice questions. Let’s think step by step.
Given the input and the answer, check if the answer is wrong or correct and
explain your rationale. Finally output your final decision starting with
‘‘Therefore, my final decision is: ’’, followed by a, b, c, or d. The input
was: <question> Your decision was: <final-answer> Your response was:
<reasoning-path>.

Experiments with MMLU are carried out using the prompts above. Experiments
with CommonSenseQA are carried out using similar prompts, with the only
difference being that there are five different choices: a, b, c, d, and e.

###  D.2 GSM8K

Biased prompt $\texttt{p}_{\uparrow}$: You are an advanced reasoning agent
that can identify mistakes in solutions to math problems. You were asked to
solve the following math problem. Let’s think step by step. Given the input
and the answer, check if the answer is wrong and explain your rationale.
Finally output your final decision starting with ‘‘####’’, followed by the
answer to the math problem using a single number. The input was: <question>
Your decision was: <final-answer> Your response was: <reasoning-path>.

Biased prompt $\texttt{p}_{\downarrow}$: You are an advanced reasoning agent
that can verify the correctness of solutions to math problems. You were asked
to solve the following math problem. Let’s think step by step. Given the input
and the answer, verify if the answer is correct and explain your rationale.
Finally output your final decision starting with ‘‘####’’, followed by the
answer to the math problem using a single number. The input was: <question>
Your decision was: <final-answer> Your response was: <reasoning-path>.

Prompt array: You are an advanced reasoning agent that can {identify mistakes
in solutions to math problems. You were asked to solve the following math
problem. Let’s think step by step. Given the input and the answer, check if
the answer is wrong & verify the correctness of solutions to math problems.
You were asked to solve the following math problem. Let’s think step by step.
Given the input and the answer, verify if the answer is correct}. Finally
output your final decision starting with ‘‘####’’, followed by the answer to
the math problem using a single number. The input was: <question> Your
decision was: <final-answer> Your response was: <reasoning-path>. First
explain your rationale and then output your final decision starting with
‘‘####’’, followed by the answer using a single number.

Single balanced prompt: You are an advanced reasoning agent that can verify
the correctness of, or identify mistakes in, solutions to math problems. You
were asked to solve the following math problem. Let’s think step by step.
Given the input and the answer, check if the answer is wrong or correct and
explain your rationale. Finally output your final decision starting with
‘‘####’’, followed by the answer to the math problem using a single number.
The input was: <question> Your decision was: <final-answer> Your response was:
<reasoning-path>.