# Creating Trumps Critiquing: LLM’s Struggle with Self-Evaluation

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Quick Jog Turning into a Marathon:_
LLMs Struggle with Self-Evaluation

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Molehill turned into a Mountain:_
LLMs Struggle with Self-Evaluation

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Clear Path Turned Labyrinth:_
LLMs Struggle with Self-Evaluation

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Simple Start but a Twisted Ending:_
LLMs Struggle with Self-Evaluation

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _When Creating Trumps Critiquing: (Production over precision?)_
LLMs Struggle with Self-Discrimination

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
# Self-(In)Correct: LLMs Struggle with Self-Discrimination

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
# Self-(In)Correct: Can LLMs Correct their Self-Generated Responses?

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  Self-[In]Correct:
LLMs Struggle with Correcting Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  Self-[In]Correct: Large Language Models Struggle with
Correcting Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  Self-[In]Correct Hypothesis: Large Language Models
Struggle with Correcting Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Lost in Own Thoughts_ : Large Language Models
Struggle with Correcting Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _Tangled in Own Thoughts_ : Large Language Models
Struggle with Refining Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _When Creating Trumps Critiquing_ : Large Language Models
Struggle with Refining Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  _When Creating Trumps Critiquing_ :
LLMs Struggle with Refining Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
#  Self-[In]Correct:
LLMs Struggle with Refining Self-Generated Responses

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper. Dongwei Jiang Jingyu Zhang Orion
Weller Nathaniel Weir
Benjamin Van Durme Daniel Khashabi
Johns Hopkins University
{djiang21<EMAIL_ADDRESS>
###### Abstract

Can LLMs continually improve their previous outputs for better results? An
affirmative answer would require LLMs to be better at _discriminating_ among
previously-generated alternatives, than _generating_ initial responses. We
explore the validity of this hypothesis in practice.

We first introduce a unified framework that allows us to compare the
generative and discriminative capability of any model on any task. Then, in
our resulting experimental analysis of several LLMs, we do not observe those
models’ performance on discrimination to be reliably better than generation.
We hope these findings inform the growing literature on self-improvement AI
systems.

##  1 Introduction

The promise of Large Language Models (LLMs) that can self-improve has brought
both excitement and fear about the future impact of AI. On one hand, this has
captured the attention of AI practitioners excited about Artificial General
Intelligence (AGI). On the other hand, this has raised the alarms for those
already anxious about the rapid pace of developments in AI (Hunt, 2023;
Hutson, 2023) since self-improving AI could evolve in unpredictable ways.

It remains a mystery what is needed for LLMs to continually self-improve. In
human learning, improvement involves learning from one’s mistakes, a process
often supported by feedback from a coach or the environment. Needless to say,
the feedback should accurately identify the mistakes made by the individual
without mistakenly criticizing what has been done correctly. This principle
should be applicable to LLMs as well. For LLMs to reliably self-improve, i.e.
generating their own feedback for self-improvement, the ability to
_discriminate_ whether their own generations are good or not should surpass
the ability to _generate_ good solutions directly. Given the importance of
this capability, it is worth raising a question about the foundations of self-
discrimination: _Are LLMs really better at discrimination than generation?_

Figure 1: Two phases evaluated in our paper. In the generation phase, the
model is fed with generation prompt. Generated answers are collected and
randomly selected to calculate the generation score. In the discrimination
phase, the model is fed with discrimination prompt and generated answers. The
model would choose between generated answers and the score of the chosen
answer would be used for calculating the discrimination score

This paper seeks to answer this question by proposing the Self-[In]Correct
hypothesis (Hypothesis): LLMs are not better at discriminating among
previously-generated alternatives than generating initial responses.
Determining the validity of this hypothesis is crucial, as existing studies
provide initial evidence suggesting that the capability to distinguish between
LLM-generated options is both a sufficient (Tyen et al., 2023) and necessary
(Huang et al., 2023) condition for self-improvement.

It is non-trivial to compare LLMs’ generative capability with their
discriminative capability on the same footing. Recent work, such as West et
al. (2023), compares these capabilities by contrasting model’s generative
ability v.s. their accuracy of choosing the ground-truth answer among answer
options. However, this discriminative setting differs from the generative
setting (and crucially that of self-improvement settings), as the ground-truth
answer is not available during generation, potentially making the upper bound
of discrimination higher. To more clearly measure these abilities, we
implement a two-phase methodology depicted in Figure 1. First, we randomly
select one LLM generation and use its metrics as the indicator of generation
performance. Then, we instruct the LLM to identify the best answer from its
own generations and use the metrics of the selected generation as the
indicator of discrimination performance. Overall, this approach is consistent
with the actual procedure employed in various self-improvement studies (Madaan
et al., 2023; Yuan et al., 2024) and we will expand more on the validity of
this approach in §3.2.

To back up this hypothesis, we conduct experiments covering widely used LLMs
(e.g., LLaMA-2, Mixtral-8$\times$7B) on a diverse set of tasks including
mathematics, world knowledge acquisition, truthful question answering, and
instructions following. Our investigation reveals a surprising finding: while
humans find the task of discrimination simpler than generation (Alexander,
2003), on all tasks we test, LLMs’ performance on discriminating previously
generated answers is not better than randomly choosing one of the generated
answers. Further details are provided in §4.2.

In §5, we provide additional experiments aimed at a deeper understanding of
Self-[In]Correct. §5.1 describes our efforts to aid the discrimination phase
through prompt-engineering to ensure that Self-[In]Correct is not merely an
artifact of suboptimal prompt design. In §5.2, we explore models not pre-
trained with autoregressive objectives (i.e. FLAN-T5, FLAN-UL2) and we find
that Self-[In]Correct does not manifest in these models. To determine if
Self-[In]Correct is an inherent issue for LLMs pre-trained with autoregressive
objective, we simplified the discrimination task in §5.3 by using more easily
distinguishable incorrect options. Under this new experimental setting, we
observe the same models perform better at discrimination than generation. In
§6, we explore the connections and potential impacts of our research on
established self-improvement methods such as Madaan et al. (2023) and Yuan et
al. (2024).

Our contributions in this paper are three-fold:

  * •

We devised a unified framework that allows the testing of generative and
discriminative capability of any LLM on any task, which enables large-scale
evaluation

  * •

We conducted a number of experiments and collected empirical evidence to
support Self-[In]Correct

  * •

We provided additional experiments to better understand Self-[In]Correct and
its implications. Specifically, our findings indicate that Self-[In]Correct
does not occur in LLMs that are not pre-trained autoregressively

##  2 Related Work

#### Self-Improvement with LLMs.

The idea of self-improvement predates the LLM era. Earlier efforts have
utilized generative adversarial networks (GANs) (Subramanian et al., 2017; Yu
et al., 2017) to enhance NLP systems through feedback mechanisms. Welleck et
al. (2023) trains a separate corrector that learns to iteratively correct
imperfect generations.

In the era of LLM, self-improvement with self-feedback has also been studied
in various forms (Pan et al., 2023). Self-Instruct (Wang et al., 2023b)
improves the instruction-following capabilities of pre-trained language models
by bootstrapping off their own generations. Yuan et al. (2024) employs LLMs to
provide rewards for their own generation. Chen et al. (2024) uses a self-play
mechanism where the LLM refines its capability by playing against instances of
itself. Several other recent studies (Liu et al., 2023a; Butt et al., 2024)
also adopted this idea and applied it to different tasks.

Another line of research with potential for self-improvement puts emphasis at
_inference time_. This line of work does not involve updating the model’s
parameters. Instead, self-improvement is achieved by having the model generate
reflection on its own outputs and adjust future outputs based on these
reflections. Recent work in this area covers various formats from verbal
(Shinn et al., 2023) to scoring feedback (Madaan et al., 2023), and addresses
diverse tasks including autonomous operations (Shinn et al., 2023), academic
benchmarks (Madaan et al., 2023), and daily activities (Wang et al., 2023a).

When external ground-truth feedback exists, LLMs are capable of self-
improvement. Gou et al. (2024) demonstrates that LLMs can verify and correct
their initial responses through the interaction with various external tools.
Tyen et al. (2023); Shinn et al. (2023) successfully improves LLM’s
performance on various tasks with the help of ground-truth feedback. Shifting
focus to a different approach, intrinsic self-correction involves LLMs trying
to refine their initial answers based solely on their inherent capabilities.
However, this method’s effectiveness, particularly for reasoning tasks, has
been called into question. Critiques have emerged (Huang et al., 2023;
Valmeekam et al., 2023; Tyen et al., 2023) that contend the efficacy of
reported self-improvement is no better than self-consistency (Wang et al.,
2022a) and that such improvements is largely due to the inferior initial
response. The setting used in our work is similar to intrinsic self-
improvement. We add to the discourse by examining LLMs’ generative and
discriminative capacities beyond reasoning tasks.

#### Discrepancy between LLM generation and evaluation.

For humans, distinguishing a good solution from a bad one is often easier than
coming up with a solution from scratch (Alexander, 2003). However, recent
studies are starting to question if the same applies to LLMs. West et al.
(2023) investigated multiple NLP tasks , and showed that LLMs often struggle
to understand their own outputs. To evaluate the generation and discrimination
performance of LLMs, Liu et al. (2023b) conducted experiments focusing on
summarization. Arora & Kambhampati (2023) conducted similar experiments in the
domain of planning. Our work differs from previous work in that we’re
evaluating this discrepancy on a wider range of tasks using a unified metric.
We’re also trying to uncover the reasons behind the discrepancy.

#### Using LLMs for self-evaluation.

Recent studies indicate the potential of LLM evaluation that is close to human
level (Chiang & Lee, 2023; Gilardi et al., 2023). However, for the task of
self-evaluation, concerns have been raised by Valmeekam et al. (2023); Huang
et al. (2023), who pointed out that LLM encounters difficulties in self-
evaluating its generation for mathematical tasks. Further research by Stechly
et al. (2024; 2023); Valmeekam et al. (2023) has uncovered models’ limitations
on self-evaluation for tasks requiring complex reasoning and planning. Our
work seeks to explore the efficacy of LLM self-evaluation in a broader range
of tasks.

##  3 Self-[In]Correct

In this section, we clarify our definition, introduce our evaluation framework
(Figure 1), and present our hypothesis.

###  3.1 Establishing an Evaluation Criteria to Compare Generation vs.
Discrimination

Given a task $T$ with an evaluation dataset $D=\\{(x_{i},y_{i})\\}_{i=1}^{m}$
and evaluation metric $f$, we use the same LLM, denoted by $P_{\text{LM}}$,
for both generation and discrimination. For each evaluation input $x_{i}$, we
first sample $n$ candidate generations $g_{1}(x_{i}),\dots,g_{n}(x_{i})\sim
P_{\text{LM}}(x_{i})$ using the default task prompt (generation prompt). We
maintain a low temperature during sampling to ensure the generated outputs are
all highly probable.

#### Evaluating generation.

The performance of the generative phase for each evaluation sample $x_{i}$ is
defined by the metric score $S_{\text{gen}}(x_{i})$ of a randomly chosen
generation from the $n$ candidate generations
$G(x_{i})=\\{g_{1}(x_{i}),g_{2}(x_{i}),\ldots,g_{n}(x_{i})\\}$, as follows:

$S_{\text{gen}}(x_{i})=f\Bigl{(}g_{\text{rand}}(x_{i}),y_{i}\Bigr{)},$

where $g_{\text{rand}}$ is a randomly-sampled generation
$g_{\text{rand}}(x_{i})\sim G(x_{i})$. $f(g_{j}(x_{i}),y_{i})$ represents the
metric $f$ applied to the $j$-th generation output for the $i$-th evaluation
sample and $g_{\text{rand}}(x_{i})$ represents one random candidate
generations for that sample. Because the candidate generations for each sample
are produced by sampling from the language model $P_{\text{LM}}$ using the
same hyper-parameters (temperature, top-$p$, etc), choosing a random candidate
from $G(x_{i})$ is essentially equivalent to generating an output directly
from $P_{\text{LM}}(x_{i})$. The overall generation performance
$S_{\text{gen}}$ is then the average of
$\\{S_{\text{gen}}(x_{1}),S_{\text{gen}}(x_{2}),\ldots,S_{\text{gen}}(x_{m})\\}$
across all evaluation samples.

#### Evaluating discrimination.

To assess the discrimination performance, we feed the generations back to
$P_{\text{LM}}$ and prompt it to identify the most suitable answer. For each
task $T$, we construct a discriminative prompt $x_{\text{disc},T}$ and feed it
generations labeled from 1 to $n$ (the prompts we used in discrimination phase
can be found in Appendix A). Using few-shot prompting, we guide
$P_{\text{LM}}$ to output the label of the preferred chosen answer and the
label chosen in $\\{1,2,\ldots,n\\}$ is determined by greedily decoding the
output of $P_{\text{LM}}(\cdot|x_{\text{disc},T}(G(x_{i})))$. The
discrimination performance for each sample $i$ is quantified by:

$S_{\text{disc}}(x_{i})=f\Big{(}g_{\text{chosen}}(x_{i}),y_{i}\Big{)}.$

To derive an overall measure of discrimination performance $S_{\text{disc}}$,
we average the individual scores
$\\{S_{\text{disc}}(x_{1}),S_{\text{disc}}(x_{2}),\ldots,S_{\text{disc}}(x_{m})\\}$
across all evaluation samples.

###  3.2 Hypothesis Formulation

Given the above definitions, our main hypothesis becomes easy to formalize.
For any given task, denote DG-Diff as the  difference between discrimination
performance and generation performance,

$\textsc{DG-Diff}=S_{\text{disc}}-S_{\text{gen}}.$

Our main hypothesis is:

######  Hypothesis.

LLMs are not universally better at discriminating among previously generated
alternatives than generating initial responses. Specifically, $\textsc{DG-
Diff}\leq$ $\epsilon$, where $\epsilon$ is a small enough value that doesn’t
result in statistically significant superiority of discriminative performance
($S_{\text{disc}}$) over generative performance ($S_{\text{gen}}$) across the
samples.

Note the term “discrimination” is employed with precision in this context. It
is selected over alternatives such as “evaluation” which may also imply the
provision of qualitative or quantitative feedback on generated responses, or
“verification”, which typically results in a binary judgment of truthfulness.

An important design choice in our framework is that the candidate generations
$G(x_{i})$ are shared across the generative and discriminative phases. This
design choice allows us to formulate the generative phase as a random multiple
choice among pre-generated candidates. As a result, it allows apples-to-apples
comparison with the discriminative phase, where the task is using LLM for
multiple choice among the same candidates.111Under this setup, the generative
and discriminative phases will always have the same upper/lower bound of
possible scores.

Our framework applies the task’s original metrics in both the generative and
discriminative phases, ensuring consistency across assessments. By eliminating
the need for human input, our framework is more scalable and cost-effective
than methods like West et al. (2023); Zheng et al. (2023b), which depend on
human involvement. Our metrics are also closely aligned to the actual process
that’s employed in self-improvement literature (Shinn et al., 2023; Madaan et
al., 2023), where the model is asked to choose the best answer from a list of
generations. Nevertheless, we would like to mention that because generation
and discrimination are two very different processes, the metrics used in this
paper are only proxies to evaluate those two important capabilities.

##  4 Empirical Support for Self-[In]Correct

In this section, we describe our experimental setup (§4.1) and lay out the
main findings (§4.2).

Task | Split | #Eval | #Shots | Task Type | Metric $f(.)$ | Metric scale
---|---|---|---|---|---|---
GSM8K | Test | 1319 | 2 | Math Word Problem | Accuracy | $[0,100]$
TriviaQA | Val | 17944 | 2 | Question Answering | Accuracy | $[0,100]$
MT-Bench | Test | 160 | 3 | Instruction Following | GPT-4 score | $[0,10]$
TruthfulQA | Val | 817 | 2 | Question Answering | GPT-judge | $[0,100]$
Table 1: Configuration of experimental tasks. “Split” specifies which subset
the data originates from. “#Eval” indicates the number of instances used for
evaluation. “#Shots” specifies the number of few-shot examples employed for
evaluation. To evaluate TruthfulQA generations, we follow Lin et al. (2022)
and develop two “GPT-judges” by fine-tuning GPT-3 models with provided data.

###  4.1 Experimental Setup

#### Tasks.

A summary of the tasks we evaluate on is provided in Table 1. We assess our
hypothesis on a diverse set of tasks including GSM8K (Cobbe et al., 2021) for
math, TriviaQA (Joshi et al., 2017) for world knowledge, TruthfulQA (Lin et
al., 2022) for truthfulness in question answering, and MT-Bench (Zheng et al.,
2023b) for instruction following. These represent a diverse set of benchmarks
used to evaluate LLMs across various domains. For TriviaQA, we use the
rc.nocontext setup, which means the model relies solely on its parametric
knowledge to answer the question correctly without accompanying context or
documents. For TruthfulQA, we use the generation setup, where the model
generates responses to a set of questions. The metrics scale for MT-Bench is
0-10 Zheng et al. (2023b).

#### Task metrics.

The list of task-specific metrics $f(.)$ is provided in Table 1. The
evaluation for GSM8K, TriviaQA and TruthfulQA is conducted using lm-
evaluation-harness222https://github.com/EleutherAI/lm-evaluation-harness,
which provides a standardized framework for assessing model performance across
benchmarks. The evaluation for MT-Bench is done with
llm_judge333https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge,
which use GPT-4 score (Zheng et al., 2023b) to score the generated answer from
models. To evaluate TruthfulQA generations, we follow Lin et al. (2022) and
develop two “GPT-judges” by fine-tuning GPT-3 models444The original “GPT-
judges” were fine-tuned with curie models which are no longer available for
fine-tuning. Therefore, we use davinci-002 which is larger than curie. with
the provided data . Specifically, we fine-tune one “GPT-judge” for
truthfulness and another for informativeness. Finally, we report the
percentage of answers that are both truthful and informative as the final
metric for TruthfulQA.

#### Handling failure modes during evaluation.

While evaluating the discrimination phase, if the model output does not adhere
to the expected format (i.e., integers indicating the selected answer), we
deem that as a failure. However, to be generous to the discrimination step, we
avoid assigning it zero. Instead, we credit it with the generation that scores
lowest (using our metric $f$) among other candidate answers:
$S_{\text{disc}}(x_{i})=\min_{g(x_{i})\in G(x_{i})}f(g(x_{i}),y_{i})$. We also
try to make the discriminator output one of the answers directly in the case
of a failure, hoping that bypassing this extra step of identifying the
multiple-choice options would simplify the discrimination phase. However, we
observe an increased percentage of invalid discrimination output with similar
discrimination performance (see Appendix C for more details). In our
experiments, we observe that the average rate of invalid responses remained
low (~10%). Given that there is also a certain proportion of invalid outputs
in the generation phase that wouldn’t get any credit when selected, we believe
that the occurrence of invalid discrimination outputs does not significantly
impact our overall findings.

#### Models.

We employ a range of models including LLaMA-2 Base models (7B, 13B, and 70B),
LLaMA-2 Chat models (7B, 13B and 70B), Mixtral $8\times 7$B-Instruct-v0.1,
GPT-3.5-turbo, and GPT-4-turbo.555We used the latest GPT-3.5-turbo-0125 and
GPT-4-0125-preview model. The experiments were conducted in Feb 2024. For the
evaluation of each model, we adapt our prompts to be compatible with the
keywords used in their [pre-]training. For example, when prompting LLaMA-2
Chat models we use <SYS>, <INST> keywords to indicate system and instruction
prompts.

#### Model hyper-parameters.

During the generation phase, we use the default hyperparameter specified in
lm-eval-harness for all tasks, except for temperature, which we have adjusted
to $0.7$. We use an above $0$ temperature to obtain distinct generations upon
multiple rounds of sampling. At the same time, during the discrimination
phase, we set the temperature to 0 to avoid any randomness in the results.

###  4.2 Main Findings

####  DG-Diff is generally small or negative.

From extensive results in Table 2, it can be concluded that DG-Diff is
generally small or negative, i.e., $S_{\text{disc}}$ is generally lower than
$S_{\text{gen}}$. This conclusion also holds when we examine the average task
performance for all models. An ablation experiment is conducted in Appendix D
that implements several different prompts and we find prompt variations do not
significantly affect DG-Diff. Although $S_{\text{gen}}$ sometimes exceed
$S_{\text{disc}}$, in such cases DG-Diff remains quite small. All these
observations lend support for Self-[In]Correct.

####  DG-Diff remains small or negative upon fine-tuning.

LLaMA-2 Chat models are fine-tuned with both instruction-tuning and RLHF while
LLaMA-2 Base models are only pre-trained with autoregressive objective. It is
reasonable to expect instruction-tuned models would exhibit better performance
in the discrimination phase because instruction-tuning is shown to make models
better at solving a variety of tasks. Furthermore, classification tasks (that
resemble our discrimination setup) are well-represented in most instruction-
tuning datasets (Wang et al., 2022b; Bach et al., 2022; Longpre et al., 2023).
However, our empirical findings do not support it. Notably, LLaMA-2 13B and
LLaMA-2 70B models outperformed their fine-tuned counterparts in our
evaluations.

| $\textbf{{DG-
Diff}{}}_{(\textbf{S}_{\textbf{gen}}~{}\rightarrow~{}\textbf{S}_{\textbf{disc}})}$
---|---
|  GSM8K |  TriviaQA |  MT-Bench |  TruthfulQA
LLaMA-2 7B |  -0.6(9.2→8.6) |  -16.9(37.1→20.2) |  -0.09(3.34→3.25) |  -4.7(30.5→25.8)
LLaMA-2 13B |  0.0(16.8→16.8) |  1.4(45.2→46.6) |  -0.12(4.15→4.03) |  2.1(26.8→28.9)
LLaMA-2 70B |  3.2(43.0→46.2) |  3.2(53.2→56.4) |  -0.12(4.87→4.75) |  0.5(28.9→29.4)
$\textbf{LLaMA-2 7B}_{\textbf{Chat}}$ |  -2.8(20.4→17.6) |  -0.1(16.1→16.0) |  -0.13(5.45→5.32) |  1.4(48.8→50.2)
$\textbf{LLaMA-2 13B}_{\textbf{Chat}}$ |  -5.5(28.3→22.8) |  0.0(25.5→25.5) |  -0.51(5.67→5.16) |  -0.1(44.9→44.8)
$\textbf{LLaMA-2 70B}_{\textbf{Chat}}$ |  -5.9(42.5→36.6) |  -1.6(47.8→46.2) |  -0.17(6.25→6.08) |  0.9(48.6→49.5)
$\textbf{Mixtral-8x7B}_{\textbf{Instruct}}$ |  4.3(56.6→60.9) |  -3.4(58.8→55.4) |  -0.20(7.39→7.19) |  -0.4(61.1→60.7)
GPT-3.5-turbo |  1.1(75.3→76.4) |  2.1(67.1→69.2) |  0.17(7.74→7.91) |  0.4(65.7→66.1)
GPT-4-turbo |  0.7(93.6→94.3) |  0.2(79.9→80.1) |  – |  1.7(77.4→79.1)
Task Avg. |  -0.61 |  -1.67 |  -0.15 |  0.2
Table 2:  Performance change defined as $\textsc{DG-
Diff}{}:=S_{\text{disc}}-S_{\text{gen}}$ for various mainstream LLMs. We do
not test GPT-4-turbo on MT-Bench to avoid self-evaluation bias (He et al.,
2023). The generation performance and discriminative performance is shown as
subscript: $(S_{\text{gen}}~{}\rightarrow~{}S_{\text{disc}})$. For majority of
our results, DG-Diff is small or negative, indicating similar or worse
performance in the discrimination phase.

##  5 Further Analysis of Self-[In]Correct

In this section, we outline experiments designed to provide further analysis
of Self-[In]Correct.

###  5.1 Aiding Discrimination Phase via Prompt-Engineering

One might argue our current prompting setup doesn’t fully capitalize on the
model’s capacity for discrimination. To make sure Self-[In]Correct isn’t an
artifact of poor prompt engineering, we conduct additional experiments with
LLaMA-2 Chat models on GSM8K, TriviaQA, and MT-Bench because their DG-DIFF on
those tasks is mostly negative.

#### Adding in-context learning examples helps discrimination, though DG-Diff
remains small or negative.

Increasing the number of in-context learning (ICL) demonstrations is shown to
improve performance (Brown et al., 2020). Is it possible that increasing the
number of ICL examples in the discrimination phase will improve it, so much
that DG-Diff becomes consistently positive? To evaluate the effect of
increasing ICL examples, we conduct experiments where the number of ICL
examples (#Shots) during the discrimination stage is doubled or tripled
relative to the baseline established in Table 1. Note that we didn’t triple
the number of ICL examples for MT-Bench because it exceeds the context length
for LLAMA-2 Chat models (4096 tokens). The results, presented in Table 3,
indicate that while increasing the number of ICL examples tends to increase
DG-Diff, it remains small or negative. Furthermore, the performance
improvement from adding ICL examples does not exhibit a consistent monotonic
trend.

#### Adding chain-of-thought rational shows minimal impact on DG-Diff.

Recently, Stechly et al. (2024) pointed out that LLM evaluation also involves
multi-step reasoning. To help with the reasoning in the discrimination phase,
we add chain-of-thought rationals in the few-shot examples while keeping the
number of examples constant. For GSM8K, we do not report anything since our
default evaluation already contains rationales for answer selection. For
TriviaQA, the CoT rationales explain the logic behind choosing an option. For
MT-Bench, we supplement explanations for preferring one answer over another. A
comparison between our prompts (w/ and w/o CoT rationales) is available in
Appendix A. As shown in Table 3, the inclusion of CoT rational only shows
minimal impact.

Model | LLaMA-2 7B Chat | LLaMA-2 13B Chat | LLaMA-2 70B Chat
---|---|---|---
Setup | +2$\times$#ICL | +3$\times$#ICL | +CoT | +2$\times$#ICL | +3$\times$#ICL | +CoT | +2$\times$#ICL | +3$\times$#ICL | +CoT
GSM8K | -1.4 | 0.1 | - | -5.9 | -6.8 | - | -5.8 | -3.9 | -
TriviaQA | -0.4 | 0.2 | -0.3 | 0.1 | 0.1 | -0.3 | -1.7 | -0.5 | -1.8
MT-Bench | -0.09 | - | -0.06 | -0.53 | - | -0.41 | -0.19 | - | -0.18
Table 3:  DG-Diff upon various modifications with LLaMA-2 Chat models. “+ 2
$\times$ #ICL” means doubling the number of in-context demonstrations during
the discrimination phase. “+ 3 $\times$ #ICL” means tripling the number of in-
context demonstrations. “+ CoT” stands for adding Chain-of-Thought rationales
for the few-shot examples. Extra prompt-engineering techniques during
discrimination do not consistently close the performance gap.

###  5.2 Is Autoregressive Pre-training Related to Self-[In]Correct?

The majority of modern LLMs are pre-trained with an autoregressive objective.
Recent studies suggest that autoregressive objectives used during pre-training
may have unexpected impacts on LLM behavior (McCoy et al., 2023). Since the
pre-training process of autoregressive models is more akin to generation than
discrimination, we hypothesize Self-[In]Correct is also partially caused by
the use of autoregressive pre-training objective.

To test this hypothesis, we evaluated Flan-T5-XXL (11B) and Flan-UL2 (20B) on
the same tasks used in Table 2. Flan-T5-XXL is pre-trained using a span
corruption objective, where the loss is only calculated on the corrupted span
(Raffel et al., 2020). Flan-UL2 (Chung et al., 2022) is pre-trained using
mixture-of-denoisers that combines multiple denoising objective functions. Our
findings, detailed in Table 4, reveal their DG-Diff across all tasks are
positive except for Flan-T5-XXL on MT-Bench. In fact, both models also
demonstrate significantly higher DG-Diff compared to the autoregressive models
we tested in Table 2. This outcome lends empirical support to the hypothesis
that Self-[In]Correct could be related to autoregressive pre-training.

| $\textbf{{DG-
Diff}{}}_{(\textbf{S}_{\textbf{gen}}~{}\rightarrow~{}\textbf{S}_{\textbf{disc}})}$
---|---
| GSM8K | TriviaQA | MT-Bench | TruthfulQA
Flan-T5 XXL | 1.3(13.3→14.4) | 5.8(28.7→34.5) | -0.06(2.02→1.96) | 6.0(20.1→26.1)
Flan-UL2 | 0.5(21.6→22.1) | 4.2(52.7→56.9) | 0.16(1.98→2.14) | 4.8(31.3→36.1)
Task Avg. | 0.9 | 5.0 | 0.05 | 5.4
Table 4: Flan-T5-XXL and Flan-UL2 tested on the same setup as Table 2. DG-Diff
for all models across all tasks are positive except for Flan-T5-XXL on MT-
Bench. Both models demonstrate significantly higher average DG-Diff compared
to autoregressive models.

It is also important to note that the pre-training processes for these two
model classes differ from autoregressively pre-trained counterparts beyond the
objective function. For example, Flan-T5 is pre-trained with most inputs
provided, except for the corrupted spans. Furthermore, the datasets used for
pre-training these LLMs can vary. Therefore, when uncovering the underlying
reason why Self-[In]Correct does not occur on Flan-T5 and Flan-UL2, caution
should be exercised before drawing definitive conclusions.

###  5.3 Controlled modification of experimental setting: DG-Diff notably
improves when the negative candidates given to the discrimination phase are
simplified

Here we consider the extent to which Self-[In]Correct may hold. For example,
is Self-[In]Correct potentially a fundamental limitation of LLMs pre-trained
with an autoregressive objective, or can a change in data distribution alter
the outcome? To address this question, we conduct experiments in an
unconventional setting that simplifies the discrimination phase by
substituting incorrect candidates with simpler ones.

The experiments are conducted on TriviaQA and GSM8K. TriviaQA contains a wide
range of answer categories, including names, locations, historical events,
etc. For this dataset, we simplify the discrimination phase by substituting
incorrect answer generations for question $A$ with correct answer generations
from another question $B$ (left panel in Figure 2). This approach simplifies
the discrimination phase as it replaces hard-to-discriminate distractors with
simpler ones. As for GSM8K, we create simplified distractors by randomly
multiplying or dividing incorrect generated answers by $100$ (right panel in
Figure 2).

Figure 2: Left: Simplified negative candidates setup for TriviaQA, where
incorrectly generated answers are replaced with randomly selected correct
answers from another question. Right: Simplified negative candidates setup for
GSM8K, where incorrect generated answers are multiplied or divided by 100 to
simplify the discrimination process

From Figure 3, it is evident that simplifying the incorrect candidates
improves DG-Diff. For TriviaQA, ${S}_{disc}$ exceeds ${S}_{gen}$ by a large
margin. For GSM8K, all models tested also demonstrate improved DG-Diff in this
setting.

Figure 3:  “Simplifed” setting uses simplified incorrect answers for the
discrimination phase. DG-Diff improves notably with simplified negative
candidates, indicating the sensitivity of the discriminative phase to data
distribution.

##  6 Further Discussion

#### Does Self-[In]Correct contradict prior findings in self-refinement?

The process of self-refine involves utilizing the same LLM to provide feedback
for its own generation and using the feedback to refine the generation. Both
Huang et al. (2023) and Madaan et al. (2023) suggested LLMs can self-refine on
tasks other than reasoning. Does this contradict our assertions?

We replicated the experiment outlined in Madaan et al. (2023) and observed the
following (more detail in Appendix B):

(1) For some evaluated tasks, certain aspects can be exploited for
artificially amplifying task performance without actually reasoning on the
feedback.  For example, with the task of constrained generation, where the
objective is to generate sentences containing specific keywords, self-refine
with LLMs often leads to progressively longer sentences that build on previous
ones. Thus, even if the refined sentences do not incorporate new keywords and
continue to grow longer (often, ignoring the feedback from the prior round),
the task performance still shows a monotonic improvement.

(2) For some evaluated tasks, the evaluation score assigned by the model for
each iteration of self-refine is not monotonically increasing. For acronym
generation and dialogue response generation, the model often assigns a lower
score to refined outputs. Thus, the model is not effectively performing
iterative self-refinement, and any observed improvement might stem from a
lower quality of initial outputs, as noted in Huang et al. (2023).

(3) Quantifying the percentage of times models prefer self-refined subsequent
generations to the previous generation, a marginal preference for self-refined
generation was observed. We used the same models to discriminate between
previous generations and self-refined subsequent generations for tasks
mentioned in Madaan et al. (2023). Our results indicate that models prefer
self-refined generations only around 54% of the time.

In summary, these observations and the findings presented in Table 5 suggest
that the results from Madaan et al. (2023) do not directly contradict our
hypothesis.

#### Implication of Self-[In]Correct for self-rewarding LLM

Recent work (Yuan et al., 2024) uses preference pairs of the form {instruction
prompt $x$, winning response $y\\_win$, losing response $y\\_lose$} to provide
self-reward for instruction-following fine-tuning. It is important to clarify
that although the setup for this work also includes discriminating between
previously generated outputs, our findings do not undermine its efficacy. Such
techniques can be useful as long as the ranking of winning responses and
losing responses in those preference pairs is correct. This can be facilitated
when winning and losing pairs are taken from the highest and lowest-scoring
responses from the candidate responses.

However, an interesting pattern from Yuan et al. (2024) is that there seem to
be diminishing returns after a few iterations of self-rewarding. We
hypothesize this may be linked to Self-[In]Correct because if the average
ability of LLMs to discriminate is inferior to their ability to generate, it
becomes challenging to engage in a virtuous cycle that simultaneously enhances
the model’s capability to follow instructions and generate self-rewards.

#### Potential influence of lengthier discrimination prompt on
Self-[In]Correct

The prompt used in the discrimination phase is inherently lengthier than the
generation prompt as it also includes the generated candidate answers. This
increase in length may pose challenges to the model’s processing capabilities.
Investigating the impact of prompt length is complex. Simply adding
superfluous content to lengthen the generation or discrimination prompt might
not yield clear results, as such additions could unintentionally influence the
outcomes. Therefore, we highlight this area for further exploration to better
understand the implications of prompt length for Self-[In]Correct.

##  7 Limitations

One limitation of our research is the inability to test the impact of pre-
training data. The scale of pre-training data makes it difficult to assess its
effect.

Another limitation of our study is the comprehensiveness of our experiments.
Extending our experiments to more recent models and more diverse tasks can
strengthen the validation of Self-[In]Correct.

Determining a model’s preference over several candidate generations can be
challenging due to various biases. Following the methodology used in other
self-improvement studies (Yuan et al., 2024), we employ LLM-as-a-Judge
prompting (Zheng et al., 2023a) to elicit answer choice from the model. It is
conceivable that the LLMs we examine can be biased toward certain answer
options or different answer formats (e.g., labels of A/B/C/D or
[1]/[2]/[3]/[4]). Another method is to rank candidate answers using the LLM’s
assigned probability over each answer text, which we did not explore. It is
worth noting that this approach can be biased too. For example, the LLM might
be biased to evaluate text fluency instead of the correctness of the answers.

##  8 Conclusion

We focused on the question of whether language models are strictly better at
discriminating their prior generations vs. generating responses directly. We
proposed a metric for comparing these capabilities and used it to evaluate
several current LLMs. For those models and tasks, we do not observe that
discrimination is reliably better than generation, in fact, we often observed
it was worse. These results raise concerns about the potential for LLM self-
improvement on any task.

## References

  * Alexander (2003) Patricia A Alexander.  The development of expertise: The journey from acclimation to proficiency.  In _Educational researcher_ , 2003.  URL https://www.jstor.org/stable/3700080.
  * Arora & Kambhampati (2023) Daman Arora and Subbarao Kambhampati.  Learning and leveraging verifiers to improve planning capabilities of pre-trained language models.  _CoRR_ , 2023.  URL https://arxiv.org/abs/2305.17077.
  * Bach et al. (2022) Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush.  Promptsource: An integrated development environment and repository for natural language prompts.  _ArXiv_ , abs/2202.01279, 2022.  URL https://arxiv.org/abs/2202.01279.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.  URL https://arxiv.org/abs/2005.14165.
  * Butt et al. (2024) Natasha Butt, Blazej Manczak, Auke Wiggers, Corrado Rainone, David Zhang, Michaël Defferrard, and Taco Cohen.  Codeit: Self-improving language models with prioritized hindsight replay.  _arXiv preprint arXiv:2402.04858_ , 2024.  URL https://arxiv.org/abs/2402.04858.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  _CoRR_ , 2024.  URL https://arxiv.org/abs/2401.01335.
  * Chiang & Lee (2023) David Cheng-Han Chiang and Hung-yi Lee.  Can large language models be an alternative to human evaluations?  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), _ACL_ , 2023.  URL https://arxiv.org/abs/2305.01937.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models.  _CoRR_ , 2022.  URL https://arxiv.org/abs/2210.11416.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems, 2021.  URL https://arxiv.org/abs/2110.14168.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ , July 2023.  ISSN 1091-6490.  URL https://arxiv.org/abs/2303.15056.
  * Gou et al. (2024) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen.  Critic: Large language models can self-correct with tool-interactive critiquing, 2024.  URL https://arxiv.org/abs/2305.11738.
  * He et al. (2023) Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov.  On the blind spots of model-based evaluation metrics for text generation.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _ACL_ , July 2023.  URL https://arxiv.org/abs/2212.10020.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.  Large language models cannot self-correct reasoning yet, 2023.  URL https://arxiv.org/abs/2310.01798.
  * Hunt (2023) Tamlyn Hunt.  Here’s why ai may be extremely dangerous–whether it’s conscious or not.  _Scientific American_ , 25, 2023.  URL https://www.scientificamerican.com/article/heres-why-ai-may-be-extremely-dangerous-whether-its-conscious-or-not/.
  * Hutson (2023) Matthew Hutson.  Can we stop runaway a.i.?  _New Yorker_ , 2023.  URL https://www.newyorker.com/science/annals-of-artificial-intelligence/can-we-stop-the-singularity.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _ACL_ , 2017.  URL https://arxiv.org/abs/1705.03551.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans.  TruthfulQA: Measuring how models mimic human falsehoods.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _ACL_. Association for Computational Linguistics, May 2022.  URL https://arxiv.org/abs/2109.07958.
  * Liu et al. (2023a) Jiacheng Liu, Ramakanth Pasunuru, Hannaneh Hajishirzi, Yejin Choi, and Asli Celikyilmaz.  Crystal: Introspective reasoners reinforced with self-feedback.  _arXiv preprint arXiv:2301.04921_ , 2023a.  URL https://arxiv.org/abs/2310.04921.
  * Liu et al. (2023b) Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan.  Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization.  _CoRR_ , 2023b.  URL https://arxiv.org/abs/2311.09184.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ , 2023.  URL https://arxiv.org/abs/2301.13688.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  _CoRR_ , 2023.  URL https://arxiv.org/abs/2303.17651.
  * McCoy et al. (2023) R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths.  Embers of autoregression: Understanding large language models through the problem they are trained to solve.  _CoRR_ , 2023.  URL https://arxiv.org/abs/2309.13638.
  * Pan et al. (2023) Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang.  Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.  _ArXiv_ , abs/2308.03188, 2023.  URL https://api.semanticscholar.org/CorpusID:260682695.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_ , 2020.  URL https://arxiv.org/abs/1910.10683.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  In _NeuralPS_ , 2023.  URL https://arxiv.org/abs/2303.11366.
  * Stechly et al. (2023) Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati.  GPT-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems.  _CoRR_ , 2023.
  * Stechly et al. (2024) Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati.  On the self-verification limitations of large language models on reasoning and planning tasks.  _CoRR_ , 2024.  URL https://arxiv.org/abs/2402.08115.
  * Subramanian et al. (2017) Sandeep Subramanian, Sai Rajeswar, Francis Dutil, Chris Pal, and Aaron C. Courville.  Adversarial generation of natural language.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017_ , pp. 241–251. Association for Computational Linguistics, 2017.  URL https://arxiv.org/abs/1705.10929.
  * Tyen et al. (2023) Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune.  Llms cannot find reasoning errors, but can correct them!  _CoRR_ , 2023.  URL https://arxiv.org/abs/2310.01798.
  * Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati.  Can large language models really improve by self-critiquing their own plans?  _CoRR_ , 2023.  URL https://arxiv.org/abs/2310.08118.
  * Wang et al. (2022a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022a.  URL https://arxiv.org/abs/2203.11171.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ Tasks.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2022b.  URL https://arxiv.org/abs/2204.07705.
  * Wang et al. (2023a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language model with self generated instructions.  2023a.  URL https://arxiv.org/abs/2212.10560.
  * Wang et al. (2023b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-Instruct: Aligning Language Model with Self Generated Instructions.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2023b.  URL https://arxiv.org/abs/2212.10560.
  * Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi.  Generating sequences by learning to self-correct.  In _International Conference on Learning Representations (ICLR)_ , 2023.  URL https://arxiv.org/abs/2211.00053.
  * West et al. (2023) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi.  The generative AI paradox: "what it can create, it may not understand".  _CoRR_ , 2023.  URL https://arxiv.org/abs/2311.00059.
  * Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  In _AAAI_ , pp. 2852–2858. AAAI Press, 2017.  URL https://arxiv.org/abs/1609.05473.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _CoRR_ , 2024.  URL https://arxiv.org/abs/2401.10020.
  * Zheng et al. (2023a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023a.  URL https://arxiv.org/abs/2306.05685.
  * Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _CoRR_ , 2023b.  URL https://arxiv.org/abs/2306.05685.

##  Appendix A Prompts for TriviaQA and MT-Bench

In this section, we provide the original and CoT prompts for GSM8k (Figure 4),
TriviaQA (Figure 5), TruthfulQA (Figure 6) and MT-Bench (Figure 7).

Figure 4: Original prompt for discrimination phase of GSM8K. Since our default
evaluation already contains rationales for answer selection, GSM8K doesn’t
have discrimination prompt with Chain-of-Thought Figure 5: Original and Chain-
of-Thought prompt for discrimination phase of TriviaQA. In this example, both
answer 3 and answer 4 are correct Figure 6: Original prompt for discrimination
phase of TruthfulQA. In this example, answer 4 is correct Figure 7: Original
and Chain-of-Thought prompt for discrimination phase of MT-Bench. In this
example, answer 1 is the correct answer

##  Appendix B Extra analysis with Self-Refine

In this section, we provide more explanations on the issues we discovered for
tasks tested in Madaan et al. (2023). We also provide the percentage of times
the model used for generation prefers previous generations over self-refined
subsequent generations in Table 5. To better explain our point, one example
from the task of acronym generation is presented in Figure 8, and another
example from the task of constraint generation is provided in Figure 9.

Task | Issues | Detailed Explanation | Disc Pref
---|---|---|---
Sentiment Reversal | Lack of Reasoning | The refinement simply makes the | 58.7%
sentiment more and more positive
Dialogue Response | Reward Inconsistency | The reward assigned by LLM | 52.4%
Generation | doesn’t increase monotonically
Code Readability | Lack of Reasoning | The refinement only makes variable | 53.3%
Improvement | names longer and more descriptive
Acronym Generation | Reward Inconsistency | The reward assigned by the model | 46.5%
doesn’t increase monotonically
Constrained | Lack of Reasoning | The refinement simply extends | 54.7%
Generation | previous generation
Table 5:  Explanation for some of the issues on tasks that Madaan et al.
(2023) tested and the percentage of times the model prefer self-refined
subsequent generations than previous generations. GSM8K isn’t included here
because it didn’t get much improvement through self-refine. Code optimization
isn’t included either due to the complexity of running experiments.

⬇

Round 0:

0 INIT> Atmospheric Aerosols and Optics Data Library

0 GEN> AAODL TITLE> Atmospheric Aerosols and Optics Data Library

0 SCORES> * Ease of pronunciation: AAODL is pronounced as "ay-odd-ul." It is
somewhat easy to pronounce. 3/5

* Ease of spelling: AAODL is moderately easy to spell. 3/5

* Relation to title: AAODL stands for "Atmospheric Aerosols and Optics Data Library," which is directly related to the title. 5/5

* Positive connotation: AAODL has a neutral connotation. 3/5

* Well-known: AAODL is not a well-known acronym. 1/5

* Total score: 15/25

Round 1:

1 GEN> AEROPTIC TITLE> Atmospheric Aerosols and Optics Data Library

1 SCORES> * Ease of pronunciation: AEROPTIC is pronounced "air-op-tik". This
is somewhat challenging to pronounce. 2/5

* Ease of spelling: AEROPTIC is moderately difficult to spell. 3/5

* Relation to title: AEROPTIC is directly related to the title, combining "Aerosols" and "Optics". 5/5

* Positive connotation: AEROPTIC has a neutral connotation. 3/5

* Well-known: AEROPTIC is not a well-known acronym. 1/5

* Total score: 14/25

Round 2:

2 GEN> AEROLIB TITLE> Atmospheric Aerosols and Optics Data Library

2 SCORES> * Ease of pronunciation: AEROLIB is pronounced "air-oh-lib". This is
an easy acronym to pronounce. 4/5

* Ease of spelling: AEROLIB is fairly easy to spell. 4/5

* Relation to title: AEROLIB is directly related to the title as it combines "Aerosols" and "Library". 5/5

* Positive connotation: AEROLIB has a neutral connotation as it conveys a sense of data repository. 3/5

* Well-known: AEROLIB is not a well-known acronym. 1/5

* Total score: 17/25

Round 3:

3 GEN> AEROPTDATA TITLE> Atmospheric Aerosols and Optics Data Library

3 SCORES> ...

Total score: 14/25

Round 4:

4 GEN> AEROPTLIB TITLE> Atmospheric Aerosols and Optics Data Library

4 SCORES> ...

* Total score: 12/25

Figure 8: Example self-refine generation for acronym generation. The feedback
score for each iterative self-refinement is not consistently increasing.

⬇

Round 0:

Generation: Walking through the club, the uniformed motorcycle owner performed
a quick tricep kiss on the axis of the dance floor, then stopped to listen to
a report about selling hamburgers on the lawn to a group of wine enthusiasts.
Feedback: Concepts Missing from the Sentence: fruit, place, mat, world, area,
mother, pass, club, patient, owner, floor, cross, bull, friend

Round 1:

Generation: Walking through the club, the uniformed motorcycle owner performed
a quick tricep kiss on the dance floor’s central axis, then paused to listen
to a report about selling hamburgers on the lawn to a group of wine
enthusiasts, while a friend handed them a basket of fresh fruits from a nearby
market.

Feedback: Concepts Missing from the Sentence: fruit, place, mat, walk, world,
area, mother, pass, club, patient, owner, cross, bull.

Round 2:

Generation: Walking through the club, the uniformed motorcycle owner performed
a quick tricep kiss on the dance floor’s central axis, then paused to listen
to a report about selling hamburgers on the lawn to a group of wine
enthusiasts, while a friend handed them a basket of fresh fruits from a nearby
market. As they walked through the area, they noticed a mother passing by with
her child and a patient waiting for their turn at the club entrance.

Feedback: Concepts missing from the sentence are: fruit, place, mat, world,
area, mother, pass, club, patient, owner, axis, tricep, kiss, report,
hamburger, wine, cross, sell, lawn, friend

Figure 9: Example self-refine generation for constraint generation. Self-
refine with LLMs on constraint generation often results in progressively
longer sentences (in red) that build on the previous one.

##  Appendix C Predicting exact answers instead of answer options

In this section, we explore whether having discriminators provide exact
(verbatim) answers improves performance compared to selecting among predefined
answer choices (1/2/3/4). For this experiment, the discrimination prompt and
in-context-learning examples are changed accordingly (the discrimination
prompt has changed from "end your generation with 1, 2, 3, or 4" to "end your
generation with one of the generated answers").

This analysis focuses on instances where models exhibit a high percentage of
invalid responses. According to the results presented in Table 6, shifting to
exact answer generation does not significantly reduce the rate of invalid
responses. In fact, in several cases, it appears to increase the percentage of
invalid answers.

|  |  | Original | Verbatim
---|---|---|---|---
Task | Model | $\textbf{S}_{\textbf{gen}}$ | $\textbf{S}_{\textbf{disc}}$ | Invalid% | $\textbf{S}_{\textbf{disc}}$ | Invalid%
GSM8K | LLaMA-2 7B | 9.2 | 8.6 | 14.6 | 8.9 | 21.5
GSM8K | LLaMA-2 13B Chat | 28.3 | 22.8 | 13.1 | 24.8 | 14.2
GSM8K | LLaMA-2 70B | 43.0 | 46.2 | 13.3 | 42.5 | 15.3
TriviaQA | LLaMA-2 7B | 37.1 | 20.2 | 25.8 | 29.3 | 28.0
Table 6: Comparing discrimination performance and invalid% of answer option
generation. Switching to exact answer generation doesn’t reduce the percentage
of invalid answers

##  Appendix D Ablation of prompts used for discrimination

Given the sensitivity of model responses to prompt variations (i.e.,
modifications in wording can impact outcomes), we implemented several prompts
to examine if the observed Self-[In]Correct persists or if it’s merely a
byproduct of specific prompt constructions.

This evaluation was specifically carried out on the GSM8K dataset using the
LLaMA-2 13B model. This model is selected because its DG-Diff on GSM8K is a
big negative number. According to the findings presented in Table 7,
alterations in prompt wording do not significantly affect performance.

Model | $\textbf{S}_{\textbf{gen}}$ | Prompts | $\textbf{S}_{\textbf{disc}}$
---|---|---|---
LLaMA-2 13B Chat | 28.3 | “Choose the final answer from those” | 22.8
“Examine the answer choices carefully and | 23.3
identify the most valid option”
“Critically evaluate each of the four provided | 21.9
answer options, and select the one that stands out
as the most plausible” |
Table 7: Prompt variations do not have a significant effect on DG-Diff