# Automatically Correcting Large Language Models:
Surveying the landscape of diverse self-correction strategies

Liangming Pan, Michael Saxon, Wenda Xu,
Deepak Nathani, Xinyi Wang, William Yang Wang
University of California, Santa Barbara
{liangmingpan, saxon, wendaxu, dnathani<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) have demonstrated remarkable performance across a
wide array of NLP tasks. However, their efficacy is undermined by undesired
and inconsistent behaviors, including hallucination, unfaithful reasoning, and
toxic content. A promising approach to rectify these flaws is self-correction,
where the LLM itself is prompted or guided to fix problems in its own output.
Techniques leveraging automated feedback—either produced by the LLM itself or
some external system—are of particular interest as they are a promising way to
make LLM-based solutions more practical and deployable with minimal human
feedback. This paper presents a comprehensive review of this emerging class of
techniques. We analyze and taxonomize a wide array of recent work utilizing
these strategies, including training-time, generation-time, and post-hoc
correction. We also summarize the major applications of this strategy and
conclude by discussing future directions and challenges.

##  1 Introduction

Recent years have seen striking empirical successes of large language models
(LLMs), as they consistently obtain impressive results across a diverse range
of NLP benchmarks Guo et al. (2023); Suzgun et al. (2023); Qin et al. (2023),
while also showcasing surprising abilities of language understanding Wei et
al. (2022a); Begus et al. (2023), generation Pu and Demberg (2023); Lin and
Chen (2023); Lyu et al. (2023a), and reasoning Wei et al. (2022b); Kojima et
al. (2022); Dasgupta et al. (2022). However, these models are not without
their flaws. LLMs are observed to intermittently display undesired and
inconsistent behaviors such as producing seemingly convincing but inaccurate
“hallucinations” Lin et al. (2022); Zhang et al. (2023c); Min et al. (2023),
conducting unfaithful reasoning Golovneva et al. (2023); Lyu et al. (2023b);
Wu et al. (2023b), generating inappropriate or harmful content Gehman et al.
(2020); Levy et al. (2021, 2022); Shaikh et al. (2023), and failing to
trustfully follow rules and constraints Zhuo et al. (2023); Wang et al.
(2023a). Such flawed behaviors hamper the trust in LLMs and pose hurdles to
their real-world applications OpenAI (2023).

A prevailing strategy to rectify these undesired behaviors of LLMs is learning
from feedback, mirroring a typical human learning strategy where individuals
actively refine their behaviors through a cycle of trial, error, and
correction. Humans, when making mistakes, often gather feedback either from
others or through self-reflection. Such feedback offers valuable insights into
missteps and proposes potential avenues for improvement. With feedback, humans
can adapt and modify their behavior accordingly, learning to correct their
mistakes over time. Inspired by this natural learning mechanism, extensive
research Huang et al. (2022); Madaan et al. (2023); Gero et al. (2023); Jiang
et al. (2023) has been undertaken to improve LLMs through the paradigm of
learning from feedback.

One popular line of research involves the use of human feedback to evaluate
and refine models, as encapsulated in the survey by Fernandes et al. (2023).
These methods typically involve direct optimization of LLMs against human
feedback on their outputs Kreutzer et al. (2018); Glaese et al. (2022); Ouyang
et al. (2022); Scheurer et al. (2023), wherein human evaluations of output
quality serve as a reward signal for improving the model performance. However,
this approach has two primary drawbacks: it can be costly due to the manual
labor involved, and it lacks real-time capabilities as humans cannot provide
instant feedback.

Figure 1: A conceptual framework for self-correcting LLMs with automated
feedback. We identify three parties involved in the prototypical correction
pipeline that are analogous to a patient, doctor, and treatment in medicine,
respectively: a Language Model produces initial output, a Critic Model
analyzes the output and provides feedback, and a Refine Model provides
treatment to either the output or the language model. We taxonomize existing
works using this conceptualization along five key aspects: the problem to be
corrected, the source and format of the feedback, and the strategy and
learning method of the refine model.

To minimize the need for human intervention, another strategy is self-
correcting LLMs with automated feedback, where the model (iteratively) learns
from automatically generated feedback signals to understand the consequences
of its actions and adapts its behaviors. The source of automated feedback can
be multifaceted, spanning from the LLM itself acting as the feedback model
Madaan et al. (2023); Schick et al. (2023), a separately trained feedback
model Yang et al. (2022b); Paul et al. (2023), readily available external
tools Gou et al. (2023); Chen et al. (2023d), to external knowledge sources
such as Wikipedia or the internet Yu et al. (2023); Li et al. (2023b).
Different strategies have been proposed to correct LLM with automated
feedback, including self-training Huang et al. (2022); Bai et al. (2022b),
generate-then-rank He et al. (2023); Weng et al. (2023), feedback-guided
decoding Yang et al. (2022a); Xie et al. (2023), iterative post-hoc revision
Zhang et al. (2023a); Jiang et al. (2023), etc. Recently, the incorporation of
such strategies has demonstrated their effectiveness across a myriad of tasks,
from question answering Peng et al. (2023) and reasoning Pan et al. (2023) to
code generation Zhang et al. (2023b) and toxicity detection Lu et al. (2022).

In light of these advancements, our paper aims to provide a comprehensive
survey. We start by establishing the concept of self-correcting LLMs with
automated feedback and creating a taxonomy of the different methods (§ 2). We
then discuss the major techniques, categorized as training-time correction (§
3), generation-time correction (§ 4), and post-hoc correction (§ 5). We then
summarize the major application areas of this strategy (§ 6). Finally, we
discuss key future directions (§ 7).

##  2 A Taxonomy for Correcting LLMs with Automated Feedback

For the sake of clean exposition, we first present a conceptual framework
outlining the overall process of correcting LLMs with feedback, thereby
establishing the scope of this survey (§ 2.1). We then proceed to identify
five primary dimensions that serve as classification criteria for existing
works: 1) What gets corrected, 2) What is the source of the feedback, 3) What
is the format of the feedback, 4) When the feedback is used, and 5) How to
correct the model with feedback (§ 2.2–§ 2.6). Finally, we summarize existing
works in § 2.7.

###  2.1 Conceptual Framework

We formulate the general process of correcting LLMs with automated feedback in
Figure 1, using an analogy of medical treatment in our daily life. Three
parties are involved in this process:

$\bullet$ Language Model (Patient). A language model
$\mathcal{M}:\mathcal{X}\rightarrow\mathcal{Y}$ performs a specific task by
mapping an input $x\in\mathcal{X}$ to an output text $\hat{y}\in\mathcal{Y}$.
This formulation encompasses a wide range of NLP tasks, for example, in
summarization, $x$ is a passage, $\hat{y}$ is the generated summary; for
question-answering, $x$ is a question and $\hat{y}$ is the predicted answer.
The initial generation $\hat{y}$ may be imperfect and suffer from various
problems such as hallucination and incorrect reasoning.

$\bullet$ Critic Model (Doctor & Diagnosis). A critic model
$\mathcal{C}:\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{F}$ learns to
generate feedback $x,\hat{y}\rightarrow c$ where $\hat{y}\sim\mathcal{M}(x)$
is the output or partial output of the language model, and $c$ is the feedback
of some format, e.g., scalar value, or natural language. A simple example is
binary feedback of whether the output is good or bad given the input
($\mathcal{C}:\mathcal{X}\times\mathcal{Y}\rightarrow\\{0,1\\}$).

$\bullet$ Refine Model (Treatment). A refine model
$\mathcal{R}:\mathcal{X}\times\mathcal{Y}\times\mathcal{F}\rightarrow\mathcal{Y}$
learns to repair an output $x,\hat{y},c\rightarrow y_{new}$ based on the
feedback $c$, where $y_{new}$ is the revised output. Besides repairing output,
some refine models directly repair the language model $\mathcal{M}$ through
fine-tuning or reinforcement learning.

Based on the above formulation, Figure 1 illustrates the fundamental
interaction among the language model $\mathcal{M}$, the critic model
$\mathcal{C}$, and the refine model $\mathcal{R}$. However, the specific model
design in existing works varies along five crucial axes, which we will
elaborate on in the following sections.

###  2.2 What gets corrected?

LLM-based natural language systems can exhibit a variety of errors. We
summarize the four major types of errors that are targeted for correction in
existing works through automated feedback.

$\bullet$ Hallucination. An open challenge for LLMs is that they often
hallucinate by making up facts or citing sources that do not exist Li et al.
(2023a); Zhang et al. (2023c). These hallucinated contents are often quite
plausible-sounding, making it difficult even for humans to detect Clark et al.
(2021). To address this, several studies have proposed the collection of
automated feedback on potential factual inaccuracies by cross-referencing the
output generated by the model with credible knowledge sources. The gathered
feedback can then be utilized by a subsequent refinement model to correct
hallucinations Gao et al. (2023b); Peng et al. (2023).

$\bullet$ Unfaithful Reasoning. LLMs have exhibited a strong ability in
solving complex reasoning tasks with improved reasoning strategies, such as
Chain-of-Thought prompting Wei et al. (2022b). However, recent studies
Golovneva et al. (2023); Ribeiro et al. (2023); Lyu et al. (2023b) found that
LLMs occasionally make unfaithful reasoning, i.e., the derived conclusion does
not follow the previously generated reasoning chain. To address this, existing
works have proposed the use of automated feedback from external tools or
models for guiding the reasoning process Xie et al. (2023); Yao et al.
(2023a), verifying the reasoning process and rectifying errors He et al.
(2023); Pan et al. (2023), or fine-tuning LLMs with process-based feedback
Huang et al. (2022); Lightman et al. (2023).

$\bullet$ Toxic, Biased, and Harmful Contents. LLMs have been observed to
occasionally generate content that is toxic, biased, or harmful due to biases
present in the training data Shaikh et al. (2023). To rectify this,
reinforcement learning from human feedback (RLHF) Ouyang et al. (2022); Bai et
al. (2022a) has been extensively employed to train LLMs to align more closely
with human values, such as being helpful, honest, and harmless. However, RLHF
is heavily dependent on high-quality human feedback, the collection of which
can be resource-intensive. To alleviate this, recent works Lu et al. (2022);
Gou et al. (2023) have also explored collecting automated feedback to identify
and correct potentially harmful outputs.

$\bullet$ Flawed Code. Besides generating natural language text, LLMs also
show strong abilities to generate computer programs (i.e., code) Chen et al.
(2023b). However, the generated code can sometimes be flawed or incorrect. To
fix this, the approach of learning from automated feedback has been
extensively applied in code generation Chen et al. (2023d); Olausson et al.
(2023), largely facilitated by the ease of obtaining such feedback through the
execution of generated code with the corresponding compilers or interpreters.

###  2.3 What is the source of the feedback?

Feedback can be broadly divided into two categories: human feedback and
automated feedback. Fernandes et al. (2023) provided a survey on integrating
human feedback for language generation. In our survey, we focus on the
emerging research area of automated feedback, which explores the possibility
of LLMs to self-correct without constant human intervention. Automated
feedback typically originates from two sources, distinguished by their
relationship with the LLM: self-feedback (i.e., the feedback originates from
the LLM itself) and external feedback (i.e., the feedback is derived from
external models, tools, or knowledge sources).

$\bullet$ Self-Feedback. The LLM itself can be utilized as a feedback
provider. One straightforward way is to directly evaluate the quality of the
generated outputs through prompting and subsequently use this feedback to
refine the results Madaan et al. (2023); Shinn et al. (2023). This process can
be iterative, with the model continually refining its output until it meets a
certain standard. This continuous self-improvement strategy has been found
particularly useful by numerous studies Ye et al. (2023); Yan et al. (2023a),
especially in scenarios where external feedback is unavailable or limited.

$\bullet$ External Feedback. Feedback can originate from sources external to
the LLM, typically including 1) other trained models Yang et al. (2022b);
Lightman et al. (2023), 2) external tools Gou et al. (2023); Charalambous et
al. (2023), 3) external knowledge sources Gao et al. (2023b); Yu et al.
(2023), and 4) external evaluation metrics Jung et al. (2022); Welleck et al.
(2023). External feedback provides a valuable outside perspective which is
particularly useful in identifying errors that the LLM might not recognize on
its own. For example, code interpreters are widely used in programming tasks
to provide real-time error messages; while external knowledge sources can be
utilized to verify the factual accuracy of the LLM’s output.

###  2.4 What is the format of the feedback?

The selection of feedback format requires the consideration of its
expressivity, the ease of its collection, and its potential to improve systems
Fernandes et al. (2023). In existing works, automated feedback is typically in
the form of a scalar value signal or in natural language.

$\bullet$ Scalar Value Feedback. In this scenario, the critic model maps the
input and output to a single score
($\mathcal{C}:\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{N}\subseteq\mathbb{R}$).
Scalar value feedback can be easily integrated into the training/decoding
process of LLMs. For example, Self-Verification Weng et al. (2023) ranks
candidate outputs to find the optimal one based on the real-value feedback
score assigned by the critic model to each candidate. Similarly, Xie et al.
(2023) use real-value feedback for each intermediate reasoning step to guide
the model in performing a stochastic beam search for the optimal solution.
However, despite its flexibility, scalar value feedback is often less
informative to capture detailed information necessary for model correction.

$\bullet$ Natural Language Feedback. Natural language feedback offers greater
expressivity than scalar value feedback, providing richer information that can
highlight the shortcomings of the current output or suggest specific
improvements. This form of feedback is particularly crucial for certain
applications, such as text editing and code generation. For text editing, PEER
Schick et al. (2023) trains an LLM to generate detailed suggestions for edits
to the initial generated text, such as “remove unsourced claim” or “rewrote
the guacamole question for clarity”. For code generation, Self-Debug Chen et
al. (2023d) uses LLMs to generate explanations for the produced code and
utilize both the explanation and the execution results as feedback to enhance
coding solutions.

Method | Feedback | Model Refinement | Application
---|---|---|---
Source | Format | Strategy | Learning
Training-Time Correction
RLHF Ouyang et al. (2022) | Reward Model | Scalar | RLHF | RL | Multiple Tasks
Fine-Grained RLHF Wu et al. (2023a) | Reward Model | Scalar | RLHF | RL | Detoxification, Long-form QA
HH-RLHF Bai et al. (2022a) | Reward Model | Scalar | RLHF | SL & RL | Helpfulness, Harmlessness
Moral RLHF Ganguli et al. (2023) | Reward Model | Scalar | RLHF | RL | Moral Correction
Sparrow Glaese et al. (2022) | Reward Model | NL | RLHF | SL & RL | Dialogue
ILF Scheurer et al. (2023) | Human Feedback | NL | Fine-tuning | SL | Summarization
ILF-Code Chen et al. (2023a) | Human Feedback | NL | Fine-tuning | SL | Code Generation
SLT Yuan et al. (2023) | Human Feedback | NL | Fine-tuning | SL | Response Generation
Gao et al. (2023a) | Human Feedback | NL | Fine-tuning | SL & RL | Extractive QA
Chain-of-Hindsight Liu et al. (2023) | Human Feedback | NL | Fine-tuning | SL | Multiple Tasks
Quark Lu et al. (2022) | External Metrics | Scalar | Fine-tuning | RL | Toxicity, Repetition, Sentiment
SimCLS Liu and Liu (2021) | External Metrics | Scalar | Fine-tuning | SL | Summarization
BERTTune Unanue et al. (2021) | External Metrics | Scalar | Fine-tuning | RL | Machine Translation
STaR Zelikman et al. (2022) | Language Model | NL | Self-Training | SL | QA, Reasoning
Self-Instruct Wang et al. (2023d) | Language Model | NL | Self-Training | SL | Multiple Tasks
RLAIF Bai et al. (2022b) | Language Model | NL | Self-Training | SL & RL | Dialogue
SIRLC Pang et al. (2023) | Language Model | NL | Self-Training | RL | Reasoning, Translation, Summary
Self-Improve Huang et al. (2022) | Language Model | NL | Self-Training | SL | QA, Reasoning, NLI
AlpacaFarm Dubois et al. (2023) | Language Model | NL | Self-Training | SL & RL | None (Intrinsic Evaluation)
ReST Gulcehre et al. (2023) | Language Model | NL | Self-Training | RL | Machine Translation
Generation-Time Correction
Self-Verification Weng et al. (2023) | Language Model | Scalar | Re-Ranking | ICL | Arithmetic Reasoning
CodeT Chen et al. (2023b) | Program Executor | Scalar | Re-Ranking | ICL | Code Generation
LEVER Ni et al. (2023) | Program Executor | Scalar | Re-Ranking | SL | Table QA, Math QA, Program
RR He et al. (2023) | External Knowledge | Scalar | Re-Ranking | — | Reasoning
InstructScore Xu et al. (2023b) | Language Model | NL | Re-Ranking | SL | Generation Evaluation
MBR Decoding Freitag et al. (2022) | External Metrics | Scalar | Re-Ranking | SL | Machine Translation
DIVERSE Li et al. (2023d) | Trained Model | Scalar | Re-Ranking | SL | Arithmetic Reasoning
TeachMe Mishra et al. (2022) | Human Feedback | NL | Feedback-guided | SL | Proof Generation
PRM Lightman et al. (2023) | Reward Model | Scalar | Feedback-guided | SL | Arithmetic Reasoning
DiffusionLM Li et al. (2022) | Trained Model | Scalar | Feedback-guided | SL | Controlled Text Generation
PPLM Dathathri et al. (2020) | Trained Model | Scalar | Feedback-guided | SL | Controlled Text Generation
Fudge Yang and Klein (2021) | Trained Model | Scalar | Feedback-guided | SL | Controlled Text Generation
Entailer Tafjord et al. (2022) | Trained Model | Scalar | Feedback-guided | SL | Proof Generation
NLProofS Yang et al. (2022a) | Trained Model | Scalar | Feedback-guided | SL | Proof Generation
GRACE Khalifa et al. (2023) | Trained Model | Scalar | Feedback-guided | SL | Arithmetic Reasoning
CoRe Zhu et al. (2023) | Trained Model | Scalar | Feedback-guided | SL | Arithmetic Reasoning
Varshney et al. (2023) | External Knowledge | NL | Feedback-guided | ICL | Hallucination Detection
MemPrompt Madaan et al. (2022) | External Knowledge | NL | Feedback-guided | ICL | Lexical and Ethical Reasoning
Maieutic Prompting Jung et al. (2022) | External Metrics | Scalar | Feedback-guided | ICL | Commonsense Reasoning
SI Creswell and Shanahan (2022) | Language Model | Scalar | Feedback-guided | ICL | Proof Generation
RAP Hao et al. (2023) | Language Model | Scalar | Feedback-guided | ICL | Planning, Reasoning
SelfEval-Decoding Xie et al. (2023) | Language Model | Scalar | Feedback-guided | ICL | Arithmetic / Symbolic Reasoning
SelfCheck Miao et al. (2023) | Language Model | NL | Feedback-guided | ICL | Arithmetic Reasoning
Tree of Thoughts Yao et al. (2023a) | Language Model | NL / Scalar | Feedback-guided | ICL | Games, Writing

Table 1: Representative works on Training-time Correction and Generation-Time
Correction. We summarize the key features for each work: the source and format
of the feedback, the strategy and learning method of the refinement process,
and the application of the method.

Method | Feedback | Model Refinement | Application
---|---|---|---
Source | Format | Strategy | Learning | Iter.
Post-hoc Correction
Self-Refine Madaan et al. (2023) | Language Model | NL | Self-Refine | ICL |  2713 | Multiple Tasks
Clinical SV Gero et al. (2023) | Language Model | NL | Self-Refine | ICL |  2717 | Information Extraction
Reflexion Shinn et al. (2023) | Language Model | NL | Self-Refine | RL |  2713 | QA, Code Generation
IterRefinement Chen et al. (2023c) | Language Model | NL | Self-Refine | ICL |  2713 | Machine Translation
Auto-Post-Editing Raunak et al. (2023) | Language Model | NL | Self-Refine | ICL |  2717 | Machine Translation
RCI Kim et al. (2023) | Language Model | NL | Self-Refine | ICL |  2713 | Computer Tasks
SelFee Ye et al. (2023) | Language Model | NL | Self-Refine | SL |  2713 | Dialogue
SelfCheckGPT Manakul et al. (2023) | Language Model | NL | Self-Refine | ICL |  2717 | Hallucination Detection
LLM Self Defense Helbling et al. (2023) | Language Model | NL | Self-Refine | ICL |  2717 | Harmful Text Correction
Re3 Yang et al. (2022b) | Trained Model | Scalar | External Feedback | SL & ICL |  2713 | Story Generation
CodeRL Le et al. (2022) | Trained Model | Scalar | External Feedback | RL |  2717 | Code Generation
FLIRT Mehrabi et al. (2023) | Trained Model | Scalar | External Feedback | ICL |  2713 | Adversarial Prompt Generation
REFINER Paul et al. (2023) | Trained Model | NL | External Feedback | SL & ICL |  2713 | Reasoning, Moral Story
RL4F Akyürek et al. (2023) | Trained Model | NL | External Feedback | SL & RL |  2713 | Planning, Summarization
Yan et al. (2023a) | Trained Model | NL | External Feedback | SL |  2713 | Semantic Parsing
Baldur First et al. (2023) | Trained Model | NL | External Feedback | ICL |  2713 | Proof Generation
CRITIC Gou et al. (2023) | External Tools | NL | External Feedback | ICL |  2713 | QA, Program, Toxicity
FacTool Chern et al. (2023) | External Tools | NL | External Feedback | ICL |  2713 | QA, Reasoning, Generation
RARR Gao et al. (2023b) | External Knowledge | NL | External Feedback | ICL |  2717 | Open-Domain QA
LLM-Augmenter Peng et al. (2023) | External Knowledge | NL | External Feedback | RL |  2713 | Open-Domain QA
Self-Checker Li et al. (2023b) | External Knowledge | NL | External Feedback | ICL |  2717 | Fact-Checking
REFEED Yu et al. (2023) | External Knowledge | NL | External Feedback | ICL |  2717 | QA, Dialogue
Olausson et al. (2023) | Program Executor | NL | External Feedback | ICL |  2713 | Code Generation
Self-Edit Zhang et al. (2023a) | Program Executor | NL | External Feedback | ICL |  2713 | Code Generation
Self-Debug Chen et al. (2023d) | Program Executor | NL | External Feedback | ICL |  2713 | Code Generation
Self-Evolve Jiang et al. (2023) | Program Executor | NL | External Feedback | ICL |  2713 | Code Generation
Logic-LM Pan et al. (2023) | Symbolic Solver | NL | External Feedback | ICL |  2713 | Logical Reasoning
Self-Critique Saunders et al. (2022) | LLMs + Human | NL | External Feedback | SL |  2717 | Summarization
ALGO Zhang et al. (2023b) | Oracle Verifier | Scalar | External Feedback | ICL |  2713 | Code Generation
Charalambous et al. (2023) | BMC Tool | NL | External Feedback | ICL |  2717 | Software Verification
Self-Correction Welleck et al. (2023) | External Metrics | NL / Scalar | External Feedback | SL |  2713 | Reasoning, Generation, Toxicity
Multiagent Debate Du et al. (2023) | Language Model | NL | Model Debate | ICL |  2713 | Reasoning, Factuality
LM vs LM Cohen et al. (2023) | Language Model | NL | Model Debate | ICL |  2713 | Factual Error Detection
ICL-AIF Fu et al. (2023) | Language Model | NL | Model Debate | ICL |  2713 | Bargaining Game
PRD Li et al. (2023c) | Language Model | NL | Model Debate | ICL |  2713 | Open-ended QA

Table 2: Representative works on Post-hoc Correction. We summarize the key
features for each work: the source and format of the feedback, the strategy
and learning method of the refinement process, whether the refinement is
iterative (Iter.), and the application of the method.

###  2.5 When to correct the model with feedback?

Depending on the timing of using automated feedback to correct the model,
existing works can be divided into three major categories.

$\bullet$ Training-time Correction. The ideal scenario is to rectify a flawed
model during training, prior to its deployment for use. Once feedback has been
collected, it is directly used to optimize the model parameters. Human
feedback is typically used for training-time correction, as exemplified by the
widely adopted RLHF approach Ouyang et al. (2022). For leveraging automated
feedback, a common strategy is self-training Huang et al. (2022), where the
model is trained with its own generated high-quality output filtered out by
the critic model. While training-time correction is a pre-hoc strategy that
addresses problems during training, its practical application may be hindered
by three issues: 1) the infeasibility of fine-tuning gaint closed-source LLMs,
such as GPT-4 OpenAI (2023), 2) the potential unavailability of feedback
during model training, and 3) the requirement for the feedback to be
“optimizable”, e.g., a numerical score severing as the basis for model
optimization.

$\bullet$ Generation-time Correction. This strategy utilizes automated
feedback to guide the language model during generation, which allows the model
to correct errors in its outputs as it is being generated. For example, for
proof generation, several works utilize the automated feedback of the
intermediate reasoning steps to guide the model to recover from incorrect
generation and search for the optimal solution in a more efficient way Yang et
al. (2022a); Lightman et al. (2023).

$\bullet$ Post-hoc Correction. Finally, post-hoc correction involves refining
the model output after it has been generated, without updating the model
parameters. This typically involves an iterative process of generating output,
receiving feedback, and refining output. Post-hoc correction provides more
flexibility than the previous two strategies as it does not require training
the LLM or accessing its parameters. Furthermore, post-hoc correction enhances
explainability as it facilitates the incorporation of more informative natural
language feedback. This allows for a more transparent visualization and
interpretation of the self-correction process.

###  2.6 How to correct the model with feedback?

Various concrete strategies have been proposed to correct LLMs with automated
feedback, which are tailored to the different dimensions we mentioned in
previous sections. For example, self-training is often used for training-time
correction. Generate-then-rank often comes with scalar value feedback. Self-
refine is the strategy that uses the same LLM as both the critic model and the
refine model. We will cover the landscape of self-correction strategies
through Section 3 to Section 5.

###  2.7 Summary of existing works

Building upon the taxonomy established in the preceding sections, we collate
existing works on correcting LLMs with (automated) feedback in Table 1 and
Table 2. We have two major selection criteria for a work to be included in
this survey:

1\. Automated Feedback: Explicit feedback is involved to assess the quality of
the model output. We focus on automated feedback that originates from external
models, metrics, knowledge, etc. However, we will cover some representative
works of human feedback for completeness.

2\. Model Refinement: The feedback should act as a directive to enhance the
LLM, either by: 1) updating model parameters, or 2) altering the model’s
output during or post the generation process.

These works are categorized based on the three strategies introduced in
Section 2.5. We also summarize key features of each work, including: 1) the
source of feedback, 2) the format of feedback, 3) the strategy and learning
method employed for the refinement, 4) whether the refinement process is
iterative, and 5) the application of the method. Subsequently, we will delve
into a detailed review of each method type, encompassing Training-Time
Correction (§ 3), Generation-Time Correction (§ 4), and Post-hoc Correction (§
5).

Figure 2: Three typical strategies of training-time correction: directly
optimization with human feedback (a), training a reward model that
approximates human feedback (b), and self-training with automated feedback
(c).

##  3 Training-Time Correction

In this section, we delve into methodologies that rectify model behavior
during the training phase. As depicted in Figure 2, we identify three typical
strategies for training-time correction. Each strategy utilizes different
forms of feedback to modify the model parameters during training: human
feedback (a), a reward model that approximates human feedback (b), and
automated feedback (c).

###  3.1 Learning from Human Feedback

The next-word prediction objective of LLM pre-training is not inherently
designed to encapsulate human values or preferences. This misalignment can
lead to unintended consequences, such as the generation of harmful,
misleading, or biased content Kenton et al. (2021). To mitigate these issues,
many research efforts have explored the integration of human feedback to
better align LLMs with human values and expectations. Wang et al. (2023e) and
Fernandes et al. (2023) extensively reviewed this research area. Our survey,
however, focuses on automated feedback, thus we will only touch upon
representative works in this direction.

#### Direct Optimization with Human Feedback.

In an ideal scenario, we would directly leverage human feedback to optimize
the model parameters. Typically, this approach follows the framework depicted
in Figure 2(a): 1) Candidate outputs are generated by LLMs, 2) Humans provide
feedback or refinements on these outputs, and 3) LLMs are then directly
optimized on the collected (outputs, feedback) to better align with human
preferences. A simple strategy is to fine-tune the model on the outputs with
positively-labeled feedback. For example, Sparrow Glaese et al. (2022) fine-
tunes LLMs on the collected dialogues rated as preferred and rule compliant
(concerning correctness, harmfulness, and helpfulness), according to humans.
Similarly, Scheurer et al. (2023) utilizes an LLM to generate multiple
refinements of the original output based on human feedback, and then the best
refinement is picked up to finetune the original LLM. A similar idea is
adopted to fine-tune the code generation model Chen et al. (2023a). First,
human annotators write natural language feedback for incorrect codes. A
refinement model then utilizes this feedback to correct the code. Finally, the
refined code is subsequently employed to fine-tune the code-generating LLM.
However, only utilizing positive data (human-refined or positive-rated data)
for fine-tuning may constrain the model’s ability to identify and correct
negative attributes or errors. To address this, Chain-of-Hindsight Liu et al.
(2023) fine-tunes the LLM on model outputs paired with both positive and
negative feedback. Beyond fine-tuning, other optimization methods are explored
as well. For example, Gao et al. (2023a) utilizes human feedback as the reward
signal and optimizes the model with contextual bandit learning.

#### Reward Modeling and RLHF.

Employing human feedback directly to rectify model behavior may not always be
practical. The collection of human feedback can be both labor-intensive and
time-consuming. An efficient alternative is to train a reward model that
emulates human feedback. Once trained, this reward model can provide
consistent, real-time feedback for every model output, thereby circumventing
the need for constant human involvement. A prominent example of this approach
is Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. (2022), as
illustrated in Figure 2(b). It first asks human annotators to label the
preference for different LLM outputs and then train the reward model to
predict the human preference. Afterward, reinforcement learning (RL)
algorithms (e.g., Proximal Policy Optimization (PPO) Schulman et al. (2017))
are employed to optimize the model. RLHF and its variants have proven
effective in correcting LLMs to become more beneficial and less harmful Bai et
al. (2022a), as well as instilling moral correctness Ganguli et al. (2023).

###  3.2 Learning with Automated Feedback

Since collecting human feedback is quite resource-intensive, numerous studies
have explored the use of automated feedback to minimize the demand for human
intervention. To differentiate between human and automated feedback, we define
human feedback as a quality assessment performed by human evaluators on the
outputs generated by the base model. This feedback is then used for either
direct optimization or reward model learning (Section 3.1). On the other hand,
automated feedback is collected in an offline environment, without the need
for human assessment of model outputs. We mainly discuss training time
strategies utilizing two types of automated feedback: extrinsic feedback from
external metrics/models, and intrinsic feedback from the language model
itself.

#### External Metric Guidance.

Feedback provided by external metrics has been frequently used for training-
time correction. Due to the discrete nature of metric signals, most approaches
focus on non-differentiable training techniques. Minimum risk training Shen et
al. (2016) optimizes model parameters with external evaluation metrics Xu et
al. (2022, 2023a), by incorporating metric score with maximum log-likelihood
in the loss function. It can optimize metric scores during training time.
However, it can lead the model to the robustness deficiencies of some metrics
Yan et al. (2023b), such as BLEURT Sellam et al. (2020). Liu and Liu (2021)
leverages a contrastive learning framework to rerank candidates based on
metric scores, which bridges the gap between training and inference
objectives. Li et al. (2019) employs deep RL algorithm and Unanue et al.
(2021) leverage Gumbel softmax Jang et al. (2017) to build distributional
semantic reward from BERTScore Zhang et al. (2020) and mitigate exposure bias.
To stabilize gradients, Wu et al. (2021) utilizes contrastive discriminator
and PPO to imitate human texts. Recently, Chang et al. (2023) propose a more
efficient RL algorithm, RLGF, than PPO Schulman et al. (2017) to finetune LLM
with pre-defined reward. They integrate a reasonable but incomplete guide
policy into a policy gradient framework and learn a near-optimal strategy.
Different from leveraging feedback solely at fine-tuning, Korbak et al. (2023)
employs conditional training Keskar et al. (2019) and an automated classifier
to tag undesirable contents at the pretraining stage.

#### Self-Training.

Instead of leveraging external metrics as feedback, the language model itself
can be used to provide feedback for its own output. This gives rise to the
self-training strategy of self-improving LLM by bootstrapping its original
outputs, as depicted in Figure 2(c). STaR Zelikman et al. (2022) leverages the
idea of CoT by prompting LLM to generate answers with rationales. By selecting
rationales leading to the correct answer to further finetune LLM, the
performance of LLM is improved. This process can be iterated with further
performance gains. Huang et al. (2022) follows this idea by applying self-
consistency Wang et al. (2023c) to majority vote reasoning paths (the paths
that lead to the most voted answers). LLM is finetuned over selected
reasoning-answer data with augmented prompts. This strategy has also been used
to reduce the harmful responses of LLMs. RLAIF Bai et al. (2022b) adopted the
strategy of critique $\rightarrow$ revision $\rightarrow$ supervised learning.
The initial toxic responses are criticized and revised by the LLM itself
following a set of human-defined principles. Afterward, the LLM is fine-tuned
on the revised responses. AlpacaFarm Dubois et al. (2023) further shows that
LLMs can self-improve with RL. It designs LLM prompts to simulate human
feedback in RLHF and shows that the feedback is effective and greatly reduces
the cost. Gulcehre et al. (2023) further improves self-training by proposing
the Reinforced Self-Training (ReST). It iteratively performs the following two
steps to improve the LLM: 1) the Grow step produces a dataset by sampling from
the policy model (i.e., the current LLM), and 2) the Improve step optimizes
the LLM policy using offline RL algorithms.

##  4 Generation-Time Correction

Correcting LLMs at training time appears to be the ideal solution given the
principle of “an ounce of prevention is worth a pound of cure”. However, there
is no guarantee that all undesired behavior can be solved at training time.
Moreover, training-time correction might be excessively resource-intensive or
even impractical for many LLMs, e.g., closed-source LLMs where weights are
inaccessible, and colossal LLMs with billions of parameters. This motivates
the exploration of methods that seek to correct LLMs during the generation
time or after the output is generated. This section focuses on generation-time
correction techniques, in which automated feedback serves as a guiding
mechanism for LLM generation. Such a strategy allows LLMs to rectify errors
during generation without modifying the model parameters. We identify two
primary strategies for generation-time correction: Generate-then-Rank, and
Feedback-Guided Decoding.

Figure 3: The illustrations of the two typical strategies of generation-time
correction: (a) Generate-then-Rank, and (b) Feedback-Guided Decoding.

###  4.1 Generate-then-Rank

The most immediate strategy involves sampling a large number of candidate
generations and subsequently picking up the best generation based on the
feedback provided by the critic model, as illustrated in Figure 3(a). Here,
the critic model $\mathcal{C}$ aims to learn the mapping
$x,\hat{y}_{1},\cdots,\hat{y}_{N}\rightarrow y_{best}$, where $y_{best}$ is
the best output among the $N$ candidate outputs
$\hat{y}_{1},\cdots,\hat{y}_{N}\sim\mathcal{M}(x)$.

This approach is often integrated with the Chain-of-Thought (CoT) prompting
method Wei et al. (2022b) to tackle complex reasoning tasks, such as solving
math word problems as in GSM8K Cobbe et al. (2021). Given an input problem
$x$, the LLM initially generates multiple candidate solutions
${y_{1},\cdots,y_{n}}$. Each solution $y_{i}=[z_{i},a_{i}]$ comprises a
reasoning path (explanation) $z_{i}$ leading to the predicted answer $a_{i}$.
Subsequently, the critic model $\mathcal{C}$ assigns a plausibility score
$s_{i}$ to each candidate reasoning path $z_{i}$. The final selection of the
best solution from the scored set ${(z_{i},a_{i},s_{i})}_{i=1}^{n}$ is
achieved via either ranking or voting.

Different critic models have been proposed in various works. For instance,
DIVERSE Li et al. (2023d) trains a binary verifier based on DeBERTa He et al.
(2021), using reasoning paths that correspond to the correct final answer as
positive examples and the others as negative examples. The best answer is then
determined by a majority vote of positively-verified candidates. Weng et al.
(2023) introduced a training-free critic model based on the idea of self-
verification, in which the plausibility score is calculated by assessing the
consistency between the results of forward reasoning and backward reasoning.
In a different vein, the RR He et al. (2023) presented a critic model to
assess the faithfulness of each reasoning path by retrieving supporting
information from a knowledge base. LEVER Ni et al. (2023) employed this
strategy in language-to-code generation, with each solution $y_{i}$ serving as
a candidate SQL program for the question $x$. A verifier was trained to
predict the likelihood of a program’s correctness based on the program itself
and its execution results. A similar idea is adopted in CodeT Chen et al.
(2023b) where multiple code solutions and the test cases are generated by the
LLM and the best code solution is selected by a dual execution agreement.

###  4.2 Feedback-Guided Decoding

The generate-then-rank method, in which the critic model offers output-level
feedback on the entire reasoning path, has certain limitations: 1) The output-
level feedback is not fine-grained enough to pinpoint the exact error
locations, 2) The extensive length of the output can complicate its quality
assessment, and 3) This method does not facilitate fine-grained control over
the generation process. For example, the LLM cannot correct its errors during
the generation process but must wait until the entire output has been
generated.

To address these issues, several works have adopted the feedback-guided
decoding strategy shown in Figure 3(b), which relies on step-level feedback to
offer fine-grained guidance over the generation process. Here, the generation
of the output $y$ is broken down into multiple reasoning steps (or thoughts),
i.e., $y_{i}=[o_{1},o_{2},\cdots,o_{n}]$. At each individual reasoning step
$t$, the critic model provides feedback $\mathcal{C}(x,o_{1:t-1},o_{t})$ that
indicates the quality of $o_{t}$ as a candidate step. With the ability to
generate and evaluate individual steps, a search algorithm, such as beam
search or depth-first search, can be employed for a systematic exploration of
the output space, which effectively steers the decoding process toward the
generation of an optimal solution. This also allows the LLM to recover from
its early mistakes during generation and helps alleviate the reasoning
inconsistency problem Zelikman et al. (2022); Creswell and Shanahan (2022),
i.e., incorrect reasoning leads to correct final answer.

The feedback-guided decoding strategy has been applied in many recent works,
such as Tree-of-Thought Yao et al. (2023a), GRACE Khalifa et al. (2023), and
RAP Hao et al. (2023). These works mostly differ in how to obtain the critic
model that provides automated step-level feedback, the most challenging but
crucial element of this strategy. We classify their employed methods into four
categories: human feedback, a trained verifier, external metrics, external
knowledge, and self-evaluation.

$\bullet$ Reward Model from Human Feedback. One approach involves training a
step-level reward model by gathering human feedback, much like the methods
discussed in Section 3.1. Uesato et al. (2022) ask human annotators to
evaluate the correctness of each reasoning step for the problems in GSM8K and
subsequently train a binary reward model. Lightman et al. (2023) expand this
approach by annotating a larger dataset consisting of 800K instances of human
step-level feedback. Both studies discovered that step-level feedback assists
in training a more reliable reward model, which enhances the faithfulness of
reasoning.

$\bullet$ Training Verifier with Synthetic Data. Considering the high cost of
collecting human annotations and their limited scalability, some works Yang et
al. (2022a); Tafjord et al. (2022); Li et al. (2023d); Khalifa et al. (2023)
have trained a step-wise verifier using automatically constructed training
data. Positive examples are derived from ground-truth reasoning paths, while
negative examples are synthesized by proposing an alignment algorithm Khalifa
et al. (2023) or by making text perturbations on positive samples Yang et al.
(2022a).

$\bullet$ Feedback from External Metric. Several works also leverage external
metrics to re-rank or guide text generation. Freitag et al. (2022) uses
minimum bayes risk decoding on unbiased samples to optimize neural metrics as
an alternative to beam search. Plug and play Dathathri et al. (2020) combines
a pretrained model with attribute classifiers that guide text generation
without any further training of the model. It leverages the gradient of the
classifier to update LM and increase the likelihood of the desirable
attribution at the text generation of LM. FUDGE Yang and Klein (2021)
reweights the model predictions at each token and estimates the attribution
classification at each partial sequence. Following up to the gradient-based
approach, DiffusionLM Li et al. (2022) obtains a sequence of intermediate
latent variables by denoising a sequence of Gaussian vectors. It performs the
iterative gradient updates over latent representations to satisfy controlled
requirements from an attribute classifier.

$\bullet$ Feedback from External Knowledge. External knowledge sources have
also been used to guide the LLM in generation. Varshney et al. (2023)
retrieves relevant knowledge from Wikipedia as evidence to validate and
correct LLM’s generated sentences at each step. Once a non-factual sentence is
corrected, the revised sentence is added back to the input along with the
prior generations to continue generating the next sentence. In a different
approach, MemPrompt Madaan et al. (2022) leverages prior user feedback as a
knowledge source. It maintains an external pool of user feedback and searches
it for responses that match the intent of the current query. The retrieved
feedback is then concatenated with the input to guide the following
generation.

$\bullet$ Self-Evaluation. Some studies have utilized a more flexible
strategy, employing the LLM itself as the critic model by designing
appropriate prompts. For instance, in Tree-of-Thought Yao et al. (2023a), the
LLM is prompted to assess the value of the current state by producing a scalar
value (e.g., 1-10) or short phrases (e.g., sure/likely/impossible). Xie et al.
(2023) employed a similar approach by prompting the LLM with “Is the above
step of reasoning: (A) Correct (B) Incorrect”. Self-evaluation provides an
efficient evaluation method without requiring task-specific verifier fine-
tuning.

Existing works also adopted different strategies to control the decoding
process with the help of the step-level critic model. Tree-of-Thought employed
breadth-first search and depth-first search, while GRACE Khalifa et al. (2023)
and Xie et al. (2023) adopted the beam search strategy. At each step, the
top-$k$ scoring candidates are selected for subsequent generations. This
process is repeated until the final answer is generated. Instead, CoRe Zhu et
al. (2023) and RAP Hao et al. (2023) adopted the Monte Carlo Tree Search
(MCTS) to strike a proper balance between exploration and exploitation to find
the best reasoning path more efficiently.

Figure 4: Three typical strategies of post-hoc correction: self-correction
(a), post-hoc correction with external feedback (b), and multi-agent debate
(c).

##  5 Post-hoc Correction

The success of generation-time correction heavily depends on the critic
model’s capability in providing accurate quantifiable feedback for
intermediate outputs. However, this might be quite challenging for many NLP
tasks, such as summarization, due to the holistic nature of the evaluation,
i.e., the summary can only be accurately assessed after the entire summary is
generated. This motivates the employment of post-hoc correction methods, where
both the critic and refine models intervene only after the entire output is
produced. Post-hoc correction also provides a more effective interface with
various forms of insightful natural language feedback. This feedback can be as
detailed as a diagnostic report pinpointing exact error locations, or as
general as suggestions for overall writing improvement. As illustrated in
Figure 4, we summarize three primary strategies for post-hoc correction: Self-
Correction, Correction with External Feedback, and Multi-Agent Debate.

###  5.1 Self-Correction

The simplest approach to implement post-hoc correction is the “Self-
Correction” technique, where an LLM is employed to generate feedback and
refine its own output. As depicted in Figure 4(a), an LLM is initially used to
produce a initial output, and subsequently, the same model acts as a critic to
generate feedback and refine this initial output based on the received
feedback. This process is typically iterative and continues until an output of
acceptable quality is obtained or a pre-specified number of iterations are
reached.

Self-Refine (Madaan et al., 2023) proposed a simple-yet-effective self-
correction framework by simply using a single powerful pre-trained LLM to
generate output, provide feedback, and refine the output based on that
feedback. All these steps are conducted using the same LLM, guided by
different prompts. Similarly, Clinical Self-Verification (Gero et al., 2023)
employs the self-correction framework to extract patient data from clinical
notes. They specifically generate feedback to find missing elements in the
initially extracted data and to validate the generated data. The output is
then refined by eliminating unsupported elements. In contrast, Reflexion
(Shinn et al., 2023) highlighted that prior self-correction research focused
on single-turn generation tasks and failed to retain a record of past errors.
To address this, Reflexion proposed to use the same self-correction framework
with an addition of a “long-term memory” capable of storing prior feedback and
outputs, thereby avoiding the repetition of previous mistakes. Also, Reflexion
improves Self-Refine by incorporating scalar-valued feedback and other forms
of feedback.

While self-correction has shown effective for a wide variety of text-
generation tasks, this strategy requires the use of powerful, large-scale LLMs
capable of refining text based on provided feedback. As noted by Madaan et al.
(2023), smaller, open-source models often struggle to refine their output
effectively, even when the correct feedback is provided. A possible solution
involves explicitly training models for this self-correction process. SelFee
(Ye et al., 2023) proposes training a model to emulate the self-correction
process by generating output, feedback, and a refined solution in an auto-
regressive manner. They use more powerful LLMs to provide feedback and
refinement data, with data collection facilitated through ChatGPT.

###  5.2 Models/Tools as Feedback

As self-correction relies on the language model for feedback, the quality of
the feedback is inherently constrained by the inherent limitations of LLMs,
such as the inability to access up-to-date information, take actions, or
perform precise mathematical reasoning. To address this, recent works have
investigated the use of external tools for providing feedback. Illustrated in
Figure 4(b), a broad array of external tools, including trained models, code
interpreters, and search engines, can be integrated to provide specialized
feedback.

#### Code Interpreter.

In code generation, the program executor is frequently used as a source of
feedback for refining the initial code written by the model. For example,
Self-Edit Zhang et al. (2023a) and Self-Evolve execute the initial program on
example test cases and provide the execution results back as feedback.
Afterward, an LLM is prompted to refine the initial code based on the
feedback. Self-Debug Chen et al. (2023d) investigated using program
explanation, unit tests, and program interpreter as feedback types. ALGO Zhang
et al. (2023b) explored a more fine-grained feedback for code generation. For
each problem, it first generates a reference oracle program that solves the
problem with an exhaustive search. The feedback is collected by comparing the
outputs from the LLM-generated program with the oracle outputs for a given set
of test inputs. The self-correction strategy has also been adopted for formal
verification of software. Charalambous et al. (2023) employed Bounded Model
Checking to locate the software vulnerability and then used the LLM for
correction.

#### Logic Reasoner.

Tool-assisted feedback has also been used to enhance the faithfulness of LLMs’
reasoning. For example, Logic-LM Pan et al. (2023) solves a logical reasoning
problem by first translating it into logical form with LLMs and then
performing inference on it with external symbolic solvers. Due to the
complexity of correctly parsing the problem at the first attempt, a self-
refinement module is introduced to modify inaccurate logical forms using the
error messages returned by the symbolic reasoner as feedback. Similarly,
Baldur First et al. (2023) uses existing search-based proof assistants as a
source of feedback to improve language models’ ability to generate theorem
proofs.

#### External Knowledge.

External knowledge is also frequently incorporated as a source of feedback to
detect and revise factual errors in LLM’s output and to support LLM-generated
facts with evidence or citations. For example, RARR Gao et al. (2023b) and
REFEED Yu et al. (2023) directly prompt LLMs to raise questions about
different aspects of the generated output. An external retriever then searches
for evidence to investigate each query. Finally, a refine model is employed to
amend the output based on any detected discrepancies between the output and
the retrieved evidence. LLM-Augmenter Peng et al. (2023) proposes a similar
method but differentiates itself by automatically generating natural language
feedback based on the evidence retrieved. This feedback identifies error
locations and provides revision suggestions. These models are evaluated on
knowledge-intensive QA tasks. To broaden the adaptability, FACTOOL Chern et
al. (2023) extends knowledge-assisted factual error correction to a wider
range of tasks, including code generation, mathematical reasoning, and
scientific literature review.

#### Trained Model.

Existing works also fine-tune specialized models for feedback generation.
These critic models can then be paired with similar or more potent language
models in an iterative-refinement cycle. For example, CodeRL Le et al. (2022)
treats program synthesis as a reinforcement learning task and trains a critic
model whose output optimizes the main model. In contrast, REFINER Paul et al.
(2023) trains a task model to produce an intermediate representation for
problem-solving, while a critique model provides feedback on each intermediate
training step. The critique model can subsequently be employed to generate
feedback for larger task models, such as ChatGPT. Similarly, RL4F (Akyürek et
al., 2023) utilizes reinforcement learning for training a critic while keeping
the downstream task model fixed. The critic model is initially fine-tuned to
produce feedback given an initial output and then further fine-tuned using a
policy optimization method. The reward is defined by evaluating the accuracy
of the refined output or comparing it with the ground truth, assuming that the
downstream model can effectively refine the output if accurate feedback is
provided. In “red-teaming” type applications, including targeting
vulnerabilities in the content filtering capabilities of other generated
systems, feedback from the content filters themselves can be used as a signal
to guide the generation of better adversarial examples. For example, Feedback
Loop In-context Red Teaming (FLIRT) Mehrabi et al. (2023) uses the signal from
an explicit image classifier to guide an LLM to produce adversarial input
prompts for a text-to-image system to generate more unsafe images for auditing
purposes.

#### Integrating Multiple Tools.

Broadening the idea of tool-assisted feedback, CRITIC (Gou et al., 2023)
integrates a variety of tools in a unified framework, including program
interpreters for coding feedback, external knowledge and search engines for
factual information, calculators for verifying mathematical equations, and
LLM-based natural language feedback. Each tool is proficient at providing
feedback for different aspects, contributing to a more comprehensive feedback
system.

###  5.3 Multi-Agent Debate

Besides integrating external tools, recent studies have also explored the
strategy of debating between multiple LLMs, drawing inspiration from
collaborative intelligence, where diverse perspectives often converge to a
more refined solution. This approach aims to improve the output quality by
employing multiple instances of LLMs, each proposing and debating their
individual responses over multiple rounds to arrive at a common final answer.

Du et al. (2023) first applied and evaluated this strategy in arithmetic
reasoning tasks. Each agent (a duplicate of LLM) initially generates its
individual solution along with justifications. The debate phase involves
collating responses from all agents and presenting this as context to each
agent. Based on this context, each agent is then instructed to craft a revised
response. The models are found to converge on a shared solution following
multiple debate iterations. Experiments show that multi-agent debate leads to
improved performance over the self-correction strategy. Furthering this
concept, PRD Li et al. (2023c) proposed the peer rank algorithm for better
obtaining a consensus answer after debating. It considers pairwise preferences
between all possible answer pairs from individual LLMs and uses these
preferences to generate a final ranking of models.

In addition to reasoning tasks, LM vs LM Cohen et al. (2023) further
demonstrated the effectiveness of multi-agent debate for detecting factual
errors. The approach involves a generator LLM creating a claim, while an
examiner LLM probes for factual inaccuracies through a multi-turn
interaction.To broaden this concept, Fu et al. (2023) demonstrated that
interactions between different LLMs could mimic human behavior in real-world
tasks. The study showcased this through a bargaining scenario where different
LLM agents assumed the roles of buyer and seller. This further highlights the
versatile applications of multi-agent debates.

##  6 Applications

Following our above outline of automated correction techniques, we now will
briefly discuss the various application domains for which automated correction
is useful, and point out commonalities in self-correction strategies, and
discuss how improvements to performance in self- or feedback-driven correction
will give rise to downstream performance improvements.

###  6.1 Factual Correction

Many of the aforementioned automated correction strategies implicitly engage
factual correction to improve various errors in output Gao et al. (2023b);
Peng et al. (2023). It is therefore natural that these capabilities can be
directly engaged for factuality detection and factual correction as an end
task, such as in LM vs LM Cohen et al. (2023) or Multiagent Debate Du et al.
(2023). When external tool use is acceptable, retrieved facts can be leveraged
to further improve the factuality of generated text in a self-correcting
manner without pure reliance on memorized knowledge Gou et al. (2023).

In short, self-correcting is a foundational technique in many LLM-based fact
correction or fact-checking systems, and models that are better at self-
directed factual correction will probably be better in a host of other self-
correction settings.

###  6.2 Reasoning Tasks

In most reasoning tasks, no good references from which outputs can be sanity-
checked are readily available Choi (2023). This is unfortunate, as the
“reasoning” capabilities provided by LLMs—in particular, their ability to
operate on natural language from instructions specified in natural language—is
a core driver for their popularity. Reasoning tasks constitute a broad array
of problems where this capacity is most necessary. For example, more complex
multi-hop question answering tasks Yang et al. (2018); Chen et al. (2021) can
be construed as requiring both factual correction and reasoning correction
capabilities Ho et al. (2023). Thus the question becomes, how can we prompt
the LLM to identify and correct intermediate reasoning errors? Similarly to
other application areas, both pure-LLM-driven and external tool-based error
detection techniques have been proposed.

LLM-based implementations of reasoning error detection include debate-based
techniques, which can be thought of as implicitly rolling consistent reasoning
enforcement in the “critic” module Cohen et al. (2023); Li et al. (2023c), and
self-refinement techniques Manakul et al. (2023). In short, a given passage
exhibiting reasoning (eg, step-by-step Wei et al. (2022b)) is fed into an LLM,
which is prompted to check for and/or correct reasoning errors directly. The
error detection often collaborates with a decoding algorithm such as the beam
search to lead the reasoning towards the correct direction Hao et al. (2023);
Yao et al. (2023a).

External feedback using techniques such as natural language inference (NLI)
can be both directly leveraged to spot errors as a heuristic for correction,
and as a means to score the quality Yang et al. (2022a); Golovneva et al.
(2023). However, there are some open questions regarding the quality of the
supervised learning-based tools like NLI Srikanth and Rudinger (2022); Saxon
et al. (2023).

Among different types of reasoning, the self-correction strategy has been well
studied and implemented for arithmetic reasoning, as outlined in Table 1 and
Table 2. One of the reasons for this skew is the relative ease of verifying
intermediate reasoning steps within arithmetic problems. Some recent studies
Pan et al. (2023); First et al. (2023) have started to extend the application
of this strategy to deductive reasoning. However, the implementation of self-
correction in a wider array of reasoning tasks, including inductive and
abductive reasoning, is still relatively under-explored.

###  6.3 Code Synthesis

Code generation is a burgeoning application domain for LLMs for which
correction is particularly important. Even human programmers tend to write
code through an iterative process of addition and correction. For humans,
strategies such as reading linter warnings, compiler/runtime errors, and
incorrect outputs to diagnose necessary changes to the source are all employed
in the software development process. Each of these strategies has natural
analogues in the code generation pipeline.

The aforementioned warnings, errors, or outputs are usually fed directly back
into the LLM to guide the code correction process Zhang et al. (2023a); Chen
et al. (2023b). After all, compiler failures are a particularly strong signal
that a piece of code will not work, having great utility in guiding LLM self-
correction. More excitingly, other work proposed to utilize more fine-grained
feedback such as program explanations generated by LLMs Chen et al. (2023d)
and the comparison with a reference oracle program Zhang et al. (2023b).

The above works only empirically show that LLMs exhibited a remarkable
capability for self-repairing the codes. In recent work, Olausson et al.
(2023) further conducted an in-depth analysis of how and when self-repair
works effectively. They found that self-repair is bottlenecked by the feedback
stage: substantial performance improvements were only noticed when feedback
was provided by expert human programmers or GPT-4. This revelation raises
intriguing questions, such as whether self-repair is an emergent ability
exclusive to certain LLMs and how could we endow smaller models with similar
capabilities.

###  6.4 Other Applications

#### Open-ended Generation.

In addition to handling pure factuality issues, self-correction can be applied
to subjective qualities of the generated text. These interventions include
post-hoc self-correcting toxic / harmful outputs Gou et al. (2023); Welleck et
al. (2023); Helbling et al. (2023), enhancing the narrative quality in story
generation Yang et al. (2022b), and refining response generation in dialogues
Ye et al. (2023); Yu et al. (2023). Given the subjectivity involved in
assessing the outputs, these studies often rely on detailed, natural language
feedback and employ an iterative refinement strategy for post-hoc refinement.

#### Machine Translation.

The concept of post-hoc self-correction has deep roots in the field of machine
translation (MT), where it is often called Automatic Post-Editing (APE) do
Carmo et al. (2021). A long line of prior works train models to fix
translation errors by either learning from human correction data Alabau et al.
(2014) or from synthetic training data Lee et al. (2021). To minimize the cost
of data collection, recent works Chen et al. (2023c); Raunak et al. (2023)
have leveraged the in-context learning ability of LLMs for post-editing
translations. Besides post-hoc methods, training-time correction Unanue et al.
(2021) and decoding-time correction Freitag et al. (2022) are also adopted by
prior works.

#### Summarization.

Automated model correction is commonly used in summarization to ensure the
factuality of the generated summary. There are two mainstream methods: 1)
training-time correction that imposes factuality constraints during training
Liu and Liu (2021); Wan and Bansal (2022); Scheurer et al. (2023), and 2)
post-hoc correction that post-edits generated summaries to correct factual
errors Cao et al. (2020); Saunders et al. (2022). Recent works have
investigated using RL to refine the model guided by automated feedback from
either reward models Akyürek et al. (2023) or language models Pang et al.
(2023).

##  7 Research Gaps and Future Directions

###  7.1 Theoretical Justifications

Although LLMs have exhibited a remarkable capability for self-analysis and
self-improvement, there remains a lack of theoretical justifications to
uncover the mystery of such ability. Therefore, we argue that the study of
underlying theoretical principles can offer a more transparent understanding
of self-correction. Subsequently, we propose several potential directions for
such explorations.

The ability of language models to self-correct is closely associated with
their capacity to exhibit metacognitive awareness, i.e., their understanding
of their own knowledge and uncertainties (Kadavath et al., 2022). Similarly,
the notion of calibration in language models, referring to their ability to
produce well-calibrated predictions with probabilities aligning closely with
observed frequencies of outcomes, is of paramount importance (Lin et al.,
2023). Recent research by Kadavath et al. (2022) reveals that pre-trained
language models, when presented with properly formatted multiple-choice and
true/false questions, demonstrate good calibration. Particularly, language
models exhibit well-calibrated responses to self-evaluation questions in few-
shot settings. On the other hand, fine-tuned language models, such as those
incorporating RLHF, require temperature adjustments to achieve calibration
since the model distribution is tailored to optimize specific behaviors.

While language models demonstrate some capacity for self-feedback, achieving
superior performance often necessitates incorporating external feedback
signals. The integration of feedback signals is closely linked to the
alignment of language models, a domain that still lacks comprehensive
understanding. For example, in RLHF, the choice of the metric to minimize
between the reward model output and the final model output significantly
impacts downstream task performance (Go et al., 2023), yet this aspect remains
underexplored in many applications. Furthermore, the optimal method for
automatically generating prompts to instruct language models effectively, for
tasks such as evaluating and refining their outputs, remains an open question.
Although Sordoni et al. (2023) have addressed this issue by treating natural
language prompts as parameters of the language model and performing discrete
optimization, their approach still requires hand-crafted meta prompts to
implement the algorithm.

###  7.2 Measuring the Ability of Self-Correction

Despite the promising progress in enabling LLMs to self-correct and improve
their outputs, current research only provides empirical evidence of its
effectiveness across diverse applications. However, a gap remains in
establishing robust quantitative metrics to understand and evaluate the self-
correction capability of LLMs. While various strategies have been proposed to
rectify LLM outputs, a comprehensive comparative evaluation of these
approaches is still missing. This includes metrics to assess the
effectiveness, applicability, complexity, and potential upper-bound limits of
each strategy within a unified context. Future research could aim to create
comprehensive evaluation frameworks which take into account variables such as
the complexity of the task, the degree of initial error, the improvement in
quality after self-correction, etc.

In addition to this, building benchmarks to diagnose self-correction
capabilities represents another promising research direction. Such diagnostic
datasets would enable a more standardized, objective evaluation of different
LLMs and self-correction strategies, driving the development of more accurate
and efficient models.

###  7.3 Continual Self-Improvement

Another promising yet under-explored area of LLM self-correction is the idea
of continual, life-long self-improvement. As LLMs are utilized in more
diverse, dynamic, and real-time contexts, the ability to adapt and improve
continually over time becomes essential. This is closely related to the
concept of continual (life-long) learning Wang et al. (2023b), where the model
continually learns new skills and adapts to novel environments and contexts.
Translating this to self-correction implies that LLMs continuously evaluate
their outputs, learn from errors, update their knowledge, and adapt their
decision-making strategies accordingly.

Studies on self-training such as Huang et al. (2022); Zelikman et al. (2022)
have evidenced that LLMs can self-improve by continuously training on their
own outputs that are positively evaluated by humans or models. However, these
studies typically concentrate on a single, one-time correction process and
evaluate improvements in a particular aspect. The robustness and stability of
self-training under continual settings remain uncertain. For example, a major
challenge of continual learning is catastrophic forgetting Kirkpatrick et al.
(2016), where the acquisition of new skills often leads to a considerable
decrease in previous capabilities. It is unclear whether similar issues may
emerge in a continual self-improve LLM, such as whether correcting one
behavior may unintentionally alter a previously corrected behavior.

Finally, exploring how to integrate various self-correction techniques to
efficiently build a continual self-improvement LLM is also worth
investigating. For example, post-hoc correction represents a more immediate
and less costly strategy, while training-time correction addresses model
behavior more fundamentally, with a higher computational cost. To combine
these strategies, post-hoc correction could be used to collect training data
(e.g., most frequently made mistakes and their corrections), which are used to
guide the periodically training-time correction to address these recurring
issues permanently.

###  7.4 Self-Correction with Model Editing

Recent years have witnessed a surge in techniques for model editing Sinitsin
et al. (2020); Cao et al. (2021); Yao et al. (2023b), aiming to adjust the
model’s behavior for examples within the editing scope while leaving its
performance for out-of-scope examples unaltered. Model editing has been
employed in updating outdated knowledge embedded in LLMs Lee et al. (2022);
Onoe et al. (2023) and in addressing issues related to false associations
memorized during LLM training Murty et al. (2022); Tanno et al. (2022).
Current model editing methods have shown some efficacy in adjusting factual
knowledge within LLMs, yet they still suffer problems such as a lack of robust
generalization capabilities Yao et al. (2023b) and the introduction of
substantial unintended side effects Hoelscher-Obermaier et al. (2023).

Nevertheless, the advancements in model editing present promising
opportunities for the self-correction of LLMs. Primarily, model editing
enables accurate, fine-grained corrections at the level of individual neurons
or layers, circumventing the need for extensive retraining associated with
training-time correction. Moreover, through the analysis of the impact of
model edits, we can deepen our understanding of the self-correction mechanism.
Further, methods developed to curtail undesired side effects in model editing
Hoelscher-Obermaier et al. (2023) could foster more robust self-correction
strategies by mitigating the issue of inadvertently introducing new errors
while resolving existing ones. Therefore, we forecast future research to
incorporate model editing into LLM self-correction processes, an under-
explored area.

###  7.5 Multi-Modal Self-Correction

As discussed in Section 6, self-correction strategies for LLMs have been
successfully employed in an extensive array of NLP tasks. However, most
existing works are limited to the textual modality, where both the model
outputs and the feedback are in textual form. The recent surge in multi-modal
data usage, including image, audio, and video modalities, presents enticing
opportunities for expansion. These include the exploration of self-correction
capabilities within multi-modal LLMs, the incorporation of visual feedback,
and improving vision-language tasks through self-correction.

A handful of pioneering studies have investigated this domain. For example,
MaskGIT Chang et al. (2022) employed a self-refinement approach to image
generation, where the model progressively refines the generated image
conditioned on the previous generation. Ke et al. (2019) utilized a self-
correction strategy for vision-and-language navigation. FLIRT Mehrabi et al.
(2023) uses self-correction to iteratively generate and revise adversarial
input prompts for text-to-image generation. However, despite these initial
explorations, self-correction strategies are yet to be broadly adopted in
multi-modal settings. A comprehensive understanding of how self-correction
methods generalize across various modalities is crucial for improving their
robustness and versatility.

##  8 Conclusion

In this paper, we present a comprehensive survey of self-correcting large
language models with automated feedback. We broadly categorize and analyze
various self-correction strategies, including training-time, generation-time,
and post-hoc corrections. We also discuss the major application areas of self-
correction, including correcting factual errors, enhancing reasoning
abilities, and improving code generation, among others. Finally, we outline a
number of potential future directions and associated challenges in this field.
Our goal with this paper is to provide a comprehensive and useful resource for
readers interested in the development of this rapidly evolving domain. To aid
in this effort, we create a continually-updated reading list in a GitHub
repository: https://github.com/teacherpeterpan/self-correction-llm-papers.

## Acknowledgements

This work was supported by the National Science Foundation Award #2048122. The
views expressed are those of the authors and do not reflect the official
policy or position of the US government. Thanks to Xinyuan Lu for assisting
with the Github reading list repo.

## References

  * Akyürek et al. (2023) Afra Feyza Akyürek, Ekin Akyürek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023.  RL4F: generating natural language feedback with reinforcement learning for repairing model outputs.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7716–7733.
  * Alabau et al. (2014) Vicent Alabau, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes García-Martínez, Ulrich Germann, Jesús González-Rubio, Robin L. Hill, Philipp Koehn, Luis A. Leiva, Bartolomé Mesa-Lao, Daniel Ortiz-Martínez, Herve Saint-Amand, Germán Sanchis-Trilles, and Chara Tsoukala. 2014.  CASMACAT: A computer-assisted translation workbench.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , pages 25–28. The Association for Computer Linguistics.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _CoRR_ , abs/2204.05862.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b.  Constitutional AI: harmlessness from AI feedback.  _CoRR_ , abs/2212.08073.
  * Begus et al. (2023) Gasper Begus, Maksymilian Dabkowski, and Ryan Rhodes. 2023.  Large linguistic models: Analyzing theoretical linguistic abilities of llms.  _CoRR_ , abs/2305.00948.
  * Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020.  Factual error correction for abstractive summarization models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6251–6258.
  * Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6491–6506.
  * Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. 2022.  Maskgit: Masked generative image transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11305–11315. IEEE.
  * Chang et al. (2023) Jonathan D. Chang, Kianté Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. 2023.  Learning to generate better than your LLM.  _CoRR_ , abs/2306.11816.
  * Charalambous et al. (2023) Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C. Cordeiro. 2023.  A new era in software security: Towards self-healing software via large language models and formal verification.  _CoRR_ , abs/2305.14752.
  * Chen et al. (2023a) Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. 2023a.  Improving code generation by training with natural language feedback.  _CoRR_ , abs/2303.16749.
  * Chen et al. (2023b) Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023b.  Codet: Code generation with generated tests.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Chen et al. (2023c) Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2023c.  Iterative translation refinement with large language models.  _CoRR_ , abs/2306.03856.
  * Chen et al. (2023d) Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023d.  Teaching large language models to self-debug.  _CoRR_ , abs/2304.05128.
  * Chen et al. (2021) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R. Routledge, and William Yang Wang. 2021.  Finqa: A dataset of numerical reasoning over financial data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3697–3711.
  * Chern et al. (2023) I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023.  Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios.  _CoRR_ , abs/2307.13528.
  * Choi (2023) Yejin Choi. 2023.  Common sense: The dark matter of language and intelligence.  In _Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ , page 2. ACM.
  * Clark et al. (2021) Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021.  All that’s ’human’ is not gold: Evaluating human evaluation of generated text.  In _Processings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7282–7296.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _CoRR_ , abs/2110.14168.
  * Cohen et al. (2023) Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023.  LM vs LM: detecting factual errors via cross examination.  _CoRR_ , abs/2305.13281.
  * Creswell and Shanahan (2022) Antonia Creswell and Murray Shanahan. 2022.  Faithful reasoning using large language models.  _CoRR_ , abs/2208.14271.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022.  Language models show human-like content effects on reasoning.  _CoRR_ , abs/2207.07051.
  * Dathathri et al. (2020) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020.  Plug and play language models: A simple approach to controlled text generation.  In _Proceedings of the 8th International Conference on Learning Representations (ICLR)_.
  * do Carmo et al. (2021) Félix do Carmo, Dimitar Shterionov, Joss Moorkens, Joachim Wagner, Murhaf Hossari, Eric Paquin, Dag Schmidtke, Declan Groves, and Andy Way. 2021.  A review of the state-of-the-art in automatic post-editing.  _Machine Translation_ , 35(2):101–143.
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023\.  Improving factuality and reasoning in language models through multiagent debate.  _CoRR_ , abs/2305.14325.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _CoRR_ , abs/2305.14387.
  * Fernandes et al. (2023) Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. 2023.  Bridging the gap: A survey on integrating (human) feedback for natural language generation.  _CoRR_ , abs/2305.00955.
  * First et al. (2023) Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. 2023.  Baldur: Whole-proof generation and repair with large language models.  _CoRR_ , abs/2303.04910.
  * Freitag et al. (2022) Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022.  High quality rather than high model probability: Minimum bayes risk decoding with neural metrics.  _Transactions of the Association for Computational Linguistics (TACL)_ , pages 811–825.
  * Fu et al. (2023) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023.  Improving language model negotiation with self-play and in-context learning from AI feedback.  _CoRR_ , abs/2305.10142.
  * Ganguli et al. (2023) Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. 2023.  The capacity for moral self-correction in large language models.  _CoRR_ , abs/2302.07459.
  * Gao et al. (2023a) Ge Gao, Hung-Ting Chen, Yoav Artzi, and Eunsol Choi. 2023a.  Continually improving extractive QA via human feedback.  _CoRR_ , abs/2305.12473.
  * Gao et al. (2023b) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023b.  Rarr: Researching and revising what language models say, using language models.  In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369.
  * Gero et al. (2023) Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. 2023.  Self-verification improves few-shot clinical information extraction.  _CoRR_ , abs/2306.00024.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin J. Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sona Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022.  Improving alignment of dialogue agents via targeted human judgements.  _CoRR_ , abs/2209.14375.
  * Go et al. (2023) Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. 2023.  Aligning language models with preferences through f-divergence minimization.  _CoRR_ , abs/2302.08215.
  * Golovneva et al. (2023) Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023.  ROSCOE: A suite of metrics for scoring step-by-step reasoning.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023.  CRITIC: large language models can self-correct with tool-interactive critiquing.  _CoRR_ , abs/2305.11738.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023\.  Reinforced self-training (rest) for language modeling.  _CoRR_ , abs/2308.08998.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _CoRR_ , abs/2301.07597.
  * Hao et al. (2023) Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023.  Reasoning with language model is planning with world model.  _CoRR_ , abs/2305.14992.
  * He et al. (2023) Hangfeng He, Hongming Zhang, and Dan Roth. 2023.  Rethinking with retrieval: Faithful large language model inference.  _CoRR_ , abs/2301.00303.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: decoding-enhanced bert with disentangled attention.  In _Proceedings of The 9th International Conference on Learning Representations (ICLR)_.
  * Helbling et al. (2023) Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. 2023.  Llm self defense: By self examination, llms know they are being tricked.  _CoRR_ , abs/2308.07308.
  * Ho et al. (2023) Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2023.  Wikiwhy: Answering and explaining cause-and-effect questions.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Hoelscher-Obermaier et al. (2023) Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. 2023.  Detecting edit failures in large language models: An improved specificity benchmark.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 11548–11559.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _CoRR_ , abs/2210.11610.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.  In _Proceedings of 5th International Conference on Learning Representations (ICLR)_.
  * Jiang et al. (2023) Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023.  Selfevolve: A code evolution framework via large language models.  _CoRR_ , abs/2306.02907.
  * Jung et al. (2022) Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022.  Maieutic prompting: Logically consistent reasoning with recursive explanations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1266–1279.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.  _CoRR_ , abs/2207.05221.
  * Ke et al. (2019) Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha S. Srinivasa. 2019.  Tactical rewind: Self-correction via backtracking in vision-and-language navigation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6741–6749.
  * Kenton et al. (2021) Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021.  Alignment of language agents.  _CoRR_ , abs/2103.14659.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019.  CTRL: A conditional transformer language model for controllable generation.  _CoRR_ , abs/1909.05858.
  * Khalifa et al. (2023) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023\.  Discriminator-guided multi-step reasoning with language models.  _CoRR_ , abs/2305.14934.
  * Kim et al. (2023) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023.  Language models can solve computer tasks.  _CoRR_ , abs/2303.17491.
  * Kirkpatrick et al. (2016) James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016.  Overcoming catastrophic forgetting in neural networks.  _CoRR_ , abs/1612.00796.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Proceedings of the 2022 Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023.  Pretraining language models with human preferences.  _CoRR_ , abs/2302.08582.
  * Kreutzer et al. (2018) Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018.  Can neural machine translation be improved with user feedback? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HIT)_.
  * Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022.  Coderl: Mastering code generation through pretrained models and deep reinforcement learning.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Lee et al. (2022) Kyungjae Lee, Wookje Han, Seung-won Hwang, Hwaran Lee, Joonsuk Park, and Sang-Woo Lee. 2022.  Plug-and-play adaptation for continuously-updated QA.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 438–447.
  * Lee et al. (2021) WonKee Lee, Baikjin Jung, Jaehun Shin, and Jong-Hyeok Lee. 2021.  Adaptation of back-translation to automatic post-editing for synthetic data generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , pages 3685–3691.
  * Levy et al. (2022) Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, and William Yang Wang. 2022.  SafeText: A benchmark for exploring physical safety in language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2407–2421.
  * Levy et al. (2021) Sharon Levy, Michael Saxon, and William Yang Wang. 2021.  Investigating memorization of conspiracy theories in text generation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4718–4729, Online. Association for Computational Linguistics.
  * Li et al. (2023a) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  _CoRR_ , abs/2305.11747.
  * Li et al. (2023b) Miaoran Li, Baolin Peng, and Zhu Zhang. 2023b.  Self-checker: Plug-and-play modules for fact-checking with large language models.  _CoRR_ , abs/2305.14623.
  * Li et al. (2023c) Ruosen Li, Teerth Patel, and Xinya Du. 2023c.  PRD: peer rank and discussion improve large language model based evaluations.  _CoRR_ , abs/2307.02762.
  * Li et al. (2019) Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019.  Deep reinforcement learning with distributional semantic rewards for abstractive summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6037–6043.
  * Li et al. (2022) Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022.  Diffusion-lm improves controllable text generation.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Li et al. (2023d) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023d.  Making language models better reasoners with step-aware verifier.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 5315–5333.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023.  Let’s verify step by step.  _CoRR_ , abs/2305.20050.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 3214–3252.
  * Lin and Chen (2023) Yen-Ting Lin and Yun-Nung Chen. 2023.  Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.  _CoRR_ , abs/2305.13711.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023.  Generating with confidence: Uncertainty quantification for black-box large language models.  _CoRR_ , abs/2305.19187.
  * Liu et al. (2023) Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023.  Chain of hindsight aligns language models with feedback.  _CoRR_ , abs/2302.02676.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  Simcls: A simple framework for contrastive learning of abstractive summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)_ , pages 1065–1072.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022.  QUARK: controllable text generation with reinforced unlearning.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Lyu et al. (2023a) Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023a.  New trends in machine translation using large language models: Case examples with chatgpt.  _CoRR_ , abs/2305.01181.
  * Lyu et al. (2023b) Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023b.  Faithful chain-of-thought reasoning.  _CoRR_ , abs/2301.13379.
  * Madaan et al. (2022) Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022.  Memory-assisted prompt editing to improve GPT-3 after deployment.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2833–2861.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  _CoRR_ , abs/2303.17651.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _CoRR_ , abs/2303.08896.
  * Mehrabi et al. (2023) Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. 2023.  Flirt: Feedback loop in-context red teaming.  _CoRR_ , abs/2308.04265.
  * Miao et al. (2023) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023.  Selfcheck: Using llms to zero-shot check their own step-by-step reasoning.  _CoRR_ , abs/2308.00436.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  _CoRR_ , abs/2305.14251.
  * Mishra et al. (2022) Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. 2022.  Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9465–9480.
  * Murty et al. (2022) Shikhar Murty, Christopher D. Manning, Scott M. Lundberg, and Marco Túlio Ribeiro. 2022.  Fixing model bugs with natural language patches.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 11600–11613.
  * Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023.  LEVER: learning to verify language-to-code generation with execution.  In _Proceedings of the 40th International Conference on Machine Learning (ICML)_.
  * Olausson et al. (2023) Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023.  Demystifying GPT self-repair for code generation.  _CoRR_ , abs/2306.09896.
  * Onoe et al. (2023) Yasumasa Onoe, Michael J. Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. 2023.  Can lms learn new entities from descriptions? challenges in propagating injected knowledge.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 5469–5485.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Pan et al. (2023) Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023.  Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning.  _CoRR_ , abs/2305.12295.
  * Pang et al. (2023) Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. 2023.  Language model self-improvement by reinforcement learning contemplation.  _CoRR_ , abs/2305.14483.
  * Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023.  REFINER: reasoning feedback on intermediate representations.  _CoRR_ , abs/2304.01904.
  * Peng et al. (2023) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.  _CoRR_ , abs/2302.12813.
  * Pu and Demberg (2023) Dongqi Pu and Vera Demberg. 2023.  Chatgpt vs human-authored text: Insights into controllable text summarization and sentence style transfer.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL)_ , pages 1–18.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver? _CoRR_ , abs/2302.06476.
  * Raunak et al. (2023) Vikas Raunak, Amr Sharaf, Hany Hassan Awadallah, and Arul Menezes. 2023.  Leveraging GPT-4 for automatic translation post-editing.  _CoRR_ , abs/2305.14878.
  * Ribeiro et al. (2023) Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Henry Zhu, Rui Dong, Deguang Kong, Juliette Burger, Anjelica Ramos, William Yang Wang, Zhiheng Huang, George Karypis, Bing Xiang, and Dan Roth. 2023.  STREET: A multi-task structured reasoning and explanation benchmark.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022.  Self-critiquing models for assisting human evaluators.  _CoRR_ , abs/2206.05802.
  * Saxon et al. (2023) Michael Saxon, Xinyi Wang, Wenda Xu, and William Yang Wang. 2023.  PECO: Examining single sentence label leakage in natural language inference datasets through progressive evaluation of cluster outliers.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , pages 3053–3066.
  * Scheurer et al. (2023) Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023.  Training language models with language feedback at scale.  _CoRR_ , abs/2303.16755.
  * Schick et al. (2023) Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick S. H. Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2023.  PEER: A collaborative language model.  In _Proceedings of The 11th International Conference on Learning Representations (ICLR)_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020.  BLEURT: learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7881–7892.
  * Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023\.  On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 4454–4470.
  * Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Minimum risk training for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1683–1692.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023.  Reflexion: Language agents with verbal reinforcement learning.  _CoRR_ , abs/2303.11366.
  * Sinitsin et al. (2020) Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. 2020.  Editable neural networks.  In _Proceedings of the 8th International Conference on Learning Representations (ICLR)_.
  * Sordoni et al. (2023) Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. 2023.  Deep language networks: Joint prompt training of stacked llms using variational inference.  _CoRR_ , abs/2306.12509.
  * Srikanth and Rudinger (2022) Neha Srikanth and Rachel Rudinger. 2022.  Partial-input baselines show that nli models can ignore context, but they don’t. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 4753–4763.
  * Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023.  Challenging big-bench tasks and whether chain-of-thought can solve them.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13003–13051.
  * Tafjord et al. (2022) Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2022.  Entailer: Answering questions with faithful and truthful chains of reasoning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2078–2093.
  * Tanno et al. (2022) Ryutaro Tanno, Melanie F. Pradier, Aditya V. Nori, and Yingzhen Li. 2022.  Repairing neural networks by leaving the right past behind.  In _Proceedings of the 2022 Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022.  Solving math word problems with process- and outcome-based feedback.  _CoRR_ , abs/2211.14275.
  * Unanue et al. (2021) Inigo Jauregi Unanue, Jacob Parnell, and Massimo Piccardi. 2021.  Berttune: Fine-tuning neural machine translation with bertscore.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)_ , pages 915–924.
  * Varshney et al. (2023) Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023.  A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.  _CoRR_ , abs/2307.03987.
  * Wan and Bansal (2022) David Wan and Mohit Bansal. 2022.  Factpegasus: Factuality-aware pre-training and fine-tuning for abstractive summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 1010–1028.
  * Wang et al. (2023a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023a.  Decodingtrust: A comprehensive assessment of trustworthiness in GPT models.  _CoRR_ , abs/2306.11698.
  * Wang et al. (2023b) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023b.  A comprehensive survey of continual learning: Theory, method and application.  _CoRR_ , abs/2302.00487.
  * Wang et al. (2023c) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c.  Self-consistency improves chain of thought reasoning in language models.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_.
  * Wang et al. (2023d) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023d.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 13484–13508.
  * Wang et al. (2023e) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023e.  Aligning large language models with human: A survey.  _CoRR_ , abs/2307.12966.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _CoRR_ , abs/2206.07682.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b.  Chain-of-thought prompting elicits reasoning in large language models.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023.  Generating sequences by learning to self-correct.  In _Proceedings of The 11th International Conference on Learning Representations (ICLR)_.
  * Weng et al. (2023) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2023\.  Large language models are better reasoners with self-verification.  _CoRR_ , abs/2212.09561.
  * Wu et al. (2021) Qingyang Wu, Lei Li, and Zhou Yu. 2021.  Textgail: Generative adversarial imitation learning for text generation.  In _Proceedings of 35th Conference on Artificial Intelligence (AAAI)_ , pages 14067–14075.
  * Wu et al. (2023a) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023a.  Fine-grained human feedback gives better rewards for language model training.  _CoRR_ , abs/2306.01693.
  * Wu et al. (2023b) Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023b.  Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks.  _CoRR_ , abs/2307.02477.
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023.  Decomposition enhances reasoning via self-evaluation guided decoding.  _CoRR_ , abs/2305.00633.
  * Xu et al. (2023a) Wenda Xu, Xian Qian, Mingxuan Wang, Lei Li, and William Yang Wang. 2023a.  SESCORE2: learning text generation evaluation via synthesizing realistic mistakes.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 5166–5183.
  * Xu et al. (2022) Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, and William Yang Wang. 2022.  Not all errors are equal: Learning text generation metrics using stratified error synthesis.  In _Findings of the Association for Computational Linguistics (EMNLP)_ , pages 6559–6574.
  * Xu et al. (2023b) Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023b.  INSTRUCTSCORE: towards explainable text generation evaluation with automatic feedback.  _CoRR_ , abs/2305.14282.
  * Yan et al. (2023a) Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, and Ziyu Yao. 2023a.  Learning to simulate natural language feedback for interactive semantic parsing.  In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 3149–3170.
  * Yan et al. (2023b) Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. 2023b.  BLEURT has universal translations: An analysis of automatic metrics by minimum risk training.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics(ACL)_ , pages 5428–5443.
  * Yang et al. (2022a) Kaiyu Yang, Jia Deng, and Danqi Chen. 2022a.  Generating natural language proofs with verifier-guided search.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 89–105.
  * Yang and Klein (2021) Kevin Yang and Dan Klein. 2021.  FUDGE: Controlled text generation with future discriminators.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 3511–3535.
  * Yang et al. (2022b) Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022b.  Re3: Generating longer stories with recursive reprompting and revision.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4393–4479.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2369–2380.
  * Yao et al. (2023a) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a.  Tree of thoughts: Deliberate problem solving with large language models.  _CoRR_ , abs/2305.10601.
  * Yao et al. (2023b) Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023b.  Editing large language models: Problems, methods, and opportunities.  _CoRR_ , abs/2305.13172.
  * Ye et al. (2023) Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. 2023.  Selfee: Iterative self-revising llm empowered by self-feedback generation.  Blog post.
  * Yu et al. (2023) Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023.  Improving language models via plug-and-play retrieval feedback.  _CoRR_ , abs/2305.14002.
  * Yuan et al. (2023) Weizhe Yuan, Kyunghyun Cho, and Jason Weston. 2023.  System-level natural language feedback.  _CoRR_ , abs/2306.13588.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022.  Star: Bootstrapping reasoning with reasoning.  In _Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)_.
  * Zhang et al. (2023a) Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023a.  Self-edit: Fault-aware code editor for code generation.  _CoRR_ , abs/2305.04087.
  * Zhang et al. (2023b) Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023b.  Algo: Synthesizing algorithmic programs with generated oracle verifiers.  _CoRR_ , abs/2305.14591.
  * Zhang et al. (2023c) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023c.  How language model hallucinations can snowball.  _CoRR_ , abs/2305.13534.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _Proceedings of 8th International Conference on Learning Representations (ICLR)_.
  * Zhu et al. (2023) Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023.  Solving math word problems via cooperative reasoning induced language models.  In _Processings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 4471–4485.
  * Zhuo et al. (2023) Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023.  Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity.  _CoRR_ , abs/2301.12867.