OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees
Under review as a conference paper at ICLR 2025

OPENPRM: BUILDING OPEN-DOMAIN PROCESS-
BASED REWARD MODELS WITH PREFERENCE TREES

Anonymous authors
Paper under double-blind review

ABSTRACT

Scaling inference-time computation is increasingly seen as the next frontier in
scaling laws for large language models. Previous work in mathematics and coding
has demonstrated the remarkable potential for inference-time scaling. During such
scaling, fine-grained supervision through process-based reward models (PRMs) is
essential for enhancement. However, exploration of inference-time scaling and
PRMs in open-domain problems remains limited, where lacking exact answers
and obtaining process supervision prove challenging. In this paper, we explore
the construction of PRMs for open-domain tasks, specifically for instruction-
following tasks. Utilizing existing outcome-based reward models (ORMs), we
develop sentence-level preference trees based on the prefix similarity of parallel
sampled candidates from datasets like UltraFeedback. This setup allows us to
derive weak supervision for processes via back-propagation from outcome-level
rewards. Subsequently, we integrate ORMs and PRMs under the same pairwise
ranking objectives, resulting in our newly developed reward models, named Open-
PRM. This approach significantly enhances the scalability of process-level super-
vision in open domains at minimal cost. We assess the performance of OpenPRM
across various reward benchmarks, demonstrating its competitive edge over tradi-
tional ORMs in open domains and PRMs in specialized domains. Additionally, we
investigate the scalability of inference-time computation for open-domain instruc-
tions. Our results highlight the limitations of ORMs’ scalability, while OpenPRM
shows superior performance in scaled settings. Despite these advances, achiev-
ing automatic fine-grained supervision for open-domain inference-time scaling
remains a substantial challenge. We hope these findings will spur further develop-
ment of process supervision reward models in open-domain scenarios.

1

INTRODUCTION

Large language models (LLMs) such as GPT-4 (Achiam et al., 2023), Llama (Touvron et al., 2023;
Dubey et al., 2024), and Gemini (Team et al., 2023; Reid et al., 2024) have garnered interest across
various fields due to their robust performance in numerous tasks and domains. The development
of LLMs involves an official process that includes pre-training on a large-scale unlabeled cor-
pus (Brown, 2020) followed by post-training using labeled instructions derived from real-world
applications. The post-training phase is further categorized into supervised fine-tuning and rein-
forcement learning from human or model feedback (Ouyang et al., 2022), a process known as align-
ment (Ji et al., 2023; Shen et al., 2023). During this phase, reward models are crucial as they act
as human proxies, providing feedback on model behavior and adjusting the models to better align
with human values (Bai et al., 2022). Although current popular alignment algorithms like Direct
Preference Optimization (DPO) (Rafailov et al., 2024) implicitly incorporate the rewarding process
within the loss function, reward models still play a significant role in ensuring long-term alignment
through methods such as online and iterative DPO (Xiong et al., 2023; Guo et al., 2024; Pang et al.,
2024) and rejected sampling (Dong et al., 2023; Liu et al., 2023; Khaki et al., 2024).

In addition to the training process, reward models are crucial for enhancing the performance of
LLMs during inference-time (Khanov et al., 2024; Deng & Raffel, 2023). Unlike the scaling laws
applied to compute during pre-training (Kaplan et al., 2020; Hoffmann et al., 2022), there is a trend
towards scaling inference-time compute through extensive searches in the decoding space (Snell

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

et al., 2024; Brown et al., 2024). Reward models play a significant role in pruning the search space
and ultimately selecting the most accurate answers (Welleck et al., 2024).

Recent studies indicate that outcome-level reward models fail to apply scaling laws during repeated
sampling (Brown et al., 2024), particularly due to their coarse granularity on challenging tasks.
Consequently, many researchers are exploring the use of more fine-grained reward models, such
as process-level (Uesato et al., 2022; Lightman et al., 2023) or token-level (Deng & Raffel, 2023),
to enhance search performance in specialized domains like mathematics, coding, and reasoning
tasks (Wang et al., 2024b; Havrilla et al., 2024; Xin et al., 2024; Yuan et al., 2024; Chen et al.,
2024a; Setlur et al., 2024; Xie et al., 2024). These efforts often follow the paradigm established
by AlphaGo (Silver et al., 2016), integrating LLMs with Monte Carlo Tree Search (Coulom, 2006;
Kocsis & Szepesv´ari, 2006) and a value function, analogous to a reward model. However, process-
level reward models are typically tailored for specific tasks and exhibit limited generalizability in
open-domain applications such as writing and chat. Moreover, there is scant research on developing
process-level reward models for open-domain contexts, primarily due to the high cost of annotation.

Currently, outcome-level reward models (ORMs) are evolving rapidly (Wang et al., 2024a; Cai et al.,
2024; Wang et al., 2024e;c; Vu et al., 2024), prompted by the emergence of datasets and benchmarks
such as Ultrafeedback (Cui et al., 2023) and RewardBench (Lambert et al., 2024), where open-
source reward models trend to outperform proprietary ones. This development raises the question
of whether process-level reward models (PRMs) can be constructed from instance-level rewards
using a weak-to-strong framework (Burns et al., 2023).

In this paper, we propose the development of PRMs in the open domain, leveraging existing ORMs
to provide fine-grained supervision. Our contributions are summarized as follows:

• We analyze the potential for extending open-domain ORMs to PRMs, elucidating the char-
acteristic and relationship between them. This analysis inspires our proposal to develop
PRMs with outcome-level supervision through building preference trees with key process.

• We integrate the modeling of PRMs and ORMs under a unified objective and develop Open-
PRM. By leveraging only existing ORMs and employing repeated sampling on prompts,
we enhance the performance of ORMs, achieving a 3∼5% improvement on RewardBench.

• We further evaluate OpenPRM across various downstream applications, including different
inference-time scaling settings. Our findings show that ORMs struggle to provide effective
supervision, while our proposed OpenPRM outperforms previous RMs under these scaling
conditions. We also observe that there is a significant journey ahead to fully realize the
potential of RMs in open-domain tasks under the inference-time scaling law.

2 PRELIMINARY

2.1 REWARD MODELING

Reward models play a crucial role in large language models by aligning model outputs with desired
human preferences (Wang et al., 2023). There are primarily two types of reward models based on
the granularity of the supervision signal: outcome-level and process-level reward models. We will
introduce the development of these two methods as follows.

Outcome-level Reward Model (ORM) ORMs are commonly used for preference learning, partic-
ularly after supervised fine-tuning in InstructGPT, where they serve as a proxy for human feedback
to model generations (Ouyang et al., 2022; Lee et al., 2023). Although many studies explore reward
model-free preference learning, such as direct preference optimization (DPO) (Rafailov et al., 2024),
which implicitly models the reward within the policy model training, ORMs continue to be instru-
mental in further model improvements. This includes applications in online or iterative DPO (Guo
et al., 2024; Pang et al., 2024) and rejection sampling (Liu et al., 2023; Dong et al., 2024).

The primary methods for obtaining ORMs involve preparing pairwise responses with preferences
(e.g., chosen and rejected) and fine-tuning instructed models using ranking loss (Ouyang et al., 2022;
Dong et al., 2024). Some studies also consider DPO models as reward models (Lambert et al., 2024),
though the effectiveness of these models in iterative optimization still requires further exploration.
Additionally, self-play learning (Chen et al., 2024b; Tao et al., 2024) has recently been applied to

2

Under review as a conference paper at ICLR 2025

Table 1: A comparison with the most related works on process-level reward models.

Name

Data Acquisition

Training & Inference

Release

Domain

Task

Backbone

Annotation

Size

Labeling

Objective

Search

Data Model

BoN

BoN

MCTS-α
+Rollout
100×17 leaf
MCTS
16×64 leaf

DEEPMIND PRM
(Uesato et al., 2022)
OPENAI PRM
(Lightman et al., 2023)

Math

Math

TS-LLM
(Feng et al., 2023)

Math
Decision

GSM8K

N/A

Human

10k

0/1

N/A

MATH

GPT-4

Human

800k

-1/0/1

CE Loss

GSM8K,GAME24,
ProntoQA,RLHF,
Chess Endgame

LLaMA2-7B

Golden Ans. ∼150k

0∼1

MSE Loss

MATH-SHEPHERD
(Wang et al., 2024b)
GLORE
(Havrilla et al., 2024)
MIPS
(Wang et al., 2024f)

MCTS-DPO
(Xie et al., 2024)

SUPER MARIO
(Chen et al., 2024a)
STEP-DPO
(Lai et al., 2024)
OMEGAPRM
(Luo et al., 2024)
REST-MCTS*
(Zhang et al., 2024)

Math

Math

GSM8K

LLemma-7B

Golden Ans.

445k

GSM8K

Llama2-7B

Golden Ans.

N/A

0/1

0/1

CE Loss

CE Loss

BoN

Math/Code

Math
Common
-sense

GSM8K, MATH
MBPP
GSM8K,MATH
ARC,AI2Science,
OpenBookQA,
CommonSenseQA

Math

GSM8K,MATH

PaLM 2-S/L

Golden Ans. ∼14k

0∼1

MSE Loss

Mistral-7B

Golden Ans. ∼24k

0∼1

MSE Loss

DeepSeek-
MathBase-7B

Golden Ans.

15k

0∼1

MSE Loss

MCTS
32×32 leaf

MCTS
4/5 leaf

MCTS
5 leaf

Math

MetaMath, MMIQC Qwen2-7B/72B

LLM

10k

0/1

Ranking Loss

BoN

Math

Math

MATH

MATH

Gemini Pro

Golden Ans.

1.5m

0∼1

MSE Loss

Mistral-7B

Golden Ans. ∼700k

0∼1

MSE Loss

MCTS

MCTS
3 leaf

✗

✓

✗

✓

✗

✗

✗

✓

✓

✗

✓

✗

✗

✓

✓

✗

✗

✗

✗

✗

✗

✗

continually improve reward models, presenting a promising method to enhance the capabilities of
ORMs autonomously (Wang et al., 2024c). Our approach on OpenPRM can also be considered a
method for continuously improving ORMs through their own annotations.

Process-level Reward Model (PRM) The primary challenge with ORMs is their coarse-grained
nature of rewards; even if the final answer is correct, errors may still exist within the solution steps.
To address this, there is a growing trend to develop more fine-grained, process-level RMs. The main
challenge in developing PRMs lies in obtaining accurate supervision signals for each process within
a solution. There are three main approaches: 1) Human Annotation: This method requires experts
to annotate each process step as neutral, bad, or good. While human annotation can provide precise
process supervision, it is difficult to scale and very costly (Uesato et al., 2022; Lightman et al., 2023).
2) Golden Answer.: For mathematical or coding problems, accurate final answers or feedback from
exact matching or interpreters are available. Common methods involve computing the probability
of nodes along the path toward the final, accurate answers, integrating with Monte Carlo Search
methods (Wang et al., 2024b; Havrilla et al., 2024; Luo et al., 2024). 3) Model-based Judgment
or Reward Models: The final approach involves obtaining rewards from model-based judgments or
reward models (Lai et al., 2024). Some research utilizes outcome-level rewards to estimate process
rewards (Lu et al., 2024), reducing the high costs associated with extensive sampling.

For training PRMs, two main methods are used, depending on the data format required: 1) Single
Sample: Each step in the solution process is labeled, and losses such as Cross-Entropy (CE) (Light-
man et al., 2023; Wang et al., 2024b; Havrilla et al., 2024) and Mean Squared Error (MSE) (Feng
et al., 2023; Wang et al., 2024f) loss are typically used. 2) Pair Sample: Each question is associ-
ated with chosen and rejected processes, and pairwise ranking loss is employed (Lai et al., 2024).
This method is typically used in the training ORMs (Dong et al., 2024). We provide a detailed
survey in Table 1. Currently, many ORMs in open domains benefit from the development of bench-
marks (Lambert et al., 2024; Wang et al., 2024e). However, the true effectiveness of these ORMs
and the feasibility of developing PRMs from ORMs are still subjects of ongoing exploration.

2.2 DERIVATION OF PRM FROM ORM

ORM is typically trained to predict the quality of the final outcome, while PRM supervises the
intermediate steps of the process. The modeling of ORM can be formalized with a pairwise ranking
loss function based on the outcome-level feedback (Ouyang et al., 2022):

LORM(θ) = −E(x,yc,yr)∼D [log (σ (rθ(x, yc) − rθ(x, yr)))]

(1)

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

Here, rθ(x, yc) and rθ(x, yr) are the model’s scores for the chosen and rejected outcomes, respec-
tively. While ORM performs well in outcome-based supervision, it has limitations when applied
to process-level supervision, due to the cumulative error effect. This effect arises because ORM
focuses on the final result, neglecting errors in intermediate steps that can propagate through the
sequence and affect the final outcome (Lightman et al., 2023).

Previous works (Uesato et al., 2022; Havrilla et al., 2024) have shown that in cases where the base
model for response generation is sufficiently strong and the task is relatively simple, the cumulative
error may be negligible. In such scenarios, ORM can be effectively used as a substitute for PRM.

However, for more complex tasks, the cumulative error can be significant, necessitating additional
process-level supervision. To address this issue, we propose a joint modeling approach that inte-
grates both ORM and PRM by introducing supervision at key process of answer. Specifically, we
identify the most divergent process between chosen and rejected outcomes, denoted as pc and pr,
and introduce additional supervision at these critical points. The loss function can be defined as:

L(θ) = LORM(θ) + λLPRM(θ)

(2)

where λ is a hyper-parameter that balances the outcome-based and process-based losses. The
process-based loss LPRM supervises the divergent steps pc and pr, and can be expressed as:

LPRM(θ) = − log (σ (rθ(x, pc) − rθ(x, pr)))

(3)

Here, rθ(x, pc) and rθ(x, pr) represent the scores for the critical steps in the chosen and rejected
sequences, respectively. By focusing on these key steps, the cumulative error is mitigated as early
errors are corrected at critical junctures. We provide more details in Appendix A. In conclusion, the
above analysis leads to the following theorem:

Theorem 1 Given a dataset D consisting of pairs of responses (yc, yr) with outcome-based pref-
erences yc > yr, and a learned outcome-based reward model rθ(x, y), the cumulative error of
process supervision can be significantly reduced. This is achieved by identifying the key divergent
steps (pc, pr), such that ∆(pc, pr) is maximized, and incorporating these steps into a joint modeling
framework. Thus, under this framework, the cumulative error in process supervision decreases as
the discrepancy ∆(pc, pr), supervision strength S, and model sensitivity γ increase.

2.3 EMPIRICAL EVALUATION OF OPEN-DOMAIN ORM IN PROCESS ASSESSMENT

In this section, we conduct an empirical evaluation of open-domain ORMs as process evaluators and
examine some unique phenomena associated with their performance. Specifically, we assess popu-
lar reward models such as FsfirX (Wang et al., 2024a), InternRM (Cai et al., 2024), and UltraFeed-
back (Cui et al., 2023) (trained on the Llama-3 8B model (Dubey et al., 2024)) using RewardBench
(named RB) (Lambert et al., 2024). We focus on the primary categories of RB, including Chat,
Chat-Hard, Reasoning, and Safety tasks. We provide more details about experiments in § 4.1.

Figure 1: Results of reward models based on rewarding processes of varying lengths within the
evaluated content. The results indicate that the initial segments of responses are particularly critical
for challenging tasks such as Chat-Hard and Reasoning, where longer content holds length bias.

As illustrated in Figure 1, a consistent trend is observed across various reward models, indicating
that RMs perform better on simpler Chat tasks as the evaluated process lengthens. However, this

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

p=1p=2p=3p=4p=-1first-n process707580859095100AccuracyInternRMp=1p=2p=3p=4p=-1first-n process65707580859095100FsfirXp=1p=2p=3p=4p=-1first-n process707580859095LlamaUFRB. ChatRB. Chat HardRB. ReasonRB. SafetyRB. AvgUnder review as a conference paper at ICLR 2025

effect reverses in more challenging Chat and Reasoning tasks, where accuracy first increases and
then decreases as the length of the evaluated process extends. In conclusion, these results support
our Theorem 1, which posits that ORMs can function as PRMs, but their performance deteriorates
due to cumulative errors, particularly in harder tasks.

3 METHODOLOGY

Figure 2: This figure illustrates the differences between outcome-level and process-level reward
models in the top left, including their common training strategies with paired and unpaired data.
The training of OpenPRM is depicted on the right and below (mainly blue area).

3.1 PROCESS-LEVEL PREFERENCE TREE

To obtain process-level rewards, we construct process-level preference trees using readily available
outcome reward models. The pipeline for building preference trees consists of three steps:

Step 1. Repeated Sampling on Prompts We initially prompt open-source language models to
generate a large number of parallel candidate responses through repeated sampling. To ensure broad
representation, we primarily include models at the 7B and 70B parameter levels.

Step 2. Aggregation on Sentences For each output, we segment it into a collection of sentences
and construct a tree using depth-first search algorithms. We calculate the edit distance (Ristad &
Yianilos, 1998) between sentences from different outputs and merge sentences into a single node
based on a predefined threshold. This helps us reduce the cost of building prefix trees.

Step 3. Backpropagation on Rewards Once outputs with their respective rewards are segmented
into sentence collections, which serve as nodes in the preference tree, we designate the outcome
rewards for the leaf nodes. For each process node, we compute the process-level rewards using
backpropagation. Given the rewards of the leaf nodes, Rk (outcome-level rewards), we can compute
the rewards of the inner nodes, Pij, using backpropagation, as detailed in Monte Carlo Tree Search.
Notably, the rewards of the inner nodes, denoted as V (Pij), can be calculated using the formula

V (Pij) =

(cid:80)

k∈L(Pij ) Rk
|L(Pij)|

,

where L(Pij) represents the set of all leaf nodes descending from Pij.

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

PromptsModelCA1CA2CA3CA4R1R2R3R4R3R4R2R1SamplingRewardingP11P21P31P32P12P22P23P11P21P31>P11P21P32P12P22>P12P23𝑲ORMs𝑹(𝒙,𝒚){∑𝟎$𝒋$𝑲𝑹𝒊𝒋𝑲}SegmentBack-PropagationProcessPreference...Aggregate𝒙𝒊,𝒚𝒊𝒊&𝟎𝑵CandidateAnswersOpenPRMTraining𝑦"=0𝑦!=1SimilarAnswerPrefixKeyProcess𝑦+/𝑦&/𝑽𝑷𝒊𝒋=∑𝒌∈𝑳𝑷𝒊𝒋𝑹𝒌𝑳(𝑷𝒊𝒋)OutcomeRM(ORM)ProcessRM(PRM)R1R2R3ORMPRMR12R11R21R31R33Unpaired𝑳𝒐=𝑹𝒙−𝒚𝟐𝑳𝑷=/𝒊&𝟎𝑵𝑹𝒙𝒊−𝒚𝒊𝟐RewardModel(RM)𝑦!"=1𝑦!!=1𝑦!#=0𝑦!$=0𝑦""=1𝑦"!=0𝑦"#=1𝑦"$=1𝑦"=0.5𝑦%=0𝑦!=1Pairwise𝑦"=0𝑦!=1...𝑳=−𝒍𝒐𝒈(𝝈(𝒓𝜽𝒙,𝒚𝒄−𝒓𝜽(𝒙,𝒚𝒓)))𝑦&’()*%𝑦+*,*&-*.RewardModeling𝒚𝒄→𝒚𝒄𝒑𝒚𝒓→𝒚𝒓𝒑Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

About Rationality of Process Aggregation Unlike previous works in mathematics and coding that
reuse partial answers for subsequent answer generations (Lightman et al., 2023; Luo et al., 2024),
our method involves directly sampling a large number of candidates and merging identical sentences,
akin to state aggregation in Monte-carlo algorithms (Hostetler et al., 2014; Jang et al., 2021). This
approach enables the RMs to learn high-level actions and logic within the shared sentences. We
provide a real example of question-answering for reference in the Appendix B.3.

3.2 PROCESS-LEVEL REWARD MODELING

During the development of OpenPRMs, we enhance the models by integrating rewards and domains,
aiming to create more robust process-level reward models from the following two perspectives:

Mixture of Rewards Considering the completeness of outputs, output-level rewards serve as spe-
cific instances of process-level rewards, where the output encompasses the entire process. Therefore,
we blend rewards from both the process and output levels to develop more robust reward models.

Mixture of Domains Existing process-level reward models predominantly focus on domains such
as mathematics, and reasoning tasks, which provide certain answers for supervision. To leverage the
strengths of these domains, we also integrate them with general domain preferences to enhance the
versatility and applicability of OpenPRMs. We provide details about dataset in Appendix B.1.

At the training stage, we treat all preference data as a pairwise ranking task. This involves using the
input prompt along with chosen and rejected completions (including both process and output). Using
this unified format, we train the PRM with the Bradley-Terry objective, as defined in Equations 1
and 3. This formulation ensures consistent training across both process- and outcome-level datasets.

3.3 APPLICATION OF PROCESS-LEVEL REWARD MODELS

Best-of-N Sampling At inference time, we can generate a large number of candidate answers for
given questions. Subsequently, we can determine the final answer through a majority vote (James,
1998) (referred to as Vote@N); however, this method is primarily applicable to questions that re-
quire exact answers, such as those found in mathematics and reasoning tasks. For open-ended ques-
tions, it is more common to apply reward models to all answers and select the one with the highest
rewards, a method known as best-of-N sampling (Stiennon et al., 2020) (BoN@N). When imple-
menting process-level reward models in the BoN context, there are two approaches to computing
rewards: one approach treats the outcome as a special process and computes rewards directly on the
outcome, while the other calculates rewards for each process and selects the minimal one (Lightman
et al., 2023; Wang et al., 2024b) to derive the outcome rewards.

Process-level Decoding Another significant application of PRMs is in the decoding phase. By
evaluating the generated process, we can expand the beam search strategy (Sutskever et al., 2014)
from token-level to process-level. As a result, we maintain N sentences at each step and reward each
sentence during generation until the completion of the answer, a technique termed process-level
beam search (PBS@N). Additionally, we can integrate advanced operations akin to those employed
in Monte Carlo Tree Search (MCTS) (Browne et al., 2012), such as simulation, retrospection, and
memory functions. However, these operations may extend the required processing time and lead to
increased inference costs. Previous research (Chen et al., 2024a; Snell et al., 2024) has indicated
that PBS@N can achieve performance comparable to MCTS but at a reduced cost.

4 EXPERIMENTAL SETUP

4.1 DATASET

In developing OpenPRM, we first construct extensive preference trees based on open-domain in-
struction dataset, as described in § 3.1. This construction utilizes the UltraFeedback (Cui et al.,
2023) and ScienceQA (Lu et al., 2022) datasets, which provide a highly diverse and high-quality
range of instructions. Additionally, we incorporate the MATH (Hendrycks et al., 2021) dataset to
further enhance the math reasoning capabilities of our reward system. For each prompt within this
instruction pool, we sample 64 candidate responses from Llama-3 models (Dubey et al., 2024).

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

4.2 MODELS

Reward Models. We compare our reward models with state-of-the-art (SOTA) open-source reward
models. Due to concerns with inference efficiency, we primarily evaluate classifier-based models,
which perform comparably to generative models but are more scalable. We compare our models
with ORMs, such as FsfairX (Dong et al., 2024), Eurus (Yuan et al., 2024), and UltraRM (Cui et al.,
2023), and PRMs like TS-LLM (Chen et al., 2024a), MathShepherd (Wang et al., 2024b).

Chat Models. We assess the effectiveness of our reward models using state-of-the-art open-sourced
chat models, including Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct (Dubey et al., 2024), and
Mistral-Nemo-Instruct-2407 1. The latter can be regarded as an out-of-distribution evaluation.

4.3 EVALUATIONS

Evaluation of Reward Models. Given the lack of established benchmarks for evaluating process-
based reward models, we primarily compare our process-based models against established outcome-
based reward benchmarks, such as UltraFeedback (Cui et al., 2023) and RewardBench (Lambert
et al., 2024). RewardBench is designed to assess the capabilities and safety of reward models
across four categories: Chat, Hard Chat, Reasoning, and Safety. We employ the primary dataset
from RewardBench to evaluate the out-of-domain generalization capabilities of our reward mod-
els. We evaluate the effectiveness of process supervision of reward models solely on the test set
of PRM800k (Lightman et al., 2023), which features high-quality human annotations. Especially,
we evaluate PRMs using specific aggregation strategies, such as selecting the minimal reward across
steps. Detailed descriptions of the different aggregation strategies are provided in the Appendix D.1.

Evaluation of Chat Models. To comprehensively evaluate the impact of reward models on chat
models, we test the chat models across a variety of benchmarks, primarily referencing the Open
LLM Leaderboard 2. This evaluation includes benchmarks in: 1) Instruction following tasks such as
Alpaca Eval 2 (Dubois et al., 2024) and IFEval (Zhou et al., 2023); 2) General domain tasks such as
MixEval Hard (Ni et al., 2024), MMLU-Pro (Wang et al., 2024d), and GPQA (Rein et al., 2023); 3)
Specific math domain tasks like MATH500 (Lightman et al., 2023). Additional details about these
evaluation tasks and methodology are provided in Appendix C.

4.4 SETTINGS FOR INFERENCE

During inference, we primarily evaluate two methods: majority vote and best-of-N. For the best-of-
N method, we adhere to the following protocol (Chen et al., 2021): initially, we sample N responses,
where N is set to 128. We then sample K responses from these, repeating the process M times to
average the results. The values for K range from 1, 2, 4, 8, 16, 32, 64, to 128, and M is set to 5. This
approach allows us to reduce inference costs and achieve robust results through multiple averaging.
N to maintain an approximately equivalent
For process-based beam search, we set the beam size to
decoding cost with best-of-N, as described in (Snell et al., 2024).

√

5 EXPERIMENTAL RESULTS

5.1 RESULTS OF REWARD BENCHMARKS

As shown in Table 2, we compare OpenPRMs with standard ORMs and specialized PRMs across
both general outcome-based and specific process-reward benchmarks. Based on the results, we can
draw the following conclusions:

OpenPRMs outperform ORMs Utilizing off-the-shelf ORMs and corresponding preference
datasets, we have developed advanced reward models that demonstrate superior performance on
RewardBench, particularly in the Chat Hard and Reasoning tasks. Additionally, the process-based
preferences built upon our method consistently enhance the performance of the base reward models,

1https://mistral.ai/news/mistral-nemo/
2https://hf.co/spaces/open-llm-leaderboard/open_llm_leaderboard

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Results of outcome-level and process-level reward models on instance-level reward bench-
marks. Models marked with an asterisk (*) were trained using data compiled by our team.

Model / Task

Training
Data

UltraFeedback
Test

RewardBench

Overall Chat Chat Hard Safety Reasoning

PRM800k
Test

Open-Source ORMs

ULTRARM-13B

UltraFeedback (UF)

LLAMA-3-8B*

UF Binarized

EURUS-RM-7B

UltraInteract

FSFAIRX-7B

Mixture Preference

INTERN2-7B-RM

Unknown

LLAMA-3-8B*

HelpSteer (HS) 2

74.8

77.8

73.5

74.5

77.4

71.8

68.5

84.8

82.8

84.4

87.6

86.8

96.4

97.5

98.0

99.4

99.2

95.3

Open-Source PRMs (Merely Math Domain)

MS-7B-PRM

Math-Shepherd

TS-LLM

Unknown

LLAMA-3-8B*

Math-Step-DPO

OPENPRM (FsfairX)

UF Tree + HS 2

OPENPRM (InternRM)

UF Tree + HS 2

53.5

52.7

70.2

72.8

78.5

56.6

57.6

73.2

89.4

91.1

62.3

66.8

98.0

95.5

98.0

55.5

66.9

65.6

65.1

69.5

76.8

51.3

50.0

58.8

81.1

81.6

59.9

85.5

81.4

86.8

87.2

85.9

39.6

55.3

59.9

88.7

89.5

62.4

89.2

86.3

86.4

94.5

89.2

73.2

58.4

76.1

92.1

95.1

50.8

51.8

60.6

53.3

61.0

53.7

56.9

57.7

61.3

64.3

68.1

showcasing their generalization capabilities. These findings validate the effectiveness of our prefer-
ence tree construction strategy discussed in § 3.1. Moreover, the results substantiate our ability to
enhance weaker existing models, achieving weak-to-strong generalization (Burns et al., 2023).

OpenPRMs outperform specific PRMs Beyond outcome-level reward benchmarks, we also com-
pare OpenPRMs with publicly available PRMs, which predominantly originate from the math do-
main, as many previous PRMs are not available. We present some results evaluated using Math-
Shepherd (Wang et al., 2024b), TS-LLM (Feng et al., 2023) and Llama-3 trained on (Lai et al.,
2024). Due to the domain gap, these math-specific PRMs underperform in open-domain bench-
marks, whereas OpenPRMs demonstrate superior performance even on tasks like PRM800k.

5.2 RESULTS OF APPLICATIONS IN DECODING

To validate the effectiveness of PRMs, we evaluated OpenPRM under various decoding settings
across multiple popular open-domain tasks, comparing strategies such as majority vote, best-of-N,
and process-level beam search. We summarize the experimental results of OpenPRM as follows:

OpenPRM Performs Effectively with BoN and PBS As illustrated in Table 3, OpenPRM achieves
superior performance in both BoN@16 and PBS@4 compared to Vote@16 with the Llama-3.1-
8B and 70B models across nearly all tasks. These results confirm the effectiveness of OpenPRM.
Additionally, even the out-of-distribution models, such as Mistral-Nemo (compared to the Llama
series), validate the advantages of OpenPRM. We also observed that beam search algorithms outper-
form BoN, benefiting from the fine-grained evaluation of processes. However, further exploration
of decoding strategies (like MCTS) in open-domain settings will be necessary in the future.

Scaling Inference-Time Achieves a High Upper Boundary We further analyze the results of scal-
ing inference-time by progressively increasing sampling times from 1 (20) to 128 (27). The results
depicted in Figures 3 demonstrate that the models can achieve exceptional performance with opti-
mal reward models (refer to coverage@N and pass@N settings (Chen et al., 2021)). The coverage
accuracies nearly reach 100% on most open-domain tasks. These findings in open-domain tasks are
consistent with prior studies in mathematical and coding domains (Brown et al., 2024; Bansal et al.,
2024), suggesting the emergence of a new scaling law at inference time in open-domain as well. We
also include the sampling curves for the Llama 70B and Mistral model in Appendix D.4.

OpenPRMs Optimize Inference-Time Utilization Compared to the coverage accuracy depicted
by the red curve, nearly all the reward models struggle to scale as inference-time increases. This
indicates that significant advancements are still required to develop more effective reward models

8

Under review as a conference paper at ICLR 2025

Table 3: Results of majority vote, best-of-N sampling, and process-level search on open-domain
tasks. The findings from BoN@16 and PBS@16 demonstrate the effectiveness of OpenPRMs. No-
tably, we have reproduced part of the results, taking into account differences in dataset usage.

Alpaca Eval 2 MixEval

IFEval GPQA MMLU-P* MATH*

Model / Task

GPT-4O (0806)

GPT-4-TURBO

CLAUDE-3.5-SONNET

LLAMA-3.1-8B
INSTRUCT

REPORTED

REPRODUCED

VOTE@16

BON@16

PBS@4

LC% WC%

57.5

55.0

52.4

20.9

34.6

35.8

39.9

51.3

46.1

40.6

21.8

33.3

35.1

42.2

MISTRAL-NEMO
INSTRUCT-202407

LLAMA-3.1-70B
INSTRUCT

REPRODUCED

45.0

37.5

VOTE@16

BON@16

PBS@4

REPORTED

REPRODUCED

VOTE@16

48.4

53.2

38.1

42.4

39.0

49.8

29.9

41.6

Acc

64.7

62.6

68.1

45.6

39.7

46.5

47.2

35.7

36.6

42.0

55.9

59.1

Avg.

85.6

84.4

88.0

79.5

76.5

80.7

75.59

64.2

69.5

63.4

85.8

85.4

BON@16

44.4

42.9

61.9

87.7

Acc

75.9

73.4

71.1

45.0

24.3

26.8

30.8

31.3

31.9

35.2

35.5

37.9

65.8

45.7

51.4

49.8

Acc

74.68

63.71

76.12

40.8

37.2

43.4

50.5

47.8

36.6

44.4

48.2

44.0

55.0

55.4

60.3

67.1

Acc

53.1

49.3

59.4

23.7

45.6

56.4

58.8

52.8

31.4

40.7

52.3

47.8

41.9

63.6

70.7

72.0

Figure 3: Results of scaling inference-time for Llama-3.1-8B-Instruct on open-domain tasks. These
results illustrate the effectiveness of OpenPRMs relative to existing reward models, yet they also
highlight the distance to the upper boundary of coverage accuracy (red curve).

for inference-time scaling. Among these previous models, InternRM performs best on most tasks
on average, notably on MMLU-Pro and IFEval, though it still lags significantly behind the coverage
curve. In contrast, our proposed OpenPRM outperforms InternRM, showing promising results in
scaling up the best-of-N sampling. However, achieving scaling comparable to the coverage curve

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

21232527Number of Samples (N)405060708090Accuracy (%)Llama-3.1-8B-Instruct,MMLU-Pro21232527Number of Samples (N)2030405060708090100Accuracy (%)Llama-3.1-8B-Instruct,GPQA21232527Number of Samples (N)5060708090Accuracy (%)Llama-3.1-8B-Instruct,MATH21232527Number of Samples (N)788082848688909294Accuracy (%)Llama-3.1-8B-Instruct,IFEval21232527Number of Samples (N)38404244464850Accuracy (%)Llama-3.1-8B-Instruct,MMLU-Pro21232527Number of Samples (N)2224262830323436Accuracy (%)Llama-3.1-8B-Instruct,GPQA21232527Number of Samples (N)45.047.550.052.555.057.560.0Accuracy (%)Llama-3.1-8B-Instruct,MATH21232527Number of Samples (N)7778798081Accuracy (%)Llama-3.1-8B-Instruct,IFEvalCoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMUnder review as a conference paper at ICLR 2025

remains a substantial challenge. We will release all of these sampling data to the public to encourage
further study on process and outcome reward models for inference-time scaling.

6 DISCUSSION

6.1 ABLATION STUDY OF PRM TRAINING

We conducted an ablation study on OpenPRM training, focusing on data sources and model config-
urations. As illustrated in Table 4a, we compared the effects of continuously fine-tuning InternRM
and FsfairX using process pairs built upon preference trees and outcome pairs. The results indicate
that process pairs yield superior outcomes, thus validating the effectiveness of our method described
in Section 3.1. Additionally, the performance of InternRM, when using a shared prefix, was inferior
to configurations using distinct prefixes for chosen and rejected pairs, emphasizing the importance
of semantic consistency. Furthermore, while using only UltraFeedback data showed promising re-
sults in Chat tasks, maintaining diversity and generalization for open-domain applications is crucial.
Therefore, we opted to integrate additional reasoning and STEM questions.

6.2 REWARDS SHAPE AND LENGTH BIAS

Model

Data

RB Avg.

InternRM

FsfairX
Llama-3-8B-It

InternRM
FsfairX
Llama-3-8B-It

PrefTree Pairs
w/ Shared Prefix

PrefTree Pairs
PrefTree Pairs

Outcome Pairs
Outcome Pairs
Outcome Pairs

91.1
90.8

89.4
87.2

88.4
87.7
86.8

(a) Ablation Study of OpenPRM.

(b) Rewards VS. Length

Figure 4: Ablation Study of OpenPRM

We compare the reward shapes of OpenPRM with other reward models in Figure 6. We analyze
the rewards of chosen and rejected candidates for each instruction in RewardBench and observe that
while all reward models can generally distinguish between chosen and rejected candidates, indicated
by a shift in distributions, there remains some overlap. However, OpenPRM exhibits the minimal
overlap among them, similar to the parent reward model (i.e., InternRM).

Language model-based judgers often suffer from length bias, typically awarding higher rewards to
longer responses. We address this issue in our analysis of OpenPRM, visualizing the correlation
between OpenPRM and InterRM in Figure 4b. The results indicate that OpenPRM maintains a cor-
relation of 0.05, compared to 0.37 for InterRM, suggesting that process-level modeling effectively
reduces length bias. This also demonstrates the effectiveness and necessity of developing PRMs.

7 CONCLUSION

In this paper, we explore the development of process-based reward models (PRMs) in the open do-
main. We begin by generalizing rewards from outcome-level to process-level, significantly reducing
data annotation costs. We then propose the construction of preference trees with parallel candidates
for open-domain instructions, from which we derive process pairs using back-propagation. Leverag-
ing this data, our trained OpenPRM achieves excellent results on reward benchmarks and performs
well under scaling inference-time search conditions. However, our findings also highlight that there
is still considerable progress to be made in building open-domain PRMs to achieve high coverage
accuracy. In conclusion, we try to unify ORMs and PRMs in the open domain, paving a new path
for PRM development that diverges from domains such as mathematics and coding. We hope that
OpenPRM will spark new insights into this topic and stimulate further research.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

50510Reward02004006008001000LengthCorr(Orig) = 0.37Corr(Our) = 0.05ModelOrigOur0.00.10.2DensityModelOrigOur0.0000.001DensityModelOrigOurUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn
Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless
assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,
2022.

Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller,
arXiv preprint

weaker, yet better: Training llm reasoners via compute-optimal sampling.
arXiv:2408.16737, 2024.

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´e, and
Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.
arXiv preprint arXiv:2407.21787, 2024.

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp
Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey
of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in
games, 4(1):1–43, 2012.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren-
ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong general-
ization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390,
2023.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui
Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye
Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting
Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li,
Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun
Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang
Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song,
Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang,
Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong,
Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia
Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo
Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui
Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou,
Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.

Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision

without process. arXiv preprint arXiv:2405.03553, 2024a.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared
Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large
language models trained on code. arXiv preprint arXiv:2107.03374, 2021.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning
converts weak language models to strong language models. arXiv preprint arXiv:2401.01335,
2024b.

Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help

mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023.

R´emi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International

conference on computers and games, pp. 72–83. Springer, 2006.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu,
and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv
preprint arXiv:2310.01377, 2023.

Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation

with a unidirectional reward model. arXiv preprint arXiv:2310.09520, 2023.

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao,
Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative
foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen
Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf.
arXiv preprint arXiv:2405.07863, 2024.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.

Yann Dubois, Bal´azs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al-
pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.

Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-
arXiv preprint

like tree-search can guide large language model decoding and training.
arXiv:2309.17179, 2023.

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre
Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from
online ai feedback. arXiv preprint arXiv:2402.04792, 2024.

Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravin-
skyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve llm reasoning
via global and local refinements. arXiv preprint arXiv:2402.10963, 2024.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS,
2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train-
ing compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

Jesse Hostetler, Alan Fern, and Tom Dietterich. State aggregation in monte carlo tree search. In

Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.

Gareth Michael James. Majority vote classifiers: theory and applications. Stanford University,

1998.

Youngsoo Jang, Seokin Seo, Jongmin Lee, and Kee-Eung Kim. Monte-carlo planning and learning
with language action value estimates. In International Conference on Learning Representations,
2021.

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan,
Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv
preprint arXiv:2310.19852, 2023.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language
models. arXiv preprint arXiv:2001.08361, 2020.

Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection
sampling and direct preference optimization method for alignment of large language models.
arXiv preprint arXiv:2402.10038, 2024.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search.

arXiv preprint arXiv:2402.01694, 2024.

Levente Kocsis and Csaba Szepesv´ari. Bandit based monte-carlo planning. In European conference

on machine learning, pp. 282–293. Springer, 2006.

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-
wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629,
2024.

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu,
Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi.
Rewardbench: Evaluating reward models for language modeling, 2024.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton
Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning
from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint
arXiv:2305.20050, 2023.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and
arXiv preprint

Jialu Liu. Statistical rejection sampling improves preference optimization.
arXiv:2309.06657, 2023.

Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and
Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence
variation. arXiv preprint arXiv:2405.16802, 2024.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord,
Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for
science question answering. In The 36th Conference on Neural Information Processing Systems
(NeurIPS), 2022.

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun
Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated
process supervision. arXiv preprint arXiv:2406.06592, 2024.

Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and
Yang You. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint
arXiv:2406.06565, 2024.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol-
low instructions with human feedback. Advances in neural information processing systems, 35:
27730–27744, 2022.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason
Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024.

Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever-

aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.06551, 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. Advances
in Neural Information Processing Systems, 36, 2024.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-
baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem-
ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
arXiv:2403.05530, 2024.

13

Under review as a conference paper at ICLR 2025

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di-
rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench-
mark. arXiv preprint arXiv:2311.12022, 2023.

Eric Sven Ristad and Peter N Yianilos. Learning string-edit distance. IEEE Transactions on Pattern

Analysis and Machine Intelligence, 20(5):522–532, 1998.

Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on
incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint
arXiv:2406.14532, 2024.

Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu,
arXiv preprint

Yan Liu, and Deyi Xiong. Large language model alignment: A survey.
arXiv:2309.15025, 2023.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,
Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering
the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally
can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,
Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances
in Neural Information Processing Systems, 33:3008–3021, 2020.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le.

Sequence to sequence learning with neural
networks.
In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger
(eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2014/
2014.
file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf.

Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei
Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models.
arXiv preprint arXiv:2404.14387, 2024.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly
capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia
Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and
outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung.
Foundational autoraters: Taming large language models for better automatic evaluation. arXiv
preprint arXiv:2407.10817, 2024.

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong
Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment
with multi-objective rewards. arXiv preprint arXiv:2402.18571, 2024a.

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang
Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Pro-
ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 9426–9439, 2024b.

Tianlu Wang,

Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan,

Jane Dwivedi-Yu,
Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught eval-
uators. arXiv preprint arXiv:2408.02666, 2024c.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming
Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi
Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language
understanding benchmark, 2024d.

Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang,
Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint
arXiv:2307.12966, 2023.

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang,
Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training
top-performing reward models. arXiv preprint arXiv:2406.08673, 2024e.

Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang.
Multi-step problem solving through a verifier: An empirical analysis on model-induced process
supervision. arXiv preprint arXiv:2402.02658, 2024f.

Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig,
Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms
for large language models. arXiv preprint arXiv:2406.16838, 2024.

Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi,
and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning.
arXiv preprint arXiv:2405.00451, 2024.

Huajian Xin, ZZ Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue
Zhang, Xuan Lu, Qiushi Du, et al. Deepseek-prover-v1. 5: Harnessing proof assistant feedback
for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408.08152, 2024.

Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam-
pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint
arXiv:2312.11456, 2023.

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin
Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees.
arXiv preprint arXiv:2404.02078, 2024.

Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-

training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny
Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint
arXiv:2311.07911, 2023.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A DETAILS ABOUT DERIVATION OF PRM FROM ORM

The cumulative error from supervising only the outcome can be expressed as follows:

Cumulative Error =

T
(cid:88)

i=1

ϵi ·

1 − αT −i+1
1 − α

(4)

where ϵi is the error at step i, and α represents the degree to which errors propagate to subsequent
steps.

The reduction in cumulative error can be expressed as:

Cumulative Errornew =

T
(cid:88)

i=i∗

β · ϵi ·

1 − αT −i+1
1 − α

(5)

where β is a reduction factor that depends on the discrepancy between pc and pr, and the strength
of the supervision. We define β as follows:

β =

1
1 + γ · ∆(pc, pr) · S

(6)

where ∆(pc, pr) is the discrepancy between the key steps, γ is a model-dependent factor repre-
senting the model’s sensitivity to supervision, and S is the strength of the supervision signal. As
∆(pc, pr) and S increase, the factor β decreases, leading to a significant reduction in cumulative
error.

B DETAILS ABOUT OPENPRM TRAINING

B.1 DATASETS FOR PREFERENCE TREE BUILDING

Orig

Dataset

UltraFeedback

Retain Outcome?

As introduced in Section 4.1, we con-
struct preference tree data primarily
using UltraFeedback, which consists
of a mixture of instructions in the
open domain. Additionally, we incor-
porate instructions from the Math and
STEM domains to enhance the gen-
eralization and reasoning capabilities
of open-domain models. Beyond pro-
cess pairs built with preference trees,
we also include some outcome-level
preference pairs to maintain the capa-
bilities of ORMs, which can be seen
as a specific case of PRMs. We provide all the statistics of the datasets used in Table 4.

59,876
N/A
6,508
N/A
7,500
N/A
N/A

N/A
70,068
N/A
12,958
N/A
19,913
7,221

+PTS
ScienceQA
+PTS

+PTS
HelpSteer 2

✓
✗
✓
✗
✓
✗
✓

MATH

Table 4: Statistics of training datasets. PTS is preference
tree sampling strategy proposed in § 3.1.

Process?
✗
✓
✗
✓
✗
✓
✗

Training Data Format. When preparing the training data for the PRM, we reformat all process-
level and outcome-level pairs into a unified format: [Q, C, Pc, Pr], where Pc and Pl represent the
chosen (preferred) and rejected (non-preferred) answers, respectively, based on the same context C.
For outcome-level pairs, [C, Pc] and [C, Pr] represent the complete answers. For process-level pairs,
these concatenations represent partial answers.

B.2 HYPER-PARAMETERS FOR TRAINING

For building preference trees, we compute the edit distance on segments of different response can-
didates, splitting the entire responses using “.\n”. The threshold for segment aggregation is set
at 0.88 for all tasks based on the distribution of similarity as shown in Figure 5, and the rewards
gap threshold between process pairs is 1.0 for UltraFeedback and 0.2 for Math and ScienceQA.
For both Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct, we sample 64 candidate responses for

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

each instruction. Due to constraints on inference cost, we randomly sample 10,000 instructions from
UltraFeedback for the Llama-3.1-70B-Instruct model. We set the temperature to 0.5 and top-p to 1.0
for repeated sampling with vLLM engine 3.

For reproducing the UltraFeedback and HelpSteer2
reward models, we finetune Llama-3-8B-Instruct us-
ing a learning rate 5 × 10−6 over 1 epoch. Mean-
while, we finetune InternRM and FsfairX on process
pairs using a learning rate 1 × 10−6 over 1 epoch.
All models are finetuned with a batch size of 64 and
a maximum sequence length of 2048.

B.3 TREE BUILDING EXAMPLE

As illustrated in Figure 2 (mainly segment and ag-
gregate), we first split each answer into sentences
and then merge similar sentences across all answers
sequentially. For each merging operation, the can-
didate sentences are sourced from the same parent
node and indexed consistently across their respective answers. To clarify, we provide a toy example
along with the corresponding tree for merging three answers from ScienceQA in Table 5. We also
provide an example of a sentence-level preference tree in Figure 13, which includes rewards for
parallel samples at each node.

Figure 5: Distribution of Similarity.

Table 5: Example for similarity-based sentence merging for preference building.

Node(Depth0): “Question:\nWhich logical fallacy is used in the text?\nBefore I refute my
opponent’s argument, I would like to draw attention to the fact that he is sweating and clearly does
not have much experience on the debate team.\nOptions:\nA. ad hominem: an attack against
the person making the argument, rather than the argument itself\nB. bandwagon fallacy: the
assumption that the popular choice is automatically correct\nC. appeal to nature: the assumption
that natural things are always good\nAnswer: Let’s think step by step.” — Reward: [] — Reply
Indices: []

Node(Depth1): “To identify the logical fallacy used in the text, let’s analyze the situation step

by step” — Reward: [0, 1] — Reply Indices: [0, 2]

Similar sentences: “To identify the logical fallacy used in the text, let’s analyze the situation
step by step” ——— “To identify the logical fallacy in the text, let’s break down the reasoning
process step by step”

Node(Depth2): “The text states that the speaker wants to refute their opponent’s argument,
but instead of addressing the argument itself, they choose to comment on their opponent’s phys-
ical appearance (sweating) and a supposed lack of experience on the debate team. This kind of
comment is meant to discredit the opponent rather than engaging with the argument they pre-
sented” — Reward: [0] — Reply Indices: [0]

Node(Depth3): “The key phrase here is ’attack against the person making the argument,
rather than the argument itself,’ which is the definition of the ad hominem fallacy. \n\nTherefore,
the correct answer is:\nA. ad hominem: an attack against the person making the argument, rather
than the argument itself.” — Reward: [0] — Reply Indices: [0]

Node(Depth2): “The text states, ’Before I refute my opponent´s argument, I would like to
draw attention to the fact that he is sweating and clearly does not have much experience on the
debate team.’ This statement does not address the opponent´s argument itself but instead makes
a personal attack on the opponent´s appearance (sweating) and experience, implying that these
characteristics make his argument weak” — Reward: [1] — Reply Indices: [2]

3https://github.com/vllm-project/vllm

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

0.800.850.900.951.00similarity0.02.55.07.510.012.515.017.5number of sentencesDistribution of Similarity Between SentencesUnder review as a conference paper at ICLR 2025

Node(Depth3): “Now, let’s analyze the options:\n\nA. ad hominem (A): This type of
fallacy involves attacking the person making the argument rather than addressing the argument
itself. In this case, the statement attacks the opponent’s person (by mentioning their sweating
and lack of experience) rather than addressing the merits of their argument” — Reward: [1] —
Reply Indices: [2]

Node(Depth4): “B. bandwagon fallacy (B): This fallacy occurs when it is assumed
that something is correct or good because it is popular. There is no indication in the text that
the opponent’s argument is being rejected because it is unpopular, but rather because of personal
characteristics” — Reward: [1] — Reply Indices: [2]

Node(Depth5): “C. appeal to nature (C): This fallacy involves assuming that some-
thing is good or true because it is natural. The text does not mention anything about what is
natural or unnatural, so this option does not apply” — Reward: [1] — Reply Indices: [2]

Node(Depth6): “Given the analysis, the correct answer is A. ad hominem, as the
text attacks the opponent’s person rather than addressing the argument itself.” — Reward: [1] —
Reply Indices: [2]

Node(Depth1): “To identify the logical fallacy in the text, let’s break it down step by
step:\n\n1. **Identify the claim and the counterargument**: The text claims that the oppo-
nent’s argument is flawed but before refuting it, it mentions that the opponent is sweating and
lacks experience on the debate team. This is an attempt to undermine the opponent’s credibility
without addressing the argument itself” — Reward: [1] — Reply Indices: [7]

Node(Depth2): “2. **Analyze the nature of the attack**: The attack is not on the argu-
ment’s merits but on the opponent’s character (sweating, lack of experience). Sweating can be a
sign of nervousness, which isn’t inherently indicative of the validity of one’s argument. Lack of
experience is also not directly related to the quality of the argument unless it’s the first time the
opponent is making an argument, which isn’t specified” — Reward: [1] — Reply Indices: [7]

Node(Depth3): “3. **Classify the fallacy based on the analysis**: The attack is focused
on undermining the opponent personally rather than addressing the argument. This matches
the description of the ”ad hominem” fallacy, which involves attacking the person making an
argument rather than addressing the argument itself” — Reward: [1] — Reply Indices: [7]

Node(Depth4): “Therefore, the logical fallacy used in the text is **ad hominem**”

— Reward: [1] — Reply Indices: [7]

Node(Depth5): “The final answer is: A” — Reward: [1] — Reply Indices: [7]

B.4 RESULTS OF REWARD SHAPE

Figure 6: Rewards on chosen and rejected content of various RMs.

C DETAILS ABOUT EVALUATIONS

We summarize the open-domain datasets used in our evaluations as follows:

• Alpaca Eval 2 (Dubois et al., 2024): A popular benchmark for evaluating instruction-based
language models using automatic evaluators such as GPT-4. It features approximately 800
open-domain prompts. Given the length bias in GPT-4 evaluations, Alpaca Eval 2 employs
length-controlled win-rates.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

1510505Reward Model Score0255075100125FrequencyFsfairX-8BChosenRejected4020020Reward Model ScoreHelpSteer2-8B2024Reward Model ScoreIntern-7B505Reward Model ScoreOpenPRMUnder review as a conference paper at ICLR 2025

• MixEval Hard (Ni et al., 2024): A ground-truth-based dynamic benchmark derived from
established benchmark mixtures. It evaluates LLMs using a highly capable model rank-
ing system. MixEval Hard includes both free-form and multiple-choice questions, each
category containing 500 questions.

• MATH500 (Lightman et al., 2023): A subset of the MATH test dataset from OpenAI,
featuring 12,500 challenging competition mathematics problems. We use the MATH500
version, which contains 500 samples that maintain IID consistency with the original test
dataset, to evaluate the mathematics abilities of LLMs under scaled inference-time settings.
• MMLU-PRO (Wang et al., 2024d): An enhanced benchmark designed to evaluate models
across broader and more challenging tasks. Built upon the MMLU dataset, MMLU-PRO
integrates more challenging, reasoning-focused questions and increases the number of an-
swer choices per question from four to ten, significantly raising the difficulty and reducing
the chance of success through random guessing. We randomly sample 500 questions from
the test data for our evaluations.

• GPQA (Rein et al., 2023): Consists of PhD-level STEM questions generated by experts in
biology, physics, and chemistry. The original GPQA dataset is divided into main, diamond,
and extended parts. We utilize the diamond split to align with OpenAI results, which
includes about 200 questions.

• IFEval (Zhou et al., 2023): Designed to evaluate the instruction-following abilities of chat
models. It focuses on a set of verifiable instructions and includes over 500 prompts with
tasks such as “write an article with more than 800 words” and “wrap your response with
double quotation marks.”

During implementation, we use zero-shot chain-of-thought promtps for MATH and GPQA datasets
based on prompt code in openai simple-evals repository 4. For Alpaca Eval 2, MixEval, IFE-
val and MMLU-Pro dataset, we use the evaluation code from the official GitHub repository 5 6 7.
Specifically, we found that Mistral-Nemo struggles to adhere to answer format instructions; there-
fore, we opted to use few-shot Chain of Thought (CoT) examples instead of zero-shot CoT.

D ADDITIONAL RESULTS

D.1 DIFFERENT AGGREGATION STRATEGIES

For applying PRM to outcome-level pairs, we explored several aggregation strategies for calculating
step-based rewards, as detailed below:

• Last Step: Use the reward of the final step as the overall reward, similar to ORM.
• Max/Min Step: Select the maximum or minimum reward among all steps.
• Simple Average: Calculate the average reward across all steps.
• Weighted Average: Apply a positional weight, giving later steps higher importance. The

formula is: r = 1
N

(cid:80)N

i=1

i
N ri.

• Dynamic Aggregation: Utilize uncertainty-weighted optimization (UWO) (Coste et al.,
2023), which dynamically adjusts weights based on intra-ensemble variance. This penal-
izes policies generating outputs with high disagreement across steps.

• Max/Min Delta: Inspired by recent research, we also compute the delta (difference) be-

tween step rewards and use the maximum or minimum delta as the final reward.

Different Performance on Chat Category. As shown in Table 2 and 7, OpenPRM’s scores in
the Chat category decrease, while performance in the Chat Hard category improves. This reflects
the inherent trade-off between optimizing for simpler conversational tasks (Chat Easy) and more
complex reasoning tasks (Chat Hard), which is a known challenge in reward model optimization, as

4https://github.com/openai/simple-evals/
5https://github.com/tatsu-lab/alpaca_eval
6https://github.com/Psycoy/MixEval
7https://github.com/TIGER-AI-Lab/MMLU-Pro

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Table 6: Overall score of RewardBench (including Chat, Chat Hard, Safety and Reasoning) and
PRM800k Test Results with different aggregation strategies.

Models

Aggregation Overall Chat Chat Hard

Safety Reasoning PRM800k Test

intern2-7b-RM
intern2-7b-RM + FT

ORM
ORM

OpenPRM (intern)

Last Step
Min Step
Max Step
Simple Avg
Weight Avg
Dynamic
Min Delta
Max Delta

87.6
89.6

91.1
91.9
88.7
91.3
91.4
91.6
77.8
77.7

99.2
87.7

98.0
96.7
95.0
97.2
98.0
97.8
64.3
77.4

69.5
84.2

81.6
83.6
80.5
83.3
82.7
83.3
75.7
69.5

87.2
92.4

89.5
91.6
88.2
89.9
89.7
90.4
76.4
74.3

94.5
94.1

95.1
95.7
91.1
94.8
95.0
95.0
95.1
89.5

61.0
63.4

68.1
68.1
68.1
68.1
68.1
68.1
68.1
68.1

also noted in RewardBench (Lambert et al., 2024). The distinction between these categories lies in
their data sources: the Chat category includes datasets like AlpacaEval and MT Bench, while Chat
Hard is derived from MT Bench and LLMBar. The decreased scores in the Chat category are largely
due to AlpacaEval, as there is a distributional shift between this dataset and our training data, which
is sourced from UltraFeedback.

D.2 EVALUATE RMS WITH OFFSETBIAS

In addition to benchmarks like RewardBench, UltraFeedback, and PRM800k, we evaluated our
reward models using OffsetBias (Park et al., 2024), which provides a more granular assessment of
bias in reward models. The results are summarized in the table below, showcasing the effectiveness
of OpenPRM across various bias metrics in Table 7.

Table 7: OffsetBias Evaluation Results

Models

Concreteness

Content/
Continuation

Empty
Reference

Familiar Knowledge
Preference

Length
Bias

Nested
Instruction

Overall

Eurus-RM-7B
FsfairX-LLaMA3-RM
FsfairX-LLaMA3-RM
+ OffsetBias
LLaMA3-8B-Instruct
+ OffsetBias
Intern2-7b-RM
Intern2-7b-RM
+ OpenPRM (Our)

71.4
100

92.9

100

1

92.86

66.7
91.7

100

95.8

1

83.3

84.6
53.8

46.2

92.3

1

1

33.3
91.7

58.3

83.3

58.3

83.3

41.2
41.2

82.4

85.3

58.8

88.2

66.7
58.3

83.3

50

91.7

91.7

60
71.3

77.5

85

84.8

89.9

D.3 COMPARISON WITH MCTS METHODS

We compared the computational efficiency of our method with MCTS-based approaches under sim-
ilar sampling budgets. The experimental setup and results are as follows:

• OpenPRM: We start from 60k samples, with 64 responses per sample, producing approx-
imately 3.84M question-answer pairs in 24 hours. From this, 90k pairs were selected for
training.

• MCTS: We start from 10k samples, with 4 responses per sample. Responses were split into
sentences, resulting in 240k partial outputs. Sampling 8 full paths for each pair of partial
outputs produced 3.84M question-answer pairs in 24 hours. Of these, 97k pairs were used
for training.

The results of PRM trained with OpenPRM and MCTS are shown in Table 8. In fact, the MCTS
method remains a powerful baseline but is significantly more resource-intensive, requiring up to 10
times the computational cost of our approach. With a larger sampling budget, the MCTS method
could still be effectively utilized. However, our method offers a more efficient alternative and can be
further optimized with more accurate similarity computation techniques, such as embedding-based

20

Under review as a conference paper at ICLR 2025

methods. While these approaches may increase the time required to construct the tree, they hold
great potential for improving performance. We plan to explore these optimizations in future work to
further enhance the efficiency and accuracy of our method.

Table 8: Comparison with MCTS methods

Models

Aggregation Overall Chat Chat Hard

Safety Reasoning PRM800k Test

intern2-7b-RM
intern2-7b-RM
+ outcome labels
OpenPRM (intern)
MCTS (intern)

ORM

ORM

Min Step
Min Step

87.6

89.6

91.9
91.4

99.2

87.71

96.7
95.5

69.5

84.21

83.6
81.6

87.2

92.43

91.6
93.2

94.5

94.06

95.7
95.4

61

63.4

68.1
68.2

D.4 RESULTS ON MORE LANGUAGE MODELS

Scaling Effect of PBS. We have analyzed the scaling effects of Process Beam Search (PBS), where
beam search is conducted at the sentence level, selecting the top N generated outputs based on PRM
rewards. As presented in Figure 7, the results indicate that OpenPRM consistently outperforms Bag
of N-grams (BoN) in PBS settings, showcasing its effectiveness and reliability in these scenarios.
However, we also observed significant variance in the scaling effect of PBS across different tasks, in
contrast to the more consistent scaling effect seen with best-of-N methods. This variance can likely
be attributed to differences in the data distributions across tasks, which highlight the need for further
investigation into handling data mixtures effectively. We plan to continue exploring this issue in
future work to better understand and address these challenges.

Figure 7: Scaling Effect of Process Beam Search

Scaling Effect of BoN. We present the remaining tasks, in addition to those shown in Figure 3, in
Figure 8. We provide additional results for the Llama-3.1-70B-Instruct and Mistral-Nemo model
regarding inference-time scaling in Figures 9, 10, 11 and 12, which support the same conclusions as
those drawn from the Llama-3.1-8B-Instruct in Figure 3.

Figure 8: Results of scaling inference-time for Llama-3.1-8B-Instruct on the rest tasks of Figure 3.

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

22242628Number of Samples (N)17.520.022.525.027.530.032.535.037.5Accuracy (%)Llama-3.1-8B-Instruct, GPQAMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMPBS@N,OpenPRM22242628Number of Samples (N)38404244464850Accuracy (%)Llama-3.1-8B-Instruct, MMLU-ProMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMPBS@N,OpenPRM21232527Number of Samples (N)405060708090100Accuracy (%)Llama-3.1-8B-Instruct,MixEval21232527Number of Samples (N)3540455055606570Accuracy (%)Llama-3.1-8B-Instruct,AlpacaEval221232527Number of Samples (N)4041424344454647Accuracy (%)Llama-3.1-8B-Instruct,MixEval21232527Number of Samples (N)34.034.535.035.536.036.5Accuracy (%)Llama-3.1-8B-Instruct,AlpacaEval2CoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMUnder review as a conference paper at ICLR 2025

Figure 9: Results of scaling inference-time for Llama-3.1-70B-Instruct on open-domain tasks.

Figure 10: Results of scaling inference-time for Llama-3.1-70B-Instruct on open-domain tasks.

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

21232527Number of Samples (N)60708090Accuracy (%)Llama-3.1-70B-Instruct,MMLU-Pro21232527Number of Samples (N)5060708090Accuracy (%)Llama-3.1-70B-Instruct,GPQA21232527Number of Samples (N)657075808590Accuracy (%)Llama-3.1-70B-Instruct,MATH21232527Number of Samples (N)60708090100Accuracy (%)Llama-3.1-70B-Instruct,MixEval21232527Number of Samples (N)506070Accuracy (%)Llama-3.1-70B-Instruct,AlpacaEval221232527Number of Samples (N)8688909294Accuracy (%)Llama-3.1-70B-Instruct,IFEvalCoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRM21232527Number of Samples (N)54565860626466Accuracy (%)Llama-3.1-70B-Instruct,MMLU-Pro21232527Number of Samples (N)424446485052Accuracy (%)Llama-3.1-70B-Instruct,GPQA21232527Number of Samples (N)6466687072Accuracy (%)Llama-3.1-70B-Instruct,MATH21232527Number of Samples (N)56586062Accuracy (%)Llama-3.1-70B-Instruct,MixEval21232527Number of Samples (N)42.543.043.544.0Accuracy (%)Llama-3.1-70B-Instruct,AlpacaEval221232527Number of Samples (N)85.586.086.587.087.5Accuracy (%)Llama-3.1-70B-Instruct,IFEvalCoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMUnder review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Figure 11: Results of scaling inference-time for Mistral-Nemo on open-domain tasks.

Figure 12: Results of scaling inference-time for Mistral-Nemo on open-domain tasks.

23

21232527Number of Samples (N)405060708090100Accuracy (%)Mistral-Nemo-Instruct-2407,MMLU-Pro21232527Number of Samples (N)406080100Accuracy (%)Mistral-Nemo-Instruct-2407,GPQA21232527Number of Samples (N)30405060708090Accuracy (%)Mistral-Nemo-Instruct-2407,MATH21232527Number of Samples (N)405060708090Accuracy (%)Mistral-Nemo-Instruct-2407,MixEval21232527Number of Samples (N)4050607080Accuracy (%)Mistral-Nemo-Instruct-2407,AlpacaEval221232527Number of Samples (N)6570758085Accuracy (%)Mistral-Nemo-Instruct-2407,IFEvalCoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRM21232527Number of Samples (N)35.037.540.042.545.047.550.0Accuracy (%)Mistral-Nemo-Instruct-2407,MMLU-Pro21232527Number of Samples (N)32343638Accuracy (%)Mistral-Nemo-Instruct-2407,GPQA21232527Number of Samples (N)303540455055Accuracy (%)Mistral-Nemo-Instruct-2407,MATH21232527Number of Samples (N)34353637Accuracy (%)Mistral-Nemo-Instruct-2407,MixEval21232527Number of Samples (N)41424344Accuracy (%)Mistral-Nemo-Instruct-2407,AlpacaEval221232527Number of Samples (N)64666870Accuracy (%)Mistral-Nemo-Instruct-2407,IFEvalCoverageMarj@NBoN@N,fsFairxBoN@N,HS2BoN@N,UFBoN@N,InternBoN@N,OpenPRMUnder review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Figure 13: Example of sentence-level preference tree for process reward modeling.

24

What adventures did the cat ha...Reward: 0.00RIndex: []The cat woke up early....Reward: 0.98RIndex: [0, 1, 2, 4, 5]Our feline friend began the da...Reward: 1.50RIndex: [3]It stretched lazily on the win...Reward: 0.98RIndex: [0, 1, 2, 4, 5]Then, it spotted a bird outsid...Reward: 0.87RIndex: [0, 1, 2]Around noon, it moved to the l...Reward: 0.90RIndex: [4]In the backyard, it stalked a ...Reward: 1.40RIndex: [5]The cat watched intently for h...Reward: 0.70RIndex: [0, 2]In the evening, it played with...Reward: 1.20RIndex: [1]It decided to take a nap on th...Reward: 0.60RIndex: [2]There, it watched birds throug...Reward: 0.90RIndex: [4]In the afternoon, it played wi...Reward: 0.90RIndex: [4]After an exciting chase, it re...Reward: 1.40RIndex: [5]The cat spent the rest of the ...Reward: 1.40RIndex: [5]It raced around the house....Reward: 1.50RIndex: [3]Then, it climbed to the top of...Reward: 1.50RIndex: [3]From there, it surveyed its do...Reward: 1.50RIndex: [3]Later, it enjoyed some treats....Reward: 1.50RIndex: [3]Finally, it curled up in a sun...Reward: 1.50RIndex: [3]