# Automatic Model Selection with Large Language Models for Reasoning

James Xu Zhao1, Yuxi Xie1, Kenji Kawaguchi1, Junxian He2, Michael Qizhe Xie1
1 National University of Singapore 2 The Hong Kong University of Science and
Technology
<EMAIL_ADDRESS>
###### Abstract

Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two
distinct reasoning methods, each with its own strengths. CoT employs natural
language, offering flexibility and interpretability, while PAL utilizes
programming language, yielding more structured and rigorous logic. We
introduce a model selection method to combine the best of both worlds by
employing a large language model (LLM) to dynamically select between them. Our
theoretical analysis underscores the feasibility of this method, which is
further corroborated by empirical results. Our proposed method demonstrates
significant performance improvements across eight reasoning datasets with
Codex, ChatGPT, and GPT-4. Additionally, our method is complementary to self-
consistency; when integrated, it can further enhance performance while
significantly reducing computation costs. Moreover, we achieve new state-of-
the-art results on GSM8K and SVAMP, with respective accuracies of 96.8% and
93.7%.111Our code, data and prompts are available at https://
github.com/XuZhao0/Model-Selection-Reasoning

##  1 Introduction

Figure 1: We propose to perform model selection to combine two distinct
methods, CoT and PAL. The figure illustrates an example where PAL makes
mistakes about crucial information and therefore fails to answer the question
correctly. In contrast, CoT manages to correctly answer the same question. Our
selection model successfully chooses the correct solution and provides a brief
explanation to support its choice.

Large language models (LLMs) have made impressive progresses in numerous
fields (Devlin et al., 2019; Brown et al., 2020; OpenAI, 2023; Chowdhery et
al., 2022; Bubeck et al., 2023; Wei et al., 2022a) and are often powerful
enough to solve problems through a single unified method. While convenient,
this approach tends to ignore the distinct structures and variations among the
problems, which would benefit from using different methods. On the other hand,
in human society, individuals with different skill sets excel in various
roles, leading to a thriving world as a whole.

In the case of reasoning, Chain-of-Thought (CoT) Wei et al. (2022b) and
Program-Aided Language Models (PAL) (Gao et al., 2022; Chen et al., 2022) have
emerged as two effective methods that offer different strengths and
weaknesses. Essentially, CoT decomposes a reasoning problem into a series of
intermediate steps using natural language, making it more general, flexible,
and comprehensible. On the other hand, PAL offers solutions via Python
functions, with its step-by-step programming code ensuring a more rigorous and
structured methodology. The external Python interpreter further guarantees the
computation precision. Intuitively, combining the strengths of both models and
selecting between them based on their solutions to a specific problem would
lead to enhanced performance. However, without access to the ground truth,
choosing a better method itself becomes a machine learning problem.

In order to select among multiple solutions, previous studies have suggested
training a ranker (Uesato et al., 2022). While training a dedicated model
generally results in improved accuracy, it can also be somewhat cumbersome and
entail significant costs. Conversely, large language models (LLMs) have
demonstrated good calibration and have been used to assess the accuracy of
their own outputs (Guo et al., 2017; Shinn et al., 2023; Xie et al., 2023). In
light of this, we propose leveraging the in-context learning capabilities of
LLMs for model selection. We direct LLMs to choose between two distinct
reasoning models and elucidate their selection rationale.

We present a detailed theoretical analysis to validate our approach. The
analysis highlights two primary factors impacting the effectiveness of our
method: (1) the significance of the difference between the two models, i.e.
the distinct distributions of their respective probabilities of correctly
solving problems, and (2) the probability of selecting the correct model.
Particularly, a higher overall performance can be attained when there is a
substantial difference between the models being considered, and when there is
a high probability of the correct model being selected. Furthermore, our
analysis affirms that even without an exceptional model selector, we can still
achieve improvement in certain cases. This reinforces our decision to simply
employ an LLM for model selection.

We evaluate our method across eight reasoning tasks, with CoT and PAL serving
as the baseline methods. Our method consistently attains performance
improvement, when employing Codex (Chen et al., 2021), ChatGPT, and GPT-4 as
backbone LLMs. Its broad applicability is further underscored by performance
gains with open-source Llama 2 models (Touvron et al., 2023). In the context
of multiple sample aggregations, our approach significantly reduces the
computation costs while achieving notable performance enhancements. Moreover,
our approach attains new state-of-the-art accuracies of 96.8% and 93.7% on
GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021), respectively.

##  2 Automatic Model Selection with Large Language Models

In this study, we examine reasoning tasks using two baseline models: CoT and
PAL. To tackle complex reasoning tasks, CoT leverages an LLM to generate
several intermediate reasoning steps before arriving at a final answer. Due to
its reliance on natural language, the reasoning steps are clearly explained.
Furthermore, natural language supports a broad range of reasoning that may
involve common sense and confidence in the reasoning steps. But it might
struggle with intricate logical relationships. In contrast, PAL takes a
structured and accurate approach, breaking down reasoning problems using
Python code. However, the deterministic program format of PAL constrains its
generalizability and flexibility. We provide a detailed comparison of these
models in Section 2.1, drawing insights from instances where their results
differ.

###  2.1 Case Study on the Differences between CoT and PAL

Figure 2: Comparative analysis of error cases between CoT and PAL on GSM8K.

To delve deeper into the distinct strengths of CoT and PAL, we analyze 100
instances from GSM8K where CoT and PAL yield different results: 50 cases where
CoT is correct and PAL is wrong, and another 50 where the opposite is true.
All the CoT and PAL solutions are generated using ChatGPT. Then we divide the
error reasons into 5 categories. Examples of each category can be found in
Appendix E.

  * •

Computation Precision: The ability to perform accurate numerical computations.

  * •

Logical Decomposition: Breaking down complex problems into manageable parts
and handling logical relationships well.

  * •

Problem-Solving Flexibility: The adaptability in addressing problems using
different reasoning paradigms, such as forward and backward reasoning.

  * •

Semantic Understanding: Grasping and interpreting the problem accurately
without overlooking crucial information.

  * •

Others: This encompasses miscellaneous error reasons such as redundant
calculations and ambiguous questions.

Figure 2 depicts the error distribution across these categories. It reveals
notable differences between CoT and PAL. PAL, with its external Python
interpreter, ensures computation accuracy. It also excels in logical
decomposition, effectively breaking down problems and employing variables to
map logical relationships. On the other hand, CoT is more versatile and
flexible, allowing for both forward and backward logical reasoning, while PAL
is less adept at backward reasoning scenarios. CoT’s strength also lies in its
superior grasp of natural language, aiding in semantic understanding. This
analysis highlights the distinct proficiencies of both models in reasoning,
reinforcing our hypothesis.

###  2.2 Method

Given the distinct advantages of CoT and PAL, it would be beneficial to
combine the strengths of both. We propose a systematic approach that combines
these models through model selection. Specifically, it contains two stages:
solution generation and model selection.

For a given reasoning question $Q$, an LLM is prompted to generate reasoning
chains for CoT, symbolized as $C_{CoT}$ along with its answer $A_{CoT}$.
Simultaneously, the LLM is prompted to derive reasoning chains for PAL,
denoted as $C_{PAL}$. Employing a Python interpreter, we then get its answer
$A_{PAL}$. When $A_{CoT}$ and $A_{PAL}$ yield different results, we move to
the model selection phase. The LLM is prompted again, provided with
$(Q,C_{CoT},A_{CoT},C_{PAL})$, and tasked to select the better method,
alongside generating a brief explanation. Specifically for Llama 2 selection
prompts, the LLM is prompted with $(Q,C_{CoT},A_{CoT},C_{PAL},A_{PAL})$ pairs.
This selection result, together with the explanation, is represented as $S$.
Ultimately, based on $S$, the final answer is derived as
$A_{final}\in\\{A_{CoT},A_{PAL}\\}$.

During the selection phase, LLMs leverage their in-context learning abilities
for model selection, i.e., we present the LLM with a few in-context exemplar
pairs $(Q,C_{CoT},A_{CoT},C_{PAL},S)$. An example of the proposed method is
illustrated in Figure 1, excluding the few-shot examples to save space.
Specifically, we provide the LLM with two distinct reasoning solutions in a
multiple-choice question format. We expect the LLM to identify errors or gaps
within the chains and determine the correct method. Note that there are very
few instances where the LLM fails to make a choice; in these cases, we
randomly select a method. The effect and qualitative analysis of the generated
explanation can be found in Appendix C.

Owing to the in-context learning capabilities of LLMs, we find that they
exhibit reasonable accuracy in selecting the appropriate method. Furthermore,
as our subsequent theoretical analysis in Section 2.3 reveals, a highly
accurate method selection is not actually necessary for the algorithm to
perform well. Our empirical findings corroborate this, showing that even if
there are situations where the model selection is poor, the overall
improvement remains substantial.

###  2.3 Theoretical Analysis

In this section, we conduct a theoretical analysis to determine under which
condition the proposed method could work (and fail).

#### Quantifying error rates

Let us denote the error rates of the two base methods, $m_{1}$ and $m_{2}$, by
$err_{1}$ and $err_{2}$, respectively. Without loss of generality, let $m_{1}$
be a better base method in the overall performance: i.e., $err_{1}\leq
err_{2}$. For a given question $x$, we define $\rho_{x}$ as the probability of
choosing a more accurate method, either $m_{1}$ or $m_{2}$, for the given $x$
using the proposed approach. Define $R(x)=p(\mathrm{correct}\mid
x,m_{2})-p(\mathrm{correct}\mid x,m_{1})$ where $p(\mathrm{correct}\mid
x,m_{i})$ represents the probability of outputting a correct prediction given
input $x$ with method $m_{i}$. Then we can quantify the final error rate $err$
of the proposed method as follows:

######  Proposition 1.

For any methods $m_{1}$ and $m_{2}$ with any combining probability function
$\rho_{x}$,

$err=err_{1}-\mathbb{E}_{x}\left[|R(x)|\left(\rho_{x}-\mathbbm{1}\\{R(x)<0\\}\right)\right]$

We refer readers to Appendix B for the full proof. Proposition 1 decomposes
the possible improvement (or deterioration) over base methods in terms of
$R(x)$ and $\rho_{x}$. It quantitatively shows when and how we can expect
improvement (or deterioration) based on these two factors. For example, to
improve over the best base method $m_{1}$, Proposition 1 suggests us to choose
another base method $m_{2}$ such that $|R(x)|$ is not too small and $\rho_{x}$
is high when $|R(x)|$ is large. In other words, it discourages us from
choosing too similar methods as base methods, because for similar methods,
$|R(x)|$ tends to be small and it is challenging to increase $\rho_{x}$ even
when $|R(x)|$ is large due to the similarity. This provides a theoretical
motivation for us to use CoT and PAL, instead of combining CoT with another
CoT.

#### On the accuracy of selection

Define $\rho$ to be the overall probability of selecting a better method:
$\rho=\mathbb{E}_{x}[\rho_{x}]$. Theorem 1 shows that $\rho$ can be much worse
than that of a random guess to achieve the improvement over the base methods
$m_{1}$ and $m_{2}$; i.e., $\mathrm{err}<\mathrm{err}_{1}$ and
$\mathrm{err}_{1}\leq\mathrm{err}_{2}$ can happen with $\rho<0.5$:

######  Theorem 1.

For any $\epsilon>0$, there exist data distribution over $x$, two base methods
($m_{1},m_{2}$), and combining probability ($\rho_{x}$) such that
$\mathrm{err}<\mathrm{err}_{1}$, $\mathrm{err}_{1}\leq\mathrm{err}_{2}$, and
$\rho<\epsilon$.

We provide a stronger version of Theorem 1 in Appendix A (that implies Theorem
1) and its proof in Appendix B.

Theorem 1 supports our proposition that, despite not training a new model for
the selection process and with the in-context learning limited to a few-shot
prompt, it is possible to achieve improvement, even if we do not achieve
$\rho_{x}>0.5$ in some instances. This theoretical analysis offers deeper
insights into the conditions and strategies for effective performance
improvement with the proposed methodology.

##  3 Experiments

###  3.1 Setup

Datasets and backbones We conduct experiments on eight datasets that span a
range of arithmetic and symbolic reasoning tasks. 7 of these datasets,
including GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), ASDIV (Miao
et al., 2020), SingleOP, SingleEQ, AddSub and MultiArith (Koncel-Kedziorski et
al., 2016), are about arithmetic reasoning, while Date Understanding
(Srivastava et al., 2022) focuses on symbolic reasoning. To comprehensively
evaluate the effectiveness of our approach, we employ three proprietary LLMs
as backbone systems: Codex (code-davinci-002), ChatGPT (gpt-3.5-turbo-0301)
and GPT-4 (gpt-4-0314). We also evaluate our method with open-source Llama 2
models on GSM8K.222Codex results are obtained in February and March, ChatGPT
in April and May, GPT-4 in May and June 9-13, Llama 2 in September 2023.
Unless stated otherwise, we always utilize the same LLM for both base models
as well as for the model selector.

Prompt design To effectively exploit the in-context learning abilities of the
LLMs, we create a set of few-shot examples by manually creating an error in
one model’s reasoning chain. Few-shot examples in our prompts are based on
those in previous works. For each backbone LLM, we use a fixed prompt for all
the arithmetic benchmarks without specific customizations. Examples of the
prompts can be found in Appendix D.

Hyperparameters For the results derived using greedy decoding, we use a
temperature of 0 during both solution generation and model selection. For
experiments with self-consistency in Section 3.2 and 3.3, we follow the
settings in prior works by setting the temperature at 0.5 for CoT and 0.8 for
PAL during solution generation. During model selection, the temperature is set
to 0 for a deterministic result. We refrain from further hyperparameter tuning
to emphasize the simplicity and efficacy of our method.

Backbone | Method | Arithmetic | Symbolic
---|---|---|---
GSM8K | SVAMP | ASDIV | SingleOP | SingleEQ | AddSub | MultiArith | Date
Codex | CoT | 64.4 | 77.6 | 80.2 | 92.7 | 93.5 | 88.4 | 95.7 | 64.5
PAL | 71.5 | 79.6 | 79.1 | 95.4 | 96.5 | 91.9 | 99.7 | 77.5
Ours | 74.7 | 82.2 | 81.6 | 96.3 | 96.9 | 91.6 | 99.7 | 79.4
ChatGPT | CoT | 80.8 | 83 | 89.3 | 94.8 | 97.4 | 90.4 | 98.7 | 69.1
PAL | 79.2 | 80.3 | 83 | 90.7 | 97.6 | 89.4 | 96.3 | 68.3
Ours | 82.6 | 84.3 | 89.4 | 94.8 | 97.8 | 90.6 | 98.7 | 70.2
GPT-4 | CoT | 94.6 | 91.9 | 92.7 | 97.2 | 97.2 | 93.9 | 98 | 90
PAL | 94.0 | 92.2 | 90.2 | 95.2 | 98.8 | 94.9 | 98.5 | 88.1
Ours | 95.6 | 93.7 | 93.5 | 97.3 | 98.6 | 95.7 | 99 | 90.5
Table 1: Results comparison (Accuracy %) on 7 arithmetic datasets and 1 symbolic dataset with greedy decoding. We evaluate our methods on Codex, ChatGPT, and GPT-4. The best results are highlighted in bold. Backbone | Method | $\mathrm{SC@5}$ | $\mathrm{SC@15}$
---|---|---|---
ChatGPT | CoT | 85.4 | 87.4
PAL | 80.9 | 82.4
Ours |  88.2 (+2.8) |  89.2 (+1.8)
GPT-4 | CoT | 95.6 | 95.8
PAL | 94.7 | 95.5
Ours |  96.5 (+0.9) |  96.8 (+1.0)
Table 2: Results comparison (Accuracy %) on GSM8K with the integration of the
Self-Consistency (SC). $\mathrm{SC@5}$ and $\mathrm{SC@15}$ represents $5$ and
$15$ sampled paths respectively. The previous state-of-the-art on GSM8K is
95.5%, achieved by Zheng et al. (2023a).

###  3.2 Main Results

The results of our experiments with greedy decoding are shown in Table 1.
First, we find that our proposed method effectively and robustly enhances
performance in most settings across various datasets of different difficulties
and with different backbone LLMs, simply by combining two base models. For
example, with GPT-4, we achieve an accuracy of 95.6% on GSM8K and 93.7% on
SVAMP without self-consistency.

Second, our results show a considerable improvement even when one of the base
models performs much worse than the other. For instance, we observe a
significant 3.2% improvement over PAL’s 71.5% accuracy on GSM8K, even though
CoT has a lower accuracy of 64.4%.

Third, our model’s general applicability is further underscored by its 1.9%
improvement on the symbolic date understanding task when utilizing Codex as
the backbone. In fact, even if the accuracy difference between two base models
is as large as 13% on this task, our proposed method still improves the
accuracy from 77.5% to 79.4%. Additionally, our method also achieves
respectable performance gains with both ChatGPT and GPT-4 on this symbolic
reasoning task. It serves as evidence of our method’s applicability to a
spectrum of reasoning tasks, extending beyond mathematical reasoning.

Method | Acc. | Cost | # Tokens | # Generated
---|---|---|---|---
CoT@15 | 87.4 | 33.69 | 21.56M | 2.72M
PAL@15 | 82.4 | 41.64 | 27.13M | 1.90M
Ours@5 | 88.2 | 29.29 | 18.99M | 1.61M
CoT@40 | 88.3 | 89.84 | 57.32M | 7.08M
PAL@40 | 83.5 | 111.04 | 72.35M | 5.07M
Ours@15 | 89.2 | 87.87 | 56.97M | 4.83M
CoT@80 | 88.2 | 179.68 | 115.04M | 14.48M
PAL@80 | 83.3 | 222.08 | 144.64M | 10.16M
Ours@15 | 89.5 | 117.16 | 75.96M | 6.44M
Table 3: Computation cost comparison with ChatGPT on GSM8K (1319 data samples). CoT@15 represents CoT with self-consistency (SC) with 15 sampled paths. Cost in USD is calculated based on OpenAI Pricing444https://openai.com/pricing. # Tokens denotes the total number of tokens, including inputs and generation tokens. # Generated denotes the number of generated tokens.  Baseline LLM | Selector LLM | $\mathrm{A}cc_{\mathrm{CoT}}$ | $\mathrm{A}cc_{\mathrm{PAL}}$ | Ours | Improvement
---|---|---|---|---|---
Llama 2 7B | Llama 2 7B | 15.0 | 13.7 | 16.2 | +1.2
Llama 2 7B | Llama 2 13B | 15.0 | 13.7 | 16.8 | +1.8
Llama 2 13B | Llama 2 13B | 29.0 | 33.3 | 35.3 | +2.0
Table 4: Results with Llama 2 7B/13B on a subset of GSM8K. The reference CoT
accuracies in the Llama 2 paper (Touvron et al., 2023) with Llama 2 7B and 13B
are 14.6% and 28.7% respectively. Baseline LLM refers to the backbone LLM used
for both CoT and PAL solution generation. Selector LLM refers to the backbone
LLM used for model selection.

#### Experiments with self-consistency

We aim to investigate the relationship between self-consistency (Wang et al.,
2022b) and model selection with multiple samples, and whether they complement
each other. The results of CoT and PAL with self-consistency are obtained by
sampling multiple times with the temperature at 0.5 and 0.8 respectively. For
our method with self-consistency, we execute our algorithm repeatedly, using
the hyperparameter settings specified in Section 3.1. Subsequently, we
aggregate results across all samples through majority voting, arriving at the
final answer. As demonstrated in Table 2, we achieve substantial improvements
over the self-consistency results of both CoT and PAL.

Employing ChatGPT, we attain a pronounced performance improvement with only 5
samples, reflecting a 2.8% leap in accuracy. With GPT-4, even though both base
models already score around 95%, integrating them with our method leads to a
96.5% accuracy on GSM8K. Furthermore, we establish the new state-of-the-art
results on GSM8K at 96.8% using 15 samples.

###  3.3 Computation Cost Comparison

We extend our analysis to compare the computation costs of model selection
with multiple samples versus using self-consistency alone. As depicted in
Table 4, our method not only showcases notable performance improvement but
also significantly reduces computation costs, underscoring its efficacy and
efficiency.

Our method consistently outperforms both CoT and PAL with SC in terms of
accuracy while demanding less token computation and cost. For instance, using
only 5 paths, our method achieves comparable performance to CoT@80 while only
costs 29 USD, which is merely 16 % of expense associated with CoT@80.
Moreover, when aggregating 15 samples, our method surpasses the performance of
CoT@80 at only half the cost.

Furthermore, our method also elevates the performance ceilings. While the
result of CoT with SC plateaus around 88.2% and that of PAL with SC saturates
around 83.3%, our method breaks through these ceilings, achieving an
impressive 89.5% accuracy using just 20 samples. This result underscores the
power and potential of integrating different reasoning models.

###  3.4 Experiments with Llama 2 Models

To explore the broad applicability of our method, we conduct experiments with
open-source Llama 2 7B/13B models using a subset of the GSM8K, comprising 600
data samples. We generate CoT and PAL solutions with Llama 2 7B/13B
models555https://huggingface.co/meta-llama/Llama-2-7b-hf
https://huggingface.co/meta-llama/Llama-2-13b-hf. For model selection, we use
Llama 2 7B/13B Chat models666 https://huggingface.co/meta-
llama/Llama-2-7b-chat-hf https://huggingface.co/meta-llama/Llama-2-13b-chat-
hf. The experiments are carried out by using greedy decoding without tuning
hyperparameters.

As shown in Table 4, our method consistently yields performance enhancements
when applied to Llama 2 models. Specifically, compared to the baseline results
on the 13B model, our method achieves respectable 2% performance gains,
improving accuracy from 33.3% to 35.3%. On the smaller 7B model, we observe
the performance improvement from 15.0% to 16.2% when utilizing the 7B model as
the selector. Moreover, when employing the 13B model as the selector, the
performance is further improved to 16.8%. These results affirm the broad
applicability of our method.

##  4 Analysis

In this section, we provide a few analyses to see when and how our method
works.

Backbone | Metric | Arithmetic | Symbolic
---|---|---|---
GSM8K | SVAMP | ASDIV | SingleOP | SingleEQ | AddSub | MultiArith | Date
Codex | $\Delta_{\mathrm{UpperBound}}$ | 10 | 8.1 | 6.5 | 1.6 | 0.9 | 2.0 | 0.3 | 4.6
Success Rate | 74.8 | 72.5 | 63.7 | 87.9 | 88 | 70 | 92.9 | 87.8
Improvement | +3.2 | +2.6 | +1.4 | +0.9 | +0.4 | -0.3 | 0 | +1.9
ChatGPT | $\Delta_{\mathrm{UpperBound}}$ | 8.6 | 6.0 | 3.4 | 2.4 | 1.4 | 2.5 | 0.6 | 9.8
Success Rate | 60.4 | 66.4 | 69.8 | 58 | 57.1 | 60.9 | 75 | 53.6
Improvement | +1.8 | +1.3 | +0.1 | 0 | +0.4 | +0.2 | 0 | +1.1
GPT-4 | $\Delta_{\mathrm{UpperBound}}$ | 2.5 | 3.6 | 2.2 | 2.4 | 0.6 | 1.3 | 0.5 | 1.1
Success Rate | 72.6 | 68.7 | 64.2 | 57 | 69.2 | 85.7 | 100 | 86.7
Improvement | +1.0 | +1.8 | +0.8 | +0.1 | -0.2 | +0.8 | +0.5 | +0.5
Table 5: We define the following terms: $\Delta_{\mathrm{UpperBound}}=\mathrm{Acc}_{\mathrm{Upper\ Bound}}-\mathrm{Acc}_{m_{1}}$, where $\mathrm{Acc}_{\mathrm{Upper\ Bound}}$ is the upper bound accuracy where we assume a perfect model selection and ${m_{1}}$ is the stronger one of the two base models. $\Delta_{\mathrm{UpperBound}}$ reflects the expected performance difference between the two base models. Success Rate calculates the correct selection rates when either CoT is correct or PAL is correct, i.e., we ignore the cases where both methods are either correct or wrong. Improvement is the performance improvement achieved over the performance of the stronger base model. Backbone | ChatGPT | GPT-4
---|---|---
$m_{1}$ | CoT | CoT
$m_{2}$ | PAL | $\mathrm{CoT}^{\prime}$ | PAL | CCoT
$\mathrm{A}cc_{m_{1}}$ | 80.8 | 80.8 | 94.6 | 94.6
$\mathrm{A}cc_{m_{2}}$ | 79.2 | 79.2 | 94.0 | 95.1
Ours | 82.6 | 80.8 | 95.6 | 95.2
Improvement | (+1.8) | (+0) | (+1.0) | (+0.1)
$\Delta_{\mathrm{Upper\ Bound}}$ | 8.6 | 7.5 | 2.5 | 1.7
Success Rate | 60.4 | 52.2 | 72.6 | 58.8
Table 6: Other model combinations results on GSM8K. $\mathrm{CoT}^{\prime}$
denotes the base CoT model with the temperature of $0.1$. CCoT denotes
ComplexCoT (Fu et al., 2022).

###  4.1 Influencing Factors

To better understand the reasons for the performance improvement across
various datasets and backbone LLMs, we present the performance improvement,
and the associated influencing factors in Table 5.

As demonstrated in our theoretical analysis, the performance improvement is
jointly determined by the difference between the two combined methods,
$|R(x)|$, and the effectiveness of the model selection, $\rho$. The results
from Table 5 showcase a high expected performance difference between CoT and
PAL, and decent selection accuracy with all backbone LLMs, which clarifies the
significant performance gains across the datasets.

Firstly, the $\Delta_{\mathrm{Upper\ Bound}}$ of CoT and PAL reflects how
differently the two base models behave across questions. A larger
$\Delta_{\mathrm{Upper\ Bound}}$ signifies a larger room for potential
improvement. Specifically, on GSM8K with ChatGPT, $\Delta_{\mathrm{Upper\
Bound}}$ stands at 8.6%, even though the accuracies of CoT and PAL are closely
matched at 80.8% and 79.2%, respectively. Similarly, with GPT-4,
$\Delta_{\mathrm{Upper\ Bound}}$ is 2.5% while the accuracies of the two base
models are close (94.6% vs 94.0%).

Secondly, across the eight datasets with varied complexities, each of the
three LLMs exhibits commendable selection accuracy. Notably, both Codex and
GPT-4 show a relatively high success selection rate. For example, their
success rates on GSM8K surpass 70%, and on the simpler MultiArith dataset, the
selection accuracies exceed 90%.

Furthermore, as detailed in Theorem 1, performance improvements are attainable
even without a high success selection rate. Our empirical results corroborate
it. For instance, on the date understanding task employing ChatGPT, despite a
modest success rate of 53.6%, we still achieve 1.1% performance enhancement
with a large $\Delta_{\mathrm{Upper\ Bound}}$ of 9.8%.

CoT LLM | PAL LLM | Selector LLM | $\mathrm{A}cc_{\mathrm{CoT}}$ | $\mathrm{A}cc_{\mathrm{PAL}}$ | Ours | Success Rate | $\Delta_{\mathrm{UpperBound}}$
---|---|---|---|---|---|---|---
ChatGPT | ChatGPT | ChatGPT | 80.8 | 79.2 |  82.6 (+1.6) | 60.4 | 8.6
ChatGPT | Codex | ChatGPT | 80.8 | 71.5 |  81.2 (+0.4) | 70.6 | 7.0
ChatGPT | Codex | GPT-4 | 80.8 | 71.5 |  84.4 (+3.6) | 84.8 | 7.0
Table 7: Results with the combination of different backbone LLMs on GSM8K. CoT LLM refers to the backbone LLM used for CoT solution generation. Metric | Codex | ChatGPT | GPT-4
---|---|---|---
CoT-PAL | PAL-CoT | CoT-PAL | PAL-CoT | CoT-PAL | PAL-CoT
$\mathrm{A}cc_{\mathrm{CoT}}$ | 64.4 | 64.4 | 80.8 | 80.8 | 94.6 | 94.6
$\mathrm{A}cc_{\mathrm{PAL}}$ | 71.5 | 71.5 | 79.2 | 79.2 | 94.0 | 94.0
Ours | 69.9(-1.6) | 74.7(+3.2) | 82.6(+1.8) | 81.6(+0.8) | 95.6(+1.0) | 95.1(+0.5)
Success Rate | 56.7 | 75 | 60.4 | 54.6 | 72.6 | 63
$\mathrm{Ratio_{CoT}}$ | 71.9 | 17.3 | 89.9 | 79.7 | 60.3 | 53.4
$\mathrm{Ratio_{PAL}}$ | 28.1 | 82.7 | 10.1 | 20.3 | 39.7 | 46.6
Table 8: The effect of option order for different backbones on GSM8K. CoT-PAL
represents CoT and PAL being placed as choice A and B respectively.
$\mathrm{Ratio_{CoT}}$ indicates the ratio of CoT selected out of all the
selections. As stated previously, during analysis, we ignore the cases where
both CoT and PAL are either correct or wrong.

###  4.2 Combination between Similar Methods

We choose CoT and PAL as our two base models due to the motivation of
combining different strengths of distinct models. We conduct experiments to
examine whether the performance improves when we combine two similar base
models. We use two variants of CoT: $\mathrm{CoT}^{\prime}$ where we set the
temperature at $0.1$, ComplexCoT (Fu et al., 2022) where we use more complex
few-shot examples in the prompt. Both of these methods’ accuracies are similar
or higher than the accuracy of PAL. The results are shown in Table 6.

From Table 6, we can find that model selection between CoT and
$\mathrm{CoT^{\prime}}$, or CoT and ComplexCoT, does not lead to substantial
performance gains, even though the accuracy of $\mathrm{CoT^{\prime}}$ and
ComplexCoT is on par with PAL. On the other hand, model selection between CoT
and PAL results in consistent performance improvements. To understand the
reasons behind these outcomes, we further investigate the
$\Delta_{\mathrm{Upper\ Bound}}$ and the success selection rate.

Firstly, $\Delta_{\mathrm{Upper\ Bound}}$ of CoT-PAL exceeds that of other
combinations, CoT-$\mathrm{CoT}^{\prime}$ and ComplexCoT-CoT, despite their
employing two stronger or equivalent two base models. This observation
suggests a larger absolute value of the accuracy difference per question for
CoT-PAL. It indicates that CoT and PAL perform more dissimilarly than other
model combinations, which represents a larger $|R(x)|$. As Proposition 1
highlights, without a substantial $|R(x)|$, it is unlikely to achieve
significant performance gain since the improvement component is factored by
$|R(x)|$.

Secondly, the success selection rate of CoT-PAL surpasses that of other model
combinations. It means that the selector is more likely to select the correct
choice when one solution derives from CoT and the other from PAL. In theory,
this higher success rate implies that when $|R(x)|$ is high for a given
question $x$, the success selection probability $\rho_{x}$ for CoT-PAL is
higher than others.

These findings support our initial motivation and hypothesis. We choose CoT
and PAL as our two base models because they represent distinct reasoning
approaches using natural language and programming language. We expect these
models to exhibit a significant difference in errors and accuracies, indicated
by a high $\Delta_{\mathrm{Upper\ Bound}}$. Moreover, we posit that the
considerable difference in errors for a particular question makes it easier
for LLMs to select the correct option, leading to a higher success rate
compared to selecting between two similar base models like
CoT-$\mathrm{CoT^{\prime}}$. This holds true even when different prompts or
temperature settings are used.

###  4.3 Combination between Different Backbone LLMs

Considering that Codex is optimized for code generation and ChatGPT performs
better in natural language reasoning, we embark on experiments integrating
various backbone LLMs, to harness the strengths of distinct backbone LLMs and
reasoning models.

As shown in Table 7, we achieve a 0.4% performance improvement with ChatGPT as
the selector, despite a notable 9.3% accuracy disparity between PAL from Codex
and CoT from ChatGPT. More impressively, when utilizing GPT-4 as the model
selector, we get a significant 3.6% performance boost, with an 84.8% success
selection rate. These results hint the potential benefits when integrating
multiple models tailored for specialized domains or tasks.

###  4.4 The Effect of Option Order

To understand the influence of option order in a multiple-choice question
format, we perform experiments by modifying the order of CoT and PAL options.
The results in Table 8 show a performance impact from simply changing the
choice order. We attribute this to the inherent bias within the backbone LLMs.
A similar phenomenon is reported by Zheng et al. (2023b).

Each of the three backbones seems to exhibit a preference for the first
option. Particularly noticeable on Codex, in which when CoT is positioned
first, it is selected in 71.9% of all selections. However,
$\mathrm{Ratio_{CoT}}$ significantly drops to 17.3% when putting CoT second.
ChatGPT shows a preference for natural language reasoning as it always selects
CoT solutions more frequently regardless of its position in the options.
Moreover, when CoT is the second choice, $\mathrm{Ratio_{CoT}}$ decreases from
89.9% to 79.7%, which we speculate demonstrates its preference for the first
option. GPT-4 has similar biases akin to ChatGPT, though it appears fairer.
The option order also affects the success selection rate and further
influences the overall performance.

While Zhao et al. (2021) attempts to address the inherent bias through
probability calibration, the task of mitigating the bias in the absence of
underlying token probabilities remains a challenge. Therefore, despite the
wide use of LLMs in data annotation and task evaluation(Xu et al., 2023; Wang
et al., 2023; Chiang et al., 2023), addressing and eliminating inherent biases
is a critical area that deserves further exploration.

##  5 Related Work

Ensemble learning. In machine learning, the strategy of combining various
models to address a single problem is exemplified in techniques such as
bagging (Breiman, 1996), boosting (Freund and Schapire, 1997; Chen and
Guestrin, 2016; Ke et al., 2017), and random forest (Ho, 1995; Breiman, 2001).
The underlying idea in these methods is that a group of weak learners can
collectively manifest as a strong learner. This concept has also found its
place in deep learning through the use of ensembles. For reasoning, self-
consistency samples diverse reasoning paths and chooses the most consistent
answer through majority voting (Wang et al., 2022b). Wang et al. (2022a) takes
it a step further by introducing rationale-augmented ensembles, emphasizing
rationale sampling in the output space. However, typically, ensembling places
equal weights on models through majority voting, which may restrict the full
potential of the diverse strengths that each model offers.

Reasoning. The research community has made tremendous progress in the field of
reasoning. Apart from CoT (Wei et al., 2022b) and PAL (Gao et al., 2022; Chen
et al., 2022), Zhou et al. (2022) simplifies complex problems by breaking them
down into a series of sub-problems. Kojima et al. (2022) shows that by simply
adding "Let’s think step by step" before each answer, LLMs can be competent
zero-shot reasoners. Creswell et al. (2022) alternates between selection and
inference stages, generating causal reasoning steps to the final answer.
Kazemi et al. (2022) proposes a backward chaining algorithm that breaks
reasoning down into four sub-models. Paul et al. (2023); Xie et al. (2023);
Yao et al. (2023) refine intermediate reasoning steps by leveraging another
critic model or the self-evaluation ability. Recently, Zhou et al. (2023)
utilizes GPT-4 Code Interpreter and code-based-self-verification, achieving a
97% accuracy on GSM8K, but their work is concurrent with ours. More
importantly, the contributions of these methods are distinct from our
approach, and the progress made by them could potentially be seamlessly
integrated using our method.

Self-Evaluation. LLM calibration studies reveal that the probabilistic
predictions made by current LLMs closely align with the actual frequencies of
token occurrences, hence producing well-calibrated predictions for certain
tasks (Guo et al., 2017; Kadavath et al., 2022; Jiang et al., 2020). As LLMs
exhibit reliable calibration, there is a growing number of research
emphasizing the use of self-evaluation for verification. Shinn et al. (2023)
proposes an approach to provide an agent with dynamic memory and self-
reflection capabilities. Madaan et al. (2023) proposes a method to generate
outputs from LLMs and refine its previously generated output given its own
feedback. Different from these works where the underlying method is the same,
in this work, we are interested in combining systems with different strengths
and weaknesses through self-evaluation model selection.

##  6 Conclusion

We introduce a method that effectively combines two distinct base models, CoT
and PAL, by enabling an LLM to make the selection. We provide a theoretical
analysis that supports the feasibility of such a model combination, which is
further validated by empirical results. Our method achieves performance
improvements across eight datasets with various backbone LLMs. We reduce the
computation cost while attaining notable performance improvement with self-
consistency. We set new SOTA performance on GSM8K and SVAMP. This research
represents a significant step towards tapping into the potential of diversity
and collaboration among models in LLMs.

In our future work, we aim to expand this framework to more reasoning tasks
and possibly other domains. An intriguing concept involves exploring the use
of diverse system instructions to elicit varying model behaviors for model
combinations. Furthermore, a genuine combination of specialized models, each
with its strengths and expertise, offers a promising avenue for exploration.

##  7 Limitation

This work focuses on reasoning tasks, but we believe that extending the
exploration of model selection into other domains could yield valuable
insights. Further, due to the inherent bias of LLMs, our method is affected by
the order of options. We encourage the community to explore solutions for bias
mitigation. We also anticipate subsequent studies to identify more robust and
potent model selectors. Finally, our combination strategy currently comprises
only the CoT and PAL base models. Future research could investigate the
incorporation of more base models with distinct strengths, with the aim of
creating a more powerful synthesis that leverages their unique advantages.

## References

  * Breiman (1996) Leo Breiman. 1996.  Bagging predictors.  _Machine learning_ , 24:123–140.
  * Breiman (2001) Leo Breiman. 2001.  Random forests.  _Machine learning_ , 45:5–32.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _ArXiv_ , abs/2303.12712.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pages 785–794.
  * Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _ArXiv_ , abs/2211.12588.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _ArXiv_ , abs/2204.02311.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _ArXiv_ , abs/2110.14168.
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _ArXiv_ , abs/2205.09712.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Freund and Schapire (1997) Yoav Freund and Robert E Schapire. 1997.  A decision-theoretic generalization of on-line learning and an application to boosting.  _Journal of computer and system sciences_ , 55(1):119–139.
  * Fu et al. (2022) Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022.  Complexity-based prompting for multi-step reasoning.  _ArXiv_ , abs/2210.00720.
  * Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022.  Pal: Program-aided language models.  _ArXiv_ , abs/2211.10435.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _International Conference on Machine Learning_.
  * Ho (1995) Tin Kam Ho. 1995.  Random decision forests.  In _Proceedings of 3rd international conference on document analysis and recognition_ , volume 1, pages 278–282. IEEE.
  * Jiang et al. (2020) Zhengbao Jiang, J. Araki, Haibo Ding, and Graham Neubig. 2020.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.  _ArXiv_ , abs/2207.05221.
  * Kazemi et al. (2022) Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xinyuan Xu, and Deepak Ramachandran. 2022.  Lambada: Backward chaining for automated reasoning in natural language.  _ArXiv_ , abs/2212.13894.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ , 30.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _ArXiv_ , abs/2205.11916.
  * Koncel-Kedziorski et al. (2016) Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016.  Mawps: A math word problem repository.  In _North American Chapter of the Association for Computational Linguistics_.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  _ArXiv_ , abs/2303.17651.
  * Miao et al. (2020) Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020.  A diverse corpus for evaluating and developing english math word problem solvers.  _ArXiv_ , abs/2106.15772.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Patel et al. (2021) Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?  In _North American Chapter of the Association for Computational Linguistics_.
  * Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023.  Refiner: Reasoning feedback on intermediate representations.  _ArXiv_ , abs/2304.01904.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _ArXiv_ , abs/2303.11366.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C’esar Ferri Ram’irez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz’alez, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan J. Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fatemeh Siar, Fernando Mart’inez-Plumed, Francesca Happ’e, François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-L’opez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik-Soon Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco’n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg, Jos Rozen, José Hernández-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng’ Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col’on, Luke Metz, Lutfi Kerem cSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, M’aty’as Schubert, Medina Baitemirova, Melissa Arnaud, Melvin Andrew McElrath, Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pi-Bei Hwang, P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ram’on Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, S. Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, T. N. Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler O’Brien Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, W Vossen, Xiang Ren, Xiaoyu Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song, Yasaman Bahri, Ye Ji Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yuntao Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang, Zijie J. Wang, Zirui Wang, Ziyi Wu, Sahib Singh, and Uri Shaham. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _ArXiv_ , abs/2206.04615.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv_ , abs/2307.09288.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, L. Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022.  Solving math word problems with process- and outcome-based feedback.  _ArXiv_ , abs/2211.14275.
  * Wang et al. (2022a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022a.  Rationale-augmented ensembles in language models.  _ArXiv_ , abs/2207.00747.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022b.  Self-consistency improves chain of thought reasoning in language models.  _ArXiv_ , abs/2203.11171.
  * Wang et al. (2023) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xingxu Xie, Wei Ye, Shi-Bo Zhang, and Yue Zhang. 2023.  Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization.  _ArXiv_ , abs/2306.05087.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _Trans. Mach. Learn. Res._ , 2022.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022b.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023.  Decomposition enhances reasoning via self-evaluation guided decoding.  _ArXiv_ , abs/2305.00633.
  * Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _ArXiv_ , abs/2304.01196.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _ArXiv_ , abs/2305.10601.
  * Zhao et al. (2021) Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_.
  * Zheng et al. (2023a) Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023a.  Progressive-hint prompting improves reasoning in large language models.  _ArXiv_ , abs/2304.09797.
  * Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. 2023b.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _ArXiv_ , abs/2306.05685.
  * Zhou et al. (2023) Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023.  Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.  _ArXiv_ , abs/2308.07921.
  * Zhou et al. (2022) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _ArXiv_ , abs/2205.10625.

##  Appendix A A detailed version of Theorem 1

In this appendix, we provide a detailed version of Theorem 1. Whereas Theorem
1 only states the existence of problem instances, Theorem 2 constructs such
instances concretely: i.e., Theorem 2 implies Theorem 1. Define
$\mu_{x}[\mathcal{X}]$ to be the distribution for the expected errors: i.e.,
an expected error can be written by
$\mathbb{E}_{x\sim\mu_{x}[\mathcal{X}],y,f}[\mathbbm{1}[y\neq f(x)]]$ for some
function $f$. Define $S[\mathcal{X}]=\\{x\in\mathcal{X}:R(x)<0\\}$. Let us
denote $U[\mathcal{X}]$ as the uniform distribution over $\mathcal{X}$. Given
any $\mathcal{X}$, we write $n=|\mathcal{X}|$, $T=|S[\mathcal{X}]|$,
$\alpha=T/n$. Assume that $1\leq T<n$.

######  Theorem 2.

Let $\mu_{x}[\mathcal{X}]=U[\mathcal{X}]$ and $\mathcal{X}$ be given such that
$|\mathcal{X}|<\infty$. Let $\epsilon,\delta\in(0,1)$ and $\lambda\in(0,1]$
such that $\beta=\frac{\epsilon T}{n-T}\in(0,1)$ and $\lambda\geq
1-\frac{\beta}{\epsilon T}(n-T-\delta)$. Let $R$ and $\rho_{x}$ be set such
that $R(x)=-\epsilon$ for $x\in S[\mathcal{X}]$, $R(x)=\beta$ for
$x\in\mathcal{X}\setminus S[\mathcal{X}]$, $(1/T)\sum_{x\in
S[\mathcal{X}]}\rho_{x}=\lambda$, and $(1/(n-T))\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}\rho_{x}=\epsilon(T/(n-T))(1-\lambda)\beta^{-1}+\delta/(n-T)$.
Then, we have that $\mathrm{err}<\mathrm{err}_{1}$,
$\mathrm{err}_{1}\leq\mathrm{err}_{2}$, and

$\rho=1-\alpha+\lambda[2\alpha-1]+\frac{\delta}{n}.$

In particular, when $\alpha\geq 0.5$, we have $\rho\rightarrow 0$ as
$\alpha\rightarrow 1$ and $(\delta/(n-T))\rightarrow 0$ (with
$\lambda=1-\frac{\beta}{T\epsilon}(n-T-\delta)$); when $\alpha<0.5$, we have
$\rho\rightarrow 0$ as $\alpha\rightarrow 0$ and $(\delta/n)\rightarrow 0$
(with $\lambda=1$).

The proof of Theorem 2 is presented in Appendix B. Theorem 2 shows that the
overall success probability of the selection process can be much worse than a
random guess to achieve the improvement over the base methods $m_{1}$ and
$m_{2}$; i.e., $\mathrm{err}<\mathrm{err}_{1}$ and
$\mathrm{err}_{1}\leq\mathrm{err}_{2}$ can happen with $\rho<0.5$. Indeed, it
is possible to have $\rho\rightarrow 0$ with the improvement
($\mathrm{err}<\mathrm{err}_{1}$ and $\mathrm{err}_{1}\leq\mathrm{err}_{2}$)
when the size of $\mathcal{X}$ is large: when $\alpha\geq 0.5$, we can choose
$\lambda=1-\frac{\beta}{T\epsilon}(n-T-\delta)$ with which
$\mathrm{err}<\mathrm{err}_{1}$, $\mathrm{err}_{1}\leq\mathrm{err}_{2}$, and
$\rho\rightarrow 0$ as $\alpha\rightarrow 1$ and $(\delta/(n-T))\rightarrow
0$. When $\alpha<0.5$, we can choose $\lambda=1$ with which
$\mathrm{err}<\mathrm{err}_{1}$, $\mathrm{err}_{1}\leq\mathrm{err}_{2}$, and
$\rho\rightarrow 0$ as $\alpha\rightarrow 0$ and $(\delta/n)\rightarrow 0$.
This supports our proposition that, despite not training a new model for the
selection process and with the in-context learning limited to a few-shot
prompt, it is possible to achieve improvement, even if we do not achieve
$\rho_{x}>0.5$ in some instances.

Theorem 2 also suggests that if the overall performance of two base methods is
similar, captured by $\epsilon$, the overall selection process can be weak to
achieve some improvement, as long as the success selection probability is
relatively high when the two methods have very different expected errors (or
accuracies) for a given question. In essence, Theorem 2 suggests a trade-off:
we want $|R(x)|$ to be larger when deciding which two base methods $m_{1}$ and
$m_{2}$ to choose, implying that we prefer base methods to perform
dissimilarly on $\mathcal{X}$. On the other hand, if two base methods exhibit
a substantial expected accuracy difference, then the selection process needs
to be stronger to improve the performance (i.e., $\rho$ needs to be larger).
However, if the expected accuracy difference between the two base methods is
relatively small, increasing the power of the selection process is not that
necessary to boost performance.

##  Appendix B Proofs

###  B.1 Proof of Proposition 1

###### Proof.

Define $\mathrm{acc}=1-\mathrm{err}$ and $\mathrm{acc}_{i}=1-\mathrm{err}_{i}$
for $i\in\\{1,2\\}$. Since $\text{expected
error}=\mathbb{E}[\mathbbm{1}[\text{incorrect prediction}]]=P(\text{incorrect
prediction})=1-P(\text{correct prediction})$, we have that

$\mathrm{acc}_{i}=p(\mathrm{correct}|m_{i})=\mathbb{E}_{x}[p(\mathrm{correct}|x,m_{i})]$

where $\mathrm{correct}$ represents the event of the correct prediction.
Similarly,

$\mathrm{acc}=\mathbb{E}_{x}\left[\sum_{i=1}^{2}p(m_{i}|x)p(\mathrm{correct}|x,m_{i})\right]$

where $p(m_{i}|x)$ represents the probability of selecting method $m_{i}$
given $x$ via the proposed method. Thus,

$\displaystyle\mathrm{acc}-\mathrm{acc}_{1}$
$\displaystyle=\mathbb{E}_{x}\left[p(m_{1}|x)p(\mathrm{correct}|x,m_{1})+p(m_{2}|x)p(\mathrm{correct}|x,m_{2})-p(\mathrm{correct}|x,m_{1})\right]$
$\displaystyle=\mathbb{E}_{x}\left[(p(m_{1}|x)-1)p(\mathrm{correct}|x,m_{1})+p(m_{2}|x)p(\mathrm{correct}|x,m_{2})\right]$
$\displaystyle=\mathbb{E}_{x}\left[(p(m_{2}|x)p(\mathrm{correct}|x,m_{2})-(1-p(m_{1}|x))p(\mathrm{correct}|x,m_{1})\right]$

Since $1-p(m_{1}|x)=p(m_{2}|x)$,

$\displaystyle\mathrm{acc}-\mathrm{acc}_{1}$
$\displaystyle=\mathbb{E}_{x}\left[(p(m_{2}|x)p(\mathrm{correct}|x,m_{2})-p(m_{2}|x)p(\mathrm{correct}|x,m_{1})\right]$
$\displaystyle=\mathbb{E}_{x}\left[(p(m_{2}|x)R(x)\right].$

Here, we notice that

$\displaystyle p(m_{2}|x)$ $\displaystyle=\begin{cases}\rho_{x}&\text{ if
}R(x)\geq 0\\\ 1-\rho_{x}&\text{ if }R(x)<0\\\ \end{cases}$
$\displaystyle=\mathbbm{1}\\{R(x)\geq
0\\}\rho_{x}+\mathbbm{1}\\{R(x)<0\\}(1-\rho_{x}).$

By plugging this into the above equation,

$\displaystyle\mathrm{acc}-\mathrm{acc}_{1}$
$\displaystyle=\mathbb{E}_{x}\left[(\mathbbm{1}\\{R(x)\geq
0\\}\rho_{x}+\mathbbm{1}\\{R(x)<0\\}(1-\rho_{x}))R(x)\right]$
$\displaystyle=\mathbb{E}_{x}[R(x)\mathbbm{1}\\{R(x)\geq
0\\}\rho_{x}]+\mathbb{E}_{x}[R(x)\mathbbm{1}\\{R(x)<0\\}(1-\rho_{x})]$

Since $R(x)\mathbbm{1}\\{R(x)\geq 0\\}=|R(x)|\mathbbm{1}\\{R(x)\geq 0\\}$ and
$R(x)\mathbbm{1}\\{R(x)<0\\}=-|R(x)|\mathbbm{1}\\{R(x)<0\\}$, we have that

$\displaystyle\mathrm{acc}-\mathrm{acc}_{1}$
$\displaystyle=\mathbb{E}_{x}[|R(x)|\mathbbm{1}\\{R(x)\geq
0\\}\rho_{x}]-\mathbb{E}_{x}[|R(x)|\mathbbm{1}\\{R(x)<0\\}(1-\rho_{x})]$
$\displaystyle=\mathbb{E}_{x}[|R(x)|((\mathbbm{1}\\{R(x)\geq
0\\}+\mathbbm{1}\\{R(x)<0\\})\rho_{x}-\mathbbm{1}\\{R(x)<0\\})]$

Since $(\mathbbm{1}\\{R(x)\geq 0\\}+\mathbbm{1}\\{R(x)<0\\})=1$ for any $x$,

$\displaystyle\mathrm{acc}-\mathrm{acc}_{1}=\mathbb{E}_{x}[|R(x)|(\rho_{x}-\mathbbm{1}\\{R(x)<0\\})].$

∎

###  B.2 Proof of Theorem 2

###### Proof.

We first confirm that $R(x)$ and $\rho_{x}$ define valid probabilities under
the condition of this statement. For $R(x)$, since $\epsilon\in(0,1)$ and
$\beta\in(0,1)$, it defines valid probabilities for methods $m_{1}$ and
$m_{2}$. For $\rho_{x}$, since $\lambda\in[0,1]$, it also defines valid
probabilities for the case of $x\in S[\mathcal{X}]$. For the case of
$x\in\mathcal{X}\setminus S[\mathcal{X}]$, since
$\epsilon(T/(n-T))(1-\lambda)\beta^{-1}+\delta/n\geq 0$, we need to show that
$\epsilon(T/(n-T))(1-\lambda)\beta^{-1}+\delta/n\leq 1$. That is,

$\displaystyle\epsilon(T/(n-T))(1-\lambda)\beta^{-1}+\delta/(n-T)\leq 1$
$\displaystyle\Longleftrightarrow$ $\displaystyle\epsilon
T(1-\lambda)\beta^{-1}\leq n-T-\delta$ $\displaystyle\Longleftrightarrow$
$\displaystyle 1-\frac{\beta(n-T-\delta)}{\epsilon T}\leq\lambda,$

which is satisfied by the condition on $\lambda$ that $\lambda\geq
1-\frac{\beta}{\epsilon T}(n-T-\delta)$. Thus, the condition on $\rho_{x}$
defines the valid probabilities for both cases of $x\in S[\mathcal{X}]$ and
$x\in\mathcal{X}\setminus S[\mathcal{X}]$.

We now show that $\mathrm{err}<\mathrm{err}_{1}$. Invoking Proposition 1,

$\mathrm{err}=\mathrm{err}_{1}-\mathbb{E}_{x}\left[|R(x)|\left(\rho_{x}-\mathbbm{1}\\{R(x)<0\\}\right)\right].$

Thus, we have $\mathrm{err}<\mathrm{err}_{1}$ if
$\mathbb{E}_{x}\left[|R(x)|\left(\rho_{x}-\mathbbm{1}\\{R(x)<0\\}\right)\right]>0$.
This condition can be rewritten as

$\displaystyle\mathbb{E}_{x}\left[|R(x)|\left(\rho_{x}-\mathbbm{1}\\{R(x)<0\\}\right)\right]>0$
$\displaystyle\Longleftrightarrow\ $
$\displaystyle\frac{1}{n}\sum_{x\in\mathcal{X}}\left[|R(x)|\left(\rho_{x}-\mathbbm{1}\\{R(x)<0\\}\right)\right]>0$
$\displaystyle\Longleftrightarrow\ $
$\displaystyle\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}|R(x)|\rho_{x}+\sum_{x\in
S[\mathcal{X}]}|R(x)|\rho_{x}>\sum_{x\in S[\mathcal{X}]}|R(x)|$
$\displaystyle\Longleftrightarrow\ $
$\displaystyle\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}|R(x)|\rho_{x}+\epsilon T\lambda>\epsilon T$
$\displaystyle\Longleftrightarrow$
$\displaystyle\beta\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}\rho_{x}>\epsilon T-\epsilon T\lambda=\epsilon T(1-\lambda)$
$\displaystyle\Longleftrightarrow$
$\displaystyle\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}\rho_{x}>\frac{\epsilon T(1-\lambda)}{\beta}$

This is satisfied by the condition on $\rho$ that
$(1/(n-T))\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}\rho_{x}=\epsilon(T/(n-T))(1-\lambda)\beta^{-1}+\delta/(n-T)$
for some $\delta>0$: i.e.,

$\sum_{x\in\mathcal{X}\setminus S[\mathcal{X}]}\rho_{x}=\frac{\epsilon
T(1-\lambda)}{\beta}+\delta.$

Therefore, we have that $\mathrm{err}<\mathrm{err}_{1}$.

We now show that $\mathrm{err}_{1}\leq\mathrm{err}_{2}$. Similarly to the
proof of Proposition 1, we define $\mathrm{acc}_{i}=1-\mathrm{err}_{i}$ for
$i\in\\{1,2\\}$. Then, the inequality $\mathrm{err}_{1}\leq\mathrm{err}_{2}$
holds if $\mathrm{acc}_{1}\geq\mathrm{acc}_{2}$. By using $\mathrm{correct}$
to represent the event of the correct prediction, this condition can be
rewritten as

$\displaystyle\mathrm{acc}_{1}\geq\mathrm{acc}_{2}$
$\displaystyle\Longleftrightarrow$
$\displaystyle\sum_{x\in\mathcal{X}}p(\mathrm{correct}|x,m_{1})]\geq\sum_{x\in\mathcal{X}}p(\mathrm{correct}|x,m_{2})]$
$\displaystyle\Longleftrightarrow$ $\displaystyle
0\geq\sum_{x\in\mathcal{X}}R(x)=(n-T)\beta-\epsilon T$
$\displaystyle\Longleftrightarrow$ $\displaystyle\frac{\epsilon
T}{n-T}\geq\beta$

This is satisfied by $\beta=\frac{\epsilon T}{n-T}$. Thus, we have that
$\mathrm{err}_{1}\leq\mathrm{err}_{2}$.

Using these, we now compute the $\rho$ as

$\displaystyle\rho$
$\displaystyle=\mathbb{E}_{x\sim\mu_{x}[\mathcal{X}]}[\rho_{x}]$
$\displaystyle=\frac{1}{n}\sum_{x\in\mathcal{X}\setminus
S[\mathcal{X}]}\rho_{x}+\frac{1}{n}\sum_{x\in S[\mathcal{X}]}\rho_{x}$
$\displaystyle=\frac{1}{n}\left(\frac{\epsilon
T(1-\lambda)}{\beta}+\delta\right)+\alpha\lambda$
$\displaystyle=\frac{\epsilon T(1-\lambda)}{\beta
n}+\alpha\lambda+\frac{\delta}{n}$ $\displaystyle=\frac{\epsilon
T(1-\lambda)(n-T)}{\epsilon Tn}+\alpha\lambda+\frac{\delta}{n}$
$\displaystyle=(1-\lambda)-\alpha(1-\lambda)+\alpha\lambda+\frac{\delta}{n}$
$\displaystyle=1-\alpha+\lambda[2\alpha-1]+\frac{\delta}{n}.$

Finally, we prove the asymptotic behavior using this equation. When
$\alpha<0.5$, by setting $\lambda=1$, we have that

$\rho=1-\alpha+\lambda[2\alpha-1]+\frac{\delta}{n}=1-\alpha+2\alpha-1+\frac{\delta}{n}=\alpha+\frac{\delta}{n}\rightarrow
0$

as $\alpha\rightarrow 0$ and $(\delta/n)\rightarrow 0$. When $\alpha\geq 0.5$,
by setting $\lambda=1-\frac{\beta}{T\epsilon}(n-T-\delta)$, we have that

$\displaystyle\rho$
$\displaystyle=1-\alpha+\left(1-\frac{\beta}{T\epsilon}(n-T-\delta)\right)[2\alpha-1]+\frac{\delta}{n}$
$\displaystyle=1-\alpha+2\alpha-1-[2\alpha-1]\frac{\beta}{T\epsilon}(n-T-\delta)+\frac{\delta}{n}$
$\displaystyle=\alpha-[2\alpha-1]\frac{\beta}{T\epsilon}(n-T-\delta)+\frac{\delta}{n}$

By defining $Q=\frac{\beta}{T\epsilon}(n-T-\delta)$, we have

$\rho=\alpha-[2\alpha-1]Q+\frac{\delta}{n}.$

Here,

$Q=\frac{\beta}{T\epsilon}(n-T-\delta)=\frac{\epsilon
T}{n-T}\frac{1}{T\epsilon}(n-T-\delta)=\frac{1}{n-T}(n-T-\delta)=1-\frac{\delta}{n-T}$

Thus,

$\displaystyle\rho$
$\displaystyle=\alpha-[2\alpha-1]\left(1-\frac{\delta}{n-T}\right)+\frac{\delta}{n}$
$\displaystyle=\alpha-2\alpha+1+\delta\frac{2\alpha-1}{n-T}+\frac{\delta}{n}$
$\displaystyle=1-\alpha+\delta\left(\frac{2\alpha-1}{n-T}+\frac{1}{n}\right)\rightarrow
0$

as $\alpha=T/n\rightarrow 1$ and $(\delta/(n-T))\rightarrow 0$: e.g., by
setting $\delta=\zeta(n-T)$ and take $\zeta\rightarrow 0$, with which
$(\delta/(n-T))=\zeta\rightarrow 0$. ∎

##  Appendix C The Effect and Qualitative Analysis of Explanation

Backbone | $\mathrm{A}cc_{\mathrm{CoT}}$ | $\mathrm{A}cc_{\mathrm{PAL}}$ | Explanation? | Ours Acc. |  | Success
---
Rate
Codex | 64.4 | 71.5 | w/o exp |  74.7 (+3.2) | 74.9
64.4 | 71.5 | w/ exp | 74.6 (+3.1) | 74.2
ChatGPT | 80.8 | 79.2 | w/o exp | 81.8 (+1.0) | 55.9
80.8 | 79.2 | w/ exp |  82.6 (+1.8) | 60.4
GPT-4 | 94.6 | 94.0 | w/o exp | 95.5 (+0.9) | 69.9
94.6 | 94.0 | w/ exp |  95.6 (+1.0) | 72.6
Table 9: Accuracy and success selection rate with and without explanation on
GSM8K.

To perform model selection, we provide explanations in the few-shot examples
and ask the LLMs to generate explanations after making a choice, as we expect
to improve the model’s selection ability by pointing out why the other choice
is incorrect. To investigate the potential role of explanations in enhancing
the model’s selection capability, we conduct experiments on GSM8K by excluding
explanations from the answer.

The results in Table 9 reveal that for backbone LLMs with more powerful in-
context learning abilities, such as Codex and GPT-4, the inclusion of
explanations within the answer seems to have negligible influence on the
model’s performance. In other words, Codex and GPT-4 have a relatively high
success selection rate even when explanations are not given. However, for
ChatGPT, which possesses relatively low few-shot in-context learning
capability, the integration of explanations yields a larger improvement in the
success selection rate. This observation suggests that for backbone LLMs with
weaker in-context learning abilities, explanations can serve as a valuable
tool to improve the success selection rate.

#### Qualitative studies on the explanations

We are also interested in gaining an intuitive understanding of the algorithm,
so we conduct some qualitative analysis on the selection explanations. As
displayed in Table 10, in the first example, both ChatGPT and GPT-4 can
accurately identify that the PAL solution is correct. However, ChatGPT
provides an incorrect explanation, while GPT-4 offers a valid one. In the
second example, which is relatively simpler, both ChatGPT and GPT-4
successfully perform accurate model selection and provide valid explanations.
In the first example, we can see that GPT-4 actually possesses exceptional
reasoning capabilities and provides reliable explanations.

Question: Courtney attended a concert and reported that the audience was 48 in
number. However, Kelly went to the same concert and said that Courtney had
made the mistake of overstating the number of people in attendance by 20%. If
Kelly was right, how many people really attended the concert?
---
(A)
Answer:
Courtney reported 48 people in attendance.
If this number is overstated by 20%, then we need to find the actual number of
people.
First, we need to find 20% of 48.
20% of 48 is (20/100) * 48 = 9.6.
Now, we subtract this number from Courtney’s reported number to find the
actual number of people.
48 - 9.6 = 38.4.
Since we cannot have a fraction of a person, we round the number to the
nearest whole number.
So, the actual number of people in attendance was approximately 38.
So the answer is 38. |  (B) ⬇ def solution(): courtney_report = 48 overstatement_percentage = 0.20 actual_attendance = courtney_report /(1 + overstatement_percentage) result = int(actual_attendance) return result
ChatGPT: (B) can correctly answer the math problem. Because (A) rounds up the
result to the nearest whole number, which is not necessary.
GPT-4: (B) can correctly answer the math problem. Because (A) incorrectly
calculated the actual number of people by subtracting the overstatement
percentage from the reported number instead of dividing it.
Question: How much does it cost you for lunch today at Subway if you pay $40
for a foot-long fish sub and thrice as much for a six-inch cold-cut combo sub?
(A)

Answer:
If the foot-long fish sub costs $40, then the six-inch cold-cut combo sub
costs 3 * $40 = $120.
So the total cost for both subs is $40 + $120 = $160.
Therefore, it costs $160 for lunch today at Subway. |  (B) ⬇ def solution(): cost_footlong_fish = 40 cost_sixinch_coldcut = cost_footlong_fish / 3 total_cost = cost_footlong_fish + cost_sixinch_coldcut result = total_cost return result
ChatGPT: (A) can correctly answer the math problem. Because (B) calculates the
cost of the six-inch cold-cut combo sub as one-third of the cost of the foot-
long fish sub instead of three times the cost of the foot-long fish sub.
GPT-4: (A) can correctly answer the math problem. Because (B) incorrectly
calculates the cost of the six-inch cold-cut combo sub as one-third of the
foot-long fish sub instead of thrice as much.
Table 10: In the first example, ChatGPT performs model selection correctly,
but gives a wrong explanation, while GPT-4 gives a correct explanation. The
second example shows where both ChatGPT and GPT-4 select correctly and give
the correct explanation.

##  Appendix D Examples of Prompts

We show examples of model selection prompts used on different tasks with
different backbones. For arithmetic reasoning tasks, we employ an 8-shot
example for both Codex and Llama 2, and a 5-shot example for ChatGPT and
GPT-4. For the date understanding task, we use 6-shot examples for Codex,
ChatGPT, and GPT-4. We only show a few examples for each case. Full prompts
can be found in our code.

Math Problem: Olivia has $23. She bought five bagels for $3 each. How much
money does she have left?
---
Question: Which of the following two choices can correctly answer the math
problem?
(A) ⬇ def solution(): money_initial = 23 bagels = 5 bagel_cost = 3 money_spent = bagels * bagel_cost money_left = money_initial - money_spent result = money_left return result |  (B)

Answer:
Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 * 3 = 15 dollars.
So she has 23 - 5 = 18 dollars left.
The answer is 18.
Answer: (A)
Table 11: An example of 8-shot model selection prompts used on 7 arithmetic
datasets with Codex. Date Understanding Problem: 2015 is coming in 36 hours.
What is the date one week from today in MM/DD/YYYY?
---
Question: Which of the following two choices can correctly answer the date
understanding problem?
(A) ⬇ def solution(): # If 2015 is coming in 36 hours, then today is 36 hours before. today = datetime(2015, 1, 1) - relativedelta(hours=36) # One week from today, one_week_from_today = today + relativedelta(weeks=1) # The answer formatted with %m/%d/%Y is result = one_week_from_today.strftime(’%m/%d/%Y’) return result |  (B)

A:
If 2015 is coming in 36 hours, then it is coming in 2 days.
2 days before 01/01/2015 is 12/30/2014, so today is 12/30/2014.
So one week from today will be 01/06/2015.
So the answer is 01/06/2015.
Answer: (A)
Table 12: An example of 6-shot model selection prompts used on Date
Understanding task with Codex. System: You are a helpful assistant that can
identify the correct answer to the math problem.
---
There are two choices to the same math problem. One uses natural language to
answer the question, while the other uses Python program to answer it. Either
of them can correctly answer the math problem. You need to identify which
choice can correctly answer the math problem. Here is one example how to do
it,
Math problem: Olivia has $23. She bought five bagels for $3 each. How much
money does she have left?
(A)

Answer:
Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 * 3 = 15 dollars.
So she has 23 - 15 = 8 dollars left.
So the answer is 8.
|  (B) ⬇ def solution(): money_initial = 23 bagels = 5 bagel_cost = 3
money_spent = bagels + bagel_cost money_left = money_initial - money_spent
result = money_left return result
Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) adds the number of
bagels to the cost of each bagel instead of multiplying them.
Now it’s your turn. Here is another math problem and two choices.
Math Problem: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On
wednesday, he lost 2 more. How many golf balls did he have at the end of
wednesday?
(A)

Answer:
Michael started with 58 golf balls.
Then after losing 23 on tuesday, he had 58 -23 = 35.
After losing 2 more, he had 35 + 2 = 37 golf balls.
So the answer is 37.
|  (B) ⬇ def solution(): golf_balls_initial = 58 golf_balls_lost_tuesday = 23
golf_balls_lost_wednesday = 2 golf_balls_left = golf_balls_initial - \
golf_balls_lost_tuesday - \golf_balls_lost_wednesday result = golf_balls_left
return result
Which of the above two choices can correctly answer the math problem?
(B) can correctly answer the math problem. Because (A) adds 2 more balls after
losing 2 more on Wednesday instead of subtracting them.
Table 13: Two examples of 5-shot model selection prompts used on 7 arithmetic
datasets with ChatGPT. System: You are a helpful assistant that can identify
the correct answer to the math problem.
---
There are two choices to the same math problem. One uses natural language to
answer the question, while the other uses Python code to answer it. Either of
them can correctly answer the math problem. You need to identify which choice
can correctly answer the math problem. Here is one example how to do it,
Math problem: There were nine computers in the server room. Five more
computers were installed each day, from monday to thursday. How many computers
are now in the server room?
(A)

Answer:
There were originally 9 computers.
For each of 4 days from monday to thursday, 5 more computers were added.
So 5 * 4 = 20 computers were added.
So there are 9 + 20 = 29 computers now.
So the answer is 29.
|  (B) ⬇ def solution(): computers_initial = 9 computers_added = 5
computers_total = computers_initial + computers_added result = computers_total
return result
Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) missed the fact that
computers were added each day from monday to thursday.
Now it’s your turn. Here is another math problem and two choices.
Math Problem: A piece of square paper has a perimeter of 32 centimeters.
Nicky’s dog, Rocky, tore off 1/4 of the paper. What is the area of the
remaining paper?
(A)

Answer:
A square has 4 equal sides.
The perimeter of the square paper is 32 centimeters.
So each side of the square is 32 / 4 = 8 centimeters.
The area of the whole square paper is side * side = 8 * 8 = 64 square
centimeters.
Rocky tore off 1/4 of the paper.
So The area of the remaining paper is 1/4 * 64 = 16 square centimeters.
So the answer is 16.
|  (B) ⬇ def solution(): perimeter = 32 fraction_torn = 1 / 4 area_total =
(perimeter / 4) ** 2 area_remaining = (1 - fraction_torn) * area_total result
= area_remaining return result
Which of the above two choices can correctly answer the math problem?
(B) can correctly answer the math problem. Because (A) incorrectly calculated
the area of the torn-off portion instead of the remaining portion.
Table 14: Two examples of 5-shot model selection prompts used on 7 arithmetic
datasets with GPT-4. System: You are a helpful assistant that can identify the
correct answer to the date understanding problem.
---
There are two choices to the same date understanding problem. One uses natural
language to answer the question, while the other uses Python program to answer
it. Either of them can correctly answer the date understanding problem. You
need to identify which choice can correctly answer the problem. Here is one
example how to do it,
Date Understanding Problem: 2015 is coming in 36 hours. What is the date one
week from today in MM/DD/YYYY?
(A)

Answer:
If 2015 is coming in 36 hours, then it is coming in 2 days.
And 2 days before 01/01/2015 is 12/30/2014, so today is 12/30/2014.
So one week from today will be 01/05/2015.
So the answer is 01/05/2015.
|  (B) ⬇ def solution(): # If 2015 is coming in 36 hours, then today is 36
hours before. today = datetime(2015, 1, 1) + relativedelta(hours=36) # One
week from today, one_week_from_today = today + relativedelta(weeks=1) # The
answer formatted with %m/%d/%Y is result =
one_week_from_today.strftime(’%m/%d/%Y’) return result
Which of the above two choices can correctly answer the date understanding
problem?
(A) can correctly answer the date understanding problem. Because (B)
incorrectly calculates the date 36 hours later instead of 36 hours before.
Now it’s your turn. Here is another date understanding problem and two
choices.
Date Understanding Problem: The first day of 2019 is a Tuesday, and today is
the first Monday of 2019. What is the date today in MM/DD/YYYY?
(A)

Answer:
If the first day of 2019 was Tuesday, then 01/01/2019 was a Tuesday.
And today is the first monday, would be 5 days later.
So today is 01/06/2019.
So the answer is 01/06/2019.
|  (B) ⬇ def solution(): # If the first day of 2019 is a Tuesday, and today is
the first Monday of 2019, then today is 6 days later. today = datetime(2019,
1, 1) + relativedelta(days=6) # The answer formatted with %m/%d/%Y is
result=today.strftime(’%m/%d/%Y’) return result
Which of the above two choices can correctly answer the date understanding
problem?
(B) can correctly answer the problem. Because (A) missed the fact that there
are 6 days between the first day of 2019 and the first Monday of 2019.
Table 15: Two examples of 6-shot model selection prompts used on Date
Understanding with ChatGPT and GPT-4. System: You are a helpful and brilliant
assistant. You are a math expert who can identify the correct answer to the
math problem.
---
There are two choices to the same math problem. One uses natural language to
answer the question, while the other uses Python program to answer it. *ONLY
ONE* of them can correctly answer the math problem. You need to check these
two solutions carefully and identify which choice can correctly answer the
math problem. Here is the math problem and two solutions to it.
Math problem: Olivia has $23. She bought five bagels for $3 each. How much
money does she have left?
(A) ⬇ def solution(): money_initial = 23 bagels = 5 bagel_cost = 3 money_spent = bagels * bagel_cost money_left = money_initial - money_spent result = money_left return result The result is 8. |  (B)

Answer:
Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 + 3 = 8 dollars.
So she has 23 - 8 = 15 dollars left.
The answer is 15.

Which of the above two choices can correctly answer the math problem? (A) or
(B)?
(A) can correctly answer the math problem. Because (B) adds the number of
bagels to the cost of each bagel instead of multiplying them.
Table 16: One example of 8-shot model selection prompts on GSM8K with Llama 2
7B as the model selector. System: You are a helpful and brilliant assistant.
You are a math expert who can identify the correct answer to the math problem.
---
There are two choices to the same math problem. One uses natural language to
answer the question, while the other uses Python program to answer it. *ONLY
ONE* of them can correctly answer the math problem. You need to check these
two solutions carefully and identify which choice can correctly answer the
math problem. Here is the math problem and two solutions to it.
Math problem: Olivia has $23. She bought five bagels for $3 each. How much
money does she have left?
(A)

Answer:
Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 * 3 = 15 dollars.
She has 23 - 15 = 8 dollars left..
The answer is 8.
|  (B) ⬇ def solution(): money_initial = 23 bagels = 5 bagel_cost = 3
money_spent = bagels + bagel_cost money_left = money_initial - money_spent
result = money_left return result The result is 15.
Which of the above two choices can correctly answer the math problem? (A) or
(B)?
(A) can correctly answer the math problem. Because (B) adds the number of
bagels to the cost of each bagel instead of multiplying them.
Table 17: One example of 8-shot model selection prompts on GSM8K with Llama 2
13B as the model selector.

##  Appendix E Examples of Failure Cases Comparison

In Section 2.1, we divide the error reasons of CoT and PAL on GSM8K into 5
categories. Detailed examples of 4 primary categories are presented in the
following tables.

Question: It costs $194 per meter to repave a street. Monica’s street is 150
meters long. How much more does it cost to repave Lewis’ street, which is 490
meters long?
---
Answer:
The cost to repave Monica’s street is 194 * 150 = $29100.
The cost per meter is the same for Lewis’ street, so the cost to repave it is
194 * 490 = $94860.
The difference in cost is $94860 - $29100 = $65760.
So it costs $65760 more to repave Lewis’ street. ✗ |  ⬇ def solution(): monica_street_length = 150 lewis_street_length = 490 cost_per_meter = 194 monica_street_cost = monica_street_length * cost_per_meter lewis_street_cost = lewis_street_length * cost_per_meter cost_difference = lewis_street_cost - monica_street_cost result = cost_difference return result >65960.0 ✓
Table 18: A CoT failure case due to Computation Precision. PAL can easily
solve this question with the help of an external Python interpreter. >65960.0
is obtained with the Python interpreter. Question:Kim has started his own
housekeeping business and is calculating how much profit he will make from his
clients. He already has 3 clients, but is talking to another 5 potential
clients and feels confident enough to include them in his calculations. Each
client’s home will need 2 bottles of bleach and a pack of cloths to clean.
Bottles of bleach will cost $2 each and packs of cloths will cost $5 each.
These are his only expenses. He calculates that his total income each week
will be $92. Profit is the difference between total income and total expenses,
so how much profit, in dollars, will Lucas make each week?
---
Answer:
Kim has 3 clients and is talking to 5 more potential clients, so he will have
3 + 5 = 8 clients in total.
Each client needs 2 bottles of bleach, so Kim will need 2 * 8 = 16 bottles of
bleach each week.
Each client also needs a pack of cloths, so Kim will need 8 * 5 = 40 packs of
cloths each week.
The cost of the bleach will be 16 * 2 = $32.
The cost of the cloths will be 40 * 5 = $200.
So Kim’s total expenses will be 32 + 200 = $232.
Kim’s total income each week will be $92.
So his profit will be 92 - 232 = -$140.
Therefore, Kim will lose $140 each week. ✗ |  ⬇ def solution(): num_clients = 8 bleach_cost = 2 cloths_cost = 5 total_expenses = num_clients * (2 * bleach_cost + cloths_cost) total_income = 92 profit = total_income - total_expenses result = profit return result >20.0 ✓
Table 19: A CoT failure case due to Logical Decomposition. PAL breaks down the
problem into step-by-step Python code, and utilizes variables to map logical
relationships. Question:Seth is twice as old as Brooke. In 2 years, the sum of
their ages will be 28. How old is Seth?
---
Answer:
Let’s assume Brooke’s age to be x.
Then Seth’s age would be 2x (as he is twice as old as Brooke).
In 2 years, Brooke’s age would be x+2 and Seth’s age would be 2x+2.
The sum of their ages in 2 years would be (x+2) + (2x+2) = 3x+4.
We know that the sum of their ages in 2 years will be 28.
So, 3x+4 = 28.
Solving for x, we get x = 8.
Therefore, Brooke’s age is 8 and Seth’s age is 2x = 16.
So the answer is 16. ✓ |  ⬇ def solution(): sum_of_ages_in_two_years = 28 age_difference = 2 * 2 # Seth is twice as old as Brooke age_of_seth_in_two_years = (sum_of_ages_in_two_years + age_difference) / 2 age_of_seth = age_of_seth_in_two_years - 2 result = age_of_seth return result >14.0 ✗
Table 20: A PAL failure case due to Problem-Solving Flexibility. CoT employs
both forward and backward reasoning paradigms and solves this problem
correctly. Question: Gloria is shoe shopping when she comes across a pair of
boots that fit her shoe budget. However, she has to choose between the boots
and two pairs of high heels that together cost five dollars less than the
boots. If one pair of heels costs $33 and the other costs twice as much, how
many dollars are the boots?
---
Answer:
Let’s start by finding the total cost of the two pairs of high heels.
One pair of heels costs $33, and the other pair costs twice as much, which is
$33 x 2 = $66.
So the total cost of the two pairs of high heels is $33 + $66 = $99.
We also know that the cost of the boots is $5 more than the cost of the two
pairs of high heels.
Therefore, the cost of the boots is $99 + $5 = $104.
So the answer is $104. ✓ |  ⬇ def solution(): heels1 = 33 heels2 = 2 * heels1 heels_total = heels1 + heels2 boots = heels_total - 5 result = boots return result >94.0 ✗
Table 21: A PAL failure case due to Semantic Understanding. PAL fails to
understand the crucial information ‘five dollars less than boosts’, whereas
CoT exhibits a more advanced semantic understanding capability.