# Evaluating LLMs with Multiple Problems at once:
A New Paradigm for Probing LLM Capabilities

Zhengxiang Wang  Jordan Kodner  Owen Rambow
Department of Linguistics & Institute for Advanced Computational Science
Stony Brook University, Stony Brook, NY, USA
<EMAIL_ADDRESS>
###### Abstract

Current LLM evaluation predominantly performs evaluation with prompts
comprising single problems. We propose multi-problem evaluation as an
additional approach to study the multiple problem handling capabilities of
LLMs. We present a systematic study in this regard by comprehensively
examining 7 LLMs on 4 related types of tasks constructed from 6 classification
benchmarks. The 4 task types include traditional single-problem tasks,
homogeneous multi-problem tasks, and two index selection tasks that embed the
multi-problem tasks. We find that LLMs are competent multi-problem solvers:
they generally perform (nearly) as well on multi-problem tasks as on single-
problem tasks. Furthermore, contrary to common expectation, they often do not
suffer from a positional bias with long inputs. This makes multi-problem
prompting a simple and cost-efficient prompting method of practical
significance. However, our results also strongly indicate that LLMs lack true
understanding: they perform significantly worse in the two index selection
tasks than in the multi-problem task under various evaluation settings,
although they can indeed do index selection in general.

capbtabboxtable[][]

Evaluating LLMs with Multiple Problems at once:
A New Paradigm for Probing LLM Capabilities

Zhengxiang Wang and Jordan Kodner and Owen Rambow Department of Linguistics &
Institute for Advanced Computational Science Stony Brook University, Stony
Brook, NY, USA<EMAIL_ADDRESS>

##  1 Introduction

In recent years, large language models (LLMs) have demonstrated remarkable
natural language understanding and reasoning capabilities by achieving often
superhuman performance on a wide range of NLP benchmarks and human exams
(OpenAI, 2023; Beltagy et al., 2020; Gemini-Team, 2023; Anthropic, 2024). With
the advances in both GPU hardware and algorithms (Dai et al., 2019; Beltagy et
al., 2020; Dao et al., 2022; Ding et al., 2024; Chen et al., 2024, inter
alia), LLMs are being developed with larger and larger context windows (e.g.,
8K, 128K, and even 2M) to support diverse real-world use cases, such as in
retrieval augmented generation Lewis et al. (2020); Borgeaud et al. (2022).
Correspondingly, there is a growing body of research that evaluates LLMs when
presented with tasks in long inputs (e.g., Shaham et al., 2023; Li et al.,
2024a; Levy et al., 2024).

Figure 1: Standard single-problem evaluation versus multi-problem evaluation.

While these efforts investigate what is possible with larger context windows,
they still focus on a single problem per prompt. Little is known how well LLMs
can handle prompts comprising multiple problems simultaneously. This is an
important question because in order to solve multiple problems presented in a
single input that can be sufficiently long, LLMs need to locate each problem
in the input when generating answers to these problems in a correct order.
This helps us better understand how well LLMs can use information from the
inputs with varying numbers of problems.

To this end, we present the first study that systematically and
comprehensively examines the multiple problem handling capabilities of 7 LLMs
on 4 related types of tasks constructed from 6 classification benchmarks. The
task types in our experiments include traditional single-problem tasks (i.e.,
the standard use of these benchmarks), homogeneous multi-problem tasks of
varying sizes, and two index selection tasks that embed the multi-problem
tasks with additional selection steps. These task types are designed to be
minimally different from one another and we test the LLMs under zero-shot,
few-shot, and few-shot-CoT (chain of thought, Wei et al., 2023) settings. We
find that:

LLMs can handle multiple problems of the same type at once with comparable or
even superior performance to traditional single-problem presentation. This is
often the case with even 100 problems per prompt. This indicates that LLMs do
not necessarily always show declining performance in response to longer
inputs, as may be commonly expected. Furthermore, they also do not necessarily
show the primacy and recency biases that have been observed in other contexts
Liu et al. (2024a).

This multi-problem prompting substantially reduces costs. For example,
prompting many homogeneous problems with shared task instructions can reduce
the actual cost of accessing GPT-4 to perform inference by up to 82% per
problem when performing multiple classifications per prompt rather than
separately, based on our experiments.

Nevertheless, LLMs still display a lack of true understanding, as they do not
solve multiple problems equally well when presented in a different form. More
concretely, they perform significantly worse in selecting indices of text
falling into each label, either independently or altogether, than in making
multiple classifications at once directly, although these are conceptually
equivalent. The issue, as we show, is not the index selection task, but the
combination of the actual classification task (which is hard) and the index
selection task (which is easy). Furthermore, we find that selecting indices of
text that belong to each label one at a time is significantly more challenging
than generating the indices for all labels at once under all evaluation
settings. In the former scenario, LLMs consistently make more contradiction
errors and non-excluded middle errors, compared to the latter scenario,
because they select indices for each label independently rather than
sequentially. This again highlights the probabilistic aspect of the LLM answer
generation, as opposed to true understanding.

Our paper proceeds as follows. We first review related work in Section 2. We
then motivate and propose multi-problem evaluation in Section 3, followed by
an introduction to the experimental setups in Section 4. We report and discuss
our main results in Section 5 and follow-up results in Section 6. Section 7
concludes. We release our code, data, and results at
https://github.com/jaaack-wang/multi-problem-eval-llm.

##  2 Related Work

While there have been several surveys dedidcated to the evaluation of specific
topics, e.g., hallucination Huang et al. (2023); Rawte et al. (2023), bias and
fairness Gallegos et al. (2024); Li et al. (2024b), and alignment Wang et al.
(2023); Liu et al. (2024b), we note that current LLM evaluation has
predominantly focused on LLM’s performance on prompts consisting of single
problems. Each of such prompts presents a single problem and expects one
specific answer to that problem.

To the best of our knowledge, Laskar et al. (2023) is the only study that
analyzes the ability of GPT-series models to solve 5 short questions sampled
from two open-domain QA benchmarks Min et al. (2021); Berant et al. (2013) at
once. They observe that only instruction fine-tuned GPT models can handle such
multi-problem prompts, but with varying performance drops. However, since
evaluating LLMs with multiple problems presented together is not the focus of
Laskar et al. (2023), the authors do not provide any fine-grained analysis nor
conduct their experiments under different conditions, e.g., different numbers
of problems in each prompt, non-GPT LLMs, and prompting strategies other than
zero-shot prompting. It remains unknown how well LLMs in general can do
multiple problems at once.

Another line of research is the evaluation of LLMs presented with tasks
requiring long inputs. These long-input tasks include long text understanding
and reasoning Shaham et al. (2023); Li et al. (2023); Liu et al. (2024a),
many-label classification with long task description Li et al. (2024a), and
short question answering extended with irrelevant distractor information Levy
et al. (2024). It is generally found that long-input tasks overall remain a
challenge, with the most capable proprietary LLMs such as GPT-4 often
achieving the best absolute performance. These studies also identify
positional biases of LLMs over long inputs, such as primacy bias Liu et al.
(2024a) and recency bias Liu et al. (2024a); Levy et al. (2024).

##  3 Multi-Problem Evaluation

###  3.1 Single- vs. Multi-Problem Evaluation

As illustrated in Fig 1, the difference between a single-problem evaluation
and a multi-problem evaluation lies in the number of problems placed under a
task instruction (extending general prompt design from (Brown et al., 2020)).
Multi-problem evaluation places several parallel problems in prompts and
expects an answer to each problem in sequence, whereas single-problem
evaluation only provides one problem per prompt. A simple example of a multi-
problem task would bundle multiple classification or QA problems together and
ask LLMs to solve them sequentially.

Evaluation in a multi-problem setting has several advantages over typical
single-problem evaluation. First, it is less likely for LLMs to encounter
exact long multi-problem prompts during pre-training because of the
combinatory nature of constructing prompts from multiple problems. This helps
to mitigate a growing data contamination concern in modern large-scale pre-
training (Jacovi et al., 2023; Sainz et al., 2023). Second, since we can
manipulate what kind of problems and how many problems to include, we know
exactly which problem a LLM gets wrong or right across positions in the
prompts. This enables us to construct a well controlled and easily
interpretable evaluation. Third, our study demonstrates that leveraging the
rich existing benchmarks to create a new multi-problem task is cheap, easy to
implement, and highly adaptable. The most laborious component is the prompt
design, which, once done, can easily be applied to a set of benchmarks with
minimal adaptation.

Input Format | Benchmark | # Samples | Objective
---|---|---|---
Single-text | SST-2 | 1,821 | Sentiment analysis
| CoLA | 1,043 | Grammatical acceptability
| AGNews | 1,000 | Topic classification
Text-pair | MRPC | 1,725 | Paraphrase detection
| SNLI | 1,000 | Natural language inference
| WiC | 1,400 | Word sense disambiguation
Table 1: Classification benchmarks used in the study. We use the test splits
wherever possible, except for CoLA, for which we use the dev split, since the
test split is not publicly available. For AGNews and SNLI, we randomly sample
1,000 examples from the test splits. Figure 2: An example prompt for a
5-problem classification task constructed from CoLA, divided into shared
instructions and a series of problems. Figure 3: The 4 types of evaluation
tasks designed for classification benchmarks illustrated in the form of a
<task, input, output> triplet. We use the darker colors to indicate the
_anticipated_ increasing task complexity.

###  3.2 Benchmarks

To facilitate comparison, we construct homogeneous multi-problem tasks from
existing single-problem benchmarks. We consider the following 6 classification
benchmarks, as described in Table 1: SST-2 (Socher et al., 2013), CoLA
(Warstadt et al., 2019), AGNews (Gulli, 2004), MRPC (Dolan and Brockett,
2005), SNLI (Bowman et al., 2015), and WiC Pilehvar and Camacho-Collados
(2019). These benchmarks are commonly used for NLP evaluation, with SST-2,
CoLA, and MRPC appearing in GLUE Wang et al. (2019) and WiC in SuperGLUE
(Sarlin et al., 2020). They cover two classification paradigms: single-text
classification and text-pair classification. The objective of each benchmark
is also different. A detailed description of these benchmarks is provided in
Appendix A.

###  3.3 Evaluation Tasks and Prompt Design

We conduct our multi-problem evaluation on 4 related types of tasks,
conceptualized in Fig 3, using the 6 classification benchmarks introduced
above. We employ the “task instruction + problem” structure illustrated in Fig
1 to design the prompts and to maximize the prompt similarity across
tasks.111The prompt templates used are presented in Appendix A.

#### Single Classification (SingleClf)

We start with Single Classification, which is the standard way of evaluating
LLM’s classification capabilities. The prompt indicates the task objective and
the possible label choices in the task instruction, followed by one target
classification problem. For example, we use the following task instruction for
CoLA: “ _Indicate the grammatical acceptability for the following line of
text. The acceptability shall be either ‘Acceptable’ or ‘Unacceptable.’_ ”

#### Batch Classification (BatchClf)

This is the multi-problem version of SingleClf. The BatchClf prompt consists
of a shared task instruction with multiple target classification problems
following. Fig 2 provides an example of a 5-problem classification task based
on CoLA. The task instructions for a SingleClf prompt and for the
corresponding BatchClf prompt are nearly identical to maximize comparability.

#### Index Selection One Category (SelectOne)

We additionally introduce two index selection tasks that conceptually embed
BatchClf. SelectOne differs from BatchClf by having an additional reasoning
step, namely index selection. More concretely, for each label one at a time,
SelectOne prompts LLMs to select all indices of text that should be classified
as label separately. The expected output for SelectOne is “None,” “All,” or a
list of applicable indices.

#### Index Selection All Categories (SelectAll)

SelectAll prompts a system to do SelectOne for each possible label all at
once. The expected output for a single prompt consists of, for every possible
label, the label followed by the SelectOne output for that label. For both
SelectOne and SelectAll, we ask LLMs to produce the output in JSON
format.222This is for ease of parsing SelectAll outputs and for comparability,
but we found in our early experiments that having free-form responses for
SelectOne led to similar results.

We design the two selection tasks to test LLM’s understanding of the
classifications performed under BatchClf. Since selection tasks of size $n$
may have anywhere from 0 to $n$ correct indices per class, spurious
correlations are less likely during our evaluation, given the combinatory
answer space.

##  4 Experimental Setup

#### Summary of terminology

We refer to prompting an LLM under one experimental configuration (e.g.,
benchmark, model) as a task. We have 4 task types: SingleClf, BatchClf,
SelectOne, and SelectAll (Section 3). Each task contains one or more problems
for the LLM to respond to. The problems are the examples from the existing 6
benchmarks listed in Table 1. Three of these benchmarks are single-text
problems (classification of properties of a single text), and 3 are text-pair
inputs (classification of the relationship between two texts).

#### LLMs and evaluation settings

We evaluate 7 LLMs: Vicuna (Chiang et al., 2023), Mistral 7B (Jiang et al.,
2023), Mixtral 8x7B (Jiang et al., 2024), Llama-3 8B and
70B,333https://ai.meta.com/blog/meta-llama-3/ GPT-3.5, and GPT-4 (OpenAI,
2023). We conduct the main experiments under the zero-shot setting with greedy
decoding. We selectively experiment with other prompting strategies in the
follow-up experiments in Section 6. Details about these models, such as model
versions and context windows, are in Appendix B.

#### LLM output parsing code

We develop an extensive set of regular expressions to parse the LLM outputs
for the 4 task types in Fig 3, taking into account such factors as the task
type, the benchmark, and the LLM model, among others. The parsing code is
capable of handling cases where LLMs do not generate outputs in a format as
intended, e.g., an undefined label, or an answer with un-requested
explanations. Details can be found in Appendix B.

#### Task size

We refer to the number of classification problems contained in a prompt as the
task size. The task size for a single-problem prompt (SingleClf) is 1. In our
experiments, we use 5 different task sizes for all benchmarks: 1, 5, 10, 20,
and 50. We also test task size 3 for text-pair benchmarks, and task size 100
for single-text benchmarks. We use all problems from each benchmark for
SingleClf.

For each benchmark, we randomly sample 100 distinct sets of 100 problems, then
select the first $n$ for task size $n$, so that larger task sizes only differ
from smaller task sizes by having additional problems. For SelectOne, we re-
use each set of multiple problems $m$ times to construct $m$ prompts targeting
each of the $m$ possible labels for those problems. This results in $m$ times
more prompts for SelectOne than BatchClf and SelectAll under the same
conditions. However, the problems are identical across the three task types.

#### Performance metric

We measure the average per-problem accuracy to unify the evaluation across the
four task types in Fig 3. Per-problem accuracy (PPA) is the average accuracy
of classifying $n$ problems in a single prompt or, in the case of SelectOne,
in a set of directly related prompts targeting different labels. It is defined
in Equation 1:

$\mathrm{PPA}=\frac{1}{n}\sum_{i=1}^{n}\delta(I(\mathcal{P}_{i}),\mathcal{A}_{i})$
(1)

$I(\mathcal{P}_{i})$ is the inferred LLM-generated answer to the $i$th problem
in the input prompt, $\mathcal{A}_{i}$ is the ground truth, and $\delta$ is
the Kronecker delta function such that $\delta(i,j)=1$ iff $i=j$ and 0
otherwise. For the two index selection tasks, $I(\mathcal{P}_{i})$ is
determined by considering the LLM’s assignments of indices to all possible
labels. LLMs may produce contradictory index assignments across different
labels, or not select certain indices at all. In other words,
$I(\mathcal{P}_{i})$ may be mapped to a list of labels or an empty list,
instead of a single label. In both cases, $I(\mathcal{P}_{i})\neq A_{i}$ and
therefore $\delta(I(\mathcal{P}_{i}),\mathcal{A}_{i})=0$.

To measure an LLM’s performance on a task type under some conditions
(benchmark and task size), we average PPA on all the related prompts for that
task. Since we are mainly interested in comparing the LLM’s performance across
task types, accuracy alone suffices for our purpose.

##  5 Main Results

| > 90% SCAcc | > 80% SCAcc | > 75% SCAcc
---|---|---|---
Vicuna 13B | 79.3 | 93.1 | 93.1
Mistral 7B | 76.7 | 83.3 | 100.0
Mixtral 8x7B | 63.3 | 83.3 | 86.7
Llama-3 8B | 73.3 | 90.0 | 100.0
Llama-3 70B | 80.0 | 100.0 | 100.0
GPT-3.5 | 56.7 | 83.3 | 90.0
GPT-4 | 100.0 | 100.0 | 100.0
Overall | 75.6 | 90.4 | 95.7
Table 2: Percent of time that BatchClf performance surpasses a threshold
percent of SingleClf accuracy (SCAcc) across benchmarks. Figure 4: Aggregated
average accuracy of the 7 LLMs on the 4 types of tasks across various task
sizes for each benchmark. See Appendix C for the full results for each LLM.
Figure 5: Cost/Accuracy Ratio (lower is better) for the 7 LLMs on the 6
benchmarks for SingleClf (task size=1) and BatchClf (otherwise). We use the
average (input + output) token count per classification as the proxy for the
actual inference costs, calculated on the basis of the input and output
tokens. The plot for MRPC is log-scale for the y-axis. Figure 6: Cost saving
rate (%) per classification based on our experiments. The cost is estimated by
both the input and output token counts (using the respective tokenizers),
weighted according to the pricing policy from OpenAI and TogetherAI (for non-
GPT LLMs) websites. Figure 7: Cumulative error density across positions
averaging results from the benchmarks. Task size 3/100: the 3 text-
pair/single-text benchmarks; otherwise: all the 6 classification benchmarks.
See Appendix C for full results.

###  5.1 LLMs are Multi-Problem Solvers

We aggregate the zero-shot results across the 7 LLMs on the 4 task types for
each benchmark in Fig 4 with the related full results for each LLM provided in
Appendix C. We exclude the results of Vicuna on AGNews when task size is 100
because the prompts exceed the model’s context length. We focus on the general
patterns across LLMs and benchmarks, since model-level or benchmark-level
variations are not the focus of the study. To compare performance difference,
we use Mann-Whitney U tests for significant testing and Cohen’s $d$ Cohen
(1969) for measuring effect size.

#### LLMs can handle multiple classifications at once with minimal performance
loss.

Although the BatchClf accuracy generally declines as the task size increases
(Fig 4), all 7 LLMs achieve accuracy of at least 90% that of SingleClF across
the benchmarks most of the time. Table 2 indicates the proportion of BatchClf
tasks for which each LLM surpasses a threshold percent of corresponding
SingleClf performance. Interestingly, for SNLI almost all LLMs perform better
in BatchClf than in SingleClf across all the task sizes (3-50) and GPT-4
consistently achieves a BatchClf accuracy near or better than the SingleClf
accuracy under all conditions (see Appendix C). Overall, the SingleClf
accuracy for the 7 LLMs on the 6 benchmarks is 75.5% and the BatchClf
counterpart is 72.3%, a minor 3.2% absolute drop from the former.

#### Multi-problem prompting is cost-efficient.

Single-problem prompting can waste input tokens by redundantly repeating a
shared task instruction. Multi-problem prompting reduces this redundancy, and
this saving is larger the more problems are combined in a single prompt.
Because performance tends to decline slowly as the number of tasks increases,
this yields a favorable cost-accuracy ratio as the number of tasks increases
(Fig 5). We only encountered two outliers, Vicuna on MRPC with task size 50
where the average input is 3,645 tokens and the context window is 4,096
tokens, and Mixtral 8x7B at $\geq$ 50 on AGNews. While it is of course up to
downstream users to determine what cost-accuracy is right for them, this is
likely beneficial for many use cases where similar prompts are repeated
frequently.

To further illustrate, we estimate the potential cost saving rate for BatchClf
based on our experiments. Fig 6 shows the cost saving rate for each model/task
combination at the largest BatchClf task size that achieves at least 95%
SingleClf accuracy for that pair. We observe that the cost saving rate is
substantial for all LLMs run on all the 6 benchmarks, ranging from from 30.7%
to 82.0%. The average cost saving rate for each LLM on the 6 benchmark is at
least 44.7% (GPT-3.5) and up to 68.5% (Mixtral 8x7B). We provide more fine-
grained analysis in Appendix C.

#### LLMs make errors evenly across positions under multiple problems.

Fig 7 shows that LLMs do not have clear positional bias, also known as the
serial position effect in psychology Murdock (1962), when solving multiple
problems of the same type at once (i.e., BatchClf). While some positional bias
may surface for some LLMs on some benchmarks, such biases are very mild or
non-existent most of times (see breakdown in Appendix C). This is in contrast
to the previous single-problem evaluation results, where LLMs are found to be
better at using information from the beginning (primacy bias) or the end
(receny bias) of the prompt Liu et al. (2024a); Levy et al. (2024). The lack
of a positional bias supports the claim that LLMs can use information equally
well across different positions under multiple homogeneous problems.

Figure 8: Pairwise comparisons of SelectOne and SelectAll for each LLM across different task sizes averaging over results from the 6 benchmarks. |  BatchClf vs
SelectOne |  BatchClf vs
SelectAll |  SelectOne vs
SelectAll
---|---|---|---
Mean Acc Dif |  32.0 |  12.1 |  -19.9
Std Dev |  16.9 |  15.3 |  12.0
Cohen’s $d$ |  1.8 |  0.8 |  -1.0
Table 3: Pairwise accuracy difference (x vs y = x - y). All the differences are statistically significant and with a large effect size (| Cohen’s $d$ | $\geq 0.8$).

###  5.2 LLMs Lack True Understanding

While there is no consensus about what constitutes “understanding” in LLMs, it
is typically assumed that true understanding of a problem involves the ability
to solve the problem under various forms, e.g., question answering, fill-in-
the-blank, and multiple choice questions. We adopt this invariance assumption
for the purposes of our analysis. In this sense, our results show that LLMs
lack true understanding.

#### LLMs perform significantly worse on the selection tasks.

In our experiments, LLMs nearly always perform much better in BatchClf than in
SelectOne and SelectAll under the same conditions with a consistent and stable
margin, even when the task size is just 3 or 5 (Fig 4). The overall
discrepancy in accuracy between BatchClf and the two tasks is large and
statistically significant (32% for SelectOne and 10% for SelectAll), as shown
in Table 3. Moreover, the selection tasks have a sharper downward performance
curve (Fig 4) as the task size increases, compared to that of BatchClf.

Although it is expected that the selection tasks are harder because BatchClf
is embedded by them, the sharp drop in accuracy even with a rather small task
size suggests that LLMs do not really “understand” many of the texts they can
otherwise classify correctly under BatchClf.

#### LLMs lack internalized planning and consistently select inconsistently.

Surprisingly, such a consistent and rather stable performance gap also exists
between SelectOne and SelectAll in favor of the latter, largely independently
of the task size (Fig 4). On average, the SelectOne accuracy is 19.9% lower
than the SelectAll accuracy, also with a large and significant effect size
(Table 3).

We investigate the reasons in Fig 8, which shows that when asked to select
text indices for one label at a time independently, LLMs almost always assign
more indices to more than one label (i.e., contradiction) and leave more
indices unselected (i.e., non-excluded middles). This showcases a lack of
internalized planning with modern LLMs with zero-shot prompting, although
different LLMs may make these two types of errors in different proportions. In
contrast, when LLMs have to select indices for all labels at once, they are
less likely to generate directly illogical answers in a single output because
what they answer to the $i_{th}$ label puts a condition on their answer to the
$(i+1)_{th}$ label. This highlights the probabilistic aspect of LLM answer
generation, which may deviate from what is considered true and invariant
understanding.

#### LLMs can do index selection.

While we argue that the LLMs’ weaker performance on SelectOne and SelectAll is
due to a weakness in combining classification with index selection, an
alternative explanation would be that they just generally struggle at index
selection. To exclude this possibility, we simplify the two index selection
tasks by directly replacing each line of text with its gold standard label in
the prompts for AGNews, which has most labels. We then ask the LLMs to select
the indices of lines containing each label with minimal modifications to the
original task instructions.

Figure 9: Average accuracy of the 7 LLMs on the two simplified index selection
tasks based on AGNews. We provide the original results from BatchClf,
SelectOne, and SelectAll for easy comparisons.

Fig 9 shows that LLMs can perform in the simplified selection tasks much
better than the original ones, with Llama-3 70B, GPT-3.5, and GPT-4 even
achieving (nearly) 100% accuracy across task sizes. The 7 LLMs’ overall
performance in the simplified selection tasks is even slightly higher than
their overall BatchClf accuracy. The previously observed performance gap
between SelectOne and SelectAll also disappears for almost all LLMs (see
Appendix D for full results). In general, LLMs can indeed do index selection.

In conclusion, it appears that the tested LLMs perform less well on SelectOne
and SelectAll as compared to BatchClf because they cannot adequately combine
the index selection task and the classification tasks in response to a zero-
shot prompt. Put differently, the LLMs lack true understanding of the problems
presented in different forms, even when the number of problems is quite small
(e.g., 3 or 5), which may not be human-like.

##  6 Follow-up Experiments

###  6.1 CoT Helps the Index Selection Tasks Slightly but the Gap Remains

We use 1-shot-CoT to prompt LLMs to do BatchClf first and then perform the two
index selection tasks on the 6 benchmarks with a fixed task size 10. Table 4
shows that while overall LLMs benefit from CoT for both SelectOne and
SelectAll, the benefits are generally larger for SelectOne than SelectAll
(3.1% versus 1.0% improvement) and not consistent across models with Mixtral
8x7B performing even worse with CoT. The full results provided in Appendix D
also show mixed effects across benchmarks. Interestingly, however, the task
complexity hierarchy among BatchClf, SelectOne, and SelectAll remains in
virtually all cases. We leave to future studies the investigation of how CoT
may further help with the two selection tasks.

###  6.2 Few-Shot Prompting Does Not Help

To test the generality of the main findings, we first re-run all experiments
for CoLA providing 2 exemplars in the prompts and with a fixed task size 5 for
tasks other than SingleClf. We find that few-shot prompting is mostly
detrimental across LLMs, particularly so for SelectOne and BatchClf. As a
result, SelectAll shows an overall better performance than BatchClf and the
performance gap between SelectOne and SelectAll becomes much larger.
Nevertheless, our main findings still hold most of times for the 7 LLMs. Full
results are provided in Appendix D for space reasons.

###  6.3 Testing on a Novel Benchmark

We generate a novel benchmark with 1,000 distinct and label-balanced text
pairs sampled from SST-2. The task objective is to determine if each text pair
shares the same sentiment, which we believe is unlikely to appear in the
training data of LLMs. We run experiments on this benchmark using the same
experimental setup described in Section 4 for the text-pair benchmarks.
Possibly due to the novel use of SST-2, Mistral 7B only achieves 29.6%
accuracy in SingleClf, well below the 50.0% random baseline and close to the
SelectOne performance. Other than that, the results (see Appendix D) are
consistent with our main results.

Task | SelectOne | \+ CoT | SelectAll | \+ CoT | BatchClf
---|---|---|---|---|---
Model |  |  |  |  |
Vicuna | 23.0 | 25.8 | 53.2 | 57.6 | 68.7
Mistral 7B | 38.3 | 47.5 | 56.8 | 60.5 | 71.2
Mixtral 8x7B | 47.4 | 38.7 | 65.1 | 58.4 | 73.4
Llama-3 8B | 39.0 | 41.8 | 62.4 | 59.0 | 73.3
Llama-3 70B | 59.4 | 67.1 | 72.9 | 79.2 | 79.4
GPT-3.5 | 45.5 | 47.5 | 66.7 | 66.3 | 71.9
GPT-4 | 66.3 | 71.8 | 78.8 | 81.8 | 81.9
Overall | 45.5 | 48.6 | 65.1 | 66.1 | 74.3
Table 4: Aggregate average accuracy of SelectOne and SelectAll with and
without 1-shot-CoT for each LLM. BatchClf performance is also provided for
comparisons.

##  7 Conclusion

We propose multi-problem evaluation and present the first systematic and
comprehensive study in this regard. Our results provide new insights into the
multiple problem handling capabilities of LLMs: LLMs are competent multi-
problem solvers for multiple homogeneous problems, but they lack true
understanding of the problems because they perform significantly worse when
the problems are presented in different forms. Our study showcases the
potential of multi-problem evaluation as a new paradigm for probing LLM
capabilities.

There are several directions worth future explorations. For example, to better
understand how well how LLMs can handle multiple problems in general, it is
important to test LLMs with other types of problems (e.g., reasoning problems)
and with multiple heterogeneous problems (e.g., mixing different
benchmarks/tasks). It is also important to understand what causes LLMs to
perform worse or better when prompted with multiple problems, such as
benchmark, task size, and model’s context length. In particular, model-level
ablation studies are needed if we want to know how LLMs obtain the ability to
handle multiple problems at once and how to improve their understanding
capabilities.

## Acknowledgement

We thank the Institute for Advanced Computational Science (IACS) at Stony
Brook University for providing us free access to the GPT models used in our
experiments. Zhengxiang Wang and Owen Rambow were supported in part by funding
from the Defense Advanced Research Projects Agency (DARPA) under Contracts No.
HR01121C0186, No. HR001120C0037, and PR No. HR0011154158. Any opinions,
findings and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of DARPA.

## Limitations

While we have done our best to make our experiments as comprehensive as
possible, there is always more work that can be done in a large comparison
study, including the addition of even more benchmarks and language models. We
were limited by cost, time, and space and have attempted to select an
informative and representative sample of experiments. Since we used pre-
existing benchmarking data sets, we inherit any labeling errors that they main
contain. Finally, despite our efforts to compare different prompts, as is the
case with all prompt-based LLM studies, we cannot guarantee that slight
differences in the prompts would not meaningfully alter the results.

## Ethical Concerns

To the best of our knowledge, all results published in this paper are
accurate. All data sources are free, publicly available, and cited in the
article. No sensitive data was used which could violate individuals’ privacy
or confidentiality.

## References

  * Anthropic (2024) Anthropic. 2024.  The Claude 3 model family: Opus, Sonnet, Haiku.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022\.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chen et al. (2024) Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024.  Longlora: Efficient fine-tuning of long-context large language models.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Cohen (1969) J. Cohen. 1969.  _Statistical Power Analysis for the Behavioral Sciences_.  Academic Press.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 16344–16359. Curran Associates, Inc.
  * Ding et al. (2024) Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024.  Longrope: Extending llm context window beyond 2 million tokens.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Gallegos et al. (2024) Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024.  Bias and fairness in large language models: A survey.
  * Gemini-Team (2023) Gemini-Team. 2023.  Gemini: A family of highly capable multimodal models.
  * Gulli (2004) Antonio Gulli. 2004.  AG’s corpus of news articles.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.
  * Jacovi et al. (2023) Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023.  Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5075–5084, Singapore. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024\.  Mixtral of experts.
  * Laskar et al. (2023) Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023.  A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 431–469, Toronto, Canada. Association for Computational Linguistics.
  * Levy et al. (2024) Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024.  Same task, more tokens: the impact of input length on the reasoning performance of large language models.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Li et al. (2023) Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023.  Loogle: Can long-context language models understand long contexts?
  * Li et al. (2024a) Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024a.  Long-context LLMs struggle with long in-context learning.
  * Li et al. (2024b) Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2024b.  A survey on fairness in large language models.
  * Liu et al. (2024a) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a.  Lost in the Middle: How Language Models Use Long Contexts.  _Transactions of the Association for Computational Linguistics_ , 12:157–173.
  * Liu et al. (2024b) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2024b.  Trustworthy LLMs: a survey and guideline for evaluating large language models’ alignment.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih. 2021.  Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.  In _Proceedings of the NeurIPS 2020 Competition and Demonstration Track_ , volume 133 of _Proceedings of Machine Learning Research_ , pages 86–111. PMLR.
  * Murdock (1962) Bennet B. Murdock. 1962.  The serial position effect of free recall. _Journal of Experimental Psychology_ , 64(5):482–488.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A survey of hallucination in large foundation models.
  * Sainz et al. (2023) Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10776–10787, Singapore. Association for Computational Linguistics.
  * Sarlin et al. (2020) Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020\.  SuperGlue: Learning feature matching with graph neural networks.
  * Shaham et al. (2023) Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023.  ZeroSCROLLS: A zero-shot benchmark for long text understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 7977–7989, Singapore. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.
  * Wang et al. (2023) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023.  Aligning large language models with human: A survey.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.

Task |  Prompt template
---|---
Single Classification |  Indicate the sentiment for the following line of text. The sentiment shall be either ‘Positive’ or ‘Negative.’

Text: $text
Sentiment:
Batch Classification |  Indicate the sentiment for each of the $num following lines of text. The sentiment shall be either ‘Positive’ or ‘Negative.’

Texts, one per line:

$texts

The sentiments for each of the $num lines of text, one per line:
Index Selection One Category |  Go over the $num lines of text below and list the index numbers of the lines with $polarity sentiment according to the following instructions: If none of the texts show $polarity sentiment, write ‘None.’ If all the texts show $polarity sentiment, write ‘All.’ Otherwise, provide the index numbers for each text with $polarity sentiment.

Output your responses in JSON format with the key ‘$polarity’. A formatted
example output is provided below.
{‘$polarity’: [None/All or index numbers for the texts with $polarity
sentiment]}

Texts, one per line:

$texts

JSON output:
Index Selection All Categories |  Go over the $num lines of text below. First, list the index numbers of the lines with positive sentiment. Then, list the index numbers of the lines with negative sentiment. If none of the texts show a particular sentiment, write ‘None.’ If all the texts show a particular sentiment, write ‘All.’ Otherwise, provide the index numbers of the texts that fit a particular category.

Output your responses in JSON format with two keys: ‘positive’ and ‘negative.’
A formatted example output is provided below.
{‘positive’: [None/All or index numbers of positive sentences], ‘negative’:
[None/All or index numbers of negative sentences]}

Texts, one per line:

$texts

JSON output:
Table 5: Prompt templates for SST-2. Words immediately preceded by the dollar sign $ are placeholders. Task size | SST-2 | CoLA | AGNews | MRPC | SNLI | WiC
---|---|---|---|---|---|---
1 | 59 | 54 | 77 | 106 | 94 | 95
3 | - | - | - | 240 / 318 / 345 | 184 / 244 / 305 | 184 / 250 / 304
5 | 183 / 246 / 275 | 127 / 182 / 218 | 236 / 338 / 406 | 356 / 434 / 461 | 253 / 312 / 374 | 254 / 320 / 374
10 | 305 / 368 / 397 | 184 / 239 / 275 | 403 / 504 / 573 | 646 / 724 / 751 | 421 / 480 / 542 | 423 / 489 / 543
20 | 547 / 610 / 639 | 300 / 355 / 391 | 736 / 838 / 906 | 1221 / 1298 / 1326 | 757 / 816 / 878 | 761 / 827 / 881
50 | 1290 / 1353 / 1382 | 647 / 702 / 738 | 1741 / 1842 / 1911 | 2948 / 3025 / 3053 | 1763 / 1822 / 1884 | 1772 / 1838 / 1892
100 | 2515 / 2578 / 2607 | 1223 / 1278 / 1314 | 3421 / 3523 / 3591 | - | - | -
Table 6: Average prompt token count for the main experiments under zero-shot scenario. For simplicity, the counts here are calculated based on GPT-3.5 tokenizer, but using other tokenizers from other LLMs used in this study yields mostly similar counts. For task size greater than 1, we report the token counts for BatchClf, SelectOne, and SelectAll in sequence separated by ‘/’. Model | Version | Open Weights | With RLHF | MoE | # Parameter | Context Window Size
---|---|---|---|---|---|---
Vicuna (Chiang et al., 2023) | v1.5 | ✓ | ✗ | ✗ | 13B | 4,096
Mistral 7B (Jiang et al., 2023) | Instruct-v0.2 | ✓ | ✗ | ✗ | 7B | 8,192
Mixtral 8x7B (Jiang et al., 2024) | Instruct-v0.1 | ✓ | ✗ | ✓ | 47B | 8,192
Llama-3 8B | Instruct | ✓ | ✓ | ✗ | 8B | 8,192
Llama-3 70B | Instruct | ✓ | ✓ | ✗ | 70B | 8,192
GPT-3.5 | turbo-0125 | ✗ | ✓ | ✗ | - | 16,385
GPT-4 (OpenAI, 2023) | turbo-2024-04-09 | ✗ | ✓ | ✗ | - | 128,000
Table 7: The 7 LLMs we use, all instruction finetuned. For the two GPT LLMs,
it is commonly assumed that both are larger than GPT-3 (a 175B LLM) with GPT-4
being the largest. For Mixtral 8x7B, a Mixture of Experts (MoE) LLM, although
each token has access to 47B parameters, but only uses 13B active parameters
during inference.

##  Appendix A Data

###  A.1 Benchmarks

  * •

SST-2 (Socher et al., 2013) or Stanford Sentiment Treebank is a binary
sentiment benchmark created from movie reviews. The sentiment is either
positive or negative. License unspecified.

  * •

CoLA (Warstadt et al., 2019) or Corpus of Linguistic Acceptability is a binary
grammaticality judgement benchmark created by aggregating grammaticality
examples used in previous linguistics literature. The grammaticality is either
acceptable or unacceptable. MIT License.

  * •

AGNews (Gulli, 2004) is a corpus of news articles collected by Antonio Gulli,
covering four general news categories: “Business,” “Sports,” “World,” and
“Sci/Tech.” License unspecified.

  * •

MRPC (Dolan and Brockett, 2005) or Microsoft Research Paraphrase Corpus is a
binary paraphrase detection benchmark comprising sentence pairs collected from
news articles, either paraphrases or non-paraphrases. License unspecified.

  * •

SNLI (Bowman et al., 2015) or Stanford Natural Language Inference is a three-
way textual inference benchmark containing human-written sentence pairs for
image captions. The inference relation is either entailment, neutral, or
contradiction. Creative Commons Attribution-ShareAlike 4.0 International
License.

  * •

WiC Pilehvar and Camacho-Collados (2019) or Word-in-Context is a binary word
sense disambiguation benchmark where a target word is used either with same
meaning or with different meanings in two provided contexts for each example.
Examples are taken from lexical resources like WordNet and Wiktionary.
Creative Commons Attribution-NonCommercial 4.0 License.

###  A.2 Prompt Templates

Table 5 provides the prompt templates for the 4 task types constructed from
SST-2. The prompt templates for the other 5 benchmarks follow the same “task
instruction + problem” structure, with the task instruction tailored for the
task objective of each benchmark. Check https://github.com/jaaack-wang/multi-
problem-eval-llm for full prompt templates and all prompts created in this
study.

Throughout our research project, we have also tried prompts with different
wordings and structures until we finally unified the prompt designs presented
in Table 5. For example, we initially asked LLMs to directly generate indices
line by line instead of a JSON output for SelectOne and we did not provide any
formatted example for SelectAll. We also put the output format instruction in
the end of each prompt for SelectAll, instead of in the beginning. Although we
observed certain task-level performance variations, which are expected, the
overall complexity among the 4 task types (SelectOne > SelectAll > BatchClf >
SingeClf) remains unchanged, despite the variations in the prompts. This
indicates the overall limited effects of rewording and restructuring prompts.

###  A.3 Average Prompt Length

Table 6 details the average prompt token count for each task across different
task sizes and benchmarks based on the GPT-3.5 tokenizer. Using other LLMs’
tokenizers yields mostly similar counts.

##  Appendix B Experimental details

###  B.1 LLM Details

Table 7 describes the specific versions for the 7 LLMs we use in the paper and
highlights their differences in terms of open weights, training with
Reinforment Learning from Human Feedback or RLHF (Christiano et al., 2017),
architecture, number of parameters, and context window size. We use OpenAI
API444https://platform.openai.com/docs/overview and TogetherAI
API555https://www.together.ai/ to call GPT LLMs and non-GPT LLMs,
respectively.

###  B.2 LLM Output Parsing Code

The LLM output parsing code contains over 200 lines of regular expressions to
parse LLM outputs. The code takes into accounts the following five variables
during parsing: task type, benchmark, model, task size, and target label (for
SelectOne). This is to ensure our code can handle cases when LLMs generate
undefined labels, un-instructed explanations, or even wrong answer format
(e.g., a non-JSON output for SelectOne and SelectAll).

Overall, our code achieves about 99.9% overall parsing rate. The unparsable
cases mostly come from SingleClf outputs where LLMs output undefined labels,
such as “Mixed” or “Netual” for SST-2. For BatchClf outputs, we implement a
series of rules to extract both defined and undefined labels, because of the
order of the extracted labels affects the final evaluation. For SelectOne and
SelectAll, our code extracts JSON object, fixes cases where the JSON object
has formatting issues, or extracts a series of text indices when there is no
JSON object identified in the output.

Check https://github.com/jaaack-wang/multi-problem-eval-llm for details.

##  Appendix C Main Experiments

###  C.1 Full Results

The full results obtained from the main experiments are visualized in Fig 10.
We exclude the results of Vicuna on AGNews when task size is 100 because the
prompts exceed the model’s context length.

###  C.2 Task Complexity Hierarchy

In Table 3 from Section 5, we demonstrate the overall task complexity
hierarchy among BatchClf, SelectOne, and SelectAll. It is found that SelectOne
> SelectAll > BatchClf, where “>” denotes a “more complex than” relationship.

Similar to Fig 4, which describes the aggregate average accuracy across
benchmarks, Fig 11 describes the aggregate average accuracy across LLMs.
Although there are few cases shown in Fig 10 where the overall complexity
hierarchy among BatchClf, SelectOne, and SelectAll does not hold (e.g., Vicuna
on MRPC for task size 50), these cases are exceptional and likely due to the
interactions of multiple factors in play, such as model, benchmark, input
length, and context windows. Nevertheless, the overall task complexity
hierarchy is clear both in Fig 4 (benchmark-level) and in Fig 11 (model-
level).

Figure 10: Full average accuracy of the 7 LLMs on the 4 tasks across the 6
benchmarks. Figure 11: Aggregate average accuracy on the 4 tasks averaging
over results from the 6 benchmarks for each LLM. “Overall” presents the
average results across all LLMs for each one of the 4 task. Figure 12:
Cost/Accuracy Ratio (lower is better) for the 7 LLMs on the 6 benchmarks for
SingleClf (task size=1) and BatchClf (otherwise). The upper MRPC plot (input-
wise) is log-scale for the y-axis. Note that the scale of y-axis tends to be
smaller in the output-wise plots than the input-wise plots because the outputs
are usually much shorter than the inputs for zero-shot classification
problems.

###  C.3 Cost/Accuracy Ratio: Input-wise and Output-wise

In Fig 5 from Section 5, we calculate the cost/accuracy ratio based on both
the input and output tokens, because this is how the actual cost is billed. We
use token counts as the proxy for the actual costs to avoid the huge
discrepancies in the inference costs among LLMs (e.g., GPT-4-Turbo is 20 times
more expensive than GPT-3.5-Turbo) and for a better visualization.

To further investigate the sources of saving for multi-problem prompting, we
also visualize the input-wise and output-wise cost/accuracy ratio in Fig 12,
considering only input length and output length as proxies for costs,
respectively. As expected, the input-wise cost/accuracy ratio goes down more
dramatically and consistently than the output-wise one as the task size
increases. This is because multi-problem prompting is meant to reduce
redundancy of repeating a shared task instruction in the input rather than in
the output and the average input token count is smaller the more
classifications are performed at once.

Interestingly, we also observe an overall lower output-wise cost/accuracy
ratio across benchmarks, mostly from non-GPT models. For example, on WiC, the
output-wise cost/accuracy ratio for Llama-3 8B shows a smooth downward curve.
We note that this is possible because non-GPT models tend to generate lengthy
reasoning and explanations when not instructed to, but multi-problem prompting
somehow suppresses such a tendency.

###  C.4 Positional Errors under BatchClf

Fig 13 shows the full results regarding the positional errors the 7 LLMs make
across benchmarks and task sizes. We note that (1) distribution of the
positional errors becomes more random (or even) as the task size increases for
all LLMs; (2) in most cases, the positional errors distribute nearly randomly,
showing no evidence of obvious positional biases, if any; (3) some LLMs may
display more severe positional biases on some benchmarks with a certain task
size, such as GPT-3.5 on SST-2 with task size 50, but overall this is rare.

##  Appendix D Follow-up Experiments

###  D.1 Simplified Index Selection Tasks

Fig 14 shows the full results for the two simplified index selection tasks,
along with results for the two original selection tasks and BatchClf, based on
AGNews. Clearly, (1) all LLMs perform much better in the simplified tasks than
the original ones; (2) Mixtral 8X7B, Llama-3 70B, GPT-3.5, and GPT-4 can do
these simplified selection tasks even better than their BatchClf performance
consistently, with the latter three achieving nearly 100% accuracy in most
cases; (3) the task complexity hierarchy between SelectOne and SelectAll
nearly disappears for all LLMs except Vicua, implying that it may be
challenging for Vicuna to perform index selection tasks in general.

###  D.2 1-shot-CoT Results

Table D.4 presents the full 1-shot-CoT results across benchmarks and LLMs with
a fixed task size 10. As demonstrated in Section 6, CoT improves the LLMs’
performance in SelectOne and SelectAll with an overall larger positive effect
on the former, but the improvement is not consistent both across benchmarks
and across LLMs. For examples, LLMs tend to benefit from CoT for both
selection tasks constructed from SST-2, CoLA, and SNLI (see the “Overall”
results in Table D.4).

###  D.3 Few-shot Results

Table D.4 shows the full 2-shot results on CoLA for each LLM, which shows a
general negative effect of few-shot prompting. According to these results,
LLMs performs much worse in SelectOne with an overall accuracy going down to
25.2% from 58.1%. Similarly, the negative effect is larger on BatchClf than on
SelectAll, making the overall BatchClf accuracy lower than SelectAll, although
the former is still often higher than the latter under the same test
conditions.

###  D.4 Full Results on the Novel Benchmark

Fig 15 shows the full results on the novel benchmark created on top of SST-2.

Figure 13: Cumulative error density across positions for all benchmarks and
LLMs across different task sizes. The task size “M/N” on the left side of the
plots denotes the task size for the 3 text-pair benchmarks (i.e., MRPC, SNLI,
and WiC) and for the 3 single-text benchmarks (i.e., SST-2, CoLA, AGNews),
respectively. Figure 14: Full results for the two simplified index selection
tasks versus the original tasks based on AGNews.

| Benchmark | SST-2 | CoLA | AGNews | MRPC | SNLI | WiC
---|---|---|---|---|---|---|---
Model | Task |  |  |  |  |  |
Vicuna | SelectOne | 38.5 | 22.2 | 16.8 | 15.8 | 9.9 | 34.5
| \+ CoT | 52.6 | 53.9 | 12.1 | 12.1 | 11.5 | 12.5
| SelectAll | 66.7 | 65.7 | 42.2 | 67.4 | 28.2 | 48.8
| \+ CoT | 81.7 | 69.2 | 40.9 | 63.9 | 40.9 | 48.9
| BatchClf | 89.1 | 73.2 | 70.4 | 66.7 | 59.3 | 53.4
Mistral 7B | SelectOne | 53.3 | 52.8 | 41.5 | 36.6 | 23.3 | 22.2
| \+ CoT | 66.5 | 55.1 | 44.8 | 59.4 | 29.5 | 29.5
| SelectAll | 68.3 | 63.6 | 48.2 | 58.4 | 54.7 | 47.9
| \+ CoT | 74.4 | 68.2 | 57.0 | 68.5 | 47.6 | 47.3
| BatchClf | 79.6 | 76.8 | 78.1 | 74.0 | 69.1 | 49.6
Mixtral 8x7B | SelectOne | 76.5 | 61.0 | 46.2 | 45.2 | 23.8 | 32.0
| \+ CoT | 41.5 | 67.3 | 12.9 | 48.4 | 25.1 | 37.2
| SelectAll | 85.3 | 68.4 | 62.9 | 65.9 | 56.2 | 52.0
| \+ CoT | 49.0 | 67.4 | 57.8 | 66.5 | 58.9 | 51.1
| BatchClf | 80.2 | 80.9 | 71.6 | 70.8 | 73.3 | 63.7
Llama-3 8B | SelectOne | 67.8 | 50.8 | 55.1 | 21.7 | 21.4 | 17.3
| \+ CoT | 85.0 | 48.9 | 22.1 | 50.7 | 22.3 | 22.0
| SelectAll | 85.0 | 62.9 | 59.6 | 64.8 | 50.1 | 52.2
| \+ CoT | 85.8 | 66.5 | 45.3 | 64.5 | 39.8 | 52.3
| BatchClf | 92.0 | 76.8 | 79.5 | 62.8 | 69.2 | 59.5
Llama-3 70B | SelectOne | 83.6 | 72.7 | 67.4 | 61.0 | 32.3 | 39.3
| \+ CoT | 94.7 | 79.7 | 75.2 | 67.9 | 69.4 | 15.8
| SelectAll | 87.3 | 79.6 | 76.2 | 71.7 | 68.6 | 53.9
| \+ CoT | 96.1 | 83.7 | 85.5 | 73.9 | 77.3 | 58.4
| BatchClf | 96.4 | 82.8 | 86.8 | 73.5 | 75.2 | 61.7
GPT-3.5 | SelectOne | 71.9 | 65.8 | 47.6 | 25.2 | 28.4 | 33.8
| \+ CoT | 80.2 | 71.7 | 57.9 | 26.8 | 26.5 | 22.1
| SelectAll | 83.1 | 79.7 | 68.3 | 61.5 | 55.8 | 51.6
| \+ CoT | 89.5 | 77.5 | 58.8 | 70.2 | 50.5 | 51.2
| BatchClf | 95.0 | 81.8 | 86.4 | 52.7 | 57.4 | 58.0
GPT-4 | SelectOne | 88.6 | 80.8 | 66.4 | 58.1 | 53.8 | 50.0
| \+ CoT | 84.5 | 81.9 | 67.8 | 60.1 | 77.0 | 59.6
| SelectAll | 92.4 | 84.0 | 79.2 | 73.5 | 80.8 | 62.8
| \+ CoT | 91.3 | 83.7 | 85.4 | 76.2 | 83.3 | 71.0
| BatchClf | 96.0 | 84.0 | 86.7 | 72.6 | 83.8 | 68.4
Overall | SelectOne | 68.6 | 58.0 | 48.7 | 37.7 | 27.6 | 32.7
| \+ CoT | 72.1 | 65.5 | 41.8 | 46.5 | 37.3 | 28.4
| SelectAll | 81.2 | 72.0 | 62.4 | 66.2 | 56.3 | 52.7
| \+ CoT | 81.1 | 73.7 | 61.5 | 69.1 | 56.9 | 54.3
| BatchClf | 89.8 | 79.5 | 79.9 | 67.6 | 69.6 | 59.2

Table 8: Full 1-shot-CoT results for all benchmarks and LLMS. The task size is
fixed at 10.

| Task | SingleClf | BatchClf | SelectOne | SelectAll
---|---|---|---|---|---
Model | Benchmark |  |  |  |
Vicuna | CoLA | 78.8 | 71.0 | 48.6 | 59.6
| \+ 2-shot | 69.0 | 50.4 | 14.8 | 64.0
Mistral 7B | CoLA | 81.5 | 74.4 | 47.4 | 59.4
| \+ 2-shot | 80.2 | 65.0 | 25.0 | 60.4
Mixtral 8x7B | CoLA | 84.4 | 81.2 | 65.0 | 70.6
| \+ 2-shot | 81.8 | 73.2 | 19.8 | 72.4
Llama-3 8B | CoLA | 80.2 | 79.4 | 33.4 | 61.2
| \+ 2-shot | 74.3 | 46.6 | 0.6 | 63.8
Llama-3 70B | CoLA | 82.8 | 83.4 | 71.2 | 79.6
| \+ 2-shot | 81.8 | 82.0 | 29.2 | 80.8
GPT-3.5 | CoLA | 84.2 | 79.6 | 63.4 | 76.2
| \+ 2-shot | 79.5 | 76.8 | 38.2 | 72.8
GPT-4 | CoLA | 85.1 | 83.8 | 77.8 | 83.6
| \+ 2-shot | 85.7 | 80.8 | 48.6 | 81.0
Overall | CoLA | 82.4 | 79.0 | 58.1 | 70.0
| \+ 2-shot | 78.9 | 67.8 | 25.2 | 70.7

Table 9: Full 2-shot results on CoLA. We use a fixed task size 5 for tasks
other than SingleClf, whose task size is 1 by default. We provide the related
zero-shot results for easy comparisons.

Figure 15: Full results on the novel benchmark created on top of SST-2 where
the task objective is to decide if a text pair shares the same sentiment.