# ‘One size doesn’t fit all’: Learning how many Examples to use for In-Context
Learning for Improved Text Classification

Manish Chandra, Debasis Ganguly, Yiwen Li  Iadh Ounis
University of Glasgow
Glasgow, United Kingdom
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Predictive models in natural language processing (NLP) have evolved from
training models from scratch to fine-tuning pre-trained models with labelled
data. An extreme form of this fine-tuning involves in-context learning (ICL),
where the output of a pre-trained generative model (frozen decoder parameters)
is controlled only with variations in the input strings (called instructions
or prompts). An important component of ICL is the use of a small number of
labelled data instances as _examples_ in the prompt. While existing work uses
a static number of examples during inference for each data instance, in this
paper we propose a novel methodology of dynamically adapting the number of
examples as per the data. This is analogous to the use of a variable-sized
neighborhood in $k$-nearest neighbors ($k$-NN) classifier. In our proposed
workflow of adaptive ICL (AICL), the number of demonstrations to employ during
the inference on a particular data instance is predicted by the Softmax
posteriors of a classifier. The parameters of this classifier are fitted on
the optimal number of examples in ICL required to correctly infer the label of
each instance in the training set with the hypothesis that a test instance
that is similar to a training instance should use the same (or a closely
matching) number of few-shot examples. Our experiments show that our AICL
method results in improvement in text classification task on several standard
datasets.

‘One size doesn’t fit all’: Learning how many Examples to use for In-Context
Learning for Improved Text Classification

Manish Chandra, Debasis Ganguly, Yiwen Li and Iadh Ounis University of Glasgow
Glasgow, United Kingdom<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) have demonstrated remarkable abilities to model
text semantics in an abstract and general manner without any task specific
training Radford et al. (2018). In-context learning (ICL) makes use of this
abstract representation and knowledge representation capabilities of LLMs
Brown et al. (2020b); Arora et al. (2022); Weidinger et al. (2022) as
effective zero or few-shot predictors. For instance, such models can solve
specific tasks including assessing reviews, answering questions, recommending
relevant documents etc. by utilising only a small number of (even zero)
examples without any task-specific training Mysore et al. (2023); Ni et al.
(2021); Pradeep et al. (2023).

Figure 1: Example workflow of In-Context Learning for sentiment
classification. The illustrative example shows a sample test instance for
which a single demonstration (as retrieved from the training set) does not
result in a correct prediction (prediction workflow of the red arrows). The
example also shows that increasing the number of demonstrations from one to
two results in a correct prediction (prediction workflow of the green arrows).
We propose a method to estimate the number of examples that are likely to
yield a correct prediction.

More formally, ICL refers to the process of conditioning an LLM’s decoder
(frozen parameters) towards generating potentially relevant outputs that could
then be interpreted in the context of a specific task, e.g., words such as
‘great’ and ‘wow’ can be mapped to a positive sentiment Schick et al. (2020).
The output of an LLM’s decoder is controlled by varying the input to the LLM,
which is usually structured in the form of an instruction (the task
description), and a small number of representative examples, similar to the
instance for which prediction is to be made Ni et al. (2022). Figure 1 shows
an example of how ICL works for the downstream task of movie review sentiment
analysis. It can be seen that both demonstrations (shown in blue) and the
current instance form a part of the input to an LLM.

Researchers have worked towards adapting an ICL workflow in several ways,
which ranges from using localised examples Nie et al. (2022) instead of random
ones, to diversifying these examples Levy et al. (2023), and also ideally
ordering them Kumar and Talukdar (2021); Rubin et al. (2022).

What prior work has not addressed thus far is the effect of dynamically
selecting the number of examples used to infer the class label of a test
instance. Our idea takes motivation from information retrieval (IR), where
different queries exhibit different levels of retrieval performance mainly due
to the inherent characteristic of the information need itself Kanoulas et al.
, or how well-formulated is the query Datta et al. (2022).

Drawing the parallel that a query in IR is analogous to a test instance in ICL
and that the localised examples are potentially relevant documents Rubin et
al. (2022), we hypothesise that some test instances are associated with better
candidates for training examples (i.e., examples which are useful in the sense
that including them as a part of the prompt leads to correct predictions), and
hence including a small number of them should be adequate. On the other hand,
the retrieval quality for some test instances used as queries do not yield
good candidates, as a result of which, one needs to further look down the
ranked list to collect the useful ones Bahri et al. (2020); Ganguly and Yilmaz
(2023). The parallel with IR means that the notion of relevance of a document
to a query needs to be replaced by the downstream usefulness of an example to
a test instance.

Figure 2:  Motivation behind using a variable number of examples for ICL
across the test data points: 1) The test sample ‘?’ is located within a
homogeneous neighborhood of negative data points indicating that an LLM may
perform well with only a few nearest neighbour demonstrations. 2) The test
instance ‘?’ is located within a more heterogeneous neighborhood, as a result
of which, an LLM may require a higher number of nearest neighbour
demonstrations to correctly predict its class.

This idea of choosing a variable number of localised examples is also somewhat
similar to selecting a variable-sized neighborhood for $k$-NN classification
Zhong et al. (2017), where the key idea is that a homogeneous neighborhood is
likely to require a relatively small-sized neighborhood for a correct
prediction, and a non-homogeneous one would likely require a larger one (see
Figure 2). The same idea can be applied to ICL, where a test instance that is
similar to a number of training instances with conflicting labels may require
a higher number of examples.

We suggest a relatively simple way of finding out the ideal number of examples
for a test instance, akin to predicting the cut-off rank of a list of top-
documents retrieved for a query. This involves predicting the class posteriors
of each instance from a training set of labelled data with a progressively
increasing number of examples and then deciding on an optimal one based on
some criteria that involves checking the true label of the instance. Some
example criteria are i) the minimum number of examples leading to the correct
prediction, ii) the number of examples leading to the correct prediction with
least uncertainty, and so on. This is schematically depicted in Figure 1,
where the red path (a single example) leads to an incorrect prediction,
whereas the green path (two examples) leads to a correct prediction, and as
per the first example criterion, this value becomes the ground-truth number of
examples to be used for this instance.

The ideal number of examples to use for each instance in the training data can
then be used as samples to learn a classifier to generalise this to unseen
data, leveraging the hypothesis that similar data should use a closely
matching number of examples for effective downstream performance. Again, this
is rooted to the theory of supervised query performance prediction (QPP) in
IR, where the hypothesis is that two similar queries should yield similar
retrieval effectiveness Bahri et al. (2020); Datta et al. (2022).

In addition to increasing the prediction effectiveness of a downstream task,
dynamically choosing the number of examples can potentially lead to more
computationally efficient predictions, since using a smaller number of
examples reduces the length of the input string to an LLM thus leading to a
reduced decoding time.

####  Our contributions.

In summary, our contributions are as follows: We propose Adaptive In-Context
Learning (AICL), which for each test instance, first predicts the appropriate
number of examples (within an upper bound), and then feeds in the predicted
number of most similar examples into an LLM for the downstream prediction. Our
experiments on text classification show that AICL is more effective than
static ICL (SICL), i.e., $k$-shot for fixed values of $k$ both in terms of
prediction effectiveness and run-time efficiency.

##  2 Related Work

####  Prompt tuning and searching.

Previous research studies the problem of constructing an appropriate prompt
typically from two perspectives: (1) prompt tuning in the embedding space Li
and Liang (2021); Liu et al. (2022b); Qin and Eisner (2021); Liu et al.
(2023), and (2) prompt searching in the text space Lu et al. (2022); Zhang et
al. (2023); Diao et al. (2023); Liu et al. (2022a); Shi et al. (2023). Prompt
tuning is a lightweight alternative to fine-tuning, which keeps language model
parameters frozen and instead optimizes a sequence of continuous task-specific
vectors. The key idea of prompt tuning is to inject task-specific embedding
into hidden layers and then tune these embeddings using gradient-based
optimization. However, these methods require to modify the original inference
process of the model, which is impractical for the case of black-box LLM
services such as GPT3 Brown et al. (2020a). Furthermore, prompt tuning
introduces additional computational and storage costs, which is typically
expensive for LLMs. A more feasible and efficient way is to optimize prompting
via searching approximate demonstration samples and ordering them in the
original text space.

Figure 3: Schematic diagram of Adaptive In-Context Learning (AICL) workflow.

####  In-context Learning (ICL).

LLMs, when scaled from millions to billions of parameters, have been
demonstrated to be adaptable to a broad set of tasks due to instruction tuning
Ouyang et al. (2022); Brown et al. (2020b), in the sense that they are not
only able to produce semantically correct and coherent text, but are also able
to adapt themselves surprisingly well with small changes in contexts supplied
as inputs, commonly called prompts Arora et al. (2022).

In ICL, a small number of labeled examples from a training set are appended to
a prompt instruction to control the text generation of LLM in a desirable way
that is beneficial to the downstream task Mysore et al. (2023); Li et al.
(2022); Ni et al. (2021); Pradeep et al. (2023). In addition to leveraging ICL
for a purely generative task, e.g., question answering or abstractive
summarisation Brown et al. (2020a); Li et al. (2023); Tang et al. (2023), a
more common use is in a predictive task, such as text classification Lu et al.
(2022); Milios et al. (2023); Wei et al. (2021), where each class is specified
by a set of words, commonly called a verbaliser Schick and Schütze (2021).
Once each class for a predictive task is well-defined, the generated text can
be mapped to the most likely class by using the probabilities of the tokens
generated by the decoder.

####  Demonstration selection for ICL.

ICL has advanced the utilization of LLMs for task-specific applications with
minimal examples, forming a solid foundation for subsequent investigations
into optimizing this learning paradigm Dong et al. (2022). A crucial aspect of
ICL is the adept selection and utilization of the demonstrations for task
comprehension. Recent work has explored the dynamics of in-context examples,
elucidating how ICL enhances the language model’s performance across various
NLP tasks by providing a minimal set of examples at inference time Han et al.
(2023). Furthermore, the efficacy of ICL is significantly shaped by the
strategies for selecting and ordering demonstrations. For instance, Luo et al.
(2023) expands the applicability of retrieval-based ICL approaches by
demonstrating that even simple word-overlap similarity measures such as BM25
outperform randomly selected demonstrations. Methods such as KATE Liu et al.
(2022a) show that localised examples work better than random ones.

Different from existing approaches, we propose to modify a standard ICL
workflow by employing a variable number of examples.

##  3 Proposed Methodology

###  3.1 Standard In-Context Learning (ICL)

In-context learning (ICL), unlike supervised learning, does not involve
training a set of parameters $\theta$ on labeled examples. Rather, the
posteriors are now a function of the following: a) text of the input test
instance, b) the decoder parameters of a pre-trained LLM, c) a prompt
instruction, and d) optionally, a set of $k$ input examples (commonly called
$k$-shot learning). Formally,

$P(y|\mathbf{x})=f(\mathbf{x},\pazocal{N}_{k}(\mathbf{x});\phi_{\text{LLM}}),$
(1)

where, different from a supervised setup, the function $f$ does not have a
parameterized representation that can be learned using a training set with
gradient descent. The function itself depends on the pre-trained parameters
$\phi_{\text{LLM}}$ of an LLM, the current inputs for which a label is to be
predicted, and a prompt comprising a set of $k$ text units denoted by
$\pazocal{N}_{k}(\mathbf{x})$.

Since the decoder of an LLM generates a sequence of words of the form of
$w_{1},\ldots,w_{l}$ ($l$ being the maximum length of a sequence), the class
posterior likelihoods are computed by mapping, say $p$ possible classes into
$p$ different equivalent sets of words, say $V(y)$, where $y\in\mathbb{Z}_{p}$
– these sets are commonly called verbalisers Hu et al. (2021). For instance,
for a binary classification problem (e.g., that of a movie review as shown in
Figure 1), $p=2$ (i.e., $y\in\\{0,1\\}$), and a reasonable way to define the
verbaliser sets could be via the following words:
$V(0)=\\{\text{`bad'},\text{`negative'}\\}$, and
$V(1)=\\{\text{`good'},\text{`positive'}\\}$.

One of the most important components of ICL is the search component that
outputs the top-$k$ candidate set of similar instances from the training set,
i.e., $\pazocal{N}_{k}(\mathbf{x})$ of Equation (1). Although, in principle,
it is possible to include random examples from the training set in the prompt,
it has been shown that localised examples (i.e., examples that are topically
similar to the current instance) yield a better performance Liu et al.
(2022a); Luo et al. (2024). Let us denote the $k$-neighborhood of similar
localised examples for a data instance $\mathbf{x}$ as
$\pazocal{N}_{k}(\mathbf{x})=\\{(\mathbf{z}_{1},y(\mathbf{z}_{1})),(\mathbf{z}_{2},y(\mathbf{z}_{2})),\ldots,(\mathbf{z}_{k},y(\mathbf{z}_{k}))\\}$,
which are then concatenated together to constitute a context of the prompt to
an LLM. In our experiments, we employ BM25 Robertson and Jones (1976) for the
neighborhood computation.

###  3.2 Adaptive ICL

We call this methodology of using a variable number of demonstrations for ICL
inference by the name ‘Adaptive In-Context Learning’, or AICL for short. The
idea of AICL centres around choosing the context $\pazocal{N}_{k}(\mathbf{x})$
in a data-driven manner, i.e., making $k$ a function of the data (the current
instance $\mathbf{x}$) itself. This is analogous to choosing a different value
of $k$ for a $k$-NN based non-parametric model or choosing a different rank
cut-off for top-retrieved set of documents for different queries Bahri et al.
(2020); Ganguly and Yilmaz (2023). The motivation is that classifying some
instances would be more difficult than others, in which cases they are
potentially to be benefited from a larger value of $k$ (more context). On the
other hand, for relatively easy data instances using too much context may be
detrimental for effective prediction.

Formally speaking, the difference of AICL with that of ICL (Equation (1)) is
that the value $k$, indicating the size of the neighborhood, is no longer a
constant. Instead, we denote it by a parameterised function
$\kappa(\mathbf{x})$ such that

$P(y|\mathbf{x})=f(\mathbf{x},\pazocal{N}_{\kappa(\mathbf{x})}(\mathbf{x});\phi_{\text{LLM}}),$
(2)

where $\kappa:\mathbf{x}\mapsto\\{0,\ldots,M\\}$, and $M$ being an upper bound
on the number of example instances.

A schematic diagram depicting the overarching idea is shown in Figure 3.
Before describing the details of each component, we now briefly explain the
idea as follows: in contrast to ICL, AICL involves an additional phase of
training a classifier, $\kappa(\mathbf{x})$ of Equation 2, to predict an ideal
number of examples from each training instance (the top rectangle in Figure
3). The bottom rectangle shows the inference phase, where, given a test
instance, we first predict the number of examples to be used and then follow
the standard ICL procedure with those many examples fed as the part of the
additional context to an LLM.

####  Selecting the number of examples .

Instead of devising a heuristic to predict the number of training examples to
use for a test instance $\mathbf{x}$, a supervised approach can be applied to
solve this as a classification problem.

We employ a supervised approach for predicting the number of examples for each
test instance. Information from a training set $\pazocal{T}$, of labelled
examples can potentially be used to estimate the ideal number of examples for
a given data instance. This is carried out as follows for each training set
instance $\mathbf{x}\in\pazocal{T}$. We first execute $k$-shot inference with
progressively increasing values of $k$ within $\\{0,\ldots,M\\}$, where $M$ is
an upper bound on the number of examples. This is depicted by the
progressively increasing dotted regions within the top rectangle of Figure 3.

After we obtain the posteriors $P(\hat{y}|\mathbf{x})$ for an instance
$\mathbf{x}$ of the training set, we need to select the ideal number. This we
decide by employing two different heuristic. Obviously, as the first filtering
criterion, we select only those values of $k$ that lead to correct predictions
(since $\mathbf{x}$ is a training set instance, its true label $y(\mathbf{x})$
is known to us), i.e., we first obtain a candidate set as

$\pazocal{K}(\mathbf{x})=\\{k:y(\mathbf{x})=\operatorname*{arg\,max}f(\mathbf{x},\pazocal{N}_{k}(\mathbf{x};\phi_{\text{LLM}}))\\},$
(3)

where
$\operatorname*{arg\,max}f(\mathbf{x},\pazocal{N}_{k}(\mathbf{x};\phi_{\text{LLM}}))$
is the most likely class from the posterior distribution obtained from an LLM
$\phi$ (see Equation 1). Note that it is possible that the candidate set
$\pazocal{K}=$ which can happen if all the predictions for $\mathbf{x}$ are
wrong. In such a case, we simply set $\pazocal{K}(\mathbf{x})=\\{M\\}$, i.e.,
we select the upper bound of examples as the only candidate.

We then apply two different heuristics on the candidate set obtained via
Equation 3 to converge on a single value as the ground-truth label for
$\mathbf{x}$. The first of this heuristics simply chooses the minimum of this
set, i.e.,

$k^{*}(\mathbf{x})=\min_{k}\pazocal{K}(\mathbf{x}).$ (4)

The second heuristic allows provision to consider the confidence of
classification, similar to Sorensen et al. (2022). More specifically,

$k^{*}(\mathbf{x})\\!\\!=\\{k\in\pazocal{K}(\mathbf{x})\\!\\!:\hat{y}(\mathbf{x},k)>\hat{y}(\mathbf{x},j)\,\forall
j\in\pazocal{K}(\mathbf{x})\\},$ (5)

where
$\hat{y}(\mathbf{x},j)=\operatorname*{arg\,max}f(\mathbf{x},\pazocal{N}_{j}(\mathbf{x};\phi_{\text{LLM}}))$
denotes a short notation for the most likely class predicted with $j$
examples.

Input: $\pazocal{T}$ – a training set of labelled instances with $p$ classes

Output: $\pazocal{K}=\cup_{\mathbf{x}\in\pazocal{T}}k^{*}(\mathbf{x})$ –
Number of examples yielding the most confident and correct predictions for
each instance $\mathbf{x}\in\pazocal{T}$

begin

$\pazocal{K}\leftarrow\phi$

for  _$\mathbf{x}\in\pazocal{T}$_ do

max_conf $\leftarrow$ 0; $k^{*}\leftarrow 0$

for  _$j\leftarrow 0$ to $M$_ do

$\Delta_{p}\leftarrow$ LLM-predict($\mathbf{x},j$)

$\hat{y}(\mathbf{x})\leftarrow\text{argmax}\Delta_{p}$

conf$\leftarrow\Delta_{\hat{y}(\mathbf{x})}\mathbb{I}(\hat{y}(\mathbf{x})=y(\mathbf{x}))$

if  _conf $>$ max_conf_ then

max_conf $\leftarrow$ conf

$k^{*}\leftarrow j$

$\pazocal{K}\leftarrow\pazocal{K}\cup k^{*}$

return _$\pazocal{K}$_

Algorithm 1 Optimal number of examples

####  Learning to predict the number of examples.

Algorithm 1 shows the pseudo-code to obtain the ground-truth labels (ideal
number of ICL examples) for the entire training set involving the more complex
heuristic of Equation 5. After executing Algorithm 1, we obtain a set of
ground-truth labels $\pazocal{K}$. This we then utilise, generally speaking,
to train a classifier by optimising the cross-entropy loss $\pazocal{L}$:

$\theta=\text{argmin}_{\theta}\sum_{\mathbf{x}\in\pazocal{T},k^{*}\in\pazocal{K}}\pazocal{L}(\mathbf{x}^{\mathrm{T}}\theta,k^{*}).$
(6)

Equation 6 leverages the underlying hypothesis that with enough training data
for the appropriate number of ICL examples to use over a broad range of
topical content, we should be able to learn to predict the likely number of
ICL examples to use for an unseen text (but similar topic seen during
training) during inference time.

In particular for our experiments, the parameterised classifier of Equation 6
corresponds to that of a standard BERT model’s classification head.

####  Inference.

During inference, for each instance $\mathbf{x}\in\pazocal{E}$, we apply the
classifier $\theta:\mathbf{x}\mapsto\\{0,\ldots,M\\}$ as trained by Equation
(6) to predict the number of examples $\kappa(\mathbf{x})$. Consequently, we
execute a $\kappa(\mathbf{x})$-shot prediction on $\mathbf{x}$, as shown in
the bottom rectangle of Figure 3.

##  4 Evaluation

###  4.1 Datasets

We conduct experiments on three text classification datasets Wang et al.
(2018) that recent works have used Ma et al. (2023); Zhang et al. (2022b),
namely AGNews Del Corso et al. (2005), Jigsaw Toxic Comment jig (2017) and
SST2 Socher et al. (2013).

####  AGNews.

AGNews is a topic classification dataset constituting news articles from the
Web. Each document in the dataset belongs to one of the following 4 classes:
World, Sports, Business, and Sci/Tech. The total number of training instances
is $120,000$, while the test set size is $7,600$. Each class contains $30,000$
samples from the train set and $1,900$ instances from the test set.

####  Jigsaw Toxic Comment.

Due to its societal impact, toxicity prediction is a problem of considerable
practical interest. This dataset, released by Jigsaw and Google as a part of a
Kaggle competition, comprises of comments extracted from Wikipedia’s talk
page, each being annotated by human evaluators across six categories
representing toxic behaviors: toxic, ‘severe toxic’, obscene, threat, insult,
and ‘identity hate’. We group together the toxic behaviours into one category
toxic and thus end up with a binary classification task. Henceforth, we refer
to this dataset as ‘Toxicity’.

####  SST2.

The Stanford Sentiment Treebank (SST) is a corpus with fully labeled parse
trees that allows for a complete analysis of the compositional effects of
sentiment in a language. The corpus consists of $11,855$ sentences extracted
from movie reviews. Being parsed with the Stanford parser, it constitutes a
total of $215,154$ unique phrases from the parse trees, each annotated by 3
human judges. The SST2 (also called SST-binary) dataset is a subset of SST,
specifically prepared for the task of binary classification. More precisely,
neutral sentences from SST were discarded, and two level, each for the
negative and the positive classes were merged thus yielding two classes in
total.

Dataset |  Instruction |  Verbaliser
---|---|---
AGNews |  The topic of this news is <MASK> |  [world], [sports], [trade], [technology, science]
Toxicity |  This sentence is <MASK> |  [non-toxic, harmless, benign],[toxic, harmful, malignant]
SST2 |  This movie review is <MASK> |  [negative], [positive]
Table 1: Instruction templates and verbalisers used in our experiments for
each dataset. Table 2:  A comparison of our proposed adaptive ICL vs. the
baselines on several datasets. $k$: #few-shot examples for baselines. For
AICL, this denotes the average number of examples used. AIS: average input
size (#tokens) rounded off to the nearest integer. RRT: run-time relative to
$5$-shot (lower the RRT, the faster is the inference).

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.7326 0.7222
0.7103 60 0.16 RICL 1 0.4154 0.4211 0.4145 123 0.25 RICL 3 0.6023 0.6028
0.5982 246 0.42 RICL 5 0.6548 0.6401 0.6395 370 0.61 SICL 1 0.9016 0.9017
0.9016 124 0.62 SICL 3 0.9058 0.9046 0.9044 252 0.80 SICL 5 0.8994 0.8945
0.8941 380 1.00 AICL-L 1.06 0.8907 0.8885 0.8883 129 0.64 AICL-U 1.91 0.9109
0.9088 0.9097 178 0.71 Oracle 1.08 0.9470 0.9467 0.9467 130 0.65

(a) OPT-2.7B on AGNews

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.6569 0.5932
0.5849 60 0.25 RICL 1 0.4585 0.4650 0.4522 123 0.31 RICL 3 0.6338 0.6189
0.5916 246 0.45 RICL 5 0.6746 0.6547 0.6354 370 0.58 SICL 1 0.9015 0.9017
0.9016 124 0.70 SICL 3 0.9006 0.8993 0.8985 252 0.86 SICL 5 0.8974 0.8938
0.8925 380 1.00 AICL-L 1.05 0.8725 0.8716 0.8709 128 0.71 AICL-U 1.87 0.9080
0.9096 0.9067 175 0.79 Oracle 1.08 0.9521 0.9518 0.9517 130 0.71

(b) GPT-J on AGNews

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.5498 0.5767
0.5553 103 0.17 RICL 1 0.5193 0.5424 0.5022 199 0.28 RICL 3 0.5250 0.5529
0.5129 367 0.50 RICL 5 0.5262 0.5562 0.5132 488 0.72 SICL 1 0.5860 0.6938
0.5876 195 0.75 SICL 3 0.5961 0.6889 0.6091 335 0.88 SICL 5 0.6023 0.6915
0.6184 431 1.00 AICL-L 1.24 0.5983 0.6881 0.6128 223 0.79 AICL-U 4.12 0.6164
0.6693 0.6223 398 0.93 Oracle 1.32 0.7939 0.8715 0.8263 231 0.80

(c) OPT-2.7B on Toxicity

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.5689 0.6238
0.5769 103 0.22 RICL 1 0.5222 0.5573 0.4837 206 0.31 RICL 3 0.5221 0.5485
0.5153 410 0.48 RICL 5 0.5272 0.5642 0.5052 610 0.65 SICL 1 0.5751 0.6967
0.5490 203 0.69 SICL 3 0.6169 0.7223 0.6367 391 0.85 SICL 5 0.6078 0.7248
0.6217 431 1.00 AICL-L 1.26 0.6020 0.7159 0.6134 227 0.75 AICL-U 3.46 0.6197
0.7237 0.6401 359 0.90 Oracle 1.32 0.7939 0.8715 0.8263 231 0.76

(d) GPT-J on Toxicity

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.7541 0.5170
0.3698 30 0.32 RICL 1 0.8601 0.8260 0.8216 61 0.44 RICL 3 0.8465 0.8314 0.8295
124 0.68 RICL 5 0.8988 0.8886 0.8877 187 0.91 SICL 1 0.8495 0.8150 0.8102 60
0.56 SICL 3 0.8413 0.8270 0.8251 120 0.78 SICL 5 0.8947 0.8885 0.8880 180 1.00
AICL-L 1.11 0.8700 0.8600 0.8589 65 0.59 AICL-U 4.93 0.9061 0.9013 0.9012 178
0.98 Oracle 1.22 0.9635 0.9616 0.9615 67 0.60

(e) OPT-2.7B on SST2

Settings Evaluation Method $k$ Precision$\uparrow$ Recall$\uparrow$
F-score$\uparrow$ AIS$\downarrow$ RRT$\downarrow$ 0-shot 0 0.7503 0.5022
0.3379 30 0.54 RICL 1 0.8277 0.8258 0.8256 61 0.59 RICL 3 0.9181 0.9159 0.9158
124 0.74 RICL 5 0.9278 0.9269 0.9269 187 0.91 SICL 1 0.8420 0.8412 0.8411 60
0.69 SICL 3 0.9151 0.9148 0.9148 120 0.84 SICL 5 0.9261 0.9253 0.9252 180 1.00
AICL-L 1.09 0.8907 0.8885 0.8883 64 0.70 AICL-U 4.12 0.9302 0.9304 0.9302 154
0.90 Oracle 1.18 0.9792 0.9791 0.9791 66 0.71

(f) GPT-J on SST2

Figure 4:  Sensitivity of AICL (LLM: OPT-2.7B) to the size of training data
used to learn the number of examples on the AGNews, Toxicity and SST2 datasets
(left to right). The red line indicates the best performing baseline (SICL).

###  4.2 Methods Investigated

####  Variants of AICL.

We employ the following variants for our proposed methodology of employing a
variable number of ICL examples.

  * •

AICL-L: This applies the heuristic that trains the $\kappa(\mathbf{x})$
classifier by considering the ideal number of examples to be the smallest
number of ICL examples that lead to correct predictions in the training set
(see Equation 4).

  * •

AICL-U: This applies the confidence-based heuristic of Equation 5 to train the
classifier for predicting the number of ICL examples.

We compare both the variants of AICL with the following baselines:

  * •

0-shot: This approach tests the model’s innate capabilities, i.e., operating
without any reference to training data.

  * •

Static ICL (SICL): This refers to the standard method of feeding in a fixed
number of examples as input Liu et al. (2022a). For a fair comparison with the
AICL methods, we report the results obtained with three different values of
$k$: $1$, $\lceil\frac{M}{2}\rceil$ and $M$ representing the most conservative
(in terms of the input size), average, and least conservative situations. For
our experiments, $M$, the upper bound on the number of ICL examples, is 5.
This means that our standard ICL experiments operate with the 1-shot, 3-shot
and 5-shot settings.

  * •

RICL: This is ICL with a fixed number of random examples sampled from the
training set. Similar to SICL, we employ three different settings of $k$ \-
$1$, $\lceil\frac{M}{2}\rceil$ and $M$.

###  4.3 Model and hyper-parameter settings

Among a relatively large number of available choices for available LLMs, we,
in particular, conduct our experiments on GTP-J Wang (2021); Su et al. (2023)
and OPT-2.7B Zhang et al. (2022a). OPT is a collection of open-source auto-
regressive language models ranging in size from 125M to 175B parameters under
a decoder-only structure akin to GPT-3 Brown et al. (2020a). OPT-175B is
comparable to GPT-3, while requiring only $1/7^{th}$ of the carbon footprint.
GPT-J is another open-source GPT-3-like model trained on the Pile dataset Gao
et al. (2020). GPT-J-6B reportedly yields performance comparable to the 6.7
billion parameter GPT-3 on a variety of tasks Wang (2021). For both these
LLMs, the max context length is 2048.

In our experiments, we set $M$ \- the maximum number of examples, to $5$.
Table 1 enlists the task-specific instructions that we use for each dataset.

####  Evaluation metrics.

As effectiveness measures, we report the macro-averaged precision, recall and
F-scores. Additionally, as efficiency measures we report the average input
size (AIS) in terms of the number of tokens supplied as input to the LLM for
the predictions (we compute the average over the test set and then round-off
to the nearest integer). We also measure the relative run-time (RRT) benefits
gained relative to executing $M$-shot predictions. The lower the RRT ratio,
the faster is the inference.

###  4.4 Results

####  Main observations.

Table 2 shows a comparison between the different ICL approaches on the three
datasets. For the experiments involving random sampling of demonstrations
(RICL), the evaluation metrics were obtained by averaging over $3$ runs.

It can be seen that AICL-U outperforms all the baselines across the three
datasets. The likely reason it outperforms the best baseline (SICL) is that
AICL-U is able to effectively adapt the number of examples to use, thereby
preventing itself from the degradation effects of non-relevant (not useful)
examples. In effect, it learns a latent relationship between the topical
content and the quantity of context required to guide the decoder’s output in
the right direction effectively.

The average values of $k$ reported for the AICL approaches show that it is
able to estimate the optimal number of ICL examples without conducting a grid
search on a dataset, e.g., Table 2(e) shows that AICL-U outperforms SICL with
3 examples (the best setting for SICL) without having any prior knowledge on
the optimal number of examples that should be used for this particular
dataset. The practical implication is that AICL can automatically adapt itself
to a particular domain without requiring any domain-specific tuning.

Moreover, AICL-U is able to operate more effectively with smaller input sizes
(see the average value of $k$ and also the AIS values), which means that it is
computationally faster (see the RRT values) as compared to SICL with a closely
matching number of examples. We observe that the variant AICL-L is worse than
its AICL-U counterpart. This indicates that there the posteriors provide a
useful signal (in terms of classifier confidence) that should be used for more
effective results.

As a part of the analysis, we report the performance of an oracle method (with
an access to the test set labels). It chooses the smallest value of $k$ that
results in the correct prediction, which means that it provides an upper bound
both in terms of effectiveness and efficiency. AICL does not substantially lag
behind the oracle measures, specifically on the topic classification and movie
review detection tasks (AGNews and SST2, respectively). A relatively large
margin between AICL’s and the oracle’s performance on toxicity classification
indicates that it is a more challenging task perhaps due to the inherent human
subjectivity of what is a toxic comment.

####  Practical considerations for training $\kappa$.

LLM inferences on the entire training data for learning the $\kappa$
classifier (Equation 6) may not be practical for large training data. An
interesting question is thus to investigate whether the $\kappa$ classifier
can be effectively trained on smaller data. To this end, we run a set of
experiments using the OPT-2.7B model with different proportions of the
training data, with the results shown in Figure 4. It can be observed, in
general, that 80% of the training data suffices to outperform SICL (the red
baseline).

##  5 Conclusion and Future work

We proposed a novel supervised learning approach of predicting the number of
demonstrations to be included in the prompt to an LLM with an objective to
improve the downstream task effectiveness. Our experiments showed that using a
data-driven number of demonstrations yields results superior to that of using
a constant number of examples.

Future work will explore alternative ways of adapting an ICL pipeline in a
data-driven manner, e.g., adapting the verbaliser and the prompt templates.

## References

  * jig (2017) 2017\.  Toxic comment classification challenge.
  * Arora et al. (2022) Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022.  Ask me anything: A simple strategy for prompting language models.
  * Bahri et al. (2020) Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, and Andrew Tomkins. 2020.  Choppy: Cut transformer for ranked list truncation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 1513–1516, New York, NY, USA. Association for Computing Machinery.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Brown et al. (2020b) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Datta et al. (2022) Suchana Datta, Debasis Ganguly, Derek Greene, and Mandar Mitra. 2022.  Deep-qpp: A pairwise interaction-based deep learning model for supervised query performance prediction.  In _WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 - 25, 2022_ , pages 201–209. ACM.
  * Del Corso et al. (2005) Gianna M. Del Corso, Antonio Gullí, and Francesco Romani. 2005.  Ranking a stream of news.  In _Proceedings of the 14th International Conference on World Wide Web_ , WWW ’05, page 97–106, New York, NY, USA. Association for Computing Machinery.
  * Diao et al. (2023) Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023.  Active prompting with chain-of-thought for large language models.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Ganguly and Yilmaz (2023) Debasis Ganguly and Emine Yilmaz. 2023.  Query-specific variable depth pooling via query performance prediction.  In _SIGIR_ , pages 2303–2307. ACM.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Han et al. (2023) Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023.  Understanding in-context learning via supportive pretraining data.  _arXiv preprint arXiv:2306.15091_.
  * Hu et al. (2021) Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2021.  Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification.  _arXiv preprint arXiv:2108.02035_.
  * (14) Evangelos Kanoulas, Ben Carterette, Paul D. Clough, and Mark Sanderson.  Evaluating multi-query sessions.  In _Proc. SIGIR 2011_ , pages 1053–1062.
  * Kumar and Talukdar (2021) Sawan Kumar and Partha P. Talukdar. 2021.  Reordering examples helps during priming-based few-shot learning.  In _ACL/IJCNLP (Findings)_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 4507–4518. Association for Computational Linguistics.
  * Levy et al. (2023) Itay Levy, Ben Bogin, and Jonathan Berant. 2023.  Diverse demonstrations improve in-context compositional generalization.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1401–1422, Toronto, Canada. Association for Computational Linguistics.
  * Li et al. (2022) Minghan Li, Xueguang Ma, and Jimmy Lin. 2022.  An encoder attribution analysis for dense passage retriever in open-domain question answering.  In _Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)_ , pages 1–11, Seattle, U.S.A. Association for Computational Linguistics.
  * Li et al. (2023) Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023.  Few-shot in-context learning on knowledge base question answering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6966–6980, Toronto, Canada. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2022a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Liu et al. (2022b) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022b.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023.  Gpt understands, too.  _AI Open_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Luo et al. (2023) Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. 2023.  Dr. icl: Demonstration-retrieved in-context learning.  _arXiv preprint arXiv:2305.14128_.
  * Luo et al. (2024) Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024.  In-context learning with retrieved demonstrations for language models: A survey.
  * Ma et al. (2023) Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. 2023.  Fairness-guided few-shot prompting for large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)_.
  * Milios et al. (2023) Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. 2023.  In-context learning for text classification with many labels.  In _Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP_ , pages 173–184, Singapore. Association for Computational Linguistics.
  * Mysore et al. (2023) Sheshera Mysore, Andrew Mccallum, and Hamed Zamani. 2023.  Large language model augmented narrative driven recommendations.  RecSys ’23, page 777–783, New York, NY, USA. Association for Computing Machinery.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ni et al. (2021) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021.  Large dual encoders are generalizable retrievers.
  * Nie et al. (2022) Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022.  Improving few-shot performance of language models via nearest neighbor calibration.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Pradeep et al. (2023) Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023.  How does generative retrieval scale to millions of passages?
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5203–5212, Online. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding with unsupervised learning.
  * Robertson and Jones (1976) S. E. Robertson and Sparck K. Jones. 1976.  Relevance weighting of search terms.  _Journal of the American Society for Information Science_ , 27(3):129–146.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
  * Schick et al. (2020) Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020.  Automatically identifying words that can serve as labels for few-shot text classification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5569–5578, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023.  Large language models can be easily distracted by irrelevant context.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Sorensen et al. (2022) Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022.  An information-theoretic approach to prompt engineering without ground truth labels.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 819–862, Dublin, Ireland. Association for Computational Linguistics.
  * Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023.  Roformer: Enhanced transformer with rotary position embedding.
  * Tang et al. (2023) Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023.  In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis.  In _Proceedings of the 4th New Frontiers in Summarization Workshop_ , pages 56–67, Singapore. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang (2021) Ben Wang. 2021.  Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wei et al. (2021) Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. 2021.  Few-shot text classification with triplet networks, data augmentation, and curriculum learning.  _arXiv preprint arXiv:2103.07552_.
  * Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022.  Taxonomy of risks posed by language models.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 214–229.
  * Zhang et al. (2022a) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a.  Opt: Open pre-trained transformer language models.
  * Zhang et al. (2022b) Yiming Zhang, Shi Feng, and Chenhao Tan. 2022b.  Active example selection for in-context learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9134–9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023.  Automatic chain of thought prompting in large language models.  In _The Eleventh International Conference on Learning Representations (ICLR 2023)_.
  * Zhong et al. (2017) Xiao-Feng Zhong, Shi-Ze Guo, Liang Gao, Hong Shan, and Jing-Hua Zheng. 2017.  An improved k-nn classification with dynamic k.  In _Proceedings of the 9th International Conference on Machine Learning and Computing_ , ICMLC ’17, page 211–216, New York, NY, USA. Association for Computing Machinery.