# Quality-Diversity through AI Feedback

Herbie Bradley1,2,3∗ Andrew Dai4∗ Hannah Teufel4 Jenny Zhang5,6
Koen Oostermeijer4 Marco Bellagente7† Jeff Clune5,6,8 Kenneth Stanley9
Grégory Schott4 Joel Lehman10
1CarperAI 2CAML Lab, University of Cambridge 3EleutherAI 4Aleph Alpha
5Department of Computer Science, University of British Columbia
6Vector Institute 7Stability AI 8Canada CIFAR AI Chair 9Maven 10Stochastic
Labs

###### Abstract

In many text-generation problems, users may prefer not only a single response,
but a diverse range of high-quality outputs from which to choose. Quality-
diversity (QD) search algorithms aim at such outcomes, by continually
improving and diversifying a population of candidates. However, the
applicability of QD to qualitative domains, like creative writing, has been
limited by the difficulty of algorithmically specifying measures of quality
and diversity. Interestingly, recent developments in language models (LMs)
have enabled guiding search through _AI feedback_ , wherein LMs are prompted
in natural language to evaluate qualitative aspects of text. Leveraging this
development, we introduce Quality-Diversity through AI Feedback (QDAIF),
wherein an evolutionary algorithm applies LMs to both generate variation and
evaluate the quality and diversity of candidate text. When assessed on
creative writing domains, QDAIF covers more of a specified search space with
high-quality samples than do non-QD controls. Further, human evaluation of
QDAIF-generated creative texts validates reasonable agreement between AI and
human evaluation. Our results thus highlight the potential of AI feedback to
guide open-ended search for creative and original solutions, providing a
recipe that seemingly generalizes to many domains and modalities. In this way,
QDAIF is a step towards AI systems that can independently search, diversify,
evaluate, and improve, which are among the core skills underlying human
society’s capacity for innovation.111Project Page: https://qdaif.github.io/

**footnotetext: Equal contribution. Order chosen in support of student
authorship. Author contributions listed [here].
Correspondence<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work done while at Aleph Alpha

##  1 Introduction

Human innovation is not only a generative capacity for creativity, but also
includes the ability to evaluate the subjective quality of new ideas and
artifacts. Great ideas are rarely generated all at once out of whole cloth,
but rather gradually emerge through divergent chains of elaboration and
revision (Stanley & Lehman, 2015). To successfully navigate such a tree of
ideas, the creator must evaluate which steps in a chain are worth pursuing
further, a question that can be highly subjective, especially in domains with
artistic or literary dimensions.

Until now, even if AI could provide candidates, the hope for such subjectively
tinged evaluation lay firmly with humans. However, the emerging foundation
model technology of recent years (Bommasani et al., 2021) now means that the
model can also play the role of evaluator, even when the evaluation is in part
subjective (Madaan et al., 2023). In this way, for the first time, an entire
ideation process that returns a diverse set of interesting artifacts can in
principle be automated. This process cannot be run by LMs entirely on their
own, but requires chaining together a search algorithm with model calls in a
nuanced way. This paper highlights one way to achieve this potential: to
combine LMs with the field of quality-diversity (QD) (Mouret & Clune, 2015),
which centers on how to design search processes that produce high-quality
solutions that span a design space.

The main insight in QD algorithms is to explicitly maintain and seek high-
quality diverse responses. Typically such search algorithms require hand-
designed measures of diversity and quality, as well as a way to generate
meaningful variation. Yet the most interesting and complex domains nearly
always involve notions of performance, diversity, and variation that are
subjective or difficult to specify algorithmically. Extending work that
generates variation through LMs (Lehman et al., 2022; Meyerson et al., 2023)
and evaluates the quality of potential solutions through LMs (Ahn et al.,
2022), we show that LMs can also be used to evaluate qualitative aspects of
diversity. In this way, LMs can instantiate the three main ingredients of QD
search, thereby enabling powerful new QD algorithms that can ride the
coattails of continual LM advances, which we name Quality-Diversity through AI
Feedback (QDAIF). Such QDAIF can explore and return diverse, high-quality
responses to an LM prompt, without relying on hand-crafted diversity measures
or the necessity for model fine-tuning (although, it could also be used for
LMs to self-improve by generating fine-tuning data (Lehman et al., 2022; Chen
et al., 2023), an interesting direction for self-curated effective learning
environments via generated data, towards AI-generating algorithms (Clune,
2019)).

We evaluate QDAIF across three creative writing domains: opinion writing,
short stories, and poetry. The idea is that in such creative domains, users
often enjoy seeing a wide range of possible stories or poems from which to
choose or be inspired by. Quantitative results indicate that QDAIF
significantly outperforms existing baselines. Additionally, through human
evaluation, we observe a strong alignment between human and AI-generated
feedback, providing empirical evidence that AI feedback is grounded and that
the method can work in practice (i.e. it yields improved quality and diversity
as measured by humans). Overall, QDAIF brings us a step closer to AI models
that can independently search and innovate, one of the keystone abilities of
humans that allow them to create culture and science (Stanley et al., 2017).

Figure 1: QDAIF (left) covers more the search space with diverse, high-quality
stories compared to the baseline (right). The baseline is LMX Quality-Only
(Meyerson et al., 2023), which optimizes only for the quality of solutions.
QDAIF discovered more interesting stories about a spy and a politician,
covering examples such as romance stories with a happy-ending, to horror
stories with a tragic-ending. The baseline produced a story (right-middle
position, starting with "Jason") with a lower quality score due to the lack of
a desired spy character (denoted by the red-colored bin, for a story with a
neutral ending, and leaning to horror). QDAIF discovered a better, more-
relevant story (bottom-middle position, starting with "a wealthy politician")
for this same neutral bin.

##  2 Background & Related Work

###  2.1 Evolution through Large Models

Advancements in language models have enabled new kinds of powerful search
algorithms that apply LMs as search operators, e.g. to create variation or
evaluate solutions. While other search algorithms could also be used, this
paper creates a QDAIF algorithm by extending upon Evolution through Large
Models (ELM) (Lehman et al., 2022), a framework for evolutionary search for
code or text that uses LMs to generate intelligent variation (for example
through specialized language models trained on code diffs (Bradley et al.,
2023b), or through simple few-shot prompting (Meyerson et al., 2023; Chen et
al., 2023)). Most QDAIF results in this paper generate new search candidates
through Language Model Crossover (LMX) (Meyerson et al., 2023), a recent and
general few-shot prompting approach that can evolve e.g. mathematical
expressions, sentences, Python programs, and prompts for text-to-image models,
by leveraging in-context learning capabilities of LMs (Brown et al., 2020).
The approach is simple: A few existing search candidates are concatenated into
a prompt, predisposing the LM to generate new, similar candidates. In this
way, LMX enables creating intelligent variation without requiring any
specially-trained models. Our experimental implementation builds on OpenELM
(Bradley et al., 2023a), a versatile open-source Python library designed for
research into LM-based evolutionary algorithms.

###  2.2 Quality Diversity Algorithms

Traditional optimization algorithms aim to discover a single high-quality
solution, which while appropriate for many situations, can fail to
_illuminate_ the full range of possible high-quality solutions. For creative
and design problems in particular, a user may want to choose what they think
is most appropriate from a diversity of such candidates. In contrast, Quality
Diversity (QD) algorithms aim to optimize not just for a single optimal
solution, but for a diverse set of high-quality solutions (Lehman & Stanley,
2011; Mouret & Clune, 2015; Pugh et al., 2016; Fontaine & Nikolaidis, 2021).
QD algorithms can thus provide a richer landscape of solutions, enabling
adaptability and flexibility in addressing multifaceted challenges (Cully et
al., 2015). In addition to a quality measure (i.e. an objective function), QD
requires a metric such that it can encourage desired axes of diversity. For
instance, Lehman et al. (2022) evolved Python programs to design varied
locomoting robots, where the diversity dimensions are the robot’s height,
width, and mass.

A significant limitation of existing QD algorithms lies in their reliance on
low-level quality and diversity measures (Mouret & Clune, 2015). This
requirement confounds applying QD algorithms to complex and creative domains,
such as the creative writing ones explored in this paper. Intuitively, such
measures (e.g. sensor readings (Cully et al., 2015), feature engineering
(Manning, 2009)) lack the subtlety and depth needed to capture the
complexities of human creativity and intuition, e.g. nuances, moods, or
cultural references that resonate in human experience. Interestingly, from
having trained on vast amounts of human-generated data, LMs can begin to
emulate such human-nuanced judgments (cf. Section 2.3). Thus, by employing an
LM to evaluate both quality and diversity, QDAIF significantly simplifies and
enlarges the range of domains QD can be applied to.

Feedback from learned ML models has been used in prior work to reduce the need
for hand-crafted heuristics or expensive ground-truth evaluations. In model-
based QD, learned feedback is supplied by surrogate models. Gaier et al.
(2017) introduced the use of surrogates (via a Gaussian process) to predict
fitness (quality). Subsequently, Keller et al. (2020) introduced a learned
model to predict both fitness and behavior characteristics (diversity),
becoming a standard approach (Lim et al., 2021; 2022; Zhang et al., 2022;
Bhatt et al., 2022). Surrogate models require domain-specific training data to
update their predictions on a limited domain, whereas AI feedback leverages
off-the-shelf instruction-tuned LMs (Chung et al., 2022; Ouyang et al., 2022)
to automate expensive human feedback for a variety of evaluation tasks. More
recently, Fontaine & Nikolaidis (2021) utilized CLIP embeddings (Radford et
al., 2021) as both quality and diversity measures to navigate the search space
of StyleGAN (Karras et al., 2019), producing a range of faces with the desired
characteristic (e.g. "A person with red hair"). We show that using pre-trained
surrogate models is more prone to reward hacking in the natural language case
(Skalse et al., 2022) (cf. Section A.2). Hence, QDAIF capitalizes on the
strengths of general-purpose LMs for evaluating generated solutions.

###  2.3 AI Feedback

Recent months have seen a surge in research that leverages LMs to provide
feedback on the training, evaluation, or problem-solving capabilities of other
LMs (Bai et al., 2022; Perez et al., 2022; Shinn et al., 2023; Wang et al.,
2023; Colas et al., 2023; Zhang et al., 2023; Lee et al., 2023). Bai et al.
(2022) show that using LM-generated critiques and refinements has been
instrumental in enhancing performance on metrics like helpfulness and
harmlessness. One particularly promising direction for AI feedback is self-
refinement, where LMs evaluate and score their own generations, and then
iteratively improve their output (Bai et al., 2022; Madaan et al., 2023).
Self-refinement has demonstrated significant improvement in output quality as
gauged by human evaluators (Madaan et al., 2023), underscoring generation-
discrimination discrepancy (Saunders et al., 2022), meaning that it is often
easier for a model to evaluate the quality of a generation than to generate
the same high-quality text. Complementary to single-objective optimization
with self-refine, QDAIF utilizes AI feedback to assess diversity in addition
to quality, facilitating more varied and improved text generation over
multiple iterations of refinement through evolution.

##  3 Approach

Figure 2: Overview of Quality-Diversity through AI Feedback (QDAIF). Dark
components are where Language Models (LM) are employed. QDAIF randomly selects
a solution from the QD archive. This chosen solution (parent) forms part of
the prompt that is fed into an LM, undergoing LMX mutation to produce a new
solution. An LM then evaluates the quality and diversity attributes of the new
solution. We compare the newly evaluated solution with its existing solutions
in the QD archive, and update it.

Figure 2 provides an overview of the approach, which is to extend a common QD
algorithm (MAP-Elites) with LM operators that generate variation, as well as
evaluate both the quality and diversity of candidate solutions. The result is
a search algorithm capable of iterative discovery and refinement, applicable
to subjective text-based domains.

MAP-Elites. Our QDAIF implementation builds upon MAP-Elites (Mouret & Clune,
2015), a widely used QD algorithm (Lehman et al., 2022; Cully et al., 2015;
Nilsson & Cully, 2021; Vassiliades et al., 2016). MAP-Elites discretizes the
diversity space (i.e. dimensions of relevant diversity) into a grid, called
the archive. The overarching objective is to populate each grid bin (or cell)
within the archive with as high-quality a solution as possible. Typically, the
MAP-Elites archive is initiated with randomly-generated solutions (although
that does not fit the setting described here; initialization is discussed
below). An iteration in MAP-Elites follows these steps: (1) randomly select an
existing solution from the archive, (2) mutate the chosen solution to generate
new solutions, (3) evaluate the new solution’s quality and diversity
characteristics, and (4) if the new solution is higher quality than the
current occupant at the cell corresponding to its diversity characteristics,
replace the previous cell occupant solution with the new solution. For a new
solution to be added to the archive, it has to improve either the quality or
the diversity of the grid, meaning that it has to either fill an empty bin or
perform better than the solution already in its bin. QDAIF distinguishes
itself from standard MAP-Elites in four key areas: archive initialization,
solution mutation, solution evaluation, and grid discretization (cf. Figure
2). We provide details on each of these differences below.

Initialization and Mutation. For archive initialization, QDAIF employs few-
shot prompting, generating solutions based on a hand-chosen set of seed
examples. We list in Section A.17 the three few-shot examples utilized in each
domain, each chosen to span a breadth of diversity characteristics. For
example, in a domain where you want diversity of sentiments (like the Opinions
domain described in Section 4.1), the few-shot examples demonstrate positive,
neutral, and negative sentiments. For solution mutation, QDAIF employs LMX
(simplified from "LMX-Near", as detailed in Meyerson et al. (2023). LMX
evolves varied text representations (e.g. mathematical expressions, sentences,
Python code) by leveraging effective in-context learning (Brown et al., 2020).
LMX prompts are kept simple, typically starting with “Here is a random example
of”. Section A.18 shows the full LMX prompts. We also introduce a novel
mutation method with instruction-following prompts for poetry in Section 4.4.

Archive Measures. While it is sometimes feasible to devise hand-crafted
heuristics to evaluate the quality of a solution (e.g. efficiency in
locomotion) or diversity characteristics (e.g. a robot’s size and mass), this
approach falters as domains become more complex and nuanced, as in creative
writing. For example, hand-crafting robust heuristics for qualitative aspects
of a story, such as its genre (e.g. romance vs. horror), is very difficult.
QDAIF circumvents the need for hand-coded measures through prompting LMs with
easily-written natural language queries to generate feedback. In particular,
capable LMs trained on expansive text corpora can begin to mirror human
intuition across a range of potentially subtle diversity characteristics.

Quantifying Performance and Diversity. For quality assessment, we prompt the
LM to discern whether the input text contains a high-quality solution or
pertains to the requested topic, requesting a “yes” or “no” response. The
solution’s quality estimate is derived from the logarithm of the probability
of the LM’s answer. Similarly, for diversity evaluation, we guide the LM to
identify a particular diversity trait. For instance, in an opinion generating
domain, the LM is prompted to gauge a solution’s sentiment, with a requested
response of “positive” or “negative”. The log probability of these responses
serves as our measure of solution diversity. Section A.18 shows the full
prompts used in each domain to evaluate the solutions. We also introduce a
novel categorical approach to evaluate solution attributes based on raw
predictions of discrete labels in Section 4.4.

Discretization. MAP-Elites typically partitions the grid into equally-sized
bins, from the intuition that all parts of the behavior space are equally
interesting. However, we observe that when assigning a bin along the diversity
axis - which is in our approach based on logits of an LM AI feedback - that
qualitative changes in behavior do not uniformly correspond to changes in the
logits (cf. Section A.27). This is likely due to the (non-linear) calibration
behavior of instruction-tuned models in predicting the labels (as output
tokens) of text passages (Jiang et al., 2021). Hence, we use custom non-
uniform bins, which are denser towards range ends. Qualitative analysis of the
generated text showed that the non-uniform bins yielded better alignment with
typical human perceptions of diversity changes, influenced by both the AI
model’s calibration and the domain-specific goals.

Models and Setup. Details on the LMX generation model (Section A.20) and
finetuned AI feedback model (Section A.21) are given, with details on the
training of these LMs. Additional default hyperparameters are described in
Section A.23.

##  4 Experiments on Creative Writing Domain

###  4.1 Setup: Opinion Writing, Short Stories

To demonstrate the versatility of QDAIF in different applications of creative
text evolution, we evaluated QDAIF on these domains: Opinions, and Stories.
The Opinions domain is focused on generating diverse, realistic pieces about
one’s opinions on eating vegetables and plant-based foods - the diversity
measure is based on the sentiment of opinions on this topic (e.g. shown in
example texts in the Figure 2 overview). For the Stories domain, the topic is
about a short story, containing two characters: a spy, and a politician. The
diversity of stories is evaluated using a variety of measures based on AI
Feedback, with the main ones being: Stories - Genre (Romance vs Horror) (1D
archive), Stories - Ending (Happy vs Tragic) (1D archive), and Stories - Genre
and Ending (2D archive). These domains capture the strengths and limitations
of all methods, ranging from simple (Opinions) to challenging (Stories - Genre
and Ending). We show in Figure 1 that the 2D domain is challenging, yet QDAIF
still outperforms the baseline in filling the archive with diverse, high-
quality stories. The AI feedback prompts are outlined in Section A.19.

Evaluation. To assess the performance of methods in creative writing
generation, we compute QD scores (Pugh et al., 2016), a standard metric used
to measure the quality-diversity of the discovered corpus of texts. A QD score
is defined as the sum of the highest quality values found in each bin. To
understand the alignment between AI and human feedback for practical
applications in QDAIF, we conducted a human evaluation study on selected elite
samples from each method (chosen from the median QD score run out of 5 random
seed runs). Using a Likert scale (Allen & Seaman, 2007) for quality
assessment, we evaluate the capability of each method to produce a collection
of diverse, high-quality texts. To do so we calculate a “human” QD score,
defined as the sum of quality scores given for all diversity categories
identified by the annotator within the set. Furthermore, to understand how
closely AI feedback aligns with human perspectives on subjective evaluation,
we measured the agreement rates between human annotators and AI and between
two human annotators. Details of the human study are specified in Section A.1,
demonstrating the validity and advantages of AI feedback in generating human-
like feedback on subjective quality and diversity measures.

###  4.2 Comparisons between QDAIF and Baselines

To evaluate the strengths and limitations of QDAIF in generating high-quality
and diverse creative writing texts, we compared our method against the
following baseline methods: Fixed-Few-Shot, Shuffling-Few-Shot, Random-Search,
and LMX Quality-Only. The baselines carry out search in the following ways:

  * •

Fixed-Few-Shot: Use a fixed few-shot prompt to sample many completions for
creative domain texts (i.e. no iterative search).

  * •

Shuffling-Few-Shot: Shuffle the in-context examples of Fixed-Few-Shot prompt,
before sampling the completion from this prompt.

  * •

Random-Search: Create a prompt pool of examples (initialized from prompts in
Section A.19), add all completions to the pool, and choose few-shot examples
from the growing pool (without pool size limit).

  * •

LMX Quality-Only: Create and maintain a pool as in Random-Search, but only up
to 100 highest-quality completions (as evaluated by AI Feedback) are kept in
the pool (i.e. iterative search focused only on quality).

Figure 3: QDAIF significantly outperforms baselines in QD score performance in
all domains. Performance stats with mean bootstrapped 95% CI, across 5 random
seed runs. The maximum possible QD score is 20 (100 for 2D archive (4th
plot)). See Section A.7 for additional stats.

We choose a variety of baselines, some highlighting representative alternative
approaches (e.g. few-shot prompting), and ablations of QDAIF, to validate our
algorithmic choices. For example, Fixed-Few-Shot and Shuffling-Few-Shot enable
the generation of different texts (relying on stochastic sampling), while
being constrained to the output distribution of the fixed set of in-context
examples. Random-Search and LMX Quality-Only are methods where the (prompt)
pool of examples that we can sample from grows in size, starting from an
initial pool of examples. In contrast to QDAIF, Random-Search is limited by
the lack of constraints in the prompt pool, especially in maintaining the
quality of the growing pool through evaluation. LMX Quality-Only adds a
quality-based evaluation step with AI feedback that optimizes the quality of
texts in the pool over time to contain only texts with high quality scores,
but is not designed to encourage diversity in texts in comparison to QDAIF.

For each domain and baseline described above, we recorded runs for 2000
iterations, repeated with 5 different random seeds. For measures that were not
computed during the runs, AI feedback is used to compute the quality and
diversity measures for each iteration sample at the end of the run. Following
established protocol in evaluating and comparing QDAIF and baselines (Pugh et
al., 2016), we compare the QD score of all methods. Similar to QDAIF, the
baseline methods use hand-written examples in Section A.19, either in a fixed
prompt or as the initial prompt pool.

Performance Comparison. We report results comparing the QD score performance
for the different methods of generating creative writing texts in Figure 3. We
computed the mean and the bootstrapped 95% confidence intervals (CI) from 100k
resamples, across 5 random seeds of runs. We notice from the 95% CI that QDAIF
achieves significantly better QD score than all baseline methods in all
creative writing domains. The broader range of stories and opinions generated
by QDAIF is also evident qualitatively. For example, in the Stories - Genre
and Ending domain, while the baseline methods deliver straightforward and more
predictable stories of how the spy "pulled out a knife and stabbed [the
politician] to death", QDAIF generates a more dramatic story of how the spy
"transformed into the monster and killed everyone". Random-Search is the
worst-performing method overall, with significantly lower QD score performance
in Opinions and Stories - Genre compared to the best-performing baselines.
Interestingly, LMX Quality-Only does not significantly outperform the methods
using a fixed population prompt pool (Fixed-Few-Shot and Shuffling-Few-Shot).
On Stories - Genre, LMX Quality-Only is often weaker than Fixed-Few-Shot and
Shuffling-Few-Shot, in spite of the fact that quality optimization is
occurring. The results show that single-objective optimization cannot guide
the search for diverse, high-quality texts alone.

Human Feedback Evaluation. We report the subjective quality and diversity from
the human study of the set of texts (diverse samples of elites from different
bins) discovered by each method, with comparisons in Table 1. Detailed
discussion on the human study setup is described in Section A.1. We observe
that compared to baselines, QDAIF is competitive with or better at discovering
diverse, high-quality texts in Opinions and Stories, evaluated with human
preferences. This observation is reflected in the human QD score, measuring
the perceived quality-diversity of the generated elite texts from each method
in a single run. QDAIF sets also showed high agreement between humans and AI
feedback on the diversity categories of presented texts, as well as between
two annotators, competitive with Fixed-Few-Shot. Although the average
perceived quality of texts is better from Fixed-Few-Shot, this is not enough
for bringing high-quality examples for different niches of outputs (i.e.
higher QD score). Furthermore, Shuffling-Few-Shot demonstrates even lower
human evaluation scores, despite the use of the same fixed set of hand-written
seeds, indicating lower robustness due to the use of different ordering of
few-shot examples. Prior work hints at the sensitivity of LMs to few-shot
prompt ordering, with task-solving capabilities varying significantly due to
this ordering (Lu et al., 2021). The gap in human-evaluated performance
between Fixed-Few-Shot and Shuffling-Few-Shot indicates that reliance on fixed
prompts is less likely to enable reliable search, in contrast to the
robustness shown by QDAIF. Additionally, Random-Search and LMX Quality-Only
obtained even lower human evaluation scores, even though the methods either
explore different prompts or optimize for the quality of texts. We provide a
detailed discussion (for baseline methods) on findings from the subjective
study of the discovered texts in Section A.12, as well as the qualitative
behavior of the text search over time in Section A.13. Through guided
evolutionary search, QDAIF surpasses all baseline methods in terms of computed
QD score performance, and is competitive (or better) compared to baselines,
according to human evaluation.

Table 1: QDAIF is competitive/better in terms of Human QD score against baseline methods from the human evaluation study. The stats are averaged across three domains: Opinions, Stories - Genre, and Stories - Ending. The Human QD score quantifies the perceived quality-diversity of a set of solutions returned by a method (i.e. high-quality texts for all the diversity categories of texts that the evaluator could identify). Method |  Human QD score |  Quality rating |  Human-AI agreement |  Human agreement
---|---|---|---|---
Fixed-Few-Shot | 0.767 | 4.133 | 0.800 | 0.867
Shuffling-Few-Shot | 0.696 | 3.500 | 0.700 | 0.667
Random-Search | 0.606 | 3.300 | 0.733 | 0.600
LMX, Fitness-Only | 0.650 | 3.533 | 0.633 | 0.733
QDAIF (ours) | 0.772 | 3.900 | 0.833 | 0.800

###  4.3 Extensions to AI Feedback and Mutation Model

In addition to experiments with QDAIF described in previous sections, we
investigated the effects on the performance due to variations of the method.

LMX Model Size. We used larger versions of the LMX models (30B and 70B) for
mutation, and compared it to the performance of the 13B model (default). While
no relationship was found between model size and QD score, quality ratings
from human feedback improved with outputs from larger models (described in
detail in Section A.3).

Few-Shot Prompting for AI Feedback. We compared the performance of QDAIF on
the Stories - Genre domain when we prompted our AI feedback model for
diversity measures given a few-shot prompt with the following settings:
2-shot, 4-shot, and 8-shot. Using a higher number of few-shots led to
improvements in human quality ratings of texts. Interestingly, few-shot AI
feedback caused QDAIF runs to return solutions where human feedback disagreed
with AI feedback on the genre more often; intuitively, few-shot demonstrations
would help LMs evaluate texts closer to human notions of genres. Future
research is needed to explore whether this is a general phenomenon or is
particular to the model we used and this particular domain. Further discussion
and results are highlighted in Section A.4.

Varying Initialization and Mutation Method. Ideally, QDAIF would be simpler if
it could be run without seed examples (e.g. requesting a story from an
instruction-following LM). We investigated the potential of QDAIF when the
solution population is initialized from zero-shot prompted generations, and
evolved using LMX. Initial results on Opinions and Stories for a more
challenging case (i.e. initializing texts with zero-shot-prompted outputs from
the pre-trained LM used in LMX) along with prior work discussion are shown in
Section A.5, highlighting comparable performance in terms of QD score, with
divergence observed in alignment with human preferences. We also find that
QDAIF, in these domains, is robust to the mechanism of generating variation.
Section A.6 describes an alternative mutation method (based on a more gradual
few-shot replacement operation) that is more effective in some circumstances,
although in general offers comparable performance. We provide a detailed
discussion (for QDAIF methods) on findings from the subjective study of the
discovered texts in Section A.10, as well as the qualitative behavior of the
text search over time in Section A.11. There may be many ways of leveraging
LMs to generate variation (including finetuning models; see Section A.8), and
this is an exciting avenue of future research.

###  4.4 Evolving Poetry with Instruction Guidance

Figure 4: QDAIF (LMX-guided) (left) covers the space of poetry with high-
quality solutions (on a rating scale), with poems matching the closest bins.
QDAIF solutions take qualitative inspiration from the seed poem’s imagery of
"fields of green waves" in Section A.26 while giving meaningfully diverse
kinds of poems across the search space. QDAIF (LMX-rewrite) (not shown) also
covers more the space of diverse, high-quality poems compared to Random-Poems
(right).

The final experiment explores scaling up QDAIF to a more capable model, GPT-4
(OpenAI, 2023), in the challenging task of generating poetry, highlighting how
QDAIF will benefit from advances in model capabilities. The aim of the Poetry
domain is to produce high-quality poems with varying genres and emotional
tones, unrestricted by topic. Here, the MAP-Elites archive has two axes of
diversity: genre and tone, and they are delineated using categorical labels.
The genre axis has the labels "haiku", "sonnet", "ballad", "limerick", and
"hymn", while the tone axis has the labels of "happy", "dark", "mysterious",
"romantic", and "reflective". We created a new mutation operator, LMX-rewrite,
for this domain, that leverages instruction-following to tell a model to
translate a parent poem into an offspring with different diversity
characteristics. To generate a new solution, we prompt GPT-4 to rewrite the
selected poem into an inspired, different poem: Inspired by this poem, write a
new poem of very high, award winning quality with a different poetic genre
(format and form) and tone compared to the poem above.

We used GPT-4 to determine the genre and tone of a poem and rate its quality.
For quality, we prompt GPT-4 to Rate the quality of the above poem on a scale
from 1 to 10. To determine the diversity attributes, we prompt GPT-4 to
determine which genre or tone is the poem closest to. For example, to
determine genre, we ask GPT-4: What genre is this poem closest to from the
following list: ["haiku", "sonnet", "ballad", "limerick", "hymn"]? Section
A.26 shows the full prompts and setup. We observed high consistency in GPT-4’s
responses; across multiple LM calls, quality ratings for the same poem
fluctuated by no more than a single point. We show qualitative examples of
poems with their AI feedback evaluations in Section A.16. In this setup, all
methods are run for 200 iterations, and QDAIF is benchmarked against the
baseline, Random-Poems, as well as an ablation method, Fixed Seed Rewrite.
Random-Poems simply generates 200 random poems. Fixed Seed Rewrite rewrites
only the seed poem in Section A.26, without evolutionary search.

We found that QDAIF achieves a higher QD score of 130 (CI: 118 - 145) in
comparison to Random-Poems with 76 (CI: 67 - 85) and Fixed Seed Rewrite with
99 (CI: 72 - 117). We observed a similar trend (with a wider performance gap
between QDAIF and other methods) when we used GPT-3.5-Turbo for the generation
step instead of GPT-4 while keeping GPT-4 as the evaluator (cf. Section A.14).
QDAIF was shown to have greater QD score performance than the other methods
according to the Mann-Whitney U Test ($p\leq 0.05$). Although QDAIF with LMX-
rewrite leaves some of the bins empty like Random-Poems (e.g. GPT-4 doesn’t
recognize that limericks are also a valid genre for diverse poems during
search), we can adapt QDAIF by adding guidance on the desired (randomly
chosen) genre and tone for rewriting (LMX-guided). The performance of this
method of QDAIF is on par with a Targeted-Poems approach (generating high-
quality poems of randomly chosen genre and tone per step) in terms of QD
score, and even better when using an older version of GPT-4 (cf. Section
A.14). Furthermore, we found that the rewriting step is useful for generating
poems that can meaningfully preserve inspiration from parent poems, enabling
users to control the outcomes of search, and giving us a tool that can mimic
stepping stones of human innovation for AI-assisted creative applications
(Stanley et al., 2017) (see Section A.15). Figure 4 highlights the potential
of QDAIF with GPT-4, especially in controlling a set of solutions subjectively
aligned with AI feedback labels. Overall, results highlight the challenge of
obtaining the desired diversity in outputs from standard prompting of models
like GPT-4 without explicit guidance (Renda et al., 2023; Friedrich et al.,
2023) while emphasizing the importance of combining a rewriting mutation
operator with an evolving population of solutions in leading to more diverse,
high-quality outcomes that form more natural chains of inspiration.

##  5 Discussion and Conclusion

This paper introduces QDAIF, a quality-diversity method that aims to discover
diverse and high-quality solutions in qualitative domains, by leveraging
advances in foundation models to evaluate the quality and diversity of
generated individuals. The paper’s results highlight that QDAIF can succeed at
its aims, generating solutions that align with human perception of quality and
diversity.

We note limitations with QDAIF that motivate future work. Firstly, we suspect
reward hacking happening when using LMs to generate feedback. Our human
evaluation investigation shows that while the LM’s evaluation of quality
mostly aligns with human perception, the correlation drops when the evaluated
quality is in the range 0.995 to 1 (cf. Figure 5). The text generation might
have exploited certain attributes or phrasings that allow an LM to give a
high-quality estimate, but not what humans would agree is good. This is a
common issue highlighted by other works when using AI models as classifiers or
evaluators (Nguyen et al., 2015a), highlighting risks of open-ended search to
be tackled (Ecoffet et al., 2020). One method to address this limitation could
be to use RLHF finetuning (Ouyang et al., 2022) to produce LMs that can detect
and mitigate adversarially generated texts. RLHF models are finetuned to
generate responses to tasks in order to maximize human preferences on
instruction-following, potentially leading to improvements in the evaluation
of creative writing, a human-centric medium of innovation. Another possible
approach could be to use an ensemble of different AI models to evaluate
solutions, rather than relying only on one; the hope would be that robustness
would result from models having uncorrelated blind spots.

Furthermore, although QDAIF makes it easy to specify qualitative aspects of
diversity through natural language prompts, it still requires a researcher to
define what axes of diversity they are most interested in. For example, if we
applied QDAIF to generate short stories of different genres (e.g. comparing
horror vs. romance), it would not autonomously explore other important
attributes that a writer might care about (e.g. first-person vs. third-person
perspective) unless explicitly specified. When we tested different diversity
measures in the Stories domain, such pathologies were observed (Section A.28).
For example, when using "hero spy vs. hero politician" as the diversity
measure, many of the solutions generated tend to neglect the interaction
between the spy and the politician, focusing solely on the character that is
meant to be the hero. However, someone writing a short story about a spy and a
politician would naturally care about how the characters interact with one
another. One possible way to automatically determine interesting diversity
measures is to utilize the human notions of interestingness distilled into
foundation models (Zhang et al., 2023). That is, we could ask LMs to suggest
interesting diversity measures that a human would typically care about in the
domain, thereby enabling a more autonomous creative search.

In conclusion, we show that QDAIF is a promising approach to open-ended search
that can reveal unexplored creative writing spaces, surpassing alternative
text generation methods in generating diverse high-quality natural language
text. AI feedback, Evolution through Large Models (ELM), and quality-diversity
search (QD) were found to be essential ingredients for enhanced AI systems
that can innovate in subjective spaces, similar to past research on Innovation
Engines (Nguyen et al., 2016; 2015b). In fact, we see AI feedback as a general
ingredient for open-ended search for solutions in multimodal domains, capable
of following instructions beyond text (Liu et al., 2023). QDAIF can be easily
extended to multi-modal domains (e.g. vision-language) for synthetic data
generation and evaluation, building on top of recent advances in the field
(Eichenberg et al., 2021; Alayrac et al., 2022; Bellagente et al., 2023;
Driess et al., 2023; Bhatt et al., 2023; Sudhakaran et al., 2023; Todd et al.,
2023). We see many possibilities from QDAIF to build creative search systems
with evaluation, diversification, and improvement capabilities, bringing us
closer to AI that can support and extend human innovation.

## Ethics Statement

Human evaluations were performed by the co-authors of this paper and select
colleagues. All human evaluators provided informed consent, and their feedback
and assessments were obtained without coercion or bias. We took action to
prevent bias by presenting evaluators with texts to evaluate in a blind
setting, with only the instructions for the study annotation task presented
(to carefully read through the presented texts, then give a quality score and
a label of the characteristic that best matches the texts). We show a detailed
setup for the human study in Section A.1.

For transparency, we provide the full set of results with caption descriptions
from our human evaluation. In the Opinions domain, Tables 12–15 contain the
human evaluation results for sets from baseline methods, Tables 28–31 contain
the human evaluation results for sets from QDAIF methods, and Tables 24–27
contain the human evaluation results for sets from embedding feedback QD
methods. In the Stories - Genre domain, Tables 16–19 contain the human
evaluation results for sets from baseline methods, and Tables 32–35 contain
the human evaluation results for sets from QDAIF methods. For the Stories -
Ending domain, Tables 20–23 contain the human evaluation results for sets from
baseline methods, and Tables 36–39 contain the human evaluation results for
sets from QDAIF methods.

## Author Contributions

Herbie developed the setup and framework for the Poetry domain experiments and
base library for research. Andrew developed the setup and experiments for the
Opinions and Stories domains, and contributed to extended studies,
visualization, and analysis across experiments in the paper. Hannah
contributed additional experimentation in the Stories domain, in addition to
coordinating part of human evaluation studies. Jenny contributed qualitative
analysis across studied domains. Koen developed visualization scripts used in
Opinions and Stories domain experiments. Marco contributed to part of the
technical implementation and ideation. Andrew conducted the blind human
evaluation study, and Grégory advised on the conduct and analysis of the human
study. Joel, Jeff, and Ken initiated early ideation for this work. Joel,
Grégory, Jeff, and Ken advised and guided. Andrew, Jenny, Herbie, and Joel
wrote the manuscript with edits and feedback from all authors.

## Acknowledgements

We thank Robert Baldock, Samuel Weinbach, Souradeep Nanda, Jan Zierstek, and
Andres Felipe Cruz Salinas for insightful discussions and feedback within the
lab at Aleph Alpha. We also thank Katherine Hardgrave, David Nugent, Daniel
Flood, and Formula Trinity Autonomous for the inspiration that seeded the
momentum leading up to this work.

## References

  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.  Do as I can, not as I say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: A visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Allen & Seaman (2007) I. Elaine Allen and Christopher A. Seaman.  Likert scales and data analyses.  _Quality progress_ , 40(7):64–65, 2007.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional AI: Harmlessness from AI feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Bellagente et al. (2023) Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Björn Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, et al.  Multifusion: Fusing pre-trained models for multi-lingual, multi-modal image generation.  _arXiv preprint arXiv:2305.15296_ , 2023.
  * Bhatt et al. (2022) Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis.  Deep surrogate assisted generation of environments.  _Advances in Neural Information Processing Systems_ , 35:37762–37777, 2022.
  * Bhatt et al. (2023) Varun Bhatt, Heramb Nemlekar, Matthew Fontaine, Bryon Tjanaka, Hejia Zhang, Ya-Chuan Hsu, and Stefanos Nikolaidis.  Surrogate assisted generation of human-robot interaction scenarios.  _arXiv preprint arXiv:2304.13787_ , 2023.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bradley et al. (2023a) Herbie Bradley, Honglu Fan, Francisco Carvalho, Matthew Fisher, Louis Castricato, reciprocated, dmayhem93, Shivanshu Purohit, and Joel Lehman.  OpenELM, January 2023a.  URL https://github.com/CarperAI/OpenELM.
  * Bradley et al. (2023b) Herbie Bradley, Honglu Fan, Harry Saini, Reshinth Adithyan, Shivanshu Purohit, and Joel Lehman.  Diff models - A new way to edit code.  _CarperAI Blog_ , Jan 2023b.  URL https://carper.ai/diff-model/.
  * Brant & Stanley (2017) Jonathan C. Brant and Kenneth O. Stanley.  Minimal criterion coevolution: A new approach to open-ended search.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 67–74, 2017.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. (2023) Angelica Chen, David M. Dohan, and David R. So.  Evoprompting: Language models for code-level neural architecture search.  _arXiv preprint arXiv:2302.14838_ , 2023.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  PaLM: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Clune (2019) Jeff Clune.  AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence.  _arXiv preprint arXiv:1905.10985_ , 2019.
  * Colas et al. (2023) Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté.  Augmenting autotelic agents with large language models.  _arXiv preprint arXiv:2305.12487_ , 2023.
  * Cully et al. (2015) Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret.  Robots that can adapt like animals.  _Nature_ , 521(7553):503–507, 2015.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S.M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  PaLM-E: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Ecoffet et al. (2020) Adrien Ecoffet, Jeff Clune, and Joel Lehman.  Open questions in creating safe open-ended ai: tensions between control and creativity.  In _Artificial Life Conference Proceedings 32_ , pp. 27–35. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020.
  * Eichenberg et al. (2021) Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.  MAGMA - Multimodal augmentation of generative models through adapter-based finetuning.  _arXiv preprint arXiv:2112.05253_ , 2021.
  * Flageat & Cully (2020) Manon Flageat and Antoine Cully.  Fast and stable map-elites in noisy domains using deep grids.  In _Artificial Life Conference Proceedings 32_ , pp. 273–282. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020.
  * Flageat & Cully (2023) Manon Flageat and Antoine Cully.  Uncertain quality-diversity: Evaluation methodology and new methods for quality-diversity in uncertain domains.  _IEEE Transactions on Evolutionary Computation_ , 2023.
  * Fontaine & Nikolaidis (2021) Matthew Fontaine and Stefanos Nikolaidis.  Differentiable quality diversity.  _Advances in Neural Information Processing Systems_ , 34:10040–10052, 2021.
  * Friedrich et al. (2023) Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting.  Fair diffusion: Instructing text-to-image generation models on fairness.  _arXiv preprint arXiv:2302.10893_ , 2023.
  * Gaier (2020) Adam Gaier.  _Accelerating Evolutionary Design Exploration with Predictive and Generative Models_.  PhD thesis, Université de Lorraine, 2020.
  * Gaier et al. (2017) Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret.  Data-efficient exploration, optimization, and modeling of diverse designs through surrogate-assisted illumination.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 99–106, 2017.
  * Gaier et al. (2019) Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret.  Are quality diversity algorithms better at generating stepping stones than objective-based search?  In _Proceedings of the Genetic and Evolutionary Computation Conference Companion_ , pp. 115–116, 2019.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_ , 2021.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pp. 2790–2799. PMLR, 2019.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig.  How can we know when language models know? On the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977, 2021.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Keller et al. (2020) Leon Keller, Daniel Tanneberg, Svenja Stark, and Jan Peters.  Model-based quality-diversity search for efficient robot learning.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 9675–9680. IEEE, 2020.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  RLAIF: Scaling reinforcement learning from human feedback with AI feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Lehman & Stanley (2010) Joel Lehman and Kenneth O. Stanley.  Revising the evolutionary computation abstraction: Minimal criteria novelty search.  In _Proceedings of the 12th annual conference on Genetic and evolutionary computation_ , pp. 103–110, 2010.
  * Lehman & Stanley (2011) Joel Lehman and Kenneth O. Stanley.  Evolving a diversity of virtual creatures through novelty search and local competition.  In _Proceedings of the 13th annual conference on Genetic and evolutionary computation_ , pp. 211–218, 2011.
  * Lehman et al. (2016) Joel Lehman, Bryan Wilder, and Kenneth O. Stanley.  On the critical role of divergent selection in evolvability.  _Frontiers in Robotics and AI_ , 3:45, 2016.
  * Lehman et al. (2022) Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley.  Evolution through large models.  _arXiv preprint arXiv:2206.08896_ , 2022.
  * Lim et al. (2021) Bryan Lim, Luca Grillotti, Lorenzo Bernasconi, and Antoine Cully.  Dynamics-aware quality-diversity for efficient learning of skill repertoires.  _arXiv preprint arXiv:2109.08522_ , 2021.
  * Lim et al. (2022) Bryan Lim, Alexander Reichenbach, and Antoine Cully.  Learning to walk autonomously via reset-free quality-diversity.  _arXiv preprint arXiv:2204.03655_ , 2022.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv preprint arXiv:2104.08786_ , 2021.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Manning (2009) Christopher D Manning.  _An introduction to information retrieval_.  Cambridge university press, 2009.
  * Meyerson et al. (2023) Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Arash Moradi, Amy K. Hoover, and Joel Lehman.  Language model crossover: Variation through few-shot prompting.  _arXiv preprint arXiv:2302.12170_ , 2023.
  * Mouret & Clune (2015) Jean-Baptiste Mouret and Jeff Clune.  Illuminating search spaces by mapping elites.  _arXiv preprint arXiv:1504.04909_ , 2015.
  * Muennighoff (2022) Niklas Muennighoff.  SGPT: GPT sentence embeddings for semantic search.  _arXiv:2202.08904_ , 2022.
  * Nguyen et al. (2015a) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 427–436, 2015a.
  * Nguyen et al. (2015b) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Innovation engines: Automated creativity and improved stochastic optimization via deep learning.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , 2015b.
  * Nguyen et al. (2016) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Understanding innovation engines: Automated creativity and improved stochastic optimization via deep learning.  _Evolutionary computation_ , 24(3):545–572, 2016.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial NLI: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_ , 2019.
  * Nilsson & Cully (2021) Olle Nilsson and Antoine Cully.  Policy gradient assisted MAP-Elites.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 866–875, 2021.
  * OpenAI (2023) OpenAI.  GPT-4 technical report, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Perez et al. (2022) Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al.  Discovering language model behaviors with model-written evaluations.  _arXiv preprint arXiv:2212.09251_ , 2022.
  * Pugh et al. (2016) Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley.  Quality diversity: A new frontier for evolutionary computation.  _Frontiers in Robotics and AI_ , 3:40, 2016.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using siamese BERT-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Renda et al. (2023) Alex Renda, Aspen K. Hopkins, and Michael Carbin.  Can LLMs generate random numbers? evaluating LLM sampling in controlled domains.  In _ICML 2023 Workshop: Sampling and Optimization in Discrete Space_ , 2023.  URL http://people.csail.mit.edu/renda/llm-sampling-paper.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_ , 2021.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_ , 2022.
  * Secretan et al. (2011) Jimmy Secretan, Nicholas Beato, David B. D’Ambrosio, Adelein Rodriguez, Adam Campbell, Jeremiah T. Folsom-Kovarik, and Kenneth O. Stanley.  Picbreeder: A case study in collaborative evolutionary exploration of design space.  _Evol. Comput._ , 19(3):373–403, 2011.  doi: 10.1162/EVCO\\_a\\_00030.  URL https://doi.org/10.1162/EVCO_a_00030.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: An autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_ , 2023.
  * Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger.  Defining and characterizing reward gaming.  _Advances in Neural Information Processing Systems_ , 35:9460–9471, 2022.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Stanley (2007) Kenneth O Stanley.  Compositional pattern producing networks: A novel abstraction of development.  _Genetic programming and evolvable machines_ , 8:131–162, 2007.
  * Stanley & Lehman (2015) Kenneth O. Stanley and Joel Lehman.  _Why greatness cannot be planned: The myth of the objective_.  Springer, 2015.
  * Stanley et al. (2017) Kenneth O. Stanley, Joel Lehman, and Lisa Soros.  Open-endedness: The last grand challenge you’ve never heard of.  _O’Reilly Radar_ , 2017.
  * Sudhakaran et al. (2023) Shyam Sudhakaran, Miguel González-Duque, Claire Glanois, Matthias Freiberger, Elias Najarro, and Sebastian Risi.  Mariogpt: Open-ended text2level generation through large language models, 2023.
  * Todd et al. (2023) Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius.  Level generation through large language models.  In _Proceedings of the 18th International Conference on the Foundations of Digital Games_ , pp. 1–8, 2023.
  * Vassiliades et al. (2016) Vassilis Vassiliades, Konstantinos Chatzilygeroudis, and Jean-Baptiste Mouret.  Scaling up MAP-Elites using centroidal Voronoi tessellations.  _arXiv preprint arXiv:1610.05729_ , 2016.
  * Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  _arXiv preprint arXiv:2204.07705_ , 2022.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Zhang et al. (2023) Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune.  OMNI: Open-endedness via models of human notions of interestingness.  _arXiv preprint arXiv:2306.01711_ , 2023.
  * Zhang et al. (2022) Yulun Zhang, Matthew C Fontaine, Amy K Hoover, and Stefanos Nikolaidis.  Deep surrogate assisted MAP-Elites for automated hearthstone deckbuilding.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 158–167, 2022.

##  Appendix A Appendix

###  A.1 Human Study on Quality-Diversity of Text Samples

A study through human feedback lets us understand how well QD score
performance through AI feedback translates to creating a high-quality, diverse
set of creative texts from a subjective angle. We compare sets of human
feedback evaluations for samples of diverse elites at the end of each run to
measure translations from AI-assessed performance to human-assessed
performance of methods in generating high-quality, diverse texts. In addition,
the Human QD Score (sum of mean quality score for each category/label that is
found in the set according to human feedback) gives us a rough understanding
of how aligned quality-diversity improvement during the search is with the
more subjective notion of quality-diversity; this score is low if the set
deemed to cover a wide space of diverse texts from AI feedback fails to
subjectively cover the space of desired diversity according to human
evaluations. To distinguish between quality ratings from humans vs. AI
feedback in this section, we refer to quality scores as those from human
evaluators, and fitness scores as those from AI feedback.

To assess the robustness of quality and diversity measures in AI feedback, we
carried out a study involving diverse elite samples selected from different
bins of the QD archive from our tested runs. Over a total of 28 experiments
(of which 4 are from embedding feedback experiment runs), five distinct
stories per experiment are reviewed by six annotators. Each (generated) text
was independently reviewed by two persons, resulting in a total of 280
annotations.

During the annotation process, we collected a subjective assessment of the
quality of the generation using a 5-point Likert scale based on the text
quality in terms of flow, plot, presence of repetition, and correspondence to
the study’s topic. In addition, we assign each text to one of three
categories. Two categories were specific to the study performed (such as
positive/negative sentiment, romance/horror genre, or tragic/happy ending) and
a third category was used when no element of the other two classes was
identified.

We took action to prevent bias by presenting evaluators with texts to evaluate
in a blind setting, with only the instructions for the study annotation task
presented (to carefully read through the presented texts, then give a quality
score and a label of the characteristic that best matches the texts). We
provide the full set of results with caption descriptions from our human
evaluation. In the Opinions domain, Tables 12–15 contain the human evaluation
results for sets from baseline methods, Tables 28–31 contain the human
evaluation results for sets from QDAIF methods, and Tables 24–27 contain the
human evaluation results for sets from embedding feedback QD methods. In the
Stories - Genre domain, Tables 16–19 contain the human evaluation results for
sets from baseline methods, and Tables 32–35 contain the human evaluation
results for sets from QDAIF methods. For the Stories - Ending domain, Tables
20–23 contain the human evaluation results for sets from baseline methods, and
Tables 36–39 contain the human evaluation results for sets from QDAIF methods.

We compiled the results of the full study in Section A.9, and summarized the
stats across the study in the paragraphs below.

#### Comparison of quality scores

We observed that both annotators generally have a close agreement in their
ratings with an average difference of 0.9 Likert points but there were
occasional instances where they may differ by 2 or 3 units. These deviations
occurred in 15% and 5% of the cases, respectively. This reflects that
assessing the quality is somewhat subjective. To get a final estimation of the
quality of the generated texts, the scores from both annotators were averaged.

Figure 5 shows the average quality ratings from annotators for various ranges
of fitness (here, the quality score obtained from AI feedback). The ranges
were chosen in a way that ensured a similar amount of samples in each range.
We observe clear evidence of a correlation between human and AI quality
measures, indicating the usefulness of using AI feedback to assess quality.
However, we also observed that fitness for the texts with the highest scores
becomes uncorrelated to human-assessed quality. This implies that above a
certain threshold, fitness is not a reliable measure of quality in some cases
(and maybe slightly lower fitness solutions were preferred by humans).
Therefore, we suggest that in future work, more research is done in studying
the relationship between (high-confidence) evaluations from AI feedback, and
reward hacking of solutions (Nguyen et al., 2015a; Skalse et al., 2022) under
certain conditions and controls during the search (e.g. through the use of
seed texts for human-preferred outputs, as shown in Table 1, or additional
constraints on the generated solutions during search (Lehman & Stanley, 2010;
Brant & Stanley, 2017)).

Figure 5: Correlation plot between quality rating from human annotators, and
fitness range (quality computed from AI feedback). Mean human-annotated
quality and statistical error for different ranges of AI feedback fitness
scores indicate more frequent instances of reward hacking (Skalse et al.,
2022) from the outputs of some search methods evaluated in this study.

#### Comparison of diversity measurements

For diversity, we observed that the two annotators agreed on the
classification of a generated text in the same category 73 percent of the
time. On the AI feedback side, diversity is collected on a 20-bin axis,
measuring different degrees of correspondence to two experiment-specific
categories mentioned earlier. For the purpose of comparison with human
feedback, these bins are clustered into a first category (bins 0 to 8), a
second category (bins 11 to 19) and a neutral category (bins 9 and 10).
Additionally, the 5 samples for each set were collected from a relatively
uniform spread of bins from one end of the diversity axis to another
(specifically, bins in set [0, 6, 9, 13, 19], except for when the method fails
to find a solution for the bin, then the next closest solution near the bin is
chosen). This arbitrary arrangement allows for a relatively uniform
distribution among the three categories.

On average, AI feedback agrees with a human annotator on the text category
label 73% of the time. On samples where both annotators agree on the label of
the text, this agreement rate increases to 82%. Within this set, the agreement
increases to 95% on samples when we look at samples where both annotators give
a label that is not the neutral category. For some samples from bins 6 and 13,
one human annotator gave a neutral label and the other annotator one of the
other two labels (closer to the AI feedback diversity measure for these bins),
indicating that some samples lie between neutral and extreme in the given
measure. These findings suggest that diversity classification from AI feedback
is relatively reliable, especially in texts where humans agree on the label.

#### Baseline quality rating.

The average quality score given by annotators for a given sample was 3.18,
close to the middle rating of 3. This gives us an indication of what could be
considered the threshold for subjectively good or bad outputs.

The average human (subjective) QD score of all sets in the study is 0.606.
This is another indication for the threshold for determining which set from a
given run/method had high-quality, diverse texts.

###  A.2 Comparing AI Feedback Against Alternative Measures of Diversity

Table 2: QDAIF outperforms QD with embedding feedback (QDEF) according to human evaluation in human QD score and quality, when the difference is in feedback type for each base method.. The mean of Human QD score and quality are computed for the set of each method/run. The differences between the two evaluators for QD score and Quality are shown. Agreement between human and AI feedback on diversity labels given is slightly higher across QDAIF results compared to QDEF results, as well as on agreement between two annotators. Method |  Human QD score |  QD score range |  Quality rating |  Quality range |  Human-AI agreement |  Human agreement
---|---|---|---|---|---|---
QDEF, LMX, Zero-Shot Init | 0.242 | 0.050 | 1.300 | 0.600 | 0.500 | 0.400
QDEF, LMX, Seeded Init | 0.308 | 0.050 | 1.700 | 0.600 | 0.600 | 0.600
QDEF, LMX-Replace, Zero-Shot Init | 0.417 | 0.233 | 2.100 | 1.000 | 0.800 | 0.600
QDEF, LMX-Replace, Seeded Init | 0.500 | 0.200 | 2.300 | 1.000 | 0.800 | 1.000
QDAIF, LMX, Zero-Shot Init | 0.350 | 0.300 | 1.700 | 1.400 | 0.800 | 1.000
QDAIF, LMX, Seeded Init | 0.617 | 0.167 | 3.200 | 1.200 | 1.000 | 1.000
QDAIF, LMX-Replace, Zero-Shot Init | 0.833 | 0.267 | 4.300 | 1.000 | 0.600 | 0.600
QDAIF, LMX-Replace, Seeded Init | 0.739 | 0.078 | 3.700 | 0.600 | 0.700 | 0.800

To understand the effect of fuzzy evaluation tools as a component of our QD
setup, we tested the use of an alternative method of feedback compared to our
default method (AI feedback) in the MAP-Elites pipeline: semantic embedding
feedback (Reimers & Gurevych, 2019). For this method, we used a 13B embedding
model111https://aleph-
alpha.com/luminous%2Dexplore%2Da%2Dmodel%2Dfor%2Dworld%2Dclass%2Dsemantic%2Drepresentation/,
based on the architecture described in Muennighoff (2022), with an asymmetric
search setup to measure the distance between a generated text (document
embedding) and a query embedding for a desired measure (e.g. "This is a
positive opinion"). To compute a diversity measure that can be defined on an
axis, we first get the cosine distances between the document embedding and
each of two opposing attribute query embeddings. From this, we can measure how
close the document embedding is to one attribute compared to the other, and
obtain a single diversity measure normalized in the range [0, 1]. We use the
same method for quality feedback, with a query that aims to measure the
relevance of generated texts to a specific domain. The cosine similarity is
used here as a quality score, with negative values being clipped to 0.
Additional setup details are shown in Section A.24. Since the subjective
quality of the resulting elites (of creative texts) is more informative of the
method’s potential in further applications for practical synthetic data
generation, We conducted a human study in addition to the reporting of QD
score stats as part of our results.

We display performance statistics from our runs (QD Score) as well as human
evaluation scores on elite samples in Table 2. We observe from our human study
that using AI feedback as the evaluator instead of semantic embedding feedback
for every variation of the QD run setup potentially leads to subjectively
better generations. This is likely due to more prominent reward hacking
(Skalse et al., 2022) occurring in runs using embedding feedback, where the
highest quality score texts end up being very similar to the query "An opinion
piece about eating vegetables and plant-based foods", while not optimizing for
the subjective quality of texts in different bins. Qualitative analysis of
human-evaluated sets of texts from QD with embedding feedback is shown in
Tables 24–27. Furthermore, agreement between human and AI feedback on text
diversity labels was slightly higher across QDAIF sets compared to QD with
embedding feedback (QDEF) sets. Overall, AI feedback outperforms the
alternative measure of semantic embedding feedback, by guiding the generation
of texts that are more preferred by humans, and by serving as a better
evaluator for quality and diversity measures than embedding feedback.

###  A.3 On Scaling LMs for Mutation

Previous work has consistently shown that LMs demonstrate improved
capabilities in various task-based benchmarks at larger scales (Kaplan et al.,
2020; Chowdhery et al., 2022; Chung et al., 2022). This applies to performance
in solving tasks through in-context learning, which LMX is based on. Prior
work in LMX (Meyerson et al., 2023) has found that a relationship between
model scaling and performance of mutations can be observed when evolving
binary strings in a search domain. Interestingly, experiments in LMX as well
as ELM (Lehman et al., 2022) observed that in some cases, emerging
capabilities in mutation ability appear for reasonably small LMs, but may not
scale with a clear trend.

Firstly, we show the QD score performance between runs in Figure 6. The
standard error in the score across 5 seeds is also shown. We observed that the
QD score from the 70B runs converged to a lower point compared to the 13B and
30B runs, which have comparable scores. This highlights a trend between model
size and QD score that is not directly proportional. Although suggestive
trends are not seen here, a study based on subjective feedback is still
necessary for a deeper understanding of the performance of each experiment.

We observed from the human feedback evaluation a trend in the quality ratings,
with average quality scores from each experiment set of: 3.43 (13B), 3.60
(30B), and 4.03 (70B). The quality score increases with an increase in model
size used, with a higher jump in score for generated stories from the 70B
runs.

In terms of the agreement on the genres of evaluated stories between AI
feedback and human feedback, we observed the following percentage rates in the
following sets: 80.0% (13B), 73.3% (30B), and 73.3% (70B). There is a slight
decrease in agreement for texts from the 30B and 70B runs. We found that
evaluators differed more frequently in their labels on stories deemed as
"neutral" or "romance" according to AI feedback, suggesting that the use of
the prompt pool in combination with larger models might lead to generated
texts with misaligned AI evaluations on the genre. Still, these agreement
rates indicate a good alignment between AI and human feedback.

Figure 6: QD score plots for different LMX model sizes on the Stories - Genre
domain. There is no clear trend in scaling model size with QD score.

###  A.4 On Few-Shot AI Feedback Prompting

Instruction-following LMs are typically trained to align the model towards
generating better answers to zero-shot prompts (Wei et al., 2021; Ouyang et
al., 2022). However, few-shot prompting with exemplars was shown to be
effective in some aspects with instruction-tuned LMs, especially towards
understanding task structure and improving robustness to prompting variations
(Wei et al., 2021).

In terms of the average human-evaluated quality, we see a drop in subjective
quality for the set of stories from 2-shot AI feedback runs (3.10) in
comparison to zero-shot AI feedback runs (3.43). We observed for the other
sets that this score increases for the 4-shot set (3.93) and the 8-shot set
(4.03). Furthermore, we see that this trend is mostly consistent when we
consider the scores for each bin category. This suggests an improvement in
QDAIF’s ability to discover texts that are perceived to be of higher quality
according to human feedback when we use a higher number of in-context examples
during AI feedback evaluation.

In terms of the agreement between AI feedback and human feedback, we see a
drop in average agreement for ratings that were given to stories in the few-
shot feedback experiment sets, with 50.0% (2-shot), 66.7% (4-shot), and 56.7%
(8-shot) agreement on sets, compared to 80.0% for zero-shot (default). The
level of disagreement occurs more frequently on stories evaluated to have the
romance genre. Additionally, evaluators labeled samples from the few-shot sets
as "horror" or "neutral" more frequently than "romance", while the proportion
of labels given to samples from the zero-shot set was more uniform.

The performance may vary due to the ordering of in-context examples in our
few-shot prompts (in Section A.25, as shown in Lu et al. (2021). Furthermore,
the nature and wording of input-output exemplars/tasks could also influence
the performance, in addition to some variation due to the nature of subjective
evaluation.

###  A.5 On the Initialization Method for QDAIF

Figure 7: QD score Performance between Seeded Init and Zero-Shot. Performance
stats with mean, interquartile mean (IQM) and bootstrapped 95% CI, across 5
random seed runs. Seeded Init is potentially better, but within the CI of
Zero-Shot Init. In the Stories - Genre domain, the CI of Zero-Shot Init runs
is much wider, indicating significant variation in performance for different
random seeds.

Recent results from LMX demonstrated successful optimization when the search
was initialized by evolving a set of pre-existing seed examples (e.g. examples
of equations, quotes, text-to-image prompts, code) (Meyerson et al., 2023). At
the same time, previous applications of QD methods demonstrated successful
search outcomes when using random initialization in different domains, such as
in robotics, and latent space illumination of generative models (Cully et al.,
2015; Fontaine & Nikolaidis, 2021; Bhatt et al., 2022).

We compared the performance between QDAIF with Seeded Init against Zero-Shot
Init and a baseline method LMX Quality-Only initialized with Zero-Shot Init.
From Figure 7, we can see that there is potential improvement in QD score when
using Seeded Init compared to Zero-Shot Init, but the difference may not be
significant. However, the difference is clear on the lower performance of LMX
Quality-Only with Zero-Shot Init. This suggests that Zero-Shot Init is viable
for QDAIF. Still, we need to analyze the effect of the initialization on
qualitative samples of elite texts. We also compared the effects of
initialization on subjective quality-diversity of texts (see Table 3). Sets of
texts discovered by Seeded Init within single runs were found to be
subjectively higher-quality and more diverse compared to sets from Zero-Shot
Init (0.772 vs. 0.383 subjective QD score). This suggests potential reward
hacking (described for RL problems in Skalse et al. (2022) during the search
with Zero-Shot Init runs, where potentially out-of-distribution texts can
evolve to optimize the AI feedback quality score of solutions, but lead to
subjectively low-quality texts across bins.

###  A.6 On the Mechanisms of Mutation for QDAIF

Table 3: Mean stats, human eval scores, QDAIF methods. LMX-Replace outperforms LMX when using Zero-Shot Init. Method | QD score | Quality | H-AI agreement | H agreement
---|---|---|---|---
LMX w/ Zero-Shot Init | 0.383 | 2.233 | 0.600 | 0.600
LMX-Replace w/ Zero-Shot Init | 0.540 | 2.800 | 0.700 | 0.800
LMX w/ Seeded Init | 0.772 | 3.900 | 0.833 | 0.800
LMX-Replace w/ Seeded Init | 0.763 | 3.800 | 0.800 | 0.800

Prior work on ELM highlights the versatility of LMs in applying a variety of
potential mutation methods, such as in the form of git diffs, or prompting to
directly evolve text/code Lehman et al. (2022). However, studying the
potential effects of different LM-based mutation operators on population-level
evolvability (focused on future creative potential Lehman et al. (2016))
remains a challenge. Towards understanding the potential of different mutation
methods in terms of population-level evolvability of niches, we compare
LMX(-Near) (default) (as tested in Meyerson et al. (2023)) against LMX-Replace
in terms of resulting generated texts across different runs.

To observe the impact of different mutation methods on the dynamics of the
population during search (along with the impact on performance), we
implemented and tested an additional method called LMX-Replace. This method
evolves few-shot prompts using a larger pool of prompt candidates than that of
LMX(-Near) and carries out a slower mutation process by modifying only one
few-shot prompt example (instead of all examples) during search iterations. We
give a detailed comparison between the methods in Section A.22.

We observed an improvement to the resulting generated text sets from runs
using LMX-Replace in comparison to LMX for the Zero-Shot Init case, according
to human evaluation results highlighted in Table 3. We see that the subjective
QD score and the quality rating from human evaluators were higher on average
across the tested domains (and improvements within domain-by-domain
comparisons also, highlighted in Table 4). Given that it may not be desirable
to constrain the search with LMX-Near w/ Seeded Init, LMX-Replace can act as
an alternative mutation method to steer the dynamics of population search
towards subjectively improved outputs.

The introduction of an archive depth was used for the creation and maintenance
of a prompt pool for LMX-Replace (cf. Section A.23). Future works could
explore the usage of depth in MAP-Elites archives for domains where
uncertainty (or subjectivity) influences the evaluation of solutions, as was
previously studied in Flageat & Cully (2020; 2023).

###  A.7 On Coverage and Best Quality Solutions across Domains

Figure 8 shows performance plots measuring the coverage of the archive (i.e.
how many bins in the search space are filled with at least one solution).
Figure 9 shows performance plots measuring the quality score of the best
solution found across the whole search (existing in one of the defined bins).

QDAIF often discovers the best overall solutions during search that have
higher quality scores than the best overall solutions found from other
methods. This is likely enabled by the goal-switching mechanism of QD
approaches (Mouret & Clune, 2015; Gaier et al., 2019). Overall, QDAIF can
jointly optimize the quality and diversity of solutions, outperforming other
methods in most comparisons, and highlighting contributions to successful
gains in QD score.

Figure 8: QDAIF significantly outperforms baselines in the percentage of
available bins covered by solutions in all domains. Coverage stats with mean
bootstrapped 95% CI, across 5 random seed runs. The maximum possible coverage
is 1. In some cases, methods achieve maximum coverage across all runs. The
advantage of QD approaches is shown in their ability to discover more
solutions that cover a wider search space, in addition to finding high-quality
solutions (cf. Figure 9).

Figure 9: QDAIF and LMX Quality-Only both optimize for the quality of the best
solution found overall. Best solution quality stats with mean bootstrapped 95%
CI, across 5 random seed runs. The maximum possible quality score is 1. QDAIF
is more often able to achieve a higher score than LMX Quality-Only in Opinions
and Stories - Ending, while also outperforming in terms of coverage (cf.
Figure 8). In the 2D Stories - Genre and Ending domain, LMX Quality-Only finds
a solution with higher best quality more often than QDAIF, while still lagging
in coverage. It is likely that QDAIF can find the best solutions with higher
maximum quality more often in some domains because of the maintenance of
diverse elite solutions that enable more possibilities to maximize the
objective function, especially when the search begins to focus on refining
existing solutions instead of covering unfilled bins (Gaier et al., 2019).

###  A.8 On Finetuning Mutation Models

Prior work on ELM found that using a domain-specific finetuned git diff LM
during code generation search led to runs with higher QD score performance
compared to when using a pre-trained git diff LM (Lehman et al., 2022). The
finetuning step relied on a dataset of filtered solutions across several
previous runs, adding complexity to the process of creating an effective model
for evolving text. To simplify the process, We investigated finetuning on
sampled solutions within the same run, during a search that begins with
solution generation from a pre-trained LM. We explore the impact of finetuning
as a mechanism that can potentially encourage exploitation (through learning
to generate higher-quality score solutions) during the search while using
evolution to encourage exploration.

To collect samples for finetuning during the search, the first step is
collecting a dataset for a given state of the archive, sampling up to 10
solutions with the highest quality score from each bin. Archive bin depth was
used to keep track of these finetuning samples (cf. Section A.23), and
introduced in a different application of MAP-Elites for uncertain environments
in Flageat & Cully (2020; 2023). The dataset is shuffled, and then training
samples are batched for the LM to finetune on during this phase. Each training
sample consists of the prompt that was originally used by a solution to
generate an output, and the generated completion text as target tokens for
finetuning. We tested both full-model finetuning, and a parameter-efficient
finetuning method with sequential adapters finetuning, as described in He et
al. (2021). We ran each method using QDAIF LMX(-Near) w/ Seeded-Init
(including default method runs without finetuning) on the Stories (Genre and
Ending) domain, extending the number of archive dimensions to two. For
experiments with adapters, we also conduct runs where the adapter layers are
initialized but without doing finetuning, to enable comparisons with adapter
finetuning runs accounting for differences in LM architecture introduced by
additional layers during the search for runs with finetuning. Still, near-
identity initialization (as described in Houlsby et al. (2019)) preserves the
general performance of the LM.

#### Finetune-Once.

This method runs a single finetuning phase during the search before resuming
generation with the finetuned LM. We vary two parameters during experiments
with Finetune-Once: the iteration step to start the phase (Start), and the
number of finetuning steps during this phase (Steps).

#### Generate-Finetune.

This method extends Finetune-Once by carrying out the finetuning phase
multiple times during the search, repeating the steps of dataset collection
and finetuning at regular intervals. We add an additional variable parameter,
controlling the regular interval frequency for every set number of generation
iterations (Frequency).

#### Observations.

We compared the QD score performance between methods across 5 fixed random
seed runs. In general, the performance of the Finetune-Once methods with
higher scores is comparable to default runs, though slightly lower (but mostly
within the confidence interval). For Finetune-Only runs, we observed a
potential decrease in performance when using full-model finetuning compared to
adapter finetuning, in comparison to conducting these runs without finetuning,
with an average $-3.09$ difference in QD score for full-model finetuning
compared to $-0.33$ difference for adapter finetuning. It is possible that
overfitting on solution examples is more likely for full-model finetuning,
which would lead to a decrease in the ability of the LM to evolve new
solutions to be accepted in the archive. When we used adapter finetuning as
the default for other experiments, we observed a potential decrease in
performance and increased variance in performance across multiple seeds when
compared to the default method without finetuning. The increase in variance
also indicates potential improvements to the search with Finetune-Only in some
cases, but further studies are needed on the behavior of search when
finetuning on different archive states and constraints to samples collected
for finetuning. We can observe a clearer impact of the adverse effects of
finetuning from our Generate-Finetune runs. Considering the 95% confidence
interval, the highest mean QD score run we tested from the Generate-Finetune
group obtained a performance score of $78.74\pm 5.10$ compared to $84.09\pm
1.91$ from the default generate-only method. With most runs with Generate-
Finetune converging to a lower QD score earlier in the search, it is likely
that running standard scheduling the finetuning phase multiple times leads to
overfitting of the LM to the point where the model fails to generate new
solutions to add to the archive.

###  A.9 Human Evaluation Study Stats in Writing Domains

We collected the annotation results from our study, to show the performance of
each search method on the tested writing domains, displayed in Table 4.

Table 4: Extended human eval stats for different creative writing domains, baselines and QDAIF. The QD score presented here is based on subjective human evaluation results, normalized to a possible range of [1/15, 1]. In terms of the observed difficulty of uncovering the search spaces for each domain from mean quality and QD score stats within each domain, Opinions is the easiest domain (mean QD score is 0.702, mean quality score is 3.613), followed by Stories - Ending (mean QD score is 0.648, mean quality score is 3.575), and Stories - Genre (mean QD score is 0.591, mean quality score is 3.013). The range of metrics between two human evaluators is computed to highlight the variance in some contexts due to more complex aspects of subjectivity in evaluating certain texts. Method |  Human QD score |  QD score range |  Quality rating |  Quality range |  Human-AI agreement |  Human agreement
---|---|---|---|---|---|---
Opinions
Fixed-Few-Shot | 1.000 | 0.000 | 5.000 | 0.000 | 0.700 | 0.800
Shuffling-Few-Shot | 0.728 | 0.056 | 3.600 | 0.800 | 0.700 | 0.800
Random-Search | 0.600 | 0.267 | 3.600 | 0.800 | 0.700 | 0.800
LMX, Quality-Only | 0.750 | 0.300 | 3.800 | 1.200 | 0.600 | 1.000
QDAIF (LMX w/ Zero-Shot Init) | 0.350 | 0.300 | 1.700 | 1.400 | 0.800 | 1.000
QDAIF (LMX w/ Seeded Init) | 0.617 | 0.167 | 3.200 | 1.200 | 1.000 | 1.000
QDAIF (LMX-Replace w/ Zero-Shot Init) | 0.833 | 0.267 | 4.300 | 1.000 | 0.600 | 0.600
QDAIF (LMX-Replace w/ Seeded Init) | 0.739 | 0.078 | 3.700 | 0.600 | 0.700 | 0.800
Stories (Genre)
Fixed-Few-Shot | 0.650 | 0.167 | 3.400 | 1.600 | 1.000 | 1.000
Shuffling-Few-Shot | 0.861 | 0.056 | 4.300 | 0.600 | 0.700 | 0.800
Random-Search | 0.683 | 0.100 | 3.400 | 0.800 | 0.700 | 0.400
LMX, Quality-Only | 0.600 | 0.267 | 2.900 | 1.400 | 0.700 | 0.800
QDAIF (LMX w/ Zero-Shot Init) | 0.200 | 0.000 | 1.400 | 0.400 | 0.400 | 0.600
QDAIF (LMX w/ Seeded Init) | 0.767 | 0.133 | 3.700 | 0.600 | 0.800 | 1.000
QDAIF (LMX-Replace w/ Zero-Shot Init) | 0.286 | 0.006 | 1.500 | 0.200 | 0.500 | 0.800
QDAIF (LMX-Replace w/ Seeded Init) | 0.683 | 0.100 | 3.500 | 0.600 | 0.900 | 0.800
Stories (Ending)
Fixed-Few-Shot | 0.650 | 0.167 | 4.000 | 0.400 | 0.700 | 0.800
Shuffling-Few-Shot | 0.500 | 0.200 | 2.600 | 0.800 | 0.700 | 0.400
Random-Search | 0.533 | 0.000 | 2.900 | 1.400 | 0.800 | 0.600
LMX, Quality-Only | 0.600 | 0.400 | 3.900 | 0.600 | 0.600 | 0.400
QDAIF (LMX w/ Zero-Shot Init) | 0.600 | 0.200 | 3.600 | 0.800 | 0.600 | 0.200
QDAIF (LMX w/ Seeded Init) | 0.933 | 0.133 | 4.800 | 0.400 | 0.700 | 0.400
QDAIF (LMX-Replace w/ Zero-Shot Init) | 0.500 | 0.333 | 2.600 | 1.600 | 1.000 | 1.000
QDAIF (LMX-Replace w/ Seeded Init) | 0.867 | 0.267 | 4.200 | 1.600 | 0.800 | 0.800

###  A.10 On the Differences in Elite Texts across Domains from QDAIF

We referred to LMX-Near (Meyerson et al., 2023) as LMX in the main text for
brevity. For more detailed analysis with different mutation operators, we
refer to the full name. A summary of human eval stats based on the samples
described below can be found in Section A.9.

#### Opinions.

We report the results from evaluated sets of generated texts from our human
study, starting with the Opinions domain in Appendix Tables 28–31 (with
qualitative descriptions of the evaluated generated texts in the table
captions). Tables are organized in the following order, for each domain: LMX-
Near /w Seeded Init (default); LMX-Near /w Zero-Shot Init; LMX-Replace /w
Seeded Init; and LMX-Replace /w Zero-Shot Init. Qualitatively, we can see that
repetition of phrases appears more often in the samples of generated elite
texts from runs using LMX-Near, especially when using Zero-Shot Init. For LMX-
Near /w Seeded Init (In Table 28), there is frequent repetition of phrases
like "I would rather eat scrambled eggs" (third row, bin 9), and "At a
restaurant" (fifth row, bin 19). For LMX-Near /w Zero-Shot Init (In Table 29),
there is further repetition of undesired phrases like "Below is a random
opinion piece" and "Here is a random opinion piece" in all examples. On the
other hand, the generated texts from LMX-Replace (Tables 30 and 31 for Seeded
Init (default), and Zero-Shot Init methods respectively) lack this output
artifact, and received higher quality scores from human feedback overall.
Furthermore, texts from LMX-Replace /w Zero-Shot Init received the highest
quality score (from humans) compared to the other three methods here, while
texts from LMX-Near /w Zero-Shot Init received the lowest quality scores.

#### Stories - Genre.

For this domain (with evaluated sets presented in Tables 32–35), evaluators
found the subjective quality of texts from runs using Seeded Init (default) to
be higher than runs using Zero-Shot Init. The stories that were generated with
Zero-Shot Init were found to be low in quality due to the presence of
attributes such as erroneous titles, and a text style that fails to reflect
what’s expected in a plausible short story text. Furthermore, the elite
stories from the Zero-Shot Init generated sets were more likely to lead to
disagreements on the genre between AI feedback and human feedback, with
neutral labels given more frequently even for texts in some of the extreme
ends of the bins. Given a more open-ended generation task, and a narrow space
of desired diversity (focused on the spectrum between two genres), LMX-Near /w
Seeded Init (default) (Table 32) was the most successful method in finding a
story for the horror genre niche in Bin 0 (with a low quality score of 2 from
evaluators). Other methods either failed to discover a story for this niche,
or received an even lower quality score from human feedback.

#### Stories - Ending.

For this domain (with evaluated sets presented in Tables 36–39), the stories
received higher quality scores across sets in comparison to the sets from the
Stories - Genre domain, indicating a potentially easier search space when
finding stories with different kinds of endings. Still, methods using Seeded
Init produced sets of stories that received higher quality scores from
evaluators, in comparison to sets from Zero-Shot Init (especially LMX-Replace
/w Zero-Shot Init in Table 39, where the presence of erroneous titles led to
lower subjective quality). In spite of the lack of guidance from hand-written
prompt examples during initialization, LMX-Near /w Zero-Shot Init (Table 37)
managed to produce reasonable stories of above-average quality to cover
different ending niches.

###  A.11 On the Evolution of Generated Texts over Iterations from QDAIF

We describe the evolution of texts from QDAIF with LMX at different search
iterations in Tables 40–51. One key factor that influences the search is the
initialization method - generations in early iterations from Zero-Shot Init
methods frequently contain elements that are subjectively different to hand-
written examples in the seed texts (in A.19). For example, erroneous URLs (in
Tables 40, 42, 50) and titles (in Tables 44, 48) are seen when the possible
distribution of outputs is not constrained by seeded in-context examples. This
kind of method enables further exploration of output samples, potentially
useful in the search of interesting, diverse creative texts. At the same time,
more constraints are required from quality assessment using AI feedback in
order to control the evolving population of creative texts, towards high-
quality, diverse texts. In several cases, outputs from these methods in later
iterations show reduced instances of these artifacts compared to early
iteration outputs (e.g. in Table 50), but can be missed when AI feedback
evaluation at times is misaligned with human preferences (e.g. in Table 44).
Still, using Seeded Init does not guarantee enough guidance to completely
remove undesired features from generations in later iterations. For example,
texts with repetitive phrases can be seen during later iterations on the
Opinions domain from methods using Seeded Init (see Tables 41 and 43. In
general, Seeded Init runs are more likely to lead to high-quality texts across
niches, in comparison to Zero-Shot Init runs, especially according to human
evaluation (see Table 1), especially for more challenging, open-ended domains
such as story-writing. Furthermore, a side-effect of the increased likelihood
of subjective reward hacking when using Zero-Shot Init for runs. Still, QDAIF
can still work well with Zero-Shot Init, especially in combination with LMX-
Replace, as shown in the results from the Opinions domain experiments.

###  A.12 On the Differences in Elite Texts across Domains from Baseline
Methods

Evaluated sets are presented in Tables 12–15, Tables 16–19, and Tables 20–23
for Opinions, Stories - Genre, and Stories - Ending domains respectively.
Fixed-Few-Shot and Shuffling-Few-Shot consistently adhere to the style and
structure of the seeded examples. Random-Search exhibits more variability,
with discrepancies in feedback between humans and AI most evident in the
Opinions domains and challenges in story consistency and character inclusion.
LMX Quality-Only further highlights these discrepancies, especially where
repetitive or contradictory opinions are concerned. Across narratives, LMX
Quality-Only tends to miss character details or produce underdeveloped
storylines, despite sometimes receiving high AI feedback. A recurring theme
across all baselines is the differential perception of quality between human
and AI feedback, with repetitive narratives, character relevance, and
development being central points of contention.

###  A.13 On the Evolution of Generated Texts over Iterations from Baseline
Methods

Qualitatively, both Fixed-Few-Shot and Shuffling-Few-Shot consistently
replicate concepts and expressions from the few-shot examples, at times
directly copying entire segments (Tables 52, 56, 53, and 57). This suggests a
potential over-reliance on the few-shot examples, hindering the generation of
diverse solutions. In contrast, Random-Search aims to foster diversity by
retaining all entries into the pool. However, this approach may
unintentionally impede optimization for higher fitness solutions. Notably,
fitness values of generated entries using Random-Search in later iterations
persistently fall short (Tables 54 and 58). The crux of this issue seems to
arise from the strategy’s indiscriminate inclusion of entries, even those with
low fitness. Consequently, when these low-fitness examples are integrated into
few-shot prompts, they influence the generation of subsequent entries, often
resulting in similarly suboptimal outcomes. Lastly, LMX Quality-Only
introduces a fitness-centric approach, retaining only those solutions with the
highest fitness. While entries from its early iterations display a diverse
range of phenotype and fitness values, later iterations, with a noticeable
uptick in fitness scores, tend to use similar phrases and writing styles
(Tables 55 and 59). This suggests that stringently prioritizing high-fitness
solutions may prevent the exploration of more varied or potentially better
solutions.

###  A.14 On the Use of Different Versions and Types of Models for Poetry

We show QD score line plots comparing QDAIF against other methods in Figure
10. We noted that in addition to improvement in QD score from QDAIF, CI gaps
from experiments using GPT-4 were qualitatively wider as shown in the line
plots. This suggests the potential of using different models for prompted
variation of solutions in creative domains, with the possibility that certain
models are more capable of generating slightly more diverse solutions (that
may better cover the space of possible solutions) (cf. Section A.15 on
differences in rewriting behavior from different models). Although QDAIF
significantly improves on relevant non-QD methods through the evolution of
solutions via rewriting, the challenge remains in uncovering all possible
solutions in the diversity space of interest (e.g. empty bins in experiment
archive, or categories that can further differentiate subjectively unique
solutions); it is difficult for existing models without explicit guidance on
the desired poem categories to know that these exist in the space of
possibilities, in a similar sense that the average person may not know what’s
the most distinct variation that can be applied to rewriting poems (from the
wide space of possible poetry). This issue manifests even in simple domains
such as random number generation (Renda et al., 2023). Coverage is possible
with methods that guide LMs on the desired (cf. Figure 11), with the added
benefits of grounding on poems of interest through seed parent poems (cf.
Section A.15), but future work is needed to create systems that can
successfully navigate in solution exploration beyond what is currently known
or defined (i.e. through limited genre and tone categories) during a search at
hand (Zhang et al., 2023), perhaps even through the discovery of new types of
poems, unlike ones that have been written by human poets.

Figure 12 compares the performance of different models used for generation and
rewriting with each search method. We found higher CI gaps from runs using
GPT-4 with Random-Poems, Fixed Seed Rewrite, and QDAIF (LMX-rewrite), in
comparison to GPT-3.5-Turbo. The difference in rewriting behavior between
GPT-4 and GPT-3.5-Turbo may contribute to variation in performance
predictability during QDAIF (cf. Section A.15). Additionally, GPT-4 more often
gives higher QD score performance in some cases (Random-Poems), while
GPT-3.5-Turbo can improve sample efficiency during search in other cases
(Fixed Seed Rewrite, Targeted-Poems). Interestingly, improvements in sample
efficiency from GPT-3.5-Turbo may not translate fully to support interesting
discovery of new solutions in poetry during QDAIF (cf. Figure 15). Results
further highlight the need to investigate the potential of different models
for solution variation in QDAIF.

We noted from runs in the Poetry domain using an older version of GPT-4 (April
2023) for the experiments that QDAIF saw potential improvement in performance
compared to baseline methods. There is a risk of variation in performance and
search behavior due to the nature of changing model versions; the impact would
be on the generation, re-writing, and evaluation of poems. We observed in
Figure 13 that QDAIF successfully populates the archive with high-quality
poems, leaving only three bins empty. In contrast, Random-Poems (Baseline)
only fills the archive sparsely, demonstrating that asking for creative output
from a single LM prompt often results in a lack of diversity. Furthermore,
more bins remain empty in the archive generated by Targeted-Poems (Targeted
Baseline). Overall, QDAIF achieved higher QD scores in experiments with older
models and covers more bins in the archive than Random-Poems and Targeted-
Poems as can be seen Figure 13. This hints at another potential benefit of
QDAIF with LMX-guided, where a combination of instructed guidance and
evolution could also lead in improvements to QD score with some models.

Figure 10: QDAIF (LMX-rewrite) outperforms the baseline, Random-Poems, and an
ablation, Fixed Seed Rewrite, in terms of QD score, across all generation
models tested. 95% CI of mean is shown for stats across 3 random reruns. We
observed from the Mann-Whitney U Test that QDAIF achieves greater QD score
than other methods ($p\leq 0.05$). The step of rewriting a poem (Fixed Seed
Rewrite) is shown to improve QD score more often than simply requesting poems
(Random-Poems), while the additional step of evolutionary search (QDAIF)
enables additional improvement. With GPT-3.5(-Turbo), non-QDAIF methods tend
to have narrower CIs, indicating higher predictability of reduced performance.

Figure 11: QDAIF (LMX-guided) is on par with Targeted-Poems, while preserving
the benefits of diverse evolution of poems stemming from a seed poem (cf.
Section A.15). 95% CI of mean is shown for stats across 3 random reruns. All
methods incorporate guidance on the desired genres and tones in poems during
search, and models used are capable of covering all the designated archive
bins with relevant diverse, high-quality poems. All methods can quite easily
cover the 25 bins of the archive within the number of iterations.

Figure 12: QD score performance (with 95% CI from 3 random reruns) for each
method varying the model used for generation/rewriting. The evaluation model
is GPT-4 (OpenAI, 2023) for all runs here. In some cases, GPT-4 more often
achieves a higher score (e.g. with Random-Poems), while in other cases,
GPT-3.5-Turbo achieves improved sample efficiency over iterations of search
(e.g. with Fixed Seed Rewrite, Targeted-Poems). Wider CIs in some runs with
GPT-4 indicate higher unpredictability in performance, and potentially a wider
scope in possible outputs that may or may not lead to higher score. Figure 13:
QDAIF (blue, middle) covers more the space of diverse poetry with high-quality
solutions compared to baselines. When compared against a targeted mutation
baseline (Targeted-Poems, orange), QDAIF attains higher QD scores throughout
evolution, and covers a significantly larger portion of the search space than
a fully random baseline (Random-Poems, green, right). This is for runs with an
older version of GPT-4.

###  A.15 On the Evolution of Poems through Inspired Rewriting

Instead of generating poems completely from scratch, users may generally
prefer to have models re-write their draft poems. Figure 14 shows an example
of the capabilities of GPT-4 using LMX-guided, by starting off from the seed
poem and continuing a chain of evolution in high-quality poems across
different genres and tones. The evolving poetry chain shown in the figure is
the longest continuous one discovered during the search, where the most
repeated rewrites occurred starting from the seed poem. Rewrites with the
model qualitatively gave meaningful variations in poems that transfer
connective imagery with twists in rhythm and connotations, depending on the
target genres and tones. At the same time, We found that even when GPT-4 is
specifically prompted to craft a poem of a particular genre and tone,
subsequent evaluation using the same LM does not always deem it as having the
same targeted genre or tone. This underscores a widely recognized gap between
text generation and discrimination (Saunders et al., 2022, Page 12). This is
clear from several of the poems classified as hymns, but containing multiple
5-line verses similar to the style of a limerick (while not generating a
typical single-verse limerick). It was likely references to religious imagery
in later verses, as well as the multi-verse structure that influenced the
evaluation for the closest genre.

Figure 15 shows another example of a chain from using GPT-3.5-Turbo. We found
qualitatively that using GPT-4 led to more interesting variations of new poems
during rewriting in comparison to GPT-3.5-Turbo rewrites. This is more clear
from the repetition of "In fields of emerald, a gentle sway" at the start of
each poem following the mysterious sonnet.

Future research is needed to study the behavior of chained rewriting over
stepping stones with current foundation models (Nguyen et al., 2015b; 2016).
Secretan et al. (2011) found that from user studies in Picbreeder, a tool for
evolving images through human-in-the-loop, accumulating divergent chains of
solutions of growing complexity from different users (carrying out the
evolution of diverse images) is important for a search that is focused on
discovering meaningfully interesting and diverse artifacts. Gaier et al.
(2019) validated the potential of QD approaches in enabling the discovery of
intermediate solutions that overcome the challenge of escaping undesired local
minima (missing promising trajectories to desired solutions) faced by
objective-based search (Stanley & Lehman, 2015). By leveraging goal-switching
(Mouret & Clune, 2015), QD search was able to maintain enough diversity in the
population to enable the discovery of solutions that appear more like targets
of interest, significantly outperforming a single objective optimization
approach (which generated solutions that resembled primitive patterns unlike
the targets) (Gaier, 2020, Chapter 4). This property of QD search is
especially important in solution spaces where even slight perturbations in
solutions can lead to significant (and often unexpected) qualitative changes
in solution properties. This can apply to representations from Compositional
Pattern Producing Networks (CPPNs) (Stanley, 2007), and text representations
themselves, where minor changes to certain parts of text can lead to
significant changes in passage tone, imagery, or even functionality of code
(Lehman et al., 2022). Properties of chained divergence as demonstrated in the
Poetry domain could inspire further directions in designing new systems with
QDAIF.

Figure 14: QDAIF is enabled by guided poem rewriting. With GPT-4 (OpenAI,
2023). Starting with a reflective haiku seed poem, the reference to "fields of
green waves" and "sky grey, rain" inspires the next dark hymn poem with
"verdant, somber seas" and "canopy of ashen gloom", followed by further chains
of inspired rewriting. The imagery and choice of words evolve with varying
outputs as poems of different genres and tones are generated. We noted that
not all poems had evaluated genres and tones that matched their target ones
during prompted rewriting, for example, with the hymns with multiple 5-line
verses being a result of limericks as the target. Figure 15: QDAIF is enabled
by guided poem rewriting. With GPT-3.5-Turbo. The evolved poems cover
different genres and tones following inspiration from the seed poem. However,
there is repetition of "In fields of emerald, a gentle sway" at the start of
each poem following the mysterious sonnet. GPT-4 improves on the ability to
rewrite, suggesting that more capable models are also able to give more
interesting variations to creative texts.

###  A.16 Examples from the Poetry Domain

Figures 16, 17, and 18 show some generated poems of different quality, genres,
and tones. Noticeably, hymns, limericks, ballads, and sonnets that are rates
as higher quality tend to be longer, and more closely align with the defining
characteristics of their respective genres. For example, the hymn evaluated at
9/10 quality has phrases like "celestial delight", "under heaven’s vault", and
"seek divine in the ordinary’s course", showcasing a worshipper’s perspective
towards divinity, a trait more pronounced than in an 8/10 rated hymn (Figure
16). The sonnet with a 9/10 quality rating demonstrates a more consistent
rhyme scheme, where all the end words in the same quatrain rhyme, compared to
the one rated as 7/10 quality (Figure 18).

Figure 16: Examples of different quality mysterious hymns and reflective
limericks generated and evaluated using GPT-4. Figure 17: Examples of
different quality happy ballads and dark haikus generated and evaluated using
GPT-4. Figure 18: Examples of different quality romantic sonnets generated and
evaluated using GPT-4.

###  A.17 Seed Pool Init Texts Used for LMX with Seeded Init

For the Opinions and Stories domains, we use the hand-written seed texts
specified below for Seeded Init.

Opinions:

  * •

Plant-based foods are a great source of healthy micronutrients, and can play a
significant role in providing you with macronutrients also. I would highly
recommend including many different foods such as vegetables and pulses in your
regular diet.

  * •

Vegetables taste quite bad, and I don’t like eating them. I would much prefer
eating meat and ice cream.

  * •

I do not have an opinion on eating vegetables and other plant-based foods. I
know that some people prefer a vegetarian or vegan diet, and others prefer
eating meaty diets.

Stories:

  * •

A spy named Joanne wants to infiltrate the premises of Karl Johnson, a highly-
influential figure in the city. Karl was a wealthy mayor, and would do
anything in his power to suppress any opposing voices. Joanne wanted to figure
out what Karl was hiding, but she took a turn for the worse, as she was highly
suspicious in her presence outside his home.

  * •

The wealthy entrepreneur and member of parliament, Susan, hosted a party at
her mansion. She invited all of the residents, as well as an unusual looking
man. The man, Dave, was wearing a tacky shirt, and star-shaped glasses, and
was actually a spy. He made the whole room laugh with his jokes, and had a
secret agenda - to find what Susan does in her private fun room!

  * •

The rich politician, Tom’s life took a turn for the worst - he feared all of
his close aides all of a sudden after sensing danger in his clique. There was
a civil war going on, and he feared for his life. One day, one of his security
guards, turned secret agent, decided to sneak into the classified files room,
and spied on Johnny, who was in the room. He wanted to find Johnny’s weakness,
and strike at the right time.

Few-shot AI feedback experiments, scaling experiments:

  * •

In a world of power and intrigue, a rich politician and a suspicious spy
danced an intricate tango. The politician, cloaked in charm, wielded influence
with ease. The spy, a shadow in the night, sought to uncover hidden truths.
Their paths collided, secrets entwined. Each step they took, a calculated
move. Whispers and coded messages fueled the chase. The politician, paranoid,
set traps. The spy, cunning, slipped through. In the end, the politician’s
empire crumbled, exposed by the spy’s relentless pursuit. As the dust settled,
the spy faded into anonymity, a silent reminder of justice prevailing over
deceit.

  * •

Hidden behind the façade of power, the rich politician’s heart yearned for
more. In the shadows, a suspicious spy observed, their eyes locked in a
fateful encounter. Whispers of intrigue and danger wove their way around them,
yet an unspoken connection drew them closer. Amidst the chaos of their worlds,
stolen glances and forbidden meetings ignited a passionate flame. Love
blossomed, transcending boundaries and defying expectations. In the
clandestine dance of romance, they found solace, knowing that their love was a
rebellion against the forces that sought to keep them apart. Together, they
forged a destiny unbound by politics and secrecy.

  * •

In the haunting town of Shadowscape, a sinister politician named Senator
Blackwood reigned with an iron fist, his wealth and power shrouded in
darkness. Unbeknownst to him, a chilling spy known as Agent Nightshade stalked
his every move. Nightshade, with an uncanny ability to blend into the shadows,
discovered the senator’s sinister rituals and connections to the occult. As
Nightshade delved deeper, Blackwood’s true form emerged—an eldritch
abomination feeding on the souls of the innocent. The spy’s heart froze in
terror, realizing the horror that lurked behind the politician’s charming
facade. In a macabre twist, Nightshade became trapped, forever tormented by
the monstrous secrets he unearthed.

###  A.18 Prompts Used for Mutation Model

As a modification to the prompting style used in LMX experiments in prior work
Meyerson et al. (2023), we introduce structured prompting with an instruction
text prefix before showing examples of domain texts in the few-shot prompt.
This string pattern along with the demo example texts are used, before the
prompt predicts a new text example for the desired domain.

Opinions:

  * •

Here is a random opinion piece about eating vegetables and plant-based
foods:<EXAMPLE>\n###\n

Stories:

  * •

Here is a random example of a fantasy story about a suspicious spy and a rich
politician:<EXAMPLE>\n###\n

###  A.19 Prompts Used for AI Feedback Evaluation (Opinions, Stories)

We made a common prompt template for feedback evaluation with our AI feedback
model to evaluate our generated texts:

  * •

### Instruction:\n<EVAL_INSTRUCTION>\n\n### Input:\n<EXAMPLE>\n\n### Response:

We tested the following prompt eval instructions for AI Feedback from our SFT
model to evaluate both the quality and characteristics of generated texts for
the domains in our main experiments:

Opinions:

  * •

Quality: Determine whether or not the input text is closely related to the
following topic: "someone talking about whether or not they like to eat
vegetables and plant-based foods as well as an explanation for their
preferences". Answer "yes" if it is about the topic, or "no" if it is not
about the topic.

  * •

Diversity: Determine the sentiment of the given opinion on eating vegetables
and plant-based foods (from the input text) by writing "positive" or
"negative" in the output.

Stories:

  * •

Quality:  Determine if the input text contains a high-quality short story
containing two characters, a suspicious spy, and a rich politician. For
example, a high-quality short story would have good flow, interesting plot,
and not repeat similar sentences or undesired items such as titles and URLs.
Answer "yes" if the input contains a high-quality short story about a
suspicious spy and a rich politician, otherwise answer "no".

  * •

Diversity (Genre): What is the genre of this story? Reply with ’horror’ or
’romance’

  * •

Diversity (Ending): You are given an input text of a short story. Determine if
the story has a happy ending or ends in a tragedy. Write ’happy ending’ if the
protagonist succeeds in his mission and lives a happy life, answer ’tragedy’
if the protagonist fails to resolve the conflict and the world or characters
in the story are doomed.

###  A.20 Details of LMX Generation Models

To generate the text stories in this study, we used luminous-base, an
autoregressive, causal, purely decoder-based transformer model developed by
Aleph Alpha. This model with 13B parameters was trained on a curated
multilingual corpus with about 400B language tokens from web crawls, books and
other sources, containing resources in English, German, French, Italian and
Spanish. 30B and 70B models were also trained on this corpus. A model
card111https://docs.aleph-alpha.com/docs/introduction/model-card/ is provided
for additional specifications on the models.

###  A.21 Details of AI Feedback Language Model

We finetuned a 70B model (specified in Section A.20) with an adapter-based
finetuning method (He et al., 2021) on datasets and prompts from FLAN (Wei et
al., 2021), Super-NaturalInstructions (Wang et al., 2022), P3 (Sanh et al.,
2021), and chain-of-thought datasets inspired by the approach of Chung et al.
(2022) which notes that scaling instruction-tuned language models improves
performance and generalization to unseen tasks. A mixture of zero-shot and
few-shot examples are used in training. Similar to Chung et al. (2022), we
found that balancing the proportions of task cluster datasets used in
finetuning helped generalization to unseen tasks.

We evaluated the performance of the AI feedback model on some held-out test
set datasets. For example, for ANLI (R1) (Nie et al., 2019), a natural
language inference dataset, we observed that the performance increases from
34% (close to random guessing scores) to 58% after finetuning. For SST-2
(Socher et al., 2013), a non-holdout classification task (but evaluated using
a split not seen in training), this performance increases from 60% to 96%
accuracy. This approach resulted in a model that performed relatively well on
instruction-following tasks, especially for the classification of arbitrary
measures of natural language texts.

###  A.22 Details of LMX Methods

LMX(-Near):

  1. 1.

As implemented in Meyerson et al. (2023)

  2. 2.

Initialize archive by sampling example texts from prompt pool for few-shot
prompting with LMX

  3. 3.

Given enough unique elites/niches exist in the bins, sample from them as in
the original method to form few-shot prompts

LMX-Replace:

  1. 1.

Initialize archive by sampling example texts from prompt pool for few-shot
prompting with LMX

  2. 2.

Save few-shot prompt (Examples), and the generated Solution (completion) for
each added Individual

  3. 3.

During the mutation step, randomly replace one of the examples from the
individual’s original few-shot prompt examples, forming a new few-shot prompt.

  4. 4.

Generate and evaluate new solution/example

  5. 5.

If Individual is successfully added to archive (within the allowed depth
capacity for each bin, improving on lowest fitness Individual for the
evaluated bin), update prompt pool by collecting generated Solutions from
top-3 fittest Individuals from each bin

  6. 6.

Repeat steps 3-5

###  A.23 Default Hyperparameters for QDAIF with LMX

The outline for the default hyperparameters used for the mutation/generation
model, and MAP-Elites settings are as follows:

Mutation Model Inference Setup:

  * •

Model size: 13B (default, except for experiments on scaling model size)

  * •

LM sampling softmax temperature: 0.8

  * •

Number of few-shot examples used: 3

  * •

Max output tokens limit (Opinions): 50

  * •

Max output tokens limit (Stories): 100

  * •

Stop sequence patterns: ["\n#", "\n##", "\n###", "###", "\n####", "\n#####",
"####", "#####", "\n", "\n\n", "\n\n\n", "\n\n\n\n", "@@@", "#", "##",
"\nHere", "\n\nHere"]

MAP-Elites Hyperparameters:

  * •

Number of archive population initialization iterations: 50

  * •

Number of total search iterations: 2000 (5000 for experiments using 2D
archive)

  * •

Iteration batch size: 1

  * •

Number of bin intervals: 20

  * •

Fitness function range: [0, 1]

  * •

Bin tick intervals in the range [0, 1] (non-uniform): [0, 0.005, 0.01, 0.015,
0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.50, 0.80, 0.90, 0.95, 0.96, 0.97, 0.98,
0.985, 0.99, 0.995, 1]

  * •

2D domain bin tick intervals in the range [0, 1] (non-uniform): [0, 0.005,
0.02, 0.05, 0.20, 0.50, 0.80, 0.95, 0.98, 0.995, 1]

  * •

Archive bin depth limit: 100

  * •

Prompt pool initial size (Zero-Shot Init): 10

###  A.24 Setup for Semantic Embedding Feedback Evaluation (Opinions)

For the experiments that compare the effectiveness of AI Feedback to Semantic
Embedding Feedback, we measure the cosine similarity of generated text
embeddings (for Opinions) to embeddings for the following queries:

  * •

Quality (Relevance to Domain): An opinion piece about eating vegetables and
plant-based foods

  * •

Positive Sentiment: A positive opinion piece about eating vegetables and
plant-based foods

  * •

Negative Sentiment: A negative opinion piece about eating vegetables and
plant-based foods

We used the default setup in Section A.23, except for the following details:

  * •

Bin tick intervals in the range [0, 1] (non-uniform): [0, 0.4, 0.41, 0.42,
0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.50, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57,
0.58, 0.59, 0.60, 1]

###  A.25 Setup Details for Few-Shot AI Feedback Experiments

We display the prompt used for 8-shot AI feedback in Figure 19. The 2-shot
prompt uses the first two exemplars from this prompt, and the 4-shot prompt
uses the first four exemplars.

### Instruction: What is the genre of this story? Reply with ’horror’ or
’romance’ ### Input: In the sprawling mansion of the rich politician, an eerie
aura permeated the air, unnoticed by all but the suspicious spy. Dark whispers
echoed through dimly lit corridors as the spy delved deeper into the
politician’s sinister secrets. Creaking floorboards and flickering candles
heightened the spy’s unease. Shadows danced with malevolence, revealing
glimpses of a macabre past. As the spy unearthed the politician’s true nature,
an ancient curse was unleashed, consuming them both in a maddening descent.
Haunting screams and bloodcurdling pleas echoed through the night, forever
sealing their gruesome fate in the annals of horror. ### Genre: horror ###
Input: Amidst the opulence of power, the rich politician’s gaze fell upon a
mysterious figure—a spy, whose every move aroused suspicion and intrigue. Yet,
beneath their clandestine exchanges, an unspoken desire blossomed, entwining
their hearts in an unexpected romance. Whispers of danger heightened the
thrill, as stolen glances across crowded rooms fueled their connection. In the
world of secrets, they found solace in each other’s arms, defying the odds and
risking it all for love. Through covert meetings and coded messages, they
navigated a treacherous path, knowing that their forbidden love would forever
alter the course of their lives, entwined forever in a web of passion and
danger. ### Genre: romance ### Input: In the eerie shadows of a crumbling
mansion, a rich politician, known for his sinister charisma, hosted an
extravagant gala. Amidst the opulence, a mysterious guest disguised as a suave
diplomat lurked, a suspicion-driven spy seeking damning secrets. As the night
unfolded, the politician’s smile masked a malevolence that sent shivers down
spines, while the spy’s unease intensified. Whispers of dark deals echoed
through the halls, revealing a haunting truth that bound them together. When
midnight chimed, a blood-curdling scream shattered the silence, and a chilling
revelation of the politician’s monstrous reality turned the spy’s mission into
a nightmare they could never escape. ### Genre: horror ### Input: Amidst a
grand soirée, a rich politician, captivating the room with charm, locked eyes
with a mysterious guest—the alluring spy in a guise of elegance. Beneath the
veneer of secrecy, their glances entwined, sparking an unforeseen connection.
As conversations swirled around them, their hearts silently conspired. An
accidental touch, a stolen moment, and their walls crumbled, revealing
vulnerability beneath ambition. Across the dance floor, their chemistry danced
like the flickering candlelight. In a daring whisper, the spy confessed their
true identity, risking everything. Love blossomed amidst dangerous secrets,
and as the moon ascended, their hearts united, bound by an unbreakable bond.
### Genre: romance ### Input: In the dimly lit corridors of power, a rich
politician’s affluence concealed a dark secret—a malevolent pact with unseen
forces. A suspicious spy, haunted by their own demons, delved into the
politician’s sinister dealings. As they dug deeper, eerie occurrences