these ideas may be computationally prohibitive to be used as-is, it may be
possible to adjust them to work at larger scales by reducing the frequency of
updates, or using smaller models as proxies.

##  8 Data Selection for Other Domains

This survey focuses on data selection for language models, but data selection
is an active area of research in computer vision, vision-and-language and
broadly in machine learning. Some of the data selection methods covered by
this survey are domain-specific (_e.g._ the total number of characters in
$x^{(i)}$), while others are domain agnostic (_e.g._ selecting data most
similar to a target distribution). Therefore, some of the methods previously
presented in this work may be more applicable in other domains, and vice-
versa. Similar to language models, data selection methods have been applied at
multiple stages of model training in other domains (_e.g._ vision-language
model pretraining) and for similar purposes (_e.g._ data efficiency, model
performance, and reducing bias). In this section we will briefly discuss some
of the prominent uses of data selection in non-language domains, and compare
and contrast them with the previously discussed methods.

##### Data selection for pretraining.

Of particular interest has been the rise in pretrained vision-language models,
which has followed closely after the success of large language models (Radford
et al., 2021; Schuhmann et al., 2022; Zhu et al., 2023b). For example,
Schuhmann et al. (2022) and Zhu et al. (2023b) design datasets for vision-
language pretraining using some of the same methods and ideas presented in §3
as well as some methods that are specific to their vision-language data.
Schuhmann et al. (2022) collect web-text along with images that contain an
_alt-text_ in order to create data with image-text pairs. They utilize some
heuristic filters on the text, similar to those from §3, but they also remove
data with less than 5kb of image data, as well as image-text pairs that have
cosine similarity less than 0.28 for english data and 0.26 for other languages
(image and text separately encoded using OpenAI’s ViT-B/32 CLIP model). The
similarity step removed significant quantities of data, reducing the original
corpus from around 50 billion images to 6 billion. While LAION-5B is a corpus
entirely of image-text pairs, Multimodal C4 (Zhu et al., 2023b) introduces a
dataset of interleaved sentences and images, where each data point can contain
multiple sentences and images. Zhu et al. (2023b) use a preprocessed English-
only C4 dataset and gather image files (png/jpeg/jpg) that were found on the
source websites for C4. They perform an image-deduplication step to ensure
that a single image is not overrepresented within a single document or across
the entire corpus. Then, they discard images with a height or width smaller
than 150 pixels, which accounts for many small icons and navigation buttons,
as well as images with aspect ratios greater than 2 or less than 0.5 (removes
many banners at the top/bottom or sides of websites). Finally, to interleave
images into the existing text, they use cosine similarity between images and
individual texts, interleaving only the images which have a similarity score
of at least 0.15. Another unique filtering done for this domain is a step of
removing images that contain faces.

Recently, Bruce et al. (2024) collect a dataset of publicly available internet
videos and train a _foundation world model_ to generate action-controllable
worlds given image inputs. They filter their dataset using a mixture of
heuristics (_e.g._ video titles must contain certain keywords such as
“speedrun” or “playthrough”) and quality filtering. Specifically, they hand
label 10,000 videos and train a classifier on their labeled data.

These methods mainly filter data with heuristic approaches, and have the same
drawbacks as heuristic approaches for language datasets; namely, the
difficulty of validating a method, and slow iteration time as it requires
training multiple models. _Data Filtering Networks_ aim to overcome these
weaknesses by filtering data not with heurstics, but with a trained network
(Fang et al., 2023). While Fang et al. (2023) demonstrate that it is possible
to train a filtering network that can select data points which lead to a
better model, in a cruel twist, they find that data quality is key to training
a good filtering network, leading to a circular problem.

In addition to data collection and filtering, data mixing is also a topic of
concern for non-language pretraining. Piergiovanni et al. (2023) propose an
online data mixing method for vision-language models that mixes a variety of
tasks together (_e.g._ image captioning or visual question answering). Their
method, dynamic difficulty sampling, calculates the loss of each task $t$,
$\mathcal{L}_{t}$, and weights each task proportionally with the total loss
$L$, getting the final sampling weight as $S_{t}=\dfrac{\mathcal{L}_{t}}{L}$.
This mixing method maintains some similarities to the online data mixing
methods from language model pretraining (Xia et al., 2023a; Albalak et al.,
2023a; Schioppa et al., 2023).

##### Data selection for task-specific fine-tuning.

The dominant paradigm in machine learning’s history has been task-specific
fine-tuning, and many methods of data selection have been proposed for this
setting. For the most part, the methods of data selection for non-language
domains are quite similar to the methods for language. Generally speaking,
these methods use the output of a model as the representation upon which the
utility function is built. In an early work on support vector machines, Wang
et al. (2005) devise a method of selecting data that reduces the training set
size while maintaining generalization by filtering out data points which do
not affect the final decision function. This idea resonates in the language
domain, with similarities to methods that use loss (Albalak et al., 2023a) or
methods that attempt to estimate data difficulty (Mindermann et al., 2022). A
more recent direction has been how to best utilize synthetic data, and Yuan et
al. (2024a) demonstrate that generating large quantities of synthetic images
and selecting data using distribution matching (matching the synthetic data to
the target data using maximum mean discrepancy) leads to models that perform
well for image classification with synthetic data only, as well as when mixing
real and synthetic data.

##### Adaptive batch selection.

A number of methods have been developed to improve the data selection on the
level of individual batches, sometimes called data difficulty estimation. One
of the motivations of these methods is that they use the current model’s state
to determine which data to select, but this can add significant computational
overhead. Additionally, because these methods are run alongside the model
training, their cost is not amortized across training runs as it would be for
static datasets. Chang et al. (2017) propose two methods for Active Bias that
emphasize training on data points where the model is uncertain, based on their
prediction variance or closeness to the decision threshold. Similarly, Song et
al. (2020) propose Recency Bias, a method that further emphasizes training on
uncertain data points that were predicted inconsistently in recent iterations
of training. Finally, (Mindermann et al., 2022) present _Reducible Holdout
Loss Selection_ (RHO-Loss), a method of adaptive batch selection that,
intuitively, avoids selecting data points that are redundant, noisy, or less
relevant. RHO-Loss was validated on a very small sample of NLP tasks, task-
specific fine-tuning on SST2 (Socher et al., 2013) and CoLA (Warstadt et al.,
2019), but has not been demonstrated to work in other learning settings.

Adaptive batch selection methods are similar to online data selection, except
that they are often performed on the level of individual data points, rather
than groups, domains, or tasks. This particular direction has found minimal
success with LLMs, in particular for pretraining, but the success of such
methods in other domains suggests that it would be worthwhile adapting these
methods to language.

##### Bias, fairness, and robustness.

Reducing a models bias towards or away from any particular class is a long-
standing challenge in machine learning, related to the desire to create fair
datasets and improve a model’s robustness to distribution shifts and out-of-
distribution data.

There have been a number of recent works that aim to prepare fair datasets.
Biswas & Rajan (2021) analyze stage-specific fairness (SF), measuring fairness
metrics with and without certain data preprocessing stages, and provided
experimental results. González-Zelaya et al. (2023) introduced FairPipes, a
heuristic approach to find an optimal data preparation pipeline configuration.
A number of works have also developed software tools for tracing distribution
distortions linked to fairness in the data preparation workflow (Yang et al.,
2020a; Grafberger et al., 2021; Schelter et al., 2019).

Additionally, there exist fairness-aware balancing techniques (Yan et al.,
2020; Sonoda, 2023; Chakraborty et al., 2021; Salazar et al., 2021; Yang et
al., 2020b) as well as reweighting/resampling methods aimed at ensuring
fairness (Kamiran & Calders, 2012; Jiang & Nachum, 2020). In NLP, using
balanced training for fairness has been explored (Wang et al., 2019; Han et
al., 2022). Jeong et al. (2022b) showed that different proportions of
demographics can lead to nontrivial differences in fairness. To alleviate the
challenge of gathering additional data, recent research has focused on
designing adversarial networks. These networks are developed to acquire a
balanced representation, which helps in minimizing the adverse effects of
sensitive attributes (Madras et al., 2018; Adel et al., 2019).

There are a number of works presented in this survey that discuss filtering
and cleaning data, but another common issue when collecting new data is
missing values. When a data entry contains missing values, should the entry be
dropped, or can the missing value be imputed? Recent and concurrent research
papers that study fairness in missing values (Caton et al., 2022; Schelter et
al., 2019; Fernando et al., 2021; Mansoor et al., 2022; Jeanselme et al.,
2022; Zhang & Long, 2021), which primarily focus on empirically assessing how
dropping or various imputation algorithms affect fairness in downstream task.
Jeong et al. (2022a) theoretically showed how discrimination risks can arise
during data imputation and developed an algorithm called Fair MIP Forest that
integrates missing value treatment and learning into a single process, and
impose a fairness regularizer in an end-to-end process.

Learning representations that are robust to distribution shifts, or out-of-
distribution data, is very challenging, and a very active area of research.
Deng et al. (2023) propose _Progressive Data Expansion_ (PDE), a method for
learning robust representations by progressively expanding the training
dataset. While naively training on all data creates models that are
susceptible to learning non-generalizable spurious features, by progressively
expanding the training dataset, Deng et al. (2023) demonstrate that PDE
improves the model’s robustness for a better worst-group performance. Nguyen
et al. (2022) demonstrate, on six publicly available image-text datasets, that
each individual dataset can lead to robustness to certain distribution shifts,
but no individual dataset dominates the robustness. They explore the
interactions between datasets by combining multiple datasets and find that
this does not lead to an overall better model, rather it dilutes the
robustness from each component dataset. Their results demonstrate that simply
gathering large amounts of data is not optimal.

##### Increasing the rate of data selection research.

Recently, the importance of data selection and quality has gained notoriety,
and thus a few efforts have been developed to try and lower the barrier to
entry for research to be done. One such effort is DataComp (Gadre et al.,
2023), a benchmark for multimodal dataset design. DataComp provides two tracks
for participation: the filtering track, and the bring-your-own-data track. The
filtering track is of particular interest as it gives participants a common
dataset (12.8B image-text pairs from Common Crawl), and the participants goal
is to determine the best subset to train a model on. Additionally, Gadre et
al. (2023) provide over 300 baseline experiments, and find that, similar to
the language-only setting, a smaller, more selective dataset can lead to
models that generalize better than larger datasets. Of course, this still
comes with the same limitations as all previously discussed sections, where
the exact evaluation setting plays a large role in what data is “best”.
Specifically, DataComp evaluates models’ abilities to perform image
classification and retrieval tasks, but does not evaluate generative
capabilities. This suggests that distribution matching methods may fare well
on DataComp, while an evaluation of generative capabilities may prefer data
selection methods that favor distribution diversification. DataPerf (Mazumder
et al., 2023) provides another entry for data selection research. The DataPerf
benchmark contains 4 data selection settings: vision, speech, debugging, and
language. The various settings allow researchers to test their algorithms for
selection, augmentation, quality assesment, and cleaning. Mazumder et al.
(2023) find that automated methods tended to perform better than manual
cleaning or filtering, specifically, they found that submissions succeeded
with automated methods for recognizing noisy images and labels, identifying
mislabeled images, correcting class imbalance, and selecting and enhancing
images from the long-tailed distribution of classes.

By providing a shared resource where researchers can test their methods,
benchmarks and competitions like these can help to increase the pace of
research, as well as giving a direct comparison between methods. Specifically,
DataComp has evaluated 21 submissions181818As of 02/06/2024, found at
https://www.datacomp.ai/leaderboard.html. and DataPerf has evaluated 54
submissions.191919Across vision, speech, debugging, and language tasks, as of
02/06/2024, found at https://www.dataperf.org/ Some of the promising methods
from these benchmarks may be useful for language domains, but their transfer
may be limited due to their domain-specific nature. Benchmarking data-centric
methods, and data selection in particular, for the language domain has not
been fully utilized and should be in the future to accelerate the pace of data
selection research.

##  9 Related Topics

There are a number of data-focused topics in machine learning that are related
to data selection, but have separate aims and use cases.

##### Data cleaning.

As discussed in sections 3.2 and 3.5, some utility functions can determine
both whether an entire data point should be removed from the dataset and also
whether a chunk of text should be removed from a data point. Data cleaning
refers to the latter scenario. While some of the selection methods described
in this work can also be used for cleaning pretraining data, there are
additional data cleaning methods not covered here. For example, in supervised
settings data can be labeled incorrectly and methods for detecting incorrectly
labeled data often assume that such data are outliers, utilizing outlier
detection methods to clean (or remove) the incorrectly labeled data (Abedjan
et al., 2016; Narayan et al., 2022). Data cleaning can also refer to imputing
missing values or finding inconsistencies across data points (see Li et al.
(2021) for more details on data cleaning in non-language domain data).

##### Dataset distillation and coreset selection.

Dataset distillation (also known as _dataset condensation_) and coreset
selection (also known as _data pruning_) have been extensively studied for
reducing the size of training data. Dataset distillation aims to synthesize a
small dataset such that the model trained on that synthetic dataset achieve
performance akin to one trained on the full dataset. Dataset distillation is
first introduced by Wang et al. (2018b), where the model weights are treated
as a function of the distilled dataset (outer optimization) and the distilled
dataset is optimized (inner optimization) so that randomly initialized models
trained on it with one-step gradient descent could achieve good performance.
Such a bi-level meta-learning framework often incurs substantial computational
costs. To improve the scalability and effectiveness of dataset distillation,
subsequent research (Zhao et al., 2020; Lee et al., 2022b; Zhao & Bilen, 2021)
has employed gradient matching as the supervision signal, aligning the
gradients of model parameters trained on the original dataset and the
distilled dataset. Moreover, researchers have proposed techniques such as
feature distribution matching (Wang et al., 2022a; Zhao & Bilen, 2023; Zhao et
al., 2023), training trajectory matching (Cazenavette et al., 2022; Cui et
al., 2023b; Guo et al., 2024), and kernel methods (Nguyen et al., 2020; 2021;
Zhou et al., 2022).

In contrast, the objective of coreset selection (Mirzasoleiman et al., 2020a;
Borsos et al., 2020; Killamsetty et al., 2021b; Guo et al., 2022; Killamsetty
et al., 2021a) is to select a subset from the given dataset so that a model
trained on it achieves optimal performance, which better aligns with the
topics discussed in this paper. Many coreset selection methods design a series
of scores and rank the data instances accordingly. Within this realm,
geometry-based coreset selection methods selects the examples based on their
distances from k-means cluster centers (Chen et al., 2012; Sener & Savarese,
2018; Sorscher et al., 2022; Xia et al., 2023b). Additionally, predictive
uncertainty and model loss metrics can also serve as criteria for ranking data
instances (Coleman et al., 2019; Pleiss et al., 2020; Paul et al., 2021; Zheng
et al., 2022).

There also exist coreset selection methods that are optimization-driven. Craig
(Mirzasoleiman et al., 2020a) optimizes the coreset selection so that the sum
of the gradients on the subsets closely match the gradients on the full
dataset. This approach has been extended to handle noisy data settings in the
subsequent work (Mirzasoleiman et al., 2020b). Glister (Killamsetty et al.,
2021b) and Grad-Match (Killamsetty et al., 2021a) focus on coreset selection
on noisy and imbalanced datasets and propose to match the gradient of the
coreset with the training of a held-out validation set. The AdaCore
(Pooladzandi et al., 2022) method incorporates the second-order gradient
information for better gradient matching. Notably, the techniques of coreset
selection have found application in LLM fine-tuning (Liu et al., 2023b; Xia et
al., 2024). For example, the LESS (Xia et al., 2024) method selects
influential data for supervised fine-tuning based on the similarity of
gradients evaluated on the coreset and the validation set.

##### Data attribution and valuation.

Data valuation and attribution focus on understanding how each training data
point affects a models predictions. Methods for valuation and attribution can
help to identify potentially mislabeled data, identify brittle predictions,
quantify the affect of a training data point on an evaluation data point, and
even quantify train-test contamination. These methods tend to be
computationally expensive, although recent progress has improved their
efficiency.

In recent work, Ghorbani & Zou (2019) propose Data Shapley, a principled
framework for data valuation, where the goal is to quantify the value of each
data point to a predictors performance. The Data Shapley value for each data
point can then be used as the utility function in a data selection method for
either distribution matching or diversification purposes. Ilyas et al. (2022)
present datamodels, a framework used to predict a models performance on a data
point $x^{(i)}$, given that the model is trained on a known subset of training
data. They demonstrate that even using a linear function as the predictor,
their datamodelling framework is capable of accurately predicting model
outputs and counterfactuals, as well as encoding a metric for data point
similarity. Similarly, Park et al. (2023) propose TRAK, a data attribution
method that can make accurate counterfactual predictions, trying to answer the
question “why did my model make this prediction”. They show that TRAK is a
plausible method for determining which training data was most impactful on a
models prediction.

##### Data augmentation.

Data augmentation methods are used to generate new data by modifying the
existing data points in a dataset. Similar to data selection, data
augmentation also aims to shift the training distribution. Data selection is
often used to improve the data coverage in domains where there is limited data
(Li et al., 2022b; Chen et al., 2023a) and can be used in conjunction with
existing data selection methods (_e.g._ domain-specific selection in §3.4 or
“selecting from auxiliary data to improve generalization” in §7). For example,
Albalak et al. (2022a) use a policy gradient along with information from an
auxiliary task to improve their main model. However, the quality of the
coverage produced from augmentation may be limited, as the semantics of
augmented data can diverge from the underlying distribution of language,
leading to an undesirable distribution shift and possibly a decrease in
performance (Li et al., 2022b; Chen et al., 2023a). Thus, care should be taken
when performing data augmentation to ensure that the augmentations do not
stray too far from the desired distribution. Nonetheless, data augmentation
can still be a useful tool, and we refer readers to existing surveys on the
many methods for augmentation in NLP (Chen et al., 2023a; Feng et al., 2021)
and computer vision (Yang et al., 2023).

##### Data curation.

Data curation is a set of processes surrounding the discovery, organization,
integration, annotation, cleaning, storage and maintenance of data (McLure et
al., 2014; Freitas & Curry, 2016; Thirumuruganathan et al., 2020). Some of the
methods that are used for curation overlap with those from data selection
(_e.g._ cleaning and filtering), but there are also methods from data
selection which are not included in data curation (_e.g._ heuristic filters,
data mixing, and filtering toxic and explicit content). Data selection is,
generally speaking, a process that occurs after data curation, once a
practitioner is prepared to train a model. After the data has been curated,
then data selection methods can be used to create a dataset for the desired
purpose.

##### Curriculum learning.

While data selection aims to determine which data points a model should train
or evaluate on, and how many times they may be repeated for training,
curriculum learning aims to improve a model through an orthogonal goal.
Curriculum learning is often motivated by the idea that the training data
should be increasing in complexity according to the models current capability,
and aims to determine _when_ a data point should be trained on.

Curriculum learning has found limited success in the natural language domain.
For example, Graves et al. (2017) use multi-armed bandits to determine an
appropriate curriculum for language modelling with LSTMs. Xu et al. (2020)
demonstrate that curricula can be used to improve learning in natural language
understanding tasks for a BERT model. More recently, Fan & Jaggi (2023)
propose _irreducible curriculum_ for language model pretraining, which
prioritizes training on samples with higher learnability (motivated by RHO-
Loss (Mindermann et al., 2022)). Curriculum learning is one aspect of data-
centric AI that has not yet seen a large adoption in large language models, in
part due to its limited success.

##  10 Discussion

###  10.1 Test set decontamination

Understanding a model’s capabilities requires evaluation on a fair benchmark,
but the training of language models on web-scale data has raised concerns and
speculation about the possible inclusion of evaluation benchmarks in training
data. Given the scale of data it is not feasible to manually check, so
automated methods have been designed to detect the existence of evaluation
data in the training set (Rae et al., 2022; Li et al., 2022a; 2023a; Magnusson
et al., 2023; Elazar et al., 2023), as well as detecting models that have
already been trained on evaluation data (Carlini et al., 2018; Jagielski,
2023; Marone & Durme, 2023). Jacovi et al. (2023) on the other hand, suggest
three strategies for reducing the risks of contamination. Data decontamination
is an important step in ensuring the integrity of model evaluation, and is
included in the discussion section because this critical step is relevant, and
should be required, for all stages of model training.

##### Detecting test set contamination with n-grams.

Test set contamination methods often follow a similar form to deduplication
methods. When decontaminating MassiveText, Rae et al. (2022) compute the
13-gram Jaccard similarity between train and test documents (as originally
proposed by Lee et al. (2022a)) and remove train documents that have a Jaccard
similarity over 0.8, the same exact method that they use to detect duplicated
documents. Similarly, Brown et al. (2020) and Gao et al. (2020) use 13-gram
overlap filtering to detect contamination in their respective training sets.
To make decontamination easier, some tools have been developed. For example,
CarperAI create a github repository with code for decontaminating evaluation
data from a training dataset using MinHashLSH.202020CarperAI decontamination
github repository Additionally, Marone & Durme (2023) propose _Data Portraits_
, a method of investigating the data that a model was trained on. Data
portraits are an artifact that records training data, allows downstream
inspection, and enables answering questions about test set leakage and model
plagiarism through a novel membership testing tool, a strided bloom filter. In
addition to the new method, they release a demo of their
tool212121https://dataportraits.org/ which uses 50-character grams to detect
overlap between training data and an input sequence.

##### Protecting test data with canaries.

A canary string is a unique sequence of characters that is included within a
dataset, or a document, used to empirically evaluate, or audit, whether a
language model was trained on that document (Carlini et al., 2018). By
including canary strings within a benchmark, they can protect the integrity of
an evaluation metric, demonstrating the likelihood of a model having been
trained on that evaluation benchmark. For example, BIG-bench (Srivastava et
al., 2023) includes a canary GUID string in each task to allow for detecting
whether models were trained on the evaluation data.222222BIG-bench canary GUID
README To detect canaries, Carlini et al. (2018) propose the _secret sharer_
framework for efficiently extracting unique, secret sequences such as
personally identifiable information, and demonstrate that such information can
be extracted from a model. Jagielski (2023) expand on the secret sharer
framework, and demonstrate how to interpret canary exposure by relating it to
membership inference attacks and differential privacy.

##### Detecting models trained on test data.

Proving that black-box models have trained on a data point can be very
challenging, but also important to ensure that their results can be trusted.
Because their data is not disclosed, it is impossible to know with certainty
what their training data was, but some methods have been proposed as efforts
to address this issue. Shi et al. (2023a) propose min-k% prob, using the
intuition that unseen examples are more likely to contain outlier words with
low probabilities according to the model, whereas unseen examples are less
likely to exhibit such low probability words. While this is not explicitly a
method of decontamination, it is a useful tool for determining how
contaminated a pretrained models data was. This tool can be used for benchmark
example contamination detection, privacy auditing for machine unlearning, and
copyrighted text detection. Oren et al. (2023) present another method for
detecting contaminated data based on the observation that models have a
tendency to memorize the order of examples as they exist in the dataset, and
provide provable guarantees of test set contamination through a framework of
statistical tests.

##### Decontaminating code data.

Decontamination in code data is particularly important, as Li et al. (2022a)
suggest that competitors in coding competitions tend to publish their
solutions online after a competition. One solution to this problem is to use a
temporal split when dividing data into training and evaluation. To ensure that
test data does not exist in the training dataset, the training dataset can be
constructed only from files that originate prior to the test competition. This
ensures that the training data includes only information that would be
available for human participants of the competition. In a more traditional
method, StarCoder (Li et al., 2023a) decontaminates The Stack by removing
files that contained either exact matching docstrings or solutions from the
HumanEval (Chen et al., 2021) or MBPP (Austin et al., 2021) datasets,
docstrings from APPS (Hendrycks et al., 2021), questions from GSM8K (Cobbe et
al., 2021), or prompts from DS1000 (Lai et al., 2023).

###  10.2 Tradeoffs between memorization and generalization

While most methods of data selection discussed here, data deduplication in
particular, suggest that memorization is a negative model property, this is
not always the case. There are truths about the world which always hold and
memorization is indeed a desirable property in those cases. For example, when
answering the question “what is the formula for Pythagoras’ theorem?” we
expect a model’s answer to deviate very minimally from “$a^{2}+b^{2}=c^{2}$”
and a serious deviation can have real world consequences if such a model was
used in an education setting. However, generalization is still desirable in
this situation as changing the variable names should still be recognized as
correct (_e.g._ “$s_{1}^{2}+s_{2}^{2}=s_{3}^{2}$”). Furthermore, memorization
is also desirable for code data, such as specific API calls in tool usage for
LLMs (Schick et al., 2023; Pan et al., 2023), and syntax for coding languages
(Chen et al., 2021; Li et al., 2022a; Allal et al., 2023). Memorization is
desirable in many other contexts as well, and Brown et al. (2021) have
theoretically proven that in many cases memorization is a requirement of any
reasonably accurate training algorithm.

On the other hand, it has been demonstrated that not only do models memorize
the desirable information, but also the undesirable information such as
personally identifiable information, phone numbers, credit card information
and much more. Carlini et al. (2020) demonstrate this with a simple attack for
extracting verbatim sequences of an LLMs training data, finding that although
data points from the training set do not have noticeably lower loss than test
data points on average, there are some individual examples that are indeed
memorized. Biderman et al. (2023a) study whether it is possible to predict
which sequences from the training data will be memorized by using smaller
models, or partially trained models. Carlini et al. (2023) demonstrate that
memorization grows significantly with three factors: the capacity of a model,
the number of times a sample is repeated in the training data, and the number
of tokens used to prompt the model to emit the memorized data. While the third
factor, the number of tokens in a prompt, is not in the control of model
developers, the first two are. Model developers can choose to use smaller
models to satisfy the needs for a given use case, rather than relying on
massive models which generalize across many use cases, but tend to memorize
data. Furthermore, deduplication has been demonstrated as a strong method for
minimizing memorization. However, as previously discussed, deduplication needs
to be intelligently applied so that “good” memorization can occur (_e.g._
facts), while reducing “bad” memorization (_e.g._ PII).

###  10.3 There’s no free lunch for filtering

Overly filtering the data can lead to undesirable biases in the model. In
particular, Dodge et al. (2021) find that blocklist filtering used to create
C4 (Raffel et al., 2020) disproportionately removes text written by, and
about, minority individuals. Additionally, Welbl et al. (2021) find that
removing toxic content (as determined by the Jigsaw Perspective
API232323https://perspectiveapi.com/ can lead to worse evaluation loss and
disproportionately affects texts about and by marginalized groups. This
finding was also replicated by Longpre et al. (2023c). Additionally, in spite
of the filtering performed in C4, Luccioni & Viviano (2021) still find
significant quantities of sexually explicit content and hate speech. These
works demonstrate the need for filtering strategies with improved precision
(remove less text that is desirable) and recall (remove more text that is
undesirable).

###  10.4 Tools for data selection

When performing data selection, a first step should generally be to understand
the raw dataset, and a number of tools have been developed to assist in data
exploration and auditing. For example, researchers at AI2 developed a search
engine over the C4 dataset,242424https://c4-search.apps.allenai.org/ and
Piktus et al. (2023) create a search engine over C4, the Pile, ROOTS, and text
captions from LAION.252525code at https://github.com/huggingface/gaia,
demonstration at https://huggingface.co/spaces/spacerini/gaia is not
functioning as of 02/07/2024 Elazar et al. (2023) describe their _What’s in my
big data_ tool, which can search for and count n-grams in a number of popular
pretraining corpora. Additionally, their online demo contains tools for
exploring the internet domains included in datasets, a visualization of the
amount of text overlap between corpora, and a visualization to understand the
length of documents in each corpus.262626https://wimbd.apps.allenai.org/
Finally, Liu et al. (2024a) developed the Infini-gram which uses suffix arrays
as a method for language modeling, and provide a demonstration of the Infini-
gram trained on the RedPajama, the Pile, and Dolma corpora and is capable of
very quickly searching for n-grams within any of those corpora.
272727https://huggingface.co/spaces/liujch1998/infini-gram

Once the data has been explored and audited, the next step of data selection
is to devise the appropriate selection mechanisms and apply them over the
data. Some of the methods discussed in this work have been implemented as
open-sourced tools, allowing anyone to easily make use of them. The CCNet
pipeline (Wenzek et al., 2020)
282828https://github.com/facebookresearch/cc_net is a commonly used tool for
downloading and cleaning Common Crawl data. The CCNet pipeline performs
deduplication at the line level, performs language identification with a
fastText linear classifier, and can train an n-gram language model to
calculate perplexity per document. Soldaini et al. (2024) open-source tools
for identifying non-language text, language identification, personally
identifiable information, toxic text, text quality, and duplicated
data.292929https://github.com/allenai/dolma Xie et al. (2023b) provide a pip-
installable tool for selecting data with a distribution similar to a target
dataset, that can be used either for domain-specific selection, or for quality
filtering.303030https://github.com/p-lambda/dsir Lee et al. (2022a) provide
the code to run their ExactSubstr
deduplication,313131https://github.com/google-research/deduplicate-text-
datasets and Computer (2023) provide tools for computing exact duplicates with
a bloom filter, and fuzzy deduplication with locality sensitive
hashing.323232https://github.com/togethercomputer/RedPajama-Data Additionally,
the tools provided by Computer (2023) contain code for calculating a wide
variety of data signals including heuristics, quality, duplication, and
toxicity. Finally, CarperAI provide a pip-installable package for
decontaminating datasets from a number of openly available
benchmarks.333333https://github.com/CarperAI/decontamination/tree/main

###  10.5 Considerations when applying data selection to your setting

The first step in determining appropriate actions for data selection is to
determine whether you are in a setting where you wish to increase the
distribution coverage from your data, or if you wish to select a subset from
your data. For pretraining, the goal will usually be to increase coverage,
unless the model has a well defined downstream use case (_e.g._ for medical
domain), in which case the goal can be to select a subset of general data
which is most similar to the downstream use case. For multitask training and
instruction tuning, the goal is generally to increase coverage because the
goal of these settings is to produce a capable general purpose model.
Additionally, depending on the number of parameters in the model you plan to
train, you may have a range for the desired number of training tokens
according to your compute budget (Kaplan et al., 2020; Hoffmann et al., 2022;
Penedo et al., 2023). The various filter sensitivities will need to be
adjusted depending on the number of tokens in the raw data in order to achieve
the desired number of training tokens. For task-specific fine-tuning, the
setting depends on how large the task-specific dataset is. Generally, if the
dataset is relatively small, the goal of data selection will be to increase
coverage over the expected test distribution. However, if the task-specific
dataset is very large then models can plausibly model the distribution with
fewer data points by removing redundant information from the dataset, and
costs can be reduced.

The next step in the decision making is with regards to how much information
you have on the test distribution as this will determine which utility
functions available to you. Depending on whether you have data sampled from
the test distribution, you may be able to find an appropriate model-based
utility function, otherwise you will need to determine appropriate heuristics.
Furthermore, the available utility functions may be limited by the amount of
compute available.

Cost is another consideration when determining how to apply data filtering.
Cost can be calculated in money, time, or effort. The ordering of filters used
can have a great impact on the final cost. An appropriate choice of filtering
order is from least expensive to most expensive, so that the more expensive
selection methods will be executed on fewer data points. Additionally, Coleman
et al. (2020) demonstrated that data selection methods requiring a model can
be improved in efficiency by using proxy models much smaller than the actual
model being trained, to further reduce costs.

##  11 Future Directions: Challenges and Opportunities

###  11.1 Accelerating research on data selection

Research on data can be a very slow and tedious process, potentially requiring
significant investments of time, compute, and human effort. Here we describe
three directions of future research that can reduce the effort required to do
research in data selection.

##### Metrics that directly evaluate data.

One very high impact, but challenging, way to accelerate research on data
selection is to develop metrics that directly evaluate the data chosen by a
selection method, thereby removing (or reducing) the need for expensive model
training. Recalling the definition of data selection in §2.2, the goal of a
data selection method is inherently tied to it’s ability to minimize or
maximize the objective function of a trained model. The dependence of data
selection on model training significantly increases the iteration time for
experiments, reducing the amount of experimentation that researchers can
perform.343434For reference, training a 1 billion parameter model on 50
billion tokens takes roughly 5 days to train on 8 A100 GPUs using GPT-NeoX
(Andonian et al., 2023; Albalak et al., 2023a).

Preliminary works have been done in this direction, including some works on
data attribution and valuation (Ghorbani & Zou, 2019; Ilyas et al., 2022), and
data measurements (Mitchell et al., 2023). Works on data valuation and
attribution (discussed in §9) attempt to better understand how each data point
affects a models predictions, but may be used as inspiration for methods that
directly evaluate the data itself. Additionally, there are a number of metrics
that directly measure the intrinsic characteristics of data including
distance, density, diversity, tendencies (mean, median, mode), and association
(correlation, mutual information), but these measures have yet to be connected
with a model’s performance on downstream data (Mitchell et al., 2023).

Developing metrics that directly evaluate data can significantly increase the
time spent on method development and increase the amount of exploratory work
possible. Furthermore, removing the need to train models (which are very
costly) can allow for lower-resourced organizations and individuals to
contribute to the field, improving diversity and inclusion in the process of
large model development.

##### Data-centric benchmarks and challenges.

Another high-impact direction for increasing the rate of research on data
selection is the development of data-centric benchmarks and challenges. One
hurdle of advancing data selection is that works are done across a wide
variety of settings and cannot be directly compared. Specifically, much of
data selection research using heuristics filters is performed as part of
collecting a new dataset. This makes it impossible to compare selection
methods with another dataset as they may contain different quantities of data,
and sometimes even use varying model architectures and sizes. In other
domains, such as vision and vision+language, challenges and benchmarks have
been developed that provide well-founded evaluation settings that allows
researchers from many institutions to participate in a combined effort to push
the frontier of knowledge and capabilities in data selection (discussed in
"Increasing the rate of data selection research" in §8). Creating similar
challenges for language models can go a long way in encouraging progress.

One challenge for developing benchmarks is there is no current consensus for
what the appropriate toy settings are for data selection research. With the
exception of the _Loose_ track from BabyLM (Warstadt et al., 2023), no prior
benchmarks or challenges have been created for data selection (BabyLM is also
not specifically for data selection, but rather to improve the speed of
development on architectures and hyperparameters for language models). The
DataComp (Gadre et al., 2023) and DataPerf (Mazumder et al., 2023) challenges
can provide inspiration for creating similar challenged and future benchmarks
in language.

Due to the large number of training settings (pretraining, instruction-tuning,
alignment, in-context, task-specific) and goals (general-purpose, knowledge-
intensive, chat, etc.) for language models, there are a wide variety of
benchmarks and challenges that would be beneficial for the community. Creating
challenges and benchmarks allows for direct comparison between methods, lowers
the bar for entry into the field, and encourages open progress on this
important task.

##### Open-sourcing tools and best practices.

As the field develops in the open, new rules and best practices will emerge.
Open sourcing the tools that implement these best practices is crucial. The
combination of new best practices being developed in the open, and the open-
sourcing of such tools can significantly reduce the overhead required to get
started into research on data. This also allows researchers to focus on
specific components of the pipeline, further accelerating research progress.

###  11.2 Better understanding the properties of the target distribution

Having a clear goal is crucial to defining and developing successful data
selection methods. Using the terminology and conceptual framework developed in
this work can improve the understanding of what data selection methods may be
appropriate depending on the properties of the desired target distribution.
For example, the goal of data selection is very clear for task-specific fine-
tuning, but less clear for preference fine-tuning (PreFT). Because the
definition of success is less specific for PreFT, developers will likely
benefit from data diversification methods. For example, it may be possible to
utilize existing auxiliary data for preferences by drawing inspiration from
methods in §7 (“Selecting from auxiliary data to improve generalization”), or
improving robustness across preference groups by utilizing methods for
robustness (§7 “Selecting data for robustness” or §8 “Bias, fairness, and
robustness”).

###  11.3 Shifting compute time from model training to data processing

It is now well known that training data plays a very significant importance in
the performance of a model, therefore creating an increased desirability to
spend more compute on data processing and selection (and possibly away from
model training). To improve beyond what is possible only through heuristics,
it may be necessary to utilize more expensive methods.

One example of how this may be accomplished is through the use of methods
which have previously only been used in low-data settings (_e.g._ alignment)
and applying them to settings with greater amounts of data (_e.g._
pretraining). For example, methods inspired by model-based filtering, or
reward model-reweighting techniques can be used to improve the “quality” of
pretraining data. As compute gets cheaper, it becomes more feasible to spend
the cost of directly applying the existing methods. However, another avenue
worth exploring is using cheap approximations for utility functions, such as
computing similarity using bag-of-n-grams and hash functions rather than from
model representations.

Another method for improving data that would require large amounts of compute
is additive data selection. Currently, most data selection methods (especially
for pretraining) remove undesirable content from data points. However, it is
possible that in the future it will be beneficial to have additive data
selection methods. For example, it would be possible to use a language model
to infill information within a data point. One example would be to include
explanations for some phrases: “ _e.g._ the basketball player wasn’t able to
move _due to the double dribble rule_ so he passed the ball to his teammate”
where the _italicized_ text has been added by an infilling model.

##  12 Conclusion

In this work, we have compiled an in-depth review on the current state of data
selection and present a unified framework with which to consider and compare
the wide variety of methods. This survey has described the current best
practices and considerations when selecting data for training a language
model. Additionally, the unified framework has allowed us to define and
describe some potentially fruitful future directions of research which we have
highlighted throughout the paper. However, there is much more room for
innovation and improvement in data selection.

#### Acknowledgments

This material is based on work that is partially funded by an unrestricted
gift from Google. This work was supported by the National Science Foundation
award #2048122. The views expressed are those of the authors and do not
reflect the official policy or position of the US government. The work of BH
and SC is partially supported by the UC Santa Barbara IEE IGSB SW Impact
Grant.

## References

  * Abbas et al. (2023) Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication.  _arXiv preprint arXiv:2303.09540_ , 2023.
  * Abedjan et al. (2016) Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang.  Detecting data errors: Where are we and what needs to be done?  _Proc. VLDB Endow._ , 9(12):993–1004, aug 2016.  ISSN 2150-8097.  doi: 10.14778/2994509.2994518.  URL https://doi.org/10.14778/2994509.2994518.
  * Adel et al. (2019) Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller.  One-network adversarial fairness.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 2412–2420, 2019.
  * Agarwal et al. (2009) Amit Agarwal, Hema Swetha Koppula, Krishna P. Leela, Krishna Prasad Chitrapura, Sachin Garg, Pavan Kumar GM, Chittaranjan Haty, Anirban Roy, and Amit Sasturkar.  Url normalization for de-duplication of web pages.  In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_ , CIKM ’09, pp. 1987–1990, New York, NY, USA, 2009. Association for Computing Machinery.  ISBN 9781605585123.  doi: 10.1145/1645953.1646283.  URL https://doi.org/10.1145/1645953.1646283.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta.  Muppet: Massive multi-task representations with pre-finetuning.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5799–5811, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.468.  URL https://aclanthology.org/2021.emnlp-main.468.
  * Aghajanyan et al. (2022) Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer.  HTLM: Hyper-text pre-training and prompting of language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=P-pPW1nxf1r.
  * Albalak et al. (2022a) Alon Albalak, Varun Embar, Yi-Lin Tuan, Lise Getoor, and William Yang Wang.  D-REX: Dialogue relation extraction with explanations.  In Bing Liu, Alexandros Papangelis, Stefan Ultes, Abhinav Rastogi, Yun-Nung Chen, Georgios Spithourakis, Elnaz Nouri, and Weiyan Shi (eds.), _Proceedings of the 4th Workshop on NLP for Conversational AI_ , pp. 34–46, Dublin, Ireland, May 2022a. Association for Computational Linguistics.  doi: 10.18653/v1/2022.nlp4convai-1.4.  URL https://aclanthology.org/2022.nlp4convai-1.4.
  * Albalak et al. (2022b) Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang.  FETA: A benchmark for few-sample task transfer in open-domain dialogue.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 10936–10953, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.751.  URL https://aclanthology.org/2022.emnlp-main.751.
  * Albalak et al. (2023a) Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang.  Efficient online data mixing for language model pre-training, 2023a.
  * Albalak et al. (2023b) Alon Albalak, Colin Raffel, and William Yang Wang.  Improving few-shot generalization by exploring and exploiting auxiliary data.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=JDnLXc4NOn.
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ , 2023.
  * Andonian et al. (2023) Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach.  GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 9 2023.  URL https://www.github.com/eleutherai/gpt-neox.
  * Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al.  Ext5: Towards extreme multi-task scaling for transfer learning.  _arXiv preprint arXiv:2111.10952_ , 2021.
  * Auer et al. (2002) Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire.  The nonstochastic multiarmed bandit problem.  _SIAM Journal on Computing_ , 32(1):48–77, 2002.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.  Program synthesis with large language models, 2021.
  * Axelrod (2017) Amittai Axelrod.  Cynical selection of language model training data, 2017.
  * Azeemi et al. (2023) Abdul Azeemi, Ihsan Qazi, and Agha Raza.  Data pruning for efficient model pruning in neural machine translation.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 236–246, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.18.  URL https://aclanthology.org/2023.findings-emnlp.18.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Bartolome et al. (2023) Alvaro Bartolome, Gabriel Martin, and Daniel Vila.  Notus.  https://github.com/argilla-io/notus, 2023.
  * Beirami et al. (2024) Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh.  Theoretical guarantees on the best-of-n alignment policy.  _arXiv preprint arXiv:2401.01879_ , 2024.
  * Bejan et al. (2023) Irina Bejan, Artem Sokolov, and Katja Filippova.  Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 10107–10121, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.625.  URL https://aclanthology.org/2023.emnlp-main.625.
  * Biderman et al. (2023a) Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Gregory Anthony, Shivanshu Purohit, and Edward Raff.  Emergent and predictable memorization in large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.  URL https://openreview.net/forum?id=Iq0DvhB4Kf.
  * Biderman et al. (2023b) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal.  Pythia: A suite for analyzing large language models across training and scaling.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 2397–2430. PMLR, 23–29 Jul 2023b.  URL https://proceedings.mlr.press/v202/biderman23a.html.
  * Biswas & Rajan (2021) Sumon Biswas and Hridesh Rajan.  Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline.  In _Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering_ , pp. 981–993, 2021.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor.  Demographic dialectal variation in social media: A case study of African-American English.  In Jian Su, Kevin Duh, and Xavier Carreras (eds.), _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 1119–1130, Austin, Texas, November 2016. Association for Computational Linguistics.  doi: 10.18653/v1/D16-1120.  URL https://aclanthology.org/D16-1120.
  * Bloom (1970) Burton H. Bloom.  Space/time trade-offs in hash coding with allowable errors.  _Commun. ACM_ , 13(7):422–426, jul 1970.  ISSN 0001-0782.  doi: 10.1145/362686.362692.  URL https://doi.org/10.1145/362686.362692.
  * Bommasani et al. (2023) Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang.  The foundation model transparency index.  _arXiv preprint arXiv:2310.12941_ , 2023.
  * Borsos et al. (2020) Zalán Borsos, Mojmir Mutny, and Andreas Krause.  Coresets via bilevel optimization for continual learning and streaming.  _Advances in neural information processing systems_ , 33:14879–14890, 2020.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In Lluís Màrquez, Chris Callison-Burch, and Jian Su (eds.), _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 632–642, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1075.  URL https://aclanthology.org/D15-1075.
  * Broder (1997) A.Z. Broder.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171)_ , pp. 21–29, 1997.  doi: 10.1109/SEQUEN.1997.666900.
  * Brown et al. (2021) Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar.  When is memorization of irrelevant training data necessary for high-accuracy learning?  In _Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing_ , STOC 2021, pp. 123–132, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380539.  doi: 10.1145/3406325.3451131.  URL https://doi.org/10.1145/3406325.3451131.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Bruce et al. (2024) Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel.  Genie: Generative interactive environments, 2024.
  * Cao et al. (2023) Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun.  Instruction mining: When data mining meets large language model finetuning, 2023.
  * Carlini et al. (2018) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Xiaodong Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _USENIX Security Symposium_ , 2018.  URL https://api.semanticscholar.org/CorpusID:170076423.
  * Carlini et al. (2020) Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel.  Extracting training data from large language models.  In _USENIX Security Symposium_ , 2020.  URL https://api.semanticscholar.org/CorpusID:229156229.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=TatRHT_1cK.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=bx24KpJ4Eb.  Survey Certification.
  * Castricato et al. (2024) Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, and Stella Biderman.  Suppressing pink elephants with direct principle feedback, 2024.
  * Caswell et al. (2020) Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna.  Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus.  _arXiv preprint arXiv:2010.14571_ , 2020.
  * Caton et al. (2022) Simon Caton, Saiteja Malisetty, and Christian Haas.  Impact of imputation strategies on fairness in machine learning.  _Journal of Artificial Intelligence Research_ , 74:1011–1035, 2022.
  * Cazenavette et al. (2022) George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu.  Dataset distillation by matching training trajectories.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4750–4759, 2022.
  * Chakraborty et al. (2021) Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies.  Bias in machine learning software: why? how? what to do?  In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2021, pp. 429–440, New York, NY, USA, August 2021. Association for Computing Machinery.  ISBN 978-1-4503-8562-6.  doi: 10.1145/3468264.3468537.  URL https://dl.acm.org/doi/10.1145/3468264.3468537.
  * Chang et al. (2017) Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum.  Active bias: Training more accurate neural networks by emphasizing high variance samples.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 1002–1012, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/2f37d10131f2a483a8dd005b3d14b0d9-Abstract.html.
  * Chang & Jia (2022) Ting-Yun Chang and Robin Jia.  Careful data curation stabilizes in-context learning.  _arXiv preprint arXiv:2212.10378_ , 2022.
  * Charikar (2002) Moses S. Charikar.  Similarity estimation techniques from rounding algorithms.  In _Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing_ , STOC ’02, pp. 380–388, New York, NY, USA, 2002. Association for Computing Machinery.  ISBN 1581134959.  doi: 10.1145/509907.509965.  URL https://doi.org/10.1145/509907.509965.
  * Chayka (2023) Kyle Chayka.  Is A.I. Art Stealing from Artists?  _The New Yorker_ , February 2023.  ISSN 0028-792X.  URL https://www.newyorker.com/culture/infinite-scroll/is-ai-art-stealing-from-artists.  Section: infinite scroll.
  * Chen et al. (2023a) Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang.  An Empirical Survey of Data Augmentation for Limited Data Learning in NLP.  _Transactions of the Association for Computational Linguistics_ , 11:191–211, 03 2023a.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00542.  URL https://doi.org/10.1162/tacl_a_00542.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * Chen et al. (2023b) Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré.  Skill-it! a data-driven skills framework for understanding and training language models, 2023b.
  * Chen et al. (2012) Yutian Chen, Max Welling, and Alex Smola.  Super-samples from kernel herding.  _arXiv preprint arXiv:1203.3472_ , 2012.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  _arXiv preprint arXiv:2401.01335_ , 2024.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems, 2021.
  * Coleman et al. (2019) Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia.  Selection via proxy: Efficient data selection for deep learning.  In _International Conference on Learning Representations_ , 2019.
  * Coleman et al. (2020) Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia.  Selection via proxy: Efficient data selection for deep learning.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=HJg2b0VYDr.
  * Computer (2023) Together Computer.  Redpajama: an open dataset for training large language models, October 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Conneau & Lample (2019) Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Cui et al. (2023a) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2023a.
  * Cui et al. (2023b) Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh.  Scaling up dataset distillation to imagenet-1k with constant memory.  In _International Conference on Machine Learning_ , pp. 6565–6590. PMLR, 2023b.
  * David (2023) Emilia David.  Ai image training dataset found to include child sexual abuse imagery.  _The Verge_ , December 2023.  URL https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford.  7:57 AM PST.
  * Deng et al. (2023) Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu.  Robust learning with progressive data expansion against spurious correlation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=9QEVJ9qm46.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _NAACL_ , 2019.  URL https://aclanthology.org/N19-1423.
  * Dhole et al. (2021) Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al.  Nl-augmenter: A framework for task-sensitive natural language augmentation.  _arXiv preprint arXiv:2112.02721_ , 2021.
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 1286–1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.98.  URL https://aclanthology.org/2021.emnlp-main.98.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  GLaM: Efficient scaling of language models with mixture-of-experts.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 5547–5569. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/du22c.html.
  * Elazar et al. (2023) Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge.  What’s in my big data?  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Endrédy & Novák (2013) István Endrédy and Attila Novák.  More Effective Boilerplate Removal - the GoldMiner Algorithm.  _Polibits - Research journal on Computer science and computer engineering with applications_ , 1(48):79–83, 2013.  ISSN 1870-9044.  URL http://polibits.gelbukh.com/2013_48.
  * Engstrom et al. (2024) Logan Engstrom, Axel Feldmann, and Aleksander Madry.  Dsdm: Model-aware dataset selection with datamodels, 2024.
  * Ethayarajh et al. (2022) Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.  Understanding dataset difficulty with $\mathcal{V}$-usable information.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 5988–6008. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/ethayarajh22a.html.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization.  _arXiv preprint arXiv:2402.01306_ , 2024.
  * Fan & Jaggi (2023) Simin Fan and Martin Jaggi.  Irreducible curriculum for language model pretraining, 2023.
  * Fan et al. (2023) Simin Fan, Matteo Pagliardini, and Martin Jaggi.  Doge: Domain reweighting with generalization estimation.  _arXiv preprint_ , 2023.
  * Fang et al. (2023) Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar.  Data filtering networks, 2023.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.  A survey of data augmentation approaches for NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pp. 968–988, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.84.  URL https://aclanthology.org/2021.findings-acl.84.
  * Feng et al. (2022) Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc.  Automatic document selection for efficient encoder pretraining.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 9522–9530, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.647.  URL https://aclanthology.org/2022.emnlp-main.647.
  * Fernando et al. (2021) Martínez-Plumed Fernando, Ferri Cèsar, Nieves David, and Hernández-Orallo José.  Missing the missing values: The ugly duckling of fairness in machine learning.  _International Journal of Intelligent Systems_ , 36(7):3217–3258, 2021.
  * Freitas & Curry (2016) André Freitas and Edward Curry.  Big data curation.  _New horizons for a data-driven economy: A roadmap for usage and exploitation of big data in Europe_ , pp. 87–118, 2016.
  * Gadre et al. (2023) Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt.  Datacomp: In search of the next generation of multimodal datasets.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=dVaWCDMBof.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The pile: An 800gb dataset of diverse text for language modeling, 2020.
  * Gao et al. (2023a) Leo Gao, John Schulman, and Jacob Hilton.  Scaling laws for reward model overoptimization.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023a.  URL https://dl.acm.org/doi/10.5555/3618408.3618845.
  * Gao et al. (2023b) Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Bendersky.  Ambiguity-aware in-context learning with large language models.  _arXiv preprint arXiv:2309.07900_ , 2023b.
  * Ghorbani & Zou (2019) Amirata Ghorbani and James Zou.  Data shapley: Equitable valuation of data for machine learning.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 2242–2251. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/ghorbani19c.html.
  * González-Zelaya et al. (2023) Vladimiro González-Zelaya, Julián Salas, Dennis Prangle, and Paolo Missier.  Preprocessing matters: Automated pipeline selection for fair classification.  In _International Conference on Modeling Decisions for Artificial Intelligence_ , pp. 202–213. Springer, 2023.
  * Grafberger et al. (2021) Stefan Grafberger, Julia Stoyanovich, and Sebastian Schelter.  Lightweight inspection of data preprocessing in native machine learning pipelines.  In _Conference on Innovative Data Systems Research (CIDR)_ , 2021.
  * Grangier & Iter (2022) David Grangier and Dan Iter.  The trade-offs of domain adaptation for neural language models.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3802–3813, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.264.  URL https://aclanthology.org/2022.acl-long.264.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov.  Learning word vectors for 157 languages.  In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).  URL https://aclanthology.org/L18-1550.
  * Graves et al. (2017) Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu.  Automated curriculum learning for neural networks.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 1311–1320. PMLR, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/graves17a.html.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al.  Olmo: Accelerating the science of language models.  _arXiv preprint arXiv:2402.00838_ , 2024.
  * Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al.  Textbooks are all you need.  _arXiv preprint arXiv:2306.11644_ , 2023.
  * Guo et al. (2022) Chengcheng Guo, Bo Zhao, and Yanbing Bai.  Deepcore: A comprehensive library for coreset selection in deep learning.  In _International Conference on Database and Expert Systems Applications_ , pp. 181–195. Springer, 2022.
  * Guo et al. (2024) Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You.  Towards lossless dataset distillation via difficulty-aligned trajectory matching.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Gupta et al. (2023) Shivanshu Gupta, Sameer Singh, and Matt Gardner.  Coverage-based example selection for in-context learning.  _arXiv preprint arXiv:2305.14907_ , 2023.
  * Gururangan et al. (2022) Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith.  Whose language counts as high quality? measuring language ideologies in text data selection.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 2562–2580, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.165.  URL https://aclanthology.org/2022.emnlp-main.165.
  * Haghtalab et al. (2022) Nika Haghtalab, Michael Jordan, and Eric Zhao.  On-demand sampling: Learning optimally from multiple distributions.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 406–419. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/02917acec264a52a729b99d9bc857909-Paper-Conference.pdf.
  * Han et al. (2022) Xudong Han, Timothy Baldwin, and Trevor Cohn.  Balancing out bias: Achieving fairness through balanced training.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 11335–11350, 2022.
  * Hashimoto et al. (2023) Kazuma Hashimoto, Karthik Raman, and Michael Bendersky.  Take one step at a time to know incremental utility of demonstration: An analysis on reranking for few-shot in-context learning.  _arXiv preprint arXiv:2311.09619_ , 2023.
  * Heafield (2011) Kenneth Heafield.  KenLM: Faster and smaller language model queries.  In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pp. 187–197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics.  URL https://aclanthology.org/W11-2123.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps.  In J. Vanschoren and S. Yeung (eds.), _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran, 2021.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/c24cd76e1ce41366a4bbe8a49b02a028-Paper-round2.pdf.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.  Unnatural instructions: Tuning language models with (almost) no human labor.  _arXiv preprint arXiv:2212.09689_ , 2022.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Ilyas et al. (2022) Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Understanding predictions with data and data with predictions.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 9525–9587. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/ilyas22a.html.
  * Iter & Grangier (2021) Dan Iter and David Grangier.  On the complementarity of data selection and fine tuning for domain adaptation, 2021.
  * Ivison et al. (2023a) Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi.  Data-efficient finetuning using cross-task nearest neighbors.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 9036–9061, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.576.  URL https://aclanthology.org/2023.findings-acl.576.
  * Ivison et al. (2023b) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023b.
  * Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_ , 2022.  URL https://arxiv.org/abs/2212.12017.
  * Jacovi et al. (2023) Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg.  Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 5075–5084, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.308.  URL https://aclanthology.org/2023.emnlp-main.308.
  * Jagielski (2023) Matthew Jagielski.  A note on interpreting canary exposure, 2023.
  * Jansen et al. (2022) Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez.  Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data, 2022.
  * Jeanselme et al. (2022) Vincent Jeanselme, Maria De-Arteaga, Zhe Zhang, Jessica Barrett, and Brian Tom.  Imputation strategies under clinical presence: Impact on algorithmic fairness.  In _Machine Learning for Health_ , pp. 12–34. PMLR, 2022.
  * Jeong et al. (2022a) Haewon Jeong, Hao Wang, and Flavio P Calmon.  Fairness without imputation: A decision tree approach for fair prediction with missing values.  In _AAAI Conference on Artificial Intelligence_ , volume 36, pp. 9558–9566, 2022a.
  * Jeong et al. (2022b) Haewon Jeong, Michael D Wu, Nilanjana Dasgupta, Muriel Médard, and Flavio Calmon.  Who gets the benefit of the doubt? racial bias in machine learning algorithms applied to secondary school math education.  In _Math AI for Education: Bridging the Gap Between Research and Smart Education_. AIED, 2022b.
  * Jiang & Nachum (2020) Heinrich Jiang and Ofir Nachum.  Identifying and correcting label bias in machine learning.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 702–712. PMLR, 2020.
  * John & Draper (1975) R. C. St. John and N. R. Draper.  D-optimality for regression designs: A review.  _Technometrics_ , 17(1):15–23, 1975.  doi: 10.1080/00401706.1975.10489266.  URL https://www.tandfonline.com/doi/abs/10.1080/00401706.1975.10489266.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov.  Fasttext.zip: Compressing text classification models, 2016.
  * Kamiran & Calders (2012) Faisal Kamiran and Toon Calders.  Data preprocessing techniques for classification without discrimination.  _Knowledge and information systems_ , 33(1):1–33, 2012.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.  URL https://arxiv.org/abs/2001.08361.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher.  Unifying question answering, text classification, and regression via span extraction.  _arXiv preprint arXiv:1904.09286_ , 2019.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.  UnifiedQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020.  URL https://aclanthology.org/2020.findings-emnlp.171.
  * Killamsetty et al. (2021a) Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer.  Grad-match: Gradient matching based data subset selection for efficient deep model training.  In _International Conference on Machine Learning_ , pp. 5464–5474. PMLR, 2021a.
  * Killamsetty et al. (2021b) Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer.  Glister: Generalization based data subset selection for efficient and robust learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 8110–8118, 2021b.
  * Kim et al. (2023) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al.  Prometheus: Inducing fine-grained evaluation capability in language models.  _arXiv preprint arXiv:2310.08491_ , 2023.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_ , 2023.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, 2022.  doi: 10.1162/tacl_a_00447.  URL https://aclanthology.org/2022.tacl-1.4.
  * Kudugunta et al. (2023) Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al.  Madlad-400: A multilingual and document-level large audited dataset.  _arXiv preprint arXiv:2309.04662_ , 2023.
  * Kung et al. (2023) Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng.  Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks, 2023.
  * Lai et al. (2023) Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu.  Ds-1000: a natural and reliable benchmark for data science code generation.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023.
  * Lambert et al. (2023a) Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick.  The history and risks of reinforcement learning and human feedback.  _arXiv preprint arXiv:2310.13595_ , 2023a.
  * Lambert et al. (2023b) Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush.  Huggingface h4 stack exchange preference dataset, 2023b.  URL https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences.
  * Lang et al. (2022) Hunter Lang, Aravindan Vijayaraghavan, and David Sontag.  Training subset selection for weak supervision.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 16023–16036. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/66720ca4e5a09ff83b55a117a6b2a86c-Paper-Conference.pdf.
  * Laurençon et al. (2022) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite.  The bigscience roots corpus: A 1.6tb composite multilingual dataset.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 31809–31826. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ce9e92e3de2372a4b93353eb7f3dc0bd-Paper-Datasets_and_Benchmarks.pdf.
  * Lee et al. (2023) Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen White, and Sujay Jauhar.  Making large language models better data creators.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 15349–15360, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.948.  URL https://aclanthology.org/2023.emnlp-main.948.
  * Lee et al. (2022a) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8424–8445, Dublin, Ireland, May 2022a. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.577.  URL https://aclanthology.org/2022.acl-long.577.
  * Lee et al. (2022b) Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon.  Dataset condensation with contrastive signals.  In _International Conference on Machine Learning_ , pp. 12352–12364. PMLR, 2022b.
  * Levy et al. (2023) Itay Levy, Ben Bogin, and Jonathan Berant.  Diverse demonstrations improve in-context compositional generalization.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1401–1422, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.78.  URL https://aclanthology.org/2023.acl-long.78.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.  Solving quantitative reasoning problems with language models.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=IFXTZERXdM7.
  * Li et al. (2021) Peng Li, Xi Rao, Jennifer Blase, Yue Zhang, Xu Chu, and Ce Zhang.  Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pp. 13–24, 2021.  doi: 10.1109/ICDE51399.2021.00009.
  * Li et al. (2023a) Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries.  Starcoder: may the source be with you!  _Transactions on Machine Learning Research_ , 2023a.  ISSN 2835-8856.  URL https://openreview.net/forum?id=KoFOg41haE.  Reproducibility Certification.
  * Li & Qiu (2023) Xiaonan Li and Xipeng Qiu.  Finding supporting examples for in-context learning.  _arXiv preprint arXiv:2302.13539_ , 2023.
  * Li et al. (2023b) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu.  Unified demonstration retriever for in-context learning.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 4644–4668, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.256.  URL https://aclanthology.org/2023.acl-long.256.
  * Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, December 2022a.  ISSN 1095-9203.  doi: 10.1126/science.abq1158.  URL http://dx.doi.org/10.1126/science.abq1158.
  * Li et al. (2022b) Zekun Li, Hong Wang, Alon Albalak, Yingrui Yang, Jing Qian, Shiyang Li, and Xifeng Yan.  Making something out of nothing: Building robust task-oriented dialogue systems from scratch.  In _Alexa Prize TaskBot Challenge 1 Proceedings_ , 2022b.  URL https://www.amazon.science/alexa-prize/proceedings/making-something-out-of-nothing-building-robust-task-oriented-dialogue-systems-from-scratch.
  * Lian et al. (2023) Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium".  Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023.  URL https://https://huggingface.co/Open-Orca/SlimOrca.
  * Liu et al. (2023a) Hao Liu, Matei Zaharia, and Pieter Abbeel.  Exploration with principles for diverse ai supervision.  _arXiv preprint arXiv:2310.08899_ , 2023a.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pp. 100–114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.deelio-1.10.  URL https://aclanthology.org/2022.deelio-1.10.
  * Liu et al. (2024a) Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi.  Infini-gram: Scaling unbounded n-gram language models to a trillion tokens, 2024a.
  * Liu et al. (2024b) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  In _The Twelfth International Conference on Learning Representations_ , 2024b.  URL https://openreview.net/forum?id=xbjSwwrQOe.
  * Liu et al. (2023b) Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.  What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.  _arXiv preprint arXiv:2312.15685_ , 2023b.
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Multi-task deep neural networks for natural language understanding.  In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4487–4496, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1441.  URL https://aclanthology.org/P19-1441.
  * Longpre et al. (2023a) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts.  The flan collection: Designing data and methods for effective instruction tuning, 2023a.
  * Longpre et al. (2023b) Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al.  The data provenance initiative: A large scale audit of dataset licensing & attribution in ai.  _arXiv preprint arXiv:2310.16787_ , 2023b.
  * Longpre et al. (2023c) Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito.  A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity, 2023c.
  * Lu et al. (2023a) Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu.  Self: Language-driven self-evolution for large language model.  _arXiv preprint arXiv:2310.00533_ , 2023a.
  * Lu et al. (2023b) Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou.  #instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023b.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8086–8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.556.  URL https://aclanthology.org/2022.acl-long.556.
  * Luccioni & Viviano (2021) Alexandra Luccioni and Joseph Viviano.  What’s in the box? an analysis of undesirable content in the Common Crawl corpus.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 182–189, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.24.  URL https://aclanthology.org/2021.acl-short.24.
  * Luukkonen et al. (2023) Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al.  Fingpt: Large generative models for a small language.  _arXiv preprint arXiv:2311.05640_ , 2023.
  * Madras et al. (2018) David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel.  Learning adversarially fair and transferable representations.  In _International Conference on Machine Learning_ , pp. 3384–3393. PMLR, 2018.
  * Magnusson et al. (2023) Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge.  Paloma: A benchmark for evaluating language model fit, 2023.
  * Maharana et al. (2023) Adyasha Maharana, Prateek Yadav, and Mohit Bansal.  D2 pruning: Message passing for balancing diversity and difficulty in data pruning, 2023.
  * Manber & Myers (1993) Udi Manber and Gene Myers.  Suffix arrays: A new method for on-line string searches.  _SIAM Journal on Computing_ , 22(5):935–948, 1993.  doi: 10.1137/0222058.  URL https://doi.org/10.1137/0222058.
  * Mansoor et al. (2022) Haris Mansoor, Sarwan Ali, Shafiq Alam, Muhammad Asad Khan, Umair Ul Hassan, and Imdadullah Khan.  Impact of missing data imputation on the fairness and accuracy of graph node classifiers.  In _IEEE International Conference on Big Data_ , pp. 5988–5997, 2022.
  * Marone & Durme (2023) Marc Marone and Benjamin Van Durme.  Data portraits: Recording foundation model training data, 2023.
  * Mavromatis et al. (2023) Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, Huzefa Rangwala, Christos Faloutsos, and George Karypis.  Which examples to annotate for in-context learning? towards effective and efficient selection.  _arXiv preprint arXiv:2310.20046_ , 2023.
  * Mazumder et al. (2023) Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William A Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Will Cukierski, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Smriti Raje, Max Bartolo, Sabri Eyuboglu, Amirata Ghorbani, Emmett Daniel Goodman, Addison Howard, Oana Inel, Tariq Kane, Christine Kirkpatrick, D. Sculley, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Y. Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi.  Dataperf: Benchmarks for data-centric AI development.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=LaFKTgrZMG.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_ , 2018.
  * McLure et al. (2014) Merinda McLure, Allison V Level, Catherine L Cranston, Beth Oehlerts, and Mike Culbertson.  Data curation: A study of researcher practices and needs.  _portal: Libraries and the Academy_ , 14(2):139–164, 2014.
  * Min et al. (2022) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi.  MetaICL: Learning to learn in context.  In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2791–2809, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.201.  URL https://aclanthology.org/2022.naacl-main.201.
  * Mindermann et al. (2022) Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal.  Prioritized training on points that are learnable, worth learning, and not yet learnt.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 15630–15649. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/mindermann22a.html.
  * Mirzasoleiman et al. (2020a) Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec.  Coresets for data-efficient training of machine learning models.  In _International Conference on Machine Learning_ , pp. 6950–6960. PMLR, 2020a.
  * Mirzasoleiman et al. (2020b) Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec.  Coresets for robust training of deep neural networks against noisy labels.  _Advances in Neural Information Processing Systems_ , 33:11465–11477, 2020b.
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  _arXiv preprint arXiv:2104.08773_ , 2021.
  * Mitchell et al. (2023) Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela.  Measuring data, 2023.
  * Moore & Lewis (2010) Robert C. Moore and William Lewis.  Intelligent selection of language model training data.  In Jan Hajič, Sandra Carberry, Stephen Clark, and Joakim Nivre (eds.), _Proceedings of the ACL 2010 Conference Short Papers_ , pp. 220–224, Uppsala, Sweden, July 2010. Association for Computational Linguistics.  URL https://aclanthology.org/P10-2041.
  * Mou (2023) Chenghao Mou.  Large-scale near-deduplication behind bigcode, May 2023.  URL https://huggingface.co/blog/dedup.  Accessed: 2023-12-06.
  * Muennighoff (2022) Niklas Muennighoff.  Sgpt: Gpt sentence embeddings for semantic search.  _arXiv preprint arXiv:2202.08904_ , 2022.
  * Muennighoff et al. (2022a) Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.  Mteb: Massive text embedding benchmark.  _arXiv preprint arXiv:2210.07316_ , 2022a.
  * Muennighoff et al. (2022b) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_ , 2022b.
  * Muennighoff et al. (2023a) Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models.  _arXiv preprint arXiv:2308.07124_ , 2023a.
  * Muennighoff et al. (2023b) Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.  Scaling data-constrained language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=j5BuTrEj35.
  * Muennighoff et al. (2024) Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela.  Generative representational instruction tuning.  _arXiv preprint arXiv:2402.09906_ , 2024.
  * Murphy (2012) Kevin P. Murphy.  _Machine Learning: A Probabilistic Perspective_.  The MIT Press, 2012.  ISBN 0262018020.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * Narayan et al. (2022) Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré.  Can foundation models wrangle your data?, 2022.
  * Nguyen & Wong (2023) Tai Nguyen and Eric Wong.  In-context example selection with influences.  _arXiv preprint arXiv:2302.11042_ , 2023.
  * Nguyen et al. (2022) Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt.  Quality not quantity: On the interaction between dataset design and robustness of CLIP.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=LTCBavFWp5C.
  * Nguyen et al. (2020) Timothy Nguyen, Zhourong Chen, and Jaehoon Lee.  Dataset meta-learning from kernel ridge-regression.  In _International Conference on Learning Representations_ , 2020.
  * Nguyen et al. (2021) Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee.  Dataset distillation with infinitely wide convolutional networks.  _Advances in Neural Information Processing Systems_ , 34:5186–5198, 2021.
  * OpenAI et al. (2023) OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2023.
  * Oren et al. (2019) Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust language modeling.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 4227–4237, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1432.  URL https://aclanthology.org/D19-1432.
  * Oren et al. (2023) Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto.  Proving test set contamination in black box language models, 2023.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi (eds.), _Proceedings of the Workshop on Challenges in the Management of Large Corpora_ , Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pp. 9 – 16, Mannheim, 2019. Leibniz-Institut für Deutsche Sprache.  doi: 10.14618/ids-pub-9021.  URL http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.  URL https://arxiv.org/abs/2203.02155.
  * Pace et al. (2024) Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn.  West-of-n: Synthetic preference generation for improved reward modeling, 2024.
  * Pan et al. (2023) Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang.  Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 3806–3824, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.248.  URL https://aclanthology.org/2023.findings-emnlp.248.
  * Park et al. (2023) Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry.  Trak: Attributing model behavior at scale, 2023.
  * Paul et al. (2021) Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=Uj7pF-D-YvT.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho.  True few-shot learning with language models.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=ShnM-rRh4T.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  _NAACL_ , 2018.  URL https://aclanthology.org/N18-1202.
  * Phang et al. (2019) Jason Phang, Thibault Févry, and Samuel R. Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks, 2019.
  * Piergiovanni et al. (2023) AJ Piergiovanni, Weicheng Kuo, Wei Li, and Anelia Angelova.  Dynamic pretraining of vision-language models.  In _ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls_ , 2023.  URL https://openreview.net/forum?id=meQWVbMCqXr.
  * Piktus et al. (2023) Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, and Jimmy Lin.  GAIA search: Hugging face and pyserini interoperability for NLP training data exploration.  In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pp. 588–598, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-demo.57.  URL https://aclanthology.org/2023.acl-demo.57.
  * Pleiss et al. (2020) Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger.  Identifying mislabeled data using the area under the margin ranking.  _Advances in Neural Information Processing Systems_ , 33:17044–17056, 2020.
  * Pooladzandi et al. (2022) Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman.  Adaptive second order coresets for data-efficient machine learning.  In _International Conference on Machine Learning_ , pp. 17848–17869. PMLR, 2022.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman.  Intermediate-task transfer learning with pretrained language models: When and why does it work?  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5231–5247, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.467.  URL https://aclanthology.org/2020.acl-main.467.
  * Radford & Narasimhan (2018) Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training, 2018.  URL https://api.semanticscholar.org/CorpusID:49313245.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.  URL https://api.semanticscholar.org/CorpusID:160025533.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2022.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.