# Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in
ultra low-data regimes

Nabeel Seedat
University of Cambridge
<EMAIL_ADDRESS>&Nicolas Huynh ∗
University of Cambridge
<EMAIL_ADDRESS>Boris van Breugel
University of Cambridge
<EMAIL_ADDRESS>&Mihaela van der Schaar
University of Cambridge
<EMAIL_ADDRESS>denotes equal contribution

###### Abstract

Machine Learning (ML) in low-data settings remains an underappreciated yet
crucial problem. Hence, data augmentation methods to increase the sample size
of datasets needed for ML are key to unlocking the transformative potential of
ML in data-deprived regions and domains. Unfortunately, the limited training
set constrains traditional tabular synthetic data generators in their ability
to generate a large and diverse augmented dataset needed for ML tasks. To
address this challenge, we introduce CLLM, which leverages the prior knowledge
of Large Language Models (LLMs) for data augmentation in the low-data regime.
However, not all the data generated by LLMs will improve downstream utility,
as for any generative model. Consequently, we introduce a principled curation
mechanism, leveraging learning dynamics, coupled with confidence and
uncertainty metrics, to obtain a high-quality dataset. Empirically, on
multiple real-world datasets, we demonstrate the superior performance of CLLM
in the low-data regime compared to conventional generators. Additionally, we
provide insights into the LLM generation and curation mechanism, shedding
light on the features that enable them to output high-quality augmented
datasets.

###  1 Introduction

No data, No Machine Learning. Machine learning (ML) has transformed numerous
industries, but its wider adoption is hindered by a pervasive roadblock:
insufficient data. Specifically, the use of ML algorithms presumes the
availability and access to large datasets for training, be it labeled or
unlabeled. Unfortunately, real-world domains are often data scarce: (i) in
healthcare and finance, collecting annotations can be expensive or practically
impossible; (ii) in developing and low-to-middle income countries (LMICs),
digital infrastructure (such as electronic healthcare records (EHRs)) can be
limited or nonexistent (Ade-Ibijola and Okonkwo, 2023; Asiedu et al., 2023;
Owoyemi et al., 2020; Mollura et al., 2020; Alami et al., 2020; Ciecierski-
Holmes et al., 2022) and (iii) within large datasets, there can be (ethnic)
minorities that are underrepresented. This lack of data has serious
consequences: to sideline these settings to the peripheries of ML advancements
and prevent the development of accurate models. How can we build a reliable ML
model in this _low-data regime_ , with so few samples? Solving this problem is
a major opportunity that would unlock the potential of ML across society,
domains, and regions.

Aim. To address this important yet undervalued low-data problem, we aim to
augment the _small labeled dataset_ ($n<100$) with synthetic samples. We focus
on tabular data, as defining augmentations is non-trivial and can easily
result in nonsensical or invalid samples. Moreover, tabular domains like
healthcare are often where data scarcity is acute.

Figure 1: CLLM uses a small dataset $D_{\mathrm{train}}$ and a frozen black-
box LLM to generate a larger synthetic set $D_{\mathrm{syn}}$. The curator
computes the learning dynamics of samples in $D_{\mathrm{syn}}$, assessing
samples based on their aleatoric uncertainty and predictive confidence, then
curates $D_{\mathrm{syn}}$ with the goal that a downstream model trained on
the curated $D_{\mathrm{curated}}$ will have improved performance.

Related work. Data augmentation is a widely used and different approach to
address data scarcity in tabular data contexts. Methods are either based on
generative models (Ghosheh et al., 2023; Biswas et al., 2023; Wang and Pai,
2023; Machado et al., 2022; Tanaka and Aranha, 2019) such as GANs (Xu et al.,
2019), VAEs (Xu et al., 2019), Normalizing Flows (Papamakarios et al., 2021),
Score-based models (Kotelnikov et al., 2022; Kim et al., 2022), or
alternatively traditional methods such as SMOTE (Chawla et al., 2002; Wang and
Pai, 2023; Machado et al., 2022). However, in low-data regimes ($n<100$), the
training data may not describe the full data distribution well, despite it
being i.i.d. draws. Consequently, this harms conventional methods since the
augmented data may not be sufficiently diverse and accurate, restricting the
generalizability of predictive models trained on such data. Recent work has
shown the potential of fine-tuning Large Language Models (LLMs) for tabular
data generation (Borisov et al., 2023). While LLMs offer some degree of prior
knowledge, there are two challenges in our setting. First, it is
computationally expensive to fine-tune LLMs, while needing specialized
hardware —luxuries often not available in LMICs, thereby limiting
applicability in such settings. Second, fine-tuning often assumes a large
number of samples. In our low-data setting, it could lead to overfitting and
low-quality generated samples, and hence poor downstream models—as we show for
Borisov et al. (2023) in Sec. 3. Tangential to data augmentation, prior work
has tackled data scarcity in the tabular setting via the lens of transfer
learning or few-short learning, by using a knowledge graph (Margeloiu et al.,
2022; Ruiz et al., 2023) (which might not be available) or a pretrained model
(Levin et al., 2022; Jin and Ucar, 2023; Hegselmann et al., 2023). However,
unlike data augmentation, these approaches tie the data customer to use a
certain downstream predictor.

Curated LLMs. To address the shortcomings of the aforementioned augmentation
approaches, we propose Curated LLM (CLLM). First, CLLM leverages the in-
context capabilities of LLMs for generation, thereby reducing the
computational burden compared to fine-tuning. We also posit for the low-data
regime; the diverse pretraining corpus of LLMs carries valuable prior
knowledge, which may offer more diversity in their generation compared to
other conventional tabular generators. _Of course, LLMs are not perfect_.
Consequently, balancing the utility of LLMs against the risk of noisy,
irrelevant data is important to ensure reliable downstream performance. Hence,
this necessitates systematic assessment of the generated data. In fact, this
issue is vital for _any_ generative model.

This motivates the second key aspect of CLLM, i.e. a post-generation data
curation mechanism. This addresses the _overlooked_ aspect that not all of the
synthetic samples are useful to downstream model performance, with some
samples even harmful. We anchor our approach with ideas from learning theory
that show the behavior of individual data samples during training, called
learning dynamics, provides a salient signal about the value of samples to a
learner (Arpit et al., 2017; Arora et al., 2019; Li et al., 2020). To provide
intuition, samples with variable predictions might be considered ambiguous or
other samples might never be learned correctly and could harm a model. In
CLLM, we study the learning dynamics of the synthetic data samples, with
respect to a model trained on the small real dataset. We then analyze these
dynamics by computing two key metrics: confidence and aleatoric (data)
uncertainty. These metrics form the basis for curating the synthetic samples.
We then aim to enable a highly performant downstream model when trained on the
curated dataset.

Contributions: CLLM is a novel data augmentation approach allying the
strengths of LLMs with a robust data curation mechanism to improve data
augmentation in the _low-data regime_ $(n<100)$, bringing several
contributions: ① Improved performance: we empirically demonstrate on 7 real-
world datasets that CLLM enables superior downstream performance compared to 6
widely used tabular data generative models and data augmentation techniques. ②
Value of curation: we show the _overlooked_ aspect of synthetic data curation
improves downstream performance across the generative models. This highlights
the flexibility and broad utility of our curation mechanism for data
augmentation. ③ Insights: we dissect the two aspects of CLLM (LLM and data
curation) along a variety of dimensions, providing insights and understanding
into why the approach is beneficial. We show the largest gains are for
underrepresented subgroups and in low-data settings. These contributions,
which address the data logjam, pave the way towards wider usage of ML across
society, domains and regions.

###  2 CLLM: Synergy of LLM Generation and Data Curation

Set-up. Given feature space $\mathcal{X}$, and label space
$\mathcal{Y}=\\{1,...,k\\}$, we assume that we only have a small labeled
dataset $D_{\mathrm{train}}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, with
$x_{i}\in\mathcal{X}$, $y_{i}\in\mathcal{Y}$ and $n<100$ (low data setting).
Assume $D_{\mathrm{train}}$ is drawn i.i.d. from the real distribution
$p_{R}(X,Y)$. We also assume access to a pretrained LLM to generate samples.
We denote the output distribution of the LLM as $p_{\Phi}(X,Y)$, with $\Phi$
containing parameters that we control (e.g., input prompts). Our goal is to
generate a dataset to augment the small $D_{\mathrm{train}}$, and subsequently
use it to train a classifier $f:\mathcal{X}\rightarrow\mathcal{Y}$. Successful
augmentation will provide a better classifier $f$, than if we had trained $f$
on the small $D_{\mathrm{train}}$ itself. We measure downstream performance on
a separate held-out dataset of real data, $D_{\mathrm{test}}$.

Our Approach. To address this challenge, we introduce CLLM, an approach for
data augmentation in low-data regimes. As shown in Figure 1, CLLM leverages
LLMs to generate a synthetic dataset $D_{\mathrm{syn}}$ using a small dataset
$D_{\mathrm{train}}$ (Sec. 2.1). It exploits the LLMs’ prior knowledge via in-
context learning (ICL) and contextual information. CLLM then curates
$D_{\mathrm{syn}}$ by analyzing the learning dynamics of samples in
$D_{\mathrm{syn}}$ based on predictive confidence and aleatoric (data)
uncertainty. These metrics are obtained by training a supervised model on
$D_{\mathrm{train}}$. We leverage them to define a curated dataset
$D_{\mathrm{curated}}$, which is used to train a downstream classifier (Sec.
2.2).

In each sub-section we describe and motivate the design of the different
aspects of CLLM (LLM and curation mechanism). Furthermore, we provide insights
and understanding into their role in improving data utility, which we later
quantify on multiple real-world datasets in Sec. 3.

####  2.1 Data generation with LLMs based on a small $D_{\mathrm{train}}$

As outlined in Sec. 1, in the low-data regime, conventional tabular generative
models (e.g. CTGAN, TVAE) are constrained by the limited $D_{\mathrm{train}}$
and may not generate sufficiently diverse and/or accurate synthetic data. To
address this, we propose to leverage LLMs, building on their large-scale
pretraining. We first outline the appealing properties of LLMs for tabular
data generation when we have very few samples, then describe design choices to
exploit these.

  * •

Prior knowledge. LLMs are pretrained with a vast corpus of information
(Chowdhery et al., 2022; Singhal et al., 2023). When prompted to generate
samples with limited real data, LLMs can leverage this encoded prior
information about similar problems and feature-label relationships to enhance
both accuracy and diversity of generation.

  * •

Contextual understanding. LLMs can process background and contextual
information about the problem via natural language (Yang et al., 2023). For
example, a high-level description of the task, features and their meanings can
be conveniently described through natural language. Such information is
unavailable to conventional generators that only utilize numerical examples.

  * •

Few-shot capabilities. LLMs have demonstrated proficiency in generalizing to
tasks with just a few examples (Brown et al., 2020; Wei et al., 2023;
Mirchandani et al., 2023). In the context of generation, we envision the idea
of in-context generation using limited real examples.

To benefit from these capabilities, we craft the LLM prompt with three
different parts (see Fig. 1): (1) _Background_ : text description of the
dataset and task (e.g. predict Covid mortality). Additionally, we include a
description of what each feature means, explicitly prompting the LLM to use
prior knowledge about these features. (2) _Examples_ : we serialize the
samples in $D_{\mathrm{train}}$ as example demonstrations and provide both the
features and the label in text format. (3) _Instructions_ : To generate a
synthetic dataset $D_{\mathrm{syn}}$, we instruct the LLM to leverage the
contextual information and provided examples as an i.i.d. draw from the
distribution. We instruct the LLM to identify structural and feature-label
relationships in the data and generate diverse data following the structure
and format of the provided examples. We provide more details on the prompts in
Appendix B.

Motivation for a frozen LLM. Using a frozen black-box LLM (e.g. GPT-4 or
GPT-3.5) is computationally cheaper and requires less specialized hardware
(i.e. GPUs) compared to fine-tuning. This relates to settings described in
Sec. 1, such as LMICs, where we may not have the computational resources to
fine-tune an LLM. Even in settings where fine-tuning is possible, we show
empirically in Sec. 3 that LLM fine-tuning (e.g. GReaT baseline) is suboptimal
in low-data settings ($n<100$) compared to providing in-context examples
coupled with curation.

Dissecting the LLM’s generative features.  We now investigate various
dimensions to understand and illustrate empirically the appealing features of
LLMs as data generators in the low-data regime, and how our design choices
unlock them. We use the Brazil _Covid-19_ dataset (Baqui et al., 2020) as a
running example and focus on GPT-4 as the LLM.

$\blacktriangleright$ Extrapolation to unseen regions of the manifold. We
compare the samples generated by GPT-4 to TVAE, a widely used tabular data
generator. We consider $D_{\mathrm{oracle}}$, a held-out dataset from the same
distribution as $D_{\mathrm{train}}$, such that
$|D_{\mathrm{oracle}}|\gg|D_{\mathrm{train}}|$, thereby providing an
approximation for the true manifold. The t-SNE plots in Fig. 2 shows, when
$D_{\mathrm{train}}$ is very small ($n=20$ samples), that its samples do not
cover all regions of $D_{\mathrm{oracle}}$. For example, $D_{\mathrm{train}}$
does not contain samples from specific demographic subgroups (e.g. people with
age 40 or below). As expected, TVAE only generates samples constrained by the
limited $D_{\mathrm{train}}$. In contrast, GPT-4 is able to extrapolate and
generate samples even in unseen regions of $D_{\mathrm{train}}$, thereby
better covering $D_{\mathrm{oracle}}$. This stems from its _contextual
understanding_ of the features, unlocking the use of its _prior knowledge_. It
leads to better coverage in the low-data regime, consequently aiding in
superior downstream performance, as shown in Table 3. As $n$ increases ($\geq
100$), $D_{\mathrm{train}}$ provides better coverage, which naturally benefits
both GPT-4 and TVAE. Overall, this result shows how prior knowledge encoded in
LLMs addresses shortcomings of conventional generative approaches (e.g. TVAE)
in the low-data regime.

Figure 2: GPT-4 is able to extrapolate to regions of the oracle (true
manifold) even where there is no training data covering them, as can be seen
by the overlap with the turquoise dots, with the effect more pronounced when
$D_{\mathrm{train}}$ is small

$\blacktriangleright$ GPT-4 benefits underrepresented groups the most. Having
illustrated the extrapolation capabilities of GPT-4, we now ask:  _where does
augmentation benefit downstream performance the most?_ We evaluate performance
gains for different demographic subgroups, such as age groups and ethnic
groups (Amarela, Prada). Fig. 3 shows the performance gain obtained by
training a classifier on data generated by GPT-4 compared to training on the
small $D_{\mathrm{train}}$. The greatest gains, on average, are for subgroups
for which we have _no data_ in $D_{\mathrm{train}}$, yet GPT-4 can extrapolate
and generate samples for these subgroups. This further validates the rationale
of extrapolation via prior knowledge as a key source of gain for GPT-4. Table
2.1 shows fine-grained results (across 10 different seeds) for the 5 subgroups
that benefit the most from data augmentation, which are small-sized
demographic subgroups. This finding has real-world implications for equity,
showing we can improve performance for underrepresented subgroups even when we
lack data or collecting data is difficult/costly.

Figure 3: Subgroups with fewest samples in $D_{\mathrm{train}}$ benefit the
most from data augmentation, on average.

Table 1: Deep dive into the top 5 demographic subgroups in the Covid dataset
with the largest gains, across 10 seeds, for $|D_{\mathrm{train}}|=20$. GPT-4
improves performance on the smallest groups.

Subgroup |  $n_{\mathrm{samples}}$ in $D_{\mathrm{train}}$ | Avg. Acc. Gain v. $D_{\mathrm{train}}$
---|---|---
| (min - max) | GPT-4 | TVAE
Age_40 | 0-6 |  6.38 +- 2.09 | -3.37 +- 2.86
Liver | 0-1 |  3.85 +- 3.37 | -13.1 +- 3.38
Renal | 0-3 |  4.52 +- 2.01 | -18.0 +- 3.22
Amarela | 0-1 |  8.71 +- 1.40 | -2.03 +- 2.88
Parda | 3-11 |  5.07 +- 1.50 | -6.57 +- 1.61

$\blacktriangleright$ Importance of contextual information in the prompt.  A
natural question is:  _how important is the prompt to elicit the prior
knowledge of the LLM?_ We explore two variants: (1) _Prompt w/ context_ :
provides contextual information including background about the dataset,
feature names and descriptions (our approach) and (2) _Prompt w/ no context_ :
only provides the numerical in-context examples (ablation). Fig. 4
qualitatively shows that not including contextual knowledge in the prompt
gives lower coverage of $D_{\mathrm{oracle}}$ with less extrapolation beyond
$D_{\mathrm{train}}$. We quantify this in Table 2.1 using Precision (Quality)
and Recall (Diversity) metrics (Sajjadi et al., 2018), as well as Utility
(Downstream performance). GPT-4 _with contextual information_ has superior
precision and recall in the low data setting. Furthermore, we show that _the
lack_ of contextual information in the prompt significantly harms the
precision (quality) of the data even compared to TVAE. This highlights that
LLMs need guidance, as we are only able to get the extrapolation and
performance benefits by including contextual information, further motivating
our design choices in the prompt. We conduct the same experiment with the
Compas dataset in Appendix C.3.

Figure 4: Contextual information in the prompt is important for extrapolation.

Table 2: Including contextual information in the prompt improves precision
(P), recall (R), and utility (U) in low-sample settings (results shown for the
Covid dataset).

$n_{\mathrm{samples}}$ in $D_{\mathrm{train}}$ |  GPT-4 w/ context |  GPT-4 no context | TVAE
---|---|---|---
| P | R | U | P | R | U | P | R | U
20 | $\bf 0.41_{(0.04)}$ | $\bf 0.87_{(0.03)}$ | $\bf 0.74_{(0.01)}$ | $0.13_{(0.0)}$ | $0.82_{(0.01)}$ | $0.66_{(0.01)}$ | $0.33_{(0.07)}$ | $0.50_{(0.03)}$ | $0.59_{(0.02)}$
40 | $\bf 0.40_{(0.01)}$ | $\bf 0.91_{(0.01)}$ | $\bf 0.76_{(0.0)}$ | $0.11_{(0.0)}$ | $0.89_{(0.0)}$ | $0.69_{(0.0)}$ | $0.27_{(0.01)}$ | $0.68_{(0.01)}$ | $0.62_{(0.03)}$
100 | $\bf 0.42_{(0.01)}$ | $0.86_{(0.02)}$ | $\bf 0.75_{(0.01)}$ | $0.11_{(0.01)}$ | $\bf 0.90_{(0.01)}$ | $0.74_{(0.01)}$ | $0.39_{(0.02)}$ | $0.67_{(0.03)}$ | $0.64_{(0.06)}$
200 | $0.44_{(0.02)}$ | $0.85_{(0.02)}$ | $\bf 0.75_{(0.0)}$ | $0.08_{(0.01)}$ | $\bf 0.90_{(0.0)}$ | $0.60_{(0.01)}$ | $\bf 0.47_{(0.0)}$ | $0.73_{(0.01)}$ | $0.65_{(0.02)}$

####  2.2 Data curation with learning dynamics

When prompted with $\Phi$ (which contains the in-context samples of
$D_{\mathrm{train}}$), the LLM generates samples from a distribution
$p_{\Phi}(X,Y)$ that approximates $p_{R}(X,Y)$, implicitly exploiting its
large-scale pretraining and few-shot capabilities. LLMs are of course not
perfect and could generate noisy samples, hence this distribution may be
inaccurate 111We could finetune the model on the scarce $D_{\mathrm{train}}$
we have, but is likely to still lead to overfitting due to the extreme data
scarcity and LLM parameter size.. To make this distribution more relevant to
the downstream task, we include a data curation mechanism. Specifically, we
focus on the noisy feature-label relationship $p_{\Phi}(Y|X)$, for which we
expect $p_{\Phi}(Y|X)\neq p_{R}(Y|X)$ given the small size of
$D_{\mathrm{train}}$. This motivates us to curate $D_{\mathrm{syn}}$ and
discard likely mislabeled samples.

We anchor our approach with ideas from learning theory that show that the
behavior of individual samples during model training (called _learning
dynamics_) contains signal about the nature of the samples themselves (Arpit
et al., 2017; Arora et al., 2019; Li et al., 2020). Some samples are easily
and confidently predicted over different model checkpoints, whereas other
samples might be challenging (e.g. due to mislabeling) and hence might be
incorrectly predicted for the given label. Consequently, we operationalize
_learning dynamics_ as the basis of our proposed curation mechanism.
Specifically, we analyze samples in $D_{\mathrm{syn}}$ by studying their
learning dynamics computed with a classifier trained on $D_{\mathrm{train}}$.
We then categorize and filter samples in $D_{\mathrm{syn}}$, and produce a
curated dataset $D_{\mathrm{curated}}\subset D_{\mathrm{syn}}$.

Learning dynamics. We now formalize how we compute learning dynamics for
individual samples. Assume that a classifier $f$ is trained in an iterative
scheme (e.g. neural networks or XGBoost trained over iterations) on
$D_{\mathrm{train}}$, which makes it possible to analyze the learning dynamics
of samples in $D_{\mathrm{syn}}$ over these iterations. The classifier $f$
should be at least as flexible as the model that the practitioner intends to
use for the downstream task. $f$ is trained from scratch on
$D_{\mathrm{train}}$ and goes through $e\in[E]$ different checkpoints leading
to the set $\mathcal{F}=\\{f_{1},f_{2},\dots,f_{E}\\}$, such that $f_{e}$ is
the classifier at the $e$-th checkpoint. Let $[f_{e}(x)]_{y}$ denote the
predicted probability for class $y$ and sample $x$. Our goal is to assess the
learning dynamics of samples in $D_{\mathrm{syn}}$ over these $E$ training
checkpoints, while we train $f$ on $D_{\mathrm{train}}$. For this, we define
$H$, a random variable following a uniform distribution
$\mathcal{U}_{\mathcal{F}}$ over the set of checkpoints $\mathcal{F}$.
Specifically, given $H=h$ and a sample $(x,y)$, we define the correctness in
the prediction of $H$ as a binary random variable $\hat{Y}_{\mathcal{F}}(x,y)$
with the following conditional:
$P(\hat{Y}_{\mathcal{F}}(x,y)=1|H=h)=[h(x)]_{y}$ and
$P(\hat{Y}_{\mathcal{F}}(x,y)=0|H=h)=1-P(\hat{Y}_{\mathcal{F}}(x,y)=1|H=h)$.

Curation metrics. Equipped with a probabilistic interpretation of the
predictions of a model, we now define two characterization metrics that we use
for curation: (i) average confidence and (ii) aleatoric (data) uncertainty,
inspired by (Kwon et al., 2020; Seedat et al., 2022a).

######  Definition 2.1 (Average confidence).

For any set of checkpoints $\mathcal{F}=\\{f_{1},...,f_{E}\\}$, the average
confidence for a sample $(x,y)$ is defined as the following marginal:

$\displaystyle\bar{P}_{\mathcal{F}}(x,y)$
$\displaystyle:=P(\hat{Y}_{\mathcal{F}}(x,y)=1)$
$\displaystyle=\mathbb{E}_{H\sim\mathcal{U}_{\mathcal{F}}}[P(\hat{Y}_{\mathcal{F}}(x,y)=1|H)]$
$\displaystyle=\frac{1}{E}\sum_{e=1}^{E}[f_{e}(x)]_{y}$

######  Definition 2.2 (Aleatoric uncertainty).

For any set of checkpoints $\mathcal{F}=\\{f_{1},...,f_{E}\\}$, the aleatoric
uncertainty for a sample $(x,y)$ is defined as:

$\displaystyle v_{al,\mathcal{F}}(x,y)$
$\displaystyle:=\mathbb{E}_{H\sim\mathcal{U}_{\mathcal{F}}}[Var(\hat{Y}_{\mathcal{F}}(x,y)|H)]$
$\displaystyle=\frac{1}{E}\sum_{e=1}^{E}[f_{e}(x)]_{y}(1-[f_{e}(x)]_{y})$

Intuitively, for binary classification ($k=2$), the aleatoric uncertainty for
a sample $x$ is maximized when $[f_{e}(x)]_{y}=\frac{1}{2}$ for all
checkpoints $f_{e}$, akin to random guessing. Recall aleatoric uncertainty
captures the inherent data uncertainty, hence is a principled way to capture
issues such as mislabeling. This contrasts epistemic uncertainty, which is
model-dependent and can be reduced simply by increasing model parameterization
(Hüllermeier and Waegeman, 2021).

Having defined sample-wise confidence and aleatoric uncertainty, we categorize
samples in $D_{\mathrm{syn}}$ as Selected or Discarded: for a sample $(x,y)$,
a set of training checkpoints $\mathcal{F}$, and two thresholds
$\tau_{\mathrm{conf}}$ and $\tau_{\mathrm{al}}$, we define the category
$c(x,y,\mathcal{F})$ as Discarded if
$\bar{P}_{\mathcal{F}}(x,y)<\tau_{\mathrm{conf}}$ and
$v_{al,\mathcal{F}}(x,y)<\tau_{\mathrm{al}}$, and Selected otherwise.

Hence, a Discarded sample is one for which we have a very low confidence in
predicting its associated label whereas we also have low inherent data
uncertainty. Finally, given a function $f$ associated with the set of
checkpoints $\mathcal{F}$, we define the curated set
$D_{\mathrm{curated}}=\\{(x,y)|(x,y)\in
D_{\mathrm{syn}},c(x,y,\mathcal{F})=\textit{Selected}\\}$. We also define
$D_{\mathrm{discarded}}=D_{\mathrm{syn}}\setminus D_{\mathrm{curated}}$.

To summarize, the objective of the curation step is that training on the
curated synthetic data leads to a better classifier $f_{D_{\mathrm{curated}}}$
for the downstream task, compared to training on the uncurated synthetic data,
i.e. $M(f_{D_{\mathrm{curated}}})>M(f_{\mathcal{D_{\mathrm{syn}}}})$, where
$M$ is a performance measure (for example accuracy). In Sec. 3, we empirically
show how performance on this curated dataset is superior both for LLM
generated data, as well as other classes of generative models.

Dissecting the role of curation. We now empirically demonstrate the role of
curation in correcting the noisy feature-label relationship present in
$D_{\mathrm{syn}}$, highlighting two insights: (i) curation discards samples
which are atypical in their label with respect to their neighbors in
$D_{\mathrm{syn}}$ (ii) discarded samples can be considered “mislabeled”, and
we quantify their atypicality using a large held-out dataset
$D_{\mathrm{oracle}}$.

$\blacktriangleright$ Discarded samples conflict on the label with their
neighbors in $D_{\mathrm{syn}}$. We audit every synthetic sample $(x,y)$
generated by GPT-4 (across 7 datasets) and compute the proportion of its $k$
nearest neighbors in $D_{\mathrm{syn}}$ which share the same label $y$. The
agreement with the neighbors assesses the typicality of a sample’s $y$ given
$x$, where naturally lower agreement is linked to mislabeling, which we aim to
detect via curation. Taking $k$ = 10, we obtain an average agreement of
$a_{\mathrm{curated}}=\textbf{0.74}$ for $D_{\mathrm{curated}}$, compared to
$a_{\mathrm{discarded}}=\textbf{0.58}$ for $D_{\mathrm{discarded}}$. This
shows that the samples removed by our curation mechanism are those which,
despite having similar features $x$, do not agree with the labels of their
surrounding neighbors. This corroborates ideas in (Ashmore et al., 2021) of
how proximity violations are useful to guide remedial action to improve
models. Not removing these mislabeled samples injects noise into the
downstream classifier, thus reducing performance.

$\blacktriangleright$ Assessing discarded samples with $D_{\mathrm{oracle}}$.
Ideally, the samples we select should better align with the true feature-label
distribution. Since we don’t have access to this distribution explicitly, we
compute a proxy for $\eta(x)=\arg\max_{y}p(Y=y|X=x)$, which we call
$\hat{\eta}$. It is obtained by training a classifier on a held-out dataset
$D_{\mathrm{oracle}}$—the same size as $D_{\mathrm{test}}$ and an order of
magnitude larger than $D_{\mathrm{train}}$. For each synthetic method, we then
report the accuracy of $\hat{\eta}$ on both the curated $D_{\mathrm{curated}}$
and discarded $D_{\mathrm{discarded}}$ datasets —see Fig. 5.

We highlight two key observations. First, the curated datasets, for all the
generative models, exhibit a higher agreement with the proxy $\hat{\eta}$ than
the discarded datasets. This aligns with the desideratum of only keeping
samples that exhibit the correct feature-label relationships.

This provides a rationale for why curation helps improve discriminative
performance, as samples in $D_{\mathrm{curated}}$ are much more likely to have
the correct feature-label relationship.

Second, GPT-4 has a higher agreement with $\hat{\eta}$ on
$D_{\mathrm{discarded}}$, compared to other generators. This illustrates that
GPT-4’s prior knowledge enables it to better capture the distribution
$p(Y|X=x)$. Note that generative baselines (e.g. TVAE) model the joint
$p(X,Y)$, _without any context_ of which is the set of features and which is
the label. In contrast, we can define in the LLM prompt which column is the
target $Y$, allowing the LLM to better capture the feature-label
relationships. This complements the findings from Fig. 2, which showed that
GPT-4 extrapolates to unseen regions of the feature manifold, captured by the
support of $p(X)$.

Figure 5: $\hat{\eta}$ aligns more with $D_{\mathrm{curated}}$ than
$D_{\mathrm{discarded}}$ for each generative model: the curation step keeps
high quality samples tailored to the downstream task.

###  3 Curated LLMs for Better Data Augmentation

We now perform an end-to-end quantitative evaluation of CLLM across multiple
real-world datasets, for downstream utility, demonstrating the value of
allying the generative capabilities of LLMs with our curation mechanism.

Sec. 3.1 compares the downstream performance of models when trained on
uncurated vs curated data for a variety of augmentation approaches. Having
evaluated CLLM on a range of datasets, we also demonstrate how we can leverage
information extracted during curation to characterize datasets via a hardness
proxy. Sec. 3.2 illustrates how our characterization of samples during the
curation step can help to flag synthesized datasets (e.g via the LLM) which,
if used for training, will result in poor downstream performance.

Experimental setup. We compare CLLM (with GPT-4 (OpenAI, 2023) and GPT-3.5
(Brown et al., 2020)) against a variety of baselines for tabular data
generation and augmentation: CTGAN (Xu et al., 2019), TVAE (Xu et al., 2019),
Normalizing Flows (Papamakarios et al., 2021), TabDDPM (Kotelnikov et al.,
2022), SMOTE (Chawla et al., 2002) and GReaT (Borisov et al., 2023), which
fine-tunes an LLM. We evaluate performance on 7 real-world datasets with
different feature counts and representative of the diverse domains where CLLM
can have impact. For each dataset, we vary the number of samples available in
$D_{\mathrm{train}}$, repeating each experiment for $10$ seeds.

While we do not know the exact makeup of the pretraining data for LLMs like
GPT-4, there is the possibility that open-source data might be included. This
poses the risk of memorization as the primary source of performance gain. To
disentangle the role of memorization, we select 4 real-world medical datasets
(Maggic (Pocock et al., 2013), Covid (Baqui et al., 2020), SEER (Duggan et
al., 2016), CUTRACT (PCUK, 2019)) that require an authorization process to
access, hence are unlikely to form part of the LLMs training corpus. We use
common open-source datasets (Adult and Drug from the UCI repository (Asuncion
and Newman, 2007) and Compas (Angwin et al., 2016)) that are highly reflective
of data scarce domains. Further experimental details can be found in Appendix
B.

Table 3: AUC averaged over 4 downstream models on $D_{\mathrm{test}}$ where
curation improves performance for all methods across all sample sizes $n$, as
indicated by $\uparrow$. CLLM w/ GPT-4 (Curated) dataset provides the
strongest performance for both private/proprietary datasets and public
datasets

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 74.41 | 68.50 | 73.78 | 73.87 $\uparrow$ | 69.85 | 71.41 $\uparrow$ | 59.00 | 63.67 $\uparrow$ | 66.84 | 66.85 $\uparrow$ | 57.38 | 66.46 $\uparrow$ | 62.87 | 68.56 $\uparrow$ | 66.95 | 66.82 | 61.69 | 66.11 $\uparrow$
cutract (n=20) | 72.23 | 70.12 | 71.15 | 72.50 $\uparrow$ | 69.97 | 71.54 $\uparrow$ | 64.01 | 67.98 $\uparrow$ | 66.05 | 66.59 $\uparrow$ | 52.38 | 67.02 $\uparrow$ | 64.44 | 70.42 $\uparrow$ | 68.41 | 69.24 $\uparrow$ | 68.94 | 70.22 $\uparrow$
maggic (n=20) | 67.41 | 57.13 | 60.70 | 61.48 $\uparrow$ | 57.54 | 58.69 $\uparrow$ | 52.75 | 54.51 $\uparrow$ | 54.59 | 55.39 $\uparrow$ | 50.29 | 55.64 $\uparrow$ | 54.72 | 57.38 $\uparrow$ | 55.84 | 56.15 $\uparrow$ | 54.08 | 56.19 $\uparrow$
seer (n=20) | 87.92 | 80.67 | 84.53 | 84.82 $\uparrow$ | 83.34 | 83.71 $\uparrow$ | 74.34 | 78.73 $\uparrow$ | 80.59 | 80.60 $\uparrow$ | 47.57 | 74.43 $\uparrow$ | 76.06 | 79.98 $\uparrow$ | 79.23 | 80.02 $\uparrow$ | 74.53 | 78.73 $\uparrow$
compas (n=20) | 67.51 | 63.11 | 68.01 | 67.91 | 62.07 | 64.43 $\uparrow$ | 55.67 | 62.56 $\uparrow$ | 57.67 | 60.87 $\uparrow$ | 53.33 | 63.59 $\uparrow$ | 59.49 | 64.62 $\uparrow$ | 61.06 | 61.59 $\uparrow$ | 58.30 | 62.58 $\uparrow$
adult (n=20) | 84.17 | 77.45 | 50.39 | 71.48 $\uparrow$ | 49.23 | 72.37 $\uparrow$ | 72.23 | 76.86 $\uparrow$ | 74.35 | 75.04 $\uparrow$ | 67.00 | 77.25 $\uparrow$ | 67.46 | 76.48 $\uparrow$ | 73.75 | 73.67 | 73.20 | 76.90 $\uparrow$
drug (n=20) | 77.81 | 70.84 | 75.08 | 75.29 $\uparrow$ | 71.68 | 72.14 $\uparrow$ | 68.31 | 72.65 $\uparrow$ | 68.12 | 69.68 $\uparrow$ | 58.78 | 68.89 $\uparrow$ | 62.13 | 67.75 $\uparrow$ | 70.16 | 70.16 | 66.60 | 69.18 $\uparrow$
covid (n=40) | 75.02 | 70.77 | 73.40 | 73.95 $\uparrow$ | 70.42 | 71.93 $\uparrow$ | 63.63 | 68.46 $\uparrow$ | 70.50 | 70.44 | 56.50 | 68.68 $\uparrow$ | 66.41 | 70.48 $\uparrow$ | 68.66 | 68.44 | 61.03 | 67.35 $\uparrow$
cutract (n=40) | 72.57 | 69.18 | 69.87 | 71.72 $\uparrow$ | 68.47 | 69.56 $\uparrow$ | 63.01 | 67.87 $\uparrow$ | 65.63 | 67.27 $\uparrow$ | 54.39 | 68.44 $\uparrow$ | 61.40 | 67.98 $\uparrow$ | 67.86 | 67.95 $\uparrow$ | 59.79 | 66.62 $\uparrow$
maggic (n=40) | 67.50 | 58.26 | 59.29 | 60.77 $\uparrow$ | 57.50 | 59.15 $\uparrow$ | 55.00 | 56.78 $\uparrow$ | 55.24 | 56.94 $\uparrow$ | 48.81 | 56.64 $\uparrow$ | 54.68 | 58.58 $\uparrow$ | 57.40 | 57.44 $\uparrow$ | 55.04 | 57.33 $\uparrow$
seer (n=40) | 87.90 | 82.93 | 84.29 | 84.93 $\uparrow$ | 83.46 | 84.44 $\uparrow$ | 80.05 | 83.67 $\uparrow$ | 82.59 | 81.37 | 54.93 | 81.11 $\uparrow$ | 79.88 | 84.36 $\uparrow$ | 80.79 | 82.21 $\uparrow$ | 78.69 | 83.62 $\uparrow$
compas (n=40) | 67.35 | 62.34 | 67.57 | 67.85 $\uparrow$ | 61.34 | 62.84 $\uparrow$ | 56.29 | 61.02 $\uparrow$ | 58.85 | 60.11 $\uparrow$ | 58.88 | 64.37 $\uparrow$ | 58.61 | 63.54 $\uparrow$ | 60.83 | 60.95 $\uparrow$ | 55.94 | 61.04 $\uparrow$
adult (n=40) | 84.43 | 79.44 | 48.31 | 73.82 $\uparrow$ | 49.21 | 74.27 $\uparrow$ | 71.82 | 79.11 $\uparrow$ | 71.51 | 77.99 $\uparrow$ | 66.77 | 78.81 $\uparrow$ | 71.13 | 79.71 $\uparrow$ | 77.90 | 78.84 $\uparrow$ | 72.58 | 80.02 $\uparrow$
drug (n=40) | 77.71 | 71.86 | 74.30 | 75.79 $\uparrow$ | 71.33 | 72.76 $\uparrow$ | 69.46 | 72.74 $\uparrow$ | 71.08 | 73.07 $\uparrow$ | 64.89 | 73.64 $\uparrow$ | 62.51 | 70.97 $\uparrow$ | 69.23 | 69.78 $\uparrow$ | 65.22 | 70.30 $\uparrow$
covid (n=100) | 74.52 | 71.57 | 73.77 | 74.71 $\uparrow$ | 70.71 | 72.76 $\uparrow$ | 69.05 | 72.13 $\uparrow$ | 71.60 | 73.22 $\uparrow$ | 63.52 | 72.04 $\uparrow$ | 64.25 | 72.64 $\uparrow$ | 70.08 | 70.78 $\uparrow$ | 69.05 | 71.96 $\uparrow$
cutract (n=100) | 72.36 | 70.96 | 70.20 | 72.51 $\uparrow$ | 69.97 | 71.94 $\uparrow$ | 67.94 | 72.42 $\uparrow$ | 70.53 | 71.98 $\uparrow$ | 55.72 | 69.14 $\uparrow$ | 67.59 | 72.42 $\uparrow$ | 68.79 | 69.68 $\uparrow$ | 66.89 | 71.52 $\uparrow$
maggic (n=100) | 67.46 | 59.65 | 58.98 | 61.32 $\uparrow$ | 55.71 | 58.90 $\uparrow$ | 57.20 | 59.34 $\uparrow$ | 57.26 | 58.28 $\uparrow$ | 49.54 | 57.91 $\uparrow$ | 56.36 | 60.11 $\uparrow$ | 58.89 | 58.99 $\uparrow$ | 56.17 | 58.86 $\uparrow$
seer (n=100) | 87.79 | 83.95 | 84.45 | 85.37 $\uparrow$ | 83.92 | 85.08 $\uparrow$ | 81.60 | 85.14 $\uparrow$ | 83.04 | 84.83 $\uparrow$ | 70.32 | 83.83 $\uparrow$ | 81.16 | 85.03 $\uparrow$ | 81.82 | 82.49 $\uparrow$ | 78.88 | 84.50 $\uparrow$
compas (n=100) | 67.18 | 62.56 | 68.02 | 68.19 $\uparrow$ | 60.10 | 62.47 $\uparrow$ | 60.01 | 63.73 $\uparrow$ | 58.32 | 61.34 $\uparrow$ | 59.97 | 64.19 $\uparrow$ | 60.02 | 64.04 $\uparrow$ | 61.44 | 61.73 $\uparrow$ | 59.97 | 62.82 $\uparrow$
adult (n=100) | 84.34 | 81.24 | 46.09 | 74.57 $\uparrow$ | 47.56 | 73.97 $\uparrow$ | 74.29 | 80.45 $\uparrow$ | 75.93 | 78.22 $\uparrow$ | 77.09 | 81.66 $\uparrow$ | 70.70 | 81.04 $\uparrow$ | 80.56 | 81.10 $\uparrow$ | 74.04 | 80.23 $\uparrow$
drug (n=100) | 78.00 | 73.58 | 76.24 | 76.74 $\uparrow$ | 69.46 | 71.05 $\uparrow$ | 68.19 | 73.28 $\uparrow$ | 72.43 | 73.79 $\uparrow$ | 67.26 | 75.28 $\uparrow$ | 62.67 | 73.12 $\uparrow$ | 70.90 | 71.53 $\uparrow$ | 68.22 | 73.59 $\uparrow$
covid (n=200) | 74.69 | 72.33 | 73.40 | 74.62 $\uparrow$ | 70.70 | 73.12 $\uparrow$ | 71.07 | 73.89 $\uparrow$ | 72.47 | 74.44 $\uparrow$ | 65.55 | 73.07 $\uparrow$ | 65.04 | 72.90 $\uparrow$ | 71.68 | 71.87 $\uparrow$ | 67.89 | 72.38 $\uparrow$
cutract (n=200) | 72.52 | 71.75 | 71.39 | 73.01 $\uparrow$ | 70.28 | 72.39 $\uparrow$ | 69.28 | 72.41 $\uparrow$ | 71.83 | 74.03 $\uparrow$ | 66.66 | 72.49 $\uparrow$ | 68.77 | 73.16 $\uparrow$ | 70.23 | 70.80 $\uparrow$ | 66.61 | 71.87 $\uparrow$
maggic (n=200) | 67.37 | 61.39 | 58.92 | 61.41 $\uparrow$ | 57.33 | 60.16 $\uparrow$ | 58.48 | 61.33 $\uparrow$ | 56.26 | 57.20 $\uparrow$ | 50.74 | 59.60 $\uparrow$ | 55.95 | 60.75 $\uparrow$ | 60.73 | 60.78 $\uparrow$ | 57.18 | 60.23 $\uparrow$
seer (n=200) | 87.84 | 84.63 | 84.39 | 85.56 $\uparrow$ | 83.48 | 84.80 $\uparrow$ | 82.04 | 85.34 $\uparrow$ | 84.39 | 86.57 $\uparrow$ | 82.15 | 86.03 $\uparrow$ | 77.73 | 85.19 $\uparrow$ | 83.38 | 84.15 $\uparrow$ | 79.71 | 85.26 $\uparrow$
compas (n=200) | 67.14 | 63.27 | 67.02 | 68.15 $\uparrow$ | 60.48 | 63.39 $\uparrow$ | 60.58 | 64.32 $\uparrow$ | 60.60 | 63.52 $\uparrow$ | 61.11 | 65.08 $\uparrow$ | 56.58 | 63.60 $\uparrow$ | 61.99 | 62.80 $\uparrow$ | 60.15 | 63.99 $\uparrow$
adult (n=200) | 84.25 | 82.12 | 40.96 | 75.84 $\uparrow$ | 49.89 | 76.11 $\uparrow$ | 78.18 | 82.32 $\uparrow$ | 81.66 | 83.17 $\uparrow$ | 80.06 | 83.32 $\uparrow$ | 74.31 | 82.64 $\uparrow$ | 82.26 | 82.39 $\uparrow$ | 75.21 | 82.02 $\uparrow$
drug (n=200) | 77.36 | 76.10 | 75.58 | 76.06 $\uparrow$ | 70.66 | 72.81 $\uparrow$ | 71.31 | 75.98 $\uparrow$ | 69.61 | 71.79 $\uparrow$ | 72.35 | 77.41 $\uparrow$ | 65.25 | 75.26 $\uparrow$ | 74.38 | 74.78 $\uparrow$ | 68.39 | 74.33 $\uparrow$

####  3.1 Overall performance: downstream utility

We assess overall performance based on _Utility_ of the augmented data, which
we evaluate in terms of AUC on the real $D_{\mathrm{test}}$, when using four
different types of downstream models (see Appendix B). This setup mirrors the
widely adopted Train-on-synthetic-Test-on-real (TSTR) (Esteban et al., 2017).
Additionally, we compare the performance to training on the small
$D_{\mathrm{train}}$, as well as training on the large held-out
$D_{\mathrm{oracle}}$, the latter serving as an upper bound.

GPT-4 + Curation has best overall performance. Table 3 shows the performance
of the proposed CLLM (GPT- 4 and GPT-3.5) vs baselines — both _with_ and
_without_ our curation mechanism. We find that the GPT-4 + Curation variant of
CLLM outperforms baselines in almost all settings ($20/28$). Interestingly,
its performance is close to or even exceeds the performance of
$D_{\mathrm{oracle}}$. Table 4 further shows that GPT-4 + Curation ranks first
on average vs all generative methods.

Sample size sensitivity. We now investigate the performance gains of CLLM as
we vary the number of samples $n$ in $D_{\mathrm{train}}$, in Table 3 and
Table 4. Performance improvements and high ranking across datasets for CLLM
(GPT-4+Curation) are especially noticeable in the low-data regime (i.e.
$n<100$). In this regime, the limited size of $D_{\mathrm{train}}$ severely
constrains the other baseline methods. In contrast, as illustrated in Sec.
2.1, CLLM can leverage GPT-4’s prior knowledge to extrapolate beyond the small
$D_{\mathrm{train}}$, thereby improving downstream performance. As expected,
the performance gap between CLLM and other methods decreases as the size of
$D_{\mathrm{train}}$ grows (e.g. $n=200$), where sufficient training data
helps other generators achieve good performance. We further decouple the prior
knowledge of the LLM and the number of in-context samples in Appendix C.1,
showing the importance of the in-context samples to guide the LLM’s
generation.

Table 4: Average rank of approaches across the different datasets and seeds.
CLLM w/ GPT-4 ranks first across all $n$ and curation improves all the
generative models.

Method | n=20 | n=40 | n=100 | n=200
---|---|---|---|---
CLLM w/ GPT-4 | 2.71 $\pm$ 1.44 | 2.14 $\pm$ 1.06 | 2.29 $\pm$ 1.19 | 3.29 $\pm$ 1.38
GPT-4 | 3.86 $\pm$ 1.73 | 4.29 $\pm$ 1.83 | 6.00 $\pm$ 1.77 | 7.57 $\pm$ 1.65
CLLM w/ GPT-3.5 | 4.14 $\pm$ 0.94 | 4.14 $\pm$ 0.71 | 6.86 $\pm$ 1.24 | 7.57 $\pm$ 0.70
NFLOW (curated) | 6.00 $\pm$ 1.21 | 4.71 $\pm$ 0.80 | 4.00 $\pm$ 0.57 | 4.71 $\pm$ 0.63
GPT-3.5 | 6.71 $\pm$ 1.52 | 7.29 $\pm$ 1.26 | 11.57 $\pm$ 0.94 | 12.57 $\pm$ 0.57
TVAE (curated) | 7.14 $\pm$ 1.17 | 7.86 $\pm$ 1.30 | 6.43 $\pm$ 0.40 | 6.71 $\pm$ 0.52
SMOTE (curated) | 7.71 $\pm$ 0.33 | 8.14 $\pm$ 0.91 | 7.71 $\pm$ 1.19 | 7.43 $\pm$ 1.07
SMOTE | 7.86 $\pm$ 0.55 | 9.57 $\pm$ 0.80 | 9.57 $\pm$ 1.09 | 9.00 $\pm$ 1.03
TabDDPM (curated) | 8.29 $\pm$ 0.98 | 8.00 $\pm$ 0.93 | 6.00 $\pm$ 0.95 | 5.14 $\pm$ 1.68
CTGAN (curated) | 8.29 $\pm$ 1.42 | 7.14 $\pm$ 0.91 | 4.14 $\pm$ 0.62 | 3.71 $\pm$ 0.39
GReaT (curated) | 8.57 $\pm$ 1.50 | 6.57 $\pm$ 1.21 | 6.29 $\pm$ 1.38 | 3.57 $\pm$ 0.92
TabDDPM | 10.14 $\pm$ 1.19 | 9.86 $\pm$ 1.15 | 10.00 $\pm$ 1.03 | 10.29 $\pm$ 1.02
TVAE | 12.14 $\pm$ 0.89 | 14.00 $\pm$ 0.70 | 13.71 $\pm$ 0.39 | 14.43 $\pm$ 0.40
NFLOW | 12.86 $\pm$ 0.47 | 14.14 $\pm$ 0.37 | 14.00 $\pm$ 0.45 | 15.29 $\pm$ 0.33
CTGAN | 13.86 $\pm$ 0.68 | 13.14 $\pm$ 0.47 | 12.86 $\pm$ 0.37 | 12.00 $\pm$ 0.53
GReaT | 15.71 $\pm$ 0.26 | 15.00 $\pm$ 0.53 | 14.57 $\pm$ 1.03 | 12.71 $\pm$ 0.96

Curation generally helps all generative models. Our curation mechanism
consistently benefits all generative models for the different $n$. It ensures
that only high-quality samples are retained, which is crucial for good data
augmentation and downstream performance and has been overlooked in previous
works. This explains why the combination of the best generative model and
curation (CLLM) gives the best results and highest rankings in the low-data
regime (e.g. $n=20$). In addition to GPT-4 and GPT-3.5, we show the
versatility of our proposed curation mechanism to provide benefit with other
LLM backbones (cf. Appendix C.2).

Performance benefits maintained for private and public datasets. One may
hypothesize that the strong LLM (e.g. GPT-4) performance is explained by
datasets being part of the LLMs’ training corpus, hence possibly being
memorized. We show in Table 3 that it is unlikely, as we retain strong
performance for both open-source datasets, as well as private medical datasets
which require authorization processes for access and are unlikely to be part
of the LLM pretraining dataset.

Remark on ICL versus fine-tuning. Our results in Table 3 and Table 4 indicate
that ICL is better than fine-tuning (GReaT baseline) in the low-data regime.
This highlights the difficulty of fine-tuning in this regime, where it is easy
to overfit to $D_{\mathrm{train}}$. As we increase the number of samples, this
baseline, coupled with curation, improves to the level of CLLM (GPT-4).

####  3.2 Hardness: a proxy signal to flag poor quality synthetic datasets

Figure 6: The proportion of discarded samples $D_{\mathrm{syn}}$ is a proxy
for test performance. This negative linear relationship where each point is a
synthetic dataset generated by GPT-4 (e.g. Adult, Covid, Compas) allows us to
flag datasets that will lead to unreliable downstream performance.

Having a systematic way to assess datasets generated by LLMs like GPT-4 is
important because their black-box nature provides little control on their
generation quality. This contrasts conventional generators for which training
loss is an exploitable signal. Hence, we ask: could we have a signal to
identify a potential problematic dataset generated by an LLM without an
exhaustive manual review? For example, GPT-4 produced low-quality synthetic
data for the Adult dataset (across the different sample sizes) resulting in
poor downstream performance. While curation improves it, downstream
performance is still suboptimal.

Addressing this question is important, since datasets are rarely created by
the ML model builder in real-world ML workflows, but rather by specialist data
teams or data owners (Gebru et al., 2021; Sambasivan et al., 2021; Goncalves
et al., 2020). Thus, having a signal to preemptively flag a potentially
suboptimal generated dataset spares investment in both storing the subpar data
and/or training a model likely to underperform on real data. To address this,
we posit that $D_{\mathrm{syn}}$ should intuitively be considered imperfect if
curation discards many of its samples, since the number of discarded samples
measures the quality of samples with respect to the small but gold-standard
$D_{\mathrm{train}}$.

Hence, we investigate the relationship between test performance (AUC) and the
proportion of samples discarded by curation. Fig. 6, where each point is a
synthetic dataset generated by GPT-4 (e.g. Adult, Compas), shows a strong
negative linear relationship between these two quantities. This holds across
the different $n$ with slopes fairly stable around $-1.4$. This relationship
corroborates the poor quality of the dataset generated by GPT-4 on the Adult
dataset, providing a useful proxy that $D_{\mathrm{syn}}$ is unlikely to lead
to good downstream performance.

###  4 Discussion

We introduce CLLM, an approach for data augmentation in the low-data setting.
CLLM exploits the prior knowledge of LLMs along with our curation mechanism
for improved downstream performance. As empirically shown, CLLM outperforms
traditional generative models—most noticeably on underrepresented subgroups,
for which data augmentation is of utmost importance. CLLM is grounded in the
ICL capability of LLMs. Further improvements may be achieved through different
tuning and prompting of the LLM, as shown in different domains (Meng et al.,
2023; Liu et al., 2023). Improving LLM tuning and prompting is beyond the
scope of our work, but we regard this as a promising avenue for future work.

While the key contribution of this work is the curation of LLM outputs,
overall downstream performance gains are still fundamentally tied to the LLM
backbone. Specifically, a practical consideration is that using less
parameterized LLMs leads to poorer uncurated data. That said, our curation
mechanism naturally addresses this aspect and improves downstream performance
(see Appendix C.2).

Finally, while CLLM addresses the data logjam issue, increasing access to ML
across regions, domains and societies is also about more than just technology.
We believe broader engagement and discussion with various stakeholders is
crucial to responsibly expand ML access, thereby realizing the benefits of ML
in an equitable way.

Data scarcity and computational limitations are deterrents for developing ML.
These challenges should inspire cutting-edge ML research (De-Arteaga et al.,
2018). We believe CLLM takes a step in this direction toward improving the use
of ML in low-data settings, across society (e.g. underrepresented subgroups
(Suresh and Guttag, 2021)), domains (e.g. healthcare (Alami et al., 2020;
Owoyemi et al., 2020)) and regions (e.g. LMICs).

Ethics. In this work, we evaluate CLLM using multiple real-world datasets. The
private datasets are _de-identified_ and used in accordance with the guidance
of the respective data providers. We follow recommendations to use the Azure
OpenAI service when using GPT-4 and GPT-3.5 models, where via the agreement we
ensure the medical data is not sent for human review or stored, hence
respecting the guidelines given by the dataset providers.  LLMs may make
errors and may reflect or exacerbate societal biases that are present in their
data (Li et al., 2023). Though the curation in CLLM improves synthetic data
quality, it does not directly aim to remove biases. The quality and fairness
of generated data should always be evaluated. More research into LLM bias is
required before methods like CLLM should be applied to real-world sensitive
settings like healthcare and finance.

### References

  * Ade-Ibijola and Okonkwo [2023] Abejide Ade-Ibijola and Chinedu Okonkwo.  Artificial intelligence in africa: Emerging challenges.  In _Responsible AI in Africa: Challenges and Opportunities_ , pages 101–117. Springer International Publishing Cham, 2023.
  * Asiedu et al. [2023] Mercy Nyamewaa Asiedu, Awa Dieng, Abigail Oppong, Maria Nagawa, Sanmi Koyejo, and Katherine Heller.  Globalizing fairness attributes in machine learning: A case study on health in africa.  _arXiv preprint arXiv:2304.02190_ , 2023.
  * Owoyemi et al. [2020] Ayomide Owoyemi, Joshua Owoyemi, Adenekan Osiyemi, and Andy Boyd.  Artificial intelligence for healthcare in africa.  _Frontiers in Digital Health_ , 2:6, 2020.
  * Mollura et al. [2020] Daniel J Mollura, Melissa P Culp, Erica Pollack, Gillian Battino, John R Scheel, Victoria L Mango, Ameena Elahi, Alan Schweitzer, and Farouk Dako.  Artificial intelligence in low-and middle-income countries: innovating global health radiology.  _Radiology_ , 297(3):513–520, 2020.
  * Alami et al. [2020] Hassane Alami, Lysanne Rivard, Pascale Lehoux, Steven J Hoffman, Stéphanie Bernadette Mafalda Cadeddu, Mathilde Savoldelli, Mamane Abdoulaye Samri, Mohamed Ali Ag Ahmed, Richard Fleet, and Jean-Paul Fortin.  Artificial intelligence in health care: laying the foundation for responsible, sustainable, and inclusive innovation in low-and middle-income countries.  _Globalization and Health_ , 16:1–6, 2020.
  * Ciecierski-Holmes et al. [2022] Tadeusz Ciecierski-Holmes, Ritvij Singh, Miriam Axt, Stephan Brenner, and Sandra Barteit.  Artificial intelligence for strengthening healthcare systems in low-and middle-income countries: a systematic scoping review.  _npj Digital Medicine_ , 5(1):162, 2022.
  * Ghosheh et al. [2023] Ghadeer O Ghosheh, C Louise Thwaites, and Tingting Zhu.  Synthesizing electronic health records for predictive models in low-middle-income countries (lmics).  _Biomedicines_ , 11(6):1749, 2023.
  * Biswas et al. [2023] Angona Biswas, MD Nasim, Al Imran, Anika Tabassum Sejuty, Fabliha Fairooz, Sai Puppala, and Sajedul Talukder.  Generative adversarial networks for data augmentation.  _arXiv preprint arXiv:2306.02019_ , 2023.
  * Wang and Pai [2023] Winston Wang and Tun-Wen Pai.  Enhancing small tabular clinical trial dataset through hybrid data augmentation: Combining smote and wcgan-gp.  _Data_ , 8(9):135, 2023.
  * Machado et al. [2022] Pedro Machado, Bruno Fernandes, and Paulo Novais.  Benchmarking data augmentation techniques for tabular data.  In _International Conference on Intelligent Data Engineering and Automated Learning_ , pages 104–112. Springer, 2022.
  * Tanaka and Aranha [2019] Fabio Henrique Kiyoiti Dos Santos Tanaka and Claus Aranha.  Data augmentation using gans.  _arXiv preprint arXiv:1904.09135_ , 2019.
  * Xu et al. [2019] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling tabular data using conditional gan.  _Advances in neural information processing systems_ , 32, 2019.
  * Papamakarios et al. [2021] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _The Journal of Machine Learning Research_ , 22(1):2617–2680, 2021.
  * Kotelnikov et al. [2022] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko.  Tabddpm: Modelling tabular data with diffusion models.  _arXiv preprint arXiv:2209.15421_ , 2022.
  * Kim et al. [2022] Jayoung Kim, Chaejeong Lee, and Noseong Park.  Stasy: Score-based tabular data synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357, 2002.
  * Borisov et al. [2023] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci.  Language models are realistic tabular data generators.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Margeloiu et al. [2022] Andrei Margeloiu, Nikola Simidjievski, Pietro Lio, and Mateja Jamnik.  Graph-conditioned mlp for high-dimensional tabular biomedical data.  _arXiv preprint arXiv:2211.06302_ , 2022.
  * Ruiz et al. [2023] Camilo Ruiz, Hongyu Ren, Kexin Huang, and Jure Leskovec.  Enabling tabular deep learning when d $\backslash\gg n$ with an auxiliary knowledge graph.  _arXiv preprint arXiv:2306.04766_ , 2023.
  * Levin et al. [2022] Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum.  Transfer learning with deep tabular models.  _arXiv preprint arXiv:2206.15306_ , 2022.
  * Jin and Ucar [2023] Qixuan Jin and Talip Ucar.  Benchmarking tabular representation models in transfer learning settings.  In _NeurIPS 2023 Second Table Representation Learning Workshop_ , 2023.
  * Hegselmann et al. [2023] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag.  Tabllm: Few-shot classification of tabular data with large language models.  In _International Conference on Artificial Intelligence and Statistics_ , pages 5549–5581. PMLR, 2023.
  * Arpit et al. [2017] Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al.  A closer look at memorization in deep networks.  In _International conference on machine learning_ , pages 233–242. PMLR, 2017.
  * Arora et al. [2019] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang.  Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.  In _International Conference on Machine Learning_ , pages 322–332. PMLR, 2019.
  * Li et al. [2020] Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang.  Learning over-parametrized two-layer neural networks beyond ntk.  In _Conference on learning theory_ , pages 2613–2682. PMLR, 2020.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Singhal et al. [2023] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  _Nature_ , pages 1–9, 2023.
  * Yang et al. [2023] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Wei et al. [2023] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al.  Larger language models do in-context learning differently.  _arXiv preprint arXiv:2303.03846_ , 2023.
  * Mirchandani et al. [2023] Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng.  Large language models as general pattern machines.  _arXiv preprint arXiv:2307.04721_ , 2023.
  * Baqui et al. [2020] Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar.  Ethnic and regional variations in hospital mortality from covid-19 in brazil: a cross-sectional observational study.  _The Lancet Global Health_ , 8(8):e1018–e1026, 2020.
  * Sajjadi et al. [2018] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly.  Assessing generative models via precision and recall.  _Advances in neural information processing systems_ , 31, 2018.
  * Kwon et al. [2020] Yongchan Kwon, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik.  Uncertainty quantification using bayesian neural networks in classification: Application to biomedical image segmentation.  _Computational Statistics & Data Analysis_, 142:106816, 2020.
  * Seedat et al. [2022a] Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar.  Data-iq: Characterizing subgroups with heterogeneous outcomes in tabular data.  In _Advances in Neural Information Processing Systems_ , 2022a.
  * Hüllermeier and Waegeman [2021] Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110:457–506, 2021.
  * Ashmore et al. [2021] Rob Ashmore, Radu Calinescu, and Colin Paterson.  Assuring the machine learning lifecycle: Desiderata, methods, and challenges.  _ACM Computing Surveys (CSUR)_ , 54(5):1–39, 2021.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Pocock et al. [2013] Stuart J Pocock, Cono A Ariti, John JV McMurray, Aldo Maggioni, Lars Køber, Iain B Squire, Karl Swedberg, Joanna Dobson, Katrina K Poppe, Gillian A Whalley, et al.  Predicting survival in heart failure: a risk score based on 39 372 patients from 30 studies.  _European Heart Journal_ , 34(19):1404–1413, 2013.
  * Duggan et al. [2016] Máire A Duggan, William F Anderson, Sean Altekruse, Lynne Penberthy, and Mark E Sherman.  The surveillance, epidemiology and end results (SEER) program and pathology: towards strengthening the critical relationship.  _The American Journal of Surgical Pathology_ , 40(12):e94, 2016.
  * PCUK [2019] Prostate Cancer UK PCUK.  Cutract.  _https://prostatecanceruk.org_ , 2019.
  * Asuncion and Newman [2007] Arthur Asuncion and David Newman.  UCI machine learning repository, 2007.
  * Angwin et al. [2016] Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu.  Machine bias.  _ProPublica: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing_ , May 2016.
  * Esteban et al. [2017] Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch.  Real-valued (medical) time series generation with recurrent conditional gans.  _arXiv preprint arXiv:1706.02633_ , 2017.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021.
  * Sambasivan et al. [2021] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–15, 2021.
  * Goncalves et al. [2020] Andre Goncalves, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle, and Ana Paula Sales.  Generation and evaluation of synthetic patient data.  _BMC medical research methodology_ , 20(1):1–40, 2020.
  * Meng et al. [2023] Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han.  Tuning language models as training data generators for augmentation-enhanced few-shot learning.  In _International Conference on Machine Learning_ , pages 24457–24477. PMLR, 2023.
  * Liu et al. [2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35, 2023.
  * De-Arteaga et al. [2018] Maria De-Arteaga, William Herlands, Daniel B Neill, and Artur Dubrawski.  Machine learning for the developing world.  _ACM Transactions on Management Information Systems (TMIS)_ , 9(2):1–14, 2018.
  * Suresh and Guttag [2021] Harini Suresh and John Guttag.  A framework for understanding sources of harm throughout the machine learning life cycle.  In _Equity and access in algorithms, mechanisms, and optimization_ , pages 1–9. 2021.
  * Li et al. [2023] Hanzhou Li, John T Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy W Gichoya.  Ethics of large language models in medicine and medical research.  _The Lancet Digital Health_ , 5(6):e333–e335, 2023.
  * Pan and Yang [2009] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 2009.
  * Farahani et al. [2021] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  _Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020_ , pages 877–894, 2021.
  * Wang et al. [2020] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.  Generalizing from a few examples: A survey on few-shot learning.  _ACM Computing Surveys (CSUR)_ , 2020.
  * Settles [2009] Burr Settles.  Active learning literature survey.  Technical report, University of Wisconsin-Madison, 2009.
  * van Engelen and Hoos [2019] Jesper E. van Engelen and Holger H. Hoos.  A survey on semi-supervised learning.  _Machine Learning_ , 109:373–440, 2019.
  * Chapelle et al. [2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors.  _Semi-Supervised Learning_.  The MIT Press, 2006.  ISBN 9780262033589.  URL http://dblp.uni-trier.de/db/books/collections/CSZ2006.html.
  * Mussmann and Liang [2018] Stephen Mussmann and Percy Liang.  On the relationship between data efficiency and error for uncertainty sampling.  In _International Conference on Machine Learning_ , pages 3674–3682. PMLR, 2018.
  * Houlsby et al. [2011] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.  Bayesian active learning for classification and preference learning.  _arXiv preprint arXiv:1112.5745_ , 2011.
  * Kirsch et al. [2019] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal.  Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Nguyen et al. [2022] Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier.  How to measure uncertainty in uncertainty sampling for active learning.  _Machine Learning_ , 111(1):89–122, 2022.
  * Liu et al. [2021] Xingyu Liu, Fan Zhang, Zengfu Hou, Lodhi Mian, Zhihui Wang, Jian Zhang, and Jinhui Tang.  Self-supervised learning: Generative or contrastive.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Seedat et al. [2022b] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar.  Dc-check: A data-centric ai checklist to guide the development of reliable machine learning systems.  _arXiv preprint arXiv:2211.05764_ , 2022b.
  * Liang et al. [2022] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou.  Advances, challenges and opportunities in creating data for trustworthy ai.  _Nature Machine Intelligence_ , 4(8):669–677, 2022.
  * Polyzotis and Zaharia [2021] Neoklis Polyzotis and Matei Zaharia.  What can data-centric ai learn from data and ml engineering?  _arXiv preprint arXiv:2112.06439_ , 2021.
  * Zha et al. [2023] Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu.  Data-centric artificial intelligence: A survey.  _arXiv preprint arXiv:2303.10158_ , 2023.
  * Seedat et al. [2023] Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela van der Schaar.  Triage: Characterizing and auditing training data for improved regression.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Dong et al. [2022] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ , 2022.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Fehrman et al. [2017] Elaine Fehrman, Awaz K Muhammad, Evgeny M Mirkes, Vincent Egan, and Alexander N Gorban.  The five factor model of personality and evaluation of drug consumption risk.  In _Data science: innovative developments in data analysis and clustering_ , pages 231–242. Springer, 2017.
  * Vanschoren et al. [2013] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo.  Openml: Networked science in machine learning.  _SIGKDD Explorations_ , 15(2):49–60, 2013.  doi:10.1145/2641190.2641198.  URL http://doi.acm.org/10.1145/2641190.2641198.
  * Qian et al. [2023] Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar.  Synthcity: facilitating innovative use cases of synthetic data in different data modalities, 2023.  URL https://arxiv.org/abs/2301.07573.
  * Lemaître et al. [2017] Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas.  Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning.  _Journal of Machine Learning Research_ , 18(17):1–5, 2017.  URL http://jmlr.org/papers/v18/16-365.html.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.

## Appendix: Curated LLM: Synergy of LLMs and Data Curation for tabular
augmentation in ultra low-data regimes

parttoc3

###  Appendix A Extended related work

This paper primarily engages with the work on data augmentation when we have
limited data, where our primary goal is synthetic data generation to augment
the dataset. Generating synthetic datasets not only helps improve downstream
performance, but it is also a flexible solution as it doesn’t tie the data
consumer to any particular downstream model. That said, beyond the major
difference of synthetic data generation, for completeness we contrast our
setting of learning from limited data with other seemingly similar settings
and highlight their differences.

Contrasting learning w/ limited data vs other settings. The challenge of
learning from limited data, while seemingly related to several other learning
paradigms, presents distinct differences and unique intricacies that warrant
dedicated study.

Transfer learning [Pan and Yang, 2009], domain adaptation [Farahani et al.,
2021], and few-shot learning [Wang et al., 2020] employ additional data
resources or rely on specific task-related assumptions to improve learning
performance. These methods exploit large labeled data from a source domain,
unlabeled data in a target domain, or leverage knowledge from related tasks
respectively. For example, Levin et al. [2022] and Jin and Ucar [2023] use
models trained on labeled data from a source domain, while Ruiz et al. [2023]
and Margeloiu et al. [2022] leverage knowledge-graphs. This is in contrast to
our setting, considered of learning with limited data, which must function
with whatever scarce labeled data it has, without making any assumptions about
the availability of additional data or tasks.

Active learning [Settles, 2009] and semi-supervised learning [van Engelen and
Hoos, 2019, Chapelle et al., 2006] also operate under the premise of having
access to plentiful unlabeled data and the capacity to interactively query
labels. However, in our setting, considered learning with limited data does
not inherently assume such capabilities, focusing instead on limited labeled
data only.

Furthermore, active learning primarily focuses on the iterative process of
selecting data samples that, when labeled, are expected to most significantly
improve the model’s performance. This selection is typically based on criteria
such as uncertainty sampling which focuses on epistemic uncertainty [Mussmann
and Liang, 2018, Houlsby et al., 2011, Kirsch et al., 2019, Nguyen et al.,
2022]. The primary objective is to minimize labeling effort while maximizing
the model’s learning efficiency. Additionally, active learning would aims to
label instances based on epistemic uncertainty where the model struggles to
make accurate predictions, yet the samples themselves are correct. In
contrast, CLLM leverage training dynamics based on aleatoric uncertainty and
confidence and is designed to discard samples that might jeopardize the
downstream accuracy. These samples can be considered to have inherent issues
or are erroneous, such as being ”mislabeled”. To summarize, in active
learning, epistemic uncertainty is used to identify data points that, if
labeled, would yield the most significant insights for model training. In our
approach, they serve to identify and exclude/filter data points that could
potentially deteriorate the model’s performance.

Self-supervised learning [Liu et al., 2021] leverages large amounts of
unlabeled data to learn useful representations for downstream tasks. However,
in our setting, considered learning with limited data does not inherently
assume such access to vast amounts of unlabeled data.

Data-centric AI. Ensuring high data quality is a critical but often overlooked
problem in ML, where the focus is optimizing models [Sambasivan et al., 2021].
Even when it is considered, the process of assessing datasets is adhoc or
artisanal [Seedat et al., 2022b]. However, the recent push of data-centric AI
[Liang et al., 2022, Polyzotis and Zaharia, 2021, Zha et al., 2023] aims to
develop systematic tools to curate existing datasets. Our work contributes to
this nascent body of work [Seedat et al., 2023] – presenting CLLM, which, to
the best of our knowledge, is the first systematic data-centric framework
looking at how we can tailor synthetic datasets) rather than real datasets) to
downstream task use with data curation.

Why Data Augmentation? Data augmentation is a flexible approach to address the
ultra low-data regime. An alternative might be to resort to a pretrained
black-box model for classification, which could be for example via in-context
learning for classification [Dong et al., 2022]. However, such a solution is
inadequate for several reasons, many of which would prevent real-world utility
(e.g. in LMICs):

$\blacktriangleright$ _Not economical over the long term:_ While using an LLM
like GPT for classification may seem attractive due to its few-shot
capabilities, it is likely not economically viable in real-world settings,
especially in LMICs. The reason is classifying each sample will incur a cost
to call the LLM, hence scales linearly with the number of test samples. Over
time, the cumulative cost of these calls will surpass the once-off fixed cost
associated with generating data. With data augmentation, once the dataset is
augmented, there are no additional deployment time costs associated with the
LLM. Indeed, the downstream models e.g. a random forest or XGBoost have
negligible inference costs.

$\blacktriangleright$ _Control, interpretability and auditability:_ Relying on
a large, pre-trained LLM as a black-box classifier raises several concerns.
(1) we have no control over our downstream classifier and its architecture,
(2) lack of interpretability and auditability of the LLM when issuing
predictions. In contrast, training a downstream model on augmented data
maintains the ability to understand and explain how the model is making
decisions (e.g. feature importance). This is especially crucial in contexts
where accountability, transparency, and validation of machine learning
processes are paramount.

$\blacktriangleright$ _Independence and self-sufficiency:_ Relying on third-
party services for continuous classification means being dependent on their
availability, pricing models, and potential changes in the LLM version. By
augmenting data and training a downstream classifier on the augmented dataset,
we ensure that there is no external dependencies such as increasing costs or
reduced performance with LLM version updates.

$\blacktriangleright$ _Hardware and financial constraints:_ Even if we opt for
an open-source LLM (e.g. Falcon [Penedo et al., 2023] or LLaMA-2 [Touvron et
al., 2023]), deploying and running it locally demands significant
computational resources. Typically, these models require GPUs with high
amounts of VRAM for optimal performance (e.g. needing around 40 GB hencing
requiring an A100 GPU for Falcon-40b and LLaMA-2 65B). Such high-end GPUs are
expensive, and are likely to be inaccessible in a LMIC setting. Furthermore,
renting hardware by the hour can quickly become prohibitively expensive. Data
augmentation, on the other hand, can often be performed on modest hardware,
and once the augmented dataset is created, many classifiers can be trained
without the need for high-end GPUs, making the entire process more financially
accessible.

In conclusion, while large language models offer vast knowledge, for low-data
settings in low-income countries, data augmentation provides a more cost-
effective, controllable, and interpretable solution for building robust
classifiers.

###  Appendix B Experimental details

We provide details on our datasets used, as well as, other experimental
specifics including: generation, curation, downstream model, prompt template.

####  B.1 Datasets

We summarize the different datasets we use in this paper in Table 5. The
datasets vary in number of samples, number of features and domain.

Table 5: Summary of the datasets used. * Denotes private/proprietary datasets.

Name |  $n$ samples |  $n$ features | Domain
---|---|---|---
Adult Income [Asuncion and Newman, 2007] | 30k | 12 | Finance
Compas [Angwin et al., 2016] | 5k | 13 | Criminal justice
Covid-19 [Baqui et al., 2020] | 7k | 29 | Healthcare/Medicine
CUTRACT Prostate [PCUK, 2019] | 2k | 12 | Healthcare/Medicine
Drug [Fehrman et al., 2017] | 2k | 27 | Healthcare/Medicine
MAGGIC [Pocock et al., 2013] | 41k | 29 | Healthcare/Medicine
SEER Prostate [Duggan et al., 2016] | 20k | 12 | Healthcare/Medicine

We detail the dataset splits used in Sec. 3.1. For each dataset and number of
samples $n\in\\{20,40,100,200\\}$, we sample a training set
$D_{\mathrm{train}}$ such that $|D_{\mathrm{train}}|=n$, and each target class
has the same number of samples. We then split the remaining samples into two
non-overlapping datasets, $D_{\mathrm{oracle}}$ and $D_{\mathrm{test}}$, which
have the same cardinality. This procedure is repeated $n_{\mathrm{seed}}=10$
times, thus leading to different training and test sets. Note that the
different generative models use the same $D_{\mathrm{train}}$ and
$D_{\mathrm{test}}$ for a given seed.

Motivation for the choice of datasets.

  1. 1.

Open-source: Adult, Drug and Compas are widely used open-source datasets used
in the tabular data literature. Adult and Drug are both UCI datasets that have
been used in many papers, while Compas is part of OpenML Vanschoren et al.
[2013]. Our reason for selecting them is that, despite them being open-source,
they are highly reflective of domains in which we might be unable to collect
many samples — hence in reality would often be in an ultra-low data regime.

  2. 2.

Private datasets: We wanted to disentangle the possible role of memorization
in the strong performance of the LLM. To ensure the datasets are not in the
LLMs training corpus, we selected 4 private medical datasets that need an
authorization process to access. Hence, these datasets would not be part of
the LLMs training corpus given their proprietary nature and hence would be
unseen to the LLM. While the private and unseen aspect was the main
motivation, we also wish to highlight that these are real-world medical
datasets. Consequently, this allows us to test a highly realistic problem
setting.

####  B.2 Data generation.

GPT-4 and GPT-3.5 We access GPT-4 [OpenAI, 2023] and GPT-3.5-Turbo [Brown et
al., 2020] through the API. We use a temperature of $0.9$.

GReaT. GReaT Borisov et al. [2023] is a generative model which fine-tunes an
LLM based on a training set. We use the implementation provided by authors.

Generative model based approaches. For the other baselines used in 3.1, we use
the library SynthCity [Qian et al., 2023], using the defaults. We detail each
next.

  * •

TVAE: this is a conditional Variational Auto Encoder (VAE) for tabular data
and is based on Xu et al. [2019]

  * •

CTGAN: A conditional generative adversarial network which can handle tabular
data and is based on Xu et al. [2019]

  * •

NFLOW: Normalizing Flows are generative models which produce tractable
distributions where both sampling and density evaluation can be efficient and
exact.

  * •

TabDDPM: A diffusion model that can be universally applied to any tabular
dataset, handles any type of feature and is based on Kotelnikov et al. [2022]

Traditional Data Augmentation. We use SMOTE [Chawla et al., 2002] which
augments data by considering nearest neighbors and performing linear
interpolations. We use the implementation provided by Lemaître et al. [2017],
and set the number of neighbors $k$ to $5$.

####  B.3 Data curation

##### Learning dynamics computation

We train an XGBoost with $100$ estimators on $D_{\mathrm{train}}$. We then
compute predictive confidence and aleatoric uncertainty for the samples in
$D_{\mathrm{syn}}$. The motivation for the choice of an XGBoost backbone is
that we cannot expect good performance by choosing “any” curation model, but
rather we require a curation model with enough capacity and generalization
properties — where boosting methods like XGBoost used in our work have shown
to achieve best performance on tabular data. This leads to our guideline for
the curation step: the model used for curation should be at least as flexible
as the model that the practitioner intends to use for the downstream task.

##### Learning dynamics thresholds

Recall that CLLM has two thresholds $\tau_{\mathrm{conf}}$ and
$\tau_{\mathrm{al}}$ on the predictive confidence and aleatoric uncertainty
respectively, as defined in 2.2. We set $\tau_{\mathrm{conf}}=0.2$, in order
to select high confidence samples. We adopt an adaptive threshold for
$\tau_{\mathrm{al}}$ based on the dataset, such that
$\tau_{\mathrm{al}}=0.75\cdot(\max(v_{al}(D_{\mathrm{syn}}))-\min(v_{al}(D_{\mathrm{syn}})))$.
Note that by definition $v_{al}(D_{\mathrm{syn}})$ is bounded between $0$ and
$0.25$.

##### Example of learning dynamics

We include examples of learning dynamics computed for $20$ samples in Fig. 7.

Figure 7: Learning dynamics computed for $20$ samples

####  B.4 Downstream task

We compute downstream performance in Sec. 3.1 using four different downstream
models: XGBoost, Random Forest, Decision tree, and Logistic Regression.

####  B.5 Prompt example

We include the template of the prompts used throughout the paper. We show how
we include (1) in-context examples (demonstrations), (2) contextual
information including dataset background and feature information and (3) the
instruction.

⬇

1 System role: ’You are a tabular synthetic data generation model.’

2

3 You are a synthetic data generator.

4 Your goal is to produce data which mirrors \

5 the given examples in causal structure and feature and label distributions \

6 but also produce as diverse samples as possible.

7

8 I will give you real examples first.

9

10 Context: Leverage your medical knowledge about covid and Brazil to generate
1000 realistic but diverse samples.

11

12 example data: {data}

13

14 The output should be a markdown code snippet formatted in the following
schema:

15

16 "Sex_male": string // feature column

17 "Age": string // feature column

18 "Age_40": string // feature column

19 "Age_40_50": string // feature column

20 "Age_50_60": string // feature column

21 "Age_60_70": string // feature column

22 "Age_70": string // feature column

23 "Fever": string // feature column

24 "Cough": string // feature column

25 "Sore_throat": string // feature column

26 "Shortness_of_breath": string // feature column

27 "Respiratory_discomfort": string // feature column

28 "SPO2": string // feature column

29 "Dihareea": string // feature column

30 "Vomitting": string // feature column

31 "Cardiovascular": string // feature column

32 "Asthma": string // feature column

33 "Diabetis": string // feature column

34 "Pulmonary": string // feature column

35 "Immunosuppresion": string // feature column

36 "Obesity": string // feature column

37 "Liver": string // feature column

38 "Neurologic": string // feature column

39 "Renal": string // feature column

40 "Branca": string // feature column

41 "Preta": string // feature column

42 "Amarela": string // feature column

43 "Parda": string // feature column

44 "Indigena": string // feature column

45 "is_dead": string // label if patient dead or not, is_dead

46

47 DO NOT COPY THE EXAMPLES but generate realistic but new and diverse samples
which have the correct label conditioned on the features.

Listing 1: Template of the prompt

###  Appendix C Additional results

####  C.1 Decoupling prior knowledge and data model

Two components can be attributed to the good performances of CLLM: the
background knowledge of the LLM, and its capacity to build a strong data
model. In this subsection, we provide insights to understand the effect of the
LLM’s background knowledge (e.g. prior).

Table 6: Downstream accuracy when varying the number of in-context samples in the prompt to generate the augmented datasets. In-context samples | Downstream accuracy
---|---
$n=1$ (Prior) | $70.20\pm 1.60$
$n=20$ | $73.87\pm 0.50$
$n=40$ | $73.95\pm 0.67$
$n=100$ | $74.71\pm 0.34$
$D_{\mathrm{oracle}}$ | $74.6\pm 0.15$

We considered the Covid dataset (private medical dataset, to avoid
memorization issues) and generated data with GPT-4 (same as Section 2.1). We
ablate the prompt used in our work (detailed in Appendix B.5), and solely
provide one in-context example in the prompt, in order to give the LLM the
minimal amount of information about the desired structure of the dataset. This
forces the LLM to rely on its own prior (background knowledge), and removes
the effect of in-context examples which could be used to build a data model.
We report the results for the prior and CLLM in Table 6.

From these results, we conclude that the LLM prior permits to obtain good
downstream performance, but is outperformed by $\mathcal{D_{\mathrm{oracle}}}$
by a margin of $4.4\%$. Hence, we cannot solely rely on the prior.
Furthermore, downstream performance increases as the number of in-context
samples increases. This shows it is important to include in-context samples if
we wish to obtain downstream performance close to
$\mathcal{D}_{\mathrm{oracle}}$, as the LLM can build a good data model. This
implies that while the LLM uses background knowledge of similar datasets, it
still needs in-context samples to refine its prior by creating a good data
model.

We then quantify and visualize the strength of the prior, by studying how much
the LLMs output distribution adapts to the in-context samples provided. We
evaluate data generated by the prior of the LLM ($n=1$), and for $n=20,40,100$
on the Covid dataset. In particular, we observe in Figure 8 that there is a
region in the oracle data which is not captured by the LLM’s prior output (the
left part of the leftmost blob, circled in blue in Figure 8). However, as the
number of in-context real examples increases in the prompt of the LLM, we
observe that this steers the LLM to generate data which covers this region.
This region is associated to the subgroup of people older than $87$ years old,
and having many severe comorbidities (e.g. Diabetes, Cardiovascular diseases)
and many respiratory symptoms. This subgroup, in the Oracle dataset,
represents less than $3.5\%$ of the data, and is completely ignored by the
GPT-4 prior. In particular, the prior defaults to more typical patients in the
range $70$-$80$ years old. On the contrary, as $n$ increases, the LLM is
guided by the in-context samples and generates samples from this subgroup,
which are ”rarer” or different from the general population.

This shows that the LLM captures the distinct features of this particular
region, and is not overwhelmed by the prior. Instead, the data in the form of
in-context samples adapts it, and aligns the augmented dataset with the
ground-truth distribution.

Figure 8: The data generated by the LLM captures the distinct features in
”atypical” regions of the Oracle manifold, as in-context samples are added to
the prompt. This shows that it is flexible enough to adapt its prior knowledge
to the nuances of the data. The group encircled in blue represents patients
who are $>88$ years old, representing around $3.5\%$ of the Oracle. This
illustrates the added in-context samples can successfully guide the LLM to
generate these rare samples.

####  C.2 Curation mechanism also improves open-source LLM generated data

In this subsection, we demonstrate the wide applicability of the CLLM
framework. Different factors may affect the choice of the LLM backbone, such
as operational costs, and the desired parameterization of the LLM (for quality
of generation). In addition to GPT-4 and GPT-3.5, we use the open source
models Mistral-7b Jiang et al. [2023] and LLaMa-13b (4-bit quantized) Touvron
et al. [2023] to generate augmented datasets. We then compute the downstream
performance when training a model on both the uncurated and curated data. As
can be seen in Table 7, downstream performance with uncurated is lower for
these two models compared to GPT-4 — which is expected given their
significantly smaller size (i.e. parameter count). However, the curation
mechanism, which is the key contribution of CLLM, almost always improves
downstream performance for all LLM backbones investigated. Overall, this
demonstrates the versatility and wide applicability of CLLM for tabular data
augmentation in low-data regimes.

Table 7: AUC averaged over 4 downstream models on $D_{\mathrm{test}}$ for
GPT-4, GPT-3.5, Mistral-7b and LLAMA-13b (Curated and Uncurated)

| Open AI | Open source
---|---|---
Dataset | GPT-4 | GPT-3.5 | Mistral-7b | LLAMA-13b
| Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 73.78 | 73.87 $\uparrow$ | 69.85 | 71.41 $\uparrow$ | 71.47 | 72.80 $\uparrow$ | 63.25 | 67.24 $\uparrow$
cutract (n=20) | 71.15 | 72.50 $\uparrow$ | 69.97 | 71.54 $\uparrow$ | 69.60 | 71.34 $\uparrow$ | 67.84 | 66.71
maggic (n=20) | 60.70 | 61.48 $\uparrow$ | 57.54 | 58.69 $\uparrow$ | 53.64 | 52.06 | 53.30 | 53.96 $\uparrow$
seer (n=20) | 84.53 | 84.82 $\uparrow$ | 83.34 | 83.71 $\uparrow$ | 83.60 | 85.18 $\uparrow$ | 80.94 | 82.84 $\uparrow$
compas (n=20) | 68.01 | 67.91 | 62.07 | 64.43 $\uparrow$ | 56.26 | 59.95 $\uparrow$ | 60.08 | 60.34 $\uparrow$
adult (n=20) | 50.39 | 71.48 $\uparrow$ | 49.23 | 72.37 $\uparrow$ | 47.68 | 65.84 $\uparrow$ | 48.82 | 66.00 $\uparrow$
drug (n=20) | 75.08 | 75.29 $\uparrow$ | 71.68 | 72.14 $\uparrow$ | 74.14 | 75.21 $\uparrow$ | 67.96 | 67.87
covid (n=40) | 73.40 | 73.95 $\uparrow$ | 70.42 | 71.93 $\uparrow$ | 69.61 | 71.47 $\uparrow$ | 61.32 | 65.57 $\uparrow$
cutract (n=40) | 69.87 | 71.72 $\uparrow$ | 68.47 | 69.56 $\uparrow$ | 68.74 | 72.36 $\uparrow$ | 64.34 | 67.46 $\uparrow$
maggic (n=40) | 59.29 | 60.77 $\uparrow$ | 57.50 | 59.15 $\uparrow$ | 52.43 | 53.79 $\uparrow$ | 52.61 | 53.45 $\uparrow$
seer (n=40) | 84.29 | 84.93 $\uparrow$ | 83.46 | 84.44 $\uparrow$ | 83.88 | 85.21 $\uparrow$ | 78.82 | 82.65 $\uparrow$
compas (n=40) | 67.57 | 67.85 $\uparrow$ | 61.34 | 62.84 $\uparrow$ | 57.07 | 60.87 $\uparrow$ | 59.52 | 61.46 $\uparrow$
adult (n=40) | 48.31 | 73.82 $\uparrow$ | 49.21 | 74.27 $\uparrow$ | 48.80 | 74.13 $\uparrow$ | 54.34 | 69.31 $\uparrow$
drug (n=40) | 74.30 | 75.79 $\uparrow$ | 71.33 | 72.76 $\uparrow$ | 73.12 | 74.12 $\uparrow$ | 69.84 | 72.50 $\uparrow$
covid (n=100) | 73.77 | 74.71 $\uparrow$ | 70.71 | 72.76 $\uparrow$ | 71.02 | 73.37 $\uparrow$ | 63.76 | 70.68 $\uparrow$
cutract (n=100) | 70.20 | 72.51 $\uparrow$ | 69.97 | 71.94 $\uparrow$ | 68.92 | 71.17 $\uparrow$ | 64.81 | 69.85 $\uparrow$
maggic (n=100) | 58.98 | 61.32 $\uparrow$ | 55.71 | 58.90 $\uparrow$ | 52.53 | 53.36 $\uparrow$ | 54.27 | 53.65
seer (n=100) | 84.45 | 85.37 $\uparrow$ | 83.92 | 85.08 $\uparrow$ | 82.23 | 84.36 $\uparrow$ | 81.00 | 81.99 $\uparrow$
compas (n=100) | 68.02 | 68.19 $\uparrow$ | 60.10 | 62.47 $\uparrow$ | 53.74 | 61.28 $\uparrow$ | 57.90 | 61.79 $\uparrow$
adult (n=100) | 46.09 | 74.57 $\uparrow$ | 47.56 | 73.97 $\uparrow$ | 40.51 | 71.08 $\uparrow$ | 48.85 | 72.91 $\uparrow$
drug (n=100) | 76.24 | 76.74 $\uparrow$ | 69.46 | 71.05 $\uparrow$ | 74.02 | 76.55 $\uparrow$ | 66.86 | 75.31 $\uparrow$
covid (n=200) | 73.40 | 74.62 $\uparrow$ | 70.70 | 73.12 $\uparrow$ | 70.81 | 73.26 $\uparrow$ | 62.53 | 70.67 $\uparrow$
cutract (n=200) | 71.39 | 73.01 $\uparrow$ | 70.28 | 72.39 $\uparrow$ | 67.69 | 70.29 $\uparrow$ | 65.77 | 69.11 $\uparrow$
maggic (n=200) | 58.92 | 61.41 $\uparrow$ | 57.33 | 60.16 $\uparrow$ | 52.78 | 52.40 | 52.56 | 52.90 $\uparrow$
seer (n=200) | 84.39 | 85.56 $\uparrow$ | 83.48 | 84.80 $\uparrow$ | 83.15 | 84.18 $\uparrow$ | 80.88 | 82.74 $\uparrow$
compas (n=200) | 67.02 | 68.15 $\uparrow$ | 60.48 | 63.39 $\uparrow$ | 53.96 | 59.22 $\uparrow$ | 57.56 | 59.97 $\uparrow$
adult (n=200) | 40.96 | 75.84 $\uparrow$ | 49.89 | 76.11 $\uparrow$ | 44.13 | 74.23 $\uparrow$ | 48.25 | 78.35 $\uparrow$
drug (n=200) | 75.58 | 76.06 $\uparrow$ | 70.66 | 72.81 $\uparrow$ | 71.89 | 76.54 $\uparrow$ | 70.88 | 75.46 $\uparrow$

####  C.3 Ablation for contextual information on Compas

We conduct a similar experiment as in Table 2.1, and use the dataset Compas.
We report the results in Table C.3.

Table 8: Including contextual information in the prompt improves precision
(P), recall (R), and utility (U) in low-sample settings (results shown for
Compas).

$n_{\mathrm{samples}}$ in $D_{\mathrm{train}}$ |  GPT-4 w/ context |  GPT-4 no context | TVAE
---|---|---|---
| P | R | U | P | R | U | P | R | U
20 | $\bf 0.69_{(0.02)}$ | $\bf 0.88_{(0.02)}$ | $\bf 0.69_{(0.02)}$ | $0.27_{(0.03)}$ | $\bf 0.89_{(0.03)}$ | $0.60_{(0.03)}$ | $0.43_{(0.02)}$ | $0.43_{(0.05)}$ | $0.55_{(0.04)}$
40 | $\bf 0.70_{(0.0)}$ | $\bf 0.92_{(0.01)}$ | $\bf 0.65_{(0.03)}$ | $0.31_{(0.06)}$ | $0.84_{(0.03)}$ | $0.57_{(0.01)}$ | $0.54_{(0.02)}$ | $0.80_{(0.02)}$ | $0.50_{(0.04)}$
100 | $\bf 0.69_{(0.02)}$ | $\bf 0.89_{(0.02)}$ | $\bf 0.69_{(0.01)}$ | $0.34_{(0.1)}$ | $0.85_{(0.05)}$ | $0.62_{(0.01)}$ | $0.60_{(0.03)}$ | $0.86_{(0.02)}$ | $0.59_{(0.03)}$
200 | $\bf 0.70_{(0.01)}$ | $\bf 0.89_{(0.02)}$ | $\bf 0.69_{(0.01)}$ | $0.31_{(0.05)}$ | $0.87_{(0.03)}$ | $0.58_{(0.05)}$ | $0.65_{(0.02)}$ | $0.88_{(0.01)}$ | $0.63_{(0.01)}$

These results highlight the importance of incorporating contextual information
in the prompt, as it enables to exploit the prior knowledge of the LLM.

####  C.4 Comparison to random noise baseline

We compare CLLM to a random noise baseline, where we augment the dataset with
random additive Gaussian noise. In order to capture the correlations between
the different features, we fit a Kernel Density Estimator with a Gaussian
kernel and bandwidth given by Scott’s rule. We then sample $1000$ points to
create an augmented dataset $D_{\mathrm{syn}}$. We report the performance gap
between CLLM and this baseline (with and without curation) for the Covid and
Compas datasets in Figure 9. We observe that the random noise baseline does
not match the performance of CLLM (i.e. has a performance gap), although the
baseline naturally improves as the dataset $D_{\mathrm{train}}$ grows in size.

Figure 9: The random noise baseline does not match the performance of CLLM

####  C.5 Detailed results for Section 3.1

We report additional results for Sec. 3.1, showing the AUC for each downstream
model (XGBoost, Random forest, Logistic regression, Decision tree). As we can
see, the conclusion that curation helps improve downstream performance holds
for each of these various downstream models, as is indicated by the green
arrows in Tables 9, 10, 11, 12.

Table 9: AUC for the RF model on $D_{\mathrm{test}}$ where curation improves
performance for all methods across all sample sizes $n$, as indicated by
$\uparrow$.

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 76.11 | 72.52 | 75.67 | 75.59 | 72.37 | 73.32 $\uparrow$ | 61.66 | 65.24 $\uparrow$ | 70.22 | 70.13 | 57.78 | 68.24 $\uparrow$ | 65.54 | 70.72 $\uparrow$ | 71.40 | 71.42 $\uparrow$ | 63.90 | 67.69 $\uparrow$
cutract (n=20) | 74.16 | 73.34 | 73.45 | 74.24 $\uparrow$ | 72.22 | 73.51 $\uparrow$ | 66.24 | 70.45 $\uparrow$ | 69.73 | 70.49 $\uparrow$ | 52.61 | 68.11 $\uparrow$ | 66.63 | 72.39 $\uparrow$ | 71.42 | 72.34 $\uparrow$ | 71.18 | 71.89 $\uparrow$
maggic (n=20) | 71.28 | 59.18 | 62.92 | 63.97 $\uparrow$ | 60.10 | 61.25 $\uparrow$ | 53.21 | 55.14 $\uparrow$ | 55.61 | 56.50 $\uparrow$ | 50.92 | 57.23 $\uparrow$ | 55.91 | 59.07 $\uparrow$ | 58.20 | 58.55 $\uparrow$ | 55.43 | 57.71 $\uparrow$
seer (n=20) | 90.09 | 85.22 | 86.30 | 86.82 $\uparrow$ | 85.86 | 85.73 | 78.11 | 80.52 $\uparrow$ | 82.61 | 82.74 $\uparrow$ | 47.57 | 75.70 $\uparrow$ | 77.35 | 81.45 $\uparrow$ | 82.09 | 83.00 $\uparrow$ | 77.63 | 81.16 $\uparrow$
compas (n=20) | 66.79 | 64.47 | 68.40 | 67.85 | 61.93 | 64.29 $\uparrow$ | 56.87 | 63.21 $\uparrow$ | 58.56 | 61.26 $\uparrow$ | 52.23 | 63.30 $\uparrow$ | 59.60 | 64.79 $\uparrow$ | 61.05 | 62.18 $\uparrow$ | 59.31 | 63.37 $\uparrow$
adult (n=20) | 85.83 | 83.31 | 51.35 | 73.45 $\uparrow$ | 49.17 | 74.27 $\uparrow$ | 75.63 | 78.99 $\uparrow$ | 77.27 | 77.64 $\uparrow$ | 69.35 | 78.79 $\uparrow$ | 68.32 | 78.27 $\uparrow$ | 77.75 | 78.34 $\uparrow$ | 76.13 | 79.02 $\uparrow$
drug (n=20) | 83.12 | 77.07 | 79.23 | 78.90 | 77.11 | 76.95 | 72.81 | 77.24 $\uparrow$ | 72.00 | 74.03 $\uparrow$ | 63.34 | 73.29 $\uparrow$ | 65.47 | 70.19 $\uparrow$ | 75.48 | 75.36 | 72.22 | 74.32 $\uparrow$
covid (n=40) | 76.11 | 75.21 | 75.63 | 75.70 $\uparrow$ | 72.27 | 73.59 $\uparrow$ | 66.64 | 70.75 $\uparrow$ | 75.56 | 75.39 | 56.91 | 70.88 $\uparrow$ | 70.47 | 73.15 $\uparrow$ | 73.59 | 73.63 $\uparrow$ | 64.27 | 69.71 $\uparrow$
cutract (n=40) | 74.16 | 72.31 | 71.67 | 73.20 $\uparrow$ | 69.92 | 71.01 $\uparrow$ | 64.67 | 69.21 $\uparrow$ | 69.16 | 69.67 $\uparrow$ | 53.41 | 68.93 $\uparrow$ | 61.23 | 69.21 $\uparrow$ | 70.82 | 71.21 $\uparrow$ | 59.99 | 67.56 $\uparrow$
maggic (n=40) | 71.28 | 60.91 | 61.80 | 63.11 $\uparrow$ | 59.38 | 61.50 $\uparrow$ | 56.49 | 58.17 $\uparrow$ | 56.50 | 58.21 $\uparrow$ | 48.43 | 57.84 $\uparrow$ | 55.47 | 59.94 $\uparrow$ | 59.82 | 60.11 $\uparrow$ | 56.76 | 58.92 $\uparrow$
seer (n=40) | 90.09 | 86.96 | 86.01 | 86.29 $\uparrow$ | 86.73 | 87.09 $\uparrow$ | 83.46 | 86.62 $\uparrow$ | 85.86 | 85.19 | 54.71 | 83.22 $\uparrow$ | 83.23 | 86.81 $\uparrow$ | 83.76 | 84.82 $\uparrow$ | 80.34 | 85.82 $\uparrow$
compas (n=40) | 66.79 | 62.73 | 68.06 | 68.13 $\uparrow$ | 61.17 | 62.27 $\uparrow$ | 56.05 | 60.92 $\uparrow$ | 59.76 | 61.03 $\uparrow$ | 57.82 | 64.25 $\uparrow$ | 58.89 | 63.79 $\uparrow$ | 61.20 | 61.25 $\uparrow$ | 55.89 | 60.54 $\uparrow$
adult (n=40) | 85.83 | 83.61 | 50.26 | 75.55 $\uparrow$ | 46.68 | 76.63 $\uparrow$ | 75.13 | 81.56 $\uparrow$ | 76.10 | 80.40 $\uparrow$ | 68.04 | 81.08 $\uparrow$ | 73.29 | 81.46 $\uparrow$ | 80.35 | 81.10 $\uparrow$ | 75.81 | 82.11 $\uparrow$
drug (n=40) | 83.12 | 78.81 | 78.35 | 79.54 $\uparrow$ | 77.33 | 77.94 $\uparrow$ | 74.68 | 76.64 $\uparrow$ | 74.50 | 77.18 $\uparrow$ | 70.25 | 77.77 $\uparrow$ | 65.86 | 74.55 $\uparrow$ | 74.84 | 75.26 $\uparrow$ | 71.29 | 75.98 $\uparrow$
covid (n=100) | 76.11 | 75.78 | 75.86 | 76.33 $\uparrow$ | 73.02 | 74.40 $\uparrow$ | 72.00 | 74.74 $\uparrow$ | 74.64 | 75.74 $\uparrow$ | 65.76 | 74.69 $\uparrow$ | 67.05 | 75.47 $\uparrow$ | 74.00 | 74.09 $\uparrow$ | 72.11 | 74.44 $\uparrow$
cutract (n=100) | 74.16 | 73.93 | 72.92 | 74.73 $\uparrow$ | 72.51 | 73.93 $\uparrow$ | 70.18 | 74.26 $\uparrow$ | 71.71 | 73.69 $\uparrow$ | 55.48 | 70.79 $\uparrow$ | 69.79 | 74.75 $\uparrow$ | 71.18 | 72.47 $\uparrow$ | 68.47 | 73.78 $\uparrow$
maggic (n=100) | 71.28 | 63.06 | 60.99 | 63.36 $\uparrow$ | 57.97 | 60.86 $\uparrow$ | 58.98 | 60.76 $\uparrow$ | 58.88 | 60.13 $\uparrow$ | 49.66 | 59.57 $\uparrow$ | 57.82 | 62.24 $\uparrow$ | 61.55 | 61.90 $\uparrow$ | 57.46 | 60.55 $\uparrow$
seer (n=100) | 90.00 | 87.53 | 86.33 | 87.31 $\uparrow$ | 86.40 | 87.06 $\uparrow$ | 84.59 | 87.27 $\uparrow$ | 85.89 | 87.00 $\uparrow$ | 70.32 | 85.97 $\uparrow$ | 84.30 | 87.52 $\uparrow$ | 85.12 | 85.82 $\uparrow$ | 81.92 | 86.41 $\uparrow$
compas (n=100) | 66.79 | 63.18 | 68.44 | 68.67 $\uparrow$ | 59.41 | 62.00 $\uparrow$ | 60.28 | 64.34 $\uparrow$ | 60.17 | 62.63 $\uparrow$ | 59.32 | 63.98 $\uparrow$ | 60.10 | 65.19 $\uparrow$ | 61.34 | 61.37 $\uparrow$ | 59.58 | 63.37 $\uparrow$
adult (n=100) | 85.83 | 84.38 | 46.37 | 76.29 $\uparrow$ | 47.30 | 75.74 $\uparrow$ | 77.45 | 82.63 $\uparrow$ | 80.88 | 81.69 $\uparrow$ | 79.23 | 83.33 $\uparrow$ | 73.52 | 83.38 $\uparrow$ | 82.54 | 83.05 $\uparrow$ | 76.60 | 82.28 $\uparrow$
drug (n=100) | 83.12 | 80.31 | 79.29 | 79.75 $\uparrow$ | 74.55 | 76.46 $\uparrow$ | 74.54 | 78.01 $\uparrow$ | 78.34 | 79.88 $\uparrow$ | 73.19 | 79.72 $\uparrow$ | 67.01 | 76.46 $\uparrow$ | 76.92 | 77.57 $\uparrow$ | 74.53 | 78.82 $\uparrow$
covid (n=200) | 76.11 | 76.08 | 75.23 | 75.84 $\uparrow$ | 72.62 | 74.99 $\uparrow$ | 74.04 | 76.25 $\uparrow$ | 75.28 | 76.82 $\uparrow$ | 67.08 | 75.62 $\uparrow$ | 67.73 | 75.67 $\uparrow$ | 74.82 | 75.27 $\uparrow$ | 70.64 | 75.03 $\uparrow$
cutract (n=200) | 74.16 | 74.25 | 73.48 | 75.28 $\uparrow$ | 72.01 | 74.37 $\uparrow$ | 71.99 | 74.93 $\uparrow$ | 74.33 | 76.19 $\uparrow$ | 68.05 | 74.68 $\uparrow$ | 71.47 | 75.67 $\uparrow$ | 72.44 | 72.85 $\uparrow$ | 68.55 | 74.43 $\uparrow$
maggic (n=200) | 71.28 | 64.77 | 61.56 | 63.78 $\uparrow$ | 59.76 | 62.53 $\uparrow$ | 60.85 | 63.57 $\uparrow$ | 58.23 | 58.85 $\uparrow$ | 50.84 | 61.49 $\uparrow$ | 57.52 | 63.00 $\uparrow$ | 63.72 | 64.01 $\uparrow$ | 58.92 | 62.08 $\uparrow$
seer (n=200) | 90.09 | 88.13 | 86.96 | 87.72 $\uparrow$ | 85.28 | 86.70 $\uparrow$ | 85.05 | 87.83 $\uparrow$ | 87.14 | 88.80 $\uparrow$ | 84.86 | 88.14 $\uparrow$ | 80.80 | 87.46 $\uparrow$ | 86.57 | 87.11 $\uparrow$ | 82.30 | 87.59 $\uparrow$
compas (n=200) | 66.79 | 63.61 | 67.68 | 68.80 $\uparrow$ | 60.67 | 63.43 $\uparrow$ | 61.20 | 65.11 $\uparrow$ | 60.88 | 63.87 $\uparrow$ | 60.10 | 64.91 $\uparrow$ | 56.94 | 64.76 $\uparrow$ | 61.57 | 62.76 $\uparrow$ | 59.95 | 64.14 $\uparrow$
adult (n=200) | 85.83 | 85.16 | 40.18 | 77.61 $\uparrow$ | 46.74 | 78.71 $\uparrow$ | 80.92 | 84.26 $\uparrow$ | 84.50 | 85.44 $\uparrow$ | 82.73 | 85.05 $\uparrow$ | 77.48 | 84.72 $\uparrow$ | 84.26 | 84.33 $\uparrow$ | 78.12 | 83.94 $\uparrow$
drug (n=200) | 83.12 | 81.47 | 78.70 | 78.98 $\uparrow$ | 75.87 | 77.59 $\uparrow$ | 77.19 | 80.03 $\uparrow$ | 73.25 | 75.75 $\uparrow$ | 78.50 | 81.13 $\uparrow$ | 67.83 | 78.81 $\uparrow$ | 79.50 | 80.16 $\uparrow$ | 74.81 | 79.45 $\uparrow$

Table 10: AUC for the XGB model on $D_{\mathrm{test}}$ where curation improves
performance for all methods across all sample sizes $n$, as indicated by
$\uparrow$.

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 77.59 | 69.70 | 75.59 | 75.49 | 71.27 | 72.25 $\uparrow$ | 61.40 | 65.96 $\uparrow$ | 69.39 | 68.21 | 55.99 | 67.06 $\uparrow$ | 62.28 | 69.67 $\uparrow$ | 68.49 | 68.43 | 62.49 | 67.16 $\uparrow$
cutract (n=20) | 73.14 | 71.04 | 73.26 | 74.30 $\uparrow$ | 71.62 | 73.04 $\uparrow$ | 65.13 | 69.67 $\uparrow$ | 68.46 | 68.87 $\uparrow$ | 51.99 | 68.31 $\uparrow$ | 62.93 | 70.87 $\uparrow$ | 68.28 | 70.72 $\uparrow$ | 69.96 | 71.98 $\uparrow$
maggic (n=20) | 71.38 | 57.98 | 61.54 | 62.81 $\uparrow$ | 58.76 | 60.01 $\uparrow$ | 53.62 | 55.44 $\uparrow$ | 55.97 | 57.06 $\uparrow$ | 50.14 | 56.96 $\uparrow$ | 54.91 | 58.22 $\uparrow$ | 56.79 | 57.15 $\uparrow$ | 55.41 | 57.72 $\uparrow$
seer (n=20) | 91.80 | 80.41 | 87.00 | 86.56 | 85.67 | 85.42 | 76.84 | 80.00 $\uparrow$ | 81.79 | 80.21 | 45.06 | 75.39 $\uparrow$ | 76.25 | 81.45 $\uparrow$ | 78.57 | 79.78 $\uparrow$ | 75.69 | 81.01 $\uparrow$
compas (n=20) | 69.17 | 65.49 | 69.71 | 68.79 | 62.55 | 64.68 $\uparrow$ | 56.34 | 63.92 $\uparrow$ | 58.19 | 61.42 $\uparrow$ | 51.73 | 63.51 $\uparrow$ | 59.01 | 64.84 $\uparrow$ | 62.35 | 63.46 $\uparrow$ | 59.18 | 63.62 $\uparrow$
adult (n=20) | 88.64 | 79.21 | 52.12 | 74.28 $\uparrow$ | 47.70 | 72.91 $\uparrow$ | 71.70 | 78.26 $\uparrow$ | 76.50 | 77.22 $\uparrow$ | 67.79 | 78.67 $\uparrow$ | 66.79 | 77.77 $\uparrow$ | 77.11 | 77.97 $\uparrow$ | 75.46 | 79.35 $\uparrow$
drug (n=20) | 81.06 | 69.26 | 78.17 | 77.28 | 75.23 | 75.71 $\uparrow$ | 70.60 | 74.31 $\uparrow$ | 70.29 | 71.25 $\uparrow$ | 57.18 | 69.67 $\uparrow$ | 62.96 | 69.12 $\uparrow$ | 70.84 | 72.30 $\uparrow$ | 69.78 | 72.45 $\uparrow$
covid (n=40) | 77.59 | 71.57 | 75.23 | 75.36 $\uparrow$ | 71.55 | 73.37 $\uparrow$ | 66.17 | 70.85 $\uparrow$ | 72.21 | 72.68 $\uparrow$ | 56.03 | 70.80 $\uparrow$ | 67.39 | 71.82 $\uparrow$ | 70.02 | 69.96 | 61.19 | 69.77 $\uparrow$
cutract (n=40) | 73.14 | 69.50 | 71.26 | 73.57 $\uparrow$ | 69.66 | 71.34 $\uparrow$ | 62.52 | 68.69 $\uparrow$ | 64.50 | 68.05 $\uparrow$ | 51.23 | 68.23 $\uparrow$ | 60.74 | 68.91 $\uparrow$ | 69.42 | 69.19 | 59.34 | 67.80 $\uparrow$
maggic (n=40) | 71.38 | 58.76 | 60.27 | 61.88 $\uparrow$ | 57.87 | 60.29 $\uparrow$ | 55.87 | 58.47 $\uparrow$ | 55.38 | 58.03 $\uparrow$ | 48.50 | 57.02 $\uparrow$ | 54.53 | 59.01 $\uparrow$ | 58.12 | 58.29 $\uparrow$ | 56.19 | 58.33 $\uparrow$
seer (n=40) | 91.80 | 84.51 | 86.63 | 87.40 $\uparrow$ | 86.27 | 86.80 $\uparrow$ | 82.48 | 86.55 $\uparrow$ | 81.62 | 81.58 | 52.80 | 83.03 $\uparrow$ | 80.04 | 85.84 $\uparrow$ | 82.53 | 83.62 $\uparrow$ | 79.35 | 85.81 $\uparrow$
compas (n=40) | 69.17 | 62.90 | 69.24 | 68.90 | 61.74 | 63.21 $\uparrow$ | 56.63 | 62.55 $\uparrow$ | 59.07 | 59.38 $\uparrow$ | 58.01 | 64.95 $\uparrow$ | 58.44 | 64.44 $\uparrow$ | 61.81 | 61.89 $\uparrow$ | 56.60 | 62.26 $\uparrow$
adult (n=40) | 88.64 | 82.16 | 48.51 | 76.59 $\uparrow$ | 45.92 | 76.92 $\uparrow$ | 73.38 | 82.53 $\uparrow$ | 74.03 | 79.84 $\uparrow$ | 66.60 | 80.98 $\uparrow$ | 69.47 | 81.56 $\uparrow$ | 80.01 | 80.51 $\uparrow$ | 74.20 | 82.53 $\uparrow$
drug (n=40) | 81.06 | 72.72 | 77.03 | 78.85 $\uparrow$ | 76.37 | 76.48 $\uparrow$ | 70.99 | 74.48 $\uparrow$ | 74.65 | 76.53 $\uparrow$ | 64.52 | 75.09 $\uparrow$ | 62.58 | 73.14 $\uparrow$ | 71.80 | 72.15 $\uparrow$ | 67.61 | 74.04 $\uparrow$
covid (n=100) | 77.59 | 73.16 | 75.55 | 76.56 $\uparrow$ | 72.69 | 74.36 $\uparrow$ | 70.54 | 73.99 $\uparrow$ | 72.99 | 74.50 $\uparrow$ | 63.40 | 73.98 $\uparrow$ | 63.50 | 73.79 $\uparrow$ | 72.64 | 73.35 $\uparrow$ | 70.45 | 73.76 $\uparrow$
cutract (n=100) | 73.14 | 70.49 | 72.29 | 74.01 $\uparrow$ | 71.51 | 73.48 $\uparrow$ | 68.64 | 73.63 $\uparrow$ | 71.32 | 74.02 $\uparrow$ | 54.74 | 69.69 $\uparrow$ | 66.97 | 72.98 $\uparrow$ | 67.71 | 69.42 $\uparrow$ | 66.44 | 72.32 $\uparrow$
maggic (n=100) | 71.38 | 60.55 | 60.04 | 62.57 $\uparrow$ | 56.98 | 60.49 $\uparrow$ | 58.21 | 60.42 $\uparrow$ | 58.11 | 59.42 $\uparrow$ | 49.71 | 58.77 $\uparrow$ | 55.36 | 60.58 $\uparrow$ | 60.02 | 60.15 $\uparrow$ | 56.47 | 60.28 $\uparrow$
seer (n=100) | 91.80 | 85.52 | 86.81 | 87.64 $\uparrow$ | 86.39 | 87.21 $\uparrow$ | 83.74 | 87.29 $\uparrow$ | 84.66 | 86.85 $\uparrow$ | 68.20 | 85.49 $\uparrow$ | 82.37 | 86.80 $\uparrow$ | 83.18 | 83.87 $\uparrow$ | 80.45 | 86.49 $\uparrow$
compas (n=100) | 69.17 | 62.71 | 69.37 | 69.37 | 59.83 | 62.80 $\uparrow$ | 60.56 | 64.90 $\uparrow$ | 58.20 | 62.00 $\uparrow$ | 58.51 | 64.60 $\uparrow$ | 59.76 | 64.68 $\uparrow$ | 61.33 | 62.03 $\uparrow$ | 60.37 | 63.42 $\uparrow$
adult (n=100) | 88.64 | 82.78 | 44.60 | 76.75 $\uparrow$ | 46.34 | 76.83 $\uparrow$ | 74.97 | 82.89 $\uparrow$ | 77.38 | 81.42 $\uparrow$ | 78.84 | 83.96 $\uparrow$ | 68.31 | 82.87 $\uparrow$ | 82.58 | 83.24 $\uparrow$ | 76.15 | 82.47 $\uparrow$
drug (n=100) | 81.06 | 75.49 | 79.12 | 79.81 $\uparrow$ | 72.52 | 74.78 $\uparrow$ | 68.89 | 75.91 $\uparrow$ | 76.88 | 78.33 $\uparrow$ | 66.38 | 77.02 $\uparrow$ | 62.27 | 75.13 $\uparrow$ | 74.40 | 74.49 $\uparrow$ | 71.02 | 76.85 $\uparrow$
covid (n=200) | 77.59 | 73.06 | 75.12 | 76.64 $\uparrow$ | 71.74 | 74.66 $\uparrow$ | 72.57 | 75.50 $\uparrow$ | 73.56 | 75.88 $\uparrow$ | 66.02 | 74.51 $\uparrow$ | 63.29 | 74.62 $\uparrow$ | 73.34 | 73.89 $\uparrow$ | 68.14 | 74.21 $\uparrow$
cutract (n=200) | 73.14 | 71.68 | 73.43 | 75.14 $\uparrow$ | 71.76 | 74.31 $\uparrow$ | 69.35 | 73.80 $\uparrow$ | 72.76 | 75.33 $\uparrow$ | 65.83 | 73.43 $\uparrow$ | 68.05 | 74.27 $\uparrow$ | 70.06 | 71.35 $\uparrow$ | 66.65 | 73.34 $\uparrow$
maggic (n=200) | 71.38 | 62.40 | 59.46 | 63.15 $\uparrow$ | 58.05 | 61.68 $\uparrow$ | 59.04 | 62.58 $\uparrow$ | 57.63 | 59.23 $\uparrow$ | 50.63 | 60.09 $\uparrow$ | 55.55 | 61.36 $\uparrow$ | 61.73 | 61.88 $\uparrow$ | 57.09 | 61.15 $\uparrow$
seer (n=200) | 91.80 | 85.87 | 86.75 | 88.08 $\uparrow$ | 85.74 | 86.97 $\uparrow$ | 83.97 | 87.64 $\uparrow$ | 86.21 | 88.46 $\uparrow$ | 83.21 | 87.65 $\uparrow$ | 76.91 | 87.16 $\uparrow$ | 84.70 | 85.59 $\uparrow$ | 81.45 | 87.46 $\uparrow$
compas (n=200) | 69.17 | 63.14 | 68.80 | 69.54 $\uparrow$ | 61.62 | 64.59 $\uparrow$ | 60.65 | 65.67 $\uparrow$ | 60.97 | 65.12 $\uparrow$ | 60.66 | 65.67 $\uparrow$ | 56.49 | 64.34 $\uparrow$ | 61.68 | 62.93 $\uparrow$ | 60.32 | 64.18 $\uparrow$
adult (n=200) | 88.64 | 84.12 | 40.38 | 77.39 $\uparrow$ | 47.40 | 79.40 $\uparrow$ | 80.00 | 84.61 $\uparrow$ | 83.73 | 85.73 $\uparrow$ | 81.92 | 85.76 $\uparrow$ | 74.90 | 85.13 $\uparrow$ | 84.78 | 84.45 | 75.54 | 84.39 $\uparrow$
drug (n=200) | 81.06 | 78.41 | 78.25 | 79.09 $\uparrow$ | 73.31 | 76.76 $\uparrow$ | 72.82 | 79.00 $\uparrow$ | 72.62 | 74.89 $\uparrow$ | 74.30 | 79.75 $\uparrow$ | 63.91 | 77.60 $\uparrow$ | 77.64 | 77.92 $\uparrow$ | 70.16 | 77.89 $\uparrow$

Table 11: AUC for the DT model on $D_{\mathrm{test}}$ where curation improve
performance for all methods across all sample sizes $n$, as indicated by
$\uparrow$.

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 63.46 | 61.92 | 67.52 | 67.67 $\uparrow$ | 61.07 | 64.89 $\uparrow$ | 52.94 | 56.62 $\uparrow$ | 58.95 | 59.19 $\uparrow$ | 51.83 | 58.73 $\uparrow$ | 54.37 | 61.53 $\uparrow$ | 59.56 | 59.01 | 56.46 | 59.49 $\uparrow$
cutract (n=20) | 62.49 | 64.62 | 62.40 | 66.03 $\uparrow$ | 62.10 | 65.15 $\uparrow$ | 55.67 | 61.73 $\uparrow$ | 57.91 | 58.97 $\uparrow$ | 51.89 | 62.76 $\uparrow$ | 57.12 | 64.85 $\uparrow$ | 62.85 | 62.38 | 60.25 | 62.34 $\uparrow$
maggic (n=20) | 56.53 | 54.80 | 54.67 | 55.03 $\uparrow$ | 52.54 | 53.54 $\uparrow$ | 50.90 | 52.08 $\uparrow$ | 52.39 | 52.70 $\uparrow$ | 49.70 | 53.08 $\uparrow$ | 51.52 | 53.23 $\uparrow$ | 52.55 | 52.44 | 50.91 | 51.52 $\uparrow$
seer (n=20) | 78.86 | 71.09 | 76.17 | 77.17 $\uparrow$ | 73.92 | 75.58 $\uparrow$ | 64.18 | 70.69 $\uparrow$ | 74.78 | 74.72 | 46.60 | 68.79 $\uparrow$ | 65.77 | 72.25 $\uparrow$ | 70.54 | 71.19 $\uparrow$ | 64.47 | 70.11 $\uparrow$
compas (n=20) | 61.05 | 58.73 | 62.07 | 63.91 $\uparrow$ | 57.17 | 60.85 $\uparrow$ | 53.83 | 58.83 $\uparrow$ | 51.77 | 56.11 $\uparrow$ | 52.39 | 60.96 $\uparrow$ | 54.64 | 60.27 $\uparrow$ | 58.16 | 57.57 | 53.52 | 58.97 $\uparrow$
adult (n=20) | 73.84 | 68.03 | 52.80 | 67.10 $\uparrow$ | 49.15 | 66.52 $\uparrow$ | 61.57 | 68.93 $\uparrow$ | 70.46 | 71.19 $\uparrow$ | 62.04 | 71.87 $\uparrow$ | 59.07 | 69.16 $\uparrow$ | 65.52 | 63.33 | 64.40 | 70.41 $\uparrow$
drug (n=20) | 65.66 | 63.49 | 64.40 | 66.87 $\uparrow$ | 62.69 | 64.21 $\uparrow$ | 56.76 | 64.18 $\uparrow$ | 61.04 | 62.48 $\uparrow$ | 52.22 | 62.62 $\uparrow$ | 53.50 | 61.30 $\uparrow$ | 62.34 | 60.69 | 55.00 | 58.89 $\uparrow$
covid (n=40) | 63.46 | 65.10 | 66.55 | 67.85 $\uparrow$ | 62.54 | 64.64 $\uparrow$ | 57.16 | 63.41 $\uparrow$ | 63.15 | 60.93 | 51.66 | 61.81 $\uparrow$ | 57.04 | 63.13 $\uparrow$ | 62.19 | 60.49 | 54.51 | 60.32 $\uparrow$
cutract (n=40) | 62.49 | 63.01 | 62.49 | 65.29 $\uparrow$ | 61.61 | 63.01 $\uparrow$ | 56.40 | 62.63 $\uparrow$ | 58.18 | 58.97 $\uparrow$ | 51.41 | 62.50 $\uparrow$ | 55.64 | 62.24 $\uparrow$ | 60.93 | 61.29 $\uparrow$ | 51.94 | 60.24 $\uparrow$
maggic (n=40) | 56.53 | 53.74 | 53.20 | 54.71 $\uparrow$ | 52.33 | 52.97 $\uparrow$ | 51.85 | 53.85 $\uparrow$ | 52.48 | 53.45 $\uparrow$ | 49.61 | 52.87 $\uparrow$ | 51.24 | 54.45 $\uparrow$ | 53.69 | 53.37 | 50.60 | 52.66 $\uparrow$
seer (n=40) | 78.86 | 72.40 | 76.02 | 77.13 $\uparrow$ | 75.18 | 78.26 $\uparrow$ | 68.31 | 73.89 $\uparrow$ | 77.94 | 73.24 | 51.45 | 72.32 $\uparrow$ | 67.78 | 75.95 $\uparrow$ | 70.32 | 72.81 $\uparrow$ | 68.79 | 74.48 $\uparrow$
compas (n=40) | 61.05 | 58.23 | 62.00 | 63.43 $\uparrow$ | 56.73 | 59.19 $\uparrow$ | 52.23 | 56.84 $\uparrow$ | 56.67 | 57.55 $\uparrow$ | 54.52 | 60.09 $\uparrow$ | 54.47 | 58.92 $\uparrow$ | 56.59 | 56.47 | 52.62 | 57.20 $\uparrow$
adult (n=40) | 74.84 | 69.75 | 49.47 | 68.90 $\uparrow$ | 47.85 | 66.79 $\uparrow$ | 60.55 | 70.14 $\uparrow$ | 64.06 | 70.56 $\uparrow$ | 58.77 | 71.63 $\uparrow$ | 60.26 | 71.50 $\uparrow$ | 69.43 | 70.58 $\uparrow$ | 59.76 | 72.41 $\uparrow$
drug (n=40) | 65.66 | 64.18 | 62.86 | 65.90 $\uparrow$ | 60.09 | 63.43 $\uparrow$ | 60.84 | 65.40 $\uparrow$ | 63.05 | 64.90 $\uparrow$ | 54.80 | 65.25 $\uparrow$ | 54.68 | 62.40 $\uparrow$ | 61.21 | 61.98 $\uparrow$ | 53.77 | 60.23 $\uparrow$
covid (n=100) | 63.46 | 63.17 | 67.14 | 68.60 $\uparrow$ | 62.40 | 66.02 $\uparrow$ | 61.34 | 64.63 $\uparrow$ | 63.33 | 66.42 $\uparrow$ | 55.86 | 64.53 $\uparrow$ | 55.44 | 64.91 $\uparrow$ | 61.96 | 63.40 $\uparrow$ | 59.75 | 64.47 $\uparrow$
cutract (n=100) | 62.49 | 62.69 | 60.96 | 65.60 $\uparrow$ | 62.12 | 65.95 $\uparrow$ | 59.06 | 66.20 $\uparrow$ | 63.38 | 65.72 $\uparrow$ | 52.62 | 61.73 $\uparrow$ | 58.11 | 65.08 $\uparrow$ | 60.15 | 61.34 $\uparrow$ | 58.82 | 63.76 $\uparrow$
maggic (n=100) | 56.53 | 54.83 | 53.58 | 55.70 $\uparrow$ | 50.72 | 53.98 $\uparrow$ | 52.29 | 55.06 $\uparrow$ | 53.93 | 54.61 $\uparrow$ | 49.71 | 53.34 $\uparrow$ | 51.82 | 54.53 $\uparrow$ | 54.11 | 54.34 $\uparrow$ | 51.31 | 53.33 $\uparrow$
seer (n=100) | 78.86 | 73.83 | 76.50 | 77.84 $\uparrow$ | 74.81 | 77.80 $\uparrow$ | 69.61 | 76.91 $\uparrow$ | 74.07 | 77.21 $\uparrow$ | 59.62 | 75.40 $\uparrow$ | 69.51 | 76.54 $\uparrow$ | 71.04 | 72.63 $\uparrow$ | 66.35 | 76.44 $\uparrow$
compas (n=100) | 61.05 | 56.61 | 63.04 | 63.39 $\uparrow$ | 55.57 | 57.40 $\uparrow$ | 55.07 | 58.95 $\uparrow$ | 53.38 | 55.69 $\uparrow$ | 54.61 | 59.31 $\uparrow$ | 54.38 | 57.92 $\uparrow$ | 56.69 | 57.01 $\uparrow$ | 54.34 | 56.72 $\uparrow$
adult (n=100) | 73.84 | 72.19 | 46.68 | 66.58 $\uparrow$ | 47.94 | 67.37 $\uparrow$ | 62.42 | 71.78 $\uparrow$ | 68.06 | 70.77 $\uparrow$ | 66.29 | 74.36 $\uparrow$ | 58.28 | 72.14 $\uparrow$ | 71.51 | 72.68 $\uparrow$ | 62.11 | 72.18 $\uparrow$
drug (n=100) | 65.66 | 64.20 | 66.18 | 67.41 $\uparrow$ | 60.77 | 61.21 $\uparrow$ | 57.67 | 63.41 $\uparrow$ | 63.26 | 64.55 $\uparrow$ | 54.89 | 65.72 $\uparrow$ | 52.06 | 63.38 $\uparrow$ | 61.21 | 62.30 $\uparrow$ | 55.40 | 63.32 $\uparrow$
covid (n=200) | 63.46 | 63.45 | 67.19 | 68.74 $\uparrow$ | 63.25 | 65.73 $\uparrow$ | 61.58 | 66.27 $\uparrow$ | 63.55 | 66.98 $\uparrow$ | 57.20 | 65.54 $\uparrow$ | 55.08 | 63.77 $\uparrow$ | 63.60 | 63.40 | 60.01 | 64.27 $\uparrow$
cutract (n=200) | 62.49 | 63.53 | 63.36 | 65.33 $\uparrow$ | 63.01 | 65.47 $\uparrow$ | 60.51 | 65.01 $\uparrow$ | 62.55 | 66.94 $\uparrow$ | 57.40 | 65.14 $\uparrow$ | 59.13 | 65.20 $\uparrow$ | 61.66 | 62.40 $\uparrow$ | 58.29 | 64.85 $\uparrow$
maggic (n=200) | 56.53 | 55.06 | 52.76 | 54.41 $\uparrow$ | 52.41 | 54.34 $\uparrow$ | 52.47 | 55.21 $\uparrow$ | 52.40 | 53.39 $\uparrow$ | 50.43 | 53.66 $\uparrow$ | 51.58 | 54.82 $\uparrow$ | 54.64 | 54.36 | 51.78 | 54.22 $\uparrow$
seer (n=200) | 78.86 | 74.51 | 75.20 | 77.26 $\uparrow$ | 75.33 | 77.41 $\uparrow$ | 70.96 | 77.08 $\uparrow$ | 74.51 | 79.20 $\uparrow$ | 71.44 | 78.66 $\uparrow$ | 65.84 | 76.60 $\uparrow$ | 73.11 | 74.77 $\uparrow$ | 66.37 | 76.36 $\uparrow$
compas (n=200) | 61.05 | 57.20 | 61.09 | 62.89 $\uparrow$ | 56.15 | 59.02 $\uparrow$ | 54.20 | 57.78 $\uparrow$ | 55.72 | 58.68 $\uparrow$ | 55.83 | 59.81 $\uparrow$ | 52.13 | 57.25 $\uparrow$ | 56.55 | 57.34 $\uparrow$ | 54.07 | 58.66 $\uparrow$
adult (n=200) | 73.84 | 72.26 | 44.41 | 67.99 $\uparrow$ | 48.46 | 67.24 $\uparrow$ | 66.56 | 74.12 $\uparrow$ | 71.61 | 74.89 $\uparrow$ | 69.91 | 76.52 $\uparrow$ | 60.70 | 74.05 $\uparrow$ | 73.11 | 74.08 $\uparrow$ | 63.10 | 73.82 $\uparrow$
drug (n=200) | 65.66 | 67.04 | 66.29 | 66.57 $\uparrow$ | 61.88 | 64.34 $\uparrow$ | 59.10 | 66.06 $\uparrow$ | 63.59 | 63.40 | 59.11 | 69.04 $\uparrow$ | 53.78 | 64.55 $\uparrow$ | 64.56 | 65.35 $\uparrow$ | 56.82 | 63.02 $\uparrow$

Table 12: AUC for the LR model on $D_{\mathrm{test}}$ where curation improves
performance for all methods across all sample sizes $n$, as indicated by
$\uparrow$.

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | 80.47 | 69.85 | 76.35 | 76.73 $\uparrow$ | 74.69 | 75.18 $\uparrow$ | 59.99 | 66.87 $\uparrow$ | 68.78 | 69.84 $\uparrow$ | 63.94 | 71.81 $\uparrow$ | 69.28 | 72.32 $\uparrow$ | 68.36 | 68.44 $\uparrow$ | 63.92 | 70.12 $\uparrow$
cutract (n=20) | 79.12 | 71.47 | 75.50 | 75.41 | 73.93 | 74.47 $\uparrow$ | 69.01 | 70.06 $\uparrow$ | 68.11 | 68.02 | 53.04 | 68.90 $\uparrow$ | 71.09 | 73.55 $\uparrow$ | 71.09 | 71.53 $\uparrow$ | 74.37 | 74.69 $\uparrow$
maggic (n=20) | 70.46 | 56.55 | 63.67 | 64.12 $\uparrow$ | 58.74 | 59.98 $\uparrow$ | 53.27 | 55.37 $\uparrow$ | 54.40 | 55.28 $\uparrow$ | 50.39 | 55.27 $\uparrow$ | 56.52 | 59.00 $\uparrow$ | 55.83 | 56.45 $\uparrow$ | 54.59 | 57.82 $\uparrow$
seer (n=20) | 90.91 | 85.96 | 88.63 | 88.74 $\uparrow$ | 87.90 | 88.11 $\uparrow$ | 78.22 | 83.69 $\uparrow$ | 83.17 | 84.74 $\uparrow$ | 51.07 | 77.84 $\uparrow$ | 84.86 | 84.75 | 85.72 | 86.11 $\uparrow$ | 80.33 | 82.63 $\uparrow$
compas (n=20) | 73.02 | 63.74 | 71.88 | 71.10 | 66.62 | 67.89 $\uparrow$ | 55.62 | 64.29 $\uparrow$ | 62.15 | 64.70 $\uparrow$ | 56.97 | 66.59 $\uparrow$ | 64.71 | 68.59 $\uparrow$ | 62.68 | 63.16 $\uparrow$ | 61.19 | 64.36 $\uparrow$
adult (n=20) | 88.37 | 79.24 | 45.28 | 71.07 $\uparrow$ | 50.89 | 75.77 $\uparrow$ | 80.03 | 81.27 $\uparrow$ | 73.19 | 74.10 $\uparrow$ | 68.84 | 79.66 $\uparrow$ | 75.65 | 80.74 $\uparrow$ | 74.61 | 75.04 $\uparrow$ | 76.82 | 78.84 $\uparrow$
drug (n=20) | 81.41 | 73.56 | 78.54 | 78.10 | 71.69 | 71.68 | 73.07 | 74.88 $\uparrow$ | 69.14 | 70.97 $\uparrow$ | 62.38 | 69.99 $\uparrow$ | 66.57 | 70.37 $\uparrow$ | 71.99 | 72.28 $\uparrow$ | 69.39 | 71.08 $\uparrow$
covid (n=40) | 80.47 | 71.20 | 76.17 | 76.91 $\uparrow$ | 75.32 | 76.13 $\uparrow$ | 64.56 | 68.82 $\uparrow$ | 71.08 | 72.77 $\uparrow$ | 61.40 | 71.21 $\uparrow$ | 70.74 | 73.84 $\uparrow$ | 68.85 | 69.65 $\uparrow$ | 64.13 | 69.58 $\uparrow$
cutract (n=40) | 79.12 | 71.90 | 74.05 | 74.82 $\uparrow$ | 72.69 | 72.89 $\uparrow$ | 68.43 | 70.97 $\uparrow$ | 70.67 | 72.39 $\uparrow$ | 61.50 | 74.10 $\uparrow$ | 67.98 | 71.55 $\uparrow$ | 70.26 | 70.12 | 67.87 | 70.88 $\uparrow$
maggic (n=40) | 70.46 | 59.65 | 61.89 | 63.39 $\uparrow$ | 60.41 | 61.84 $\uparrow$ | 55.81 | 56.64 $\uparrow$ | 56.58 | 58.08 $\uparrow$ | 48.68 | 58.82 $\uparrow$ | 57.50 | 60.90 $\uparrow$ | 57.98 | 58.00 $\uparrow$ | 56.61 | 59.42 $\uparrow$
seer (n=40) | 90.91 | 87.86 | 88.50 | 88.89 $\uparrow$ | 85.66 | 85.62 | 85.95 | 87.62 $\uparrow$ | 84.92 | 85.45 $\uparrow$ | 60.75 | 85.84 $\uparrow$ | 88.46 | 88.85 $\uparrow$ | 86.56 | 87.58 $\uparrow$ | 86.30 | 88.36 $\uparrow$
compas (n=40) | 73.02 | 65.50 | 70.96 | 70.93 | 65.70 | 66.72 $\uparrow$ | 60.24 | 63.78 $\uparrow$ | 59.91 | 62.49 $\uparrow$ | 65.17 | 68.19 $\uparrow$ | 62.63 | 67.00 $\uparrow$ | 63.70 | 64.18 $\uparrow$ | 58.63 | 64.17 $\uparrow$
adult (n=40) | 88.37 | 82.23 | 44.99 | 74.25 $\uparrow$ | 56.38 | 76.76 $\uparrow$ | 78.24 | 82.22 $\uparrow$ | 71.83 | 81.15 $\uparrow$ | 73.65 | 81.54 $\uparrow$ | 81.49 | 84.34 $\uparrow$ | 81.80 | 83.16 $\uparrow$ | 80.54 | 83.02 $\uparrow$
drug (n=40) | 81.41 | 71.74 | 78.98 | 78.88 | 71.54 | 73.21 $\uparrow$ | 71.34 | 74.44 $\uparrow$ | 72.12 | 73.68 $\uparrow$ | 70.00 | 76.45 $\uparrow$ | 66.91 | 73.78 $\uparrow$ | 69.09 | 69.75 $\uparrow$ | 68.21 | 70.94 $\uparrow$
covid (n=100) | 80.47 | 74.19 | 76.53 | 77.34 $\uparrow$ | 74.74 | 76.27 $\uparrow$ | 72.30 | 75.15 $\uparrow$ | 75.45 | 76.20 $\uparrow$ | 69.05 | 74.94 $\uparrow$ | 71.01 | 76.38 $\uparrow$ | 71.71 | 72.28 $\uparrow$ | 73.90 | 75.17 $\uparrow$
cutract (n=100) | 79.12 | 76.73 | 74.62 | 75.70 $\uparrow$ | 73.75 | 74.40 $\uparrow$ | 73.89 | 75.61 $\uparrow$ | 75.70 | 74.49 | 60.03 | 74.35 $\uparrow$ | 75.49 | 76.86 $\uparrow$ | 76.13 | 75.47 | 73.85 | 76.21 $\uparrow$
maggic (n=100) | 70.46 | 60.14 | 61.31 | 63.64 $\uparrow$ | 57.17 | 60.28 $\uparrow$ | 59.32 | 61.13 $\uparrow$ | 58.12 | 58.97 $\uparrow$ | 49.10 | 59.95 $\uparrow$ | 60.43 | 63.06 $\uparrow$ | 59.87 | 59.59 | 59.43 | 61.28 $\uparrow$
seer (n=100) | 90.91 | 88.92 | 88.16 | 88.69 $\uparrow$ | 88.09 | 88.25 $\uparrow$ | 88.45 | 89.11 $\uparrow$ | 87.53 | 88.25 $\uparrow$ | 83.13 | 88.44 $\uparrow$ | 88.46 | 89.27 $\uparrow$ | 87.95 | 87.65 | 86.81 | 88.67 $\uparrow$
compas (n=100) | 73.02 | 67.75 | 71.21 | 71.34 $\uparrow$ | 65.61 | 67.67 $\uparrow$ | 64.14 | 66.72 $\uparrow$ | 61.55 | 65.03 $\uparrow$ | 67.43 | 68.88 $\uparrow$ | 65.84 | 68.39 $\uparrow$ | 66.39 | 66.53 $\uparrow$ | 65.61 | 67.77 $\uparrow$
adult (n=100) | 88.37 | 85.59 | 46.71 | 78.68 $\uparrow$ | 48.64 | 75.94 $\uparrow$ | 82.30 | 84.48 $\uparrow$ | 77.41 | 78.98 $\uparrow$ | 83.99 | 84.99 $\uparrow$ | 82.67 | 85.77 $\uparrow$ | 85.63 | 85.43 | 81.32 | 83.97 $\uparrow$
drug (n=100) | 81.41 | 74.34 | 80.35 | 80.01 | 70.00 | 71.76 $\uparrow$ | 71.66 | 75.80 $\uparrow$ | 71.26 | 72.40 $\uparrow$ | 74.57 | 78.65 $\uparrow$ | 69.34 | 77.50 $\uparrow$ | 71.04 | 71.75 $\uparrow$ | 71.91 | 75.38 $\uparrow$
covid (n=200) | 80.47 | 76.73 | 76.06 | 77.25 $\uparrow$ | 75.18 | 77.09 $\uparrow$ | 76.11 | 77.54 $\uparrow$ | 77.51 | 78.09 $\uparrow$ | 71.92 | 76.62 $\uparrow$ | 74.06 | 77.53 $\uparrow$ | 74.94 | 74.94 | 72.75 | 76.00 $\uparrow$
cutract (n=200) | 79.12 | 77.53 | 75.28 | 76.30 $\uparrow$ | 74.36 | 75.38 $\uparrow$ | 75.25 | 75.88 $\uparrow$ | 77.66 | 77.66 | 75.37 | 76.69 $\uparrow$ | 76.45 | 77.50 $\uparrow$ | 76.77 | 76.59 | 72.94 | 74.86 $\uparrow$
maggic (n=200) | 70.46 | 63.32 | 61.89 | 64.30 $\uparrow$ | 59.10 | 62.10 $\uparrow$ | 61.57 | 63.98 $\uparrow$ | 56.77 | 57.35 $\uparrow$ | 51.05 | 63.17 $\uparrow$ | 59.17 | 63.84 $\uparrow$ | 62.86 | 62.86 | 60.94 | 63.47 $\uparrow$
seer (n=200) | 90.91 | 90.01 | 88.65 | 89.20 $\uparrow$ | 87.57 | 88.14 $\uparrow$ | 88.20 | 88.80 $\uparrow$ | 89.69 | 89.81 $\uparrow$ | 89.10 | 89.68 $\uparrow$ | 87.35 | 89.54 $\uparrow$ | 89.13 | 89.14 $\uparrow$ | 88.74 | 89.65 $\uparrow$
compas (n=200) | 73.02 | 69.14 | 70.53 | 71.36 $\uparrow$ | 63.49 | 66.50 $\uparrow$ | 66.26 | 68.73 $\uparrow$ | 64.83 | 66.40 $\uparrow$ | 67.85 | 69.93 $\uparrow$ | 60.78 | 68.07 $\uparrow$ | 68.18 | 68.18 | 66.26 | 68.99 $\uparrow$
adult (n=200) | 88.37 | 86.94 | 38.89 | 80.35 $\uparrow$ | 56.97 | 79.07 $\uparrow$ | 85.25 | 86.29 $\uparrow$ | 86.81 | 86.63 | 85.70 | 85.94 $\uparrow$ | 84.18 | 86.65 $\uparrow$ | 86.88 | 86.70 | 84.09 | 85.91 $\uparrow$
drug (n=200) | 81.41 | 77.47 | 79.06 | 79.61 $\uparrow$ | 71.56 | 72.54 $\uparrow$ | 76.15 | 78.84 $\uparrow$ | 69.00 | 73.14 $\uparrow$ | 77.51 | 79.72 $\uparrow$ | 75.47 | 80.07 $\uparrow$ | 75.82 | 75.69 | 71.76 | 76.95 $\uparrow$

####  C.6 Full results for performance evaluation

We report full results with standard deviation for the results from the main
paper. The performance is AUC averaged over XGBoost, Random forest, Logistic
regression, Decision tree.

Table 13: AUC averaged over 4 downstream models on $D_{\mathrm{test}}$ where
curation improves performance for all methods across all sample sizes $n$, as
indicated by $\uparrow$. CLLM w/ GPT-4 (Curated) dataset provides the
strongest performance for both private/proprietary datasets and public
datasets

| Real data |  CLLM (OURS) | Baselines
---|---|---|---
|  | GPT-4 | GPT-3.5 | CTGAN | TabDDPM | GReaT | NFLOW | SMOTE | TVAE
Dataset | $D_{\mathrm{oracle}}$ | $D_{\mathrm{train}}$ | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur. | Uncur. | Cur.
covid (n=20) | $74.41_{(0.11)}$ | $68.50_{(1.57)}$ | $73.78_{(0.31)}$ | $\textbf{73.87}_{(0.50)}$ | $69.85_{(0.75)}$ | $71.41_{(0.92)}$ | $59.00_{(2.25)}$ | $63.67_{(2.51)}$ | $66.84_{(1.66)}$ | $66.85_{(1.56)}$ | $57.38_{(1.47)}$ | $66.46_{(0.80)}$ | $62.87_{(0.98)}$ | $68.56_{(1.07)}$ | $66.95_{(1.66)}$ | $66.82_{(1.89)}$ | $61.69_{(2.72)}$ | $66.11_{(2.79)}$
cutract (n=20) | $72.23_{(0.65)}$ | $70.12_{(1.16)}$ | $71.15_{(0.46)}$ | $\textbf{72.50}_{(0.76)}$ | $69.97_{(1.13)}$ | $71.54_{(1.38)}$ | $64.01_{(2.26)}$ | $67.98_{(1.63)}$ | $66.05_{(1.14)}$ | $66.59_{(1.49)}$ | $52.38_{(1.78)}$ | $67.02_{(0.97)}$ | $64.44_{(1.05)}$ | $70.42_{(1.25)}$ | $68.41_{(1.34)}$ | $69.24_{(1.25)}$ | $68.94_{(1.38)}$ | $70.22_{(1.12)}$
maggic (n=20) | $67.41_{(0.06)}$ | $57.13_{(0.85)}$ | $60.70_{(0.38)}$ | $\textbf{61.48}_{(0.49)}$ | $57.54_{(0.83)}$ | $58.69_{(0.76)}$ | $52.75_{(1.34)}$ | $54.51_{(1.40)}$ | $54.59_{(1.02)}$ | $55.39_{(1.07)}$ | $50.29_{(0.52)}$ | $55.64_{(0.43)}$ | $54.72_{(1.38)}$ | $57.38_{(1.06)}$ | $55.84_{(1.21)}$ | $56.15_{(1.20)}$ | $54.08_{(1.05)}$ | $56.19_{(0.76)}$
seer (n=20) | $87.92_{(0.08)}$ | $80.67_{(1.67)}$ | $84.53_{(0.33)}$ | $\textbf{84.82}_{(0.47)}$ | $83.34_{(0.95)}$ | $83.71_{(0.52)}$ | $74.34_{(4.11)}$ | $78.73_{(3.13)}$ | $80.59_{(1.32)}$ | $80.60_{(1.32)}$ | $47.57_{(2.51)}$ | $74.43_{(3.36)}$ | $76.06_{(1.68)}$ | $79.98_{(1.65)}$ | $79.23_{(2.17)}$ | $80.02_{(2.08)}$ | $74.53_{(1.37)}$ | $78.73_{(1.24)}$
compas (n=20) | $67.51_{(0.03)}$ | $63.11_{(0.96)}$ | $\textbf{68.01}_{(0.44)}$ | $67.91_{(0.55)}$ | $62.07_{(1.51)}$ | $64.43_{(1.38)}$ | $55.67_{(1.60)}$ | $62.56_{(1.61)}$ | $57.67_{(1.90)}$ | $60.87_{(1.33)}$ | $53.33_{(1.85)}$ | $63.59_{(1.30)}$ | $59.49_{(1.72)}$ | $64.62_{(1.14)}$ | $61.06_{(1.92)}$ | $61.59_{(2.15)}$ | $58.30_{(1.77)}$ | $62.58_{(1.87)}$
adult (n=20) | $84.17_{(0.10)}$ | $77.45_{(1.25)}$ | $50.39_{(3.99)}$ | $71.48_{(2.34)}$ | $49.23_{(2.57)}$ | $72.37_{(2.26)}$ | $72.23_{(1.26)}$ | $76.86_{(1.25)}$ | $74.35_{(1.48)}$ | $75.04_{(1.82)}$ | $67.00_{(4.72)}$ | $77.25_{(1.49)}$ | $67.46_{(3.76)}$ | $76.48_{(1.77)}$ | $73.75_{(1.64)}$ | $73.67_{(1.46)}$ | $73.20_{(1.51)}$ | $76.90_{(1.64)}$
drug (n=20) | $77.81_{(0.55)}$ | $70.84_{(2.25)}$ | $75.08_{(1.17)}$ | $\textbf{75.29}_{(1.11)}$ | $71.68_{(2.25)}$ | $72.14_{(2.64)}$ | $68.31_{(2.81)}$ | $72.65_{(2.00)}$ | $68.12_{(2.38)}$ | $69.68_{(2.41)}$ | $58.78_{(4.26)}$ | $68.89_{(3.57)}$ | $62.13_{(4.94)}$ | $67.75_{(3.72)}$ | $70.16_{(1.87)}$ | $70.16_{(1.75)}$ | $66.60_{(2.95)}$ | $69.18_{(2.59)}$
covid (n=40) | $74.41_{(0.11)}$ | $70.77_{(0.96)}$ | $73.40_{(0.61)}$ | $\textbf{73.95}_{(0.67)}$ | $70.42_{(0.92)}$ | $71.93_{(0.60)}$ | $63.63_{(1.15)}$ | $68.46_{(0.93)}$ | $70.50_{(1.49)}$ | $70.44_{(1.37)}$ | $56.50_{(0.83)}$ | $68.68_{(1.39)}$ | $66.41_{(2.51)}$ | $70.48_{(1.48)}$ | $68.66_{(1.48)}$ | $68.44_{(1.28)}$ | $61.03_{(0.96)}$ | $67.35_{(0.60)}$
cutract (n=40) | $72.23_{(0.65)}$ | $69.18_{(0.65)}$ | $69.87_{(0.62)}$ | $\textbf{71.72}_{(0.46)}$ | $68.47_{(0.59)}$ | $69.56_{(0.50)}$ | $63.01_{(2.33)}$ | $67.87_{(1.36)}$ | $65.63_{(2.71)}$ | $67.27_{(2.38)}$ | $54.39_{(1.59)}$ | $68.44_{(0.41)}$ | $61.40_{(1.95)}$ | $67.98_{(1.06)}$ | $67.86_{(0.72)}$ | $67.95_{(0.59)}$ | $59.79_{(1.85)}$ | $66.62_{(1.06)}$